Skip to content

Move S3 operator template validation out of __init__ (#70296)#70359

Open
AmoghAtreya wants to merge 3 commits into
apache:mainfrom
AmoghAtreya:fix/70296-operators-init-s3-and-gce
Open

Move S3 operator template validation out of __init__ (#70296)#70359
AmoghAtreya wants to merge 3 commits into
apache:mainfrom
AmoghAtreya:fix/70296-operators-init-s3-and-gce

Conversation

@AmoghAtreya

@AmoghAtreya AmoghAtreya commented Jul 24, 2026

Copy link
Copy Markdown

Description

related: #70296 for S3DeleteObjectsOperator and ComputeEngineDeleteInstanceOperator.

Changes Made:

  1. S3DeleteObjectsOperator (amazon):

    • Deferred parameter validation on template fields (keys, prefix, from_datetime, to_datetime) from __init__ into execute().
    • Updated unit tests in tests/providers/amazon/aws/operators/test_s3.py to assert parameter checks occur during execution rather than constructor time.
  2. ComputeEngineDeleteInstanceOperator (google):

    • Deferred GcpBodyFieldValidator instantiation out of __init__ and into execute() so api_version is evaluated after Jinja rendering.
    • Added unit tests in tests/providers/google/cloud/operators/test_compute.py verifying validator instantiation occurs inside execute().
  3. CI Exemptions:

    • Removed both path::ClassName entries from scripts/ci/prek/validate_operators_init_exemptions.txt.

works on: #70296


Was generative AI tooling used to co-author this PR?
  • Yes (Claude Sonnet)

Generated-by: Claude following the guidelines


@AmoghAtreya

Copy link
Copy Markdown
Author

Sorry, the issue has been linked to the PR and should be unlinked to prevent the issue from closing. My apologies.

@shahar1 shahar1 removed the backport-to-v3-3-test Backport to v3-3-test label Jul 24, 2026
@shahar1 shahar1 changed the title fix(providers): move S3 and GCE operator template validation out of __init__ (#70296) Move S3 and GCE operator template validation out of __init__ (#70296) Jul 24, 2026
@shahar1

shahar1 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Sorry, the issue has been linked to the PR and should be unlinked to prevent the issue from closing. My apologies.

No worries, it's a matter of changing the "resolves:" into "related:" :)
Also, I've modfied the title - we don't use conventional commits.
Please note that you have a failure in the static checks.

@AmoghAtreya

Copy link
Copy Markdown
Author

Thanks for updating the title. Will commit the code to fix the failure in the static checks after they all finish running. The Tests (AMD) check was failing due to an incorrect number in the code.

@AmoghAtreya
AmoghAtreya force-pushed the fix/70296-operators-init-s3-and-gce branch from 42f5c65 to 08f503f Compare July 24, 2026 08:20
@AmoghAtreya

Copy link
Copy Markdown
Author

@shahar1 please let me know if there's anything you want me to change in this PR. I fixed all the static bugs and checks from a few days ago.

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please focus the PR on a single provider? (I know that GCP compute operators are being handled in another PR, so maybe just focus on S3DeleteObjectsOperator)

A reviewer asked to keep this PR scoped to a single provider, and the
Compute Engine operators are being addressed in a separate PR. This
restores ComputeEngineDeleteInstanceOperator (and its exemption entry)
to their original state, leaving only the S3DeleteObjectsOperator fix.

Co-authored-by: Cursor <cursoragent@cursor.com>
@AmoghAtreya

AmoghAtreya commented Jul 27, 2026

Copy link
Copy Markdown
Author

Reverted changes! This PR is only focused on S3 now. @shahar1

@AmoghAtreya

Copy link
Copy Markdown
Author

Checks just passed. Please let me know what you think!

@AmoghAtreya AmoghAtreya changed the title Move S3 and GCE operator template validation out of __init__ (#70296) Move S3 operator template validation out of __init__ (#70296) Jul 27, 2026
Comment on lines -700 to -704
if not exactly_one(keys is None, all(var is None for var in [prefix, from_datetime, to_datetime])):
raise AirflowException(
"Either keys or at least one of prefix, from_datetime, to_datetime should be set."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#70505 narrows the rule: field is None checks read provision, not the un-rendered value, so they belong in init. This one qualifies.

Please keep both copies, though. keys = self.keys or self.hook.list_keys(prefix=..., ...) lists the whole bucket when every filter is None — reachable past a correct constructor check, since a templated keys can render to None under render_template_as_native_obj=True. The execute() guard catches that; the __init__ guard keeps the plain authoring mistake a parse-time error.

The exemption entry can still go — the hook just doesn't see comparisons inside a comprehension:

by_scan = prefix is not None or from_datetime is not None or to_datetime is not None
if not exactly_one(keys is not None, by_scan):
    raise ValueError("Either keys or at least one of prefix, from_datetime, to_datetime should be set.")

Identical on every input including keys=[]. Context in #70296's "False positives" section.

Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just committed the change for that. Updated 'S3DeleteObjectsOperator.init' to use explicit 'is not None' checks while retaining execution-time guards as requested by you @shahar1. This change has been committed and is dependent on #70505 being merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers provider:amazon AWS/Amazon - related issues provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants