Move S3 operator template validation out of __init__ (#70296)#70359
Move S3 operator template validation out of __init__ (#70296)#70359AmoghAtreya wants to merge 3 commits into
Conversation
|
Sorry, the issue has been linked to the PR and should be unlinked to prevent the issue from closing. My apologies. |
No worries, it's a matter of changing the "resolves:" into "related:" :) |
|
Thanks for updating the title. Will commit the code to fix the failure in the static checks after they all finish running. The Tests (AMD) check was failing due to an incorrect number in the code. |
42f5c65 to
08f503f
Compare
|
@shahar1 please let me know if there's anything you want me to change in this PR. I fixed all the static bugs and checks from a few days ago. |
shahar1
left a comment
There was a problem hiding this comment.
Could you please focus the PR on a single provider? (I know that GCP compute operators are being handled in another PR, so maybe just focus on S3DeleteObjectsOperator)
A reviewer asked to keep this PR scoped to a single provider, and the Compute Engine operators are being addressed in a separate PR. This restores ComputeEngineDeleteInstanceOperator (and its exemption entry) to their original state, leaving only the S3DeleteObjectsOperator fix. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Reverted changes! This PR is only focused on S3 now. @shahar1 |
|
Checks just passed. Please let me know what you think! |
| if not exactly_one(keys is None, all(var is None for var in [prefix, from_datetime, to_datetime])): | ||
| raise AirflowException( | ||
| "Either keys or at least one of prefix, from_datetime, to_datetime should be set." | ||
| ) | ||
|
|
There was a problem hiding this comment.
#70505 narrows the rule: field is None checks read provision, not the un-rendered value, so they belong in init. This one qualifies.
Please keep both copies, though. keys = self.keys or self.hook.list_keys(prefix=..., ...) lists the whole bucket when every filter is None — reachable past a correct constructor check, since a templated keys can render to None under render_template_as_native_obj=True. The execute() guard catches that; the __init__ guard keeps the plain authoring mistake a parse-time error.
The exemption entry can still go — the hook just doesn't see comparisons inside a comprehension:
by_scan = prefix is not None or from_datetime is not None or to_datetime is not None
if not exactly_one(keys is not None, by_scan):
raise ValueError("Either keys or at least one of prefix, from_datetime, to_datetime should be set.")Identical on every input including keys=[]. Context in #70296's "False positives" section.
Drafted-by: Claude Code (Opus 5); reviewed by @shahar1 before posting
Description
related: #70296 for
S3DeleteObjectsOperatorandComputeEngineDeleteInstanceOperator.Changes Made:
S3DeleteObjectsOperator(amazon):keys,prefix,from_datetime,to_datetime) from__init__intoexecute().tests/providers/amazon/aws/operators/test_s3.pyto assert parameter checks occur during execution rather than constructor time.ComputeEngineDeleteInstanceOperator(google):GcpBodyFieldValidatorinstantiation out of__init__and intoexecute()soapi_versionis evaluated after Jinja rendering.tests/providers/google/cloud/operators/test_compute.pyverifying validator instantiation occurs insideexecute().CI Exemptions:
path::ClassNameentries fromscripts/ci/prek/validate_operators_init_exemptions.txt.works on: #70296
Was generative AI tooling used to co-author this PR?
Generated-by: Claude following the guidelines