Skip to content

7782: Tear down can fail due to data landing in query results bucket after autodelete #7830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Rob9786 wants to merge 3 commits into
base: develop
Choose a base branch
from
+15 −6
Draft

7782: Tear down can fail due to data landing in query results bucket after autodelete #7830

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5eec90b
7782: Add dependency to lambda
Rob9786 Aug 3, 2026
e377183
7782: Added null check wrapper around add dependency
Rob9786 Aug 3, 2026
b1f70e7
Merge branch 'develop' into 7782-tear-down-can-fail-due-to-data-landi…
Rob9786 Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions java/deployment/cdk/src/main/java/sleeper/cdk/stack/SleeperCoreStacks.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import software.amazon.awscdk.CustomResource;
import software.amazon.awscdk.Stack;
import software.amazon.awscdk.services.cloudwatch.IMetric;
import software.amazon.awscdk.services.ec2.ISubnet;
Expand Down Expand Up @@ -240,8 +241,8 @@ public void grantReadTablesStatus(IGrantable grantee) {
tableIndexStack.grantRead(grantee);
}

public void addAutoDeleteS3Objects(Construct scope, IBucket bucket) {
autoDeleteS3Stack.addAutoDeleteS3Objects(scope, bucket);
public CustomResource addAutoDeleteS3Objects(Construct scope, IBucket bucket) {
return autoDeleteS3Stack.addAutoDeleteS3Objects(scope, bucket);
}

public void addAutoStopEcsClusterTasksAfterTaskCreatorIsDeleted(Construct scope, ICluster cluster, IFunction taskCreator) {
Expand Down
9 changes: 6 additions & 3 deletions java/deployment/cdk/src/main/java/sleeper/cdk/stack/core/AutoDeleteS3ObjectsStack.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,11 @@ private void createLambda(InstanceProperties instanceProperties, SleeperInstance
/**
* Adds a custom resource to delete a bucket's contents.
*
* @param scope the stack to add the custom resource to
* @param bucket the bucket to delete from
* @param scope the stack to add the custom resource to
* @param bucket the bucket to delete from
* @return a custom resource
*/
public void addAutoDeleteS3Objects(Construct scope, IBucket bucket) {
public CustomResource addAutoDeleteS3Objects(Construct scope, IBucket bucket) {

String id = bucket.getNode().getId() + "-AutoDelete";

Expand All @@ -105,6 +106,8 @@ public void addAutoDeleteS3Objects(Construct scope, IBucket bucket) {
.build();

customResource.getNode().addDependency(bucket);

return customResource;
}

}
7 changes: 6 additions & 1 deletion java/deployment/cdk/src/main/java/sleeper/cdk/stack/query/QueryStack.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import software.amazon.awscdk.CfnOutput;
import software.amazon.awscdk.CfnOutputProps;
import software.amazon.awscdk.CustomResource;
import software.amazon.awscdk.Duration;
import software.amazon.awscdk.NestedStack;
import software.amazon.awscdk.RemovalPolicy;
Expand Down Expand Up @@ -83,6 +84,7 @@ public class QueryStack extends NestedStack {

private IFunction queryExecutorLambda;
private IFunction leafPartitionQueryLambda;
private CustomResource customResource;

@patchwork01 patchwork01 Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to be named after its purpose rather than its type, since it's not clear from context what the custom resource would be for. It's for auto-delete for the results bucket.


public QueryStack(Construct scope,
String id,
Expand Down Expand Up @@ -185,6 +187,9 @@ private IFunction setupLeafPartitionQueryQueueAndLambda(
.batchSize(1)
.build();

if (customResource != null) {
lambda.getNode().addDependency(customResource);
}
lambda.addEventSource(new SqsEventSource(leafPartitionQueryQueue, eventSourceProps));

return lambda;
Expand Down Expand Up @@ -306,7 +311,7 @@ private IBucket setupResultsBucket(InstanceProperties instanceProperties, Sleepe
instanceProperties.set(CdkDefinedInstanceProperty.QUERY_RESULTS_BUCKET, resultsBucket.getBucketName());

if (removalPolicy == RemovalPolicy.DESTROY) {
coreStacks.addAutoDeleteS3Objects(this, resultsBucket);
customResource = coreStacks.addAutoDeleteS3Objects(this, resultsBucket);
}

return resultsBucket;
Expand Down
Loading