Description / Background
We've had a nightly system test failure during teardown of the main system test instance in the slow3 test suite.
The query results bucket failed to delete because it contained data. This should not be possible, because the lambda functions that publish this data should be deleted before the custom resource deletes all the data in the bucket.
Steps to reproduce
- Deploy an instance with QueryStack, and some data in a table.
- Submit many queries via SQS that will publish some data to S3. Ensure enough are submitted that they will still be processing during teardown.
- Delete the instance's CloudFormation stack while the queries are still processing.
- See error.
Expected behaviour
The instance should tear down cleanly.
The lambdas that output query results should be deleted before the data in the query results bucket is emptied.
Technical Notes / Implementation Details
The lambdas and results bucket are deployed in QueryStack. The lambda that outputs to the bucket is QueryLeafPartitionExecutorLambda. The data in the results bucket is deleted in a custom resource that's created by SleeperCoreStacks.addAutoDeleteS3Objects.
We can declare a dependency between QueryLeafPartitionExecutorLambda and the CDK custom resource that autodeletes the data.
We don't currently have a reference to the custom resource object in QueryStack. We can return that from SleeperCoreStacks.addAutoDeleteS3Objects and AutoDeleteS3ObjectsStack.addAutoDeleteS3Objects, so that we can declare the dependency.
Description / Background
We've had a nightly system test failure during teardown of the main system test instance in the slow3 test suite.
The query results bucket failed to delete because it contained data. This should not be possible, because the lambda functions that publish this data should be deleted before the custom resource deletes all the data in the bucket.
Steps to reproduce
Expected behaviour
The instance should tear down cleanly.
The lambdas that output query results should be deleted before the data in the query results bucket is emptied.
Technical Notes / Implementation Details
The lambdas and results bucket are deployed in QueryStack. The lambda that outputs to the bucket is QueryLeafPartitionExecutorLambda. The data in the results bucket is deleted in a custom resource that's created by SleeperCoreStacks.addAutoDeleteS3Objects.
We can declare a dependency between QueryLeafPartitionExecutorLambda and the CDK custom resource that autodeletes the data.
We don't currently have a reference to the custom resource object in QueryStack. We can return that from SleeperCoreStacks.addAutoDeleteS3Objects and AutoDeleteS3ObjectsStack.addAutoDeleteS3Objects, so that we can declare the dependency.