Set retention on bulk import job S3 file instead of deleting

[patchwork01]

Description / Background

In the future we want to add retries of bulk import jobs that have failed in Spark. Currently a bulk import job is submitted to SQS, which invokes a lambda that writes the job to S3 and submits it to Spark. When the Spark driver starts it reads the job from S3, and deletes the file from S3.

We'd like to give ourselves the option to retry the job from the same file after the point where it's currently deleted.

Ideally this should also give us some diagnostic capability by keeping the definitions of all the bulk import jobs that have been submitted or finished recently.

Technical Notes / Implementation Details

There are options for this:

• Start the file in a "pending" folder, then move it to a "completed" folder when the job finishes, and set retention in S3 then
• Leave the file where it is, but stop deleting it and just set retention in S3 instead

The second option seems simpler, so we'll go with that.

See the S3 documentation for expiring objects:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html

Code changes

The file is written to S3 by BulkImportExecutor, with BulkImportJobWriterToS3. It's then read with BulkImportJobLoaderFromS3 from BulkImportJobDriver, which currently also deletes the file.

Both BulkImportJobWriterToS3 and BulkImportJobLoaderFromS3 are tested in BulkImportJobLoaderFromS3IT.

It doesn't look like BulkImportExecutor or BulkImportJobDriver will need to change.

We can:

• Add an instance and table property for how long to keep bulk import jobs in S3, and default it to a month.
• Update BulkImportJobWriterToS3 to configure retention in S3.
• Update BulkImportJobLoaderFromS3 to stop it deleting the file.

Dependencies / Blockers

Conflicts with:

• Pass bulk import job S3 path to Spark explicitly #7607