Support Cross-Account S3 Replication in Data Lake Modules

[nitincs10]

Problem Statement

I am looking for a supported way to configure S3 bucket replication between multiple AWS accounts using the MDAA data lake modules.

Currently, the S3 data lake modules do not appear to support cross-account replication out of the box for two primary reasons:

Replication permissions are not included in the default bucket policy configuration.
Cross-account IAM role references are not allowed/supported by the current implementation.
Current Workaround

To enable replication, I implemented a temporary workaround by:

Extending the data lake bucket permissions to include replication-related actions.
Referencing the destination account replication role using the unique role ID from the target AWS account.

This approach works, but it feels like a quick-and-dirty solution and may not be the intended pattern.

Relevant Code

MDAA Repository:

https://github.com/aws/modern-data-architecture-accelerator/blob/main/packages/utilities/s3-bucketpolicy-helper/lib/index.ts

Current section:

static readonly READ_WRITE_ACTIONS = [
...RestrictObjectPrefixToRoles.READ_ACTIONS,
's3:PutObject',
's3:PutObjectTagging',
's3:DeleteObject',
*```
's3:Replicate',
's3:ObjectOwnerOverrideToBucketOwner',

];


Questions
- Is there a recommended or supported approach for configuring cross-account S3 replication using MDAA?
- Are there plans to add native support for replication permissions and cross-account replication roles?
- Is there a cleaner implementation pattern that avoids modifying the default bucket policy helper?


Expected Outcome

It would be useful if the framework could natively support:

S3 replication permissions in bucket policy generation.
Cross-account replication IAM role references.

Thank you for any guidance or recommendations.

[guonengamazon]

Thank you, @nitincs10, for the suggestion. We are reviewing this request and will let you know once we prioritize the request. Will give you an update soon.

[guonengamazon]

Hi @nitincs10, thanks for your patience. We'd like to add this natively so you don't have to modify s3-bucketpolicy-helper or hardcode role IDs. Cross-account replication has two sides, and a given MDAA-managed bucket is usually only one of them. Before we build, could you confirm which side your MDAA bucket is on? It would help to know whether MDAA manages the bucket that sends objects, the one that receives them, or both.

The config lives in your data lake module's config YAML (the same file with your roles, accessPolicies, and buckets). We're proposing an optional, default-off replication: block on a bucket. Nothing changes for configs that don't use it.

If your MDAA bucket RECEIVES replicated objects from another account (destination):

buckets:
  curated:
    accessPolicies:
      - Root
    replication:
      inbound:
        # The replication role in the SOURCE account, by ARN or unique ID.
        # No in-account name resolution, so a cross-account role works directly.
        sourceReplicationRoleArn: arn:aws:iam::111122223333:role/datalake-replication
        prefixFilters:
          - data

Under the hood this adds a bucket-policy statement granting that role s3:Replicate* and s3:ObjectOwnerOverrideToBucketOwner on the configured prefixes, plus an encrypt grant on the bucket's KMS key. This is the native replacement for the two things you patched (the READ_WRITE_ACTIONS edit and the raw role-ID reference).

If your MDAA bucket SENDS objects to a bucket in another account (source):

buckets:
  raw:
    accessPolicies:
      - Root
    replication:
      outbound:
        destinationBucketArn: arn:aws:s3:::dr-datalake-raw-us-west-2
        destinationAccount: '111122223333'
        destinationRegion: us-west-2
        destinationKmsKeyArn: arn:aws:kms:us-west-2:111122223333:key/abcd-1234  # optional
        prefixFilters:
          - data

Under the hood this creates the ReplicationConfiguration on the bucket and an MDAA-managed replication role (scoped to those prefixes and the destination bucket, with decrypt on the source key and encrypt on the destination key).

Could you let us know which of these matches your setup, or if you need both? And if the field names read awkwardly for your use case, we're happy to adjust before implementing.

If the proposal above doesn't align with what you need, please show with yaml examples and expected outcome, similar to how I set it up above. Thank you!

[nitincs10]

Thanks for looking into it. We have following use cases:

• MDAA manages both source and target bucket
• MDAA manages source bucket only- MDAA managed buckets want to replicate the files from s3 bucket to other non MDAA manages s3 buckets.
• MDAA manages target bucket only. (some non MDAA managed buckets outside our control wants to replicate data to us)

Would you also be able to add a functionality to create replication rules ?

[guonengamazon]

Hi @nitincs10 ,

Thanks for the feedback. We will make replication rules as part of the functionality. We will keep you informed of the release schedule.

Thanks!