[BUG] SMUS Tooling KMS key CloudWatch Logs condition blocks Bedrock Lambda log group encryption

[marcinc]

Summary

When a SageMaker Unified Studio project environment that includes AmazonBedrockKnowledgeBase (and likely other Bedrock sub-blueprints) is deployed, it fails immediately with a KMS 400 error from CloudWatch Logs. The root cause is a misconfigured condition operator on the tooling KMS key's CloudWatchLogsEncryption policy statement - ArnEquals is used where ArnLike is required - combined with log group prefixes that do not cover Bedrock Lambda log groups.

Affected version

@aws-mdaa/datazone-l3-construct v1.5.0

Component

packages/constructs/L3/governance/datazone-l3-construct/lib/private/sagemaker-domain-helper.ts
Function: createToolingResources() — CloudWatchLogsEncryption policy statement on the tooling KMS key

Error message

Project environment AmazonBedrockKnowledgeBase (<env-id>) failed to deploy in the account <account> and region <region> due to the failure: The specified KMS key does not exist or is not allowed to be used with Arn 
'arn:aws:logs:<region>:<account>:log-group:/aws/lambda/amazon-bedrock-ide-kbIngestion--<env-id>'
    CloudFormation stack ID: DataZone-Env-<env-id>
    Service: CloudWatchLogs
    Status code: 400

Project environment AmazonBedrockKnowledgeBase (<env-id>) failed to deploy in the account <account> and region <region> due to the failure: The specified KMS key does not exist or is not allowed to be used with Arn
'arn:aws:logs:<region>:<account>:log-group:/aws/lambda/amazon-bedrock-ide-opensearchIndex--<env-id>'
    CloudFormation stack ID: DataZone-Env-<env-id>
    Service: CloudWatchLogs
    Status code: 400

Root cause

The AmazonBedrockKnowledgeBase blueprint receives the tooling key ARN (KmsKeyArn) in its environment blueprint configuration regional parameters. DataZone passes this ARN into the CloudFormation stack it deploys per project environment. CloudFormation instructs CloudWatch Logs to create two encrypted log groups using that key:

• /aws/lambda/amazon-bedrock-ide-kbIngestion--<env-id>
• /aws/lambda/amazon-bedrock-ide-opensearchIndex--<env-id>

The CloudWatchLogsEncryption policy statement on the tooling key was:

"Condition": {
  "ArnEquals": {
    "kms:EncryptionContext:aws:logs:arn": [
      "arn:aws:logs:<region>:<account>:log-group:datazone-*",
      "arn:aws:logs:<region>:<account>:log-group:airflow-*"
    ]
  }
}

Two compounding problems:

1. Wrong operator: ArnEquals performs a literal string comparison - wildcards (*) are treated as literal characters, not globs. Neither entry ever matched a real log group. ArnLike is required to enable wildcard matching.

2. Wrong prefixes: Even with ArnLike, the patterns log-group:datazone-* and log-group:airflow-* do not match Bedrock Lambda log groups, which are prefixed /aws/lambda/amazon-bedrock-ide-*.

Additionally, kms:DescribeKey was absent from the allowed actions on the same statement. CloudWatch Logs requires this action to verify the key before using it for log group encryption.

Proposed Fix

Two changes in createToolingResources() in sagemaker-domain-helper.ts:

1. Switch to ArnLike and add the Bedrock Lambda log group prefix

-    conditions: {
-      ArnEquals: {
-        'kms:EncryptionContext:aws:logs:arn': [
-          `arn:${this.props.partition}:logs:${region}:${account}:log-group:datazone-*`,
-          `arn:${this.props.partition}:logs:${region}:${account}:log-group:airflow-*`,
-        ],
-      },
-    },
+    conditions: {
+      ArnLike: {
+        'kms:EncryptionContext:aws:logs:arn': [
+          `arn:${this.props.partition}:logs:${region}:${account}:log-group:datazone-*`,
+          `arn:${this.props.partition}:logs:${region}:${account}:log-group:airflow-*`,
+          `arn:${this.props.partition}:logs:${region}:${account}:log-group:/aws/lambda/amazon-bedrock-ide-*`,
+        ],
+      },
+    },

The operator change (ArnEquals → ArnLike) fixes the wildcard-matching bug. The third prefix covers the Lambda log groups created by the AmazonBedrockKnowledgeBase blueprint (kbIngestion and opensearchIndex functions). The /aws/lambda/amazon-bedrock-ide-* prefix was verified against AmazonBedrockKnowledgeBase.

Other blueprints (AmazonBedrockChatAgent, AmazonBedrockFlow, etc.) have not been explicitly verified, but they are expected to follow the same amazon-bedrock-ide- naming convention in which case no additional prefixes will be needed when new blueprints are enabled.

2. Add kms:DescribeKey to the CloudWatch Logs actions

-      actions: [...DECRYPT_ACTIONS, ...ENCRYPT_ACTIONS],
+      actions: [...DECRYPT_ACTIONS, ...ENCRYPT_ACTIONS, 'kms:DescribeKey'],

Impact

• Severity: Blocker -- AmazonBedrockKnowledgeBase environments fail immediately on creation.
• Unaffected: Other non-Bedrock blueprints that do not create Lambda log groups using the tooling key.

Workaround

Update the tooling KMS key policy directly:

# Fetch the current policy
aws kms get-key-policy \
  --region <region> \
  --key-id <tooling-key-id> \
  --policy-name default \
  --query Policy --output text > /tmp/key-policy.json

In /tmp/key-policy.json, find the statement with "Sid": "CloudWatchLogsEncryption" and apply both changes:

• Change "ArnEquals" to "ArnLike"
• Add "arn:aws:logs:<region>:<account>:log-group:/aws/lambda/amazon-bedrock-ide-*" as a third entry in the array
• Add "kms:DescribeKey" to the Action array

aws kms put-key-policy \
  --region <region> \
  --key-id <tooling-key-id> \
  --policy-name default \
  --policy file:///tmp/key-policy.json

After updating the key policy, delete the failed project environment and recreate it. CloudFormation does not retry a failed environment stack automatically.

[suddash24]

Hi @marcinc,

Thank you for another well-documented bug report. The root cause analysis clearly explains the two compounding problems: ArnEquals treating wildcards as literals instead of globs, and the missing log group prefix for Bedrock Lambda functions.

We're actively investigating this issue and evaluating the associated pull request (#49). The fix to switch to ArnLike and add the /aws/lambda/amazon-bedrock-ide-* prefix makes sense, and we appreciate you calling out the missing kms:DescribeKey action as well.

We'll review the proposed changes and follow up here once we have an update on the PR status. Thank you for including the workaround for anyone hitting this in the meantime.

[suddash24]

Hello @marcinc,

As mentioned in issue #46 , we have attempted to replicate the issue you reported; however, we were unable to reproduce it in our environment. To assist us in investigating further, could you please share your MDAA configuration along with the detailed steps to reproduce the problem?

Based on our initial assessment, it is possible that certain permissions may have been modified since your deployment attempt. For reference, we enabled the AmazonBedrockGuardrail blueprint and used it to create an environment via a profile, and the deployment completed without issue.

Please let us know if there is any additional context that might help us identify the root cause.