Skip to content

feat: add CloudFormation IAM activity alerts#1

Merged
ssilare-adobe merged 5 commits into
mainfrom
migrate-iam-alerts-pr4
May 4, 2026
Merged

feat: add CloudFormation IAM activity alerts#1
ssilare-adobe merged 5 commits into
mainfrom
migrate-iam-alerts-pr4

Conversation

@ssilare-adobe

@ssilare-adobe ssilare-adobe commented Mar 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrates the CloudFormation IAM activity alerts from adobe/mysticat-ai-native-guidelines#4 into this standalone repo, with all issues from the code review addressed.

Review feedback addressed

  • eval command injection in deploy.sh — replaced with a bash array (PARAMS=(...) + "${PARAMS[@]}") so user-supplied input never passes through eval
  • Adobe-specific service account defaults — removed portalsvc1,klam-sts-user from ExcludedUserNames default everywhere (template, script, parameters file); replaced with generic placeholders or empty
  • Lambda error handling — wrapped the Slack HTTP POST in try/except; logs HTTP errors and non-200 responses via logger.error before returning, and re-raises unexpected exceptions
  • SNS email template quoted strings — replaced the per-line quoted InputTemplate block with a single JSON string using \n escapes, so email bodies arrive as plain text without literal quote characters
  • S3 sharing --acl public-read — replaced with presigned URL instructions and StackSets as alternatives; no public ACL recommendation
  • Slack webhook NoEcho limitation — added an explicit note in both the CloudFormation parameter description and README explaining that NoEcho doesn't prevent lambda:GetFunctionConfiguration access, and recommending SSM SecureString / Secrets Manager for higher-security environments
  • Documentation redundancy — consolidated the three overlapping docs into a single README.md appropriate for a standalone repo
  • Scope — this standalone repo resolves the scope concern raised in the review (content was too large for an inline guidelines example)

Changes in this update

  • teardown.sh — new interactive teardown script that lists stack resources, confirms before deleting, and waits for completion
  • teardown.sh post-deletion verification — after stack deletion, verifies the stack is truly gone and scans for leftover CloudWatch Log Groups (which Lambda creates at runtime outside of CloudFormation); offers to delete them interactively
  • deploy.sh region validationaws configure get region now validated against the AWS region pattern before use; falls back to us-east-1 if the stored value is corrupted (e.g. accidentally set to a session token)
  • Slack message formatting — field values in Slack alerts are now wrapped in backticks for inline code rendering
  • README.md setup docs — expanded Prerequisites and Quick Start sections with step-by-step AWS CLI install, credential setup (including temporary/SSO credentials), CloudTrail verification, and a numbered walkthrough of every deploy.sh prompt
  • deploy.sh post-deployment instructions — expanded next steps to cover all three monitored events (CreateUser, CreateAccessKey, CreateLoginProfile) with ready-to-run test commands, cleanup steps, CloudWatch log tailing, and a reference to teardown.sh for stack removal

Files

  • README.md — consolidated documentation covering setup, configuration, testing, sharing, security notes, and troubleshooting
  • iam-alerts-cloudformation.yaml — EventBridge rule, SNS topic, Lambda (Slack), IAM roles
  • deploy.sh — interactive deployment script with region validation and comprehensive post-deployment instructions
  • teardown.sh — interactive stack teardown script with post-deletion verification
  • parameters-example.json — sample parameters with generic placeholders

ssilare-adobe and others added 3 commits March 22, 2026 21:33
…and README setup docs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…down script reference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ssilare-adobe ssilare-adobe changed the title Add CloudFormation IAM activity alerts feat: add CloudFormation IAM activity alerts Mar 25, 2026
… group cleanup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@solaris007 solaris007 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ssilare-adobe,

Thanks for the v2 - good job addressing the feedback from the prior review on adobe/mysticat-ai-native-guidelines#4. The eval injection fix, NoEcho documentation, Adobe-specific defaults removal, doc consolidation, error handling, email template fix, presigned URLs, and standalone repo scope are all addressed. A few things remain before this is ready.

Strengths

  • Clean conditional architecture - UseEmail/UseSlack conditions with !If + AWS::NoValue on EventBridge targets is the correct CloudFormation pattern for optional resources, avoiding dead infrastructure
  • Eval injection properly eliminated - bash array (PARAMS=(...) + "${PARAMS[@]}") in deploy.sh is the right fix
  • Least-privilege IAM role - SlackNotificationFunctionRole only attaches AWSLambdaBasicExecutionRole, no over-scoping
  • SNS topic policy correctly scoped to events.amazonaws.com with specific resource ARN
  • Honest NoEcho documentation - both the template parameter description and README Security Notes call out the lambda:GetFunctionConfiguration limitation with concrete SSM/Secrets Manager recommendations
  • Teardown with orphan detection - scanning for Lambda-created CloudWatch Log Groups after stack deletion addresses a real operational gap
  • Region validation - regex guard on aws configure get region prevents corrupted config values from being silently used
  • Lambda uses defensive .get() throughout with fallback defaults

Issues

Critical (Must Fix)

1. Lambda silently drops failed Slack notifications - alerts permanently lost
iam-alerts-cloudformation.yaml:187

When the Slack webhook returns a non-200 status (rate-limited 429, server error 500, revoked webhook 403), the Lambda logs the error but returns a success response. EventBridge considers this a successful invocation (Lambda didn't throw), so it won't retry. The security alert is permanently lost with no indication to the operator. For a security monitoring tool, silent alert loss is the worst failure mode.

Fix: Raise an exception on non-200 responses so EventBridge retries (up to 185 times over 24 hours by default):

if response.status != 200:
    msg = f'Slack webhook returned HTTP {response.status}'
    logger.error(msg)
    raise RuntimeError(msg)

Important (Should Fix)

2. EventBridge rule deploys with zero targets if both channels are empty
iam-alerts-cloudformation.yaml:28

HasNotifications condition is defined (!Or [UseEmail, UseSlack]) but never referenced. If someone deploys the template directly (bypassing deploy.sh) with both NotificationEmail and SlackWebhookUrl empty, both !If branches resolve to AWS::NoValue. The EventBridge rule is created with an empty Targets list - events match but go nowhere. No error, no warning. The operator believes they have alerting, but they don't.

Fix: Add a CloudFormation validation rule that fails fast at deploy time:

Rules:
  AtLeastOneNotification:
    Assertions:
      - Assert: !Or
          - !Not [!Equals [!Ref NotificationEmail, '']]
          - !Not [!Equals [!Ref SlackWebhookUrl, '']]
        AssertDescription: At least one notification method (email or Slack) must be configured

3. deploy.sh || true swallows stack update failures
deploy.sh:259

If the update fails (bad Lambda code, IAM error), the script prints "Stack deployment completed!" and exits 0. The operator sees success when the stack is in UPDATE_ROLLBACK_COMPLETE.

Fix: Check actual stack status after the wait:

aws cloudformation wait stack-update-complete ... 2>/dev/null
STATUS=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region "$REGION" \
  --query 'Stacks[0].StackStatus' --output text)
if [[ "$STATUS" != "UPDATE_COMPLETE" ]]; then
    print_error "Stack update failed. Status: ${STATUS}"
    exit 1
fi

4. deploy.sh overwrites the default AWS CLI profile
deploy.sh:86

aws configure set without --profile silently overwrites whatever credentials exist in the user's default profile. For security teams who likely have multiple profiles, this could replace production credentials with temporary session creds. If the new credentials are also invalid, the old working ones are already gone.

Fix: Use environment variables instead (session-scoped, no disk persistence):

export AWS_ACCESS_KEY_ID="$aws_access_key_id"
export AWS_SECRET_ACCESS_KEY="$aws_secret_access_key"
export AWS_SESSION_TOKEN="$aws_session_token"

5. Teardown log group scan is overly broad
teardown.sh:129

The JMESPath query contains(logGroupName, 'iam-') matches ANY log group with "iam-" in the name - including unrelated ones like /aws/lambda/iam-rotation-handler. A hasty "yes" at the interactive prompt deletes someone else's operational logs.

Fix: Remove the || contains(logGroupName, 'iam-') clause. Match the specific Lambda function name pattern instead: /aws/lambda/${ALERT_RULE_NAME}-slack-notifier.

6. Lambda should validate webhook URL scheme
iam-alerts-cloudformation.yaml (inline Lambda)

The Lambda POSTs to whatever URL is in the env var with no validation. A non-HTTPS URL makes it an SSRF vector, leaking event data (source IPs, principal ARNs, account IDs) to any reachable endpoint.

Fix:

if not webhook_url.startswith('https://hooks.slack.com/'):
    logger.error('Invalid webhook URL - must start with https://hooks.slack.com/')
    return {'statusCode': 400, 'body': 'Invalid webhook URL'}

Minor (Nice to Have)

7. Cross-stack exports create deletion friction - All three outputs use Export. If any other stack imports these values, this stack becomes undeletable. For a self-contained alerting tool, remove the Export blocks (keep the outputs).

8. No Slack webhook URL validation in deploy.sh - Email gets validated with a regex, but the Slack webhook URL is accepted as-is. Add a basic format check (^https://hooks\.slack\.com/services/).

9. Lambda FunctionName / SNS TopicName collision risk - Explicit names prevent CloudFormation replacement during updates and cause conflicts if two stacks use the same AlertRuleName. Consider omitting physical names.

10. Region validation regex misses GovCloud - ^[a-z]{2}-[a-z]+-[0-9]+$ doesn't match us-gov-west-1. Change to ^[a-z]{2}(-[a-z]+)+-[0-9]+$.

Recommendations

  1. Add a DLQ - An SQS dead-letter queue on the EventBridge target catches permanent failures after retries are exhausted. For a security alerting pipeline, this is the safety net.
  2. Add CloudFormation parameter validation - AllowedPattern on the webhook URL parameter catches misconfiguration at deploy time.
  3. Add cfn-lint and shellcheck to CI - catches template and script issues statically.
  4. Consider AWS::CloudFormation::Interface metadata - groups parameters logically in the AWS Console.

Assessment

Ready to merge? No - fix Critical #1 first, Important #2-6 strongly recommended.

The overall design is sound - EventBridge + conditional SNS/Lambda is the right pattern, and the prior review feedback has been well addressed. The critical issue is that the Lambda's silent failure on non-200 Slack responses means security alerts can be permanently lost. That's a one-line fix (raise instead of return). The important issues are all straightforward fixes that prevent real operational surprises.

Next Steps

  1. Fix Critical #1 (Lambda raise on non-200) - one-line change
  2. Address Important #2-6 - all small, targeted fixes
  3. Minor items and recommendations can be follow-up work

Comment thread iam-alerts-cloudformation.yaml Outdated
response.status,
response.data.decode('utf-8')
)
return {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical: This returns success to EventBridge on non-200 Slack responses, so EventBridge won't retry and the security alert is silently lost. Raise an exception instead:

if response.status != 200:
    msg = f'Slack webhook returned HTTP {response.status}'
    logger.error(msg)
    raise RuntimeError(msg)

Comment thread deploy.sh Outdated
if [ "$OPERATION" == "create-stack" ]; then
aws cloudformation wait stack-create-complete --stack-name "$STACK_NAME" --region "$REGION"
else
aws cloudformation wait stack-update-complete --stack-name "$STACK_NAME" --region "$REGION" 2>/dev/null || true

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: || true swallows real update failures - if the stack rolls back, the script still prints 'completed' and exits 0. Check actual stack status after the wait instead.

Comment thread deploy.sh Outdated
read -r -p "AWS Region [${REGION}]: " input
REGION=${input:-$REGION}

aws configure set aws_access_key_id "$aws_access_key_id"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: aws configure set without --profile silently overwrites the user's default AWS profile credentials. Consider using environment variables (export AWS_ACCESS_KEY_ID=...) instead - they're session-scoped and don't persist to disk.

Comment thread teardown.sh Outdated
log_groups=$(aws logs describe-log-groups \
--log-group-name-prefix "/aws/lambda/" \
--region "$REGION" \
--query "logGroups[?contains(logGroupName, '${STACK_NAME}') || contains(logGroupName, 'iam-')].logGroupName" \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: contains(logGroupName, 'iam-') is too broad - matches unrelated log groups like /aws/lambda/iam-rotation-handler. Match against the specific Lambda function name pattern (${ALERT_RULE_NAME}-slack-notifier) instead.

…dential handling, fix log group scope, add webhook validation
@ssilare-adobe

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review @solaris007! I've addressed all Critical and Important issues in 548f616, plus the clear-cut Minor items. Here's what was done:

Critical #1 — Lambda now raises on non-200 Slack responses
Replaced the return with raise RuntimeError(msg) so EventBridge will retry on 429/500/403 instead of silently dropping the alert.

Important #2 — CloudFormation Rules block added
Added an AtLeastOneNotification assertion that fails fast at deploy time if both email and Slack are left empty. The HasNotifications condition was already defined but unused — the Rules block closes that gap.

Important #3 — Stack update failure no longer swallowed
Replaced || true with an explicit describe-stacks status check after the wait. Script now exits 1 with the actual status if the stack ends up in anything other than UPDATE_COMPLETE.

Important #4 — Credentials via env vars, not aws configure set
Replaced all aws configure set calls with export AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN. Credentials are now session-scoped and never written to ~/.aws/credentials.

Important #5 — Teardown log group scan scoped to exact Lambda name
Captures the Lambda physical resource ID from list-stack-resources before deletion, then uses it as the --log-group-name-prefix. Falls back to stack-name substring match only if no Lambda was found. Removes the broad || contains(..., 'iam-') clause entirely.

Important #6 — Lambda validates webhook URL scheme
Added a startswith('https://hooks.slack.com/') check at the top of lambda_handler that raises immediately on an invalid URL, preventing SSRF data leakage.

Minor #7 — Removed Export blocks from Outputs
Kept the output values but dropped Export: names so the stack isn't blocked from deletion if another stack ever imports them.

Minor #8 — Slack webhook URL validated in deploy.sh
Added a loop with a ^https://hooks\.slack\.com/services/ regex check, matching the same pattern used in the Lambda.

Minor #10 — GovCloud region regex fixed
Changed ^[a-z]{2}-[a-z]+-[0-9]+$ to ^[a-z]{2}(-[a-z]+)+-[0-9]+$ in both deploy.sh and teardown.sh so us-gov-west-1 is accepted.

Minor #9 (explicit resource names) and the recommendations (DLQ, cfn-lint/shellcheck CI, AWS::CloudFormation::Interface) are noted as good follow-up work — happy to tackle them in a separate PR to keep this one focused.

@solaris007 solaris007 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @ssilare-adobe,

Excellent turnaround. All six Critical/Important findings from the prior review are resolved, plus three of the four Minor items. The fixes are clean and match the suggested approaches.

Strengths

Previously flagged issues now addressed:

  • Lambda now raises RuntimeError on non-200 Slack responses (iam-alerts-cloudformation.yaml:193-195), so EventBridge will retry failed alerts instead of silently dropping them
  • CloudFormation AtLeastOneNotification assertion fails fast at deploy time if neither email nor Slack is configured (iam-alerts-cloudformation.yaml:36-42) - a cleaner approach than gating the rule on HasNotifications because the operator gets an immediate, clear error
  • deploy.sh stack update now verifies actual StackStatus after the wait and exits 1 on rollback (deploy.sh:271-276) - no more silent success on failed updates
  • Credential handling switched to session-scoped environment variables (deploy.sh:86-90) - no more overwriting the user's default AWS profile
  • Teardown captures the Lambda function name before stack deletion (teardown.sh:89-94) and scopes the orphaned log group scan to /aws/lambda/${LAMBDA_FUNCTION_NAME}*, with a safe stack-name-only fallback when the lookup returns empty
  • Lambda validates the webhook URL scheme before processing events (iam-alerts-cloudformation.yaml:101-103) - defense in depth against SSRF via misconfiguration
  • Cross-stack exports removed from all three outputs - the stack is now cleanly deletable
  • Webhook URL validation added to deploy.sh prompts - catches typos at deploy time with a clear error message
  • Region regex updated to ^[a-z]{2}(-[a-z]+)+-[0-9]+$ in both scripts, now handles GovCloud regions like us-gov-west-1

Assessment

Ready to merge? Yes.

Reasoning: The original design was sound and the prior review's findings are all resolved. The remaining unaddressed item (explicit FunctionName/TopicName collision risk) is legitimately minor - it only surfaces if two stacks use the same AlertRuleName in the same account/region, which is unlikely for a security alerting tool typically deployed once per account. Ship it.

@ssilare-adobe
ssilare-adobe merged commit e82d99f into main May 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants