ChelseaKR · ChelseaKR · Jul 12, 2026 · Jul 12, 2026
@@ -25,6 +25,7 @@ follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   residual-risk register. Enabled GitHub private vulnerability reporting and
   converted every remaining human/account-setting conformance blocker into a
   linked issue rather than an untracked roadmap assertion.
+
 - **DORA delivery-health review + root `DEFINITION_OF_DONE.md` (2026-07-07).**
   `docs/DORA-DELIVERY-HEALTH-REVIEW.md` instantiates QM-11: Deployment Frequency and
   Change Lead Time computed from real merged-PR history (`gh pr list`), with Change
@@ -150,6 +151,16 @@ follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
   anonymous surface (HTML, JSON record/list APIs, CSV export), while the withholding is
   still acknowledged honestly without exposing the embargo date to outsiders.
 
+### Fixed
+
+- **Cloud-init secret tracing (SEV2, 2026-07-12).** Removed shell xtrace from
+  AWS first-boot provisioning after the initial synthetic demo deploy revealed
+  that expanded secret assignments reached the IAM-restricted EC2 console log.
+  Both demo credentials were rotated, the synthetic archive was rebuilt, and a
+  regression test now forbids xtrace in the secret-bearing template. See
+  incident [#86](https://github.com/ChelseaKR/ledger/issues/86) and the committed
+  postmortem under `docs/incidents/`.
+
 ### Prepared as 0.1.0 (2026-06-16) — first reference implementation, not yet tagged
 
 A small collective can install ledger, self-host it on one inexpensive box with no

@@ -0,0 +1,78 @@
+# Incident: cloud-init traced synthetic demo secrets — 2026-07-12
+
+**Severity:** SEV2
+
+**Status:** Resolved
+
+**Related issue:** [#86](https://github.com/ChelseaKR/ledger/issues/86)
+
+## Summary
+
+The first AWS showcase deployment ran its cloud-init shell with xtrace enabled.
+Expanded assignments for the generated vault key and claim secret were written to
+the IAM-restricted EC2 console log. The values were not published and the showcase
+contained synthetic data only, but operational logs are not an acceptable secret
+store. Both credentials were rotated, the synthetic archive was rebuilt, and
+shell tracing was removed from provisioning.
+
+## Timeline (UTC)
+
+| Time | Event |
+|---|---|
+| 05:39 | Terraform apply began for the first AWS showcase deployment. |
+| 05:41 | Cloud-init completed; health and TLS became ready. |
+| 05:45 | Post-deploy console review detected traced secret assignments. |
+| 05:47 | Both SSM values were rotated; the runtime environment was replaced; the synthetic archive volume was destroyed and reseeded. |
+| 05:48 | Health returned 200 under the replacement credentials. |
+
+## Impact
+
+Two L3 capability values appeared in an IAM-restricted AWS operational log. No
+real contributor or archive data existed on the instance, the values were never
+committed or printed in the task transcript, and no public route exposed them.
+Anyone with permission to read EC2 console output during the exposure window
+could have read them. Rotation invalidated both values.
+
+## Detection
+
+The deployment smoke test included a manual EC2 console-output review after
+cloud-init completed. That review searched for provisioning errors and noticed
+the use of `set -x`; a redacted occurrence count confirmed both assignments were
+present without reproducing their values.
+
+## Root cause
+
+The user-data template used the common debugging combination `set -euxo
+pipefail`. That is safe only for scripts that never handle secrets. Later work
+added SSM retrieval and runtime-environment generation without removing xtrace,
+and no infrastructure regression test prohibited it. Code review focused on
+Terraform state and file permissions but did not include the rendered execution
+log as a disclosure surface.
+
+## What went well
+
+- Post-deploy review found the issue minutes after first boot.
+- The deployment used synthetic data and IAM-restricted logs.
+- SSM-backed values could be replaced without changing Terraform state.
+- The archive volume was disposable and reseeded cleanly under the new vault key.
+
+## What went poorly
+
+- The original deployment review did not treat shell tracing as secret output.
+- There was no automated guard against xtrace in secret-bearing provisioning.
+- The initial stability watch began before console-output review completed.
+
+## Action items
+
+| Action | Owner | Due | Tracking issue |
+|---|---|---|---|
+| Rotate both exposed values and rebuild the synthetic archive | Chelsea Kelly-Reif | 2026-07-12 | [#86](https://github.com/ChelseaKR/ledger/issues/86) — complete |
+| Remove xtrace from user data | Chelsea Kelly-Reif | 2026-07-12 | [#86](https://github.com/ChelseaKR/ledger/issues/86) — complete in hotfix |
+| Add a regression test forbidding xtrace in the template | Chelsea Kelly-Reif | 2026-07-12 | [#86](https://github.com/ChelseaKR/ledger/issues/86) — complete in hotfix |
+| Include redacted console-log review before post-deploy stability monitoring | Chelsea Kelly-Reif | next deploy | [#86](https://github.com/ChelseaKR/ledger/issues/86) |
+
+## Related links
+
+- Incident issue: [#86](https://github.com/ChelseaKR/ledger/issues/86)
+- Provisioning template: `infra/aws/terraform/user_data.sh.tftpl`
+- Secret response procedure: [`docs/INCIDENT-RESPONSE.md`](../INCIDENT-RESPONSE.md)
@@ -6,7 +6,10 @@
 # enter Terraform state and survive instance replacement), writes a root-only
 # .env, and brings up the ledger + Caddy stack. Caddy obtains a Let's Encrypt
 # certificate for the domain once DNS points at this box's elastic IP.
-set -euxo pipefail
+# Never enable xtrace here. This script retrieves generated credentials below;
+# `set -x` writes expanded assignment values to the IAM-readable EC2 console log.
+# Keep fail-fast/unset/pipe handling without command tracing (incident #86).
+set -euo pipefail
 
 REGION="${region}"
 BUCKET="${bucket}"

@@ -0,0 +1,23 @@
+"""Security regressions for deployment infrastructure."""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+USER_DATA = ROOT / "infra" / "aws" / "terraform" / "user_data.sh.tftpl"
+
+
+def test_cloud_init_never_enables_shell_tracing() -> None:
+    """Provisioning must not echo expanded secret assignments to console logs."""
+    script = USER_DATA.read_text(encoding="utf-8")
+    executable = "\n".join(
+        line for line in script.splitlines() if not line.lstrip().startswith("#")
+    )
+
+    assert not re.search(r"(?:^|\n)\s*set\s+-[^\n\s]*x", executable)
+    assert not re.search(r"(?:^|\n)\s*set\s+-o\s+xtrace\b", executable)
+    assert "set -euo pipefail" in executable
+    assert "VAULT_KEY=$(get_or_create" in executable
+    assert "CLAIM_SECRET=$(get_or_create" in executable