diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a250c5..ac5be47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,7 @@ follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). residual-risk register. Enabled GitHub private vulnerability reporting and converted every remaining human/account-setting conformance blocker into a linked issue rather than an untracked roadmap assertion. + - **DORA delivery-health review + root `DEFINITION_OF_DONE.md` (2026-07-07).** `docs/DORA-DELIVERY-HEALTH-REVIEW.md` instantiates QM-11: Deployment Frequency and Change Lead Time computed from real merged-PR history (`gh pr list`), with Change @@ -150,6 +151,16 @@ follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). anonymous surface (HTML, JSON record/list APIs, CSV export), while the withholding is still acknowledged honestly without exposing the embargo date to outsiders. +### Fixed + +- **Cloud-init secret tracing (SEV2, 2026-07-12).** Removed shell xtrace from + AWS first-boot provisioning after the initial synthetic demo deploy revealed + that expanded secret assignments reached the IAM-restricted EC2 console log. + Both demo credentials were rotated, the synthetic archive was rebuilt, and a + regression test now forbids xtrace in the secret-bearing template. See + incident [#86](https://github.com/ChelseaKR/ledger/issues/86) and the committed + postmortem under `docs/incidents/`. + ### Prepared as 0.1.0 (2026-06-16) — first reference implementation, not yet tagged A small collective can install ledger, self-host it on one inexpensive box with no diff --git a/docs/incidents/2026-07-12-cloud-init-secret-tracing.md b/docs/incidents/2026-07-12-cloud-init-secret-tracing.md new file mode 100644 index 0000000..a73462a --- /dev/null +++ b/docs/incidents/2026-07-12-cloud-init-secret-tracing.md @@ -0,0 +1,78 @@ +# Incident: cloud-init traced synthetic demo secrets — 2026-07-12 + +**Severity:** SEV2 + +**Status:** Resolved + +**Related issue:** [#86](https://github.com/ChelseaKR/ledger/issues/86) + +## Summary + +The first AWS showcase deployment ran its cloud-init shell with xtrace enabled. +Expanded assignments for the generated vault key and claim secret were written to +the IAM-restricted EC2 console log. The values were not published and the showcase +contained synthetic data only, but operational logs are not an acceptable secret +store. Both credentials were rotated, the synthetic archive was rebuilt, and +shell tracing was removed from provisioning. + +## Timeline (UTC) + +| Time | Event | +|---|---| +| 05:39 | Terraform apply began for the first AWS showcase deployment. | +| 05:41 | Cloud-init completed; health and TLS became ready. | +| 05:45 | Post-deploy console review detected traced secret assignments. | +| 05:47 | Both SSM values were rotated; the runtime environment was replaced; the synthetic archive volume was destroyed and reseeded. | +| 05:48 | Health returned 200 under the replacement credentials. | + +## Impact + +Two L3 capability values appeared in an IAM-restricted AWS operational log. No +real contributor or archive data existed on the instance, the values were never +committed or printed in the task transcript, and no public route exposed them. +Anyone with permission to read EC2 console output during the exposure window +could have read them. Rotation invalidated both values. + +## Detection + +The deployment smoke test included a manual EC2 console-output review after +cloud-init completed. That review searched for provisioning errors and noticed +the use of `set -x`; a redacted occurrence count confirmed both assignments were +present without reproducing their values. + +## Root cause + +The user-data template used the common debugging combination `set -euxo +pipefail`. That is safe only for scripts that never handle secrets. Later work +added SSM retrieval and runtime-environment generation without removing xtrace, +and no infrastructure regression test prohibited it. Code review focused on +Terraform state and file permissions but did not include the rendered execution +log as a disclosure surface. + +## What went well + +- Post-deploy review found the issue minutes after first boot. +- The deployment used synthetic data and IAM-restricted logs. +- SSM-backed values could be replaced without changing Terraform state. +- The archive volume was disposable and reseeded cleanly under the new vault key. + +## What went poorly + +- The original deployment review did not treat shell tracing as secret output. +- There was no automated guard against xtrace in secret-bearing provisioning. +- The initial stability watch began before console-output review completed. + +## Action items + +| Action | Owner | Due | Tracking issue | +|---|---|---|---| +| Rotate both exposed values and rebuild the synthetic archive | Chelsea Kelly-Reif | 2026-07-12 | [#86](https://github.com/ChelseaKR/ledger/issues/86) — complete | +| Remove xtrace from user data | Chelsea Kelly-Reif | 2026-07-12 | [#86](https://github.com/ChelseaKR/ledger/issues/86) — complete in hotfix | +| Add a regression test forbidding xtrace in the template | Chelsea Kelly-Reif | 2026-07-12 | [#86](https://github.com/ChelseaKR/ledger/issues/86) — complete in hotfix | +| Include redacted console-log review before post-deploy stability monitoring | Chelsea Kelly-Reif | next deploy | [#86](https://github.com/ChelseaKR/ledger/issues/86) | + +## Related links + +- Incident issue: [#86](https://github.com/ChelseaKR/ledger/issues/86) +- Provisioning template: `infra/aws/terraform/user_data.sh.tftpl` +- Secret response procedure: [`docs/INCIDENT-RESPONSE.md`](../INCIDENT-RESPONSE.md) diff --git a/infra/aws/terraform/user_data.sh.tftpl b/infra/aws/terraform/user_data.sh.tftpl index 9f876cb..f53d505 100644 --- a/infra/aws/terraform/user_data.sh.tftpl +++ b/infra/aws/terraform/user_data.sh.tftpl @@ -6,7 +6,10 @@ # enter Terraform state and survive instance replacement), writes a root-only # .env, and brings up the ledger + Caddy stack. Caddy obtains a Let's Encrypt # certificate for the domain once DNS points at this box's elastic IP. -set -euxo pipefail +# Never enable xtrace here. This script retrieves generated credentials below; +# `set -x` writes expanded assignment values to the IAM-readable EC2 console log. +# Keep fail-fast/unset/pipe handling without command tracing (incident #86). +set -euo pipefail REGION="${region}" BUCKET="${bucket}" diff --git a/tests/test_infra_security.py b/tests/test_infra_security.py new file mode 100644 index 0000000..1663242 --- /dev/null +++ b/tests/test_infra_security.py @@ -0,0 +1,23 @@ +"""Security regressions for deployment infrastructure.""" + +from __future__ import annotations + +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +USER_DATA = ROOT / "infra" / "aws" / "terraform" / "user_data.sh.tftpl" + + +def test_cloud_init_never_enables_shell_tracing() -> None: + """Provisioning must not echo expanded secret assignments to console logs.""" + script = USER_DATA.read_text(encoding="utf-8") + executable = "\n".join( + line for line in script.splitlines() if not line.lstrip().startswith("#") + ) + + assert not re.search(r"(?:^|\n)\s*set\s+-[^\n\s]*x", executable) + assert not re.search(r"(?:^|\n)\s*set\s+-o\s+xtrace\b", executable) + assert "set -euo pipefail" in executable + assert "VAULT_KEY=$(get_or_create" in executable + assert "CLAIM_SECRET=$(get_or_create" in executable