Skip to content

cli: replay command to rebuild from historical CloudTrail logs - #56

Merged
alsmola merged 5 commits into
mainfrom
feat/replay-command
Jul 26, 2026
Merged

cli: replay command to rebuild from historical CloudTrail logs#56
alsmola merged 5 commits into
mainfrom
feat/replay-command

Conversation

@alsmola

@alsmola alsmola commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Adds trailtool replay, the second reset-and-replay primitive from #49 (design in #54, docs/design/cloudtrail-replay.md). With reset (#55), it gives a full rebuild: reset, then replay the range.

Replay lists historical CloudTrail objects in S3 and invokes the ingestor Lambda once per object with a synthetic S3 event, the same shape EventBridge delivers live. It adds no second ingestion path, so replayed and live data are identical.

Selection

  • --prefix replays every object under an explicit S3 key prefix (the escape hatch for org trails or custom layouts).
  • --from / --to (with --account and --region) expand to per-day CloudTrail key prefixes under AWSLogs/<account>/CloudTrail/<region>/YYYY/MM/DD/. Dates accept YYYY-MM-DD or RFC3339.

Listing is ascending, which for CloudTrail's date-partitioned keys is chronological. That ordering is what keeps windowed-fallback sessions and cross-file attribution behaving as they did live (design §6). Objects are filtered to gzipped event logs; Digest/Insight/Aggregated sidecars are skipped, matching the Lambda's own skips.

Execution

  • Invocations run with bounded --concurrency (default 4). Synchronous invocation lets the driver observe per-object success and apply backpressure.
  • A single object's failure is recorded, not fatal; failed keys are listed for targeted re-runs, and the command exits non-zero if any failed.
  • A local resume cursor advances only past a contiguous run of successes, so an interrupted replay resumes without stranding a failed object below the cursor. --no-resume ignores it; --dry-run lists and counts without invoking.

Structure

core/replay holds the driver behind Lister / Invoker / Cursor interfaces. The AWS-backed S3 lister and Lambda invoker live in aws.go, separated from the pure loop so the driver is tested with fakes: ordering, filtering, resume-skip, contiguous high-water on partial failure, overlapping-prefix dedup, and dry-run, all under -race. Adds the s3, lambda, and aws-lambda-go/events dependencies.

Part of #49.

@alsmola
alsmola force-pushed the feat/replay-command branch from f4adc6b to 1b0cd31 Compare July 26, 2026 15:32
@alsmola
alsmola changed the base branch from main to feat/reset-command July 26, 2026 15:32
@alsmola

alsmola commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Reworked to the corrected v1 after review. All four P1s and both P2s that touched replay are addressed:

  • [P1] Concurrency vs. ordering — removed the worker pool and --concurrency. Replay is now strictly sequential: the next object is invoked only after the previous returns, so the ingestor sees objects in chronological order. Sequential is documented as a correctness requirement, not a knob.
  • [P1] Cursor breaks the rebuild — removed the persistent FileCursor entirely. It survived a reset (a rebuild would skip everything and leave the projection empty) yet didn't survive a kill (saved only after the whole run). Replay is now stateless; a re-run relists the range and relies on the ingestor marker for in-window skip. A generation-bound checkpoint is a deferred follow-up.
  • [P2] RFC3339 replays whole days--from/--to are now date-only (YYYY-MM-DD). Sub-day instants are rejected, not silently rounded up.
  • [P2] JSON reports partial replay as success — JSON mode now encodes the result and returns a non-nil error when any object failed, so scripted callers exit non-zero. New test locks this in.

Simplifications also applied: streamed listing with no whole-range buffer, sort, or cross-prefix dedup (the CLI only emits ordered single-account/region prefixes); help documents the no-overlap requirement.

The design doc's overlap claim (#54, finding P1 #3) is corrected there: the ingestor marker is a sequential-redelivery guard, not general idempotency, so overlap with live delivery is an explicit non-goal pending an atomic claim.

Now stacked on #55 (both previously inserted at the same main.go line). New tests: order preservation, cross-prefix order, failure-continue, context cancel, dry-run, selection validation, sub-day rejection, JSON failure exit code. go test ./..., go vet ./..., and go test -race all green.

@alsmola
alsmola force-pushed the feat/replay-command branch 2 times, most recently from b487a9f to c9c5139 Compare July 26, 2026 22:28
@alsmola
alsmola changed the base branch from feat/reset-command to main July 26, 2026 22:32
Alex Smolen and others added 5 commits July 26, 2026 15:32
Adds `trailtool replay`, which lists historical CloudTrail objects in
S3 and invokes the ingestor Lambda once per object with a synthetic
S3 event, reusing the live ingestion path so replayed and live data
are identical.

Selection is by explicit --prefix or by --from/--to (with --account
and --region) expanded to per-day CloudTrail key prefixes. Listing is
ascending, hence chronological, so windowed sessions and cross-file
attribution behave as they did live. Objects are filtered to gzipped
event logs (Digest/Insight/Aggregated skipped, matching the Lambda).

Invocations run with bounded --concurrency. Failures are recorded,
not fatal, and reported for targeted re-runs. A local resume cursor
advances only past a contiguous run of successes, so an interrupted
replay resumes without stranding a failed object below the cursor.
--dry-run lists and counts without invoking.

The core/replay package holds the driver behind Lister/Invoker/Cursor
interfaces; the AWS-backed S3 lister and Lambda invoker are separated
from the pure loop so it is tested with fakes (ordering, resume,
contiguous high-water, dedup, dry-run) under -race.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reworks replay to the correct offline-rebuild v1:

- Sequential invocation only. Removes the concurrency pool and
  --concurrency; cross-file attribution is order-sensitive and never
  repaired, so objects must reach the ingestor in chronological order.
- Removes the persistent resume cursor (FileCursor and all wiring). It
  survived a reset (a rebuild would skip everything and leave the
  projection empty) yet did not survive a kill (saved only after the
  whole run). Replay is now stateless; a re-run relists the range.
- Date-only --from/--to (YYYY-MM-DD). Rejects sub-day instants instead
  of silently rounding them up to whole boundary days, which would put
  out-of-range events into the projection.
- Streamed listing: no whole-range buffer, sort, or cross-prefix dedup.
  The CLI only ever emits ordered single-account/region prefixes.
- JSON mode now returns a non-nil error when any object failed, so
  scripted callers exit non-zero (previously only text mode did).

Help documents the no-overlap requirement. Tests cover order
preservation, failure-continue, context cancel, dry-run, selection
validation, and the JSON failure exit code.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the "assumes it is the only writer / can double-count"
wording with the actual contract: replay closed days, whose objects
are disjoint from live delivery, so the two can run concurrently.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--bucket is no longer required. When omitted, replay reads the bucket
name from the deployed ingestor CloudFormation stack (the same stack
names 'trailtool status' probes), so the operator does not repeat what
the deployment already knows. --bucket remains as an override.

Adds a CloudTrailBucketName output to the ingestor template and reads
it, falling back to the stack parameter of the same name so it works
on stacks deployed before the output existed.

Selection flags are validated before any AWS call, so a bad invocation
still fails fast without hitting CloudFormation.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
validateReplaySelection already guarantees exactly one of --prefix or
--from/--to is set before this runs, so the mutual-exclusion case and
the default were dead. Reduce to a plain prefix-or-day-range split.

Part of #49

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alsmola
alsmola force-pushed the feat/replay-command branch from c9c5139 to f09af4c Compare July 26, 2026 22:33
@alsmola
alsmola merged commit 8b75561 into main Jul 26, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant