Trace JSONL retention/rotation for the us-ny1 builder
Follow-up from the PR #9 review. The trace emitter (#7) writes one hash-chained JSONL per ingest run at <store.root>/traces/<run_id>.jsonl. The builder loops zeitghost ingest every ZEITGHOST_INTERVAL (3600s default) — so over months on us-ny1, traces/ grows unboundedly.
Not urgent (each file is small — a couple events per cycle when there's new news, zero-byte/absent when nothing new), but it should be bounded before it's a year of files.
Options
- A
--max-trace-age-days cleanup pass in ingest (or a tiny zeitghost prune-traces command) that deletes/compacts JSONLs older than N days.
- Or roll traces into a single append log with periodic compaction.
- Whatever the choice, keep it aligned with the shard-store backup story (traces live under
store.root, so they're bind-mounted + rsync'd on us-ny1 — pruning should be safe w.r.t. backups).
Acceptance criteria
Relates to #7 (trace emitter), #8 (reanalyze, which also writes traces).
Trace JSONL retention/rotation for the us-ny1 builder
Follow-up from the PR #9 review. The trace emitter (#7) writes one hash-chained JSONL per ingest run at
<store.root>/traces/<run_id>.jsonl. The builder loopszeitghost ingesteveryZEITGHOST_INTERVAL(3600s default) — so over months on us-ny1,traces/grows unboundedly.Not urgent (each file is small — a couple events per cycle when there's new news, zero-byte/absent when nothing new), but it should be bounded before it's a year of files.
Options
--max-trace-age-dayscleanup pass iningest(or a tinyzeitghost prune-tracescommand) that deletes/compacts JSONLs older than N days.store.root, so they're bind-mounted + rsync'd on us-ny1 — pruning should be safe w.r.t. backups).Acceptance criteria
traces/has a bounded retention policy (age- or count-based), documented.trace_refwithin the retention window (or that's an explicit, documented tradeoff).verify-trace <run_id>audit command — don't prune what we'd want to audit.Relates to #7 (trace emitter), #8 (reanalyze, which also writes traces).