api: configurable data retention for findings and audit events (#73) - #102
Merged
Conversation
This was referenced Jul 31, 2026
Findings, their append-only trail, and the administrative audit log grow forever. This bounds that — and, mostly, defines what will never be deleted. **Every window is off by default.** This database is evidence: it records where an organisation's secrets are and who did what about each one. Deleting some of it has to be a decision somebody made, not something that starts happening because they upgraded. With nothing configured the purge runs and removes nothing. **Never eligible, whatever the window says:** * open findings — an unresolved secret is not old news; * analyst decisions (`false_positive`, `accepted_risk`, *manual* resolutions) — a judgement that expires silently means the finding returns next scan with nobody remembering it was already considered. Only `auto` resolutions, which are an inference from absence, are eligible; * suppressed findings — deleting one resurrects it as new on the next scan, the exact outcome the suppression exists to prevent (ADR 0008); * an open finding's events, however old. The finding clock runs from the decision, not the first sighting: a secret found two years ago and auto-resolved yesterday is one day old here. Purges audit themselves. Deleting evidence is an administrative action, so a round that removed anything writes `retention.purged` with the counts *and the windows that justified them*, so the row still explains itself when the settings have since changed. A round that deleted nothing writes nothing — an audit log full of "deleted 0 rows" every minute is one nobody reads. That row is subject to the audit window like any other, which is honest rather than a special case hiding from the policy it enforces. Runs last in the maintenance round, under the existing advisory lock, and deletes in batches so the first purge on a database that has never had one cannot hold locks for minutes. `python -m iceberg_api retention-purge` runs one now. `docs/retention.md` covers the windows, the never-delete rules, and guidance on choosing them (including that "you cannot get these rows back"). Closes #73 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
richardmhope
force-pushed
the
m4-retention
branch
from
July 31, 2026 08:23
9eb6603 to
c3e6daf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Findings, their append-only trail, and the administrative audit log grow forever. This bounds that
— and, mostly, defines what will never be deleted.
Off by default, on purpose
This database is evidence: it records where an organisation's secrets are and who did what about
each one. Deleting some of it has to be a decision somebody made, not something that starts
happening because they upgraded. With nothing configured the purge runs and removes nothing.
ICEBERG_RETENTION_RESOLVED_FINDINGS_DAYS0ICEBERG_RETENTION_FINDING_EVENTS_DAYS0ICEBERG_RETENTION_AUDIT_EVENTS_DAYS0audit_eventrowsNever eligible — the actual product
Most of the tests assert something was not deleted, which is the right shape here: the purge is
trivial, the eligibility rules are everything, and getting them wrong destroys evidence nobody can
get back.
false_positive,accepted_risk, manual resolutions. A judgement thatexpires silently means the finding returns next scan with nobody remembering it was already
considered. Only
autoresolutions (an inference from absence, made by a scan) are eligible.the suppression exists to prevent (ADR 0008).
The finding clock runs from the decision, not the first sighting: a secret found two years ago
and auto-resolved yesterday is one day old here.
Purges audit themselves
Deleting evidence is an administrative action. A round that removed anything writes
action = retention.purgedwith the counts and the windows that justified them, so the rowstill explains itself a year later when the settings have changed. A round that deleted nothing
writes nothing — an audit log full of "deleted 0 rows" every minute is one nobody reads.
That audit row is itself subject to the audit window. Deliberate, not an oversight: a purge record
exempt from retention would be a special case hiding from the policy it enforces. The docs say so,
and say to ship the structured logs off-box if you need purge records to outlive the window.
Operating
Runs last in the maintenance round under the existing advisory lock (one replica purges however
many are running), and deletes in batches so the first purge on a never-purged database cannot hold
locks for minutes.
python -m iceberg_api retention-purgeruns one on demand.docs/retention.mdcovers all of the above plus guidance on choosing windows — compliance anchors,data-minimisation, and the note that
finding_eventis usually what is actually growing.16 tests;
make lint,make typeand the full suite pass.Closes #73
🤖 Generated with Claude Code