Skip to content

api: configurable data retention for findings and audit events (#73) - #102

Merged
richardmhope merged 1 commit into
mainfrom
m4-retention
Jul 31, 2026
Merged

api: configurable data retention for findings and audit events (#73)#102
richardmhope merged 1 commit into
mainfrom
m4-retention

Conversation

@richardmhope

Copy link
Copy Markdown
Contributor

Stacked on #101 (rate limiting) to avoid a three-way conflict in config.py, .env.example
and docker-compose.yml. Review the second commit; the first is #101.

Findings, their append-only trail, and the administrative audit log grow forever. This bounds that
— and, mostly, defines what will never be deleted.

Off by default, on purpose

This database is evidence: it records where an organisation's secrets are and who did what about
each one. Deleting some of it has to be a decision somebody made, not something that starts
happening because they upgraded. With nothing configured the purge runs and removes nothing.

Setting Default Ages out
ICEBERG_RETENTION_RESOLVED_FINDINGS_DAYS 0 Auto-resolved findings
ICEBERG_RETENTION_FINDING_EVENTS_DAYS 0 Events on non-open findings
ICEBERG_RETENTION_AUDIT_EVENTS_DAYS 0 audit_event rows

Never eligible — the actual product

Most of the tests assert something was not deleted, which is the right shape here: the purge is
trivial, the eligibility rules are everything, and getting them wrong destroys evidence nobody can
get back.

  • Open findings. An unresolved secret is not old news.
  • Analyst decisionsfalse_positive, accepted_risk, manual resolutions. A judgement that
    expires silently means the finding returns next scan with nobody remembering it was already
    considered. Only auto resolutions (an inference from absence, made by a scan) are eligible.
  • Suppressed findings. Deleting one resurrects it as new on the next scan — the exact outcome
    the suppression exists to prevent (ADR 0008).
  • An open finding's event trail, however old.

The finding clock runs from the decision, not the first sighting: a secret found two years ago
and auto-resolved yesterday is one day old here.

Purges audit themselves

Deleting evidence is an administrative action. A round that removed anything writes
action = retention.purged with the counts and the windows that justified them, so the row
still explains itself a year later when the settings have changed. A round that deleted nothing
writes nothing — an audit log full of "deleted 0 rows" every minute is one nobody reads.

That audit row is itself subject to the audit window. Deliberate, not an oversight: a purge record
exempt from retention would be a special case hiding from the policy it enforces. The docs say so,
and say to ship the structured logs off-box if you need purge records to outlive the window.

Operating

Runs last in the maintenance round under the existing advisory lock (one replica purges however
many are running), and deletes in batches so the first purge on a never-purged database cannot hold
locks for minutes. python -m iceberg_api retention-purge runs one on demand.

docs/retention.md covers all of the above plus guidance on choosing windows — compliance anchors,
data-minimisation, and the note that finding_event is usually what is actually growing.

16 tests; make lint, make type and the full suite pass.

Closes #73

🤖 Generated with Claude Code

Findings, their append-only trail, and the administrative audit log grow forever.
This bounds that — and, mostly, defines what will never be deleted.

**Every window is off by default.** This database is evidence: it records where
an organisation's secrets are and who did what about each one. Deleting some of
it has to be a decision somebody made, not something that starts happening
because they upgraded. With nothing configured the purge runs and removes
nothing.

**Never eligible, whatever the window says:**

* open findings — an unresolved secret is not old news;
* analyst decisions (`false_positive`, `accepted_risk`, *manual* resolutions) —
  a judgement that expires silently means the finding returns next scan with
  nobody remembering it was already considered. Only `auto` resolutions, which
  are an inference from absence, are eligible;
* suppressed findings — deleting one resurrects it as new on the next scan,
  the exact outcome the suppression exists to prevent (ADR 0008);
* an open finding's events, however old.

The finding clock runs from the decision, not the first sighting: a secret found
two years ago and auto-resolved yesterday is one day old here.

Purges audit themselves. Deleting evidence is an administrative action, so a
round that removed anything writes `retention.purged` with the counts *and the
windows that justified them*, so the row still explains itself when the settings
have since changed. A round that deleted nothing writes nothing — an audit log
full of "deleted 0 rows" every minute is one nobody reads. That row is subject to
the audit window like any other, which is honest rather than a special case
hiding from the policy it enforces.

Runs last in the maintenance round, under the existing advisory lock, and
deletes in batches so the first purge on a database that has never had one cannot
hold locks for minutes. `python -m iceberg_api retention-purge` runs one now.

`docs/retention.md` covers the windows, the never-delete rules, and guidance on
choosing them (including that "you cannot get these rows back").

Closes #73

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@richardmhope
richardmhope merged commit 3641dce into main Jul 31, 2026
3 checks passed
@richardmhope
richardmhope deleted the m4-retention branch July 31, 2026 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data-retention policy for findings & audit events

1 participant