Problem
The AI review study reconstructs behavior from mutable GitHub comments, opportunistically retained local logs, and session files. This is sufficient for a bounded retrospective but not for ongoing census-quality measurement of retries, partial coverage, throttle events, provider transitions, or duplicate mechanisms.
Examples of current uncertainty include edited-in-place OCR summaries, replaced CodeRabbit throttle comments, empty retained session files, missing local run identity, and incompatible denominators between local and PR review samples.
Desired behavior
Build an append-only, privacy-preserving cross-repository AI-review event ledger and derive mechanism-specific metrics from it.
Acceptance criteria
- Define a versioned event schema containing repository, PR, reviewer, trigger, requested SHA/range, reviewed SHA/range, run/session/attempt identifiers, parent retry, terminal status, coverage, provider/model identity, non-secret rule/config hash, and GitHub object IDs/timestamps.
- Retain redacted raw-artifact hashes and provenance without publishing credentials, private paths, or unrelated source content.
- Record selected/completed/reused/failed/waived coverage when supplied by the review system.
- Separate metrics for within-run exact duplicates, adjudicated within-run semantic duplicates, cross-batch exact repetition, stale/remediated findings, cross-reviewer overlap, partial runs, hard failures, provider transitions, review completion, and CodeRabbit throttling.
- Aggregate CodeRabbit demand at the developer-and-organization scope where policy applies, while retaining repository dimensions.
- Separate availability from finding quality and finding count from reviewed-file coverage.
- Document retention policy, redaction policy, immutable corrections, and backfill limits.
- Add deterministic importers/validators and reproducible figures without requiring private credentials for validation of published aggregates.
Non-goals
- This is initially an analytics ledger, not a product-side OCR/CodeRabbit suppression service.
- It does not automatically accept, reject, or merge review findings.
- It does not pretend to reconstruct irrecoverable historical events.
Dependencies
Future records should consume the upstream coverage-manifest and finding-identity contracts:
Problem
The AI review study reconstructs behavior from mutable GitHub comments, opportunistically retained local logs, and session files. This is sufficient for a bounded retrospective but not for ongoing census-quality measurement of retries, partial coverage, throttle events, provider transitions, or duplicate mechanisms.
Examples of current uncertainty include edited-in-place OCR summaries, replaced CodeRabbit throttle comments, empty retained session files, missing local run identity, and incompatible denominators between local and PR review samples.
Desired behavior
Build an append-only, privacy-preserving cross-repository AI-review event ledger and derive mechanism-specific metrics from it.
Acceptance criteria
Non-goals
Dependencies
Future records should consume the upstream coverage-manifest and finding-identity contracts: