What happened
PR #2098 had 16 review runs over 31 days (Jun 9–Jul 10) for a 2-commit, 13-file change. The privilege-escalation finding appeared identically in 5 runs (Jun 9, Jun 9, Jun 17, Jun 17, Jun 17) — the exact pattern described in #2959. The bot approved on run 14 (Jun 29) then posted a new medium finding on run 15 (same day) with no human engagement between — matching #2992. That medium finding claimed 'The sandbox.Exec call discards the exit code' when the code actually reads _, stderr, exitCode, err := sandbox.Exec(...) with exitCode != 0 checked — a factual misread matching #2868. Conservative token cost: 240-320K tokens across 16 runs.
What could go better
Three existing issues address the underlying patterns individually. This PR demonstrates all three failure modes simultaneously on a single PR, providing concrete evidence of their combined cost: repeated findings waste ~100K tokens on identical output (#2959), continued runs after approval without human engagement waste another ~60K tokens (#2992), and a factually incorrect finding about exit-code handling created noise on a security-sensitive PR (#2868). The combined effect is a 31-day review cycle where the bot's signal-to-noise ratio degraded sharply after run 3.
Proposed change
Use this PR as a prioritization signal for the three existing issues. No new issue is needed. Specifically: (1) #2959 — implement cross-iteration dedup by hashing finding category + file + line range and suppressing previously-posted findings; (2) #2992 — implement a circuit breaker that stops review dispatch after N runs (e.g., 3) without human comment or code push; (3) #2868 — add a code-quoting requirement to the meta-prompt so sub-agents must quote relevant code before asserting what it does.
Validation criteria
After implementing all three: (1) identical findings should not appear in more than one review run on the same PR; (2) review dispatch should stop after 3 runs without human engagement; (3) findings about code behavior should include quoted code snippets that can be verified against the source.
Generated by retro agent from #2098
What happened
PR #2098 had 16 review runs over 31 days (Jun 9–Jul 10) for a 2-commit, 13-file change. The privilege-escalation finding appeared identically in 5 runs (Jun 9, Jun 9, Jun 17, Jun 17, Jun 17) — the exact pattern described in #2959. The bot approved on run 14 (Jun 29) then posted a new medium finding on run 15 (same day) with no human engagement between — matching #2992. That medium finding claimed 'The sandbox.Exec call discards the exit code' when the code actually reads
_, stderr, exitCode, err := sandbox.Exec(...)withexitCode != 0checked — a factual misread matching #2868. Conservative token cost: 240-320K tokens across 16 runs.What could go better
Three existing issues address the underlying patterns individually. This PR demonstrates all three failure modes simultaneously on a single PR, providing concrete evidence of their combined cost: repeated findings waste ~100K tokens on identical output (#2959), continued runs after approval without human engagement waste another ~60K tokens (#2992), and a factually incorrect finding about exit-code handling created noise on a security-sensitive PR (#2868). The combined effect is a 31-day review cycle where the bot's signal-to-noise ratio degraded sharply after run 3.
Proposed change
Use this PR as a prioritization signal for the three existing issues. No new issue is needed. Specifically: (1) #2959 — implement cross-iteration dedup by hashing finding category + file + line range and suppressing previously-posted findings; (2) #2992 — implement a circuit breaker that stops review dispatch after N runs (e.g., 3) without human comment or code push; (3) #2868 — add a code-quoting requirement to the meta-prompt so sub-agents must quote relevant code before asserting what it does.
Validation criteria
After implementing all three: (1) identical findings should not appear in more than one review run on the same PR; (2) review dispatch should stop after 3 runs without human engagement; (3) findings about code behavior should include quoted code snippets that can be verified against the source.
Generated by retro agent from #2098