What happened
On PR #1779, the review agent's initial review (run 29461958211) identified a medium-severity documentation accuracy error: the collection-registration mechanism was incorrectly described in docs/patterns/add-release-policy-rule.md and .claude/rules/policy-rules.md. The author pushed fix commit 375f923f correcting both files. The re-review (run 29517353505) on the final commit 4dfd4e06 posted 3 findings (an unrelated path error, stderr suppression, and tool-in-PATH assumption) but did not check whether AGENTS.md — which was also in the PR diff and describes the same collection mechanism — had the same error. It did. The incorrect guidance shipped to main.
What could go better
The re-review had access to the previous review's findings (included as "Previous run" history in the sticky comment). When a systematic documentation error is partially fixed, the re-review should verify the fix is complete by checking whether other files in the diff that describe the same concept also received the correction. In this case, the agent knew the collection mechanism was wrong in 2 files, saw those 2 files were fixed, but didn't check the 3rd file (AGENTS.md) that describes the same mechanism. Confidence: high — the error is clear and the re-review context included the original finding. Uncertainty: this may be difficult to solve generically since it requires the agent to reason about conceptual relationships between files, but for documentation-focused PRs where multiple files describe the same mechanism, it should be feasible.
Proposed change
Add guidance to the review agent's re-review prompt or skill instructing it to verify cross-file consistency of fixes: when a previous review identified a documentation accuracy or factual error and the author has pushed fix commits, the re-review should check whether other files in the PR diff that document the same concept or mechanism also need the same correction. This could be implemented as a checklist step in the re-review flow: "For each previously-identified finding that was addressed, list all files in the PR diff that describe the same topic. Verify the fix was applied to all of them."
Validation criteria
On the next 5 re-reviews where a documentation accuracy finding was partially addressed with fix commits, the review agent should explicitly check other files in the PR diff for the same error pattern and flag any remaining unfixed instances. Success means no more incomplete documentation fixes shipping to main when the error was already identified in a subset of affected files.
Generated by retro agent from conforma/policy#1779
What happened
On PR #1779, the review agent's initial review (run 29461958211) identified a medium-severity documentation accuracy error: the collection-registration mechanism was incorrectly described in
docs/patterns/add-release-policy-rule.mdand.claude/rules/policy-rules.md. The author pushed fix commit375f923fcorrecting both files. The re-review (run 29517353505) on the final commit4dfd4e06posted 3 findings (an unrelated path error, stderr suppression, and tool-in-PATH assumption) but did not check whetherAGENTS.md— which was also in the PR diff and describes the same collection mechanism — had the same error. It did. The incorrect guidance shipped to main.What could go better
The re-review had access to the previous review's findings (included as "Previous run" history in the sticky comment). When a systematic documentation error is partially fixed, the re-review should verify the fix is complete by checking whether other files in the diff that describe the same concept also received the correction. In this case, the agent knew the collection mechanism was wrong in 2 files, saw those 2 files were fixed, but didn't check the 3rd file (AGENTS.md) that describes the same mechanism. Confidence: high — the error is clear and the re-review context included the original finding. Uncertainty: this may be difficult to solve generically since it requires the agent to reason about conceptual relationships between files, but for documentation-focused PRs where multiple files describe the same mechanism, it should be feasible.
Proposed change
Add guidance to the review agent's re-review prompt or skill instructing it to verify cross-file consistency of fixes: when a previous review identified a documentation accuracy or factual error and the author has pushed fix commits, the re-review should check whether other files in the PR diff that document the same concept or mechanism also need the same correction. This could be implemented as a checklist step in the re-review flow: "For each previously-identified finding that was addressed, list all files in the PR diff that describe the same topic. Verify the fix was applied to all of them."
Validation criteria
On the next 5 re-reviews where a documentation accuracy finding was partially addressed with fix commits, the review agent should explicitly check other files in the PR diff for the same error pattern and flag any remaining unfixed instances. Success means no more incomplete documentation fixes shipping to main when the error was already identified in a subset of affected files.
Generated by retro agent from conforma/policy#1779