What happened
PR #1766 added a new redhat_security collection, tagging 124 existing release rules across 33 rego files. The review agent approved four times (commits 6963252, 2a58409, 6a4acfa, 286931c) without ever asking whether the new collection had functional test coverage. Human reviewer Acepresso asked on July 13: 'Do we have a test that covers the use of the new collection?' This led to a productive discussion where the PR author explained that CLI tests in internal/evaluator/filters_test.go cover collection filtering, and checks/annotations.rego validates dependency-collection consistency. Acepresso was satisfied, but the agent never surfaced this question.
What could go better
The review agent should have context about what testing is expected for collection changes in this repo. Currently AGENTS.md describes the 'Common Change Patterns' table with 'Add a new release policy rule → policy/release/ (rule + _test + add to collection)' but says nothing about adding a collection itself. Without this context, the agent cannot distinguish between a change that needs new tests and one that is covered by existing infrastructure tests. Confidence: medium. The review agent may still not ask this question even with the guidance, since the change was truly metadata-only. But documenting the testing expectations would help both agents and human contributors understand the coverage model.
Proposed change
Add a row to the 'Common Change Patterns' table in AGENTS.md for collection changes:
| Add a new collection | policy/*/collection/<name>/ — metadata-only definition file. No new tests needed for the collection itself; the CLI repo tests collection filtering (internal/evaluator/filters_test.go) and make conventions-check validates dependency-collection superset constraints. If new rules are added alongside the collection, those rules need _test.rego coverage as usual. |
This gives the review agent and future contributors explicit context about what is and isn't expected when introducing a new collection.
Validation criteria
On the next PR that adds or modifies a collection definition, the review agent should either (a) note that collection changes are metadata-only and covered by existing infrastructure tests, or (b) ask about rule-level test coverage only if new rules are introduced alongside the collection. Compare review agent output against human reviewer feedback on the next 2-3 collection-related PRs.
Generated by retro agent from #1766
What happened
PR #1766 added a new
redhat_securitycollection, tagging 124 existing release rules across 33 rego files. The review agent approved four times (commits 6963252, 2a58409, 6a4acfa, 286931c) without ever asking whether the new collection had functional test coverage. Human reviewer Acepresso asked on July 13: 'Do we have a test that covers the use of the new collection?' This led to a productive discussion where the PR author explained that CLI tests ininternal/evaluator/filters_test.gocover collection filtering, andchecks/annotations.regovalidates dependency-collection consistency. Acepresso was satisfied, but the agent never surfaced this question.What could go better
The review agent should have context about what testing is expected for collection changes in this repo. Currently AGENTS.md describes the 'Common Change Patterns' table with 'Add a new release policy rule → policy/release/ (rule + _test + add to collection)' but says nothing about adding a collection itself. Without this context, the agent cannot distinguish between a change that needs new tests and one that is covered by existing infrastructure tests. Confidence: medium. The review agent may still not ask this question even with the guidance, since the change was truly metadata-only. But documenting the testing expectations would help both agents and human contributors understand the coverage model.
Proposed change
Add a row to the 'Common Change Patterns' table in AGENTS.md for collection changes:
| Add a new collection |
policy/*/collection/<name>/— metadata-only definition file. No new tests needed for the collection itself; the CLI repo tests collection filtering (internal/evaluator/filters_test.go) andmake conventions-checkvalidates dependency-collection superset constraints. If new rules are added alongside the collection, those rules need_test.regocoverage as usual. |This gives the review agent and future contributors explicit context about what is and isn't expected when introducing a new collection.
Validation criteria
On the next PR that adds or modifies a collection definition, the review agent should either (a) note that collection changes are metadata-only and covered by existing infrastructure tests, or (b) ask about rule-level test coverage only if new rules are introduced alongside the collection. Compare review agent output against human reviewer feedback on the next 2-3 collection-related PRs.
Generated by retro agent from #1766