What happened
On PR #3534, the code agent implemented a camelCase conversion regex for OpenSSF metric IDs. It wrote tests that computed expected values using the same regex as production code — a tautological pattern that could never catch a bug. The review agent correctly identified both the regex bug (HIGH logic-error: CII-Best-Practices → cIIBestPractices instead of ciiBestPractices) and the tautological test (HIGH test-integrity). The human reviewer (dzemanov) later had to manually add hardcoded test assertions to catch the bug. The triage agent had flagged the OpenSSF dynamic ID generation as needing special handling, but the code agent still produced a buggy implementation that its own tests couldn't catch.
What could go better
The code agent should have written tests with hardcoded expected outputs (e.g., expect(convert('CII-Best-Practices')).toBe('ciiBestPractices')) rather than computing expected values using the same transformation logic. This is a general pattern: when implementing string transformations (regex-based conversions, case transformations, slug generation), tests that recompute the expected value using production logic are tautological — they verify the code equals itself, not that it produces correct results. High confidence this is a real and generalizable improvement. The review agent already detects this pattern (it flagged it as HIGH test-integrity), but preventing it at the code agent level would eliminate the rework cycle entirely.
Proposed change
Add guidance to the code agent's base definition (or the verification/testing phase instructions) in the agents repo. Specifically, in the section covering the verification phase, add a rule like: "When writing tests for string transformation functions (regex conversions, case changes, ID normalization), always use hardcoded expected values, never compute expected values using the same logic as the code under test. Enumerate known inputs and verify each output against a manually-determined expected value. If the set of possible inputs is documented or discoverable (e.g., a list of check names, metric IDs, or enum values), test against the full set." This targets the base code agent definition at harness/code.yaml or the agent definition it references, not a repo-specific override.
Validation criteria
On the next 5 code agent PRs that include string transformation logic, verify that (1) tests use hardcoded expected values rather than computing them from production logic, and (2) the review agent does not flag test-integrity issues of the tautological-test class. A secondary signal: reduction in fix iterations caused by test-quality findings.
Generated by retro agent from redhat-developer/rhdh-plugins#3534
What happened
On PR #3534, the code agent implemented a camelCase conversion regex for OpenSSF metric IDs. It wrote tests that computed expected values using the same regex as production code — a tautological pattern that could never catch a bug. The review agent correctly identified both the regex bug (HIGH logic-error:
CII-Best-Practices→cIIBestPracticesinstead ofciiBestPractices) and the tautological test (HIGH test-integrity). The human reviewer (dzemanov) later had to manually add hardcoded test assertions to catch the bug. The triage agent had flagged the OpenSSF dynamic ID generation as needing special handling, but the code agent still produced a buggy implementation that its own tests couldn't catch.What could go better
The code agent should have written tests with hardcoded expected outputs (e.g.,
expect(convert('CII-Best-Practices')).toBe('ciiBestPractices')) rather than computing expected values using the same transformation logic. This is a general pattern: when implementing string transformations (regex-based conversions, case transformations, slug generation), tests that recompute the expected value using production logic are tautological — they verify the code equals itself, not that it produces correct results. High confidence this is a real and generalizable improvement. The review agent already detects this pattern (it flagged it as HIGH test-integrity), but preventing it at the code agent level would eliminate the rework cycle entirely.Proposed change
Add guidance to the code agent's base definition (or the verification/testing phase instructions) in the agents repo. Specifically, in the section covering the verification phase, add a rule like: "When writing tests for string transformation functions (regex conversions, case changes, ID normalization), always use hardcoded expected values, never compute expected values using the same logic as the code under test. Enumerate known inputs and verify each output against a manually-determined expected value. If the set of possible inputs is documented or discoverable (e.g., a list of check names, metric IDs, or enum values), test against the full set." This targets the base code agent definition at
harness/code.yamlor the agent definition it references, not a repo-specific override.Validation criteria
On the next 5 code agent PRs that include string transformation logic, verify that (1) tests use hardcoded expected values rather than computing them from production logic, and (2) the review agent does not flag test-integrity issues of the tautological-test class. A secondary signal: reduction in fix iterations caused by test-quality findings.
Generated by retro agent from redhat-developer/rhdh-plugins#3534