feat(darwin/repro): non-gating symptom-binding signal for conformant repros (ADR-175 §63, #47)#122
Merged
Merged
Conversation
…repros (ADR-175 §63, #47) Downstream feedback (#47, from ruflo) flags Conformant repair's Goodhart trap: the agent's self-written reproduce_bug.py IS the contract, so it can fail for a friendlier/unrelated reason than the ticket demands. test-critic already rejects error-phase failures (Import/Syntax/collection) — only a real raised assertion/exception counts as valid. The remaining ADR-175 §63 follow-up ("assert the issue's SPECIFIC symptom") is implemented here as a MEASUREMENT, not a gate: symptomBindingScore(problemStatement, repro, logTail) → { assessable, score∈[0,1], boundExceptionType, matchedExceptions, matchedIdentifiers, ... } +0.5 when the failure trace raises an exception TYPE the issue names; +0.5 × the fraction of (≤3) salient issue identifiers the repro references; assessable=false (score null) when the issue names neither — no fabricated confidence. Deliberately NON-GATING: hard-rejecting low-binding repros would risk lowering the load-bearing conformant resolve (55.6% Verified), so we ship the signal a live A/B can act on, not a blind gate. Plumbed through buildReproTest → reproGateSolve → row.symptomBinding in the run report (all one-line, non-behavioral passthroughs; valid/invalid and the resolve path are unchanged). 7 tests (node-script style; $0, no Docker/LLM) wired into the CI node-test list. ADR-175 §63 updated to record the signal + the deferred-gating rationale. Co-Authored-By: claude-flow <ruv@ruv.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
Addresses the concrete, unbuilt technical follow-up named in ADR-175 §63 and raised by downstream feedback in #47 (from ruvnet/ruflo): Conformant repair's Goodhart trap. In oracle-OFF mode the agent's self-written
reproduce_bug.pyis the contract, so it can fail for a narrower/friendlier/unrelated reason than the ticket demands — a "fix-shaped artifact" that passes a fake test.test-criticalready closes the coarse slice (an Import/Syntax/collection error is classifiederror, not a valid repro — only a real raised assertion/exception counts). The remaining §63 ask — "assert the issue's specific symptom" — lands here as a measurement, not a gate.Change
symptomBindingScore(problemStatement, repro, logTail):+0.5when the failure trace raises an exception type the issue names (boundExceptionType)+0.5 ×the fraction of (≤3) salient issue identifiers the repro referencesassessable:false/score:nullwhen the issue names neither — no fabricated confidencebuildReproTest→reproGateSolve→row.symptomBindingin the run report — all one-line, non-behavioral passthroughs.valid/invalid,reproPassed, and the resolve path are unchanged.Why non-gating (deliberate)
Hard-rejecting low-binding repros would risk lowering the load-bearing conformant resolve (55.6% Verified, 51.3% Lite). The responsible order is: ship the measurement a live A/B on the frozen holdout can act on, then decide whether a gate is safe — not a blind gate.
meetsPromotionRuleand all scoring are untouched.Tests
test-critic.test.mjs, node-script style —$0, no Docker/LLM): exception-type binding, non-binding when the trace's exception differs, not-assessable, stopword filtering, the ≤3 identifier cap + fraction scoring, and the non-gating passthrough throughreproGateSolve(incl. the undefined-when-no-signal case).node-test list (ci.yml); existingrepro-gate.test.mjsstill green.🤖 Generated with claude-flow