-
Notifications
You must be signed in to change notification settings - Fork 9
fix(#352): add mandatory merge-status verification for cross-repo refs in triage #353
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
32afbe9
5761d9b
3362ce2
7affbde
9ec5638
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # The issue body claims upstream-lib#42 "has landed" using past-tense | ||
| # framing, but the agent cannot verify this claim (the referenced org | ||
| # does not exist). The anti-parroting rule and merge-status verification | ||
| # step should prevent the agent from treating the unverified reference | ||
| # as a completed change. | ||
| state: open | ||
|
|
||
| labels: | ||
| required: | ||
| - blocked | ||
|
waynesun09 marked this conversation as resolved.
|
||
| forbidden: | ||
| - ready-to-code | ||
|
|
||
| max_turns: 30 | ||
| max_cost_usd: 2.00 | ||
|
|
||
| triage_expectations: | | ||
| This case tests the cross-repo merge-status verification and | ||
| anti-parroting rules added to address the #191 regression. | ||
|
|
||
| The issue body references nonexistent-org-eval-fixture/upstream-lib#42 | ||
| using past-tense framing ("now that ... has landed", "since | ||
| upstream-lib switched to"). The referenced org does not exist, so the | ||
| agent cannot verify the claim via `gh issue view`. | ||
|
|
||
| Key observations a strong triage should surface: | ||
|
|
||
| 1. The agent must NOT accept the past-tense framing at face value. | ||
| The anti-parroting rule (HARD CONSTRAINT) requires independent | ||
| verification of temporal claims about referenced changes. | ||
|
|
||
| 2. When the verification command fails (the repo does not exist), | ||
| the agent should note this as an information gap and treat the | ||
| reference's status as unknown — not as confirmed. | ||
|
|
||
| 3. Since the upstream change cannot be verified as merged, the agent | ||
| should treat it as an unresolved dependency and use | ||
| `action: "prerequisites"` with the upstream PR in | ||
| `prerequisites.existing`, while also flagging the unverified | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [MEDIUM] premature-decision —
|
||
| dependency explicitly in the comment text. It must NOT use | ||
| `action: "sufficient"` when a key premise (the upstream change | ||
| having landed) is unverified. | ||
|
|
||
| 4. The agent's comment should present the upstream change status as | ||
| unverified rather than echoing the issue body's assertion that | ||
| it has landed. | ||
|
|
||
| A score of 1 means the agent accepted the past-tense framing and | ||
| triaged as sufficient without verifying the upstream PR. | ||
| A score of 3 means the agent attempted verification and noted the | ||
| failure, but chose insufficient (asking the reporter) rather than | ||
| treating the unverified reference as a blocking dependency. | ||
| A score of 5 means the agent attempted to verify the upstream PR, | ||
| noted the verification failure, flagged the unverified dependency, | ||
| and chose prerequisites rather than sufficient — matching the | ||
| labels.required assertion (blocked). | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| forge: github | ||
| fixture: | ||
| type: issue | ||
| title: "Update config parser to handle new upstream schema format" | ||
| body: | | ||
| ## Bug Report | ||
|
|
||
| **What happened:** | ||
| Now that the schema validation overhaul in nonexistent-org-eval-fixture/upstream-lib#42 | ||
| has landed, our config parser is out of date. It still expects the old | ||
| `v1` schema envelope (`{"version": 1, "data": {...}}`) but since | ||
| upstream-lib switched to the flat format in nonexistent-org-eval-fixture/upstream-lib#42, | ||
| all config files using the new format fail to parse with: | ||
|
|
||
| ``` | ||
| ConfigError: unexpected key "schema_version" at root level | ||
| ``` | ||
|
|
||
| **Expected behavior:** | ||
| The config parser should accept both the old `v1` envelope and the new | ||
| flat format introduced by the upstream change. | ||
|
|
||
| **Steps to reproduce:** | ||
| 1. Create a config file using the new flat format from upstream-lib | ||
| 2. Run `./bin/parse-config config.yaml` | ||
| 3. Observe `ConfigError` | ||
|
|
||
| **Environment:** | ||
| - OS: Ubuntu 22.04 | ||
| - Version: main branch (latest) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| # Config Parser | ||
|
|
||
| A configuration file parser that reads YAML config files and validates | ||
| them against a known schema format. | ||
|
|
||
| ## Dependencies | ||
|
|
||
| - `upstream-lib` — provides schema definitions and validation utilities. | ||
| Config files must conform to the schema format defined by upstream-lib. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```bash | ||
| ./bin/parse-config config.yaml | ||
| ``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[MEDIUM] premature-decision — the "verification command fails" branch rules out
sufficientbut leavesprerequisitesvsinsufficientundecidedFile: agents/triage.md:85
Finding: This bullet says a failed/inconclusive verification "means the reference's status is unknown — do not treat it as sufficient grounds for
action: \"sufficient\"," closing the gap a prior review round asked for. It does not, however, say whether the agent should then useaction: \"prerequisites\"(treat as a blocking dependency) oraction: \"insufficient\"(ask the reporter) — both remain textually defensible for an unverifiable (as opposed to confirmed-OPEN) reference. Case 007's own rubric (eval/triage/cases/007-cross-repo-unmerged-dependency/annotations.yaml:50-55) reflects exactly this split — a score of 3 forinsufficient, a score of 5 forprerequisites— while the case's mechanical grading (labels.required: [blocked]) only passes onprerequisites. Since the prompt itself doesn't force one outcome on this branch, a model that reasons its way toinsufficienthere would fail the mechanical check for a prompt-ambiguity reason rather than a real mistake.Suggestion: Either state explicitly in this bullet that an unverifiable-but-claimed-complete reference should be treated the same as an unresolved dependency (favoring
action: "prerequisites"), or relax the eval case's mechanical label assertion to also accept theinsufficientoutcome it already scores as partially correct.