Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions agents/triage.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,28 @@ gh pr list --repo OTHER-ORG/OTHER-REPO --state open --search "relevant keywords"

If a cross-repo search fails or returns an error (e.g., due to access restrictions), note this in your reasoning as an information gap rather than concluding no blocking work exists.

**Verify merge status of referenced PRs and issues (HARD CONSTRAINT):** For every PR or issue reference found in the issue body or its comments — whether cross-repo (e.g., `org/repo#N`) or same-repo (e.g., `#N`) — verify its current state before describing it as completed or landed. If a reference uses shorthand without an owner segment (e.g., `repo#N`), resolve the fully-qualified `owner/repo` before constructing the `--repo` argument — bare shorthand is not a valid `--repo` value.

```
# Check state (works for both issues and PRs):
gh issue view N --repo org/repo --json state,stateReason
```

Note: `gh issue view` resolves both issues and PRs, so use it as the single verification command. The `state` field returns `OPEN`, `CLOSED`, or `MERGED` (PRs only). The `stateReason` field distinguishes issue closure reasons (`COMPLETED`, `NOT_PLANNED`, `DUPLICATE`); it is empty for PRs.

Interpret the results as follows:

- **`MERGED`**: the referenced change has landed. Proceed accordingly.
- **`OPEN`**: the referenced work is still in progress — treat it as a potential blocker. Note it as a blocking dependency in your `reasoning` (and in `prerequisites.existing` if the action becomes `prerequisites`), and evaluate whether the issue should use `action: "prerequisites"` instead of `action: "sufficient"`.
- **`CLOSED` with `stateReason: "COMPLETED"`**: the referenced issue was resolved. Proceed accordingly.
- **`CLOSED` with `stateReason: "NOT_PLANNED"` or `"DUPLICATE"`**: the referenced issue was abandoned or duplicated, not completed. Treat the completion claim as contradicted.
- **`CLOSED` with empty `stateReason`**: likely a PR closed without merging — the referenced work was abandoned. The premise about this change is contradicted, not merely unverified — flag this explicitly in your `reasoning`.
- **Verification command fails** (e.g., due to access restrictions): note this in your `reasoning` as an information gap. A failed or inconclusive verification means the reference's status is unknown — do not treat it as sufficient grounds for `action: "sufficient"`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] premature-decision — the "verification command fails" branch rules out sufficient but leaves prerequisites vs insufficient undecided

File: agents/triage.md:85
Finding: This bullet says a failed/inconclusive verification "means the reference's status is unknown — do not treat it as sufficient grounds for action: \"sufficient\"," closing the gap a prior review round asked for. It does not, however, say whether the agent should then use action: \"prerequisites\" (treat as a blocking dependency) or action: \"insufficient\" (ask the reporter) — both remain textually defensible for an unverifiable (as opposed to confirmed-OPEN) reference. Case 007's own rubric (eval/triage/cases/007-cross-repo-unmerged-dependency/annotations.yaml:50-55) reflects exactly this split — a score of 3 for insufficient, a score of 5 for prerequisites — while the case's mechanical grading (labels.required: [blocked]) only passes on prerequisites. Since the prompt itself doesn't force one outcome on this branch, a model that reasons its way to insufficient here would fail the mechanical check for a prompt-ambiguity reason rather than a real mistake.
Suggestion: Either state explicitly in this bullet that an unverifiable-but-claimed-complete reference should be treated the same as an unresolved dependency (favoring action: "prerequisites"), or relax the eval case's mechanical label assertion to also accept the insufficient outcome it already scores as partially correct.


Do not treat any reference as a completed change based solely on framing in the issue body or its comments — independently confirm the state.

**Anti-parroting rule (HARD CONSTRAINT):** Do not echo temporal framing about referenced changes from the issue body or its comments without verification. Issue authors or commenters may describe unmerged or abandoned work using past tense ("since X changed…", "now that Y landed…"). Verify each such claim against the actual state of the referenced PR or issue. If you cannot verify a claim, present it as unverified rather than restating it as fact.

### 2c. Check existing prerequisites

If the issue already has a `blocked` label, check whether the previously identified prerequisites (linked in prior triage comments) are still open. Fetch the full context of each prerequisite issue or PR to understand its current state:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# The issue body claims upstream-lib#42 "has landed" using past-tense
# framing, but the agent cannot verify this claim (the referenced org
# does not exist). The anti-parroting rule and merge-status verification
# step should prevent the agent from treating the unverified reference
# as a completed change.
state: open

labels:
required:
- blocked
Comment thread
waynesun09 marked this conversation as resolved.
forbidden:
- ready-to-code

max_turns: 30
max_cost_usd: 2.00

triage_expectations: |
This case tests the cross-repo merge-status verification and
anti-parroting rules added to address the #191 regression.

The issue body references nonexistent-org-eval-fixture/upstream-lib#42
using past-tense framing ("now that ... has landed", "since
upstream-lib switched to"). The referenced org does not exist, so the
agent cannot verify the claim via `gh issue view`.

Key observations a strong triage should surface:

1. The agent must NOT accept the past-tense framing at face value.
The anti-parroting rule (HARD CONSTRAINT) requires independent
verification of temporal claims about referenced changes.

2. When the verification command fails (the repo does not exist),
the agent should note this as an information gap and treat the
reference's status as unknown — not as confirmed.

3. Since the upstream change cannot be verified as merged, the agent
should treat it as an unresolved dependency and use
`action: "prerequisites"` with the upstream PR in
`prerequisites.existing`, while also flagging the unverified

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] premature-decision — prerequisites.existing is directed to hold a reference whose existence can never be confirmed

File: eval/triage/cases/007-cross-repo-unmerged-dependency/annotations.yaml:36-39
Finding: agents/triage.md:265 defines prerequisites.existing as "issues or PRs that already exist and block this work." Case 007's own design guarantees verification of nonexistent-org-eval-fixture/upstream-lib#42 fails (the org doesn't exist), so "Key observations" item 3 directs the agent toward the one action/field pairing whose documented contract (confirmed existence) this fixture can never satisfy. A triage agent that takes triage.md's own unverified-URL guidance seriously has a textual basis for avoiding existing altogether — in tension with what the rubric scores as the top (5/5) answer. This is a downstream consequence of the eval case never reaching a genuinely-OPEN, verifiable reference (tracked separately) and will likely resolve on its own once that root gap closes — but as currently written, the annotation and the schema field it targets are in tension.
Suggestion: Fix at the root (make the fixture resolve to a real, existing, OPEN reference so existing legitimately holds a confirmed entry). If the access-failure scenario is intentionally kept as a distinct case, stop directing it at existing — route it through reasoning/comment only, or through prerequisites.create.

dependency explicitly in the comment text. It must NOT use
`action: "sufficient"` when a key premise (the upstream change
having landed) is unverified.

4. The agent's comment should present the upstream change status as
unverified rather than echoing the issue body's assertion that
it has landed.

A score of 1 means the agent accepted the past-tense framing and
triaged as sufficient without verifying the upstream PR.
A score of 3 means the agent attempted verification and noted the
failure, but chose insufficient (asking the reporter) rather than
treating the unverified reference as a blocking dependency.
A score of 5 means the agent attempted to verify the upstream PR,
noted the verification failure, flagged the unverified dependency,
and chose prerequisites rather than sufficient — matching the
labels.required assertion (blocked).
30 changes: 30 additions & 0 deletions eval/triage/cases/007-cross-repo-unmerged-dependency/input.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
forge: github
fixture:
type: issue
title: "Update config parser to handle new upstream schema format"
body: |
## Bug Report

**What happened:**
Now that the schema validation overhaul in nonexistent-org-eval-fixture/upstream-lib#42
has landed, our config parser is out of date. It still expects the old
`v1` schema envelope (`{"version": 1, "data": {...}}`) but since
upstream-lib switched to the flat format in nonexistent-org-eval-fixture/upstream-lib#42,
all config files using the new format fail to parse with:

```
ConfigError: unexpected key "schema_version" at root level
```

**Expected behavior:**
The config parser should accept both the old `v1` envelope and the new
flat format introduced by the upstream change.

**Steps to reproduce:**
1. Create a config file using the new flat format from upstream-lib
2. Run `./bin/parse-config config.yaml`
3. Observe `ConfigError`

**Environment:**
- OS: Ubuntu 22.04
- Version: main branch (latest)
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Config Parser

A configuration file parser that reads YAML config files and validates
them against a known schema format.

## Dependencies

- `upstream-lib` — provides schema definitions and validation utilities.
Config files must conform to the schema format defined by upstream-lib.

## Usage

```bash
./bin/parse-config config.yaml
```
Loading