Skip to content

test(evals): confirm the eval gate triggers, and show output when it fails#8

Merged
walkerhughes merged 1 commit into
mainfrom
test/eval-trigger-check
Jul 25, 2026
Merged

test(evals): confirm the eval gate triggers, and show output when it fails#8
walkerhughes merged 1 commit into
mainfrom
test/eval-trigger-check

Conversation

@walkerhughes

Copy link
Copy Markdown
Owner

Two purposes, one file changed.

1. Confirm the trigger

This PR touches only tastytrade-mcp/evals/validate_local.sh. If the tastytrade-mcp workflow runs and its Validate eval task verifiers step executes, the paths filter is confirmed to fire on eval-only changes. If the harbor-mcp jobs stay absent, per-subdirectory scoping is confirmed in the same run.

For reference, the step already ran green in #7, which touched tastytrade-mcp/README.md:

7. Validate eval task verifiers: success

This isolates it to an eval file specifically.

2. Make the gate useful when it fails

validate_local.sh discarded both runs with >/dev/null 2>&1, so a failure printed only a task name and two reward values:

FAIL  net-liq-value  (oracle=0 negative=0)

That is enough to know something broke and nothing about what. Diagnosing meant reproducing locally. The oracle, verifier, and negative-control output now go to files inside the existing $work tempdir and print, indented and tail-limited, only on failure. Passing runs are byte-identical.

Verified by breaking one oracle deliberately:

FAIL  net-liq-value  (oracle=0 negative=0)
      --- oracle ---
      simulated oracle failure: endpoint unreachable
      --- verify ---
      verifier error: [Errno 2] No such file or directory: '.../app/answer.json'
      reward=0
      --- negative ---
      verifier error: 'net_liquidating_value'
      reward=0

11 passed, 1 failed

The cascade is visible: the oracle exited non-zero, so no answer.json existed for the verifier to read. Restored, 12 passed, 0 failed.

The negative control was already sound and is untouched: it still requires reward 1 from the oracle and reward 0 from an empty answer, so a verifier that blindly awards 1 fails.

Opened to confirm the tastytrade-mcp eval gate triggers on a change confined to
evals/, and to make the gate useful when it does fail.

validate_local.sh discarded both runs with >/dev/null, so a CI failure printed
only a task name and the two reward values. Diagnosing meant reproducing it
locally. The oracle, verifier, and negative-control output now go to files inside
the existing $work tempdir and are printed, indented and tail-limited, only on
failure. Passing runs are unchanged.

Verified by breaking one oracle on purpose: the run reported 11 passed, 1 failed
and surfaced the real cause plus the resulting cascade (oracle exited non-zero,
so no answer.json existed for the verifier to read). Restored, 12 passed.
@walkerhughes
walkerhughes merged commit 26b8e75 into main Jul 25, 2026
1 check passed
@walkerhughes
walkerhughes deleted the test/eval-trigger-check branch July 25, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant