Summary
The Evals Nightly workflow continues to fail all save_memory attempts after the reporting fix from issue 2605.
Example run: https://github.com/vybestack/llxprt-code/actions/runs/29981538815
The model successfully calls save_memory with the exact expected fact, but the eval compares the exact expected assistant answer against TestRig's complete human-readable process result. Successful plain-text runs can append stderr diagnostics to stdout, so a correct answer is reported as an output hundreds of characters long and fails the exact comparison.
Expected behavior
- The eval validates the assistant response independently from process diagnostics.
- A successful save_memory call followed by the exact answer $blue$ passes.
- Raw stdout, stderr, exit status, timeout state, and tool telemetry remain available in eval artifacts.
- JSON output remains intact when Podman telemetry is present.
Acceptance criteria
- Evals use the CLI's structured JSON output contract.
- The response field is schema-validated before assertions run.
- TestRig captures raw stdout and stderr separately without changing existing plain-text callers.
- Podman telemetry filtering preserves legitimate CLI JSON and user output.
- Deterministic FakeProvider integration coverage exercises save_memory and the exact answer.
- Process failures and timeouts preserve diagnostic captures.
Related to issue 2605.
Summary
The Evals Nightly workflow continues to fail all save_memory attempts after the reporting fix from issue 2605.
Example run: https://github.com/vybestack/llxprt-code/actions/runs/29981538815
The model successfully calls save_memory with the exact expected fact, but the eval compares the exact expected assistant answer against TestRig's complete human-readable process result. Successful plain-text runs can append stderr diagnostics to stdout, so a correct answer is reported as an output hundreds of characters long and fails the exact comparison.
Expected behavior
Acceptance criteria
Related to issue 2605.