feat: eval suites can score any dataset - #19
Merged
Conversation
…ored Both suites already accepted datasetDir/casesFile - the CLI just never passed them, and runEngineSuite() was called with zero arguments. This wires them up so a second, harder dataset is reachable without touching the committed defaults. - --dataset <folder>, --cases-file <path> and --glossary <path> on both suites. Note --cases stays the case-id filter, hence --cases-file for the file. - loadEvalDataset() is now shared by the engine and agent suites, so the two cannot drift on which dataset or curation the engine was grounded in. - --glossary applies committed curation through the Stage-1 path. The suites read no .datactx/ cache by design, so a glossary has to be passed explicitly. - Reports carry dataset/casesFile/glossary, and the filename includes the dataset (engine-dataset-<ts>.json). Two datasets would otherwise leave indistinguishable baselines side by side in .datactx/evals/. - eval:engine:hard and eval:ab:hard scripts, pointing at the dataset that Stage 3 adds. The committed dataset and cases stay the defaults: test/evals.test.ts asserts the engine suite scores exactly 1.0 there, and an agent test hardcodes 12650/9050 from the current CSV bytes. Proof the flag is honored rather than ignored: pointing the engine suite at fixtures/data drops it from 18/18 to 3/18, and a new test asserts exactly that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Stage 2 of 3 for the trap-dataset work. A harder dataset is useless if the harness cannot be pointed at it - and today it cannot:
runEngineSuite()is called with zero arguments (eval.ts), so its existingdatasetDir/casesFileoptions are unreachable, and the agent suite's equivalents are never forwarded either.What
--dataset <folder>,--cases-file <path>,--glossary <path>on both suites.--caseskeeps its existing meaning (case-id filter), hence--cases-filefor the file.loadEvalDataset()is now shared by the engine and agent suites, so they cannot drift on which dataset or curation the engine was grounded in.--glossaryapplies committed curation through the Stage-1 path. The suites deliberately read no.datactx/cache, so a glossary must be passed explicitly.dataset/casesFile/glossary, and the filename now includes the dataset (engine-dataset-<ts>.json) - two datasets would otherwise leave indistinguishable baselines side by side in.datactx/evals/.eval:engine:hard/eval:ab:hardscripts, pointing at the dataset Stage 3 adds.The committed dataset and cases stay the defaults, deliberately:
test/evals.test.tsasserts the engine suite scores exactly1.0there, and an agent test hardcodes12650/9050from the current CSV bytes.Verification
npm run checkexit 0;npm run test:cli138 pass / 0 fail (136 + 2 new)npm run eval:enginestill 18/18 on the default pairfixtures/datadrops it from 18/18 to 3/18, and a new test asserts that plus the recorded identity--glossaryfile reaches the grounding context the model actually sees (asserting on the captured system prompt), which is the Stage-1 chain working end to end through the harness🤖 Generated with Claude Code
https://claude.ai/code/session_01CKZ9UVgw2Mhiu6T5YAG7k6