benchmarks: reproducible agent-accuracy comparison benchmark (Phase 1 MVP)#207
Draft
takashi-matsuyama wants to merge 2 commits into
Draft
benchmarks: reproducible agent-accuracy comparison benchmark (Phase 1 MVP)#207takashi-matsuyama wants to merge 2 commits into
takashi-matsuyama wants to merge 2 commits into
Conversation
…k (Phase 1 MVP) Adds a private, never-published benchmarks/ workspace package that measures how an AI agent's accuracy on business questions changes with the database context it is given, generalizing the quickstart's 4.8x revenue anecdote into a reproducible, falsifiable measurement. - 3-arm design (arm-pluggable): A0 raw DDL from the system catalogs, A1 raw COMMENT pass-through (Phase 2), A2 Kozou-compiled context driven from a real running `kozou mcp --http` via the MCP Streamable HTTP client. - Pre-registered task set (12 tasks: source-of-truth, soft-delete, status-enum, plus two no-trap controls) with ground truths verified in CI against the live quickstart fixture. - The agent answers with a single SQL statement; the harness executes it in a READ ONLY transaction as the fixture's read-only analyst role and scores the result, recording observed/expected ratios (miss severity), token usage, and full transcripts under results/. - Phase 1 runs A0 vs A2 on one model (claude-sonnet-5), 5 runs per cell; sampling parameters are unavailable on current models, so variance is reported across runs instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- string_set scoring now requires the exact row count, so join fan-out duplicates no longer score as correct (review MEDIUM) - the analyst role creation catches duplicate_object instead of check-then-create, surviving concurrent fixture loads on the shared CI Postgres (review MEDIUM) - the testcontainer is stopped even when the initial connect fails (review LOW) - per-task summary cells print the failure count next to the mean ratio so failure-heavy cells cannot look benign (review LOW) - README documents the intentionally-kept asymmetries raised by the review (tracked results/, A0 view names, A2 rowCountEstimate, effort default, scoring semantics) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a private, never-published
benchmarks/workspace package that measures how an AI agent's accuracy on business questions changes with the database context it is given — generalizing the quickstart's single "4.8x revenue" anecdote into a reproducible, falsifiable measurement.Design
Same model, same fixture, same prompt template, same tasks; the only independent variable is the context block:
kozou mcp --http, driven via the MCP Streamable HTTP clientanalystrole and scores against pre-registered ground truths (numeric tasks also record observed/expected ratios, i.e. miss severity).benchmarks/tasks/taskset.yaml; every ground truth is verified in CI against the live quickstart fixture.results/, null results are results) are documented inbenchmarks/README.md, along with intentionally-kept asymmetries raised in adversarial review.Review
Adversarially reviewed (review-only, design-challenging). Applied: exact row-count check in set scoring (join fan-out), race-safe
CREATE ROLEfor the shared CI Postgres, container cleanup on early connect failure, failure counts in per-task summary cells. Kept by design (documented in README): trackedresults/, A0 view-name visibility, A2rowCountEstimate,effort=high(the model's API default).Status
Test plan
pnpm --filter @kozou/benchmarks test(CI-safe, no API calls)pnpm -r typecheck,pnpm run lint,bash scripts/check-english-only.shall pass locally🤖 Generated with Claude Code