autodev: partner-eval comparison - #1

Open

Cl3mi wants to merge 70 commits into

mainfrom

Commits on Jul 24, 2026

chore(autodev): adapt plan paths for container + mark GPU steps [manual]
autodev worker
committed
chore: add vitest and better-sqlite3 dev deps for eval collector
autodev worker
committed
chore: add test script to package.json for vitest
autodev worker
committed
chore: add vitest.config.ts for eval collector test harness
autodev worker
committed
chore: verify vitest runner exits cleanly with no tests
autodev worker
committed
chore: mark Task 0 Step 5 (commit) complete in eval plan
autodev worker
committed
test(collector): add failing scoring.ts test suite
autodev worker
committed
chore(collector): verify scoring.ts test fails before implementation
autodev worker
committed
feat(collector): implement scoring.ts (Task 1 Step 3)
autodev worker
committed
chore(collector): verify scoring.ts test passes (Task 1 Step 4)
autodev worker
committed
chore: mark Task 1 Step 5 (commit) complete in eval plan
autodev worker
committed
chore(eval-dist): add seed questions.json fixture (Task 2 Step 1)
autodev worker
committed
test(collector): add failing test for questions.ts QuestionSet (Task 2 Step 2)

autodev worker
and
claude
committed
chore(eval-dist): verify questions.ts test fails (Task 2 Step 3)
autodev worker
committed
feat(collector): implement questions.ts QuestionSet (Task 2 Step 4)

autodev worker
and
claude
committed
test(collector): verify questions.ts test passes (Task 2 Step 5)
autodev worker
committed
docs(plan): check off Task 2 Step 6 (already committed in Steps 1-5)
autodev worker
committed
test(collector): add failing test for store.ts CollectorStore
autodev worker
committed
chore(collector): verify store.ts test fails (Task 3 Step 2)
autodev worker
committed
feat(collector): implement store.ts CollectorStore (Task 3 Step 3)
autodev worker
committed
test(collector): verify store.ts test passes (Task 3 Step 4)
autodev worker
committed
docs(plan): check off Task 3 Step 5 (already committed in Steps 2-3)
autodev worker
committed
feat(collector): serve bundled question set from /queries
autodev worker
committed
test(collector): verify /api/external/queries type-checks (Task 4 Step 2)
autodev worker
committed
docs(plan): check off Task 4 Step 3 (already committed as 621e1ee)
autodev worker
committed
feat(collector): add POST /api/external/run handler (Task 5 Step 1)
autodev worker
committed
test(collector): verify /api/external/run type-checks (Task 5 Step 2)
autodev worker
committed
docs(plan): check off Task 5 Step 3 (already committed as b79e940)
autodev worker
committed
feat(collector): assemble report.json from collector store (Task 6 Step 1)
autodev worker
committed
test(collector): verify /api/external/report type-checks (Task 6 Step 2)
autodev worker
committed
docs(plan): check off Task 6 Step 3 (already committed as 706eeea/cac34ca)
autodev worker
committed
chore(collector): remove obsolete judge/evaluate endpoints and pg external store

autodev worker
and
claude
committed
docs(plan): check off Task 7 Step 2 (verified only hooks.server.ts still imports storage, as expected)

autodev worker
and
claude
committed
docs(plan): check off Task 7 Step 3 (already committed as 00ed38f)
autodev worker
committed
feat(collector): mode-gated startup (sqlite collector vs postgres)
autodev worker
committed
docs(plan): check off Task 8 Step 2 (type-check + full test run pass)
autodev worker
committed
chore(autodev): skip sandbox-unrunnable local smoke tests (Node18/nested build)
autodev worker
committed
docs(plan): check off Task 8 Step 4 (already committed as 42c3d21)
autodev worker
committed
feat(collector): add export-questions script (Task 9 Step 1)
autodev worker
committed
chore(collector): remove old export-queries script (Task 9 Step 2)
autodev worker
committed
fix(collector): export-questions uses standalone pg client (Task 9 Step 3)
autodev worker
committed
docs(plan): check off Task 9 Step 4 (already committed across Steps 1-3)
autodev worker
committed
feat(collector): slim compose stack to single eval-server service
autodev worker
committed
chore(collector): remove obsolete ollama entrypoint and queries.sql

autodev worker
and
claude
committed
docs(plan): check off Task 10 Step 3 (already committed as 42d7f3a/5cec54e)

autodev worker
and
claude
committed
feat(collector): bundle seed/questions.json into runtime image
autodev worker
committed
feat(collector): assert better-sqlite3 loads in runtime image
autodev worker
committed
docs(plan): check off Task 11 Step 4 (already committed as 3a0e194/0507389)
autodev worker
committed
feat(collector): simplify .env.example for single-container flow
autodev worker
committed
docs(collector): rewrite README for single-container flow
autodev worker
committed
docs(plan): check off Task 12 Step 3 (already committed as 6200f3e/898dbd2)
autodev worker
committed
feat(compare): add shared report/judged types for compare feature
autodev worker
committed
test(compare): add failing test for judged-row aggregation
autodev worker
committed
docs(plan): check off Task 13 Step 3 (test fails as expected, module missing)
autodev worker
committed
feat(compare): implement aggregateJudged in aggregate.ts
autodev worker
committed
docs(plan): check off Task 13 Step 5 (aggregate.test.ts passes)
autodev worker
committed
docs(plan): check off Task 13 Step 6 (already committed as 3e038a1/5e5df06/0407f5b)
autodev worker
committed
feat(compare): add judge-report.ts to judge a report's rows
autodev worker
committed
docs(plan): check off Task 14 Step 2 (bun run check passes, 0 errors)
autodev worker
committed
docs(plan): check off Task 14 Step 3 (judge-report.ts already committed as 2d9cb5a)
autodev worker
committed
feat(compare): add store.ts to persist comparisons in postgres
autodev worker
committed
docs(plan): check off Task 15 Step 2 (bun run check passes, 0 errors)
autodev worker
committed
docs(plan): check off Task 15 Step 3 (store.ts already committed as 48d0089)
autodev worker
committed
feat(compare): add POST /api/compare endpoint
autodev worker
committed
feat(compare): add GET /api/compare/[id] endpoint
autodev worker
committed
docs(plan): check off Task 16 Step 3 (bun run check passes, 0 errors)
autodev worker
committed
docs(plan): check off Task 16 Step 4 (endpoints already committed as 8771431/ac64656)
autodev worker
committed
feat(compare): add /compare page server load for prior comparisons
autodev worker
committed
feat(compare): add /compare +page.svelte with upload + side-by-side UI
autodev worker
committed
docs(plan): check off Task 17 Step 3 (bun run check passes, 0 errors)
autodev worker
committed