autodev: partner-eval comparison - #1
Open
Cl3mi wants to merge 70 commits into
Open
Conversation
Add passWithNoTests so `bun run test` exits 0 instead of vitest's default exit-1 when no test files exist yet.
Code for this step (package.json, bun.lock, vitest.config.ts) was already committed across Steps 1-4; nothing new to commit here.
Adds normalizeUrl, scoreRow, and summarizeRows pure functions reusing metrics.ts, satisfying the previously-failing scoring.test.ts suite.
…2 Step 2) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Confirms the QuestionSet test fails with "cannot find module ./questions" before the implementation is added.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Alias $env/dynamic/private to a process.env stub in vitest.config.ts so tests for modules importing SvelteKit's private dynamic env can run under vitest without the full sveltekit vite plugin (incompatible with this environment's Node 18 runtime).
…p 2) svelte-check reports 0 errors; remaining warnings are pre-existing in settings/+page.svelte, unrelated to this file.
Computes retrieval metrics for a submitted answer and persists the row to the SQLite collector store, keyed by session and system label.
Handler file and content were already committed in Step 1; this step's git add/commit was a no-op since the working tree was clean.
…ep 1) Replace GET /api/external/report to read rows from the SQLite collector store and summarize via scoring.ts, instead of the old Postgres-backed external storage module.
…ep 3) src/lib/server/db/client.ts pulls in $env/dynamic/private, which only resolves inside the SvelteKit/Vite runtime. export-questions.ts runs via plain `bun run` from repo root, so it now opens its own postgres connection via getDatabaseUrl() instead. Verified with `bun build eval-dist/scripts/export-questions.ts --target=node ...` -> OK.
Replaces the postgres+ollama+eval-server docker-compose.yml with a single judge-free, postgres-free, ollama-free eval-server service (Task 10 Step 1).
Compose stack now runs a single vendor eval-server service, so the Ollama entrypoint script and old queries.sql seed are dead weight. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ec54e) Steps 1-2 landed individually; Step 3 was just a checkbox flip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Task 13 Step 4: write the aggregateJudged implementation per plan.
Judges both reports, aggregates, and persists the comparison via saveComparison.
Renders the compare page: file upload for two report JSONs, POST to /api/compare, summary/failure-type tables, and a list of prior comparisons.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Autonomous implementation of
docs/superpowers/plans/2026-07-23-partner-eval-comparison.mdby the autodev worker on hetzner (containerautodev-web-rag, branchauto/web-rag).Commits land per plan-step. The two GPU/Ollama smoke-test steps (Task 17 Step 4, Task 19 Step 2) are tagged
[manual]— the worker pauses and pages via ntfy (autodev-web-rag) instead of running them on the GPU-less box.🤖 Generated with Claude Code