Scale corpus to 10,000 books

[NoelHuibers]

Step two of the scaling ladder. Follow-up to #83. Adds ~9,000 books on top of the 1,000 already ingested — only the delta gets processed thanks to the skip-already-ingested check from #83.

Hard prerequisite: proper Vercel + Inngest Cloud deploy

At 1k books, pnpm seed:direct --top=1000 --prod is enough — a local serial-ish backfill against the prod DB, no deploy needed. At 10k, that breaks down:

• ~50 h of single-machine runtime at concurrency 4 — your laptop can't stay open that long.
• A laptop hiccup mid-run leaves partial state; resuming costs more script babysitting.
• No automatic retries on transient failures (PG hiccup, Claude rate-limit edge case, etc.).

So before this ticket starts, the proper deploy needs to land:

• Next app deployed to Vercel (Pro plan — Hobby's 10s function timeout will kill the inline fetchBookText on big books).
• app/api/inngest/route.ts gets export const maxDuration = 300 so each step.run invocation has headroom.
• Inngest Cloud account with the app synced — visible under "Apps" with all 6 functions listed.
• Prod env vars set in Vercel: DATABASE_URL, OPENAI_API_KEY, ANTHROPIC_API_KEY, SESSION_SECRET, INNGEST_EVENT_KEY, INNGEST_SIGNING_KEY, NEXT_PUBLIC_APP_URL.
• Local .env carries INNGEST_EVENT_KEY so pnpm seed:all --top=10000 --prod can POST events to Cloud.

Once that's in place, this ticket runs against the deployed pipeline, not from a laptop.

Cost & footprint

Item	Estimate
Ingestion total (~9,000 new books × ~$0.005 each)	~$45 one-time
Storage delta (cumulative ~150 MB)	Still fits Neon Free tier (0.5 GB)
Runtime via Inngest Cloud + Vercel	~3–5 h with the Inngest concurrency limit of 2 on ingest-book. Bump to 4–8 if rate limits hold.
OpenAI rate-limit tier	Tier 2 (5 M TPM) auto-unlocks once cumulative spend passes $50 — this ticket will trigger it. Embedding throughput goes up 5×.
Neon compute hours	Free tier gives 191 compute hours/month with 5-minute auto-suspend. At 10k books + casual traffic, expect to bump against this if the database is queried continuously. Light traffic is fine.
Vercel	Pro plan (~$20/mo) — required for the 300s function timeout. Inngest invocations run as Vercel functions; Hobby's 10s cap is insufficient.

Still free Neon, but Vercel cost moves from $0 → $20/mo as a steady-state add. Worth it; Hobby plain doesn't work for this pipeline.

What's new vs #83

• The sourcing strategy changes. Top-1000 doesn't have 10k entries — switch to the full catalog RDF dump approach. Download once, filter to English, length ≥ 10k words, has plain-text version available; pick 10k by a simple heuristic (downloads × pyramid balance of decades).
• Bump ingest-book.ts concurrency from 2 → 4 or 8 to use the increased rate-limit headroom.
• HNSW index over 10k embeddings: built once during pnpm db:migrate or recomputed periodically. Vibe search latency stays well under 100 ms.

Acceptance

• Vercel Pro + Inngest Cloud prereqs all green (see checklist above).
• Catalog-RDF parser in lib/ingestion/gutenberg-catalog.ts (or similar) that streams the dump, filters, and outputs a typed list of ~10k.
• Seed list expanded to ~10,000 entries (or replaced with the dynamic catalog fetch).
• Re-run pnpm seed:all --top=10000 --prod — only the new ~9,000 IDs should hit the pipeline. Verify the skip-check from Scale corpus to 1,000 books #83 works at this scale.
• UMAP layouts (triggered by seed:all) finish in reasonable time — likely minutes, not hours.
• Document estimated cost in the run-log / RFC.

Out of scope

• Neon plan upgrade — only needed if compute hours become a problem (defer to Scale corpus to 50,000 books #85).
• Re-tuning HNSW m / efConstruction — defaults handle 10k easily.

Depends on #83.