Scale corpus to full English Gutenberg (~70,000 books)

[NoelHuibers]

Final step of the scaling ladder. Follow-up to #85 (50k). Adds the remaining ~20,000 English books on Project Gutenberg — the catalog has roughly 70k English-language entries.

Cost & footprint

Item	Estimate
Ingestion total (~20,000 new books × ~$0.005 each)	~$100 one-time
Cumulative ingestion spend (all steps from #83 to here)	~$350
Storage delta (cumulative ~1.0 GB)	Fits Scale ($69/mo, 50 GB storage) comfortably; over Launch's 10 GB by margin
Compute hours	If Inklings sees real traffic, Scale's 750 hours/mo can also be tight — Business ($700/mo) is the next step up, but unlikely needed at student-project scale
Runtime	~100 h at concurrency 2; ~25 h at concurrency 8. Roughly one day of pipeline time with full parallelism
OpenAI tier	Tier 3 (10 M TPM) already unlocked from #85; tier-up doesn't help further at this volume

What changes vs #85

• No new DB plan if you went to Scale at the previous step. (If you stayed on Launch, upgrade now — 1 GB exceeds the 10 GB plan with margin but compute hours probably matter more.)
• Catalog completeness — the full RDF dump becomes the source of truth. Filter to English + non-trivial length + has-text; the rest stays.
• HNSW `m` tuning — at 70k vectors, defaults still work, but increasing `m` from 16 → 24 can improve recall at the cost of memory. Worth measuring.
• Layout strategy — see Discovery: how to choose the blot shape per book #62 / Discovery: what should the blot detail panel show? #63 (open design questions). A full-Gutenberg UMAP at 70k points is a one-shot computation taking minutes; running it weekly is fine.

Acceptance

• Neon Scale (or higher) confirmed as the prod tier.
• Seed list / catalog parser covers the full English Gutenberg corpus.
• HNSW parameters re-evaluated; `ef_search` tuned if latency regresses.
• Layout recompute scheduled — weekly is enough at this scale.
• Total ingestion spend documented (~$350 cumulative from a fresh start).

Cost summary across the ladder

Step	Books added	Ingestion spend	Cumulative	Neon plan
#83	1,000	~$5	~$5	Free
#84	9,000	~$45	~$50	Free
#85	40,000	~$200	~$250	Launch / Scale ($19–$69/mo)
#86 (this)	20,000	~$100	~$350	Scale ($69/mo)

Numbers are one-time API spend, not monthly. The Neon tier price is the ongoing cost.

Depends on #85 (50k).