Current corpus is ~35 hand-picked books (#10 seed list). Step one of the scaling ladder — get to ~1,000 English novels.
Cost & footprint
| Item |
Estimate |
| Ingestion total (~965 new books × ~$0.005 each) |
~$5 one-time |
| OpenAI embedding (text-embedding-3-small @ $0.020/M tokens) |
~$2 |
| Claude Sonnet 4.6 LLM colour (~600 tokens each) |
~$3 |
| Storage delta (books + features) |
~15 MB (Neon Free tier handles it: 0.5 GB) |
| Runtime |
~5 h at concurrency 2; ~1 h at concurrency 8 |
| OpenAI rate-limit tier |
Tier 1 (1 M TPM) is enough; auto-unlock to Tier 2 (5 M TPM) happens after $50 cumulative spend |
No Neon plan change. No OpenAI tier change. Single afternoon of pipeline time.
Two pieces of work
1. Sourcing the IDs
Hand-curating 1,000 books is overkill. Two viable sources:
For 1k, top-downloads is the simpler path. The list is already curated by user demand.
2. Skip-already-ingested in `scripts/seed-corpus.ts`
Critical for all future scaling tickets too — without this, every re-run of `pnpm seed:corpus` after we've grown the seed list would re-process every previously-ingested book and double-bill.
Today `seed-corpus.ts` blind-sends every event:
```ts
const events = SEED_BOOKS.map((b) => ({ name: "corpus/book.ingest", data: { gutenbergId: b.gutenbergId } }));
await inngest.send(events);
```
Change it to query `books` for existing `gutenberg_id`s first, filter `SEED_BOOKS` to the delta, and only send the new ones. The Inngest pipeline still upserts on conflict, but skipping at the script level saves the entire embedding + LLM cost for books already done.
Acceptance
Out of scope
Current corpus is ~35 hand-picked books (#10 seed list). Step one of the scaling ladder — get to ~1,000 English novels.
Cost & footprint
No Neon plan change. No OpenAI tier change. Single afternoon of pipeline time.
Two pieces of work
1. Sourcing the IDs
Hand-curating 1,000 books is overkill. Two viable sources:
For 1k, top-downloads is the simpler path. The list is already curated by user demand.
2. Skip-already-ingested in `scripts/seed-corpus.ts`
Critical for all future scaling tickets too — without this, every re-run of `pnpm seed:corpus` after we've grown the seed list would re-process every previously-ingested book and double-bill.
Today `seed-corpus.ts` blind-sends every event:
```ts
const events = SEED_BOOKS.map((b) => ({ name: "corpus/book.ingest", data: { gutenbergId: b.gutenbergId } }));
await inngest.send(events);
```
Change it to query `books` for existing `gutenberg_id`s first, filter `SEED_BOOKS` to the delta, and only send the new ones. The Inngest pipeline still upserts on conflict, but skipping at the script level saves the entire embedding + LLM cost for books already done.
Acceptance
Out of scope