Chaptered pages #0: two-tier LLM worker pool (provider RPM + workload class)

[arozumenko]

Part of #287. Prerequisite for #290 (writer parallel sub-pages).

Why now

#290 makes the writer fire 3–5 sub-page LLM calls in parallel per chapter, multiplied across all chapters in a generation run. Without a pool we stampede the provider — 429s, cost spikes, and starved interactive Q&A. This issue lands the pool BEFORE #290 turns on the parallelism.

Design

Tier 1 — Provider RPM cap (hard floor)

• One asyncio.Semaphore per (provider, model) pair, lazily constructed by llm_factory.
• Bound from env: LLM_MAX_RPM_<PROVIDER> (defaults: openai=12, anthropic=6, ollama=4, custom=8).
• Every LLM call acquires this first. Hard cap on provider-side concurrency.

Tier 2 — Workload class (fair share)

• class_pool["ingestion"] — planner, writer, verifier, evidence build, code map. Bound LLM_INGESTION_SLOTS (default 8).
• class_pool["interactive"] — ask_codebase / ask_project / research / MCP. Bound LLM_INTERACTIVE_SLOTS (default 6).
• Acquired second, after provider. Sum can exceed provider cap (8+6=14 > 12) — intentional. Provider is the truth; classes are partition hints preventing class starvation.

Routing

create_llm() gains workload_class: Literal["ingestion", "interactive"]. Callers self-declare:

• wiki_service / planner / writer / verifier → ingestion
• ask_service / research_service / qa_service → interactive
• MCP / toolkit_bridge → pass-through (depends on caller's declared class)

Implementation

• PooledChatModel wrapper around BaseChatModel. Acquires both semaphores in ainvoke / astream (reverse-release on exit). Sync invoke proxies through asyncio.run only at the boundary.
• LangChain's with_retry(stop_after_attempt=3, wait_exponential_jitter=True) wraps the pool — provider 429s still backoff, but the pool prevents most of them.

Observability

• Stream events: llm.pool_saturated emitted when acquire wait > 500ms (operators see when bound is too tight).
• /health exposes per-pool active/queued/peak counts.

Acceptance

• Unit tests: provider cap enforced under load; class cap enforced; cross-class fairness (interactive bypasses ingestion queue).
• Integration test: 20 concurrent ingestion calls + 3 interactive calls — interactive p99 latency stays bounded.
• All existing callers route through the pool (no raw BaseChatModel.invoke survives).

Out of scope

• Token-aware throttling (TPM). RPM is the v1 control; add TPM later only if 429s still leak.
• Per-tenant fairness. Single-tenant deployment today; multi-tenant is a v2 concern.
• Cross-process pools (multi-worker uvicorn). Each worker has its own pool; provider cap should be tuned per total worker count.