OCR fallback via kind='ocr_text' artifact (concrete path for #13 OCR item)

[deOliveira-R]

Refines the OCR line-item in the v2 umbrella (#13) with a concrete path now that the artifact infrastructure (commit e2b7342) is in place.

The gap

Scanned PDFs have no extractable text. PyMuPDF returns empty pages → zero chunks → invisible to both keyword and chunk-level semantic search. Only the item-level SPECTER2 embedding (title + abstract) surfaces them at all, and that relies on Crossref/GROBID having given us an abstract — which usually fails for scans.

Plan

1. New CLI: `grimoire artifacts build --kind ocr_text [--since YYYY-MM-DD] [-j N]`.
2. Walks items where:
   • `primary` artifact exists, and
   • `extract.pdf.has_extractable_text()` returns False (body is empty/degenerate), and
   • no `ocr_text` artifact yet.
3. Each worker thread:
   • Pulls the primary CAS blob.
   • Runs ocrmypdf or pure Tesseract over it — the plan chose `ocrmypdf` (plan §4 tech-stack row).
   • Stores the plain-text result as the `ocr_text` artifact (source: `tesseract-` or `ocrmypdf-`).
4. Extend `index.py` so when chunking a PDF item:
   • Prefer `ocr_text` artifact over raw PyMuPDF extraction when the latter is empty.
   • Reuse the existing chunker; no schema change (page numbers become approximate/synthetic for scans).
5. Bonus: add a soft `metadata_confidence` reduction when the source is OCR so the dedup tier-4 is more cautious.

Acceptance

• `grimoire artifacts build --kind ocr_text` walks scanned PDFs, writes `ocr_text` artifacts.
• Indexer consumes OCR text when PyMuPDF gives nothing; scanned papers become searchable.
• Throughput is bounded: 30 s – 2 min per page as the plan warns; `-j N` parallelism respected.
• Non-scanned PDFs are untouched.
• Test with a synthetic image-only PDF (built via pymupdf with `insert_image` + a pre-rendered PNG of text) + a stubbed OCR callable; assert end-to-end behavior.

Notes

• The `ocrmypdf` dependency is heavy (pulls in Tesseract, poppler). Keep it in an optional extra (`[ocr]`) so it's not mandatory for users who don't need it.
• Processing cost is bounded per-scan and embarrassingly parallel — user has explicitly said processing isn't a concern.
• This closes the "invariant 7 (paper + DOI ⇒ venue)" soft-failure path for a lot of items where OCR could have provided a first-page abstract for Crossref backfill — worth measuring on the real library once landed.

Once this ships, update the corresponding checkbox in #13.

[deOliveira-R]

Smoke test on a 1982 scan (2026-04-21)

Ran ocrmypdf 17.4.1 + Tesseract on a 1982 NSE critical review (Sanchez & McCormick, "A Review of Neutron Transport Approximations", 56 pages, 4.3 MB image-only scan).

Results

• 56 pages, 38 s wall / 170 s CPU (~4-core parallel). 0.68 s/page — the plan's "30 s – 2 min per page" warning is stale for modern ocrmypdf on a modern Mac.
• pymupdf post-OCR: 235k chars total, median 4.2k chars/page, min 343 max 5.9k. Text layer is well-formed.
• Page 1: title, authors (R. Sanchez, N.J. McCormick), venue ("Nuclear Science and Engineering, Volume 80, Number 4, April 1982") all machine-readable. Crossref backfill should work for this class.
• Body prose: dense reactor-physics text came through cleanly. Greek letters and math glyphs get mangled (ψ(r, Ω) → W(r, 2), Ω → 2). GROBID's reference extraction cares about DOIs and author/year, not math, so downstream ref-linking should be unaffected.
• Cover masthead styling produces junk strings ("PPTENGINEERINGS"). Cosmetic; not in body chunks.
• Output PDF is slightly smaller (4.07 MB vs 4.51 MB) — ocrmypdf recompresses images.

Design note: ocr_pdf vs ocr_text

The current plan stores the artifact as plain text (ocr_text). Consider storing ocrmypdf's searchable PDF output (ocr_pdf) instead:

• Preserves layout AND a text layer that pymupdf and GROBID both consume natively. processFulltext needs a PDF, not a txt blob — a text-only artifact would need to be re-wrapped to feed into the GROBID pipeline.
• Page numbers stay real (the plan currently notes they'd be "approximate/synthetic" with ocr_text — with ocr_pdf they stay accurate).
• index._extract_pages works unchanged: swap the CAS path to the OCR artifact when the primary has no text layer.
• artifacts build --kind grobid_tei can prefer the ocr_pdf artifact when present, so the OCR → GROBID → section-aware-chunking chain works without special-casing.

Trade-off: storage is larger (~MB per paper, vs ~hundreds of KB for plain text). Given scan volume is medium-low, probably fine.

Install gotcha

Fresh brew install ocrmypdf on a Mac with a recently-bumped x265 (4.2) fails on first import: libheif is linked against libx265.215.dylib but x265 4.2 ships .216. Fix is brew reinstall libheif. Worth mentioning in install docs.

Scope estimate (revised)

• migration 005 (~15 lines)
• grimoire.ocr module wrapping ocrmypdf subprocess (~80 lines)
• CLI wiring in artifacts build (~30 lines)
• index._extract_pages + GROBID pipeline prefer the OCR artifact when present (~20 lines)
• optional [ocr] extra in pyproject.toml
• 3 tests (subprocess mocked; one end-to-end with a real tiny image-only PDF)

~half a day of focused work. Backfill runs after the GROBID backfill finishes so we don't contend for the same corpus.