Add new example - OCR for structured documents via proprietary LLMs

.claude/skills/datapipe-examples/SKILL.md

@@ -19,13 +19,14 @@ Each `examples/*` pipeline has a dedicated setup skill — pick by what it does:
 | `e2e_template/image_keypoints` | YOLO-pose keypoints + Label Studio → train → FiftyOne | **setup-e2e-template** |
 | `sam_cvat` | SAM3 text-prompt boxes+masks → CVAT pre-annotations | **setup-sam-cvat** |
 | `detection_tags` | YOLO detection + **tags** (per-scenario metrics), no Label Studio / FiftyOne, GT injected | **setup-detection-tags** |
+| `ocr` | Multi-LLM structured OCR (OpenAI/Gemini/Qwen) on passport/id-doc images → FiftyOne | **setup-ocr** |
 
 ## Ask first — don't assume (only the unresolved)
 Clarify what's not obvious before acting — don't spin up services or pull data you don't need. Ask only the unresolved:
-- **Demo or your own data?** · **Provision Postgres/services or reuse existing?** (e2e ships `docker compose`; embedder/sam need external Postgres)
+- **Demo or your own data?** · **Provision Postgres/services or reuse existing?** (e2e / detection_tags ship `docker compose`; embedder/sam/ocr need external Postgres)
 - **Which Postgres + which database** for `DB_URL`? Never point it at an existing DB or drop in a `localhost` default without confirming.
-- **Reuse an existing venv / `uv` env, or create a fresh one?** · **Which GPU?** (SAM3 >8 GB; DINOv2/YOLO ~8 GB) · **Annotation backend up?** (LS for e2e / external CVAT for sam)
-- **Surface stage logs or run quiet?** · **Per-tag scenario metrics** (retrain new case, old vs new)? → the `detection_tags` example / **setup-detection-tags**
+- **Reuse an existing venv / `uv` env, or create a fresh one?** · **Which GPU?** (SAM3 >8 GB; DINOv2/YOLO ~8 GB; OCR needs none — LLM APIs) · **Annotation backend up?** (LS for e2e / external CVAT for sam)
+- **OCR engines + API keys?** (`OCR_ENGINES` → openai/gemini/qwen) · **Surface stage logs or run quiet?** · **Per-tag scenario metrics** (retrain new case, old vs new)? → the `detection_tags` example / **setup-detection-tags**
 
 ## How to work
 Read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. Trust the status table (`*_training_status`/`brain_status`), not the exit code. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors) sent to a file and `grep`ped, rather than dumping the verbose output inline.
@@ -36,12 +37,13 @@ prompt / label config, the label field, and any class filter. A mismatch runs fi
 useful results. The per-example skill lists the exact knobs.
 
 ## Universal prerequisites (every example)
-1. **PostgreSQL** at `DB_URL`. embedder + sam_cvat need an EXTERNAL Postgres (they don't start one);
-   e2e_template bundles Postgres in its `docker compose`. Empty DB is fine; tables auto-create via
+1. **PostgreSQL** at `DB_URL`. embedder + sam_cvat + ocr need an EXTERNAL Postgres (they don't start one);
+   e2e_template / detection_tags bundle Postgres in `docker compose`. Empty DB is fine; tables auto-create via
    `datapipe db create-all`. `.env.example` default `...postgres:postgres@localhost:5432/postgres`.
    External quick start: `docker run -d -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=postgres -p 5432:5432 postgres:16`
    (k8s pod / no docker → conda postgres, `initdb` as non-root with `--locale=C`.)
 2. **A GPU big enough** (`nvidia-smi`): DINOv2/YOLO fit ~8 GB; SAM3 wants >8 GB (OOM'd on an 8 GB Pascal in our tests).
+   OCR needs no GPU (calls LLM APIs; needs API keys for enabled engines).
 3. **Annotation backend, if used:** e2e_template ships Label Studio in its `docker compose`; sam_cvat
    needs an external CVAT you provide (URL + creds + a project whose labels match its config).
 4. **`uv` + Python ≥3.10,<3.13.** Each example has a `pyproject.toml` → `uv sync` (cu124 torch pinned,
@@ -52,7 +54,8 @@ useful results. The per-example skill lists the exact knobs.
 ## HF auth (only for GATED models)
 Needed for `dinov3_*` (embedder, off by default) and `sam3` (sam_cvat). Accept the model license on
 its HF page, then `export HF_TOKEN=...` with **gated-repo read access**. A plain token → `403`, often
-masked as `LocalEntryNotFoundError: check your connection`. Public models (dinov2, YOLO) need no token.
+masked as `LocalEntryNotFoundError: check your connection`. Public models (dinov2, YOLO) and the OCR
+demo HF dataset (`ud-synthetic/printed-usa-passports`) need no token.
 
 ## Universal run shape
 ```bash

.claude/skills/setup-ocr/SKILL.md

@@ -0,0 +1,81 @@
+---
+name: setup-ocr
+description: >
+  Use when working in examples/ocr, or running / debugging the OCR→FiftyOne
+  datapipe example (multi-LLM structured OCR on passport/id-doc images).
+---
+
+# ocr (LLM OCR → FiftyOne)
+
+This skill = run multi-engine LLM OCR on YOUR passport/id-doc images and publish structured JSON to FiftyOne. The HF dataset (`ud-synthetic/printed-usa-passports`) is just a smoke-test; the real goal is your own images — set the knobs below first.
+
+**Ask first — don't assume (only the unresolved):** demo (HF passports) or your own images? **which Postgres + which database** for `DB_URL` — never point it at an existing DB or drop in a `localhost` default without confirming; reuse an existing venv / `uv` env or create a fresh one? **which OCR engines** (`OCR_ENGINES` → subset of `openai,gemini,qwen`; each needs its API key)? **which model names** (defaults below — keep or change in `config.py` `ENGINE_REGISTRY`)? **cost cap** (`HF_LIMIT` in demo mode; local mode = all images × all engines)? **custom output schema** (default `IdDocument` in `config.py` or user edits `OUTPUT_MODEL`)? surface stage logs or run quiet?
+
+**How to work:** read the setup, then propose a short plan and get a go-ahead before touching anything. Prepare `.env` and **pause for the user to verify it** before running. Run each stage with its logs shown and, after each, say what you did and what changed — don't run the pipeline silently. If a stage fails and the cause isn't clear from the normal logs, re-run it with `datapipe --debug … run` (or `--debug-sql` for SQL errors); debug is very verbose, so send it to a file and `grep` it (e.g. `datapipe --debug run > /tmp/dp_debug.log 2>&1; grep -nEi "error|traceback" /tmp/dp_debug.log`) rather than dumping it inline.
+
+## Run on YOUR data
+- **Your images:** `LOCAL_IMAGES_DIR=/path` (recursive `.jpg/.jpeg/.png/.webp/.heic`; file stem → `image_id`).
+  Local wins when the dir exists and has images (`use_local_images()` in `config.py`); otherwise HF fallback.
+- **Or HF:** `HF_DATASET_NAME`, `HF_SPLIT`, `HF_LIMIT` (caps download **and** LLM calls in demo mode).
+- **Engines:** `OCR_ENGINES=openai,gemini` (comma list). Model names + inference kwargs live in `config.py`
+  `ENGINE_REGISTRY` (not `.env`) — **ask which models to use**, then edit `ENGINE_REGISTRY[..]["model"]` if needed.
+  Current defaults (verify against `config.py`): `openai` → `gpt-5.4-nano`, `gemini` → `gemini-3.5-flash`,
+  `qwen` → `qwen3.7-plus`.
+- **FiftyOne dataset:** `FIFTYONE_DATASET_NAME` (default `datapipe_ocr`).
+- **Optional:** edit `OUTPUT_MODEL` in `config.py` — `ocr_prompt()` auto-follows the pydantic schema.
+
+## Prerequisites
+Stages (labels match `app.py`): `ingest` (list images + engines) → `ocr` (LLM inference, images × engines
+cross-join, `chunk_size=2`) → `fiftyone` (per-engine `{engine_id}_ocr` StringField via Caption Viewer).
+- **External PostgreSQL** at `DB_URL` (SQLAlchemy URL). Not started by the example; an empty DB is fine —
+  `datapipe db create-all` auto-creates tables. `.env.example` default `...postgres@localhost:5432/postgres`.
+- **LLM API keys** for each enabled engine: `OPENAI_API_KEY`, `GEMINI_API_KEY` (or `GOOGLE_API_KEY`), `QWEN_API_KEY`.
+  **Geoblock:** `openai` / `gemini` call the **native** OpenAI / Google APIs — blocked from Russian IPs. Workarounds
+  (proxy/router) are possible outside this example; from RU, prefer `qwen` (DashScope) or run from a non-blocked network.
+- **FiftyOne must run on the SAME machine** — samples live in local FiftyOne/MongoDB; remote → empty panels.
+- **Caption Viewer plugin** (required for readable JSON in the App):
+  `fiftyone plugins download https://github.com/harpreetsahota204/caption_viewer`
+- **No GPU needed** — OCR calls LLM APIs, not local inference. **`uv` + Python 3.10–3.12** → `uv sync`
+  (pins CPU `torch==2.6.0` for transitive cv-pipeliner; OCR itself doesn't use it).
+- Default HF dataset (`ud-synthetic/printed-usa-passports`) is public — no `HF_TOKEN` required.
+
+## Quick demo to verify setup
+Skip if you have data. Leave `LOCAL_IMAGES_DIR` unset → HF fallback (`HF_DATASET_NAME`=ud-synthetic/printed-usa-passports,
+`HF_LIMIT=10` in `.env.example` keeps LLM cost low); run §Run as-is to confirm install/DB/API keys/FiftyOne.
+
+## Run (from examples/ocr)
+```bash
+cp .env.example .env   # DB_URL, API keys, OCR_ENGINES, LOCAL_IMAGES_DIR or HF_*, FIFTYONE_DATASET_NAME
+uv sync && source .venv/bin/activate   # else prefix each command with `uv run`
+fiftyone plugins download https://github.com/harpreetsahota204/caption_viewer   # once per venv
+datapipe db create-all && datapipe run
+# one stage: datapipe step --labels=stage=ingest run  (ingest|ocr|fiftyone)
+```
+Run from `examples/ocr/` (`load_dotenv()` in `app.py` — wrong cwd breaks `.env`).
+
+## Cost note
+OCR runs a **cross-product** of images × enabled engines. Demo: `HF_LIMIT` caps both. Local: every image hits
+every engine — confirm image count and `OCR_ENGINES` before a full run.
+
+## View in FiftyOne (same machine)
+```bash
+fiftyone app launch --remote --address 0.0.0.0 --port 5151 --wait -1   # venv active; SSH-forward 5151
+```
+Open `FIFTYONE_DATASET_NAME`. Per engine: open a sample → add panel → **Caption Viewer** → field
+`openai_ocr` / `gemini_ocr` / `qwen_ocr`. Open multiple panels to compare engines side by side.
+
+## Success criteria
+- `ocr_results` table has a row per `(image_id, engine_id)` pair with populated `output_json`.
+- FiftyOne dataset has samples with `{engine_id}_ocr` fields containing structured JSON.
+- Caption Viewer panel renders the JSON readably.
+
+## Troubleshooting (may already be fixed — verify against current files)
+- **No images found** → empty/missing `LOCAL_IMAGES_DIR` and HF misconfigured; or local dir has no supported suffixes.
+- **No OCR engines enabled** → `OCR_ENGINES` empty or unknown IDs (must match `ENGINE_REGISTRY` keys in `config.py`).
+- **API key errors** → missing/wrong key for an enabled engine (`OPENAI_API_KEY`, `GEMINI_API_KEY`, `QWEN_API_KEY`).
+- **OpenAI/Gemini timeout / region / connection refused from RU** → geoblock on native APIs (see Prerequisites); use `qwen` or a non-blocked network/proxy.
+- **Runaway LLM cost** → forgot `HF_LIMIT` in demo mode, or large local folder with all three engines enabled.
+- **FiftyOne empty / mongod bind errors** → old `~/.fiftyone` datadir (FCV mismatch / port bind). Use a fresh
+  dir: `FIFTYONE_DATABASE_DIR=/tmp/fo_db`.
+- **JSON not pretty in App** → Caption Viewer plugin not installed (see Run).
+- **Schema mismatch after editing `OUTPUT_MODEL`** → re-run `datapipe step --labels=stage=ocr run` so engines pick up the new prompt.

examples/ocr/.env.example

@@ -0,0 +1,12 @@
+DB_URL=postgresql+psycopg2://postgres:postgres@localhost:5432/postgres
+OCR_ENGINES=openai,gemini,qwen
+OPENAI_API_KEY=replace-me
+GEMINI_API_KEY=replace-me
+QWEN_API_KEY=replace-me
+# local mode: use when dir exists and contains images (optional HF fallback)
+# LOCAL_IMAGES_DIR=./data/images
+HF_DATASET_NAME=ud-synthetic/printed-usa-passports
+HF_SPLIT=train
+HF_LIMIT=10
+DATA_DIR=./data
+FIFTYONE_DATASET_NAME=datapipe_ocr

examples/ocr/.gitignore

@@ -0,0 +1 @@
+data/

examples/ocr/README.md

@@ -0,0 +1,104 @@
+# OCR + FiftyOne Pipeline
+
+**Claude Code skill:** `/setup-ocr` — auto-loads when relevant, or invoke directly.
+
+Datapipe example for:
+- ingesting passport / id-doc images (local folder or Hugging Face dataset)
+- running OCR with multiple LLM engines (OpenAI, Gemini)
+- publishing pydantic `OUTPUT_MODEL` JSON to FiftyOne StringFields
+- viewing results with the Caption Viewer plugin
+
+## Data sources
+
+Supports **local folder** or **Hugging Face dataset** fallback.
+
+- Local: `LOCAL_IMAGES_DIR` with `.jpg`/`.jpeg`/`.png`/`.webp`/`.heic` inside (recursive scan)
+- HF fallback when local dir unset, missing, or empty: `HF_DATASET_NAME` (default `ud-synthetic/printed-usa-passports`)
+- HF images cached under `DATA_DIR/<dataset>/`
+
+Use `HF_LIMIT` to cap HF download size and LLM cost (ignored for local dirs).
+
+## Engine switch
+
+Single source of truth: `config.ENGINE_REGISTRY` + `.env` `OCR_ENGINES`.
+
+```text
+OCR_ENGINES=openai,gemini,qwen
+```
+
+FiftyOne tables and pipeline steps are generated from the same enabled engine list, so config and tables stay aligned.
+
+Model names and inference kwargs (LLM `temperature`, etc.) live in `ENGINE_REGISTRY` inside `config.py`, not in `.env`.
+
+## Prerequisites
+
+### PostgreSQL
+
+Set `DB_URL` in `.env`:
+
+```text
+postgresql+psycopg2://user:password@host:port/dbname
+```
+
+Create tables (use a fresh DB or drop old tables after schema changes):
+
+```shell
+datapipe db create-all
+```
+
+### API keys
+
+- `OPENAI_API_KEY` for `openai`
+- `GEMINI_API_KEY` for `gemini`
+- `QWEN_API_KEY` for `qwen` (DashScope compatible OpenAI API)
+
+### FiftyOne App
+
+First - after creating virtual env install the community plugin for readable JSON in the FiftyOne modal:
+
+```shell
+fiftyone plugins download https://github.com/harpreetsahota204/caption_viewer
+```
+
+Launch on the same machine as the pipeline:
+
+```shell
+fiftyone app launch --remote --address 0.0.0.0 --port 5151 --wait -1
+```
+
+To visualize JSON structured output in the App: open a sample → add panel → **Caption Viewer** → select `openai_ocr`, `gemini_ocr`, or `qwen_ocr`.
+
+## Run
+
+1. Copy env: `cp .env.example .env`
+2. Edit `.env` (DB, API keys, dataset, limits)
+3. Install deps: `uv sync`
+4. From this folder:
+
+```shell
+datapipe run
+```
+
+Or run stages by label:
+
+```shell
+datapipe step --labels=stage=ingest run
+datapipe step --labels=stage=ocr run
+datapipe step --labels=stage=fiftyone run
+```
+
+## Visualize in FiftyOne
+
+Dataset name: `FIFTYONE_DATASET_NAME` (default `datapipe_ocr`).
+
+| Engine | FiftyOne field | How to view |
+|--------|----------------|-------------|
+| `openai` | `openai_ocr` | Caption Viewer panel (pretty JSON) |
+| `gemini` | `gemini_ocr` | Caption Viewer panel (pretty JSON) |
+| `qwen` | `qwen_ocr` | Caption Viewer panel (pretty JSON) |
+
+Open multiple Caption Viewer panels to compare engine fields side by side.
+
+## Customize output schema
+
+Edit `OUTPUT_MODEL` in `config.py` (default: passport fields `name`, `surname`, `birthdate`). All engines use it as output schema. OCR instruction text auto-generated via `ocr_prompt()`.

examples/ocr/app.py

@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+import logging
+
+from dotenv import load_dotenv
+
+load_dotenv()
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
+
+from datapipe.compute import Catalog, DatapipeApp, Pipeline
+from datapipe.datatable import DataStore
+from datapipe.step.batch_generate import BatchGenerate
+from datapipe.step.batch_transform import BatchTransform
+
+import data
+import steps
+from config import DBCONN, ENABLED_ENGINES
+
+pipeline_steps = [
+    BatchGenerate(
+        steps.list_images,
+        outputs=[data.images_tbl],
+        labels=[("stage", "ingest")],
+    ),
+    BatchGenerate(
+        steps.list_engines,
+        outputs=[data.engines_tbl],
+        labels=[("stage", "ingest")],
+    ),
+    BatchTransform(
+        func=steps.ocr_inference,
+        inputs=[data.images_tbl, data.engines_tbl],
+        outputs=[data.ocr_results_tbl],
+        transform_keys=["image_id", "engine_id"],
+        chunk_size=2,
+        labels=[("stage", "ocr")],
+    ),
+]
+
+for engine_id in ENABLED_ENGINES:
+    pipeline_steps.append(
+        BatchTransform(
+            func=steps.make_build_image_data(engine_id),
+            inputs=[data.images_tbl, data.ocr_results_tbl],
+            outputs=[data.fo_tables[engine_id]],
+            transform_keys=["image_id"],
+            labels=[("stage", "fiftyone")],
+        )
+    )
+
+pipeline = Pipeline(pipeline_steps)
+
+ds = DataStore(DBCONN, create_meta_table=True)
+app = DatapipeApp(ds, Catalog({}), pipeline)