feat: registered downloads + local VLM OCR (GLM/Unlimited) + aglaia ocr - #54
Open
yb85 wants to merge 37 commits into
Open
feat: registered downloads + local VLM OCR (GLM/Unlimited) + aglaia ocr#54yb85 wants to merge 37 commits into
aglaia ocr#54yb85 wants to merge 37 commits into
Conversation
Phase 1 of the download-registry refactor. Core assets and drop-in plugins now register fetchable targets in Python instead of a static model-list.json: - new aglaia/app_data/downloads.py: DownloadTarget + in-memory _REGISTRY + register_download()/registry()/target_for(); disk-truth is_downloaded(); reconciling download_status(); the sync urllib downloader; and the 5 core assets registered via _register_core_targets(). - db.py: new `downloads` table (key/status/sha/size_bytes/updated_at) + set/get/clear_download_status + download_statuses, mirroring the `plugins` table convention (absence of a row = never fetched). - models.py: reduced to a back-compat shim re-exporting the old names (ModelSpec→DownloadTarget, MODEL_SPECS, is_model_installed→is_downloaded, download_model, spec_for, _load_model_specs) so ModelDownloaderTab / OnboardingWizard / setup_cli need no change. - model-list.json deleted; removed from Aglaia.spec; paddle_vl comments updated. Disk stays ground truth: is_downloaded() is a cheap DB-free presence check; download_status() reconciles the table against disk (a model deleted out-of-band flips back to not_downloaded). platform field added for the upcoming MLX/vLLM backend filtering. 8 new tests in tests/test_downloads.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… onto it Phase 2 of the OCR refactor. Extracts paddle_vl's private _MlxVlmServer into a reusable, model-neutral serving layer so any local VLM (Paddle now; GLM / Unlimited next) runs locally on the best backend for the machine. - aglaia/workers/vlm/backends.py: VlmBackend protocol + MlxBackend (mlx_vlm.server, Apple Silicon) + VllmBackend (vllm OpenAI api_server, CUDA/other) + pick_backend() (platform preference, AGLAIA_VLM_BACKEND override). Each backend builds its own spawn argv + served-model-name. - aglaia/workers/vlm/server.py: LocalVlmServer.ensure(model_path, backend=…) — lazy spawn keyed per model, free-port pick, health-wait, log-tail through a pluggable sink, and the process-group SIGKILL teardown (killpg on the start_new_session group — avoids orphaning the backend's native threads). - paddle_vl.py: drops _MlxVlmServer (~226 lines) and routes through LocalVlmServer.ensure(..., backend=MlxBackend(), log=engine_log); pins MLX since its weights are MLX-4bit. Behaviour preserved (same argv, teardown, model-name pinning); timeout path now LocalVlmServer.stop(model_path). 12 new tests (backend selection, env override, command shapes, free-port, no-backend error, log routing) in tests/workers/test_vlm.py. Full suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ted-OCR Phase 3 of the OCR refactor. Adds two end-to-end document VLMs that run locally and auto-managed (MLX on Apple Silicon, vLLM on CUDA via the Phase-2 server). - openai_compat.py: OpenAiCompatVlmOcr base — declares mlx/vllm download targets, prompt, extra_body; does backend pick → LocalVlmServer.ensure → DPI downsample → chat → grounding-token parse → OcrResult. parse_grounded_markdown() strips <|ref|>/<|det|> to clean Markdown (meta.markdown for md_export) and decodes the det boxes to per-line bboxes (real searchable PDF text layer), with a full-page fallback. Per-page failures are isolated, not fatal. - glm_ocr.py (`--ocr glm`) and unlimited_ocr.py (`--ocr unlimited`): ~25-line config subclasses; each registers two download targets (*_mlx / *_vllm) tagged by platform. Unlimited carries the recipe's <image> prompt + skip_special_tokens + R-SWA xargs. Registered in ocr/__init__.py; both appear in `list ocr`. - Paddle stays separate (layout orchestrator, not whole-page chat). - vLLM is intentionally NOT a managed extra: it hard-pins torch/CUDA and conflicts with every other OCR stack (would destabilise the opencv pins), so it's a per-box `uv pip install vllm`; the VllmBackend only needs it importable. Also fixes a test-pollution bug: tests/test_downloads.py reloaded the downloads module, wiping engine-registered targets for later tests — now snapshots/restores _REGISTRY and relies on live env reads (no reload). 13 new tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unit tests stub the server + HTTP; this exercises the real spawn → health → chat path on actual hardware (MLX on Apple Silicon, vLLM on CUDA). Skipped unless AGLAIA_VLM_SMOKE=1; each engine self-skips when its backend/weights aren't ready. Run: AGLAIA_VLM_SMOKE=1 uv run pytest tests/workers/test_vlm_integration.py Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Model Downloader now mirrors its lifecycle (downloading / downloaded / failed) into the central downloads table via downloads.record_status — same as the CLI download path — so the registry's persisted status reflects GUI downloads, not just disk reconciliation. Also drops stale model-list.json references (message + comments) and three pre-existing unused imports. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For already-clean docs (born-digital PDFs, flat scans) that don't need dewarp/binarize/page-split. Ingests each page as the raw COLOR root node + a single branch pointing straight at it (no IntegratedProcessingChain — no worker pool, no geometric ops), then reuses run()'s OCR + export primitives verbatim. - workers/headless.py: run_ocr_only() + chain-free ingest (_ingest_images_no_chain / _ingest_pdfs_no_chain via _persist_raw + BranchRepo.upsert). A synthetic zero-step "ocr-passthrough" pipeline_version backs the nodes. Pass a .agl to re-OCR (or --check-ocr to poll a Mistral batch). - cli/commands/ocr.py + cli/shared.py ocr_config() + registered in cli/__init__.py (KNOWN_COMMANDS + app). Same OCR/export flags as run, minus --pipeline/--workers/--force-proc; engine defaults to auto. - app.py: _run_ocr_only handler (no setup gate — OCR needs no detection model). - Docs: README, docs/cli.md (new `ocr` section), CLAUDE.md. - Tests: chain-free ingest yields exactly one OCR-able branch (= raw page), engine defaults to auto, no-inputs → exit 2. Verified end-to-end with Apple Vision (image → md round-trip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The MLX VLM backend is no longer tied to the `paddle` extra — it's the serving layer for all local VLM OCR (glm / unlimited / paddle_vl), so move it to the platform extras: - `macos` += mlx-vlm (arm64 marker) → `--extra macos` now ships a working local VLM backend out of the box. - `cuda` += vllm (linux marker) → `--extra cuda` ships the Linux VLM backend (GPU with a CUDA torch, CPU otherwise via the pytorch-cpu index). - `paddle` = just paddleocr + paddlepaddle (the layout orchestrator); PaddleOCR-VL now needs `--extra macos --extra paddle`. Platform markers keep mlx-vlm (darwin) and vllm (linux) on disjoint splits, which dissolves their llguidance clash. Only surya↔cuda remains a hard conflict (vLLM's torch/openai≥2 vs surya-ocr's torch≥2.7/openai<2) — declared via [tool.uv] conflicts so the universal lock splits them. Also pin opencv-python-headless to 4.11 alongside opencv-python so a re-lock can't reintroduce the cv2 version skew. Engine "backend missing" hints + README/CLAUDE now point at `--extra macos` / `--extra cuda`. Verified: universal lock resolves, cv2 stays 4.11, mlx-vlm installs via macos, pick_backend()→mlx on this machine. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
paddleocr pulls opencv-contrib-python, a third distribution writing the shared cv2/ dir; unpinned it installed 4.10 and clobbered cv2/ to a half-written/skewed state (override only pinned opencv-python + -headless). Pin all three to 4.11.0.86 so the shared cv2/ stays one consistent payload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surya 2 is a Qwen3.5-VL document model, so serve it through the shared LocalVlmServer like glm/unlimited instead of the old surya-ocr + llama-server + GGUF path (which pinned torch / huggingface-hub<1 / openai<2 and was mutually exclusive with the MLX/vLLM stack). - surya.py: 485 lines → a ~20-line OpenAiCompatVlmOcr subclass (name unchanged). MLX on Apple Silicon, vLLM on CUDA. Output is HTML-flavoured Markdown. - downloads.py: replace the GGUF `surya` target with `surya_mlx` (aglaia-models/surya-ocr-2-mlx, pinned to an immutable commit) + `surya_vllm` (datalab-to/surya-ocr-2). DownloadTarget gains a `revision` field; the downloader fetches /resolve/<revision>/ (commit-pinned = immutable). - pyproject: drop the `surya` extra (no more surya-ocr/torch) AND the `conflicts` declaration — with torch-surya gone, surya↔cuda no longer clashes. - setup_cli / OnboardingWizard / test_downloads: surya → surya_mlx key. The MLX weight is plain safetensors + configs (no executable code), pinned by sha — verified mlx_vlm.server serves it on an OpenAI /v1 endpoint and reads a test page. Full suite green; nothing imports the surya-ocr package anymore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1) CLI downloader is now resumable (HTTP Range): a stalled read on slow/flaky Wi-Fi times out (60s) and we resume from the .partialdl byte offset instead of restarting, retrying up to 6×. Fixes the "times out at 300 MB, loses everything" failure on the ~800 MB VLM weights (mirrors the GUI downloader). Handles 416 (already complete) and a server ignoring Range (200 → restart). 2) Surya emits HTML (<p>/<table>/<div>); add OpenAiCompatVlmOcr.output_html + html_to_markdown() so its .md is clean Markdown (tables → Markdown tables), not raw HTML soup. Uses html2text (pure-Python, no compiled extension → bundle-safe for the frozen .app), with a tag-strip fallback if absent. Tests: Range resume / 416 / retry-after-timeout (mocked urlopen); HTML→MD paragraphs + tables + the surya engine's output_html path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OCR is valuable, and we now have many engines — so keep ONE layer per engine per branch (rerun same engine replaces; different engines coexist) and let export pick which layer. - OcrRepo: branches_needing_ocr(engine=…) evaluates "needs OCR" per engine, so re-running a different engine adds its layer instead of being a no-op (the old CLI no-op). finish() prunes superseded same-engine done rows → max 1 per engine. available_ocr_layers() lists engines with a done layer, latest- generated first (MAX(id)). Stale/fresh icon path (branch_status_map, mark_stale_for_engine_switch) left untouched. - Export takes an engine selector (default = latest layer, back-compat): write_markdown(engine=…) + ocr_engine_suffix(engine=…); create_pdf_from_db( engine=…) → _ocr_results_for_rows(engine=…). - CLI: `--export "pdf:g4:ocr=surya+md:ocr=apple"` (engine via the existing param parser; aliases honoured). _run_exports validates the layer and lists available ones on a miss. _run_ocr / OcrWorker pass the engine to branches_needing_ocr. - Surya HTML→MD now uses pad_tables=True (canonical GFM tables). GUI layer selector is Slice B. 5 storage tests; full suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Slice B of multi-engine OCR. ExportTab gains an "OCR layer" combo (populated from OcrRepo.available_ocr_layers, latest-generated first; hidden until ≥1 layer exists) applying to both the PDF text layer and Markdown. MainWindow populates it on every OCR-state refresh and threads the selection into create_pdf_from_db (via _run_pdf_maker) / write_markdown / ocr_engine_suffix. "Latest layer" (None) keeps the prior behaviour. Stale/fresh icon path untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The status bar shares _done_snaps between the pipeline's mark_done (branch_ready, deduped by scan_id) and OCR's mark_tick. With multiple OCR engine layers now kept, a re-OCR runs while the chain may still emit branch_ready — those mark_done()s piled into the same set as the OCR ticks, so the OCR bar read prior+current (e.g. 322 + 12 = 334/322, hitting 100% in 1s). mark_tick already kept a private monotonic _tick_count; the displayed count just read len(_done_snaps). Add a _tick_mode flag (set by mark_tick, cleared by reset) and a _done_count() that returns _tick_count in OCR mode — so a concurrent pipeline mark_done can't inflate an active OCR pass. ratio / label / is_finished / force_complete all go through _done_count(). Test: 12 OCR ticks interleaved with 12 pipeline mark_done → done stays 12. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The local VLM (Surya/glm/unlimited) runs on the GPU/ANE, not CPU, and a small model rarely saturates it — yet pages were sent to the server strictly serially, leaving throughput (and the M-series GPU) underused. Add OpenAiCompatVlmOcr concurrency: dispatch N pages to the server in parallel via a ThreadPoolExecutor (urllib releases the GIL on the socket read), order-preserving. Default 1 (safe, serialized); tune via AGLAIA_VLM_CONCURRENCY or `--ocr <engine>:concurrency=N`. Per-page errors stay isolated (a concurrent failure → empty page, not an abort), so 2–3 is safe to try even though a single mlx_vlm.server's continuous batcher can choke on mismatched-size concurrent prompts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ncy knob)" This reverts commit 3a5b2f5.
The real cause of "OCR · 322/322 in 3s": the chain is idle for the whole OCR run (OCR doesn't use the pipeline), so the pipeline-idle watchdog (_maybe_force_progress_complete → force_complete) fired after ~6s and snapped the bar to N/N — but the bar was showing OCR. force_complete is now a no-op in OCR (tick) mode; OCR drives its own completion via mark_tick / _on_ocr_finished. (Also drops the earlier mistaken tick-count snap in force_complete.) Tests: force_complete is a no-op mid-OCR, still reconciles in pipeline mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…complement The Apple Document engine offloads scripts Vision can't read (e.g. Greek) to a "complement" engine, but the eligible set was hardcoded to surya/paddle in two places (apple_docs._VALID_COMPLEMENTS + OcrTab._COMPLEMENT_CHOICES), so the new local VLMs (glm / unlimited / the MLX Surya) couldn't be picked. Add a DirectBlockOCR trait (engine recognises a cropped block directly — no layout pass) + engine.direct_block + direct_block_engines(). Compose it onto OpenAiCompatVlmOcr (glm/unlimited/surya) and paddle_vl. apple_docs now validates the complement against the registered DirectBlockOCR engines (+ none) and the OCR tab's complement dropdown lists every AVAILABLE one with its display name — so a new qualifying engine appears automatically, no edit. Default stays surya. Tests: trait membership (VLMs in, Vision/cloud out), resolve_complement accepts any direct-block engine + env + default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OCR runs in its own QThread, but every completed page fired ocr_state_changed → a main-thread refresh that runs branch_status_map() DB joins (twice — also in _update_ocr_frame_state), available_ocr_layers(), and repaints every scan widget's badge. The VLM completes pages in bursts of 4, so 4 of these heavy refreshes fired back-to-back → visible UI hitches. Coalesce: _on_ocr_state_changed now (re)starts a 350 ms single-shot timer whose timeout runs the actual refresh (_refresh_ocr_ui + _update_ocr_frame_state + _refresh_alt_views_if_visible) once. The progress tick stays immediate (cheap). No correctness change — badges just settle ~350 ms after the last event. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OCR counts per branch (one branch ≈ one page; a 2-up photo is 1 scan but 2 OCR units), so the final-state "N/N · …s/scan" was misleading. Use "page" in OCR (tick) mode, keep "scan" for the pipeline (which counts whole scans). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ETA used a moving average over the last 8 completion timestamps, but the VLM completes pages in bursts of 4 (a batch returns all at once, then a gap), so the windowed rate swung wildly across batch boundaries — "chaotic and not true". Switch to cumulative throughput (elapsed-since-start ÷ done × remaining): smooth, converges to the real pace, includes the one-time warmup so it settles rather than jumps. (Surya's model load is fast, so it's accurate within a few pages.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three OCR-correctness fixes surfaced by the delbrel-colloque test run:
- Surya `.md` was layout JSON, not text. The generic "convert to Markdown"
prompt triggered Surya 2's *layout* task ([{label,bbox,count}] JSON we were
exporting as garbage). Use its exact HIGH_ACCURACY_BBOX_PROMPT (OCR→HTML
<div data-bbox> blocks) + output_html so html_to_markdown yields real text.
- apple_docs + a recognition-only VLM complement (Surya/GLM) could splice a
repetition-loop hallucination into the result — observed 390× "(N) Ibid.,
p. 188." polluting an export (the loop, crammed onto the block's first line
by _split_block_text's count-mismatch path). Add _complement_degenerate:
reject output that explodes the block line count or collapses to few
distinct lines (after stripping a leading enumerator, so real footnotes
whose page numbers vary survive). Fail-safe → keep Vision's text.
- The "VLM OCR; dense pages can take a few minutes" / "loading the model"
progress hint fired for every engine, including fast Apple Vision. Gate it
on a new served_vlm trait (True on the local-server VLMs + PaddleOCR-VL).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The indeterminate phase before the first OCR tick is the local VLM model spinning up (or the Cloud whole-doc round-trip), so 'loading…' is more precise than 'working…' and matches the worker's first-run log line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DBnet split scans 26/31 of morales-slim wrong: the top-left page number became the "left page" and the whole spread the "right page". Apple/EAST were fine. Two root causes, both fixed: - merge_layouts grouped boxes by pairwise x-OVERLAP, starting a new group at ANY gap. DBnet placed the page-number box flush against (touching, 0 px) the body column, so it formed its own group → 3 groups, max_pages=2. Replace with xy_cut: the X-projection step of the classic recursive XY-cut. A column boundary is a vertical whitespace gutter wider than gutter_min_frac (2% of page width), not the mere absence of overlap — so a flush page number stays in its column. Single interpretable knob vs the merge scorer's weights; Y-cut omitted (a page is vertically continuous). - smart_merge's capacity-forced branch merged the highest-scoring pair, which on an over-split spread fused the two REAL pages. Now it absorbs the SMALLEST region into its best-scoring neighbour (small→large), keeping real pages intact. Belt-and-suspenders with xy_cut for >2-region cases. Verified on morales-slim scans 26/31: both backends now yield clean left/right pages, page number riding with the left page; Apple unchanged. Adds tests/processors/test_xy_cut.py + a smart_merge speck regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some pages early-exit the optimiser at the cold seed cubic_slopes=[0,0] — a flat local optimum → the page is left undewarped while its facing page fits real curl (morales-slim 163b: branch A curl≈0.18 in 1080ms, branch B ≈0 in 181ms). Curl varies slowly across a book, so seed it from recent same-side fits to lift the optimiser into the right basin. - Worker-local cache `_warm_curl[page_side] -> ring of (cubic_slopes, spline extras)`. Seed read in `_build_dewarp_problem` (feeds BOTH the inline and the batched paths); stored in `process()` + `apply_result`. Transfers ONLY the global shape — pose (rvec/tvec) + per-span keypoints stay page-specific. Respects the shape-bucket batcher (seed set before submit, cache updated as results return); no cross-process state. - Robustness: seeding the SINGLE last fit let a one-off over-curl compound page-to-page into a runaway (127a/132a/136a/137a/138a). Seed is now the median of the last 3 fits — componentwise median scaled to the MEDIAN magnitude (a lone outlier moves neither) — then hard magnitude-capped (_WARM_CURL_MAX=0.5). The cap bounds only the SEED, not the fit: a legit strong-curl page can still climb past it, a runaway can't START astray. Non-finite fits are dropped before they poison the median. Tests: test_dewarp_warmstart.py (seed/store, median-ignores-runaway, cap, side isolation, non-finite). test_dewarp_batchable.py now uses fresh instances per path (warm-start is cross-page state, broke the byte-identity invariant). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ache The round stage-toggle on a thumbnail silently stopped working after any (re)process. cell_disable_states is memoised per scan and keyed by NODE id, but invalidated ONLY in set_step_disabled. A (re)processed scan gets fresh node ids, so the cached map missed every new node → the toggle resolved to (toggleable=False) and locked → and since a locked button can't fire set_step_disabled, the only invalidator never ran: permanent lockout. Drop the scan's cached entry on branch_ready (fires when a branch finishes (re)processing — exactly when its node ids change). Boot stays fast: loaded scans don't fire branch_ready, so their cache stays warm. Test: test_step_toggle_cache.py — branch_ready invalidates the stale entry (leaving other scans'), and the no-cache path is safe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s alias
The debug viewer showed "no debug renderer for this step" for the layout step
(its page-bbox overlay never appeared — noticed while diagnosing a DBnet
mis-split). _RENDERERS is keyed by processor CLASS name ("PageDetector"), but
a node stores the CONFIGURED step name, and the shipped pipelines name the
step "LayoutDetector" → _RENDERERS.get() missed → _default_renderer.
Resolve the renderer by mapping the configured name through the registry to
its class name when the direct lookup misses (generic — fixes any aliased
step). Extracted as _resolve_renderer for testing.
Test: tests/storage/test_debug_renderer_alias.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the column-edge fit collapses, the trap rectifies a sliver quad and shears the page — a visible slant (morales-slim 288_A: quad aspect 0.23 vs ~0.6-0.9 for good pages). Add two self-checks that fall back to passthrough (identity) instead of emitting a skewed page; same philosophy as the dewarp warm-start guards — when the estimate is unreliable, don't apply it. - min_column_aspect (0.5): reject an implausibly thin column quad. Catches the obvious slant case; across all 586 trap pages this discards only 2.2%, all genuinely degenerate (good pages sit well above 0.5). Tighter than the pre-existing max_aspect_ratio floor (1/5 = 0.2) that 0.23 squeaked past. - max_added_tilt_deg (1.5): push the detected baselines through the correction and reject if it ADDS median tilt beyond the bound — guards the rotation failure mode (a correct keystone keeps text horizontal). Both gates run before the warp (cheap scalar checks). The subtle extrapolation-slant cases (geometrically-fine quad, slant only in the footer projected past the column) are deliberately NOT gated — their only signals overlap ~10-18% of good pages. Tests: test_trap_autodiscard.py. Validated against morales-slim 288_A (discarded), 145_B / 26_A (kept). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e timing Two fixes surfaced while benchmarking OCR configs on a 24-page corpus. 1. apple_docs export filename now folds in the complement engine. apple_docs alone / +surya / +glm produce DIFFERENT text but all got `_apple_docsOCR` → they silently overwrote each other. ocr_engine_suffix now appends the dominant complement (from result meta.complement_used): `_apple_docs_surya OCR`, `_apple_docs_glmOCR`, vs `_apple_docsOCR` when none. 2. Headless OCR timing separated model-load from page processing. The old per-page timer charged the first page with the whole model spin-up (VLMs load the server inside recognize). Added an OcrEngine.warmup() hook (no-op base; the served VLMs ensure their LocalVlmServer), called + timed before the page loop, and a steady-state summary: "model load Xs" + "N page(s) in Ys (mean/median s/page)". Apple Vision reports ~0 s load, as expected. Tests: test_ocr_export_naming.py (complement folded in, variants distinct, plain engines unaffected, dominant-engine path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tral) The headless OCR driver looped recognize() per page. For Mistral that meant one uploaded PDF — one billed API call — PER PAGE (24 calls for 24 pages), instead of the single whole-document request the engine is built for (and the GUI's OcrWorker already uses via recognize_rows: all pages → one PDF → one /v1/ocr call). Route any engine exposing recognize_rows() (the whole-doc interface) through a single recognize_rows call: assemble every page's image row, one request, distribute the per-page results back. Per-page engines (Apple, Surya, GLM, Paddle, Unlimited) are unchanged. Timing reports the one request + amortized s/page, kept separate from model load. NOTE: the CLI and GUI still have two separate OCR drivers (_run_ocr here vs OcrWorker) — this fix aligns the cloud path, but the divergence as a whole warrants an audit/unification (they should share one driver). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_run_ocr called recognize() without src_dpi, so downsample_to_dpi saw src_dpi=0 and used its coarse 12-inch longest-edge fallback — which barely shrinks a 300-dpi page. Result: the CLI OCR'd ~2.25× the pixels (4.1 MP vs the 1.8 MP a true 200-dpi downsample gives) at the wrong resolution, ~2× slower than the GUI (which passes the real dpi). Pass images.dpi as src_dpi. With AGLAIA_OCR_DPI honoured end-to-end, a 100/150/200 dpi sweep now actually changes the pixels the engine sees. Another CLI/GUI divergence (see audit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oops (#56) Greedy decoding (temp 0) + no repetition penalty + 8192 max_tokens let the served VLMs fall into a token loop on low-context crops (an apple_docs complement block), emitting the same line until they exhaust the budget — e.g. "(195) Ibid., p. 188." ×100. The _complement_degenerate guard discarded the garbage but the runaway generation still ran to ~8192 tokens, which is what made the complement path crawl / time out (#55 is a consequence of #56). Send repetition_penalty=1.15 + repetition_context_size=64 on every VLM chat request (the MLX server honours both via make_logits_processors). Moderate penalty over a short window breaks pathological long-sequence loops while leaving OCR's legit short repeats (spaces, digits, "Ibid.") intact; temp stays 0.0 (deterministic). Per-engine tunable via the class attrs / extra_body. Validated on the exact config that timed out (apple_docs + surya, 24 Greek pages): >1200s TIMEOUT → 218s complete (9.1s/page); degenerate-loop warnings many → 1; corrections now real (e.g. "4. Ibidem, q 39 a 8 arg 1."). Closes #56. Largely resolves #55 (the slowness was the loop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Benchmark of every OCR engine x {100,150,200,300} dpi on a French+Greek
corpus (docs/ocr-benchmark.md): timing (model-load vs page), word-overlap
accuracy vs native-300 and vs mistral@300. Conclusion: a single global 200
dpi default — 300 buys nothing over 200, 200 is safe for the local VLMs, and
Apple loses only 0.14 s/page vs 100.
Make the CLI match the GUI: resolve_ocr_dpi default 150 -> 200, and add a
--ocr-dpi override to the ocr command (sets AGLAIA_OCR_DPI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apple Vision renders ancient Greek as confusable CYRILLIC — and often with HIGH confidence, so the confidence-only gate leaks those misreads uncorrected (measured: ~10 of the residual Cyrillic lines had confidence >= 0.70). A line carrying letters from a script the document's languages don't cover is a near-certain misread — measured 100% precision / ~96% recall on this corpus — so it should go to the complement regardless of confidence. Add _has_unexpected_script(text, languages): flags a line with >=2 letters outside the scripts its OCR languages imply (Cyrillic in a fr+el doc). Wire it into the complement gate alongside the confidence test. Innocent of the loop guard: legit Greek/French never trips it; a Russian doc's Cyrillic is expected. NOTE (separate finding): the Cyrillic is entirely VISION's, not the VLMs' — standalone glm/surya emit 0-12 Cyrillic chars vs Vision's 2200+. For Greek-heavy material a standalone VLM beats Vision+complement outright; this fix improves the complement for MIXED (mostly-Latin) documents. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…guage UI
Replace the domain-default (Latin/Greek) garbage detector with a general,
script-level module backed by langcodes (CLDR) + regex \p{sc}. Apple Vision,
handed a script it can't read, falls back to a confusable one it can — ancient
Greek → Cyrillic — WITH HIGH CONFIDENCE, so the confidence gate leaks it.
- aglaia/workers/ocr/text_scripts.py: scripts_for_language() maps any BCP-47 /
ISO 639-1/2/3 code (explicit script subtag or CLDR likely-subtags, aggregate
writing systems expanded: ja→Han+Hira+Kana, ko→Hang+Han; grc→Grek quirk fix)
to ISO 15924 scripts; has_unexpected_script() flags a line with >=2 letters
outside the chosen languages' scripts (regex V1 set subtraction). Coverage is
the full ISO/Unicode set — plugin VLMs and unknown-language/known-script docs
work. NO domain default: empty languages → no judgement.
- apple_docs complement gate is now a DOUBLE gate: confidence < 0.7 OR
unexpected script.
- OpenAiCompatVlmOcr: append the chosen languages to the prompt (via
langcodes display names) so surya/glm/… know which scripts to expect.
- UI: language picker reactivated for the VLMs (was Apple-only) — it feeds
Apple recognition + the VLM prompt hint + the script gate. Dropped the
"(Apple only)" caption; LanguageTagInput now offers a 141-language completer
and accepts ANY valid BCP-47 tag (tag_is_valid); 'Auto' only for cloud
engines that ignore languages (Mistral).
Deps: langcodes + language_data. Tests: test_text_scripts.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When >30% of a page's lines are flagged for the complement (low confidence OR unexpected script), Vision is failing on the page as a whole, so one clean whole-page pass by the complement VLM replaces the per-block re-OCR + splice (fragile, and the VLM reads a full page cleanly anyway). Below the threshold, targeted block re-OCR keeps Vision's fast/good majority. Threshold 0.30, tunable via FULL_PAGE_COMPLEMENT_FRAC. Tests: test_complement_fullpage.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The complement's block re-OCR was leaking: on a line-count mismatch, _split_block_text puts the whole corrected block on the first line and blanks the rest, but the splice did 'if not txt: continue' — leaving those blanked lines as Vision's original (Cyrillic) text. So most re-OCR'd lines kept their garbage even though the block WAS corrected. Set every block line (clearing the blanks); the corrected text is already on line 1. End-to-end on the Greek corpus (apple_docs+glm@200): Cyrillic chars 2232 (Vision) → 901 (old complement) → 876 (+script gate) → 12 (+this) — a 99.5% reduction, 5 residual lines (isolated single confusables below the 2-char gate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two "engine doesn't work" bugs found while benchmarking:
- unlimited: the prompt began with a literal "<image>", but the OpenAI-compat
server ALSO inserts an image token for the image_url content part → the chat
template counted 2 image tokens for 1 image → HTTP 500 ("image tokens ... 2
!= 1") on every page. Drop the literal marker. (A deeper mlx-vlm deepseekocr
MLX-stream bug in the 4-bit build remains — upstream; try an 8-bit/bf16
conversion.)
- paddle_vl: runs its OWN PaddleOCRVL pipeline (not our openai_compat path), so
the repetition-penalty fix never reached it — on a script it can't read
(Greek) it looped, filling the token budget (95 KB of "Ở đòn" ×hundreds).
PaddleOCRVL.predict() accepts repetition_penalty/max_new_tokens; pass 1.15 /
4096. Loop markers hundreds→3, output 95 KB→69 KB; clean French + partial
Greek (recognition itself is a model limitation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #53 (base
feat/server) — this PR is just the 6 commits on top; review #53 first. Three coupled pieces, each shippable on its own.1. Central download registry (retires
model-list.json)Core assets and drop-in plugins register fetchable targets in Python instead of a static JSON catalogue.
aglaia/app_data/downloads.py:DownloadTarget+ in-memory_REGISTRY+register_download/registry/is_downloaded; reconcilingdownload_status(); sync downloader; the 5 core assets registered in Python.db.py: newdownloadstable (status/sha/size) mirroring theplugins-table convention. Disk stays ground truth;download_status()reconciles the table against it (a model deleted out-of-band flips back to not-downloaded).models.pyreduced to a back-compat shim soModelDownloaderTab/OnboardingWizard/setup_clineed no change. The GUI downloader now records its lifecycle into the table too.2. OCR-agnostic local VLM server (
aglaia/workers/vlm/)Generalised from PaddleOCR-VL's private
_MlxVlmServer.LocalVlmServer.ensure(model_path, backend=…)— lazy spawn keyed per model, free-port, health-wait, log-tail, process-group SIGKILL teardown (the orphan-worker footgun).MlxBackend(Apple Silicon) +VllmBackend(CUDA) +pick_backend()(platform preference,AGLAIA_VLM_BACKENDoverride).paddle_vlrefactored onto it (−226 lines), behaviour preserved.3. Abstract OpenAI-compat OCR + two local engines
OpenAiCompatVlmOcrbase: backend-pick → server → DPI downsample → chat → grounding-token parse →OcrResult(meta.markdownfor md_export;<|det|>boxes → per-line bboxes for the searchable PDF layer).--ocr glm(GLM-OCR) and--ocr unlimited(Baidu Unlimited-OCR), ~25-line config subclasses; each registers MLX + vLLM download targets tagged by platform.uv pip install vllm; the backend just needs it importable.Plus
aglaia ocr PATHS…— OCR PDFs/images (or re-OCR a.agl) with no processing chain, for already-clean docs. Verified end-to-end (image → md via Apple Vision).AGLAIA_VLM_SMOKE=1).Testing
Full suite green. New:
test_downloads.py,test_vlm.py,test_openai_compat_ocr.py,test_ocr_command.py,test_vlm_integration.py(opt-in). The live VLM spawn/HTTP path is stubbed in CI and exercised by the opt-in harness on real hardware.🤖 Generated with Claude Code