Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/mobile-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,9 @@ jobs:
with:
path: |
frontend/src-tauri/onnxruntime-ios
key: release-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
key: release-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
restore-keys: |
release-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-
release-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-

- name: Prepare ONNX Runtime iOS artifact
run: nix develop .#apple -c ./scripts/ci/ios-onnxruntime.sh
Expand Down Expand Up @@ -237,9 +237,9 @@ jobs:
with:
path: |
frontend/src-tauri/onnxruntime-ios
key: pr-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
key: pr-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
restore-keys: |
pr-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-
pr-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-

- name: Prepare ONNX Runtime iOS artifact
run: nix develop .#apple -c ./scripts/ci/ios-onnxruntime.sh
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/mobile-pr-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,9 @@ jobs:
with:
path: |
frontend/src-tauri/onnxruntime-ios
key: pr-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
key: pr-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
restore-keys: |
pr-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-
pr-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-

- name: Prepare ONNX Runtime iOS artifact
run: nix develop .#apple -c ./scripts/ci/ios-onnxruntime.sh
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -446,9 +446,9 @@ jobs:
with:
path: |
frontend/src-tauri/onnxruntime-ios
key: release-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
key: release-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-${{ hashFiles('flake.lock', 'flake.nix', 'scripts/ci/_common.sh', 'scripts/ci/ios-onnxruntime.sh', 'frontend/src-tauri/scripts/build-ios-onnxruntime-all.sh', 'frontend/src-tauri/scripts/canonicalize-static-archive.py', 'frontend/src-tauri/scripts/onnxruntime-pins.sh') }}
restore-keys: |
release-onnxruntime-ios-artifact-1.22.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-
release-onnxruntime-ios-artifact-1.23.2-xcode26.5-${{ steps.select_xcode.outputs.build }}-

- name: Prepare ONNX Runtime iOS artifact
run: nix develop .#apple -c ./scripts/ci/ios-onnxruntime.sh
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ bun tauri build --target universal-apple-darwin

#### Linux: ONNX Runtime Setup

Linux builds bundle `libonnxruntime.so` for local TTS (Supertonic). Before building on Linux, you must download the ONNX Runtime shared library:
Linux builds bundle ONNX Runtime 1.23.2 for local TTS and PDF OCR. `just desktop-build` provisions it automatically. Before invoking `bun tauri build` directly, download the pinned shared library:

```bash
cd frontend/src-tauri
Expand All @@ -152,7 +152,7 @@ cd frontend/src-tauri

This downloads the pinned ONNX Runtime release, verifies its SHA-256 checksum, and extracts it to `frontend/src-tauri/onnxruntime-linux/` (which is gitignored). The script is idempotent — it skips the download if the library already exists.

> **Note:** CI workflows call this script automatically. You only need to run it manually for local builds.
> **Note:** CI workflows and the desktop `just` recipes call this script automatically. Run it manually only when invoking the lower-level Tauri commands directly.

#### Linux: Running in Headless/Virtual Display Environments

Expand Down
10 changes: 5 additions & 5 deletions docs/ios-tts-local-development.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# iOS TTS Local Development Guide
# iOS ONNX Runtime Local Development Guide

This guide explains how to build and test the iOS TTS (Text-to-Speech) feature locally using the iOS Simulator or a physical device.
This guide explains how to build and test the iOS ONNX Runtime used by local TTS and PDF OCR on the iOS Simulator or a physical device.

## Overview

The iOS TTS feature uses ONNX Runtime to run Kokoro TTS models. ONNX Runtime must be built from source for iOS because:
Maple pins ONNX Runtime 1.23.2 and links it statically on iOS for both TTS and PDF OCR. ONNX Runtime is built from source because:
1. Pre-built binaries from HuggingFace are missing Abseil symbols
2. We need both device (arm64) and simulator (arm64) builds
3. The simulator build requires a workaround for a libiconv linking bug
Expand All @@ -25,7 +25,7 @@ The iOS TTS feature uses ONNX Runtime to run Kokoro TTS models. ONNX Runtime mus
just ios-build-onnxruntime
```

This builds ONNX Runtime for both device and simulator (~5-10 minutes) and automatically generates the cargo config.
This builds and hash-verifies ONNX Runtime for both device and simulator, then generates the Cargo config. A valid cached artifact is reused; a stale or differently built artifact fails verification and is rebuilt.

The output will be in `frontend/src-tauri/onnxruntime-ios/onnxruntime.xcframework/`.

Expand Down Expand Up @@ -93,7 +93,7 @@ Undefined symbols for architecture arm64:
"_AbslInternalSpinLockDelay_lts_20240722"
```

This means the ONNX Runtime library wasn't built from source. Pre-built binaries are missing these symbols. Run `./scripts/build-ios-onnxruntime-all.sh` to build from source.
This means the ONNX Runtime library wasn't built through Maple's verified source pipeline. Run `just ios-build-onnxruntime`.

### Simulator Build Fails with libiconv Error

Expand Down
72 changes: 72 additions & 0 deletions docs/pdf-ocr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# PDF extraction and OCR

Maple uses [`pdf_oxide` 0.3.74](https://crates.io/crates/pdf_oxide/0.3.74) for native PDF text extraction, page classification, rendering, and PaddleOCR preprocessing. PDF bytes and rendered pages stay on the device.

Maple pins [OpenSecretCloud's `pdf_oxide` maintenance commit](https://github.com/OpenSecretCloud/pdf_oxide/commit/f24b43ba997dd91ce60839640a8ce3ac92a87a5d) based directly on the published 0.3.74 release. The fork adds only a caller-controlled ONNX Runtime loader feature; it contains no Maple-specific parser or inference changes. This keeps iOS static linkage compatible without importing unpublished upstream work.

## One inference runtime

PDF OCR and desktop/iOS TTS use one Rust binding (`ort = 2.0.0-rc.11`) and one Microsoft ONNX Runtime version (1.23.2). Maple does not ship tract.

The runtime is part of the application package; users do not download it when opening a PDF:

| Platform | ONNX Runtime packaging |
| --- | --- |
| macOS | Microsoft's official universal2 dylib, loaded from the app Frameworks directory; both Apple silicon and Intel are retained. Minimum macOS is 13.4. |
| Linux | Microsoft's x64 or aarch64 shared library, installed under Maple's private library directory. |
| Windows | Microsoft's x64 DLL, installed next to `maple.exe` and loaded by its explicit path. |
| Android | Microsoft's official Android AAR, with `libonnxruntime.so` staged for arm64-v8a, armeabi-v7a, x86, and x86_64. |
| iOS | A deterministic device/simulator XCFramework built from the pinned Microsoft source commit and linked statically. |

Maple owns loader policy through PDFOxide's loader-neutral `ocr-ort` feature. Dynamic targets initialize the exact packaged library before either OCR or TTS creates a session. iOS initializes the same Rust API against its static library. Cargo resolves exactly one `ort` version per target.

The application constructs and reuses one OCR engine. Native-only PDFs bypass model setup. OCR-routed pages are rendered once in full, bounded to a 2,000 by 2,000-pixel box, so tiled scans, inline images, and vector content do not depend on selecting one embedded image. Mixed pages retain native text and append OCR spans not already represented in the native layer. OCR remains optional enrichment for native-readable hybrid pages: if OCR is unavailable, Maple keeps the native text.

## Model download and cache

The first PDF that actually needs OCR downloads 12,577,821 bytes of PaddleOCR model data on macOS, Windows, Linux, iOS, or Android. The files are stored under the application cache at `ocr/models/paddleocr-en-v1`. An operating system may purge that cache, in which case Maple downloads and verifies the pack again.

Maple does not use PDFOxide's mutable model downloader. Every artifact is pinned to an immutable repository revision and checked for exact length and SHA-256 before loading:

| File | Immutable source | Bytes | SHA-256 |
| --- | --- | ---: | --- |
| `det.onnx` | [SWHL RapidOCR PaddleOCR detector](https://huggingface.co/SWHL/RapidOCR/blob/1cfba2e90fc938db55889873735088de210cc173/PP-OCRv4/ch_PP-OCRv4_det_infer.onnx) | 4,745,517 | `d2a7720d45a54257208b1e13e36a8479894cb74155a5efe29462512d42f49da9` |
| `rec.onnx` | [monkt English recognizer](https://huggingface.co/monkt/paddleocr-onnx/blob/7b02d0a30a07ba2b92ad1ff5a8941ae2c633de65/languages/english/rec.onnx) | 7,830,888 | `4e16deb22c4da6468bdca539b2cd3c8687825538b67109177c47d359ab994cd7` |
| `en_dict.txt` | [monkt English dictionary](https://huggingface.co/monkt/paddleocr-onnx/blob/7b02d0a30a07ba2b92ad1ff5a8941ae2c633de65/languages/english/dict.txt) | 1,416 | `e025a66d31f327ba0c232e03f407ae8d105e1e709e7ccb3f408aa778c24e70d6` |

Downloads stream to a temporary file, enforce the expected size, verify SHA-256, sync, and then rename into place. Concurrent first-use requests share one setup lock. Existing cached files are re-verified before engine construction. The backend emits download progress while the current UI keeps the attachment in its processing state.

PaddleOCR, SWHL RapidOCR, and the monkt model repository declare Apache-2.0. On Android, the model-only HTTP client uses rustls with Mozilla WebPKI roots; other platforms retain their normal trust-store integration.

## Panic and error boundary

All PDF parsing, classification, rendering, and inference work runs in an isolated blocking task. An ordinary Rust unwind becomes a normal Tauri command error, so the frontend clears its busy state and remains usable. Process aborts and native stack exhaustion cannot be recovered by a Rust unwind boundary.

The backend serializes PDF jobs, bounds OCR renders to four megapixels, and checks source-image count and pixel budgets. It independently enforces 10 MiB limits on both the input document and extracted text, rejects locked PDFs with a specific message, and treats a document with no recognized text as an error rather than silently attaching an empty document.

If OCR is required for a scanned page and that page cannot be processed, the upload fails with the page-specific error rather than attaching an incomplete subset of the document. OCR failures remain optional only for hybrid pages whose native text is already readable.

## Current scope and known limitation

The current model pack recognizes English text. Additional language packs, password entry, and a dedicated OCR download UI are separate product work.

PDFOxide 0.3.74 cannot render some uncompressed, one-bit FlateDecode, and inline PDF images ([upstream issue #860](https://github.com/yfedoseev/pdf_oxide/issues/860)). OCR sees a blank render for those pages even when Poppler displays the scan. A genuine ten-page NASA archival scan passes Maple's full-page OCR path, while a genuine ten-page NARA one-bit Flate scan reproduces #860. The loader-policy fork intentionally does not patch this parser/renderer issue in the initial viability change.

The MVP also caps each source image at 24 million pixels before PDFOxide decodes it. A typical 300-DPI archival scan fits; some 600-DPI scans do not, even though the final OCR render would be reduced to four million pixels. Maple reports the first affected page instead of attaching partial text. Raising the budget or adding decode-time downsampling requires separate desktop/mobile memory measurements.

## Local model-backed test

Provision the desktop runtime first, then run the ignored release-profile test with a real scanned PDF and the verified model pack:

```sh
cd frontend/src-tauri
./scripts/provide-macos-onnxruntime.sh # use the platform-equivalent provider

ORT_DYLIB_PATH=/absolute/path/from/the/provider \
MAPLE_OCR_TEST_PDF=/absolute/path/to/scanned.pdf \
MAPLE_OCR_MODEL_DIR=/absolute/path/to/models \
nix develop -c cargo test --release --locked --lib \
extracts_scanned_pdf_with_real_ocr_models -- --ignored --nocapture
```

Use the release profile for model-backed timing. Debug inference is not representative of the packaged application.
6 changes: 6 additions & 0 deletions frontend/src-tauri/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,11 @@
# ONNX Runtime Linux (downloaded for bundling)
/onnxruntime-linux/

# ONNX Runtime macOS (downloaded for bundling)
/onnxruntime-macos/

# ONNX Runtime Android (verified AAR cache; JNI libraries are staged under gen/android)
/onnxruntime-android/

# Generated cargo config for iOS builds
/.cargo/
Loading
Loading