Junas

Multimodal RAG over technical PDFs — text, tables, and figures, with grounded answers and citations.

Junas ingests a technical PDF (lecture notes, DSA handbooks, OCR-benchmark docs, anything with figures and tables), indexes every region — text paragraphs, figure captions, table cells, full figures, and 2×2 page tiles — into a multi-vector store, and answers natural-language questions by fusing visual + textual + keyword retrieval. The final answer is generated by your configured LLM, grounded in the retrieved evidence.

It is designed for queries like:

• "What does figure 3 show?" — retrieves the figure region by visual similarity.
• "What's the time complexity of push on a stack?" — retrieves the figure plus the surrounding prose that states the answer.
• "Summarize the table on page 12." — retrieves the table region with its structured cells, not a paragraph that mentions the table.

---

Why Junas

Most PDF RAG systems treat a document as a stream of paragraphs. That works for prose; it fails on technical material where the answer is in a figure or a table. Junas fixes this with three things:

1. Layout-aware chunking — DocLayout-YOLO detects title, text, figure, caption, table, formula regions per page. Each region is its own chunk, not a sliding window over text.
2. Multi-vector retrieval — every region is embedded as a ColFlor multi-vector (for late-interaction MaxSim over figure regions) and a single MiniLM vector (for text), and indexed in BM25 (for keywords).
3. Three-level cascade at query time — pages → regions → 2×2 tiles. Intent from the LLM boosts region types (a "what does this figure show" query boosts figure regions; a "summarize the table" query boosts table regions), and the final top-k context is sent to the answer LLM.

The whole pipeline runs on CPU. GPU helps but isn't required.

---

Pipeline

flowchart TB
    subgraph INGEST["Ingestion (per PDF)"]
        A[PDF] --> B[pdfplumber<br/>metadata, tables, text]
        A --> C[PyMuPDF<br/>rasterize @ 150 DPI]
        C --> D[DocLayout-YOLO<br/>region detection]
        D --> E[RapidOCR<br/>recover text from crops]
        D --> F[VLM Llama.cpp<br/>describe figures]
        B --> G[SQLite<br/>chunks.db]
        E --> G
        F --> G
        D --> H[2x2 tile crops]
    end

    subgraph EMBED["Embedding"]
        G --> I[MiniLM-L12<br/>text vectors 384d]
        G --> J[ColFlor<br/>multi-vector patches]
        H --> J
        G --> K[BM25<br/>keyword index]
    end

    subgraph STORE["Storage"]
        I --> L[(Qdrant)]
        J --> L
        K --> M[(bm25.pkl)]
    end

    subgraph QUERY["Query time"]
        Q[User question] --> R[LLM<br/>decompose + intent]
        R --> S[Page-level<br/>MaxSim + MiniLM + BM25]
        S --> T[Region routing<br/>intent-boosted]
        T --> U[2x2 tile drill-down]
        U --> V[Context assembly]
        V --> W[Answer LLM<br/>grounded response]
    end

    L -.->|top-k pages| S
    M -.->|keyword| S
    L -.->|top-k regions| T
    L -.->|top-k tiles| U

Loading

---

Architecture in one breath

Stage	What happens	Output
Pass 1	pdfplumber extracts metadata, table structure, raw text	document metadata + page text
Pass 2	PyMuPDF rasterizes every page to a 150-DPI image	page PNGs in pages/
Pass 3	DocLayout-YOLO (fork with CPU fuse_custom()) detects regions: title, text, figure, caption, table, formula	region bboxes + labels
Pass 4	RapidOCR (ONNX) recovers text from each Text/Caption/Formula/Title/Table crop	OCR'd region text
Pass 5	LFM 2.5 VL 1.6B (via llama-server) eagerly describes each figure crop; produces strict JSON	summary, entities, relationships, metadata per figure
Tile split	Every page is split into a 2×2 grid of tiles for fine-grained retrieval	4 tiles per page
Embeddings	MiniLM-L12-v2 (384d) on every chunk; ColFlor multi-vector on every region + tile	Qdrant collection + BM25 pickle
Storage	Qdrant (vector store) + SQLite (chunk metadata, FK = Qdrant point id) + on-disk pickle for BM25	per-PDF output dir

Stage (query)	What happens
Decomposition	The configured LLM classifies intent (figure, table, prose, formula) and generates 1–3 query variants
Page scoring	3-way fusion: ColFlor MaxSim over pages + MiniLM cosine over pages + BM25 over page text
Region routing	Within top pages, re-rank regions with intent boost (figure query → boost figure regions)
Tile drill-down	If the top region is a figure, score its 2×2 tiles; pick the most specific one
Context assembly	Top-k (page, region, tile) triples with their text + VLM summaries, deduped
Generation	LLM produces a final answer grounded in the assembled context

---

Quick start

# 1. Clone
git clone https://github.com/ayaanmustafa/Junas.git
cd Junas

# 2. Virtual env
python -m venv env
.\env\Scripts\Activate.ps1        # Windows
# source env/bin/activate         # Linux / macOS
pip install -r requirements.txt

# 3. DocLayout-YOLO (your fork with CPU fuse_custom)
git clone https://github.com/ayaanmustafa/DocLayout-YOLO.git
cd DocLayout-YOLO && pip install -e . && cd ..

# 4. ColFlor (visual embedder)
git clone https://github.com/AhmedMasryKU/colflor.git
cd colflor && pip install -e . && cd ..

# 5. Drop the YOLO checkpoint + VLM GGUF weights into ./models/
#    (see setup.md for the exact filenames)

# 6. Start Qdrant (bundled Windows binary, or grab from qdrant/releases)
.\qdrant-x86_64-pc-windows-msvc\qdrant.exe

# 7. Start llama-server with the VLM (LFM 2.5 VL 1.6B or Qwen2.5-VL 3B)
llama-server -m models/lfm25-vl-1.6bextract/LFM2.5-VL-1.6B-Extract-Q4_0.gguf \
    --mmproj models/lfm25-vl-1.6bextract/mmproj-LFM2.5-VL-1.6B-Extract-Q8_0.gguf \
    -c 4096 --port 8081 --host 127.0.0.1

# 8. Configure your answer-generation LLM
cp config.example.json ~/.junas/config.json
# edit it — set provider + matching key/URL/model

# 9. Ingest a PDF
python junas_cli.py ingest --pdf path\to\paper.pdf --out output\my_paper

# 10. Ask
python junas_cli.py ask --out output\my_paper "What does figure 3 show?"

Full step-by-step with troubleshooting lives in setup.md.

---

Configuration

The answer-generation LLM is configured in ~/.junas/config.json. The default template is in config.example.json. Seven providers are supported out of the box — all use the OpenAI-compatible HTTP API, so the same code path handles OpenAI, Gemini, Ollama, llama.cpp, vLLM, or any third-party proxy:

{
  "provider": "openai",
  "openai_api_key": "sk-...",
  "openai_base_url": "https://api.openai.com/v1",
  "openai_model": "gpt-4o-mini"
}

Set provider to whichever you want and fill in the matching <provider>_api_key / <provider>_base_url / <provider>_model triple. Env vars (JUNAS_<PROVIDER>_<FIELD>) override the JSON. A legacy token.txt at the repo root is still honored for the generic llm provider.

The VLM (LFM or Qwen) is not in this config — it only needs a local llama-server running on :8081.

---

Project layout

junas/
├── junas_cli.py              # unified CLI entry point
├── setup.py                  # pip-installable package
├── config.example.json       # template for ~/.junas/config.json
├── requirements.txt          # runtime deps
├── setup.md                  # detailed install + run guide
│
├── core/                     # library code
│   ├── llm_provider.py       # answer-LLM provider abstraction
│   ├── lfm.py                # VLM client (llama-server OpenAI-compat)
│   ├── doclayout_yolo.py     # layout detection wrapper
│   ├── pdfplumber_pass.py    # text + table extraction
│   ├── fitz_rasterize.py     # PyMuPDF page rasterization
│   ├── rapidocr_pass.py      # ONNX OCR for crops
│   ├── embedding.py          # MiniLM wrapper
│   ├── colflor_embed.py      # ColFlor multi-vector embedder
│   ├── colflor_memmap.py     # on-disk ColFlor vectors
│   ├── bm25_index.py         # keyword index
│   ├── qdrant_store.py       # Qdrant collection helpers
│   ├── qdrant_utils.py
│   ├── sqlite_meta.py        # chunk metadata store
│   ├── region_logic.py       # page-context region processing
│   ├── retrieval.py          # 3-level cascade + intent boosting
│   ├── coord.py              # bbox coordinate helpers
│   └── prompts.py            # LLM system prompts
│
├── ingest/
│   └── run.py                # ingestion entry point (Pass 1–6)
│
├── query/
│   └── ask.py                # query-time entry point
│
├── DocLayout-YOLO/           # gitignored — clone fresh, pip install -e .
├── colflor/                  # gitignored — clone fresh, pip install -e .
├── models/                   # gitignored — YOLO .pt + VLM .gguf weights
└── qdrant-x86_64-pc-windows-msvc/   # gitignored — Windows Qdrant binary

---

Key design choices

These are the load-bearing decisions in the system — the things we picked on purpose, not by default:

• One embedding model per modality. MiniLM for text (fast, 384d, plenty for retrieval), ColFlor for visual (multi-vector, late-interaction). We deliberately do not run both on both modalities — text → MiniLM only, figure → ColFlor only.
• Eager VLM at index time, lazy only at the tile level. Figures get described once when ingested; tile-level VLM is only triggered if a query lands on a specific tile and the existing summary is too coarse.
• Trust YOLO for tables. DocLayout-YOLO's table class is reliable enough on technical PDFs that we don't run a separate table-structure pass. pdfplumber gives us the cell text within the YOLO-detected bbox.
• CPU-optimized YOLO via a maintained fork. The fuse_custom() recursive path in ayaanmustafa/DocLayout-YOLO shaves enough time on CPU to make ingest of a 50-page paper tractable without a GPU.
• Three-level cascade with intent boosting. Pages first (cheap), regions second (intent-aware), tiles third (only on drill-down). Avoids the cost of scoring every tile for every query.
• SQLite for metadata, Qdrant for vectors. FK from SQLite to Qdrant point id. SQLite gives you cheap structured queries (WHERE doc_id = ? AND page = ?); Qdrant gives you ANN. Don't conflate them.
• No reranker (yet). BGE cross-encoder is on the post-MVP list. We want end-to-end retrieval numbers first before optimizing the last mile.

---

Limitations

• Scale. MVP targets ≤ a few hundred PDFs. ColFlor multi-vectors are loaded into memory and MaxSim is computed in Python. Past ~10k regions the cascade will need a PLAID-style index.
• Multilingual. English-first. Works on CJK if the models don't mangle it, but no guarantees.
• No reranking. Recall@10 numbers without a cross-encoder are mediocre on long-tail queries.
• No streaming. Final answer comes back in one shot.
• No evaluation harness. Hand-tested on three test PDFs (test_data/). Proper Recall@k / MRR setup is post-MVP.

---

License

Add a LICENSE file before going public. Suggested: MIT for permissive, Apache-2.0 if you want patent grants, AGPL-3.0 if you want copyleft (note: the vendored DocLayout-YOLO is already AGPL-3.0, so any distribution that includes it inherits that requirement).

---

Credits

• Layout detection: DocLayout-YOLO (this fork) / opendatalab/DocLayout-YOLO (upstream)
• Visual embedder: ColFlor (Ahmed Masry)
• VLM: LFM 2.5 VL 1.6B via llama.cpp
• Vector store: Qdrant
• OCR: RapidOCR