KID PDF Pipeline

Simple Docling-based pipeline for fund KID PDFs

It does two things:

• extracts the key fields from each PDF
• answers user questions from inputs/questions.json

Extra docs:

• [PIPELINE.md](/Users/Pmmoks/Documents/Applications/FE Fund Info/ai-engineer-test/PIPELINE.md)
• [THOUGHT_PROCESS.md](/Users/Pmmoks/Documents/Applications/FE Fund Info/ai-engineer-test/THOUGHT_PROCESS.md)

What It Extracts

For each PDF the pipeline extracts:

• fund_name
• share_class
• isin
• investment_strategy
• holding_period
• risk_indicator
• ongoing_costs

Quick Start

1. Create a virtual environment

python3 -m venv .venv

2. Install dependencies

Runtime only:

.venv/bin/pip install -r requirements.txt

Runtime plus tests:

.venv/bin/pip install -r requirements-dev.txt

You can also install the package in editable mode:

.venv/bin/pip install -e ".[dev]"

3. Add your xAI credentials

Copy .env.example to .env and fill in your values:

cp .env.example .env

.env should contain:

XAI_API_KEY=your_key_here
MODEL=grok-4-1-fast-non-reasoning

Docling downloads parsing models on first use, so the first run needs network access

4. Add questions

Put a JSON list of questions in [inputs/questions.json](/Users/Pmmoks/Documents/Applications/FE Fund Info/ai-engineer-test/inputs/questions.json)

Example:

[
  "What is the ISIN?",
  "What does the fund invest in?",
  "What is the holding period?",
  "What are the fees?"
]

If this file exists and is non-empty, the pipeline will answer those questions for every processed PDF and save the results under output/answers/

5. Run the pipeline

Process one PDF:

.venv/bin/python -m kid_pipeline sample_files/239038469.pdf

Process all PDFs in a folder:

.venv/bin/python -m kid_pipeline sample_files

Use table images as well as parsed markdown:

.venv/bin/python -m kid_pipeline sample_files --content-mode image

Write the final combined JSON to a file as well:

.venv/bin/python -m kid_pipeline sample_files --output output.json

Use a different questions file:

.venv/bin/python -m kid_pipeline sample_files --questions-path my_questions.json

Hide the final JSON on stdout:

.venv/bin/python -m kid_pipeline sample_files --quiet

Override the model from the CLI:

.venv/bin/python -m kid_pipeline sample_files --grok-model grok-4-1-fast-non-reasoning

What Gets Written

Each run writes per-document files under output/

• output/parsed_markdown/<pdf-stem>.md
• output/parsed_docling_json/<pdf-stem>.json
• output/extractions/<pdf-stem>.json
• output/answers/<pdf-stem>.json if questions were provided
• output/table_images/<pdf-stem>/tables.json
• output/table_images/<pdf-stem>/images/ in image mode only

The final extraction JSON uses this shape:

{
  "file": "239038469.pdf",
  "fund_name": {
    "value": "AB China A Shares Equity Portfolio",
    "page": 1,
    "block": "block_001"
  },
  "share_class": {
    "value": "ICNH",
    "page": 1,
    "block": "block_003"
  },
  "isin": {
    "value": "LU1238072786",
    "page": 1,
    "block": "block_004"
  },
  "investment_strategy": {
    "value": "...",
    "page": 1,
    "block": "block_010"
  },
  "holding_period": {
    "value": "keep your investment in the Portfolio for 5 years",
    "page": 1,
    "block": "block_020"
  },
  "risk_indicator": {
    "value": null,
    "page": null,
    "block": null
  },
  "ongoing_costs": {
    "value": "1,19%",
    "page": 2,
    "block": "table_2_1"
  }
}

How The Questions Work

The pipeline reads the list in inputs/questions.json and sends those questions to Grok together with the parsed document content

It then writes one answers file per PDF, for example:

• output/answers/239038469.json

That file looks like:

{
  "file": "239038469.pdf",
  "answers": [
    {
      "question": "What is the ISIN?",
      "answer": "LU1238072786"
    }
  ]
}

Running Tests

Run the normal automated tests:

.venv/bin/python -m pytest

Run the live end-to-end xAI smoke test only when you want to hit the real API:

.venv/bin/python -m pytest --run-live-llm

Notes

• The pipeline always uses Grok for extraction
• By default it reads the model name from MODEL in .env
• --content-mode image uses parsed markdown plus cropped table images
• If a PDF cannot be parsed into readable text, the command exits instead of falling back to OCR
• If xAI rejects the request, for example because of a bad model name or rate limits, the command prints the provider error directly