Project Document Extraction Pipeline

A multimodal document extraction and analysis pipeline designed for architectural and engineering project folders. Extracts structured content from 50+ file formats into a unified Canonical Document Model, with a web UI, REST API, hierarchical analysis, and ML-ready export.

Overview

This tool takes a folder of project documents (PDFs, drawings, specs, models, etc.) and produces:

Structured JSON — every document parsed into a common Canonical Document Model (blocks, pages, tables, figures, metadata)
Hierarchical analysis — section trees with similarity detection across documents
ML-ready exports — flat relational schema in 4 formats (JSON, JSONL, CSV, text)
Web UI — interactive viewer with bbox overlays, D3 force graph, and export dashboard

Architecture

ingestion_tool/
├── main.py                  # FastAPI web app + REST API
├── demo_run.py              # Standalone demo script
├── src/
│   ├── models.py            # Canonical Document Model (Pydantic v2)
│   ├── pipeline.py          # ProjectPipeline orchestrator
│   ├── cli.py               # CLI entry point (python -m src)
│   ├── extractors/          # Per-format extractor plugins
│   │   ├── base.py          # BaseExtractor abstract class
│   │   ├── registry.py      # Routes extensions → extractors
│   │   ├── pdf_extractor.py
│   │   ├── office_extractor.py
│   │   ├── cad_extractor.py
│   │   ├── gis_extractor.py
│   │   ├── ifc_extractor.py
│   │   └── ...
│   ├── analysis/            # Post-extraction analysis
│   │   ├── hierarchy_analyzer.py
│   │   ├── registry.py
│   │   └── runner.py
│   ├── export/              # ML-ready export system
│   │   ├── schema.py        # 13 Pydantic export models
│   │   ├── builder.py       # Assembles unified export
│   │   ├── formats.py       # JSON / JSONL / CSV / TXT serializers
│   │   └── bundle.py        # ZIP bundle creator
│   └── utils/               # Hashing, MIME detection, output writing
├── static/
│   ├── index.html           # Landing page
│   └── extraction.html      # Full single-page extraction UI
└── tests/                   # pytest suite (97 tests)

Data flow:

Input folder
    │
    ▼
ExtractorRegistry
    │  routes by file extension
    ▼
Format Extractor (PDF / DOCX / DXF / IFC / ...)
    │  produces
    ▼
CanonicalDocument (blocks, pages, tables, figures)
    │
    ├──► canonical.json + extracted_text.txt + tables.json + metadata.json
    │
    ▼
HierarchyAnalyzer
    │  produces
    ▼
hierarchy.json  +  project_hierarchy.json
    │
    ▼
ExportBuilder
    │  produces
    ▼
unified_export.json  |  blocks.jsonl  |  tables.csv  |  text_corpus.txt  |  bundle.zip

Supported Formats

Category	Extensions
PDF	`.pdf`
Office	`.docx`, `.xlsx`, `.pptx`
Images	`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`, `.tiff`
CAD	`.dxf`, `.dwg`
BIM	`.ifc`
GIS	`.geojson`, `.shp`, `.kml`, `.kmz`
3D Mesh	`.stl`, `.obj`, `.glb`
Point Cloud	`.las`, `.laz`, `.e57`
Archives	`.zip`, `.7z`, `.rar`
Text / Data	`.txt`, `.csv`, `.xml`, `.json`, `.svg`

Run python -m src list_formats to see the full list.

Prerequisites

Python 3.10+ (tested on 3.13)
pip or any package manager
Git

Optional system-level dependencies:

libreoffice — DWG conversion (only needed for .dwg files)
ifcopenshell — BIM/IFC (pip install separately, see Installation)

Installation

1. Clone the repo

git clone https://github.com/YOUR_ORG/ingestion-tool.git
cd ingestion-tool

2. Create a virtual environment

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

3. Install dependencies

Minimum (core + web UI):

pip install -e .

With common format support (recommended):

pip install -e ".[office,cad,gis,archive,render]"

Everything:

pip install -e ".[all]"
pip install -r requirements.txt

Optional extras (install manually if needed):

# BIM / IFC support
pip install ifcopenshell

# Point cloud LAS/LAZ
pip install "laspy[lazrs]>=2.4"

# Point cloud E57
pip install pye57

# RAR archive support
pip install rarfile

4. Verify installation

python -m src list_formats

You should see a table of all supported file extensions.

Quick Start

Web UI

Start the FastAPI server:

uvicorn main:app --reload --port 8000

Then open http://localhost:8000/extraction in your browser.

The UI lets you:

Upload files — drag-and-drop or file picker
View extracted content — JSON, plain text, tables, page renders
Explore hierarchy — interactive tree and D3 force-directed graph
Browse media — extracted figures and images
Download exports — all 4 formats or a full ZIP bundle

CLI

Extract a project folder:

python -m src extract_project \
  --input  ./my_project_folder \
  --output ./extraction_output/run_001 \
  --project-id my-project-2024

Extract + run hierarchy analysis:

python -m src extract_project \
  --input  ./my_project_folder \
  --output ./extraction_output/run_001 \
  --project-id my-project-2024 \
  --analyze

Export to ML-ready formats:

# All formats
python -m src export \
  -o extraction_output/run_001 \
  -r run_001 \
  --format all

# ZIP bundle with media
python -m src export \
  -o extraction_output/run_001 \
  -r run_001 \
  --format bundle

# Bundle without renders (smaller)
python -m src export \
  -o extraction_output/run_001 \
  -r run_001 \
  --format bundle --no-renders

List available analyzers:

python -m src list_analyzers

Python API

from src.pipeline import ProjectPipeline
from pathlib import Path

pipeline = ProjectPipeline(
    project_id="my-project-2024",
    input_dir=Path("./my_project_folder"),
    output_dir=Path("./extraction_output/run_001"),
)
results = pipeline.run(analyze=True)

for r in results:
    print(f"{'OK' if r.success else 'FAIL'} | {r.file_path.name} | {r.extractor_name}")

Run the built-in demo (no files needed):

python demo_run.py

This creates synthetic sample files (TXT, CSV, GeoJSON, JSON) and runs the full pipeline on them.

Output Structure

After extraction, each file gets its own output directory:

extraction_output/
└── {run_id}/
    ├── manifest.json                   # Index of all processed files
    ├── project_hierarchy.json          # Cross-document section graph
    └── {doc_hash}/
        ├── canonical.json              # Full structured document (CDM)
        ├── extracted_text.txt          # Clean plain text
        ├── tables.json                 # Structured table data
        ├── metadata.json              # File metadata + extraction info
        ├── hierarchy.json             # Per-document section hierarchy
        ├── media/                     # Extracted figures and images
        │   ├── p1_img0.png
        │   └── ...
        └── render/                    # Page-level renders (PNG)
            ├── page_0.png
            └── ...

canonical.json schema (simplified)

{
  "doc_id": "abc123",
  "source_path": "relative/path/to/file.pdf",
  "doc_class": "document",
  "extractor": "PDFExtractor",
  "pages": [
    {
      "page_index": 0,
      "width": 612,
      "height": 792,
      "blocks": [
        {
          "block_id": "b_001",
          "type": "Heading",
          "text": "Section 1 - Introduction",
          "bbox": [72, 120, 540, 144],
          "confidence": 0.95,
          "source": { "page_index": 0, "doc_id": "abc123" }
        }
      ]
    }
  ],
  "reading_order": { "0": ["b_001", "b_002"] },
  "extracted_assets": []
}

Export System

The export system converts a completed run into ML-ready formats.

Formats

Format	File	Best for
Master JSON	`unified_export.json`	Analytics, databases, knowledge graphs
Blocks JSONL	`blocks.jsonl`	ML training, streaming pipelines
Tables CSV	`tables.csv`	SQL, spreadsheets, pandas
Text Corpus	`text_corpus.txt`	RAG pipelines, search indexing, LLMs
ZIP Bundle	`{project}_{run}_export.zip`	Full handoff — all formats + media

Loading in Python

import json, pandas as pd

# Master JSON → any array as a DataFrame
data = json.load(open("unified_export.json"))
blocks_df  = pd.json_normalize(data["blocks"])
tables_df  = pd.json_normalize(data["tables"])
figures_df = pd.json_normalize(data["figures"])

# Blocks JSONL → stream line by line
with open("blocks.jsonl") as f:
    blocks = [json.loads(line) for line in f]

# Tables CSV → pandas directly
tables = pd.read_csv("tables.csv")

# Text corpus → feed to LangChain / LlamaIndex
from langchain.document_loaders import TextLoader
docs = TextLoader("text_corpus.txt").load()

Analysis Framework

The hierarchy analyzer runs after extraction to build a cross-document knowledge graph.

What it produces:

Section hierarchy — heading → subheadings → content blocks (tree structure)
Similarity edges — sections with similar text content linked via SIMILAR_TO edges
Merged sections — duplicate headings under the same parent are de-duplicated
Cross-document links — identical headings across different files get similarity connections

Output: hierarchy.json per file, project_hierarchy.json at the run root.

Visualized in the Web UI Hierarchy tab as an interactive D3 force graph or collapsible tree.

Running Tests

# Run all tests
pytest

# With coverage
pytest --cov=src --cov-report=term-missing

# Run a specific module
pytest tests/test_pipeline.py
pytest tests/test_export.py
pytest tests/test_analysis.py
pytest tests/test_models.py

# Verbose
pytest -v

Expected: 97 tests, all passing.

API Reference

The FastAPI server exposes the following endpoints. Interactive docs available at http://localhost:8000/docs.

Extraction

Method	Path	Description
`POST`	`/api/extract`	Upload files and run extraction
`GET`	`/api/runs`	List all extraction runs
`GET`	`/api/runs/{run_id}`	Get run metadata
`GET`	`/api/formats`	List supported file formats

Per-file results

Method	Path	Description
`GET`	`/api/runs/{run_id}/file/{path}/canonical`	Canonical JSON
`GET`	`/api/runs/{run_id}/file/{path}/text`	Extracted plain text
`GET`	`/api/runs/{run_id}/file/{path}/tables`	Table data
`GET`	`/api/runs/{run_id}/file/{path}/metadata`	File metadata
`GET`	`/api/runs/{run_id}/file/{path}/render/{image}`	Page render image
`GET`	`/api/runs/{run_id}/file/{path}/media/{image}`	Extracted figure image

Analysis

Method	Path	Description
`GET`	`/api/runs/{run_id}/file/{path}/hierarchy`	Per-file hierarchy graph
`GET`	`/api/runs/{run_id}/project_hierarchy`	Cross-document hierarchy
`POST`	`/api/runs/{run_id}/analyze`	Run analysis on existing run

Export

Method	Path	Description
`GET`	`/api/runs/{run_id}/export/preview`	Export statistics (fast)
`GET`	`/api/runs/{run_id}/export/master`	unified_export.json download
`GET`	`/api/runs/{run_id}/export/blocks`	blocks.jsonl download
`GET`	`/api/runs/{run_id}/export/tables`	tables.csv download
`GET`	`/api/runs/{run_id}/export/text`	text_corpus.txt download
`GET`	`/api/runs/{run_id}/export/bundle`	ZIP bundle download

Project Structure

ingestion_tool/
├── main.py                        # FastAPI app + all API endpoints
├── demo_run.py                    # Self-contained demo (no files needed)
├── pyproject.toml                 # Package config + optional extras
├── requirements.txt               # Flat dependency list
├── src/
│   ├── __init__.py
│   ├── __main__.py                # python -m src entry point
│   ├── cli.py                     # 4 CLI commands
│   ├── models.py                  # CanonicalDocument, Page, Block, SourceRef
│   ├── pipeline.py                # ProjectPipeline
│   ├── extractors/
│   │   ├── base.py
│   │   ├── registry.py
│   │   ├── pdf_extractor.py
│   │   ├── office_extractor.py    # DOCX, XLSX, PPTX
│   │   ├── cad_extractor.py       # DXF, DWG
│   │   ├── gis_extractor.py       # GeoJSON, SHP, KML
│   │   ├── ifc_extractor.py       # IFC/BIM
│   │   ├── mesh3d_extractor.py    # STL, OBJ, GLB
│   │   ├── pointcloud_extractor.py# LAS, LAZ, E57
│   │   ├── image_extractor.py     # PNG, JPG, etc.
│   │   ├── archive_extractor.py   # ZIP, 7Z, RAR
│   │   └── unknown_extractor.py   # Fallback
│   ├── analysis/
│   │   ├── base.py
│   │   ├── registry.py
│   │   ├── models.py
│   │   ├── hierarchy_analyzer.py
│   │   └── runner.py
│   ├── export/
│   │   ├── schema.py
│   │   ├── builder.py
│   │   ├── formats.py
│   │   └── bundle.py
│   └── utils/
│       ├── hashing.py
│       ├── mime_detect.py
│       └── output_writer.py
├── static/
│   ├── index.html
│   └── extraction.html
└── tests/
    ├── test_models.py
    ├── test_pipeline.py
    ├── test_analysis.py
    └── test_export.py

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.claude		.claude
src		src
static		static
tests		tests
.gitignore		.gitignore
README.md		README.md
SETUP.md		SETUP.md
WALKTHROUGH.md		WALKTHROUGH.md
demo_run.py		demo_run.py
main.py		main.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Project Document Extraction Pipeline

Table of Contents

Overview

Architecture

Supported Formats

Prerequisites

Installation

1. Clone the repo

2. Create a virtual environment

3. Install dependencies

4. Verify installation

Quick Start

Web UI

CLI

Python API

Output Structure

canonical.json schema (simplified)

Export System

Formats

Loading in Python

Analysis Framework

Running Tests

API Reference

Extraction

Per-file results

Analysis

Export

Project Structure

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages