freight-rag-agents

A multi-agent hybrid RAG assistant for a freight brokerage / LTL operations desk. It answers natural-language ops questions by routing them across three retrieval modes — dense embeddings + sparse BM25 over policy/SOP documents (fused with Reciprocal Rank Fusion) and guarded read-only SQL over an operational warehouse — orchestrated as a small, explainable LangGraph agent graph with a grounding critic that guards against hallucination.

Every quality number below comes from an actual scripts/run_eval.py run over a committed, labeled eval set. Nothing here is hand-written.

Synthetic data. The warehouse and document corpus are generated deterministically (no real PII). Models used: OpenAI text-embedding-3-small (embeddings) and gpt-4o-mini (chat).

Eval results (real receipts)

From results/eval.json — produced by running the full graph over the 20-question labeled set in eval/eval_set.jsonl:

Metric	Score
Routing accuracy	95.0% (19/20)
Retrieval hit rate	100.0% (20/20)
Groundedness catch-rate (unanswerable)	100.0% (3/3)
Answer correctness	100.0% (20/20)

• Routing accuracy — predicted route (docs / data / both) vs the label. The single miss is q01, where the router conservatively chose both instead of docs; this is harmless (it retrieves a superset) but counts against us.
• Retrieval hit rate — for questions that name a required doc or table, was that doc actually cited / that table actually queried.
• Groundedness catch-rate — on planted unanswerable questions, did the system refuse instead of fabricating.
• Answer correctness — required key facts present in the answer (gold substrings), with refusal required on unanswerable cases.

Full per-case breakdown: results/report.md.

Architecture

Every graph node is a pure, dependency-injected function — the LLM and retrievers are passed in — so the whole graph is unit-testable with a mocked LLM and an in-memory database. No test makes a network call.

                         ┌─────────────┐
   question ───────────▶ │   router    │  classify -> docs | data | both
                         └──────┬──────┘
                                │
                  ┌─────────────┴──────────────┐
                  ▼ (docs|both)                 ▼ (data|both)
          ┌───────────────┐            ┌────────────────────┐
          │  doc_agent     │           │  analytics_agent    │
          │  hybrid RAG:   │           │  NL -> guarded SQL   │
          │  dense + BM25  │           │  (sqlglot guard +    │
          │  RRF (k0=60)   │           │   DuckDB execute)    │
          └───────┬───────┘            └─────────┬──────────┘
                  │   doc_hits (cited)           │   sql_facts
                  └──────────────┬───────────────┘
                                 ▼
                         ┌───────────────┐
                         │  synthesizer  │  cited answer from evidence only
                         └───────┬───────┘
                                 ▼
                         ┌───────────────┐
                         │    critic     │  grounded?  {true|false, reason}
                         └───────┬───────┘
                    grounded /   │   \  ungrounded & attempts < 2
                       end       │    └────────► loop back to retrieve
                                 ▼                 (with critic feedback)
                         ┌───────────────┐
                         │   finalize    │  flag low_confidence if still ungrounded
                         └───────────────┘

Orchestration: LangGraph. The graph is built with langgraph (1.x): StateGraph over a typed AgentState, a conditional edge after the critic that either loops back to retrieval (max 2 attempts) or finalizes. A behaviorally equivalent hand-rolled orchestrator (run_state_machine) over the same node functions is included as a documented fallback and is exercised in the tests to prove the two are equivalent. LangGraph worked cleanly; the fallback was not needed in anger.

Components

Module	Role
state.py	AgentState TypedDict threaded through every node
router.py	LLM classifier → docs / data / both; unknown → safe both
retrieval/dense.py	cosine over committed OpenAI embeddings (query embedded at runtime)
retrieval/bm25.py	rank_bm25 Okapi over the tokenized corpus
retrieval/hybrid.py	Reciprocal Rank Fusion (k0=60) of the two ranked lists
sql_tool.py	sqlglot read-only guard + DuckDB execution
analytics_agent.py	text-to-SQL constrained to the schema, guarded, with 1 correction retry
doc_agent.py	hybrid retrieve → cited passages
synthesizer.py	compose a cited answer from evidence only
critic.py	grounding judgment {grounded, reason}
graph.py	LangGraph wiring + hand-rolled fallback

Quickstart

# 1. Environment (Python 3.11+, Apple Silicon fine)
uv venv && . .venv/bin/activate && uv pip install -e ".[dev]"
export OPENAI_API_KEY=sk-...

# 2. Build the deterministic warehouse + doc corpus (no API needed)
make data

# 3. Embed the corpus -> committed index (needs API; already committed in repo)
make index

# 4. Ask a question
freight-rag "What is our detention free time and rate?"

# 5. Reproduce the eval (needs API)
make eval        # -> results/eval.json + results/report.md

# 6. See the grounding-critic retry loop fire end-to-end (no API)
make demo

# 7. Network-free tests
make test

The committed data/doc_embeddings.npy + data/doc_meta.json mean the corpus is searchable without re-spending embedding credits; make index only needs to be re-run if you change the documents.

SQL safety

LLM-generated SQL is never trusted. Every statement passes guard_sql (parsed with sqlglot, not regex) before touching DuckDB:

• single statement only;
• top-level must resolve to a SELECT / CTE;
• any DDL/DML node anywhere in the tree (INSERT/UPDATE/DELETE/CREATE/ DROP/ALTER/ATTACH/PRAGMA/COPY…) is rejected;
• filesystem/external functions (read_csv, read_parquet, read_text, glob, …) are rejected so a query can't escape the warehouse;
• an explicit LIMIT is appended if missing.

If the guard rejects the SQL (or DuckDB errors), the error is fed back to the analytics agent for exactly one bounded correction attempt. The DuckDB connection is also opened read-only.

The grounding critic

After synthesis, the critic checks whether every claim in the answer is supported by the retrieved passages + SQL facts and returns {grounded, reason}. If ungrounded and attempts remain, the graph loops back to retrieve/synthesize with the critic's feedback; if still ungrounded after the max attempts, the answer is returned with an explicit low-confidence flag instead of a confident hallucination.

On the labeled eval set the system is well-behaved enough that the critic rarely needs to retry (it refuses cleanly on unanswerable questions rather than over-reaching). The retry control flow is therefore demonstrated two ways that do not depend on LLM nondeterminism:

• scripts/demo_retry.py drives the real compiled LangGraph app with a scripted critic that rejects a planted hallucination once, then accepts the corrected answer — printing the node trace router → doc_agent → synthesizer#1 → critic → doc_agent → synthesizer#2 → critic.
• tests/test_graph.py asserts the same loop deterministically (exactly one retry, then either recovery or a low-confidence flag).

Interview talking points

• Why hybrid retrieval? Dense embeddings capture semantics ("late deliveries" ≈ "on-time breaches"); BM25 nails rare exact tokens (lane ids, accessorial codes, dollar figures). RRF fuses them on rank, not raw score, so the incompatible scales of cosine similarity and BM25 never need calibration — a doc both retrievers like wins. k0=60 is the canonical constant.
• Why multi-agent / routing? Documents and the warehouse need different tools. Routing keeps each agent's prompt small and its job legible, and lets a both question fan out to documents and data and be synthesized together. The router defaults to the safe both on uncertainty — better to over-retrieve than to miss a source.
• Why a grounding critic? RAG's failure mode is a fluent answer unsupported by the evidence. A separate critic node turns "don't hallucinate" from a hope in the prompt into a checked, retried, and ultimately flagged property of the pipeline.
• SQL safety as a parsed boundary. Guarding generated SQL by parsing the AST (reject DDL/DML/filesystem functions, force a LIMIT, read-only connection) is far more robust than string matching and is the kind of boundary you want between an LLM and a database.
• Eval methodology / honesty. A small labeled set spanning docs-only / data-only / both / unanswerable, scored on routing, retrieval hit-rate, groundedness, and correctness — committed as results/eval.json so the numbers in this README are receipts, not claims. The build also surfaced and fixed a bad metric definition (see notes), which is itself the point of evaluating.

Honest notes

• Numbers are from a real run of scripts/run_eval.py against live OpenAI (gpt-4o-mini + text-embedding-3-small). They can vary slightly run-to-run because the chat model is not perfectly deterministic even at temperature 0.
• The data is synthetic and deterministic (seeded). The "facts" the analytics agent returns are real queries over that synthetic warehouse.
• Metric correction during the build: the first eval run defined the groundedness catch-rate as "system ended with a low-confidence flag," which scored 0% — because the system handles unanswerable questions by refusing cleanly (a correctly grounded refusal), not by flagging low confidence. The metric was corrected to measure the thing that actually matters: on an unanswerable question, did the system avoid fabricating. This is documented rather than hidden.
• make index requires an API key; the index is committed so most users never need to run it.

License

MIT © 2026 Mark Teji