LineageRAG

Copy-aware evidence corroboration for retrieval-augmented verification

LineageRAG asks a simple question before treating agreement as confidence:

Did these documents reach the same conclusion independently, or did they copy one another?

Twenty pages can repeat one mistaken bulletin. A verifier that counts URLs sees twenty witnesses. LineageRAG infers likely derivation links, collapses dependent documents into origin families, and combines evidence at the family level.

The repository is a complete CPU-only research system. It generates controlled evidence cascades, trains a calibrated dependency classifier, infers sparse document genealogy, trains dataset-specific stance models, runs copy-aware Bayesian verification, compares baselines, and produces an interactive evidence graph. Smoke mode needs no API key, GPU, model download, or external dataset.

Why source count is not evidence count

Domain and URL deduplication address repeated locations. Near-duplicate filtering catches almost identical text. Neither is enough when copied information crosses domains, receives a heavy rewrite, preserves a rare factual error, or cites a common upstream report.

LineageRAG estimates dependence from observable signals:

• publication order
• citation and quotation links
• exact and paraphrased text overlap
• stable SimHash similarity
• entity and number overlap
• shared rare error markers
• URL, domain, and title similarity

The resulting weighted graph is clustered into likely origin families. A bounded family weight prevents copied descendants from contributing linearly to the claim posterior.

Retrieved documents
        |
        v
Observable pair features
        |
        v
Directed lineage graph
        |
        v
Evidence families
        |
        v
Family-level Bayesian verifier
        |
        +--> claim posterior
        +--> effective evidence count
        +--> false-corroboration risk
        +--> next independent search intent

Quick start

Python 3.11 or newer is required.

python -m venv .venv

Activate the environment on Linux or macOS:

source .venv/bin/activate

Activate it on Windows PowerShell:

.\.venv\Scripts\Activate.ps1

Install and validate the project:

python -m pip install -e ".[dev]"
pytest -q

Run the complete benchmark:

python -m lineagerag.cli smoke --seed 14 --out artifacts/smoke
python -m lineagerag.cli inspect-run artifacts/smoke

The smoke run creates at least 200 claims and 2,000 documents across nine controlled cascade templates. It evaluates LineageRAG and ten non-oracle baselines.

Interactive genealogy

Open the generated file in any modern browser:

artifacts/smoke/evidence_genealogy.html

The demo is standalone. Plotly is embedded into the file, so the graph works without a network connection.

Node fill identifies a predicted origin family. The border shows stance. Diamond nodes are inferred roots. The summary cards compare raw source count with effective evidence.

CLI

# Generate a reusable synthetic corpus with hidden oracle annotations
python -m lineagerag.cli generate-synthetic \
  --config configs/synthetic.yaml \
  --out data/synthetic

# Run the configured benchmark
python -m lineagerag.cli evaluate \
  --config configs/synthetic.yaml \
  --out artifacts/synthetic

# Copy a completed standalone demo
python -m lineagerag.cli demo \
  --run artifacts/smoke \
  --out artifacts/demo

# Register a separately acquired local dataset
python -m lineagerag.cli prepare averitec \
  --source /path/to/local/data \
  --out data/averitec

# Train the calibrated dependency model
python -m lineagerag.cli train-lineage \
  --out models/dependency.json

# Run both held-out real claim benchmarks
python -m lineagerag.cli evaluate-real \
  --scifact-source data/external/scifact \
  --averitec-source data/external/averitec/AVeriTeC-7c62d1ec8df3fb560d6efe2b85fa191135636f81 \
  --lineage-model models/dependency.json \
  --out artifacts/real

# Build and test chronological OpenAlex citation cascades
python -m lineagerag.cli evaluate-citations \
  --scifact-source data/external/scifact \
  --lineage-model models/dependency.json \
  --out artifacts/citations

Real-data adapters never access the network or redistribute data. The explicit PowerShell acquisition script downloads pinned sources only after license acceptance and checks each archive hash. The offline benchmark remains available without research data.

Method

For each document pair $i, j$, a transparent logistic model estimates a shared-derivation probability from observed features:

$$p_{ij} = \sigma(b + w^\top x_{ij})$$

The default trained model uses claim-disjoint synthetic graphs. A second seed provides probability calibration and an operating threshold chosen for high validation precision. An auditable hand-specified scorer remains available for offline smoke tests.

Edges above the calibrated threshold form a directed graph. Explicit citations determine direction first, followed by publication order. Copy-family inference keeps at most one parent per document to reduce bridge errors.

Within each predicted family $f$, document stances are pooled into a family score $s_f \in [-1, 1]$. The support posterior is:

$$\mathrm{logit}\,P(y=1 \mid E) = \mathrm{logit}\,P(y=1) + \sum_f 1.6\,w_f\,s_f$$

The effective family weight grows slowly with copied descendants:

$$w_f = c_f \min \left( 1 + \alpha \log(1 + |f| - 1), \tau \right)$$

Here $c_f$ combines stance confidence, lineage confidence, and configurable source reliability. The cap $\tau$ is the key safeguard. One family cannot gain unbounded influence by producing more copies.

Dataset-specific stance classifiers use TF-IDF features and logistic regression. Training, validation, and test claims are disjoint. Each method receives its own validation-fitted claim-probability calibrator.

Benchmark design

The generator provides nine graph families:

• independent witnesses
• copied bulletin star
• citation chain
• paraphrase tree
• cross-domain syndication
• shared false root
• delayed correction cascade
• two-origin merge
• mixed support and refute families

Observed document metadata and hidden oracle annotations are saved separately. Inference receives documents and extracted stances only. Oracle files are read during metric calculation and by explicit upper-bound baselines.

Synthetic text mutations include exact copying, attributed quotes, light and heavy paraphrases, entity-preserving rewrites, number changes, title rewrites, summaries, and boilerplate insertion.

Baselines

All methods use the same claims, retrieval order, stance observations, and evaluator.

Method	What it controls
Standard RAG	Counts every selected document independently
URL deduplication	Removes repeated URLs
Domain deduplication	Keeps one document per domain
Domain and near duplicate	Applies both simple controls together
Near-duplicate deduplication	Removes very high lexical similarity
MMR diversity	Trades relevance for lexical diversity
Reliability weighted	Weights sources by agreement and ignores lineage
Contradiction aware	Balances support and refuting stances
Reliability and contradiction	Combines the preceding controls
Counterfactual evidence	Preserves evidence from both sides
LineageRAG	Discounts likely derivation families

Oracle family baselines are included only as diagnostic upper bounds.

Research data

Raw external data is excluded from Git. Source URL, upstream commit, license, archive checksum, and local destination are recorded under data/manifests.

Resource	Role	Split	License
SciFact	Scientific claims and direct document stance	Official dev test	CC BY 4.0 and ODC-By 1.0
AVeriTeC	Real web claims and annotated evidence URLs	Chronological 70/15/15	CC BY-NC 4.0
OpenAlex	Citation links and publication dates	Chronological 60/20/20	CC0

Review the terms before downloading. AVeriTeC is noncommercial.

.\scripts\acquire_research_data.ps1 -AcceptResearchLicenses

See docs/real_data.md for normalization, leakage controls, and limitations.

Latest research result

The overall kill criterion fails. This is an important result, not a hidden caveat.

Evaluation	Size	LineageRAG result	Comparator or interpretation
Learned synthetic	200 claims	Brier 0.165, false corroboration 0.000	Controlled sanity check
SciFact	300 test claims	Brier 0.162	Combined dedup 0.164, paired interval excludes zero
AVeriTeC	536 test claims	Brier 0.185	Combined dedup 0.187, paired interval includes zero
OpenAlex	8 test cascades	Citation edge F1 1.000	Explicit citation is an input feature

SciFact passes the held-out calibration criterion. AVeriTeC does not establish a statistically reliable improvement. OpenAlex confirms citation metadata handling, not hidden copying detection.

On learned synthetic cascades, iterative search uses 7.58 documents per claim on average instead of all 11.15 available documents. It reaches both stopping thresholds for 67 of 200 claims. The remaining 133 claims stop at the explicit budget limit.

Classification quality is still weak. SciFact accuracy is 0.373 with 0.990 abstention. AVeriTeC accuracy is 0.347 with 0.466 abstention. This version is a research prototype, not a fact-checking product.

See reports/research_summary.md and reports/latest_results.json for exact values.

Outputs

Every completed run contains:

Artifact	Purpose
metrics.json	Run metadata and all method metrics
metrics_by_method.csv	Method comparison
metrics_by_template.csv	Breakdown by cascade type
lineage_metrics.json	Edge and family recovery
claim_results.csv	Claim-level predictions
predicted_edges.csv	Inferred directed graph edges
predicted_families.csv	Predicted family membership
trajectories.jsonl	Active-search decisions
calibration.png	Reliability diagram
false_corroboration_by_template.png	Failure-rate comparison
evidence_genealogy.html	Standalone interactive graph
summary.md	Run report and kill-criterion check

Metrics

Claim verification uses accuracy, macro F1, Brier score, negative log likelihood, expected calibration error, abstention rate, search cost, and false-corroboration rate.

Lineage recovery uses edge precision, recall, F1, area under the precision-recall curve, adjusted Rand index, normalized mutual information, pairwise family F1, and independent-family recall.

A false-corroboration event requires all of the following:

1. The method assigns high support probability
2. The claim is false, mixed, or unsupported
3. Several retrieved documents support the claim
4. Hidden lineage shows fewer than the required number of independent support origins

Reproducibility

Randomness enters only through the synthetic generator and seeded bootstrap utilities. The seed is stored in every run. Stable hashing is used for text features, so Python hash randomization does not change lineage scores.

See docs/reproducibility.md for exact commands, artifact schemas, and extension points.

Research boundaries

The bundled benchmark is a controlled engineering test. Synthetic markers make lineage recoverable and overstate performance compared with natural publication cascades.

The supervised stance models are lightweight and do not provide the language understanding needed for deployment. Source reliability estimates have limited coverage. Time metadata can be missing, delayed, or manipulated.

AVeriTeC provides evidence URLs but no gold copying families. OpenAlex citations are relationships, not copying labels. Natural news genealogy remains unverified.

The paired AVeriTeC confidence interval includes zero, so the strict project kill criterion fails for this release. News lineage inference should be redesigned rather than promoted.

Repository policy

This repository contains an original implementation, generated synthetic data, and project-authored documentation. It does not include third-party papers, external datasets, model weights, copied code, or reproduced result tables.

See NOTICE and docs/originality.md for the source boundary.

Contributing

Contributions are welcome for learned lineage models, temporal cascade adapters, calibrated stance models, and evaluation datasets with clear redistribution terms.

Read CONTRIBUTING.md before opening a pull request.

License

LineageRAG is released under the MIT License.