Phantom Merge

When Your Large Language Model Agents Pick One but Tell You About Another

Code release for diagnosing and mitigating anchor–claim binding failures in tool-using LLM agents.

Overview · Quick Start · Layout · How to Cite

---

Overview

Phantom Merge (PM) is a binding failure in multi-step agents: the run commits to a protocol anchor (a ShoppingBench product or a FHIR resource), but the final answer attributes to that anchor facts its observed evidence cannot support. Standard task-success checks may still pass.

This repository provides:

• Diagnosis — anchor–claim labeling with a shared claim taxonomy (binding/)
• Mitigation — BCP-Detect and MSPS probes (probes/)
• Reference runs — paper-aligned trajectories and table exports under results/

Evaluations use ShoppingBench and FHIR-AgentBench with Qwen3-32B and Mistral-Small-3.2-24B-Instruct-2506 agent backbones. Rollouts are produced in the upstream benchmarks; this repo consumes rollout logs and runs the PM judge plus downstream analysis.

Qualitative example of an anchor–claim binding failure (see paper for full discussion).

---

Quick Start

git clone https://github.com/LabRAI/PhantomMerge.git
cd PhantomMerge
git lfs pull

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
bash reproduce/verify.sh

verify.sh checks the sealed snapshot against the paper artifact index. To print headline metrics from sealed JSON:

python scripts/print_paper_metrics.py

Artifact index: ARTIFACT.json.

---

Repository Layout

binding/shopping/   PM judge + object KB (ShoppingBench)
binding/fhir/       PM judge (FHIR-AgentBench)
probes/             BCP-Detect, OC-BCP, MSPS
results/            Per-trajectory labels and table exports
reproduce/          verify.sh and table-aligned reproduction scripts
configs/            Reproducibility manifest (models, decoding, probe settings)
asserts/            Paper figures used in this README

Sealed claim labels use the frozen release protocol (pipeline_version shopping_pm_judge_release / fhir_pm_judge_release). Shared taxonomy: binding/shopping/src/analysis/pm_eval_release.py.

---

Benchmarks

Benchmark	Role in this repo
ShoppingBench	Product-recommendation agent rollouts → PM judge
FHIR-AgentBench	Clinical FHIR agent rollouts → PM judge

Setup and rollout commands: binding/shopping/README.md, binding/fhir/README.md.

---

Supplemental: MCR-100

results/supplemental/mcr100/mcr100.jsonl ships 100 author-curated anchor–claim reference cards (50 Shopping + 50 FHIR) for protocol walkthrough and qualitative illustration.

---

How to Cite

If you use this code release or build on the Phantom Merge diagnosis protocol, please cite the arXiv preprint (link to be added):

@article{zhu2026phantommerge,
  title   = {Phantom Merge: When Your Large Language Model Agents Pick One but Tell You About Another},
  author  = {Zhu, Wangshu and Guo, Qianwen and Dong, Yushun},
  journal = {arXiv preprint arXiv:TODO},
  year    = {2026},
  url     = {https://arxiv.org/abs/TODO}
}

See also CITATION.bib.

---

License

MIT (LICENSE).

---

Responsible AI (RAI) Lab · Florida State University

https://github.com/LabRAI