ConfusionMapper

A Python tool for classifying wrong answer choices in multiple-choice questions by the type of cognitive error they represent, and for computing how well a human researcher and an AI rater agree on those labels.

What it does

In educational research, you cannot use qualitative labels in a statistical analysis unless two independent raters can produce roughly the same labels on the same items. The usual way to check this is Cohen's kappa.

ConfusionMapper handles the full workflow. You go through a set of distractors one by one and pick a label for each. If you have an OpenAI API key, an AI rater labels the same distractors in parallel. At the end, the program computes Cohen's kappa, draws a 4x4 confusion matrix of where the two raters agreed and disagreed, and saves everything to JSON.

It was built as the reliability check for a preregistered RCT on cognitive error feedback in government junior high schools (OSF: https://doi.org/10.17605/OSF.IO/YU6P5). Data collection in that study could only start once kappa cleared 0.70.

The four error types

The taxonomy is called the Confusion Fingerprint Index (CFI). It splits wrong answers into four categories:

Code	Name	What it looks like
RF	Recall Failure	No memory trace at all; the answer is essentially random
PK	Partial Knowledge	Almost right; the direction is correct but the model is incomplete
CF	Confabulation	A coherent misconception held with confidence
INT	Interference	A correct answer pulled from the wrong topic

Install

Once a versioned release is on PyPI:

pip install confusion-mapper

From source (until the first PyPI release, this is the recommended path):

git clone https://github.com/Manik-Maurya/Confusion-Mapper.git
cd Confusion-Mapper
pip install -e .

The only runtime dependency is openai>=1.0.0. tkinter ships with Python on most systems; on a minimal Debian or Ubuntu image install it with sudo apt-get install python3-tk. If tkinter is missing, ConfusionMapper drops to console output instead of the GUI.

Python 3.9 or higher.

For the test extras:

pip install -e ".[dev]"

Run

Without an API key (you label everything by hand):

python confusion_mapper.py

With the AI rater on:

# macOS / Linux
export OPENAI_API_KEY="your-key-here"
python confusion_mapper.py

# Windows
set OPENAI_API_KEY=your-key-here
python confusion_mapper.py

For each distractor, type 1 for RF, 2 for PK, 3 for CF, or 4 for INT. When you finish, the dashboard opens and the session writes itself to JSON.

Case study

A fully reproducible worked example lives in case_study/. It runs the entire pipeline (nominal kappa, weighted kappa under linear and quadratic schemes, BCa bootstrap 95% CI, confusion matrix, per-category stats) on the bundled 30-item paired-label set with a fixed seed, then writes the results to case_study/results/ (JSON summary, CSV matrix, CSV per-type stats, full bootstrap distribution, and a Markdown report). Regenerate with one command:

python case_study/run_case_study.py

Headline result on the bundled data: nominal kappa = 0.8653, BCa 95% CI = (0.6856, 1.0000), pre-registration gate PASSES at the 0.70 threshold.

Headless example

A non-interactive demo runs the three core functions on 30 pre-labelled distractors in under a second. No API key, no display:

python examples/demo.py

It prints kappa, the full confusion matrix, per-category agreement, and a PASS / HOLD verdict against the 0.70 gate. Swap in your own paired labels by replacing sample_data/example_labels.csv.

Tests

pip install -e ".[dev]"
pytest tests/ -v

88 tests, run in under a second. Covers the kappa formula (with hand-verified worked examples), weighted kappa (nominal / linear / quadratic), bootstrap confidence intervals (percentile and BCa, seedable), PABAK and the bias / prevalence diagnostics, Krippendorff's alpha at three measurement levels, the sample-size estimator, custom-taxonomy loading, the confusion matrix, and the mathematical invariants that tie everything together, the prompt-refinement engine, Fleiss's kappa for multi-rater panels, the CLI subcommand router, and cross-validation against published reference values. Doesn't need an API key or a display.

Advanced features

All of these use only the Python standard library and have their own tests.

Weighted kappa for ordinal taxonomies. Pass weights="linear" or weights="quadratic" to penalise far-apart disagreements more than adjacent ones:

from confusion_mapper import compute_cohens_kappa
r = compute_cohens_kappa(human, ai, weights="linear")
print(r["kappa"], r["weights"])

Bootstrap 95% confidence interval (BCa or percentile). Cohen's kappa is a point estimate; this gives you uncertainty around it. The seed makes the interval bit-identically reproducible:

from confusion_mapper import bootstrap_kappa_ci
ci = bootstrap_kappa_ci(human, ai, n_resamples=10000, method="bca", seed=42)
print(f"kappa = {ci['point_estimate']} (95% CI {ci['ci_lower']} to {ci['ci_upper']})")

PABAK and the kappa paradox. When marginals are highly skewed, Cohen's kappa can be paradoxically low even though raters agree on most items. compute_kappa_diagnostics returns PABAK (2*Po - 1) along with the bias and prevalence indices (Byrt et al. 1993) so you can tell whether a low kappa reflects genuine disagreement or just a skewed taxonomy:

from confusion_mapper import compute_kappa_diagnostics
d = compute_kappa_diagnostics(human, ai)
print(d['kappa'], d['pabak'], d['bias_index'], d['prevalence_index'])

Krippendorff's alpha. An alternative IRR coefficient that generalises to more than two raters and supports nominal, ordinal, or interval-level measurements:

from confusion_mapper import krippendorff_alpha
r = krippendorff_alpha(human, ai, level='ordinal')
print(r['alpha'], r['interpretation'])

Sample-size planning. Before you collect calibration labels, estimate how many you need to bound the kappa CI within a target half-width:

from confusion_mapper import recommend_sample_size
n = recommend_sample_size(expected_kappa=0.80, ci_half_width=0.10, n_categories=4)
print(f"Need at least {n['recommended_n']} items.")

Custom taxonomy via JSON. Swap the default CFI categories for any 2-or-more category nominal scheme. A working example sits at sample_data/custom_taxonomy.json:

from confusion_mapper import load_taxonomy_from_json, compute_cohens_kappa
codes, tax = load_taxonomy_from_json("sample_data/custom_taxonomy.json")
r = compute_cohens_kappa(human, ai, categories=codes)

Auto-generated prompt refinements. Feed the confusion matrix back into the prompt: suggest_prompt_refinements ranks the largest off-diagonal cells and emits a Markdown report with concrete contrastive instructions you can paste into the AI prompt.

from confusion_mapper import suggest_prompt_refinements
r = suggest_prompt_refinements(human, ai, top_k=3)
print(r['report_markdown'])

Fleiss's kappa for three or more raters. Cohen's kappa is two-rater. When you have a panel, use fleiss_kappa:

from confusion_mapper import fleiss_kappa
ratings = [
    {'RF': 3, 'PK': 0, 'CF': 0, 'INT': 0},  # all 3 raters: RF
    {'RF': 0, 'PK': 0, 'CF': 3, 'INT': 0},  # all 3 raters: CF
    {'RF': 1, 'PK': 2, 'CF': 0, 'INT': 0},  # split 1 RF / 2 PK
]
print(fleiss_kappa(ratings)['kappa'])

Command line

ConfusionMapper has a Unix-style CLI with five subcommands. Pipe any CSV with human_label, ai_label columns through it:

python -m confusion_mapper kappa       sample_data/example_labels.csv --bootstrap 10000
python -m confusion_mapper alpha       sample_data/example_labels.csv --level ordinal
python -m confusion_mapper diagnostics sample_data/example_labels.csv
python -m confusion_mapper refine      sample_data/example_labels.csv --top 3
python -m confusion_mapper plan        --kappa 0.80 --ci 0.05 --categories 4

Every subcommand prints JSON (or Markdown for refine) on stdout, so you can redirect into a results bundle without writing any glue code.

Why a 4x4 confusion matrix instead of just kappa

The single kappa number tells you how much you and the AI agree overall. It does not tell you which category boundary is causing the disagreement. CF vs INT is the hardest distinction in the CFI taxonomy (a confident wrong belief looks a lot like a correct answer applied to the wrong topic), and the [CF, INT] cell of the confusion matrix is where most disagreement tends to land. Looking at the full 4x4 matrix lets you see exactly that, and rewrite the AI prompt or your own rubric until the cell stops glowing.

A note on ethics

The AI rater is a starting point, not a substitute for a second human. For published research you should still run kappa between two trained human raters on the same calibration set. Treat the AI labels as a way to surface boundaries that need rubric work, not as a replacement for human judgment.

How to cite

Archived release on Zenodo: https://doi.org/10.5281/zenodo.20807432

APA:

Maurya, M. (2026). ConfusionMapper: A Python Tool for AI-Assisted Distractor Classification and Inter-Rater Reliability Computation in Cognitive Error Taxonomy Research (Version 1.0.0) [Software]. Zenodo. https://doi.org/10.5281/zenodo.20807432

BibTeX:

@software{maurya2026confusionmapper,
  author    = {Maurya, Manik},
  title     = {ConfusionMapper: A Python Tool for AI-Assisted Distractor Classification
               and Inter-Rater Reliability Computation in Cognitive Error Taxonomy Research},
  year      = {2026},
  version   = {1.0.0},
  doi       = {10.5281/zenodo.20807432},
  url       = {https://doi.org/10.5281/zenodo.20807432}
}

Contributing

Bug reports, feature requests, and pull requests are welcome. See CONTRIBUTING.md and CODE_OF_CONDUCT.md.

Acknowledgements

Initial development was completed as part of Stanford Code in Place 2026. The tool is used in the Confusion Fingerprint Index research programme at the Department of Cognitive Science, IIT Kanpur.

License

MIT License © 2026 Manik Maurya. See LICENSE for details.