AgreeKit

Tool-agnostic inter-coder reliability and disagreement adjudication for qualitative coding.

AgreeKit reads coding from a tidy CSV or from REFI-QDA .qdpx exports, computes the inter-coder reliability statistics reviewers expect (percent agreement, Cohen's and Fleiss' kappa, and Krippendorff's alpha, overall and per code), lists the units where coders disagree so you can adjudicate them, and writes a publication ready report with a methods paragraph you can paste into a paper.

It is tool agnostic: any coding that can be shaped into a tidy CSV works, and it reads .qdpx files directly. It runs fully offline with no paid services.

The gap it fills

The reliability statistics exist as low level libraries (the maintained krippendorff package, textbook kappa formulas), but the workflow around them is tedious glue that nobody packaged: reading arbitrary coding exports, unitizing highlight style span coding, computing the right coefficients, listing disagreements, and adjudicating them. CAQDAS tools that include an agreement report lock you into their own project format. AgreeKit is the standalone, tool agnostic option. See ../OPPORTUNITIES.md for the evidence.

Full scope (what this project ships)

Core:

• Read coding from a tidy long CSV (unit, coder, code, and optional start/end for spans) with configurable column names, or from one REFI-QDA .qdpx per coder.
• Unitize span coding into atomic intervals (or per character) so highlight style coding becomes comparable across coders. Unit based coding is used directly.
• Compute overall reliability: percent agreement, Cohen's kappa (two coders), Fleiss' kappa (any number of coders, each unit rated by all), and Krippendorff's alpha (any number of coders, handles missing data).
• Compute per code presence agreement (alpha and percent agreement for each code treated as present or absent), with prevalence.
• List the units where coders disagree, export them to a CSV for adjudication, and reconcile the filled in decisions into a single resolved coding.
• Render a report as HTML, Markdown, or JSON, including a ready to paste methods paragraph with the coefficient values.

Install

Requires Python 3.10 or newer (developed and tested on 3.14).

python -m venv .venv
# Windows PowerShell:  .venv\Scripts\Activate.ps1
# Git Bash / Linux / macOS:  source .venv/bin/activate
pip install -e .

Exact pinned environment:

pip install -r requirements-lock.txt
pip install -e . --no-deps

Development tools: pip install -e ".[dev]" then pytest.

Usage

Report from a tidy CSV

Your CSV needs a coder column, a code column, and either a unit column (unit based coding) or start and end columns with a document column (span based coding):

unit,coder,code
r1,alice,Positive
r1,bob,Positive
r2,alice,Negative
r2,bob,Neutral

agreekit report coding.csv --md report.md --json report.json

Prints the methods paragraph and writes the reports. Example paragraph:

2 coders independently coded 6 units (6 rated by all coders). Inter-coder
reliability was assessed as follows: Krippendorff's alpha (nominal) was 0.766;
percent agreement was 83.3%; Cohen's kappa was 0.750; Fleiss' kappa was 0.745.
1 units required adjudication.

Custom column names and span data:

agreekit report spans.csv --coder-col rater --code-col label \
  --document-col doc --start-col begin --end-col finish --granularity atomic

Report from .qdpx exports (one per coder)

agreekit report --qdpx alice.qdpx --qdpx bob.qdpx --coders alice,bob --out report.html

Adjudicate disagreements

# 1. Export the units where coders disagree
agreekit disagreements coding.csv --out disagreements.csv

# 2. Open disagreements.csv and fill the empty "final" column with the agreed code

# 3. Reconcile into a single resolved coding
agreekit adjudicate coding.csv --resolved disagreements.csv --out reconciled.csv

Run python -m agreekit ... if you prefer not to install the console script.

Configuration

Column names and unitization settings come from defaults, an optional TOML file (--config), or CLI flags. AgreeKit reads no environment variables and makes no network calls. Example TOML:

[agreekit]
coder_col = "rater"
code_col = "label"
granularity = "atomic"   # auto, atomic, char, or unit

How span coding is made comparable

When coders highlight character ranges, there is no shared list of units to compare. AgreeKit builds one: it collects every distinct span boundary in a document and forms the atomic intervals between them; for each interval, each coder's code is the span that covers it, or the uncoded category if none does. This turns overlapping, unaligned highlights into a units by coders matrix that the standard coefficients can score. Use --granularity char for a finer, more conservative per character unitization.

How to run the tests

pip install -e ".[dev]"
pytest

The suite (33 tests) checks the statistics against known textbook values, builds its own .qdpx fixtures in memory, and runs fully offline. It covers the stats, unitization (unit, span, and char), the CSV and .qdpx readers, adjudication, the report renderers, the CLI, and the security properties below.

Architecture notes

• models.py segments, the units matrix, metrics, and the report.
• config.py column names and unitization settings.
• readers.py the tidy CSV reader and the per coder .qdpx reader.
• qdpx_reader.py a small hardened REFI-QDA span reader.
• unitize.py building the units matrix from segments.
• stats.py percent agreement, Cohen's and Fleiss' kappa, Krippendorff's alpha.
• adjudicate.py disagreement export, reconciliation.
• report.py HTML, Markdown, and JSON rendering plus the methods paragraph.
• safe.py formula injection safe CSV writing.
• cli.py the Typer command line interface.

Security notes

• XXE and entity expansion are blocked. .qdpx XML is parsed with defusedxml, which refuses external entities and entity expansion, so a malicious .qdpx cannot read local files or blow up memory. There is a test that an external entity payload is rejected.
• Archive limits. A .qdpx is checked for entry count and uncompressed size before it is read, guarding against zip bombs.
• Spreadsheet formula injection is prevented. Code labels and unit text are user controlled, so every cell written to the disagreements and reconciled CSVs is sanitized (dangerous leading characters are quoted). There is a test for this.
• Report output is escaped. The HTML report is rendered with Jinja2 autoescaping, so a code label containing markup cannot inject script into the report. There is a test for this.
• Bounded input and no unsafe execution. Input files over a configurable size limit are refused. There is no eval, no exec, no pickle, and no yaml.load. Configuration is read with the standard library tomllib. No network calls are made.

License

MIT. See LICENSE.

Reliability coefficients are summaries, not verdicts. Interpret them in the context of your coding scheme and the number of categories, and report the measure you used alongside the value.