Human Annotation Evaluation Toolkit

Evaluation toolkit for multi-annotator human annotation research.

The toolkit reads tidy CSV annotation files and produces agreement metrics, label-distribution analysis, disagreement tables, plots, and a Markdown report. It is designed for human-evaluation workflows in LLM evaluation, annotation pilots, post-training data quality checks, and dataset construction.

Metrics

• Cohen's kappa for each annotator pair
• mean pairwise Cohen's kappa
• Fleiss' kappa for balanced multi-annotator data
• Krippendorff's alpha for nominal labels
• label distribution analysis
• annotator-level label distribution analysis
• item-level disagreement analysis

Input Format

Default CSV schema:

item_id,annotator,label
item_001,ann_a,technical_jargon
item_001,ann_b,technical_jargon
item_002,ann_a,community_slang
item_002,ann_b,implicit_knowledge

Each row should represent one annotation decision by one annotator for one item.

Custom column names are supported:

annotation-eval annotations.csv --item-col example_id --annotator-col worker_id --label-col category

Quickstart

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
annotation-eval examples/sample_annotations.csv --out outputs/sample_report

Without installation:

python -m annotation_eval_toolkit.cli examples/sample_annotations.csv --out outputs/sample_report

Outputs

The output directory contains:

metrics.json
pairwise_cohen_kappa.csv
label_distribution.csv
annotator_label_distribution.csv
item_disagreements.csv
report.md
plots/
  label_distribution.svg
  pairwise_cohen_kappa.svg
  top_item_disagreements.svg

Disagreement Analysis

item_disagreements.csv ranks items by disagreement rate and entropy. Each row includes:

• item ID
• number of annotations
• number of unique labels
• majority label
• majority fraction
• disagreement rate
• entropy
• label counts
• annotator labels

This makes it easier to find ambiguous examples, weak rubric definitions, edge cases, and items that need adjudication.

Notes On Metrics

Fleiss' kappa is computed only when every included item has the same number of ratings. If your data has missing annotator labels, use pairwise Cohen's kappa and Krippendorff's alpha.

Krippendorff's alpha is implemented for nominal labels. Ordinal, interval, and ratio distance functions are natural future extensions.

Development

python -m unittest discover -s tests

Roadmap

• support multiple label columns in one run
• add adjudicated gold-label comparison
• add bootstrap confidence intervals
• add ordinal Krippendorff alpha
• add confusion matrices by annotator pair
• add HTML report export