Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 2 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Full conventions are in [CONTRIBUTING.md](CONTRIBUTING.md). In short:

- Conventional Commits. One logical change per commit; keep formatting-only changes in their own `style:` commit (never bundle a repo-wide reformat into a feature commit).
- Trunk-based: `main` stays green and releasable. Small changes go straight to `main`; non-trivial work uses a short-lived branch merged back promptly so `main` never goes stale.
- `main` is branch-protected: PRs required (no direct push, including admins), CI green across the 3.11/3.12/3.13 matrix, linear history, squash-merge only.
- Pre-commit and CI must pass before merging to `main`.
- Never commit scaffolding or secrets; the pre-commit guard (`scripts/check_no_scaffolding.sh`) blocks them. `AGENTS.md` is the shareable context; the gitignored `CLAUDE.md` is private.
- Tag releases `vX.Y.Z`. For any reported number, tag the exact commit that produced it so it is citeable.
Expand All @@ -39,16 +40,9 @@ Python 3.11+ (CI matrix 3.11/3.12/3.13). Strict pyright over `src` and `tests`.

## Binding rules (do not violate)

- Numerical correctness is non-compressible (SPEC R19). A wrong DIF or alpha is reputational damage. Test every statistic against a pinned external reference (the `krippendorff` package, R `irr`/`TAM` fixtures, statsmodels), not just internal consistency. Never cut the rigor pass to save time.
- Numerical correctness is non-compressible (SPEC R19): a wrong DIF or alpha is reputational damage. Test every statistic against a pinned external reference (the `krippendorff` package, R `irr`/`TAM` fixtures, statsmodels) before implementing: write the reference-value assertions first, watch them fail, then implement until they pass. When a reference and a literal disagree, the reference wins; tolerances are never loosened to pass.
- Runtime dependencies are limited to `numpy`, `pandas`, `scipy`, `krippendorff`. Any addition is justified in the PR. `pingouin` and `statsmodels` are dev/test oracles only, never runtime imports.
- Type hints on every public function, explicit return types (pyright runs strict). No bare `except`, no mutable default args, no `import *`.
- Test-first for every feature and fix. For a stats function the failing-test target is a pinned external reference value, not internal consistency: write the reference-value assertions against a known oracle up front, watch them fail, then implement until they pass.
- Git: Conventional Commits.

## Gotchas

- The numerical-reference convention is the shape of the test loop here. When a reference value and a literal disagree, the reference wins and the literal is corrected; assertion tolerances are not loosened to make a test pass.
- `--cov` is wired into pytest `addopts`, so a bare `uv run pytest` already produces coverage.

## Methods are provisional

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ It prints the report card below, with the live numbers exactly as the command pr

Reliability rests on two measures, Krippendorff's alpha and ICC. Both rise when raters converge: alpha is a chance-corrected agreement coefficient, ICC a variance-ratio reliability coefficient (the share of score variance that is rater-consistent signal). ICC(2,k) is higher than ICC(2,1) because averaging three raters cancels some of the per-rater noise.

Krippendorff's bands put the demo's 0.554 below even the tentative floor, so these coherence scores are only marginally reliable, which is the kind of result this tool exists to surface:
Krippendorff's bands put the demo's 0.554 below even the tentative floor, so these coherence scores are only marginally reliable. That is the kind of result this tool exists to surface:

| Krippendorff α | Reading |
|---|---|
Expand All @@ -86,7 +86,7 @@ The reliability estimators assume a complete crossed design. On a matrix with mi

### DIF

DIF, differential item functioning, asks whether the panel scores abstractive outputs differently from extractive outputs once you condition on the rest-score proxy for overall quality. The "item" under audit is the rubric criterion rather than a shared test item, since each stratum scores different outputs. The engine is ordinal logistic regression in the Zumbo tradition (single-pass, not lordif's iterative purification), run as three nested proportional-odds models, so it reports a uniform-DIF test, a nonuniform-DIF test, and an effect size (the Nagelkerke pseudo-R-squared change) classified A, B, or C by the Jodoin-Gierl thresholds. The demo shows why the card prints both a p-value and an effect size: at n = 4800 the uniform-DIF test is significant (p = 0.0005), but the effect size is 0.002, class A, which is negligible. The signal is detectable; the magnitude is not.
DIF, differential item functioning, asks whether the panel scores abstractive outputs differently from extractive outputs once you condition on the rest-score proxy for overall quality. The "item" under audit is the rubric criterion rather than a shared test item, since each stratum scores different outputs. The engine is ordinal logistic regression in the Zumbo tradition (single-pass, not lordif's iterative purification), run as three nested proportional-odds models. It reports a uniform-DIF test, a nonuniform-DIF test, and an effect size (the Nagelkerke pseudo-R-squared change) classified A, B, or C by the Jodoin-Gierl thresholds. The demo shows why the card prints both a p-value and an effect size: at n = 4800 the uniform-DIF test is significant (p = 0.0005), but the effect size is 0.002, class A, which is negligible. The signal is detectable; the magnitude is not.

The matching variable is a leave-one-rater-out rest score across the three expert raters, which uses the same exchangeable-rater assumption as the reliability pillar. That rest score detects bias relative to the rater panel and understates bias the whole panel shares, so the card labels this path panel-relative DIF. For a stronger DIF analysis, pass an explicit external quality conditioner: `audit(ratings, focal=..., reference=..., conditioner=...)` accepts a sample-id to quality-score mapping. External-conditioner DIF supports instrument-level interpretation only when the conditioner is valid, independent, and appropriate for the quality construct being matched. Read DIF output as a screening audit, not a confirmatory significance claim. When a p-value lands near a decision threshold, `cluster_bootstrap_dif` runs the same engine with item-block resampling and returns a 95% cluster-robust interval alongside the unchanged point estimate.

Expand Down
22 changes: 4 additions & 18 deletions docs/REFERENCES.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,6 @@
# References

The methods metajudge implements are credited to their original authors here.
Every entry is carried verbatim from the decision record where the method was
adopted; the per-method rationale and any caveats about applying a
method outside its original validation live in [decisions/](decisions/). This
file is the consolidated bibliography so the credits travel with the package
rather than sitting only inside individual ADRs.
The methods metajudge implements are credited to their original authors here. Every entry is carried verbatim from the decision record where the method was adopted; the per-method rationale and any caveats about applying a method outside its original validation live in [decisions/](decisions/). This file is the consolidated bibliography so the credits travel with the package rather than sitting only inside individual ADRs.

## Reliability: Krippendorff's alpha and ICC

Expand Down Expand Up @@ -121,15 +116,6 @@ Fabbri, A. R., Kryscinski, W., McCann, B., Xiong, C., Socher, R., and Radev, D.

## Software and reference implementations

The runtime alpha computation wraps the `krippendorff` Python package. The
statistics are pinned in the test suite against external reference
implementations: R `MASS::polr` for the proportional-odds DIF fit, and the
`pingouin` and `statsmodels` Python packages as ICC and logistic-regression
oracles. `pingouin` and `statsmodels` are test-only oracles and are never
imported at runtime.

AutoRubric: a Python library for rubric-based LLM-as-judge grading with
agreement metrics and bootstrap confidence intervals (https://autorubric.org,
https://github.com/delip/autorubric). Adjacent, shipped practitioner coverage
of the commodity reliability layer; cited in SPEC.md as prior art this
library does not compete with.
The runtime alpha computation wraps the `krippendorff` Python package. The statistics are pinned in the test suite against external reference implementations: R `MASS::polr` for the proportional-odds DIF fit, and the `pingouin` and `statsmodels` Python packages as ICC and logistic-regression oracles. `pingouin` and `statsmodels` are test-only oracles and are never imported at runtime.

AutoRubric: a Python library for rubric-based LLM-as-judge grading with agreement metrics and bootstrap confidence intervals ([https://autorubric.org](https://autorubric.org), [https://github.com/delip/autorubric](https://github.com/delip/autorubric)). Adjacent, shipped practitioner coverage of the commodity reliability layer; cited in SPEC.md as prior art this library does not compete with.
2 changes: 1 addition & 1 deletion src/metajudge/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""metajudge: audit an LLM judge/rubric instrument before you trust its numbers.
"""metajudge: A reliability and DIF report card for LLM-judge and human-rater scoring instruments.

Two pillars: inter-rater reliability (Krippendorff's alpha, ICC(2,1)/(2,k)) and
differential item functioning across output strata (ordinal logistic-regression DIF).
Expand Down
Loading