From 9b25545d182b430b390985f76526c8ffe1f887ea Mon Sep 17 00:00:00 2001 From: Brittany Reese Markides <12763491+brittanyreese@users.noreply.github.com> Date: Fri, 3 Jul 2026 15:12:32 -0500 Subject: [PATCH 1/3] docs(agents): document main branch protection, dedupe binding rules main now requires PRs (CI green across 3.11/3.12/3.13, linear history, squash-merge only) - record it in Git workflow instead of leaving the doc to imply direct push still works. Also collapses the test-first and reference-value bullets that said the same thing twice, drops the redundant Conventional-Commits bullet already covered above, and removes the Gotchas section, whose two lines duplicated the Commands comment and the binding rule. --- AGENTS.md | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a33f009..9d66dad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,6 +22,7 @@ Full conventions are in [CONTRIBUTING.md](CONTRIBUTING.md). In short: - Conventional Commits. One logical change per commit; keep formatting-only changes in their own `style:` commit (never bundle a repo-wide reformat into a feature commit). - Trunk-based: `main` stays green and releasable. Small changes go straight to `main`; non-trivial work uses a short-lived branch merged back promptly so `main` never goes stale. +- `main` is branch-protected: PRs required (no direct push, including admins), CI green across the 3.11/3.12/3.13 matrix, linear history, squash-merge only. - Pre-commit and CI must pass before merging to `main`. - Never commit scaffolding or secrets; the pre-commit guard (`scripts/check_no_scaffolding.sh`) blocks them. `AGENTS.md` is the shareable context; the gitignored `CLAUDE.md` is private. - Tag releases `vX.Y.Z`. For any reported number, tag the exact commit that produced it so it is citeable. @@ -39,16 +40,9 @@ Python 3.11+ (CI matrix 3.11/3.12/3.13). Strict pyright over `src` and `tests`. ## Binding rules (do not violate) -- Numerical correctness is non-compressible (SPEC R19). A wrong DIF or alpha is reputational damage. Test every statistic against a pinned external reference (the `krippendorff` package, R `irr`/`TAM` fixtures, statsmodels), not just internal consistency. Never cut the rigor pass to save time. +- Numerical correctness is non-compressible (SPEC R19): a wrong DIF or alpha is reputational damage. Test every statistic against a pinned external reference (the `krippendorff` package, R `irr`/`TAM` fixtures, statsmodels) before implementing: write the reference-value assertions first, watch them fail, then implement until they pass. When a reference and a literal disagree, the reference wins; tolerances are never loosened to pass. - Runtime dependencies are limited to `numpy`, `pandas`, `scipy`, `krippendorff`. Any addition is justified in the PR. `pingouin` and `statsmodels` are dev/test oracles only, never runtime imports. - Type hints on every public function, explicit return types (pyright runs strict). No bare `except`, no mutable default args, no `import *`. -- Test-first for every feature and fix. For a stats function the failing-test target is a pinned external reference value, not internal consistency: write the reference-value assertions against a known oracle up front, watch them fail, then implement until they pass. -- Git: Conventional Commits. - -## Gotchas - -- The numerical-reference convention is the shape of the test loop here. When a reference value and a literal disagree, the reference wins and the literal is corrected; assertion tolerances are not loosened to make a test pass. -- `--cov` is wired into pytest `addopts`, so a bare `uv run pytest` already produces coverage. ## Methods are provisional From 99ce016428e83aca511479172523e362f2b26242 Mon Sep 17 00:00:00 2001 From: Brittany Reese Markides <12763491+brittanyreese@users.noreply.github.com> Date: Fri, 3 Jul 2026 15:16:01 -0500 Subject: [PATCH 2/3] docs: tighten README and package docstring wording Shortens a closing sentence in the reliability section and states the package docstring as a plain description rather than an instruction. --- README.md | 4 ++-- src/metajudge/__init__.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index d9e0601..864ff0a 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ It prints the report card below, with the live numbers exactly as the command pr Reliability rests on two measures, Krippendorff's alpha and ICC. Both rise when raters converge: alpha is a chance-corrected agreement coefficient, ICC a variance-ratio reliability coefficient (the share of score variance that is rater-consistent signal). ICC(2,k) is higher than ICC(2,1) because averaging three raters cancels some of the per-rater noise. -Krippendorff's bands put the demo's 0.554 below even the tentative floor, so these coherence scores are only marginally reliable, which is the kind of result this tool exists to surface: +Krippendorff's bands put the demo's 0.554 below even the tentative floor, so these coherence scores are only marginally reliable. That is the kind of result this tool exists to surface: | Krippendorff α | Reading | |---|---| @@ -86,7 +86,7 @@ The reliability estimators assume a complete crossed design. On a matrix with mi ### DIF -DIF, differential item functioning, asks whether the panel scores abstractive outputs differently from extractive outputs once you condition on the rest-score proxy for overall quality. The "item" under audit is the rubric criterion rather than a shared test item, since each stratum scores different outputs. The engine is ordinal logistic regression in the Zumbo tradition (single-pass, not lordif's iterative purification), run as three nested proportional-odds models, so it reports a uniform-DIF test, a nonuniform-DIF test, and an effect size (the Nagelkerke pseudo-R-squared change) classified A, B, or C by the Jodoin-Gierl thresholds. The demo shows why the card prints both a p-value and an effect size: at n = 4800 the uniform-DIF test is significant (p = 0.0005), but the effect size is 0.002, class A, which is negligible. The signal is detectable; the magnitude is not. +DIF, differential item functioning, asks whether the panel scores abstractive outputs differently from extractive outputs once you condition on the rest-score proxy for overall quality. The "item" under audit is the rubric criterion rather than a shared test item, since each stratum scores different outputs. The engine is ordinal logistic regression in the Zumbo tradition (single-pass, not lordif's iterative purification), run as three nested proportional-odds models. It reports a uniform-DIF test, a nonuniform-DIF test, and an effect size (the Nagelkerke pseudo-R-squared change) classified A, B, or C by the Jodoin-Gierl thresholds. The demo shows why the card prints both a p-value and an effect size: at n = 4800 the uniform-DIF test is significant (p = 0.0005), but the effect size is 0.002, class A, which is negligible. The signal is detectable; the magnitude is not. The matching variable is a leave-one-rater-out rest score across the three expert raters, which uses the same exchangeable-rater assumption as the reliability pillar. That rest score detects bias relative to the rater panel and understates bias the whole panel shares, so the card labels this path panel-relative DIF. For a stronger DIF analysis, pass an explicit external quality conditioner: `audit(ratings, focal=..., reference=..., conditioner=...)` accepts a sample-id to quality-score mapping. External-conditioner DIF supports instrument-level interpretation only when the conditioner is valid, independent, and appropriate for the quality construct being matched. Read DIF output as a screening audit, not a confirmatory significance claim. When a p-value lands near a decision threshold, `cluster_bootstrap_dif` runs the same engine with item-block resampling and returns a 95% cluster-robust interval alongside the unchanged point estimate. diff --git a/src/metajudge/__init__.py b/src/metajudge/__init__.py index 5c72424..1b1c6a9 100644 --- a/src/metajudge/__init__.py +++ b/src/metajudge/__init__.py @@ -1,4 +1,4 @@ -"""metajudge: audit an LLM judge/rubric instrument before you trust its numbers. +"""metajudge: A reliability and DIF report card for LLM-judge and human-rater scoring instruments. Two pillars: inter-rater reliability (Krippendorff's alpha, ICC(2,1)/(2,k)) and differential item functioning across output strata (ordinal logistic-regression DIF). From 2844baaba3cff6b3e92613b4a870b67672aeb2c1 Mon Sep 17 00:00:00 2001 From: Brittany Reese Markides <12763491+brittanyreese@users.noreply.github.com> Date: Fri, 3 Jul 2026 16:54:47 -0500 Subject: [PATCH 3/3] docs(references): rejoin the Software section into single-line paragraphs Matches the rest of the file's one-paragraph-per-line convention; the Software section had been left mid-edit with paragraphs broken across several lines. --- docs/REFERENCES.md | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md index 2f3518c..ae6912a 100644 --- a/docs/REFERENCES.md +++ b/docs/REFERENCES.md @@ -1,11 +1,6 @@ # References -The methods metajudge implements are credited to their original authors here. -Every entry is carried verbatim from the decision record where the method was -adopted; the per-method rationale and any caveats about applying a -method outside its original validation live in [decisions/](decisions/). This -file is the consolidated bibliography so the credits travel with the package -rather than sitting only inside individual ADRs. +The methods metajudge implements are credited to their original authors here. Every entry is carried verbatim from the decision record where the method was adopted; the per-method rationale and any caveats about applying a method outside its original validation live in [decisions/](decisions/). This file is the consolidated bibliography so the credits travel with the package rather than sitting only inside individual ADRs. ## Reliability: Krippendorff's alpha and ICC @@ -121,15 +116,6 @@ Fabbri, A. R., Kryscinski, W., McCann, B., Xiong, C., Socher, R., and Radev, D. ## Software and reference implementations -The runtime alpha computation wraps the `krippendorff` Python package. The -statistics are pinned in the test suite against external reference -implementations: R `MASS::polr` for the proportional-odds DIF fit, and the -`pingouin` and `statsmodels` Python packages as ICC and logistic-regression -oracles. `pingouin` and `statsmodels` are test-only oracles and are never -imported at runtime. - -AutoRubric: a Python library for rubric-based LLM-as-judge grading with -agreement metrics and bootstrap confidence intervals (https://autorubric.org, -https://github.com/delip/autorubric). Adjacent, shipped practitioner coverage -of the commodity reliability layer; cited in SPEC.md as prior art this -library does not compete with. +The runtime alpha computation wraps the `krippendorff` Python package. The statistics are pinned in the test suite against external reference implementations: R `MASS::polr` for the proportional-odds DIF fit, and the `pingouin` and `statsmodels` Python packages as ICC and logistic-regression oracles. `pingouin` and `statsmodels` are test-only oracles and are never imported at runtime. + +AutoRubric: a Python library for rubric-based LLM-as-judge grading with agreement metrics and bootstrap confidence intervals ([https://autorubric.org](https://autorubric.org), [https://github.com/delip/autorubric](https://github.com/delip/autorubric)). Adjacent, shipped practitioner coverage of the commodity reliability layer; cited in SPEC.md as prior art this library does not compete with. \ No newline at end of file