docs: README pipeline diagram, tables, and PyPI install#12
Merged
Conversation
Add a mermaid flowchart of the Ratings -> two pillars -> ReportCard path, a Krippendorff-alpha interpretation table, and a human-vs-Gemini reliability comparison table so the key contrast reads at a glance. Split the dense How to read section under Reliability/DIF subheads. Add a PyPI version badge and switch the install commands to pip install metajudge now that 0.2.0 is published.
There was a problem hiding this comment.
Pull request overview
This PR refreshes the README to be more scannable and up-to-date now that metajudge is published on PyPI, without changing the technical voice.
Changes:
- Adds a Mermaid pipeline diagram that summarizes the Ratings → (Reliability | DIF) → ReportCard flow.
- Reworks the “How to read” section with Reliability/DIF subheads and adds two tables to make key interpretations/numbers easier to scan.
- Updates install instructions and adds a PyPI badge, switching primary install from GitHub
maintopip install metajudge.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ```mermaid | ||
| flowchart LR | ||
| R["Ratings<br/>rater × item × stratum<br/>ordinal scores"] | ||
| R --> REL["Reliability<br/>Krippendorff α<br/>ICC(2,1) / (2,k)"] |
| To audit a real instrument, point metajudge at the output of an existing judge runner. `Ratings.from_eval_instruments` maps the per-judge score frames produced by Epic's [`evaluation-instruments`](https://github.com/epic-open-source/evaluation-instruments) (`frame_from_evals`) into the `Ratings` the audit consumes: rater is judge, item is sample, score is one rubric criterion. It is a local DataFrame transform that adds no dependency. A runnable, no-PHI walkthrough is in [docs/interop-epic.md](https://github.com/brittanyreese/metajudge/blob/main/docs/interop-epic.md). | ||
|
|
||
| For a self-contained, end-to-end example that builds the judge panel itself, [`examples/audit_llm_judge.py`](https://github.com/brittanyreese/metajudge/blob/main/examples/audit_llm_judge.py) runs three LLM judges over 16 stratified summaries and prints the report card. Install with `pip install "metajudge[examples] @ git+https://github.com/brittanyreese/metajudge@main"`. Then pick a mode: `--mode live --provider gemini` calls Gemini models on a billed project (`GOOGLE_AI_API_KEY`); `--mode live --provider openrouter` calls free-tier OpenRouter models (`OPENROUTER_API_KEY`, capacity not guaranteed); `--mode offline` runs a seeded simulation with no key or network. | ||
| For a self-contained, end-to-end example that builds the judge panel itself, [`examples/audit_llm_judge.py`](https://github.com/brittanyreese/metajudge/blob/main/examples/audit_llm_judge.py) runs three LLM judges over 16 stratified summaries and prints the report card. Install with `pip install "metajudge[examples]"`. Then pick a mode: `--mode live --provider gemini` calls Gemini models on a billed project (`GOOGLE_AI_API_KEY`); `--mode live --provider openrouter` calls free-tier OpenRouter models (`OPENROUTER_API_KEY`, capacity not guaranteed); `--mode offline` runs a seeded simulation with no key or network. |
brittanyreese
added a commit
that referenced
this pull request
Jul 3, 2026
Add a mermaid flowchart of the Ratings -> two pillars -> ReportCard path, a Krippendorff-alpha interpretation table, and a human-vs-Gemini reliability comparison table so the key contrast reads at a glance. Split the dense How to read section under Reliability/DIF subheads. Add a PyPI version badge and switch the install commands to pip install metajudge now that 0.2.0 is published.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the README more scannable without changing the technical content:
pip install metajudgenow that 0.2.0 is published.Do not merge until the v0.2.0 publish job confirms PyPI, since the install commands now assume it.