Add Caliper (pass@k reliability harness for agent skills) to 5a

[edonadei]

What

Adds Caliper to 5a · Eval frameworks & harnesses (code-first test-runners).

What it is

Caliper is a code-first test-runner that measures the pass@k reliability of agent skills. It also ships with it's family of skills evalute-skill and grill-skill to make simpler the adoption of the framework by agents.

You describe a skill, a judge, and tasks in a git-diffable .eval.yaml spec. Caliper runs the skill k times in isolated per-attempt sandboxes and grades each attempt with an LLM-autorater (expect:) and/or deterministic Python assertions (assert:).

A --baseline flag re-runs the same tasks without the skill and reports the delta, so you can prove the skill is actually doing the work rather than the base agent passing anyway.

• Compatible harnesses: claude-code, codex, pi (You can use a different harness for eval run an judge run)
• Judging: LLM autorater over the full transcript (incl. tool-call traces) and/or local Python assertions. No detailled rubric yet, I preferred to keep it simple for now, but it will be probably needed in the future.
• Output: pass@k score + saved JSON transcripts you can diff over time
• Install: pipx install caliper-eval (Python 3.10+) or through a skill npx skills@latest add edonadei/caliper

Why it fits 5a

Like Inspect AI / promptfoo / DeepEval, it binds dataset + grading + sampling strategy into a single reusable harness. Its distinguishing angle is pass@k reliability for agent skills with a built-in baseline comparison — closest in spirit to Vercel agent-eval but Python-native with deterministic assertions and multi-agent-CLI backends.

Disclosure: I'm the author of that framework, I just think it would be worth be more visible in that list.

[edonadei]

@xdotli would that framework fit this repo? :)

[github-actions]

Thanks for the honest self-disclosure, Emrick — it makes the review clean and straightforward.

Verified: The repo exists, v0.3.0 is live on PyPI, and the code is functional — not scaffolding. The concept is also technically sound: a --baseline re-run that reports the skill-vs-no-skill delta maps directly onto the pass^k reliability pattern in PATTERNS.md — that's the right question to ask. Section 5a placement is correct.

The problem is traction. The curated bar explicitly screens for self-published low-traction tools. At 24 stars and 7 weeks old, Caliper is well below every entry currently in 5a — the lowest-star neighbor is Harbor at ~2.7k★. That gap isn't a knock on the tool; the list just earns its credibility by being selective here, especially for author submissions.

Recommendation: respectfully decline for now. The PATTERNS.md pass^k connection is a genuinely strong hook for a future annotation — worth preserving. Resubmit when you have meaningful external adoption: stars into the hundreds, issues or PRs from non-authors, or a write-up from someone who ran it on a real eval loop. At that point this fits the list well.

— awesome-evals assistant

[xdotli]

Closing, but with a genuine invitation to come back.

Nothing here is wrong: MIT, 12 releases on PyPI, 139 commits, actively pushed, and you disclosed authorship. The pass@k-for-skills idea is real and the automated review on this PR flagged the same thing I'd flag — it's a good hook for PATTERNS.md whenever it lands.

The blocker is the same one from that review on July 1: adoption. §5a's entries are frameworks with meaningful external use, and the lowest-star neighbor there is Harbor at ~2.7k★. Caliper was at 24★ then and is at 37★ now — real growth, but still a couple of orders of magnitude below the section floor, and no issues or PRs from non-authors yet.

The conditions from that review stand: stars into the hundreds, or evidence of use by people who aren't you. Reopen or file fresh when you get there and I'll take another look — I'd rather list it late than list it early and have the section stop meaning anything.

Sorry this sat a month without a human reply.

[xdotli]

Two corrections, both mine, and I'm reopening this.

1. "No issues or PRs from non-authors yet" is false. edonadei/caliper has external contributors — syf2211 (3 commits), Sehlani042, and caioribeiroclw-pixel — and issue #9 was filed by mwrshah on 2026-06-25 ("Support for Pi SDK"). That issue predates the July 1 review by six days, so the condition I restated as unmet had already been met before it was written. I repeated a stale bot verdict without checking it.

2. The star floor I cited is wrong, and wrong in an embarrassing direction. I said the lowest-star §5a neighbour was Harbor at ~2.7k★ and that Caliper was "a couple of orders of magnitude" below. Actual §5a numbers today: benchflow-ai/benchflow 305★ — our own project — then olmes 390, inspect_evals 610, agentevals 674. Harbor is 3683★, not 2.7k; the ~2.7k figure is a stale annotation in our own README that I quoted as if it were data. Caliper at 37★ is about 8× below the real floor, under one order of magnitude.

I'm not going to restate a numeric traction bar, because we don't have one written down. grep -inE "star|traction|adoption" CONTRIBUTING.md returns nothing, and any threshold that excludes Caliper at 37★ would also need applying to benchflow-ai/ClawsBench (31★), agi-inc/REAL (8★), and patronus-ai/glider (10★), all currently listed without comment. Applying an unwritten rule to your PR and not to ours isn't a defensible bar.

Reopening for a decision on the merits. The right fix on our side is to write the bar down in CONTRIBUTING and apply it uniformly, which I'm doing separately. Sorry for the runaround.

[edonadei]

Oh wow! There is still hope then :D

On my side, I'm still maintaining it. Working on integrations, skill trigger tests, and overall reliability. I'm getting some good feedback on it too.

If you have any open questions, ready to answer it.