Conversation
📝 WalkthroughWalkthroughAdded a new ChangesRubric derivation workflow
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
Adds a second skill to review-plugin-creator. create-review-plugin scaffolds a custom reviewer plugin but leaves the user to invent the judges and dimensions; derive-review-rubrics grounds them in evidence instead – target SKILL.md files, recurring PR review feedback, and agent logs showing where skills failed to activate or needed correction – then hands the rubric design to create-review-plugin for scaffolding and validation. Bumps the plugin to 0.2.0. Both new evals score 100% against claude:claude-sonnet-4-6. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@review-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.md`:
- Around line 7-13: The transcript fences in the example blocks are missing an
info string, causing the MD040 markdownlint warning. Update the fenced code
blocks in the agent log so they are explicitly tagged as text, using the
existing transcript/example sections around the hand-written route snippet and
the other affected block(s) referenced by the review comment.
In `@review-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.json`:
- Around line 16-34: The checklist in criteria.json is too permissive because it
only checks for minimum examples instead of full coverage. Update the rubric
criteria so every evidence pattern is required to be classified individually as
either a verifier or a rubric dimension, and every dimension must have its own
anchors/examples rather than relying on a global minimum. Use the existing
criteria entries like "Judges with evaluation_target," "Verifier classification
present," "Dimension vs verifier reasoning," and "Anchors from actual feedback"
as the places to tighten the language so the validation enforces
per-pattern/per-dimension completeness.
In
`@review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/api-scaffolder/SKILL.md`:
- Around line 8-13: The scaffold currently hard-codes a TypeScript/Express-style
layout, but the skill is intended to work across multiple frameworks. Update the
guidance in the SKILL.md steps to use framework-aware paths and naming based on
the detected stack, or explicitly scope the skill to JS/TS only; adjust the
route, types, router, and test references so they map to the chosen framework
instead of always using src/routes, src/types, src/router.ts, and
tests/*.test.ts.
In
`@review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/test-generator/SKILL.md`:
- Around line 8-13: The test-generation conventions are still hard-coded for
JS/TS, while the framework detection in the instructions already includes
Pytest. Update the guidance in the test generator spec so the output format
depends on the detected framework: keep `<module>.test.ts`/`.spec.ts` and
`describe` blocks for Jest/Vitest, but switch to pytest-style `test_*.py` files
and plain test functions/assertions for Python projects. Reference the existing
framework-detection step and the grouping rules so the file naming and structure
stay aligned with the selected test framework.
In `@review-plugin-creator/skills/derive-review-rubrics/SKILL.md`:
- Around line 25-33: The no-history fallback in the Stream B rubric logic is
skipping both signals when agent logs are absent, but the PR-backed "needed
correction" signal should still be derivable from evidence mapping. Update the
handling around the log lookup flow so only the log-dependent "skill didn't
activate" check is bypassed when `tessl agent-logs view` returns no entries,
while the "needed correction" dimension can still be grounded from PR feedback
as described in `evidence-to-rubric.md`. Keep the decision logic aligned with
the existing Stream A/Stream B split and the transcript scan path that uses
`entries`, `failures`, and the activation detection rules.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 230a9afc-f208-432b-9e75-647457837589
📒 Files selected for processing (18)
README.mdreview-plugin-creator/.tessl-plugin/plugin.jsonreview-plugin-creator/README.mdreview-plugin-creator/evals/derive-rubric-from-evidence/criteria.jsonreview-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.mdreview-plugin-creator/evals/derive-rubric-from-evidence/inputs/pr-feedback.mdreview-plugin-creator/evals/derive-rubric-from-evidence/inputs/target-skill/SKILL.mdreview-plugin-creator/evals/derive-rubric-from-evidence/scenario.jsonreview-plugin-creator/evals/derive-rubric-from-evidence/task.mdreview-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.jsonreview-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/pr-feedback.mdreview-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/api-scaffolder/SKILL.mdreview-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/code-formatter/SKILL.mdreview-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/test-generator/SKILL.mdreview-plugin-creator/evals/derive-rubric-no-agent-logs/scenario.jsonreview-plugin-creator/evals/derive-rubric-no-agent-logs/task.mdreview-plugin-creator/skills/derive-review-rubrics/SKILL.mdreview-plugin-creator/skills/derive-review-rubrics/references/evidence-to-rubric.md
marc-tessl
approved these changes
Jun 26, 2026
marc-tessl
left a comment
marc-tessl
left a comment
Contributor
There was a problem hiding this comment.
Approving. I gave this a thorough test pass beyond the bundled evals, and the new skill does exactly what the PR claims. Nice work.
What I tested
I ran derive-review-rubrics end-to-end against real evidence (live PRs on this repo plus tessl agent-logs view), not just the stubbed eval fixtures. It:
- gathered PR feedback and called the agent-logs mechanism for real, and its documented parsing of the
tessl agent-logs view --jsonoutput matches the actual tool output exactly; - degraded gracefully when no local logs were present, proceeding on PR feedback alone and inventing zero log-based activation evidence (the key guardrail held);
- kept the handoff boundary clean - design file only, with all scaffolding and
tessl review runleft tocreate-review-plugin, no duplication; - produced a working, validated plugin: real
tessl review runreturnedvalidation.overallPassed: true, ajudgesmap keyed by the custom stems, and a score of 90. A separate standalone run on a different target skill auto-activated the skill and scored 99.
All six bundled evals pass. derive-rubric-from-evidence shows strong, consistent uplift (57->100, 67->99 across two runs), which is the one carrying the real signal.
One thing to check before you merge
scenario-2 (a pre-existing create-review-plugin scenario, not touched by this PR) is flaky on its "Config weights sum to 1.0" check: 100->71 in one run, 75->73 in the next, failing that check in both baseline and with-context. So it is not a regression from this PR, but it points at noise in that scenario. Worth a quick glance before merge in case it is worth a separate fix.
Follow-up (not blocking this PR)
I hit two documentation gaps in create-review-plugin while validating the generated plugin: --json requires --workspace, and the workspace must be a real one from tessl workspace list. Tracked separately in OPT-1419 - no change needed here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second skill,
derive-review-rubrics, to thereview-plugin-creatorplugin.Why a second skill
create-review-pluginalready scaffolds a custom reviewer plugin, but the user has to invent the judges and dimensions themselves.derive-review-rubricsgrounds those rubrics in real evidence instead: it reads the target SKILL.md files, recurring PR review feedback (via/find-optimizations), and agent logs (viatessl agent-logs view) showing where skills failed to activate or needed correction, translates the recurring patterns into scoring dimensions with anchors drawn from the evidence itself, then hands the design tocreate-review-pluginfor scaffolding and validation. It does not scaffold or runtessl review run– that stayscreate-review-plugin's job, no duplication.Agent logs are an enhancement, not a dependency: the skill stays effective on PR feedback alone, losing only the log-grounded trigger-clarity anchor.
create-review-pluginis unchanged – this adds the new skill alongside it.Changes
skills/derive-review-rubrics/(SKILL.md+references/evidence-to-rubric.md).derive-rubric-from-evidence(logs present) andderive-rubric-no-agent-logs(graceful degradation on PR feedback alone). Both score 100% againstclaude:claude-sonnet-4-6.plugin.json: version0.1.0→0.2.0, skill registered, description expanded to cover all three entry points (fork the default rubric, build from scratch, or derive from evidence).Closes OPT-1342