feat(default-skill-review): publish the default tessl review rubric as a plugin#14
Merged
Conversation
…ugin The default reviewer existed only baked into the skill-review recipe, with nowhere addressable to reference it from. This publishes it as a first-class plugin so the default reviewer has a real, versioned identity like any other reviewer. Content is copied verbatim from the recipe's baked-in reviewer, so the rubric is byte-equivalent to today's default. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds a new ChangesDefault review plugin bundle
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Comment |
gjvis
reviewed
Jun 26, 2026
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@default-review/.tessl-plugin/plugin.json`:
- Line 4: The published plugin description and the top-level plugin table both
only mention the two LLM judges, but the reviewer also includes the validation
pass; update the wording in the plugin metadata to explicitly include validation
as part of the scoring flow. Use the existing description field in plugin.json
and the corresponding plugin table entry so both surfaces stay aligned and
accurately reflect the full rubric.
In `@default-review/skills/skill-reviewer/references/schemas/config.schema.json`:
- Around line 18-20: The `judges` schema description in `config.schema.json`
contradicts the shipped config contract because it કહે weights must sum to 1.0
while `references/config.json` intentionally reserves 0.2 for
`validation_weight`. Update the `judges` description to reflect the real
contract used by the bundled config, and make sure the wording around
`validation_weight` and equal-weight fallback stays consistent with the schema
and `references/config.json`.
In `@default-review/skills/skill-reviewer/references/schemas/results.schema.json`:
- Line 4: The results schema description in results.schema.json uses the wrong
weight source for scoring components. Update the schema text to say each
component weight comes from references/config.json via
config.judges[stem].weight, matching how SKILL.md builds scoring.components, and
remove the misleading reference to a rubric-level weight that does not exist in
rubric.schema.json. Keep the wording aligned anywhere the same contract is
described in the results schema so consumers implement the actual contract.
In `@default-review/skills/skill-reviewer/references/schemas/rubric.schema.json`:
- Around line 18-31: The scale definition in the rubric schema allows a
zero-width range because the min and max properties can be equal, which can
break score normalization in the scoring logic. Tighten the invariant in the
schema used by rubric validation so the scale enforces min to be strictly less
than max, or add a runtime check in the scoring path that validates the scale
before normalization. Use the existing scale object in rubric.schema.json and
the normalization logic that reads scale.min and scale.max to locate the fix.
In `@default-review/skills/skill-reviewer/SKILL.md`:
- Around line 14-22: Update the skill instructions to use the bundle-relative
locations that actually exist in this plugin: change rubric discovery to read
from references/rubrics, config loading to references/config.json, and schema
loading to references/schemas/results.schema.json. Keep the rest of the workflow
intact, and ensure any references in the bundle-relative file list under
references/scripts and references/assets remain consistent with the new paths.
- Around line 22-27: The scoring setup in SKILL.md assumes config.judges always
exists, but the schema allows it to be omitted and falls back to equal weights,
so the instructions and schema are inconsistent. Update the Step 2 guidance
around scoring.components to either describe how to handle the no-judges
fallback by deriving equal weights when config.judges is missing, or remove the
fallback claim from references/schemas/config.schema.json so both sources match.
Use the identifiers config.judges, scoring.components, and the rubric stem
handling to locate the affected guidance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9a1b61d2-3875-4e05-a094-b705072d0416
📒 Files selected for processing (11)
README.mddefault-review/.tessl-plugin/plugin.jsondefault-review/README.mddefault-review/skills/skill-reviewer/SKILL.mddefault-review/skills/skill-reviewer/references/config.jsondefault-review/skills/skill-reviewer/references/rubrics/content.jsondefault-review/skills/skill-reviewer/references/rubrics/description.jsondefault-review/skills/skill-reviewer/references/schemas/config.schema.jsondefault-review/skills/skill-reviewer/references/schemas/results.schema.jsondefault-review/skills/skill-reviewer/references/schemas/rubric.schema.jsondefault-review/tessl.json
…t descriptions - Rename tessl/default-review -> tessl/default-skill-review (PR review thread). - config.schema: the judge weights plus validation_weight sum to 1.0 (not the judge weights alone), and drop the unimplemented equal-weights-if-absent claim. - results.schema: component weights come from config.json's judges map, not from a rubric-level weight field (rubrics have none). - Drop the validation pass from the plugin's README and description: the reviewer agent only runs the judges; validation is a platform scoring step applied to every review, not this plugin's concern. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
colinhallett
changed the title
…M judges" The reviewer is a single agent that reads the skill and scores it against the rubrics, not a set of LLM judges. Reword the public-facing surfaces — plugin.json, both READMEs, and the skill-reviewer frontmatter description — accordingly. The prompt body and the system's internal "judge" vocabulary (config.judges, the judges map in results) are left unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…er prompt Reword the reviewer prompt's role line to describe scoring the skill against each rubric (one evaluation per rubric) rather than "rubric-based LLM judges". The structural "judge" terms that map to the config.judges and results judges schema are left intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishes the default
tessl reviewrubric as a first-class plugin,tessl/default-skill-review.Why
The default reviewer existed only baked into the skill-review recipe in the monorepo — there was nowhere addressable the system could reference it from, so a default review recorded no reviewer identity and the dedup fingerprint folded in none (changing the default could silently serve stale scores). The backend now resolves a configured default reviewer ref through the same path as a custom reviewer. Publishing this plugin gives that ref something to resolve, so the default reviewer gains a real, versioned identity that flows into the run record and the fingerprint.
Behaviour-safe
The reviewer
SKILL.md,config.json, and rubrics are copied verbatim from the recipe's baked-in reviewer, so once the default ref resolves to this plugin, scores don't change — only the reviewer's identity and fingerprint now reflect a named, versioned reviewer. The recipe keeps its baked-in copy as the fallback when the ref can't be resolved.Naming
tessl/default-skill-review(per the review thread).Description / schema corrections
The schema
descriptiontext copied from the recipe was inaccurate; corrected here for the public contract:config.schema: the judge weights plusvalidation_weightsum to 1.0 (not the judge weights alone), and dropped an equal-weights-if-judges-absent claim the reviewer never implements.results.schema: component weights come fromconfig.json's judges map, not a rubric-levelweightfield (rubrics have none).1 − Σ judge weightsand adds the component), not this plugin's concern.The same inaccurate schema descriptions still live in the monorepo recipe and
review-plugin-creator; syncing those is a follow-up.Follow-up (after this lands)
The backend's
DEFAULT_REVIEWER_PLUGIN_REFdefault is currentlytessl/default-review; it needs updating totessl/default-skill-reviewso the default resolves in prod. Until then the default falls back to the baked-in reviewer (safe, just inert).Merging publishes
tessl/default-skill-review@0.1.0via the publish-on-merge workflow.Closes OPT-1412