From 5b9a8ea6a9fc78748020c3737a7bdd8df17bdeb4 Mon Sep 17 00:00:00 2001 From: "Abheer Kolhatkar (Tessl)" Date: Fri, 26 Jun 2026 14:05:25 +0100 Subject: [PATCH] feat(review-plugin-creator): add derive-review-rubrics skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a second skill to review-plugin-creator. create-review-plugin scaffolds a custom reviewer plugin but leaves the user to invent the judges and dimensions; derive-review-rubrics grounds them in evidence instead – target SKILL.md files, recurring PR review feedback, and agent logs showing where skills failed to activate or needed correction – then hands the rubric design to create-review-plugin for scaffolding and validation. Bumps the plugin to 0.2.0. Both new evals score 100% against claude:claude-sonnet-4-6. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- .../.tessl-plugin/plugin.json | 7 +- review-plugin-creator/README.md | 8 +- .../derive-rubric-from-evidence/criteria.json | 61 ++++++++++++++++ .../inputs/agent-log.md | 31 ++++++++ .../inputs/pr-feedback.md | 32 ++++++++ .../inputs/target-skill/SKILL.md | 13 ++++ .../derive-rubric-from-evidence/scenario.json | 5 ++ .../evals/derive-rubric-from-evidence/task.md | 26 +++++++ .../derive-rubric-no-agent-logs/criteria.json | 56 ++++++++++++++ .../inputs/pr-feedback.md | 73 +++++++++++++++++++ .../sample-skills/api-scaffolder/SKILL.md | 20 +++++ .../sample-skills/code-formatter/SKILL.md | 17 +++++ .../sample-skills/test-generator/SKILL.md | 18 +++++ .../derive-rubric-no-agent-logs/scenario.json | 5 ++ .../evals/derive-rubric-no-agent-logs/task.md | 25 +++++++ .../skills/derive-review-rubrics/SKILL.md | 47 ++++++++++++ .../references/evidence-to-rubric.md | 29 ++++++++ 18 files changed, 468 insertions(+), 7 deletions(-) create mode 100644 review-plugin-creator/evals/derive-rubric-from-evidence/criteria.json create mode 100644 review-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.md create mode 100644 review-plugin-creator/evals/derive-rubric-from-evidence/inputs/pr-feedback.md create mode 100644 review-plugin-creator/evals/derive-rubric-from-evidence/inputs/target-skill/SKILL.md create mode 100644 review-plugin-creator/evals/derive-rubric-from-evidence/scenario.json create mode 100644 review-plugin-creator/evals/derive-rubric-from-evidence/task.md create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.json create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/pr-feedback.md create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/api-scaffolder/SKILL.md create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/code-formatter/SKILL.md create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/test-generator/SKILL.md create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/scenario.json create mode 100644 review-plugin-creator/evals/derive-rubric-no-agent-logs/task.md create mode 100644 review-plugin-creator/skills/derive-review-rubrics/SKILL.md create mode 100644 review-plugin-creator/skills/derive-review-rubrics/references/evidence-to-rubric.md diff --git a/README.md b/README.md index af67d66..9869d4d 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ These are R&D-maintained plugins intended for production use. | Plugin | Description | Badge | | --- | --- | --- | | [`skill-optimizer`](skill-optimizer/) | Optimize your skills and plugins: review SKILL.md quality, generate eval scenarios, run evals, compare across models, diagnose gaps, and re-run until scores improve | [![](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Fskill-optimizer%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/skill-optimizer) [![](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Fskill-optimizer)](https://tessl.io/registry/tessl/skill-optimizer/evals) | -| [`review-plugin-creator`](review-plugin-creator/) | Create a custom `tessl review` rubric, by forking the default rubric or building one from scratch | [![review-plugin-creator version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Freview-plugin-creator%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/review-plugin-creator) [![review-plugin-creator evals](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Freview-plugin-creator)](https://tessl.io/registry/tessl/review-plugin-creator/evals) | +| [`review-plugin-creator`](review-plugin-creator/) | Create a custom `tessl review` rubric – fork the default rubric, build one from scratch, or derive it from evidence (existing skills, PR feedback, agent logs) | [![review-plugin-creator version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Freview-plugin-creator%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/review-plugin-creator) [![review-plugin-creator evals](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Freview-plugin-creator)](https://tessl.io/registry/tessl/review-plugin-creator/evals) | | [`api`](api/) | Find and call Tessl API endpoints token-efficiently, without loading the whole `openapi.json` spec into context | [![api version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Fapi%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/api) [![api evals](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Fapi)](https://tessl.io/registry/tessl/api/evals) | ## Install diff --git a/review-plugin-creator/.tessl-plugin/plugin.json b/review-plugin-creator/.tessl-plugin/plugin.json index 1ef9ac5..df57cfe 100644 --- a/review-plugin-creator/.tessl-plugin/plugin.json +++ b/review-plugin-creator/.tessl-plugin/plugin.json @@ -1,9 +1,10 @@ { "name": "tessl/review-plugin-creator", - "version": "0.1.0", - "description": "Guided workflow for creating a custom Tessl reviewer plugin, by forking the default rubric or building one from scratch. Scaffolds the plugin directory structure, authors rubrics and config.json, and validates the result with tessl review run.", + "version": "0.2.0", + "description": "Create custom Tessl reviewer plugins – fork the default rubric, build one from scratch, or derive its rubrics from evidence (existing skills, PR review feedback, agent logs). Scaffolds the plugin directory structure, authors rubrics and config.json, and validates the result with tessl review run.", "private": false, "skills": [ - "skills/create-review-plugin" + "skills/create-review-plugin", + "skills/derive-review-rubrics" ] } diff --git a/review-plugin-creator/README.md b/review-plugin-creator/README.md index 774ad33..ce7d15f 100644 --- a/review-plugin-creator/README.md +++ b/review-plugin-creator/README.md @@ -12,10 +12,11 @@ tessl install tessl/review-plugin-creator `tessl review` scores a skill against a reviewer plugin. With no `--review-plugin`, it uses Tessl's default rubric (Anthropic best practices). This plugin walks you through creating your own reviewer plugin so reviews reflect your team's standard, then gating CI on that score. -Two starting points: +Three starting points: -- **Fork the default rubric** — start from Tessl's default rubric (bundled with the skill) and tweak weights, anchors, or dimensions. -- **Build from scratch** — author new judges from a blank template for a security-only or domain-specific reviewer. +- **Fork the default rubric** – start from Tessl's default rubric (bundled with the skill) and tweak weights, anchors, or dimensions. +- **Build from scratch** – author new judges from a blank template for a security-only or domain-specific reviewer. +- **Derive from evidence** – ground the rubric in how your agents actually behave: existing skills, recurring PR review feedback, and agent logs showing where skills failed to activate or needed correction. The default rubric is bundled at `skills/create-review-plugin/references/default-rubric/` so you can read exactly what `tessl review` uses out of the box before deciding which path to take. @@ -24,3 +25,4 @@ The default rubric is bundled at `skills/create-review-plugin/references/default | Skill | Description | |-------|-------------| | `create-review-plugin` | Scaffolds the plugin, writes rubric files and config.json, and validates with `tessl review run` | +| `derive-review-rubrics` | Turns evidence (existing skills, PR feedback, agent logs) into rubric dimensions and anchors, then hands the design to `create-review-plugin` | diff --git a/review-plugin-creator/evals/derive-rubric-from-evidence/criteria.json b/review-plugin-creator/evals/derive-rubric-from-evidence/criteria.json new file mode 100644 index 0000000..df5bdb3 --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-from-evidence/criteria.json @@ -0,0 +1,61 @@ +{ + "context": "Tests whether the agent, following derive-review-rubrics, correctly maps a fixed set of evidence to rubric design content. The evidence (in ./inputs/) contains: a target SKILL.md with a vague description and a vague type-sync step; PR feedback with a recurring skill-type api:sync finding, a verifier-type finding about the typed reply helper, and a discoverability complaint; and an agent log showing one session where the skill failed to activate and one where it activated but a vague step was skipped and the user corrected it. The agent must produce a design.md that translates each evidence pattern into the right rubric element (or routes it out of the rubric), with anchors grounded in the actual evidence. The strongest signals are: a trigger-clarity dimension on `description` from the didn't-activate session, an enforceability dimension on `content` (with scope + scoring_notes) from the ignored-step correction, a dimension for the recurring api:sync feedback, and the verifier-type finding routed OUT to a verifier rather than made a rubric dimension.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Produces a design file only", + "max_score": 8, + "description": "A `./design.md` file exists describing the rubric design. The agent did NOT scaffold a plugin directory, copy schemas, run `tessl review run`, or modify any file under `./inputs/`." + }, + { + "name": "Trigger-clarity dimension on description", + "max_score": 12, + "description": "The design includes a dimension that judges trigger/description clarity, and its judge's `evaluation_target` is `description`. This is the correct mapping for the Session 1 evidence where the skill failed to activate." + }, + { + "name": "Trigger-clarity anchored in the real failed description", + "max_score": 8, + "description": "The trigger-clarity dimension's low-score (e.g. score-1) example is the actual description that failed to fire — `\"Add an endpoint to the backend.\"` — and its high-score example is a description that names the situation it triggers on. The example is drawn from the evidence, not invented." + }, + { + "name": "Enforceability dimension on content", + "max_score": 12, + "description": "The design includes an enforceability/actionability dimension whose judge's `evaluation_target` is `content`, mapping the Session 2 evidence (skill activated but the vague step was skipped). Its bad example is the vague step (`\"Make sure the types are in sync.\"`) and its good example makes the step concrete (e.g. `\"Run `bun run api:sync` and commit the regenerated files.\"`)." + }, + { + "name": "Content judge carries scope and scoring_notes", + "max_score": 10, + "description": "Any judge with `evaluation_target: \"content\"` in the design also specifies a `scope` (one-line description of what it evaluates) and `scoring_notes`. A judge on `description` is not required to have them." + }, + { + "name": "Dimension for the recurring api:sync feedback", + "max_score": 10, + "description": "The design includes a dimension derived from the recurring skill-type PR finding (Finding 1): does the skill instruct the agent to run `bun run api:sync` / regenerate API types after a route change." + }, + { + "name": "api:sync anchors drawn from the PR evidence", + "max_score": 8, + "description": "The api:sync dimension's examples are grounded in the actual review comments from PRs #4821 / #4855 / #4902, not invented." + }, + { + "name": "Verifier-type finding routed out of the rubric", + "max_score": 12, + "description": "The verifier-type finding (Finding 2 — every handler returns via the typed `reply.send` helper, never raw `res.end`) is kept as a verifier (a binary pass/fail check), NOT turned into a rubric dimension. The design explicitly states this routing decision and the reason (binary invariant, not a degree of quality)." + }, + { + "name": "Dimension weights sum to 1.0 per rubric", + "max_score": 8, + "description": "Within each rubric the dimension weights sum to 1.0, and the design states this." + }, + { + "name": "Plugin-level weight split sums to 1.0", + "max_score": 6, + "description": "The plugin-level split — `validation_weight` plus each judge's weight — sums to 1.0 in the design." + }, + { + "name": "Weights reflect frequency and severity", + "max_score": 6, + "description": "The design justifies its weights by how often / how severely each pattern recurred (e.g. the api:sync pattern, recurring across three PRs and causing red CI, is weighted higher than a one-off stylistic nit)." + } + ] +} diff --git a/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.md b/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.md new file mode 100644 index 0000000..e748ce7 --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.md @@ -0,0 +1,31 @@ +# Agent-log excerpt — `tessl agent-logs view` output (provider: claude-code) + +Two sessions are reproduced below. Transcripts are trimmed to the relevant turns. + +## Session 1 — skill did NOT activate when it should have + +``` +user: Add a GET /workspaces/:id/members endpoint to the backend that returns the + members of a workspace. +assistant: I'll create the route handler and schema directly. + [no Skill tool_use block; no /add-api-endpoint slash command anywhere in the session] + ... hand-writes src/routes/workspaces/members/get.ts ... +``` + +The task squarely matches `add-api-endpoint`'s purpose, but neither activation form +(a `Skill` tool_use nor a `/add-api-endpoint` slash command) appears. The agent +re-derived the route boilerplate from scratch. + +## Session 2 — skill activated but a step was ignored, and the user corrected the agent + +``` +user: /add-api-endpoint for the billing webhook +assistant: [tool_use: Skill, input.skill = "add-api-endpoint"] + ... creates the route handler and the request/response schema ... + ... does NOT regenerate API types ... +assistant: Done — the billing webhook endpoint is in place. +user: You forgot to sync the types. CI is red on api-types-check. Run `bun run api:sync`. +``` + +The skill fired, but step 3 ("Make sure the types are in sync") was vague enough that the +agent skipped the concrete action, and the user had to redirect it. diff --git a/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/pr-feedback.md b/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/pr-feedback.md new file mode 100644 index 0000000..daf0922 --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/pr-feedback.md @@ -0,0 +1,32 @@ +# find-optimizations output — PR window: last 4 weeks, scoped to skill PRs + +Findings below were produced by `/find-optimizations` over the skill's PR history. +Each finding carries a **Type**, **Summary**, and **Evidence**. + +## Finding 1 + +- **Type**: skill +- **Summary**: The `add-api-endpoint` skill never tells the agent to regenerate the API + client types after changing a route. Reviewers repeatedly have to ask for + `bun run api:sync`, and CI fails on type drift until they do. +- **Evidence**: + - PR #4821 — reviewer: "CI is red on api-types-check. Run `bun run api:sync` and commit the regenerated files." + - PR #4855 — reviewer: "Same as last time — you changed a route schema but didn't run `bun run api:sync`." + - PR #4902 — reviewer: "Need `bun run api:sync` here too; the frontend client is out of date." + - Three PRs, same recurring comment from the same reviewer. + +## Finding 2 + +- **Type**: verifier +- **Summary**: Every route handler must return through the typed `reply.send(...)` helper, + never via raw `res.end(...)`. This is a binary, observable property of the committed file. +- **Evidence**: + - PR #4877 — reviewer: "Use `reply.send` here, not `res.end` — we lose response typing otherwise." + +## Finding 3 + +- **Type**: skill +- **Summary**: Reviewers and teammates note the skill is hard to discover — its description + doesn't say what kind of work triggers it, so agents hand-roll routes instead of using it. +- **Evidence**: + - PR #4902 — reviewer: "Was there not a skill for this? The route boilerplate is all slightly wrong." diff --git a/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/target-skill/SKILL.md b/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/target-skill/SKILL.md new file mode 100644 index 0000000..b3c5000 --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-from-evidence/inputs/target-skill/SKILL.md @@ -0,0 +1,13 @@ +--- +name: add-api-endpoint +description: Add an endpoint to the backend. +--- + +Add a new API endpoint to the Fastify backend. + +## Steps + +1. Create the route handler under `src/routes/`. +2. Add the request and response schema. +3. Make sure the types are in sync. +4. Test it. diff --git a/review-plugin-creator/evals/derive-rubric-from-evidence/scenario.json b/review-plugin-creator/evals/derive-rubric-from-evidence/scenario.json new file mode 100644 index 0000000..4b061b0 --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-from-evidence/scenario.json @@ -0,0 +1,5 @@ +{ + "type": "generic", + "description": "Derive a review-plugin rubric from gathered evidence (agent logs present)", + "include": ["./inputs"] +} diff --git a/review-plugin-creator/evals/derive-rubric-from-evidence/task.md b/review-plugin-creator/evals/derive-rubric-from-evidence/task.md new file mode 100644 index 0000000..fac73ca --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-from-evidence/task.md @@ -0,0 +1,26 @@ +# Derive a review-plugin rubric from gathered evidence + +You are deriving a custom `tessl review` plugin rubric for a single skill, grounded in +evidence about how agents actually behaved with it. The evidence has already been gathered +for you (the gathering tools that would normally collect it — `/find-optimizations` and +`tessl agent-logs view` — are not available in this environment), so work from the files +provided rather than calling those tools. + +## Inputs (in `./inputs/`) + +- `inputs/target-skill/SKILL.md` — the skill the produced plugin will score. This is the baseline. +- `inputs/pr-feedback.md` — `/find-optimizations` output: recurring review findings, each with a + Type, Summary, and Evidence. +- `inputs/agent-log.md` — an agent-log excerpt with two sessions. + +## What to produce + +Following the `derive-review-rubrics` workflow, map the evidence to rubric content and write +a single rubric **design file** to `./design.md` in your working directory. Do not scaffold a +plugin, copy schemas, or run `tessl review run` — that is `create-review-plugin`'s job, which +happens later. Do not modify any file under `./inputs/`. + +The design file should describe the judges and their `evaluation_target`s, the scoring +dimensions with anchors and examples drawn from the evidence, the `scope` and `scoring_notes` +for any `content` judge, and the weight split. It should also record any evidence pattern that +does NOT belong in the rubric and where it should go instead. diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.json b/review-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.json new file mode 100644 index 0000000..5359b4b --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.json @@ -0,0 +1,56 @@ +{ + "context": "Tests whether the agent correctly applies the derive-review-rubrics skill in a cloud sandbox scenario: proceeding on Stream A alone when no agent logs are available, not fabricating log-based activation evidence (while still deriving a description-clarity dimension from the recurring vague-description PR feedback), classifying evidence patterns as dimensions vs verifiers, grounding anchor examples in the provided PR feedback, and weighting dimensions by frequency/severity. The agent must produce a rubric-design.md suitable for handoff to /create-review-plugin without scaffolding any files.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Acknowledges no log evidence", + "description": "rubric-design.md explicitly states that agent-log evidence is unavailable (e.g. cloud sandbox) and that the design proceeds on Stream A alone", + "max_score": 10 + }, + { + "name": "No fabricated log-based activation evidence", + "description": "With no agent logs available, the design must not invent or claim log-derived activation evidence — e.g. a dimension anchored in 'the skill failed to fire in N transcripts', or a score example that purports to come from observed non-activation. Deriving a description/trigger-clarity dimension from the recurring PR feedback that descriptions are too vague to tell when a skill fires (Theme 1) is correct and expected — that is valid Stream-A evidence. Full marks: no invented log evidence, and any trigger-clarity dimension is anchored in the actual PR review comments rather than in non-existent logs.", + "max_score": 12 + }, + { + "name": "Judges with evaluation_target", + "description": "At least one proposed judge includes an explicit evaluation_target field (e.g. 'description', 'content', 'structure')", + "max_score": 8 + }, + { + "name": "Verifier classification present", + "description": "At least one evidence pattern from pr-feedback.md is classified as a verifier or lint check (binary pass/fail) rather than a rubric dimension, with a stated reason", + "max_score": 10 + }, + { + "name": "Dimension vs verifier reasoning", + "description": "The document explains WHY each classification was made — i.e. distinguishes judgmental qualities from binary pass/fail invariants", + "max_score": 8 + }, + { + "name": "Anchors from actual feedback", + "description": "At least two rubric score anchors or examples are drawn verbatim or near-verbatim from the text in pr-feedback.md (not invented examples)", + "max_score": 12 + }, + { + "name": "Frequency/severity drives weights", + "description": "The document states that dimension weights reflect the frequency and severity data from pr-feedback.md (e.g. Theme 1 at 18/40 PRs and High severity receiving more weight than Theme 6 at 5/40 PRs and Low severity)", + "max_score": 10 + }, + { + "name": "Weights sum to 1.0 per judge", + "description": "Within each proposed judge, the listed dimension weights sum to 1.0 (or the document clearly states they should)", + "max_score": 8 + }, + { + "name": "Handoff note to create-review-plugin", + "description": "The document includes a section or note explaining what /create-review-plugin would do next — without actually scaffolding directories, copying schemas, or running tessl commands", + "max_score": 10 + }, + { + "name": "No scaffolding actions taken", + "description": "The agent did NOT create any plugin directories, schema files, rubric JSON files, or run any tessl CLI commands — only rubric-design.md was produced", + "max_score": 12 + } + ] +} diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/pr-feedback.md b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/pr-feedback.md new file mode 100644 index 0000000..545a88e --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/pr-feedback.md @@ -0,0 +1,73 @@ +# Recurring PR Review Feedback — Q2 2026 + +This document collects recurring review comments gathered across ~40 skill-related PRs from January to June 2026. Comments are grouped by theme. + +--- + +## Theme 1: Vague "When to use" descriptions (appeared in 18 of 40 PRs) + +- "The trigger section just says 'when the user asks' — that's too broad. What distinguishes this skill from a general assistant reply?" +- "Reviewer couldn't tell from the description alone when this skill fires vs when the agent handles it inline." +- "Trigger context is missing entirely. What keywords or request patterns are expected?" +- "This is the third PR this sprint where the skill description gave the reviewer no basis to judge fit." + +**Severity**: High — caused wrong-skill invocations in staging; one incident resulted in the api-scaffolder being triggered by a formatting request. + +--- + +## Theme 2: Steps too vague to follow reliably (appeared in 14 of 40 PRs) + +- "Step 3 says 'apply the changes' — apply how? Which CLI? Which flags?" +- "The steps skip from reading the file to producing the output with nothing in between." +- "No mention of what to do when the expected config file is absent. Agent will halt or guess." +- "Steps 1–3 are fine; Steps 4–5 are placeholders. A reviewer can't certify this is implementable." + +**Severity**: Medium — skills with vague steps produced inconsistent agent behavior across runs. + +--- + +## Theme 3: Missing example invocations (appeared in 11 of 40 PRs) + +- "There are no examples anywhere in this skill. How does the reviewer know what 'correct' looks like?" +- "Skill has good intent but without a worked example, the LLM interpolates wildly." +- "Every other skill in this repo has at least one before/after example. This one has none." + +**Severity**: Medium — correlated with higher review score variance; graders disagree more when examples are absent. + +--- + +## Theme 4: No cleanup or size-limit guidance for generated output (appeared in 9 of 40 PRs) + +- "Skill generates a report but doesn't say anything about file size or cleanup. We've had jobs time out because 2 GB diffs were left on disk." +- "What if the formatter produces a huge unified diff? The task description says nothing about this." + +**Severity**: Medium — operational risk; jobs fail when large files cause pipeline timeouts. + +--- + +## Theme 5: Scope creep — skills doing more than stated (appeared in 7 of 40 PRs) + +- "The skill says it will format files. The agent also rewrote two test files unprompted." +- "README says 'scaffold route only' but the skill's steps also modify the main router and write tests. That's three things, not one." +- "Boundaries are not enforced. The agent interpreted 'update README' as permission to reorganize the whole docs/ folder." + +**Severity**: Low-Medium — caused reviewer confusion; one incident required a rollback. + +--- + +## Theme 6: Missing frontmatter / malformed headings (appeared in 5 of 40 PRs) + +- "No `# Title` at line 1." +- "H2 headings are inconsistent — some use '##', one uses '###'." +- "Title present but blank." + +**Severity**: Low — purely structural; lint-catchable. + +--- + +## Theme 7: No error-handling guidance (appeared in 6 of 40 PRs) + +- "What should the agent do if the formatter exits non-zero? Skill is silent on this." +- "Missing: what happens when the target file doesn't exist? The skill assumes a happy path only." + +**Severity**: Low-Medium — led to silent failures in CI. diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/api-scaffolder/SKILL.md b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/api-scaffolder/SKILL.md new file mode 100644 index 0000000..516cd8b --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/api-scaffolder/SKILL.md @@ -0,0 +1,20 @@ +# API Scaffolder Skill + +## When to use this skill +Use this skill when a user asks to create, scaffold, or bootstrap a new REST API endpoint or service. + +## Steps + +1. Ask (or infer from context) the resource name, HTTP methods needed, and whether auth is required. +2. Create the route handler file at `src/routes/.ts`. +3. Define request and response types in `src/types/.ts`. +4. Add the route to the main router in `src/router.ts`. +5. Write a brief integration test at `tests/.test.ts` that covers at least GET and POST. +6. Update `README.md` with a one-line entry in the "Endpoints" table. + +## Important notes +- Use the project's existing HTTP framework (Express, Fastify, Flask, Gin — detect from dependencies). +- Always validate incoming request bodies using the project's existing validation library. +- Return standard HTTP status codes: 200 for success, 201 for created, 400 for bad input, 404 for not found, 500 for unexpected errors. +- Never store credentials or secrets in route handlers — read from environment variables. +- If the resource name conflicts with an existing route, warn the user before proceeding. diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/code-formatter/SKILL.md b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/code-formatter/SKILL.md new file mode 100644 index 0000000..d96b55c --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/code-formatter/SKILL.md @@ -0,0 +1,17 @@ +# Code Formatter Skill + +## When to use this skill +Use this skill whenever a user asks to format, lint, or clean up source code files. + +## Steps + +1. Identify the language of the code files provided. +2. Apply the project's formatting rules (from `.editorconfig` or `.prettierrc` if present). +3. Run the formatter with `--write` to apply changes in-place. +4. Report which files were changed and how many lines were modified. + +## Important notes +- Always preserve semantic meaning — never alter logic while formatting. +- For TypeScript and JavaScript, use Prettier. For Python, use black. For Go, use gofmt. +- If no config file is present, use 2-space indentation for JS/TS, 4-space for Python. +- Do not format generated files (e.g., files in `dist/`, `build/`, or `__generated__/`). diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/test-generator/SKILL.md b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/test-generator/SKILL.md new file mode 100644 index 0000000..c537f5f --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/inputs/sample-skills/test-generator/SKILL.md @@ -0,0 +1,18 @@ +# Test Generator Skill + +## When to use this skill +Use this skill when a user asks to write, scaffold, or add test coverage to a module or function. + +## Steps + +1. Read the target module and identify public functions and exported classes. +2. For each function, generate test cases covering: happy path, edge cases (empty input, null, boundary values), and expected error conditions. +3. Group tests by function in a single test file named `.test.ts` (or `.spec.ts` per project convention). +4. Use the project's existing test framework (Jest, Vitest, Pytest — detect from `package.json` or `pyproject.toml`). +5. Add a brief comment above each `describe` block explaining what scenario it covers. + +## Important notes +- Do not add tests for private/internal helpers unless explicitly asked. +- Mock external dependencies (HTTP, file system, databases) using the framework's standard mock library. +- Each test should have exactly one assertion or a clearly documented reason for multiple assertions. +- If a module has no public exports, explain this and do nothing. diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/scenario.json b/review-plugin-creator/evals/derive-rubric-no-agent-logs/scenario.json new file mode 100644 index 0000000..73465ff --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/scenario.json @@ -0,0 +1,5 @@ +{ + "type": "generic", + "description": "Derive rubrics from evidence, cloud sandbox (no agent logs)", + "include": ["./inputs"] +} diff --git a/review-plugin-creator/evals/derive-rubric-no-agent-logs/task.md b/review-plugin-creator/evals/derive-rubric-no-agent-logs/task.md new file mode 100644 index 0000000..6df26c9 --- /dev/null +++ b/review-plugin-creator/evals/derive-rubric-no-agent-logs/task.md @@ -0,0 +1,25 @@ +# Rubric Design from PR Feedback (No Agent Logs Available) + +## Problem/Feature Description + +Your team maintains a set of skills used by AI agents in a cloud-based CI pipeline. Over the past quarter, the team has accumulated a significant backlog of PR review comments about skill quality — and your tech lead wants to introduce automated skill review to catch recurring issues before they reach code review. + +You have been asked to produce a rubric design document that can be handed off to the plugin scaffolder. The design should describe which judges to create, what each judge evaluates, the rubric dimensions and their relative weights, and example anchors drawn from real review comments. + +You are working in a cloud sandbox environment where `tessl agent-logs` is unavailable and the `tessl` CLI cannot be run. You will need to work from the provided skills and PR feedback alone. + +Three sample skills are available in `inputs/sample-skills/` — one for code formatting, one for test generation, and one for API scaffolding. A collection of recurring PR review comments is available in `inputs/pr-feedback.md`. + +## Output Specification + +Produce a single file `rubric-design.md` in your working directory. The design document should include: + +- A statement acknowledging that agent-log evidence is unavailable and describing what that means for the design +- A list of proposed judges, each with: + - A name + - An `evaluation_target` (what type of artifact it judges) + - A list of rubric dimensions with proposed weights (weights within each judge should sum to 1.0) + - For each dimension: a question the judge will answer, and example anchors drawn from the actual PR feedback provided +- For each evidence pattern from the PR feedback, a classification of what kind of quality check it represents — e.g. a subjective judgment call vs. a binary pass/fail check — with a brief reason +- A note on how weights were assigned, referencing the frequency and severity information in `inputs/pr-feedback.md` +- A final section explaining what `/create-review-plugin` would need to do next with this design document (but do NOT scaffold any directories, copy any schemas, or run any CLI commands) diff --git a/review-plugin-creator/skills/derive-review-rubrics/SKILL.md b/review-plugin-creator/skills/derive-review-rubrics/SKILL.md new file mode 100644 index 0000000..6ac92de --- /dev/null +++ b/review-plugin-creator/skills/derive-review-rubrics/SKILL.md @@ -0,0 +1,47 @@ +--- +name: derive-review-rubrics +description: Derive custom review-plugin judges, rubrics, and scoring criteria from evidence — existing skills, PR review feedback, and accumulated agent logs — by finding where agents needed correction or skills failed to activate, then translating those patterns into scoring dimensions and anchors. Hands the rubric design to create-review-plugin for scaffolding. Use when the user wants a custom skill reviewer or scoring rubric grounded in how their agents actually behave — to grade skills consistently, or build review criteria from past PR feedback and agent logs. +--- + +This skill grounds a custom reviewer plugin in real evidence. It gathers two evidence streams, translates recurring patterns into rubric dimensions with anchors drawn from the evidence itself, then hands the design to `create-review-plugin` for scaffolding and validation. It does not scaffold, copy schemas, or run `tessl review run` — that is `create-review-plugin`'s job. + +## Input + +The user provides the scope to ground the rubric in: + +- The skills to review (a directory of SKILL.md files, or "the installed skills"). These are the artefacts the produced plugin will score. +- A PR window for review feedback (a PR number or a time period like "the last 2 weeks"). +- Optionally a log window and provider filter. + +## Step 1 — Gather evidence + +### Stream A — skills and PR feedback + +1. Read each target SKILL.md for its intended behaviour (the `description` trigger and the workflow body). This is the baseline the rubric scores against. +2. Invoke `/find-optimizations` over the PR window, scoped to skill / SKILL.md PRs. Consume its findings — each has a **Type**, **Summary**, and **Evidence** (PR numbers and the specific review comments). The types come from find-optimizations' configurable list in `.tessl/memory/improvement-types.md` (default: `skill`, `rule`, `hook`, `test`, `refactor`, `verifier`); read it for the names in effect. Recurring `skill`-type feedback is the strongest rubric signal. + +### Stream B — agent logs (optional) + +Run `tessl agent-logs view --json --since ` (add `--provider ` to narrow; it covers `claude-code`, `cursor-ide`, `cursor-agent`, `tessl-agent`). It reads local session history only. Empty `entries` means no sessions were found — no local history, as in a cloud sandbox or when collection was never enabled. Then this stream has no evidence: say so, proceed on Stream A alone, and skip the two signals below. Without logs you can't ground a trigger-clarity dimension in a real failed-to-fire transcript — but recurring PR feedback that a description is too vague to tell when the skill fires still grounds that dimension from Stream A. What you must not do is invent a log-based activation example you don't have. Entries present but with empty `content` is different — it only means nothing happened after ``; widen the window rather than treating it as no history. + +When entries are present, the output is `{ entries, failures }`; each entry has `provider`, `sessionId`, and a `content` transcript. For `claude-code`, `content` is the session's raw JSONL event lines. A skill activation shows up in one of two forms: an assistant `tool_use` block named `Skill` (the skill is in `input.skill`), or a user-message slash command — text that starts with `/` (sometimes wrapped in `` tags), because Claude Code expands skill slash commands inline rather than as a tool call. Count both as activations. Other providers render the transcript as text; look for the equivalent invocation. + +Scan each transcript for two signals: + +- **Skill didn't activate** — a task that matched a skill's purpose where neither activation form for it appears. Count this only when the skill genuinely should have fired; a task its own DO-NOT-TRIGGER guidance excludes is a correct non-activation, not evidence. +- **Needed correction** — repeated tool errors and retries, or the user redirecting the agent after it acted on a skill. + +## Step 2 — Map evidence to rubric content + +Cluster and dedupe themes across both streams. For each theme, follow [references/evidence-to-rubric.md](references/evidence-to-rubric.md): + +- Classify it: a judgmental quality attribute becomes a `dimensions[]` entry; a binary pass/fail invariant routes out of the rubric — to a verifier (an LLM-judge rule) when no deterministic lint or test can express it, otherwise to a lint/test. +- Group dimensions into judges by what each evaluates. `description` and `content` are the built-in `evaluation_target`s; an evidence-derived domain rubric may define its own (any string). A judge whose `evaluation_target` is `content` also requires a `scope` and `scoring_notes` — see [references/evidence-to-rubric.md](references/evidence-to-rubric.md). +- Set dimension weights (sum to 1.0 per rubric) and the judge + `validation_weight` split (sum to 1.0) by how often and how severely the pattern recurred. +- Populate each `scores[].example` with the **actual evidence** — the description that failed to fire, the flagged review comment, the corrected diff. Grounded anchors are what this skill adds over the generic template. + +## Step 3 — Hand off to create-review-plugin + +Write the assembled design (judges, their `evaluation_target`, dimensions with grounded anchors, the `scope` and `scoring_notes` of any `content` judge, and the weight split) to a workspace file, then invoke `/create-review-plugin` and point it at that file. `create-review-plugin` scaffolds the directory, copies the schemas, writes the rubric and config files, and validates with `tessl review run`. + +Do not make any changes to the codebase in this skill beyond the design file. Gather, map, and hand off. diff --git a/review-plugin-creator/skills/derive-review-rubrics/references/evidence-to-rubric.md b/review-plugin-creator/skills/derive-review-rubrics/references/evidence-to-rubric.md new file mode 100644 index 0000000..6bafbd1 --- /dev/null +++ b/review-plugin-creator/skills/derive-review-rubrics/references/evidence-to-rubric.md @@ -0,0 +1,29 @@ +# Evidence to rubric + +How to turn a recurring evidence pattern into rubric content. A rubric judge has an `evaluation_target` (`description` and `content` are the built-ins; a domain-specific rubric may define its own, any string), one or more `dimensions`, and per-dimension `scores` with an `anchor` and an `example` at each level. See `create-review-plugin` for the full schema and weight invariant. + +## Mapping table + +| Evidence pattern (source) | Rubric translation | +|---|---| +| Skill didn't activate when it should have (logs) | A trigger-clarity dimension on `description`. Score-1 example = the actual description that failed to fire; score-3 example = a description that names the situation it triggers on. The failed-to-fire example needs logs — with no log evidence (e.g. a cloud sandbox), don't fabricate one; ground the same dimension in PR feedback instead (next row). | +| Reviewers kept flagging that a `description` is too vague to tell when the skill fires (PRs) | The same trigger-clarity dimension on `description`, grounded in Stream A. Valid with no logs: anchor the low score in the actual vague descriptions reviewers flagged and the high score in one that names its trigger. Recurring, high-severity discoverability feedback is strong signal — don't drop the dimension just because logs are absent. | +| Skill activated but its instructions were ignored and the user corrected the agent (logs / PRs) | An enforceability dimension on `content` — are the steps imperative, ordered, and checkable. Bad example = the vague step the agent skipped; good example = the same step made concrete. | +| Reviewers kept flagging the same point on skill PRs (find-optimizations) | A "Does the skill avoid X?" dimension. Bad example = the flagged code or text; good example = the fix that resolved the review thread. | +| find-optimizations proposes a `verifier` (a binary, observable invariant) | Keep it as a verifier (a pass/fail LLM-judge check on committed files), out of the rubric. A verifier scores pass/fail; a rubric dimension scores degrees of quality. | + +## Content judges need extra fields + +A judge with `evaluation_target: "content"` must also carry a `scope` (a one-line description of what it evaluates) and `scoring_notes` (with `simple_skills`, `code_vs_instruction_skills`, and `feedback_loops` guidance). The schema rejects a `content` rubric without them. A `description` judge does not need either. Include them in the design you hand to `create-review-plugin`. + +## Choosing dimension vs verifier + +A rubric dimension measures **degrees** of a judgmental quality ("how clear is the trigger"). A **binary** pass/fail invariant about a committed file ("does every route return via the typed helper") belongs outside the rubric: as a verifier — an LLM-judge rule — when no deterministic lint or test can express it, or as a lint/test when one can. If it does not admit a middle grade, it is not a rubric dimension. + +## Weights + +Frequency and severity of a pattern drive its weight. A pattern that recurred across many PRs or logs, or that caused a wrong outcome rather than a stylistic nit, carries more weight. Dimension weights within one rubric sum to 1.0. The plugin-level split — `validation_weight` plus each judge's weight — also sums to 1.0. + +## Anchors + +Each score level needs an `anchor` (what that grade means) and an `example` (a concrete instance). Draw examples from the gathered evidence rather than inventing them: the description that failed to activate, the review comment that recurred, the diff that resolved it. Evidence-grounded anchors are the reason to derive a rubric from evidence rather than author one from scratch.