feat(review-plugin-creator): publish derive-review-rubrics skill

README.md

@@ -9,7 +9,7 @@ These are R&D-maintained plugins intended for production use.
 | Plugin | Description | Badge |
 | --- | --- | --- |
 | [`skill-optimizer`](skill-optimizer/) | Optimize your skills and plugins: review SKILL.md quality, generate eval scenarios, run evals, compare across models, diagnose gaps, and re-run until scores improve | [![](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Fskill-optimizer%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/skill-optimizer) [![](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Fskill-optimizer)](https://tessl.io/registry/tessl/skill-optimizer/evals) |
-| [`review-plugin-creator`](review-plugin-creator/) | Create a custom `tessl review` rubric, by forking the default rubric or building one from scratch | [![review-plugin-creator version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Freview-plugin-creator%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/review-plugin-creator) [![review-plugin-creator evals](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Freview-plugin-creator)](https://tessl.io/registry/tessl/review-plugin-creator/evals) |
+| [`review-plugin-creator`](review-plugin-creator/) | Create a custom `tessl review` rubric – fork the default rubric, build one from scratch, or derive it from evidence (existing skills, PR feedback, agent logs) | [![review-plugin-creator version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Freview-plugin-creator%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/review-plugin-creator) [![review-plugin-creator evals](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Freview-plugin-creator)](https://tessl.io/registry/tessl/review-plugin-creator/evals) |
 | [`api`](api/) | Find and call Tessl API endpoints token-efficiently, without loading the whole `openapi.json` spec into context | [![api version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2Ftesslio%2Fproduct-plugins%2Fmain%2Fapi%2F.tessl-plugin%2Fplugin.json&query=%24.version&label=tessl&color=blue&prefix=v)](https://tessl.io/registry/tessl/api) [![api evals](https://img.shields.io/endpoint?url=https%3A%2F%2Fapi.tessl.io%2Fv1%2Fbadges%2Ftessl%2Fapi)](https://tessl.io/registry/tessl/api/evals) |
 
 ## Install

review-plugin-creator/.tessl-plugin/plugin.json

@@ -1,9 +1,10 @@
 {
   "name": "tessl/review-plugin-creator",
-  "version": "0.1.0",
-  "description": "Guided workflow for creating a custom Tessl reviewer plugin, by forking the default rubric or building one from scratch. Scaffolds the plugin directory structure, authors rubrics and config.json, and validates the result with tessl review run.",
+  "version": "0.2.0",
+  "description": "Create custom Tessl reviewer plugins – fork the default rubric, build one from scratch, or derive its rubrics from evidence (existing skills, PR review feedback, agent logs). Scaffolds the plugin directory structure, authors rubrics and config.json, and validates the result with tessl review run.",
   "private": false,
   "skills": [
-    "skills/create-review-plugin"
+    "skills/create-review-plugin",
+    "skills/derive-review-rubrics"
   ]
 }

review-plugin-creator/README.md

@@ -12,10 +12,11 @@ tessl install tessl/review-plugin-creator
 
 `tessl review` scores a skill against a reviewer plugin. With no `--review-plugin`, it uses Tessl's default rubric (Anthropic best practices). This plugin walks you through creating your own reviewer plugin so reviews reflect your team's standard, then gating CI on that score.
 
-Two starting points:
+Three starting points:
 
-- **Fork the default rubric** — start from Tessl's default rubric (bundled with the skill) and tweak weights, anchors, or dimensions.
-- **Build from scratch** — author new judges from a blank template for a security-only or domain-specific reviewer.
+- **Fork the default rubric** – start from Tessl's default rubric (bundled with the skill) and tweak weights, anchors, or dimensions.
+- **Build from scratch** – author new judges from a blank template for a security-only or domain-specific reviewer.
+- **Derive from evidence** – ground the rubric in how your agents actually behave: existing skills, recurring PR review feedback, and agent logs showing where skills failed to activate or needed correction.
 
 The default rubric is bundled at `skills/create-review-plugin/references/default-rubric/` so you can read exactly what `tessl review` uses out of the box before deciding which path to take.
 
@@ -24,3 +25,4 @@ The default rubric is bundled at `skills/create-review-plugin/references/default
 | Skill | Description |
 |-------|-------------|
 | `create-review-plugin` | Scaffolds the plugin, writes rubric files and config.json, and validates with `tessl review run` |
+| `derive-review-rubrics` | Turns evidence (existing skills, PR feedback, agent logs) into rubric dimensions and anchors, then hands the design to `create-review-plugin` |

review-plugin-creator/evals/derive-rubric-from-evidence/criteria.json

@@ -0,0 +1,61 @@
+{
+  "context": "Tests whether the agent, following derive-review-rubrics, correctly maps a fixed set of evidence to rubric design content. The evidence (in ./inputs/) contains: a target SKILL.md with a vague description and a vague type-sync step; PR feedback with a recurring skill-type api:sync finding, a verifier-type finding about the typed reply helper, and a discoverability complaint; and an agent log showing one session where the skill failed to activate and one where it activated but a vague step was skipped and the user corrected it. The agent must produce a design.md that translates each evidence pattern into the right rubric element (or routes it out of the rubric), with anchors grounded in the actual evidence. The strongest signals are: a trigger-clarity dimension on `description` from the didn't-activate session, an enforceability dimension on `content` (with scope + scoring_notes) from the ignored-step correction, a dimension for the recurring api:sync feedback, and the verifier-type finding routed OUT to a verifier rather than made a rubric dimension.",
+  "type": "weighted_checklist",
+  "checklist": [
+    {
+      "name": "Produces a design file only",
+      "max_score": 8,
+      "description": "A `./design.md` file exists describing the rubric design. The agent did NOT scaffold a plugin directory, copy schemas, run `tessl review run`, or modify any file under `./inputs/`."
+    },
+    {
+      "name": "Trigger-clarity dimension on description",
+      "max_score": 12,
+      "description": "The design includes a dimension that judges trigger/description clarity, and its judge's `evaluation_target` is `description`. This is the correct mapping for the Session 1 evidence where the skill failed to activate."
+    },
+    {
+      "name": "Trigger-clarity anchored in the real failed description",
+      "max_score": 8,
+      "description": "The trigger-clarity dimension's low-score (e.g. score-1) example is the actual description that failed to fire — `\"Add an endpoint to the backend.\"` — and its high-score example is a description that names the situation it triggers on. The example is drawn from the evidence, not invented."
+    },
+    {
+      "name": "Enforceability dimension on content",
+      "max_score": 12,
+      "description": "The design includes an enforceability/actionability dimension whose judge's `evaluation_target` is `content`, mapping the Session 2 evidence (skill activated but the vague step was skipped). Its bad example is the vague step (`\"Make sure the types are in sync.\"`) and its good example makes the step concrete (e.g. `\"Run `bun run api:sync` and commit the regenerated files.\"`)."
+    },
+    {
+      "name": "Content judge carries scope and scoring_notes",
+      "max_score": 10,
+      "description": "Any judge with `evaluation_target: \"content\"` in the design also specifies a `scope` (one-line description of what it evaluates) and `scoring_notes`. A judge on `description` is not required to have them."
+    },
+    {
+      "name": "Dimension for the recurring api:sync feedback",
+      "max_score": 10,
+      "description": "The design includes a dimension derived from the recurring skill-type PR finding (Finding 1): does the skill instruct the agent to run `bun run api:sync` / regenerate API types after a route change."
+    },
+    {
+      "name": "api:sync anchors drawn from the PR evidence",
+      "max_score": 8,
+      "description": "The api:sync dimension's examples are grounded in the actual review comments from PRs #4821 / #4855 / #4902, not invented."
+    },
+    {
+      "name": "Verifier-type finding routed out of the rubric",
+      "max_score": 12,
+      "description": "The verifier-type finding (Finding 2 — every handler returns via the typed `reply.send` helper, never raw `res.end`) is kept as a verifier (a binary pass/fail check), NOT turned into a rubric dimension. The design explicitly states this routing decision and the reason (binary invariant, not a degree of quality)."
+    },
+    {
+      "name": "Dimension weights sum to 1.0 per rubric",
+      "max_score": 8,
+      "description": "Within each rubric the dimension weights sum to 1.0, and the design states this."
+    },
+    {
+      "name": "Plugin-level weight split sums to 1.0",
+      "max_score": 6,
+      "description": "The plugin-level split — `validation_weight` plus each judge's weight — sums to 1.0 in the design."
+    },
+    {
+      "name": "Weights reflect frequency and severity",
+      "max_score": 6,
+      "description": "The design justifies its weights by how often / how severely each pattern recurred (e.g. the api:sync pattern, recurring across three PRs and causing red CI, is weighted higher than a one-off stylistic nit)."
+    }
+  ]
+}

review-plugin-creator/evals/derive-rubric-from-evidence/inputs/agent-log.md

@@ -0,0 +1,31 @@
+# Agent-log excerpt — `tessl agent-logs view` output (provider: claude-code)
+
+Two sessions are reproduced below. Transcripts are trimmed to the relevant turns.
+
+## Session 1 — skill did NOT activate when it should have
+
+```
+user: Add a GET /workspaces/:id/members endpoint to the backend that returns the
+      members of a workspace.
+assistant: I'll create the route handler and schema directly.
+  [no Skill tool_use block; no /add-api-endpoint slash command anywhere in the session]
+  ... hand-writes src/routes/workspaces/members/get.ts ...
+```
+
+The task squarely matches `add-api-endpoint`'s purpose, but neither activation form
+(a `Skill` tool_use nor a `/add-api-endpoint` slash command) appears. The agent
+re-derived the route boilerplate from scratch.
+
+## Session 2 — skill activated but a step was ignored, and the user corrected the agent
+
+```
+user: /add-api-endpoint for the billing webhook
+assistant: [tool_use: Skill, input.skill = "add-api-endpoint"]
+  ... creates the route handler and the request/response schema ...
+  ... does NOT regenerate API types ...
+assistant: Done — the billing webhook endpoint is in place.
+user: You forgot to sync the types. CI is red on api-types-check. Run `bun run api:sync`.
+```
+
+The skill fired, but step 3 ("Make sure the types are in sync") was vague enough that the
+agent skipped the concrete action, and the user had to redirect it.

review-plugin-creator/evals/derive-rubric-from-evidence/inputs/pr-feedback.md

@@ -0,0 +1,32 @@
+# find-optimizations output — PR window: last 4 weeks, scoped to skill PRs
+
+Findings below were produced by `/find-optimizations` over the skill's PR history.
+Each finding carries a **Type**, **Summary**, and **Evidence**.
+
+## Finding 1
+
+- **Type**: skill
+- **Summary**: The `add-api-endpoint` skill never tells the agent to regenerate the API
+  client types after changing a route. Reviewers repeatedly have to ask for
+  `bun run api:sync`, and CI fails on type drift until they do.
+- **Evidence**:
+  - PR #4821 — reviewer: "CI is red on api-types-check. Run `bun run api:sync` and commit the regenerated files."
+  - PR #4855 — reviewer: "Same as last time — you changed a route schema but didn't run `bun run api:sync`."
+  - PR #4902 — reviewer: "Need `bun run api:sync` here too; the frontend client is out of date."
+  - Three PRs, same recurring comment from the same reviewer.
+
+## Finding 2
+
+- **Type**: verifier
+- **Summary**: Every route handler must return through the typed `reply.send(...)` helper,
+  never via raw `res.end(...)`. This is a binary, observable property of the committed file.
+- **Evidence**:
+  - PR #4877 — reviewer: "Use `reply.send` here, not `res.end` — we lose response typing otherwise."
+
+## Finding 3
+
+- **Type**: skill
+- **Summary**: Reviewers and teammates note the skill is hard to discover — its description
+  doesn't say what kind of work triggers it, so agents hand-roll routes instead of using it.
+- **Evidence**:
+  - PR #4902 — reviewer: "Was there not a skill for this? The route boilerplate is all slightly wrong."

review-plugin-creator/evals/derive-rubric-from-evidence/inputs/target-skill/SKILL.md

@@ -0,0 +1,13 @@
+---
+name: add-api-endpoint
+description: Add an endpoint to the backend.
+---
+
+Add a new API endpoint to the Fastify backend.
+
+## Steps
+
+1. Create the route handler under `src/routes/`.
+2. Add the request and response schema.
+3. Make sure the types are in sync.
+4. Test it.

review-plugin-creator/evals/derive-rubric-from-evidence/scenario.json

@@ -0,0 +1,5 @@
+{
+  "type": "generic",
+  "description": "Derive a review-plugin rubric from gathered evidence (agent logs present)",
+  "include": ["./inputs"]
+}

review-plugin-creator/evals/derive-rubric-from-evidence/task.md

@@ -0,0 +1,26 @@
+# Derive a review-plugin rubric from gathered evidence
+
+You are deriving a custom `tessl review` plugin rubric for a single skill, grounded in
+evidence about how agents actually behaved with it. The evidence has already been gathered
+for you (the gathering tools that would normally collect it — `/find-optimizations` and
+`tessl agent-logs view` — are not available in this environment), so work from the files
+provided rather than calling those tools.
+
+## Inputs (in `./inputs/`)
+
+- `inputs/target-skill/SKILL.md` — the skill the produced plugin will score. This is the baseline.
+- `inputs/pr-feedback.md` — `/find-optimizations` output: recurring review findings, each with a
+  Type, Summary, and Evidence.
+- `inputs/agent-log.md` — an agent-log excerpt with two sessions.
+
+## What to produce
+
+Following the `derive-review-rubrics` workflow, map the evidence to rubric content and write
+a single rubric **design file** to `./design.md` in your working directory. Do not scaffold a
+plugin, copy schemas, or run `tessl review run` — that is `create-review-plugin`'s job, which
+happens later. Do not modify any file under `./inputs/`.
+
+The design file should describe the judges and their `evaluation_target`s, the scoring
+dimensions with anchors and examples drawn from the evidence, the `scope` and `scoring_notes`
+for any `content` judge, and the weight split. It should also record any evidence pattern that
+does NOT belong in the rubric and where it should go instead.

review-plugin-creator/evals/derive-rubric-no-agent-logs/criteria.json

@@ -0,0 +1,56 @@
+{
+  "context": "Tests whether the agent correctly applies the derive-review-rubrics skill in a cloud sandbox scenario: proceeding on Stream A alone when no agent logs are available, not fabricating log-based activation evidence (while still deriving a description-clarity dimension from the recurring vague-description PR feedback), classifying evidence patterns as dimensions vs verifiers, grounding anchor examples in the provided PR feedback, and weighting dimensions by frequency/severity. The agent must produce a rubric-design.md suitable for handoff to /create-review-plugin without scaffolding any files.",
+  "type": "weighted_checklist",
+  "checklist": [
+    {
+      "name": "Acknowledges no log evidence",
+      "description": "rubric-design.md explicitly states that agent-log evidence is unavailable (e.g. cloud sandbox) and that the design proceeds on Stream A alone",
+      "max_score": 10
+    },
+    {
+      "name": "No fabricated log-based activation evidence",
+      "description": "With no agent logs available, the design must not invent or claim log-derived activation evidence — e.g. a dimension anchored in 'the skill failed to fire in N transcripts', or a score example that purports to come from observed non-activation. Deriving a description/trigger-clarity dimension from the recurring PR feedback that descriptions are too vague to tell when a skill fires (Theme 1) is correct and expected — that is valid Stream-A evidence. Full marks: no invented log evidence, and any trigger-clarity dimension is anchored in the actual PR review comments rather than in non-existent logs.",
+      "max_score": 12
+    },
+    {
+      "name": "Judges with evaluation_target",
+      "description": "At least one proposed judge includes an explicit evaluation_target field (e.g. 'description', 'content', 'structure')",
+      "max_score": 8
+    },
+    {
+      "name": "Verifier classification present",
+      "description": "At least one evidence pattern from pr-feedback.md is classified as a verifier or lint check (binary pass/fail) rather than a rubric dimension, with a stated reason",
+      "max_score": 10
+    },
+    {
+      "name": "Dimension vs verifier reasoning",
+      "description": "The document explains WHY each classification was made — i.e. distinguishes judgmental qualities from binary pass/fail invariants",
+      "max_score": 8
+    },
+    {
+      "name": "Anchors from actual feedback",
+      "description": "At least two rubric score anchors or examples are drawn verbatim or near-verbatim from the text in pr-feedback.md (not invented examples)",
+      "max_score": 12
+    },
+    {
+      "name": "Frequency/severity drives weights",
+      "description": "The document states that dimension weights reflect the frequency and severity data from pr-feedback.md (e.g. Theme 1 at 18/40 PRs and High severity receiving more weight than Theme 6 at 5/40 PRs and Low severity)",
+      "max_score": 10
+    },
+    {
+      "name": "Weights sum to 1.0 per judge",
+      "description": "Within each proposed judge, the listed dimension weights sum to 1.0 (or the document clearly states they should)",
+      "max_score": 8
+    },
+    {
+      "name": "Handoff note to create-review-plugin",
+      "description": "The document includes a section or note explaining what /create-review-plugin would do next — without actually scaffolding directories, copying schemas, or running tessl commands",
+      "max_score": 10
+    },
+    {
+      "name": "No scaffolding actions taken",
+      "description": "The agent did NOT create any plugin directories, schema files, rubric JSON files, or run any tessl CLI commands — only rubric-design.md was produced",
+      "max_score": 12
+    }
+  ]
+}