From 741830843a490ebfb6ade1c34c9baae580fc4bd7 Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Wed, 1 Jul 2026 22:52:03 +0100 Subject: [PATCH 1/5] feat(plugin-creator): draft context/plugin creator for Sandra's UX review Working-name draft, not published. Six skills implementing the composition-plan flow (understand -> gather -> plan -> build); eval is handed off to skill-optimizer, not woven in. Lints clean; all skills materialise on install. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/.tessl-plugin/plugin.json | 7 ++ plugin-creator/DRAFT-NOTES.md | 46 ++++++++++++ plugin-creator/README.md | 24 +++++++ plugin-creator/TRY-THIS.md | 71 +++++++++++++++++++ .../skills/build-composition/SKILL.md | 48 +++++++++++++ .../references/plugin-anatomy.md | 40 +++++++++++ .../references/shaping-and-decomposition.md | 30 ++++++++ plugin-creator/skills/create-context/SKILL.md | 34 +++++++++ .../skills/decompose-into-skills/SKILL.md | 30 ++++++++ plugin-creator/skills/gather-context/SKILL.md | 37 ++++++++++ .../references/hunting-artifacts.md | 15 ++++ .../skills/plan-composition/SKILL.md | 30 ++++++++ .../references/choosing-the-shape.md | 31 ++++++++ plugin-creator/skills/publish-plugin/SKILL.md | 49 +++++++++++++ plugin-creator/tessl.json | 9 +++ 15 files changed, 501 insertions(+) create mode 100644 plugin-creator/.tessl-plugin/plugin.json create mode 100644 plugin-creator/DRAFT-NOTES.md create mode 100644 plugin-creator/README.md create mode 100644 plugin-creator/TRY-THIS.md create mode 100644 plugin-creator/skills/build-composition/SKILL.md create mode 100644 plugin-creator/skills/build-composition/references/plugin-anatomy.md create mode 100644 plugin-creator/skills/build-composition/references/shaping-and-decomposition.md create mode 100644 plugin-creator/skills/create-context/SKILL.md create mode 100644 plugin-creator/skills/decompose-into-skills/SKILL.md create mode 100644 plugin-creator/skills/gather-context/SKILL.md create mode 100644 plugin-creator/skills/gather-context/references/hunting-artifacts.md create mode 100644 plugin-creator/skills/plan-composition/SKILL.md create mode 100644 plugin-creator/skills/plan-composition/references/choosing-the-shape.md create mode 100644 plugin-creator/skills/publish-plugin/SKILL.md create mode 100644 plugin-creator/tessl.json diff --git a/plugin-creator/.tessl-plugin/plugin.json b/plugin-creator/.tessl-plugin/plugin.json new file mode 100644 index 0000000..774d383 --- /dev/null +++ b/plugin-creator/.tessl-plugin/plugin.json @@ -0,0 +1,7 @@ +{ + "name": "tessl/plugin-creator", + "version": "0.1.0", + "description": "Create a Tessl plugin: wrap your existing skills into a versioned, shareable bundle, decide what else it needs (rules, commands, MCP servers), validate, and publish.", + "private": true, + "skills": "./skills/" +} diff --git a/plugin-creator/DRAFT-NOTES.md b/plugin-creator/DRAFT-NOTES.md new file mode 100644 index 0000000..342dc49 --- /dev/null +++ b/plugin-creator/DRAFT-NOTES.md @@ -0,0 +1,46 @@ +# Draft notes (for Marc's review) + +Draft of the context/plugin creator plugin. Uncommitted, unpublished, untracked in git. Working name only. + +## Model (updated 2026-07-01, per Marc): composition plan + +**Problem-first, and eval is downstream.** Users are not asking to package anything. They usually already tried to solve their problem and now have a skill that is a mess, too long, or not getting enough out of Tessl. This plugin understands the problem, gathers what they have, plans the right shape, builds it, and *then* points them to eval, which stays in skill-optimizer as a separate sibling flow. We never run evals mid-creation. Sprawl/rot is skill-inventory's concern, not this plugin's. + +Note: in Tessl a "skill" is already a minimal single-skill plugin (`tessl skill new` / `import` create the `plugin.json`). So packaging-for-eval is cheap; this plugin's job ends at *composition created*. + +## The arc (and the 6 skills) + +understand → gather → plan → build → (then, separately) eval + +| Skill | Phase / role | +|---|---| +| `create-context` | Orchestrator. Runs the arc; hands off to skill-optimizer for eval at the end. | +| `gather-context` | Understand the problem; take stock of artifacts (skill, plugin, code, prompt, or "go find it"); fill gaps by asking, hunting, or inferring. | +| `plan-composition` | Decide the shape: single skill vs plugin (+ rules / MCP). Produce a short composition plan and confirm it. | +| `build-composition` | Create it: author/restructure/wrap skills, add rules/MCP per plan. Scaffolds with `tessl skill new` / `plugin new`. | +| `decompose-into-skills` | Split a big skill into focused, independently-verifiable skills. Invoked during build. | +| `publish-plugin` | Optional. Kept in for now; distribute workflow may absorb it later. | + +`author-skill-from-intent`, `find-context-opportunities`, `wrap-skills-into-plugin`, and `add-mcp-server-to-plugin` from the previous draft were **absorbed**: gathering (incl. the PR/log hunt) folded into `gather-context`; authoring/wrapping/MCP folded into `build-composition`. + +## What changed this round + +- Reframed to the composition-plan arc; eval is a downstream sibling, never woven in. +- Dropped the "skill → evals → plugin" ladder (it implied evaling mid-flow). +- Unified the entry points into `gather-context` (Marc: author-from-intent and find-opportunities are aspects of the same gathering). +- Folded MCP into `build-composition` (no standalone MCP skill). +- Kept publishing in (Marc: distribute is undefined; pull it out later if needed). +- Removed sprawl/rot as a design anchor. + +## QA status + +- `tessl plugin lint` — PASS. Clean install — PASS, all 6 skills materialise. +- Quality review (`tessl review run`) — still needs `tessl login`. + +## Parked actions / decisions + +1. **Name** (not plugin-first; Sandra may weigh in) — decide once the flow is pinned. +2. **Update skill-optimizer** so that when a user brings it a skill, it can decide to package it and call *these* skills to do so (sibling ↔ sibling). +3. **Fate of existing creators**: `tessl-labs/tile-creator` is deprecated (no tiles; Labs = prototype). The `harness-engineering/plugin-creator` placeholder is closer to a product surface (ships with Tessl agent) so replacing it needs care. +4. Does this plugin own publishing long-term, or defer to distribute? +5. Is 6 skills the right cut? diff --git a/plugin-creator/README.md b/plugin-creator/README.md new file mode 100644 index 0000000..2614eab --- /dev/null +++ b/plugin-creator/README.md @@ -0,0 +1,24 @@ +# plugin-creator (working name) + +> First draft, not published. Built for Marc to review. Working name only, this is not plugin-first. See `DRAFT-NOTES.md`. + +Turn a problem, or a messy existing skill, into well-shaped agent context. It understands what you are trying to do, takes stock of what you already have, plans the right shape (a single skill, or a plugin with rules and MCP servers), and builds it. Evaluation is a separate, downstream step, handled by `tessl/skill-optimizer`; this gets you to a well-formed composition first. + +## The arc + +understand → gather → plan the composition → build → (then, separately) eval + +## Skills + +| Skill | Description | +|-------|-------------| +| `create-context` | Orchestrator. Runs the arc and points to eval at the end. | +| `gather-context` | Understand the problem and take stock of what exists; fill gaps by asking, hunting (PRs/logs), or inferring. | +| `plan-composition` | Decide the shape: a single skill, or a plugin with the right primitives. | +| `build-composition` | Create it, scaffolding with the CLI, not hand-written manifests. | +| `decompose-into-skills` | Split a big skill into focused, independently-verifiable skills. | +| `publish-plugin` | Optional. Publish or re-publish to the registry. | + +## Companion + +For quality (review, eval scenarios, optimisation), `tessl/skill-optimizer` is the sibling flow, after a composition is built. diff --git a/plugin-creator/TRY-THIS.md b/plugin-creator/TRY-THIS.md new file mode 100644 index 0000000..3b3b07b --- /dev/null +++ b/plugin-creator/TRY-THIS.md @@ -0,0 +1,71 @@ +# Try this (Sandra) + +Thanks for kicking the tyres on this draft. It's the **context/plugin creator** (working name), a set of skills that help someone turn a problem, or a messy existing skill, into well-shaped agent context. It does **not** run evals, that stays in skill-optimizer as a separate step. + +What we most want from you: a read on the **workflow and UX**. Does the flow make sense? Does it meet a user where they are? Anything confusing, missing, or annoying? Tweak it freely, it's a draft and nothing here is precious. + +The full rationale (and the user research behind it) is in the PRD Marc shared with you. + +## The flow it runs +understand → gather → plan the composition → build → (then, separately) eval + +Six skills: `create-context` (orchestrator), `gather-context`, `plan-composition`, `build-composition`, `decompose-into-skills`, `publish-plugin`. + +## Set it up (about 5 minutes) +You've got a clone of the `product-plugins` repo on the `opt-plugin-creator-draft` branch. The plugin is at `plugin-creator/`. To try it, install it into a scratch project: + +```bash +# note the absolute path to the plugin-creator folder in your clone: +PLUGIN=/plugin-creator + +mkdir -p ~/pc-sandbox && cd ~/pc-sandbox +cat > tessl.json < "I've got a skill at `sample-messy-skill/SKILL.md` that's a mess, it does endpoints, migrations, and deploys all in one. Help me get more out of Tessl with it." + +Watch whether it understands the problem, proposes a sensible plan (it should suggest splitting into focused skills and explain why), builds it, and only *then* mentions eval, without rewriting your content unasked or dragging you into evals. + +**B - from a brain-dump (no file needed).** +> "I keep re-explaining our service architecture conventions to the agent. Help me capture this. Rough notes: [add three or four bullets]." + +Watch whether it gathers and probes for what's missing rather than dead-ending because you didn't hand it a finished skill. + +## Tweak it +The skills are plain markdown at `plugin-creator/skills/*/SKILL.md`. Edit anything, re-run `tessl install` in `~/pc-sandbox`, and try again. + +## What we'd love back +- Does the flow feel right, or does it get in the way? +- Where did it confuse you or do the wrong thing? +- Anything you'd change in how it talks to the user. + +Notes in whatever form is easiest. Marc picks this up Friday. diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md new file mode 100644 index 0000000..77234e9 --- /dev/null +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -0,0 +1,48 @@ +--- +name: build-composition +description: Use once there is a confirmed composition plan, to create it. Authors or restructures the skill(s), decomposes anything too big, wraps existing skills, and assembles the plugin with any rules or MCP servers the plan called for. Scaffolds with the CLI rather than hand-writing manifests. +--- + +# Build the composition + +Turn the confirmed plan into real files, in the right place. Do not re-litigate the plan here, build it. + +## 1. Scaffold with the CLI + +Create the skeleton with the CLI, never hand-write manifests: + +- A single skill → `tessl skill new` (creates the `SKILL.md` and its `plugin.json`). +- A richer plugin → `tessl plugin new` (can seed skills and rules). +- Wrapping an existing loose `SKILL.md` → `tessl skill import` to generate its `plugin.json`. + +Inspect `tessl skill new --help` / `tessl plugin new --help` for the current flags. See [references/plugin-anatomy.md](references/plugin-anatomy.md). + +## 2. Author or restructure the skills + +- For each skill in the plan, write a strong `description` (what + when, it drives discovery) and lean, ordered steps, with depth pushed into `references/`. See [references/shaping-and-decomposition.md](references/shaping-and-decomposition.md). +- **Honour existing content.** If you are wrapping the user's skill, restructure rather than rewrite. Change wording only if a review flags a real problem and the user agrees. Keep authorship attributed. +- If a skill is too big or does several jobs, hand to `decompose-into-skills`. + +## 3. Add the other primitives the plan called for + +Only add what the plan specified, do not pad. + +- **Rule** — a plain `.md` under `rules/` for an always-on convention. +- **MCP server** — a bundled `.mcp.json` when the plan calls for a capability. Point the manifest at it (`"mcpServers": ".mcp.json"`). Two transports: `http` (a URL) and `stdio` (a command plus args). Do not hard-code secrets. +- **Command** — a `.md` under `commands/` for a user-invoked action. + +## 4. Validate + +```bash +tessl plugin lint +``` + +Optionally install into a throwaway project (a `file:` dependency) and confirm the skills, rules, and any MCP servers materialise. + +## 5. Hand off + +The composition now exists. Point the user at the next steps: eval via `tessl/skill-optimizer` (a separate flow), and `publish-plugin` if they want to share it. Do not run evals here. + +## When to stop + +Stop when the composition is built, lints clean, and the user knows the next steps. diff --git a/plugin-creator/skills/build-composition/references/plugin-anatomy.md b/plugin-creator/skills/build-composition/references/plugin-anatomy.md new file mode 100644 index 0000000..0e0df06 --- /dev/null +++ b/plugin-creator/skills/build-composition/references/plugin-anatomy.md @@ -0,0 +1,40 @@ +# Plugin anatomy + +## Minimum structure + +``` +/ + .tessl-plugin/ + plugin.json # manifest (required) + skills/ + / + SKILL.md + references/ # optional + rules/ # optional + .md + commands/ # optional + .md + .mcp.json # optional (bundled MCP servers) +``` + +Scaffold this with `tessl plugin new` (or `tessl skill new` for a single skill) rather than writing it by hand. A single skill created with `tessl skill new` is already a minimal plugin, it gets a `.tessl-plugin/plugin.json`. + +## `plugin.json` fields + +**Required:** `name` (`/`, matches the key in `tessl.json`), `version` (semver), `description` (the registry shopfront, what others see to decide whether to install). + +**Optional metadata:** `author`, `homepage`, `repository`, `license`, `private` (true keeps it workspace-only; public is irreversible). + +**Content paths (string or array):** `skills` (defaults to `./skills/`), `rules` (defaults to `./rules/`), `commands`, `mcpServers` (literal `".mcp.json"` or `"./.mcp.json"`). + +Hooks (`hooks` / `nativeHooks`) exist in the schema but are not GA. Do not include them in a published plugin. + +## The five primitives, and who triggers each + +- **Skill** — a workflow the *model* loads when the task matches its `description`. +- **Rule** — an always-on convention the agent follows passively. Plain markdown. +- **Command** — an action the *user* invokes explicitly (a slash command). +- **MCP server** — external tools or data, declared in a bundled `.mcp.json`. +- **Hook** — a shell command at a lifecycle event. Not GA yet. + +Rule of thumb: always-on → rule; reach-for-it-when-relevant → skill; a button the user presses → command; needs live tools or data → MCP server. diff --git a/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md b/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md new file mode 100644 index 0000000..7a79401 --- /dev/null +++ b/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md @@ -0,0 +1,30 @@ +# Shaping and decomposition + +The most common authoring mistakes are cramming everything into one skill and packing unrelated skills into one plugin. Fixing both is a core part of the value this plugin adds. + +## Right-size each skill + +- **One skill = one goal-oriented workflow.** Name it as a verb (`write-endpoint`, not `endpoints`). Avoid skills that are thin wrappers over a single CLI command. +- **Keep `SKILL.md` lean.** If it is long or repetitive, move detail into `references/` files and link to them, so the core procedure stays scannable (progressive disclosure). +- **Move supporting material into the skill directory.** Templates, examples, and scripts live alongside `SKILL.md`. +- **A strong `description` is what makes a skill discoverable.** It must say what the skill does and when to use it. + +## Right-size the plugin + +- One responsibility per plugin. A plugin is a focused bundle, not a junk drawer. +- A few tightly-related skills, not fifty. +- An orchestrator skill plus a small number of sub-skills it delegates to is a good shape. + +## Decomposition (why it matters) + +A big skill is hard to get right and hard to verify. Splitting it into focused skills makes each one: + +- **Independently verifiable** — each can be evaluated on its own, later, so its quality can be proven. +- **Independently triggerable** — each has a clear, narrow description, so the agent routes accurately. +- **Easier to maintain** — a change to one does not risk the others. + +The `decompose-into-skills` skill owns this workflow. Lead with the benefit: decomposition is not tidying, it is what makes the result reliable. + +## Split signals + +Decompose a skill when it does several distinct jobs, has unrelated triggers, is long enough that the core procedure is hard to follow, or repeats content and mixes "what to do" with deep reference detail. diff --git a/plugin-creator/skills/create-context/SKILL.md b/plugin-creator/skills/create-context/SKILL.md new file mode 100644 index 0000000..d3d7821 --- /dev/null +++ b/plugin-creator/skills/create-context/SKILL.md @@ -0,0 +1,34 @@ +--- +name: create-context +description: Use when someone wants to turn a problem, a messy or sprawling skill, or some existing work into well-shaped reusable context for their agent. Triggers include "help me make a skill for X", "my skill is a mess, sort it out", "I want to get more out of Tessl with this skill", or "package this so my team can use it". Runs the arc - understand the problem, gather the artifacts, plan the right composition, build it - then points to eval as a separate next step. +--- + +# Create context + +Start from the user's problem, not from "let's make a plugin." Most users are not asking to package anything. They usually already tried to solve their problem and now have a skill that is sprawling, too long, or not getting enough out of Tessl, and they want help. Your job is to understand what they are trying to do and shape their work into the right form. + +A plugin is just the packaging mechanism. Do not lead with it. And do not run evals here, evaluation is a separate, downstream step (see the end). + +## The arc + +Run these phases in order. Each has a skill. + +1. **Understand and gather** — `gather-context`. Understand the problem, take stock of what the user already has (an existing skill or plugin, code, a prompt or transcript, or a pointer like "go look at my PRs"), and fill the gaps by asking or by hunting. The material you need may already be the thing they are trying to create. Make sure you have enough before moving on. +2. **Plan the composition** — `plan-composition`. Decide the right shape to solve the problem. Simple problems want a single skill; richer ones want a plugin, possibly with rules or an MCP server. Produce a short plan and confirm it. +3. **Build the composition** — `build-composition`. Create it: author or restructure the skill(s), decompose anything too big (`decompose-into-skills`), and assemble the plugin with the primitives the plan called for. Scaffold with the CLI, do not hand-write manifests. +4. **Publish (optional)** — `publish-plugin`, only if the user wants to share it now. + +## Then, separately: eval + +Once the composition exists and everything is in the right place, the natural next step is to prove it works. That is a **separate, sibling flow**: hand off to `tessl/skill-optimizer`. Do not pull the user into evals while they are still creating, and do not run evals as part of this arc. Offer it as the next thing once they are done here. + +## Principles + +- **Honour what the user brings.** Restructure rather than rewrite. Change a user's skill content only if a review flags a real problem and they agree. +- **Narrate the why.** Make the value of decomposition or packaging visible. +- **Default to the smallest thing that solves the problem.** If a single skill does it, stop there. A plugin must earn its place. +- **Know the real CLI.** Scaffold with `tessl skill new` / `tessl plugin new`; inspect `tessl --help` rather than inventing commands. + +## When to stop + +Stop when the user has well-shaped context, a skill or a plugin, in the right place, and knows that eval via skill-optimizer is the next step if they want it. diff --git a/plugin-creator/skills/decompose-into-skills/SKILL.md b/plugin-creator/skills/decompose-into-skills/SKILL.md new file mode 100644 index 0000000..dacffa4 --- /dev/null +++ b/plugin-creator/skills/decompose-into-skills/SKILL.md @@ -0,0 +1,30 @@ +--- +name: decompose-into-skills +description: Use during build when a skill or a problem is too big for one skill - a sprawling SKILL.md doing several jobs, or a large problem the user wants a single skill for. Splits it into focused, independently-verifiable skills, with a plugin as the superstructure that holds them together. +--- + +# Decompose into skills + +Big skills are hard to get right and hard to verify. Splitting them into focused pieces is often the most valuable thing this plugin does, so lead with the value, do not treat it as tidying. This is invoked from `build-composition` (or `plan-composition` when the plan already calls for a split). + +## 1. Spot the seams + +A skill is too big when it does several distinct jobs, has unrelated triggers, or is so long the core procedure is hard to follow. Identify the independent responsibilities inside it. + +## 2. Propose the split + +Draft a decomposition: one skill per responsibility, each with its own clear trigger. Show the user the proposed structure and the reasoning. Make the call for them, but bring them in, explain why each split makes the result more reliable and easier to prove later. This is the moment to make the benefit visible. + +## 3. Restructure + +Create the focused skills (`tessl skill new`). Move shared depth into `references/`. If skills depend on one another, make that explicit so they are not used in isolation. The plugin becomes the superstructure that bundles them, a legitimate reason to go from a single skill to a plugin. + +Honour the original content: preserve the user's wording and intent, restructure rather than rewrite, unless a review flags a real problem and the user agrees. + +## 4. Result + +Hand the decomposed set back to `build-composition` to finish assembling. Note to the user that each skill can now be evaluated independently, later, via skill-optimizer, that independence is a big part of why decomposition was worth it. Do not run evals here. + +## When to stop + +Stop when the big skill has become a set of focused, independently-triggerable skills the user is happy with. diff --git a/plugin-creator/skills/gather-context/SKILL.md b/plugin-creator/skills/gather-context/SKILL.md new file mode 100644 index 0000000..f766f16 --- /dev/null +++ b/plugin-creator/skills/gather-context/SKILL.md @@ -0,0 +1,37 @@ +--- +name: gather-context +description: Use at the start of creating context, to understand the problem the user is solving and take stock of what they already have. The user may arrive with an existing skill or plugin, code, a written prompt or transcript, or nothing but a pointer ("go look at my PRs"). Gathers what exists and fills the gaps - by asking or by hunting - until there is enough to plan a composition. +--- + +# Gather context + +Before shaping anything, understand the problem and take stock of the material. Users arrive in different states, meet them where they are. + +## 1. Understand the problem + +What does the user want the agent to do better? Often they have already tried to solve it and now have a skill that is messy, too long, or underperforming. Get the problem clear in a sentence or two. + +## 2. Take stock of what exists + +Look at what they have. The material you need may already be the thing they are trying to create. + +- **An existing skill or plugin** — read it. This is often the starting point ("my skill is a mess"). +- **Code, configs, or docs** — the conventions to capture may be implicit in the codebase. +- **A prompt, transcript, or notes** — raw intent to structure. +- **A pointer** — "go find it in my PRs / recent sessions." + +## 3. Fill the gaps + +Work out what is missing to solve the problem, then fill it the right way: + +- **Ask the user** — when the gap is knowledge only they have (the trigger, the exact steps, constraints). Probe specifically: "you have given me the what; I still need when the agent should reach for this, and the two or three steps you follow." +- **Hunt for it** — when the user wants you to find the evidence yourself (recurring PR feedback, agent logs). See [references/hunting-artifacts.md](references/hunting-artifacts.md). +- **Infer it** — when you can read it from the artifacts already in front of you. Do not ask for what you can find. + +## 4. Confirm you have enough + +Before handing to `plan-composition`, make sure you understand the problem and have the material (existing or gathered) to solve it. If not, keep gathering. Do not jump to building. + +## When to stop + +Stop when you can state the problem and have resolved the artifacts and gaps well enough to plan the composition. diff --git a/plugin-creator/skills/gather-context/references/hunting-artifacts.md b/plugin-creator/skills/gather-context/references/hunting-artifacts.md new file mode 100644 index 0000000..62e78bc --- /dev/null +++ b/plugin-creator/skills/gather-context/references/hunting-artifacts.md @@ -0,0 +1,15 @@ +# Hunting for artifacts + +When the user wants you to find the material yourself rather than hand it over ("look at my PRs and tell me what's worth a skill"), mine two evidence streams. This mirrors how `derive-review-rubrics` (in the `review-plugin-creator` plugin) grounds its work in real evidence. + +## PR review feedback + +Invoke `/find-optimizations` over a PR window (a PR number, or a period like "the last 2 weeks"), scoped to recurring review comments. Repeated corrections are the strongest signal that reusable context is missing. Each finding has a type, a summary, and evidence (PR numbers and the specific comments). + +## Agent logs (optional) + +`tessl agent-logs view --json --since ` (add `--provider` to narrow). Look for tasks where the agent needed correction, or where a skill that should have fired did not. Empty entries mean no local history (e.g. a cloud sandbox), say so and proceed on PR evidence. Do not invent log evidence you do not have. + +## Turn evidence into candidates + +Cluster recurring patterns. Each cluster is a candidate: "the agent keeps getting X wrong, context teaching Y would fix it." Rank by how often and how severely each recurred, present the top few with their evidence, and let the user choose. Feed the chosen candidate back into the problem understanding. If the evidence is thin, say so rather than manufacturing a candidate. diff --git a/plugin-creator/skills/plan-composition/SKILL.md b/plugin-creator/skills/plan-composition/SKILL.md new file mode 100644 index 0000000..cc653d8 --- /dev/null +++ b/plugin-creator/skills/plan-composition/SKILL.md @@ -0,0 +1,30 @@ +--- +name: plan-composition +description: Use once the problem and artifacts are understood, to decide the right shape for the context. Produces a short composition plan - a single skill for simple cases, or a plugin (with rules, MCP servers, or multiple skills) for richer ones - and confirms it with the user before anything is built. +--- + +# Plan the composition + +You know the problem and have the material. Now decide how best to arrange it. This is a plan, not a build, keep it short and confirm it before creating anything. + +## Decide the shape + +Default to the smallest thing that solves the problem. See [references/choosing-the-shape.md](references/choosing-the-shape.md). + +- **A single skill** — for a simple, self-contained workflow or body of knowledge. In Tessl a skill is already a minimal plugin (it gets a `plugin.json`), so this is not a lesser option; it is the right one when the problem is simple. +- **A plugin with several skills** — when the problem has distinct parts (decompose), or when related skills belong together. +- **A plugin with more than skills** — add a **rule** for an always-on convention, or an **MCP server** when the agent needs a tool or live data, not just instructions. Add these only when they clearly earn their place. + +Hooks are not GA. Note them as a follow-up if relevant; do not plan to ship them. + +## Write the plan + +In a few lines, state: the problem, the artifacts, the proposed shape (skill or plugin, plus which primitives), and whether anything needs decomposing. Explain the why, especially if you are proposing to split a big skill. Confirm with the user, or make the call and tell them your reasoning. + +## Hand off + +Pass the confirmed plan to `build-composition`. + +## When to stop + +Stop when there is a confirmed composition plan. diff --git a/plugin-creator/skills/plan-composition/references/choosing-the-shape.md b/plugin-creator/skills/plan-composition/references/choosing-the-shape.md new file mode 100644 index 0000000..9e4fda4 --- /dev/null +++ b/plugin-creator/skills/plan-composition/references/choosing-the-shape.md @@ -0,0 +1,31 @@ +# Choosing the shape + +Decide the smallest arrangement that solves the problem. Guide the user, do not interrogate. + +## Simple vs rich + +- **Simple problem → a single skill.** One workflow or one body of knowledge. It is already a minimal plugin under the hood. +- **Richer problem → a plugin.** Reach for a plugin when the problem has several distinct parts (each its own skill), related skills belong together, or you need more than instructions (a rule or an MCP server). + +Do not escalate to a plugin for its own sake. If a single skill solves it, that is the answer. + +## Which primitive for which intent + +| The user wants the agent to... | Use a... | +|---|---| +| Follow a workflow when a task comes up | Skill | +| Always obey a convention, unprompted | Rule | +| Run a named action on demand | Command | +| Use a tool or live data it does not have | MCP server | +| Run something automatically at a lifecycle event | Hook (not GA, defer) | + +## Heuristics + +- A skill is instructions; an MCP server is capability. +- Rules are for the always-on minority; task-specific guidance belongs in a skill. +- One responsibility per plugin; two unrelated jobs are two plugins. +- Do not add primitives to look complete. + +## Eval is downstream + +Evaluation is not part of choosing the shape, and not part of this plugin. Once the composition is built, skill-optimizer handles eval as a separate step. The only thing composition owes eval is a well-formed skill or plugin for it to run on. diff --git a/plugin-creator/skills/publish-plugin/SKILL.md b/plugin-creator/skills/publish-plugin/SKILL.md new file mode 100644 index 0000000..74a784c --- /dev/null +++ b/plugin-creator/skills/publish-plugin/SKILL.md @@ -0,0 +1,49 @@ +--- +name: publish-plugin +description: Use when a composition is built and the user wants to share it, to publish or re-publish it to the Tessl registry. Lints it, confirms private/public visibility, and publishes. +--- + +# Publish a plugin + +Publishing makes a plugin installable from the registry. It lives in this plugin for now; the broader distribute workflow (rollout across repos, org distribution) is a larger surface that may absorb parts of this later. Only invoke this when the user explicitly wants to publish. + +## 1. Confirm visibility, and ask + +Never publish without an explicit yes, and confirm public vs private first. The `private` field in `.tessl-plugin/plugin.json` controls it: + +- `"private": true` (default): only the workspace can see and install it. +- `"private": false`: anyone can discover and install it. **This is irreversible, a public plugin cannot be made private again.** + +If the user is new to Tessl, be extra clear about what public means. + +## 2. Handle provenance + +If the plugin wraps skills the user did not author, do not publish on their behalf without making ownership clear. Wrapping and publishing someone else's work is a real sensitivity, surface it. + +## 3. Lint and dry-run + +Inspect the live CLI first (`tessl plugin publish --help`). Then: + +```bash +tessl plugin lint +tessl plugin publish --dry-run +``` + +Fix all issues. Publishing requires Tessl auth (`tessl login`, or `TESSL_TOKEN` in CI). Published plugins should have evals (the internal bar is review and task-eval scores of 80%+); if they are missing, point the user at `tessl/skill-optimizer` first, evals are a separate, downstream flow. + +## 4. Publish + +```bash +tessl plugin publish +``` + +Bump the version each release (`--bump patch|minor|major`, or edit `version`). + +## Undo + +- `tessl plugin unpublish` works only within 2 days. +- `tessl plugin archive` is the normal way to retire a plugin: blocks new installs, keeps existing ones working. + +## When to stop + +Stop when the plugin is published at the intended visibility, or the user chooses to keep it local. diff --git a/plugin-creator/tessl.json b/plugin-creator/tessl.json new file mode 100644 index 0000000..591dae3 --- /dev/null +++ b/plugin-creator/tessl.json @@ -0,0 +1,9 @@ +{ + "name": "tessl/plugin-creator-dev", + "mode": "vendored", + "dependencies": { + "tessl/skill-optimizer": { + "version": "0.14.0" + } + } +} From 68b7c6154df6da431943c73f3ae8ba37ba6f53bf Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Sun, 5 Jul 2026 12:18:35 +0100 Subject: [PATCH 2/5] feat(plugin-creator): fix decomposition/build QA findings, add eval scenarios - build-composition / decompose-into-skills: stop instructing `tessl skill new` per sub-skill (it nests a standalone plugin that `pack` silently drops); author skills//SKILL.md directly, and verify with `pack`, not just `lint` - plan-composition / choosing-the-shape: add explicit skill-vs-rule checkpoint for always-on-convention requests - build-composition: correct `tessl plugin new` flag guidance (--skill/--rules, --workspace) - shaping-and-decomposition: distinguish peer-skills vs orchestrator shapes - add 5 generated eval scenarios (evals/) QA: sonnet-4-6 content eval 65% -> 97% avg (no regressions); skill reviews 85-100. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/.tesslignore | 2 + plugin-creator/evals/scenario-0/criteria.json | 56 +++++++ plugin-creator/evals/scenario-0/scenario.json | 1 + plugin-creator/evals/scenario-0/task.md | 25 +++ plugin-creator/evals/scenario-1/criteria.json | 56 +++++++ .../evals/scenario-1/inputs/SKILL.md | 148 ++++++++++++++++++ plugin-creator/evals/scenario-1/scenario.json | 1 + plugin-creator/evals/scenario-1/task.md | 22 +++ plugin-creator/evals/scenario-2/criteria.json | 56 +++++++ plugin-creator/evals/scenario-2/scenario.json | 1 + plugin-creator/evals/scenario-2/task.md | 21 +++ plugin-creator/evals/scenario-3/criteria.json | 46 ++++++ .../my-plugin/.tessl-plugin/plugin.json | 11 ++ plugin-creator/evals/scenario-3/scenario.json | 1 + plugin-creator/evals/scenario-3/task.md | 17 ++ plugin-creator/evals/scenario-4/criteria.json | 66 ++++++++ .../evals/scenario-4/inputs/SKILL.md | 23 +++ .../evals/scenario-4/inputs/pr-feedback.json | 38 +++++ plugin-creator/evals/scenario-4/scenario.json | 1 + plugin-creator/evals/scenario-4/task.md | 24 +++ .../skills/build-composition/SKILL.md | 46 ++++-- .../references/shaping-and-decomposition.md | 2 +- .../skills/decompose-into-skills/SKILL.md | 4 +- .../skills/plan-composition/SKILL.md | 19 ++- .../references/choosing-the-shape.md | 7 +- 25 files changed, 671 insertions(+), 23 deletions(-) create mode 100644 plugin-creator/.tesslignore create mode 100644 plugin-creator/evals/scenario-0/criteria.json create mode 100644 plugin-creator/evals/scenario-0/scenario.json create mode 100644 plugin-creator/evals/scenario-0/task.md create mode 100644 plugin-creator/evals/scenario-1/criteria.json create mode 100644 plugin-creator/evals/scenario-1/inputs/SKILL.md create mode 100644 plugin-creator/evals/scenario-1/scenario.json create mode 100644 plugin-creator/evals/scenario-1/task.md create mode 100644 plugin-creator/evals/scenario-2/criteria.json create mode 100644 plugin-creator/evals/scenario-2/scenario.json create mode 100644 plugin-creator/evals/scenario-2/task.md create mode 100644 plugin-creator/evals/scenario-3/criteria.json create mode 100644 plugin-creator/evals/scenario-3/my-plugin/.tessl-plugin/plugin.json create mode 100644 plugin-creator/evals/scenario-3/scenario.json create mode 100644 plugin-creator/evals/scenario-3/task.md create mode 100644 plugin-creator/evals/scenario-4/criteria.json create mode 100644 plugin-creator/evals/scenario-4/inputs/SKILL.md create mode 100644 plugin-creator/evals/scenario-4/inputs/pr-feedback.json create mode 100644 plugin-creator/evals/scenario-4/scenario.json create mode 100644 plugin-creator/evals/scenario-4/task.md diff --git a/plugin-creator/.tesslignore b/plugin-creator/.tesslignore new file mode 100644 index 0000000..b950ba7 --- /dev/null +++ b/plugin-creator/.tesslignore @@ -0,0 +1,2 @@ +DRAFT-NOTES.md +TRY-THIS.md diff --git a/plugin-creator/evals/scenario-0/criteria.json b/plugin-creator/evals/scenario-0/criteria.json new file mode 100644 index 0000000..1df27c6 --- /dev/null +++ b/plugin-creator/evals/scenario-0/criteria.json @@ -0,0 +1,56 @@ +{ + "context": "Tests whether the agent correctly applies Tessl plugin conventions when scaffolding a single-skill plugin: verb-form skill naming, valid plugin.json manifest fields, skill description quality (what + when), progressive disclosure via references/, and choosing the smallest appropriate shape.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Verb-form skill name", + "description": "The skill folder name is a verb or verb phrase (e.g. 'review-pr', 'audit-pull-request') — NOT a noun or noun phrase (e.g. 'prs', 'pr-review-skill', 'code-review-tool')", + "max_score": 15 + }, + { + "name": "plugin.json name format", + "description": "plugin.json contains a 'name' field in the format '/' (two slash-separated segments, no extra slashes)", + "max_score": 12 + }, + { + "name": "plugin.json semver version", + "description": "plugin.json contains a 'version' field that is a valid semver string (e.g. '1.0.0', '0.1.0')", + "max_score": 8 + }, + { + "name": "plugin.json description present", + "description": "plugin.json contains a non-empty 'description' field", + "max_score": 8 + }, + { + "name": "SKILL.md description covers what", + "description": "The SKILL.md frontmatter 'description' field states what the skill does (i.e. describes the PR review workflow or its output)", + "max_score": 10 + }, + { + "name": "SKILL.md description covers when", + "description": "The SKILL.md frontmatter 'description' field also states when to use the skill (e.g. 'when reviewing a pull request', 'use when...', or a trigger condition)", + "max_score": 10 + }, + { + "name": "References directory used", + "description": "A 'references/' subdirectory exists inside the skill folder containing at least one file", + "max_score": 12 + }, + { + "name": "SKILL.md links to references", + "description": "SKILL.md contains at least one markdown link pointing into the references/ directory (e.g. '[see checklist](references/checklist.md)')", + "max_score": 10 + }, + { + "name": "Single-skill shape chosen", + "description": "The agent produced a single skill (not a multi-skill plugin with multiple skill folders) — the problem is simple and self-contained", + "max_score": 10 + }, + { + "name": "plan.md justifies shape choice", + "description": "plan.md explains why a single skill was chosen (or if a plugin with multiple skills was chosen, gives a clear rationale beyond 'completeness')", + "max_score": 5 + } + ] +} diff --git a/plugin-creator/evals/scenario-0/scenario.json b/plugin-creator/evals/scenario-0/scenario.json new file mode 100644 index 0000000..e8eebb2 --- /dev/null +++ b/plugin-creator/evals/scenario-0/scenario.json @@ -0,0 +1 @@ +{ "description": "Tessl plugin scaffolding conventions" } diff --git a/plugin-creator/evals/scenario-0/task.md b/plugin-creator/evals/scenario-0/task.md new file mode 100644 index 0000000..d6b5f4b --- /dev/null +++ b/plugin-creator/evals/scenario-0/task.md @@ -0,0 +1,25 @@ +# Build a PR Review Skill for the Engineering Team + +## Problem Description + +Your engineering team has started using Tessl to share reusable agent workflows, and the team lead wants to add a pull-request review workflow to the shared plugin registry. Right now, developers rely on ad hoc checklists or personal notes when reviewing PRs — there is no consistent process, and newer team members miss common issues around correctness, style, and security. + +The team lead has asked you to create a Tessl skill that an agent can follow when reviewing a pull request. The skill should cover the full review process: checking for logic bugs, evaluating code style consistency, flagging potential security concerns, and summarising findings. It should be written so that another developer or an AI agent can pick it up and use it without hand-holding. + +The team lead wants this shipped as the smallest reasonable Tessl artifact — not a large multi-skill bundle unless that is genuinely needed. The skill (and any supporting files) should be placed on disk in a directory structure ready for use with Tessl tooling. + +## Output Specification + +Produce the following on disk: + +- A Tessl plugin directory (e.g. `.tessl-plugin/` or a named folder) containing: + - A `plugin.json` manifest + - A skill folder containing a `SKILL.md` entry point + - Any supporting reference files your skill links to +- A `plan.md` file at the top level of your working directory that explains: + - The shape you chose (single skill vs multi-skill plugin) and why + - How you named the skill and why + - What you put in `SKILL.md` versus any `references/` files, and why + - Any design decisions about the description field + +Do not include the actual CLI output or session logs — just the final files and the `plan.md` explaining your decisions. diff --git a/plugin-creator/evals/scenario-1/criteria.json b/plugin-creator/evals/scenario-1/criteria.json new file mode 100644 index 0000000..8339e65 --- /dev/null +++ b/plugin-creator/evals/scenario-1/criteria.json @@ -0,0 +1,56 @@ +{ + "context": "Tests whether the agent correctly decomposes a monolithic skill into focused, independently-triggerable sub-skills following the decompose-into-skills guidance. The agent must lead with the value of decomposition, use verb-form skill names, surface shared material in references/, and avoid running evals or hinting at them.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Benefit-led rationale", + "description": "decomposition-rationale.md leads with the benefit of the split (reliability, independent verifiability, or ease of maintenance) — not with mechanics or tidying framing", + "max_score": 12 + }, + { + "name": "Independent verifiability mentioned", + "description": "decomposition-rationale.md mentions that each skill can now be evaluated or verified independently (e.g., references skill-optimizer or independent evaluation)", + "max_score": 10 + }, + { + "name": "No eval-running", + "description": "decomposition-rationale.md does NOT announce plans to run evals now or as an immediate next step — evaluation is deferred or pointed at a separate tool", + "max_score": 8 + }, + { + "name": "Verb-form skill names", + "description": "All skill sub-folder names use verb form (e.g., migrate-schema, backup-database, monitor-health, manage-access) — none use noun-only names like 'migrations' or 'backups'", + "max_score": 10 + }, + { + "name": "One responsibility per skill", + "description": "Each resulting SKILL.md covers exactly one distinct responsibility area; no two skills have substantially overlapping trigger conditions", + "max_score": 12 + }, + { + "name": "Shared material in references/", + "description": "Shared reference content (e.g., environment variable requirements, safety rules) is placed in a references/ directory rather than duplicated across multiple skill files", + "max_score": 10 + }, + { + "name": "Orchestrator or explicit delegation", + "description": "Either an orchestrator skill exists that delegates to sub-skills, OR the plugin.json / rationale document makes delegation relationships explicit", + "max_score": 10 + }, + { + "name": "Cross-skill dependencies explicit", + "description": "If any skill depends on another (e.g., backup-database is a prerequisite for migrate-schema), that dependency is called out explicitly in the skill content or rationale — skills are not expected to be used in isolation when they cannot be", + "max_score": 8 + }, + { + "name": "plugin.json present", + "description": "A plugin.json manifest exists at the top of the plugin directory with at minimum name, version, and description fields", + "max_score": 8 + }, + { + "name": "Decomposition structure shown", + "description": "decomposition-rationale.md includes a proposed structure listing each skill name and its trigger or responsibility — not just prose explanation", + "max_score": 12 + } + ] +} diff --git a/plugin-creator/evals/scenario-1/inputs/SKILL.md b/plugin-creator/evals/scenario-1/inputs/SKILL.md new file mode 100644 index 0000000..f588f81 --- /dev/null +++ b/plugin-creator/evals/scenario-1/inputs/SKILL.md @@ -0,0 +1,148 @@ +--- +name: database-admin +description: Database administration tasks including schema changes, backups, monitoring, and user access management. +--- + +# Database Administration + +This skill covers everything related to running and maintaining a PostgreSQL database in production. + +## Schema Migrations + +When the user needs to change the database schema: + +1. Connect to the database using `psql $DATABASE_URL`. +2. Check pending migrations: `SELECT * FROM schema_migrations WHERE applied_at IS NULL ORDER BY version;` +3. Review the migration file carefully before applying. +4. Run the migration: `psql $DATABASE_URL -f migrations/_.sql` +5. Confirm the migration applied: `SELECT version, applied_at FROM schema_migrations ORDER BY applied_at DESC LIMIT 5;` +6. If the migration fails, roll back: `psql $DATABASE_URL -f migrations/__rollback.sql` +7. Update the application's ORM model if a column was added or removed. +8. Notify the engineering channel in Slack: "#db-changes". + +### Migration file conventions +- Filename: `_.sql` (e.g. `20240512140000_add_user_preferences.sql`) +- Always include a corresponding `_rollback.sql` file. +- Migrations must be idempotent where possible (`CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`). +- Never drop a column in the same migration that removes it from the application code — do it in a later release. + +--- + +## Database Backups + +When the user asks to back up the database or when a backup is needed before a risky operation: + +1. Identify the target database: confirm with the user if unclear. +2. Run a full dump: `pg_dump $DATABASE_URL > backup_$(date +%Y%m%d_%H%M%S).sql` +3. Compress: `gzip backup_*.sql` +4. Upload to S3: `aws s3 cp backup_*.sql.gz s3://company-db-backups/postgres//` +5. Verify the upload: `aws s3 ls s3://company-db-backups/postgres// | tail -5` +6. Delete the local copy after confirming upload: `rm backup_*.sql.gz` +7. Log the backup event in the #db-ops Slack channel. + +### Restore procedure +1. Download from S3: `aws s3 cp s3://company-db-backups/postgres//.sql.gz .` +2. Decompress: `gunzip .sql.gz` +3. Restore: `psql $DATABASE_URL < .sql` +4. Run a quick sanity check: `SELECT COUNT(*) FROM users;` (should match pre-backup count). + +--- + +## Database Monitoring + +When the user asks about database health, slow queries, or connection issues: + +1. Check active connections: `SELECT count(*), state FROM pg_stat_activity GROUP BY state;` +2. Find slow queries (running > 30s): + ```sql + SELECT pid, now() - pg_stat_activity.query_start AS duration, query + FROM pg_stat_activity + WHERE state = 'active' AND now() - query_start > interval '30 seconds' + ORDER BY duration DESC; + ``` +3. Check table bloat: + ```sql + SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size + FROM pg_tables + ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC + LIMIT 20; + ``` +4. Check index usage: + ```sql + SELECT relname, indexrelname, idx_scan, idx_tup_read, idx_tup_fetch + FROM pg_stat_user_indexes + WHERE idx_scan = 0 + ORDER BY relname; + ``` +5. Summarize findings to the user with recommendations (add index, VACUUM, kill query). +6. If any query has been running for > 5 minutes, ask the user before killing it: `SELECT pg_terminate_backend();` + +### Common fixes +- High connection count → check application connection pooling, consider PgBouncer. +- Table bloat → run `VACUUM ANALYZE ;` +- Unused indexes → flag for removal in the next maintenance window. + +--- + +## User & Access Management + +When the user needs to create, modify, or revoke database access: + +1. List current roles: `SELECT rolname, rolsuper, rolcreaterole, rolcreatedb FROM pg_roles ORDER BY rolname;` +2. To create a read-only user: + ```sql + CREATE ROLE WITH LOGIN PASSWORD ''; + GRANT CONNECT ON DATABASE TO ; + GRANT USAGE ON SCHEMA public TO ; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO ; + ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO ; + ``` +3. To create a read-write user: + ```sql + CREATE ROLE WITH LOGIN PASSWORD ''; + GRANT CONNECT ON DATABASE TO ; + GRANT USAGE ON SCHEMA public TO ; + GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO ; + GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO ; + ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO ; + ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO ; + ``` +4. To revoke all access: + ```sql + REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM ; + REVOKE CONNECT ON DATABASE FROM ; + DROP ROLE ; + ``` +5. Never share the password in Slack — send it via 1Password or a secure channel. +6. Notify the requester and log the change in the #db-access Slack channel. + +### Audit: check who has access +```sql +SELECT grantee, table_name, privilege_type +FROM information_schema.role_table_grants +WHERE table_schema = 'public' +ORDER BY grantee, table_name; +``` + +--- + +## Shared Reference: Environment Variables + +All database tasks rely on these environment variables being set: + +- `DATABASE_URL` — full connection string, e.g. `postgresql://user:pass@host:5432/dbname` +- `AWS_PROFILE` or `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` — for S3 backup operations +- `SLACK_WEBHOOK_URL` — for posting notifications to Slack channels + +If any of these are missing, ask the user to provide them before proceeding. + +--- + +## Shared Reference: Safety Rules + +These rules apply to all database operations: + +- **Always take a backup before a destructive operation** (DROP TABLE, DELETE without WHERE, major migration). +- **Never run operations on production without confirming the environment** — check `$ENVIRONMENT` variable. +- **Prefer dry-run or preview mode** when available. +- **Document every change** in the relevant Slack channel. diff --git a/plugin-creator/evals/scenario-1/scenario.json b/plugin-creator/evals/scenario-1/scenario.json new file mode 100644 index 0000000..ccb89cf --- /dev/null +++ b/plugin-creator/evals/scenario-1/scenario.json @@ -0,0 +1 @@ +{ "description": "Decompose monolithic skill into focused sub-skills", "include": ["./inputs"] } diff --git a/plugin-creator/evals/scenario-1/task.md b/plugin-creator/evals/scenario-1/task.md new file mode 100644 index 0000000..8744145 --- /dev/null +++ b/plugin-creator/evals/scenario-1/task.md @@ -0,0 +1,22 @@ +# Refactor a Sprawling Database Admin Skill + +## Problem Description + +The platform team at Fieldstone Technologies has been maintaining a single monolithic skill file that covers everything related to database administration — schema migrations, backups, health monitoring, and user access management. Over time this file has grown unwieldy: it has four clearly distinct responsibility areas with different triggers, different audiences, and different risk profiles. A junior engineer recently ran the backup steps when they meant to run the monitoring steps, causing unnecessary downtime. + +The team lead has decided it is time to split the skill into focused, independently maintainable units. The skill file lives at `inputs/SKILL.md`. There is no other infrastructure to set up; all the context you need is in that file. + +Your job is to analyze the existing skill, propose and execute a decomposition into a well-structured multi-skill plugin, and produce the resulting file structure on disk. + +## Output Specification + +Produce the following in your working directory: + +1. **A decomposed plugin directory** — a folder named `database-admin-plugin/` containing a proper plugin manifest and one sub-folder per focused skill, each with its own skill file. Organize shared material appropriately so it is not duplicated across skills. + +2. **`decomposition-rationale.md`** — a short document (a page or less) explaining: + - Why you decomposed the skill the way you did and what the agent gains from this structure + - The proposed structure: list each resulting skill, its name, and when it fires + - What happens next — any recommendations for the team after the restructure is in place + +Do not preserve the monolithic structure. The goal is a clean multi-skill plugin where each unit can stand alone. diff --git a/plugin-creator/evals/scenario-2/criteria.json b/plugin-creator/evals/scenario-2/criteria.json new file mode 100644 index 0000000..d1cc8d3 --- /dev/null +++ b/plugin-creator/evals/scenario-2/criteria.json @@ -0,0 +1,56 @@ +{ + "context": "Tests whether the agent correctly maps a set of team requirements to Tessl primitives (skill, rule, MCP server, hooks), justifies each choice, avoids padding with unneeded primitives, and correctly handles a not-GA primitive (hooks) by deferring rather than planning to ship it.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Skill for PR review", + "description": "Recommends a skill (not a rule or command) for the task-specific, on-demand PR review workflow", + "max_score": 10 + }, + { + "name": "Rule for import convention", + "description": "Recommends a rule (not a skill) for the always-on Python import ordering convention", + "max_score": 10 + }, + { + "name": "MCP server for metrics API", + "description": "Recommends an MCP server (not a skill or rule) for the live metrics API query capability", + "max_score": 10 + }, + { + "name": "Hooks deferred, not shipped", + "description": "Does NOT plan to ship hooks for the commit-triggered summaries; notes that hooks are not GA or unavailable, and defers the requirement as a follow-up", + "max_score": 12 + }, + { + "name": "Justification for each primitive", + "description": "Provides a brief justification for each primitive choice (not just a list of types), explaining why that primitive fits the requirement", + "max_score": 8 + }, + { + "name": "No padding", + "description": "Does NOT add extra primitives (e.g. a command, a second MCP server, additional rules/skills) that were not needed by the stated requirements", + "max_score": 10 + }, + { + "name": "Plan is concise", + "description": "The composition-plan.md is structured as a short plan (not a full implementation or lengthy spec) — a few lines per component rather than detailed code or exhaustive prose", + "max_score": 10 + }, + { + "name": "Plugin shape stated", + "description": "States the overall proposed shape — whether this is a single skill or a plugin — and names which primitives will be bundled together", + "max_score": 10 + }, + { + "name": "No secrets hard-coded for MCP", + "description": "Does not hard-code the bearer token or API URL secret in the plan; notes that credentials should be provided via environment variable or configuration", + "max_score": 10 + }, + { + "name": "Skill vs rule distinction clear", + "description": "The plan explicitly distinguishes between always-on (rule) and on-demand/task-specific (skill) as the reason for the primitive split, not just listing names", + "max_score": 10 + } + ] +} diff --git a/plugin-creator/evals/scenario-2/scenario.json b/plugin-creator/evals/scenario-2/scenario.json new file mode 100644 index 0000000..4042a7c --- /dev/null +++ b/plugin-creator/evals/scenario-2/scenario.json @@ -0,0 +1 @@ +{ "description": "Primitive selection and composition planning" } diff --git a/plugin-creator/evals/scenario-2/task.md b/plugin-creator/evals/scenario-2/task.md new file mode 100644 index 0000000..dbc3b85 --- /dev/null +++ b/plugin-creator/evals/scenario-2/task.md @@ -0,0 +1,21 @@ +# Plugin Shape Planning for a Backend Engineering Team + +## Problem/Feature Description + +The Meridian backend team has been using AI coding assistants for a few months and wants to make the experience more consistent across engineers. They've identified four areas where they want AI assistance baked into their workflow: + +1. **Pull request review workflow**: Engineers follow a specific internal PR review process — they check diff size, scan for missing tests, look for breaking API changes, and verify that CHANGELOG entries are present. This workflow is something they invoke on demand when reviewing a PR, not something that should fire constantly. + +2. **Import ordering convention**: The team always wants Python imports sorted in a specific order (stdlib → third-party → internal), no exceptions. This should apply whenever the agent touches a Python file — they don't want to invoke it, it should just always be enforced. + +3. **Internal metrics dashboard**: During incident response and performance reviews, engineers frequently need to query the team's internal metrics API (`https://metrics.meridian.internal/api`) to pull latency percentiles, error rates, and deployment event timelines. The agent needs to be able to call this API with a bearer token to fetch live data on demand. + +4. **Commit-triggered summaries**: The team mentioned it would be great if the agent could automatically produce a summary of changes whenever a commit lands — no manual invocation needed. + +Your job is to produce a `composition-plan.md` file that describes how you would build this out in Tessl — which components to use for each need, the overall shape of the solution, and anything the team should know or confirm before you start building. + +## Output Specification + +Produce a single file named `composition-plan.md` in the working directory. + +The plan should describe how each of the four team requirements maps to the appropriate Tessl building blocks, and note anything the team should know before work begins. diff --git a/plugin-creator/evals/scenario-3/criteria.json b/plugin-creator/evals/scenario-3/criteria.json new file mode 100644 index 0000000..fab817f --- /dev/null +++ b/plugin-creator/evals/scenario-3/criteria.json @@ -0,0 +1,46 @@ +{ + "context": "Tests whether an agent producing a plugin-publishing checklist correctly surfaces provenance sensitivity for third-party skills, warns about the irreversibility of making a plugin public, includes the required lint and dry-run steps, mentions auth requirements, bumps the version for a second release, and handles plugin retirement correctly.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Provenance concern raised", + "description": "The checklist explicitly mentions that the plugin wraps skills authored by someone else (a colleague/third party) and states this must be addressed or confirmed before publishing", + "max_score": 16 + }, + { + "name": "Public visibility confirmed", + "description": "The checklist includes a step to confirm or explicitly set the plugin's visibility to public (i.e., changing or verifying the `private` field in plugin.json)", + "max_score": 14 + }, + { + "name": "Public is irreversible warning", + "description": "The checklist states that making a plugin public is irreversible — a public plugin cannot be made private again", + "max_score": 14 + }, + { + "name": "Lint step included", + "description": "The checklist includes running `tessl plugin lint` (with the plugin directory) as a required pre-publish step", + "max_score": 12 + }, + { + "name": "Dry-run step included", + "description": "The checklist includes running `tessl plugin publish --dry-run` (or equivalent) before the actual publish command", + "max_score": 12 + }, + { + "name": "Auth requirement mentioned", + "description": "The checklist mentions that publishing requires authentication via `tessl login` or the `TESSL_TOKEN` environment variable", + "max_score": 10 + }, + { + "name": "Version bump step", + "description": "The checklist includes bumping the plugin version (e.g., using `--bump patch`, `--bump minor`, `--bump major`, or editing the `version` field in plugin.json) since this is a second release", + "max_score": 12 + }, + { + "name": "Retirement via archive, not unpublish", + "description": "The note about taking the plugin down recommends `tessl plugin archive` as the standard approach, OR clearly states that `tessl plugin unpublish` only works within 2 days of publishing — does NOT present unpublish as a general rollback strategy", + "max_score": 10 + } + ] +} diff --git a/plugin-creator/evals/scenario-3/my-plugin/.tessl-plugin/plugin.json b/plugin-creator/evals/scenario-3/my-plugin/.tessl-plugin/plugin.json new file mode 100644 index 0000000..4834000 --- /dev/null +++ b/plugin-creator/evals/scenario-3/my-plugin/.tessl-plugin/plugin.json @@ -0,0 +1,11 @@ +{ + "name": "acme/data-pipeline-tools", + "version": "0.1.0", + "description": "A plugin that bundles data pipeline authoring skills for the Acme workspace, including skills contributed by external collaborators.", + "private": true, + "skills": [ + "skills/write-pipeline", + "skills/validate-schema", + "skills/transform-csv" + ] +} diff --git a/plugin-creator/evals/scenario-3/scenario.json b/plugin-creator/evals/scenario-3/scenario.json new file mode 100644 index 0000000..609064c --- /dev/null +++ b/plugin-creator/evals/scenario-3/scenario.json @@ -0,0 +1 @@ +{ "description": "Plugin publish checklist with provenance, visibility, lint, dry-run, auth, and versioning", "include": ["./my-plugin"] } diff --git a/plugin-creator/evals/scenario-3/task.md b/plugin-creator/evals/scenario-3/task.md new file mode 100644 index 0000000..90ecaca --- /dev/null +++ b/plugin-creator/evals/scenario-3/task.md @@ -0,0 +1,17 @@ +# Publishing the data-pipeline-tools Plugin + +## Background + +You're a developer at Acme working on the `data-pipeline-tools` Tessl plugin. The plugin bundles several data pipeline authoring skills, but two of the three skills (`validate-schema` and `transform-csv`) were originally written by a colleague, Jordan, who has since moved to another team. You've been handed ownership and are now preparing the plugin for a wider release. + +The plugin started as an internal tool (`private: true`) and version `0.1.0`, but the team has decided it should be shared publicly across the organization and beyond. This is the plugin's second release — you've added improvements to `write-pipeline` since the first version. + +The plugin directory is at `my-plugin/`. The plugin manifest is at `my-plugin/.tessl-plugin/plugin.json`. + +## Your Task + +Produce a file called `publish-checklist.md` in the current directory. The checklist should walk through every step a developer needs to complete to safely and correctly publish this plugin — from pre-publish checks through to the actual publish command. + +The checklist should reflect the full context: the plugin is moving from private to public, it includes skills you didn't write yourself, and this is not the first release. Write it as an actionable, numbered list that a developer could follow step by step. + +Also include a short note at the bottom of `publish-checklist.md` about what to do if, after publishing, the team decides they want to take the plugin down or prevent new installs. diff --git a/plugin-creator/evals/scenario-4/criteria.json b/plugin-creator/evals/scenario-4/criteria.json new file mode 100644 index 0000000..ad24286 --- /dev/null +++ b/plugin-creator/evals/scenario-4/criteria.json @@ -0,0 +1,66 @@ +{ + "context": "Tests whether the agent correctly distinguishes between what can be inferred from provided artifacts (SKILL.md and PR feedback JSON) versus what must be asked of the user. Also tests that the agent does not fabricate evidence, honors the existing skill's intent, and proposes a strong skill description covering both what the skill does and when to use it.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Identifies test gap", + "description": "The 'What We Already Know' section includes missing pre-deploy test run as an inferred gap (sourced from the PR feedback — do NOT ask the user about this)", + "max_score": 8 + }, + { + "name": "Identifies migration gap", + "description": "The 'What We Already Know' section includes missing database migration step as an inferred gap (sourced from the PR feedback)", + "max_score": 8 + }, + { + "name": "Identifies smoke test gap", + "description": "The 'What We Already Know' section includes missing post-deploy smoke test / health check as an inferred gap (sourced from the PR feedback)", + "max_score": 8 + }, + { + "name": "Identifies env var gap", + "description": "The 'What We Already Know' section includes undocumented environment variables as an inferred gap (sourced from the PR feedback or SKILL.md)", + "max_score": 8 + }, + { + "name": "No redundant questions", + "description": "The 'Questions for the Team Lead' section does NOT ask about things already present in the fixtures (e.g., does not ask which steps are missing when PR feedback already lists them)", + "max_score": 10 + }, + { + "name": "Questions target unknowns only", + "description": "All questions in 'Questions for the Team Lead' concern genuinely unknown information (e.g., required env var names, rollback procedure, target environment specifics, trigger conditions) not readable from the provided files", + "max_score": 10 + }, + { + "name": "No fabricated evidence", + "description": "The output does NOT claim to have run `/find-optimizations` or `tessl agent-logs` with specific results; any mention of these tools is framed as what could be done next, not as already completed", + "max_score": 9 + }, + { + "name": "Preserves original intent", + "description": "The Problem Summary and analysis treat the existing SKILL.md content as a base to improve, not discard — it acknowledges the skill exists and describes what it currently covers", + "max_score": 7 + }, + { + "name": "Description has 'what'", + "description": "The Proposed Skill Description states what the skill does (e.g., guides production deployment of the platform service)", + "max_score": 8 + }, + { + "name": "Description has 'when'", + "description": "The Proposed Skill Description states when to use the skill (e.g., when deploying to production, before a release, etc.)", + "max_score": 8 + }, + { + "name": "Sources cited", + "description": "Items in 'What We Already Know' reference which artifact (SKILL.md or a specific PR feedback pattern) they were inferred from", + "max_score": 8 + }, + { + "name": "No invented PR feedback", + "description": "The output does not introduce additional PR feedback patterns beyond the four provided in pr-feedback.json", + "max_score": 8 + } + ] +} diff --git a/plugin-creator/evals/scenario-4/inputs/SKILL.md b/plugin-creator/evals/scenario-4/inputs/SKILL.md new file mode 100644 index 0000000..5700d3c --- /dev/null +++ b/plugin-creator/evals/scenario-4/inputs/SKILL.md @@ -0,0 +1,23 @@ +# Deploy Skill + +Use this skill when deploying. + +## Steps + +1. Make sure things are ready +2. Run the deploy command +3. Check that it worked + +## Notes + +- Sometimes you need to set environment variables first +- The deploy might take a while +- If it fails, look at the logs + +## Commands + +```bash +deploy --env production +``` + +Watch for errors in the output. diff --git a/plugin-creator/evals/scenario-4/inputs/pr-feedback.json b/plugin-creator/evals/scenario-4/inputs/pr-feedback.json new file mode 100644 index 0000000..c18d0f7 --- /dev/null +++ b/plugin-creator/evals/scenario-4/inputs/pr-feedback.json @@ -0,0 +1,38 @@ +{ + "repository": "acme-corp/platform", + "period": "last 3 months", + "recurring_review_comments": [ + { + "comment_id": "rc-001", + "count": 7, + "pattern": "Missing pre-deploy test run", + "representative_comment": "You deployed without running tests first. The test suite must pass before any deploy to production. This has caused rollbacks twice this quarter.", + "pr_examples": ["PR #412", "PR #438", "PR #451"], + "reviewer": "@sarah-ops" + }, + { + "comment_id": "rc-002", + "count": 5, + "pattern": "No database migration step", + "representative_comment": "The deploy script ran but the database migrations weren't applied. Always run `db migrate` before starting the new service version.", + "pr_examples": ["PR #419", "PR #433", "PR #447"], + "reviewer": "@sarah-ops" + }, + { + "comment_id": "rc-003", + "count": 4, + "pattern": "No smoke test after deploy", + "representative_comment": "After deploying, you need to hit the /health endpoint and verify the response. The skill says to 'check that it worked' but doesn't specify how.", + "pr_examples": ["PR #421", "PR #436", "PR #449"], + "reviewer": "@devops-bot" + }, + { + "comment_id": "rc-004", + "count": 3, + "pattern": "Environment variables not documented", + "representative_comment": "What env vars are required? The skill just says 'sometimes you need to set environment variables first' but never lists them. Reviewers keep asking the same question.", + "pr_examples": ["PR #415", "PR #429", "PR #444"], + "reviewer": "@marcus-dev" + } + ] +} diff --git a/plugin-creator/evals/scenario-4/scenario.json b/plugin-creator/evals/scenario-4/scenario.json new file mode 100644 index 0000000..b93c029 --- /dev/null +++ b/plugin-creator/evals/scenario-4/scenario.json @@ -0,0 +1 @@ +{ "description": "Infer vs ask — context analysis without over-questioning", "include": ["./inputs"] } diff --git a/plugin-creator/evals/scenario-4/task.md b/plugin-creator/evals/scenario-4/task.md new file mode 100644 index 0000000..872dc14 --- /dev/null +++ b/plugin-creator/evals/scenario-4/task.md @@ -0,0 +1,24 @@ +# Skill Improvement Context Analysis + +## Background + +The platform team at Acme Corp has a deployment skill that's been in use for several months. Recently, the team lead noticed that the same mistakes keep appearing in PRs — engineers keep skipping steps or getting confused about what the deployment process actually requires. The team wants to improve the skill so these gaps are closed. + +A colleague has handed over two artifacts for you to work with: + +- `inputs/SKILL.md` — the current deployment skill as it exists today +- `inputs/pr-feedback.json` — a summary of recurring review comments left by senior engineers on recent deployment PRs, showing patterns of what goes wrong repeatedly + +The team lead's request is: "I want to improve this skill and package it up properly. Can you analyze what we have and tell me what we know and what we still need to find out?" + +## Output Specification + +Produce a file called `context-analysis.md` in the current directory with the following sections: + +1. **Problem Summary** — One or two sentences describing what is wrong with the current skill, based on what you can see in the existing materials. + +2. **What We Already Know** — A list of gaps, missing steps, or improvements that can be inferred directly from the existing `SKILL.md` and the PR feedback. For each item, note the source (e.g., which recurring comment pattern it came from, or what in the SKILL.md reveals it). + +3. **Questions for the Team Lead** — A list of questions that genuinely cannot be answered from the available materials — things that only the team lead or domain experts would know. Do not include questions whose answers are already present in the artifacts. + +4. **Proposed Skill Description** — A revised one-sentence description for the improved skill. This should state both what the skill does and when a developer should reach for it. diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md index 77234e9..008cc8d 100644 --- a/plugin-creator/skills/build-composition/SKILL.md +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -1,48 +1,66 @@ --- name: build-composition -description: Use once there is a confirmed composition plan, to create it. Authors or restructures the skill(s), decomposes anything too big, wraps existing skills, and assembles the plugin with any rules or MCP servers the plan called for. Scaffolds with the CLI rather than hand-writing manifests. +description: Use once there is a confirmed composition plan, to create it. Scaffolds the plugin, authors or restructures the skill(s), decomposes anything too big, and adds any rules or MCP servers the plan called for. Uses the CLI to scaffold the plugin, then adds further skills by hand. --- # Build the composition Turn the confirmed plan into real files, in the right place. Do not re-litigate the plan here, build it. -## 1. Scaffold with the CLI +## 1. Scaffold the plugin once -Create the skeleton with the CLI, never hand-write manifests: +Create the plugin skeleton with the CLI. Inspect `tessl plugin new --help` first, and note two things it requires: +- Pass `--skill` (with `--skill-name` and `--skill-description`) or `--rules` to seed initial content; it errors with neither. +- Pass `--workspace ` explicitly. The workspace in `--name workspace/plugin` is **not** parsed from the name. -- A single skill → `tessl skill new` (creates the `SKILL.md` and its `plugin.json`). -- A richer plugin → `tessl plugin new` (can seed skills and rules). -- Wrapping an existing loose `SKILL.md` → `tessl skill import` to generate its `plugin.json`. +- A single standalone skill → `tessl skill new` (creates the `SKILL.md` and its `plugin.json`). +- A plugin → `tessl plugin new --skill --skill-name --skill-description "..." --workspace `. +- Wrapping an existing loose `SKILL.md` into a plugin → `tessl skill import`. -Inspect `tessl skill new --help` / `tessl plugin new --help` for the current flags. See [references/plugin-anatomy.md](references/plugin-anatomy.md). +See [references/plugin-anatomy.md](references/plugin-anatomy.md). -## 2. Author or restructure the skills +## 2. Add each further skill by hand, not with `tessl skill new` -- For each skill in the plan, write a strong `description` (what + when, it drives discovery) and lean, ordered steps, with depth pushed into `references/`. See [references/shaping-and-decomposition.md](references/shaping-and-decomposition.md). +To add more skills to an existing plugin, **create the file directly**: + +``` +skills//SKILL.md # plus a references/ folder as needed +``` + +The manifest's `skills: ./skills/` discovers them automatically. + +**Do not run `tessl skill new` again inside a plugin to add a skill.** That command always scaffolds a standalone skill-as-plugin, which nests a second plugin inside yours (`skills//.tessl-plugin/...`). `tessl plugin lint` may still call the result "valid", but `tessl plugin pack` silently drops the mis-nested skill, so the plugin ships incomplete. Only the plugin's own `plugin.json` should be CLI-scaffolded; individual `SKILL.md` files are authored directly. + +## 3. Author or restructure the skills + +- Write a strong `description` (what + when, it drives discovery) and lean, ordered steps, with depth pushed into `references/`. See [references/shaping-and-decomposition.md](references/shaping-and-decomposition.md). - **Honour existing content.** If you are wrapping the user's skill, restructure rather than rewrite. Change wording only if a review flags a real problem and the user agrees. Keep authorship attributed. - If a skill is too big or does several jobs, hand to `decompose-into-skills`. -## 3. Add the other primitives the plan called for +## 4. Add the other primitives the plan called for Only add what the plan specified, do not pad. -- **Rule** — a plain `.md` under `rules/` for an always-on convention. +- **Rule** — a plain `.md` under `rules/` for an always-on convention. If the plan identified a convention the agent should always follow (see `plan-composition`), it belongs here, not in a skill. - **MCP server** — a bundled `.mcp.json` when the plan calls for a capability. Point the manifest at it (`"mcpServers": ".mcp.json"`). Two transports: `http` (a URL) and `stdio` (a command plus args). Do not hard-code secrets. - **Command** — a `.md` under `commands/` for a user-invoked action. -## 4. Validate +## 5. Validate with lint AND pack + +Lint checks structure; pack confirms what actually ships. Because lint can pass on a mis-nested skill, always do both: ```bash tessl plugin lint +tessl plugin pack --output /tmp/check.tgz +tar tzf /tmp/check.tgz | grep SKILL.md # every intended skill must appear ``` Optionally install into a throwaway project (a `file:` dependency) and confirm the skills, rules, and any MCP servers materialise. -## 5. Hand off +## 6. Hand off The composition now exists. Point the user at the next steps: eval via `tessl/skill-optimizer` (a separate flow), and `publish-plugin` if they want to share it. Do not run evals here. ## When to stop -Stop when the composition is built, lints clean, and the user knows the next steps. +Stop when the composition is built, lints clean, packs with every intended skill, and the user knows the next steps. diff --git a/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md b/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md index 7a79401..2b07c92 100644 --- a/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md +++ b/plugin-creator/skills/build-composition/references/shaping-and-decomposition.md @@ -13,7 +13,7 @@ The most common authoring mistakes are cramming everything into one skill and pa - One responsibility per plugin. A plugin is a focused bundle, not a junk drawer. - A few tightly-related skills, not fifty. -- An orchestrator skill plus a small number of sub-skills it delegates to is a good shape. +- Two shapes are both fine: **peer skills** (several independently-triggered skills with no coordinator, e.g. separate logging / errors / config skills) and **orchestrator plus sub-skills** (one entry skill that delegates). Use an orchestrator only when the skills genuinely need coordinating; do not add one just to look complete. ## Decomposition (why it matters) diff --git a/plugin-creator/skills/decompose-into-skills/SKILL.md b/plugin-creator/skills/decompose-into-skills/SKILL.md index dacffa4..9289fb5 100644 --- a/plugin-creator/skills/decompose-into-skills/SKILL.md +++ b/plugin-creator/skills/decompose-into-skills/SKILL.md @@ -17,7 +17,9 @@ Draft a decomposition: one skill per responsibility, each with its own clear tri ## 3. Restructure -Create the focused skills (`tessl skill new`). Move shared depth into `references/`. If skills depend on one another, make that explicit so they are not used in isolation. The plugin becomes the superstructure that bundles them, a legitimate reason to go from a single skill to a plugin. +Create each focused skill as `skills//SKILL.md` directly (do **not** run `tessl skill new` per sub-skill inside a plugin, it nests a broken standalone plugin that `pack` silently drops, see `build-composition`). Move shared depth into `references/`. If skills depend on one another, make that explicit so they are not used in isolation. The plugin becomes the superstructure that bundles them, a legitimate reason to go from a single skill to a plugin. + +Not every piece is a skill. If decomposition surfaces an always-on convention (something the agent should always follow, not a workflow it triggers), that piece belongs in a **rule**, not a skill. Route it accordingly. Honour the original content: preserve the user's wording and intent, restructure rather than rewrite, unless a review flags a real problem and the user agrees. diff --git a/plugin-creator/skills/plan-composition/SKILL.md b/plugin-creator/skills/plan-composition/SKILL.md index cc653d8..fdbaf96 100644 --- a/plugin-creator/skills/plan-composition/SKILL.md +++ b/plugin-creator/skills/plan-composition/SKILL.md @@ -7,19 +7,28 @@ description: Use once the problem and artifacts are understood, to decide the ri You know the problem and have the material. Now decide how best to arrange it. This is a plan, not a build, keep it short and confirm it before creating anything. +## First, skill or rule? + +Before deciding single-skill-vs-plugin, decide what *kind* of context each part is, because it is easy to default everything to skills. See [references/choosing-the-shape.md](references/choosing-the-shape.md). + +- If the agent should **follow a workflow when a relevant task comes up**, that is a **skill**. +- If the agent should **always obey a convention, unprompted**, that is a **rule**. + +Watch for always-on-convention language: "the agent keeps getting X wrong", "every time", "it never follows our...". That is usually a **rule** (or a rule plus a skill), not a skill alone. Do not push a conventions request toward skills by default. + ## Decide the shape -Default to the smallest thing that solves the problem. See [references/choosing-the-shape.md](references/choosing-the-shape.md). +Default to the smallest thing that solves the problem. -- **A single skill** — for a simple, self-contained workflow or body of knowledge. In Tessl a skill is already a minimal plugin (it gets a `plugin.json`), so this is not a lesser option; it is the right one when the problem is simple. -- **A plugin with several skills** — when the problem has distinct parts (decompose), or when related skills belong together. -- **A plugin with more than skills** — add a **rule** for an always-on convention, or an **MCP server** when the agent needs a tool or live data, not just instructions. Add these only when they clearly earn their place. +- **A single skill** — for a simple, self-contained workflow. In Tessl a skill is already a minimal plugin (it gets a `plugin.json`), so this is not a lesser option; it is the right one when the problem is simple. +- **A plugin with several skills, and/or rules** — when the problem has distinct parts (decompose), when related skills belong together, or when it mixes workflows (skills) with always-on conventions (rules). +- **A plugin with an MCP server** — when the agent needs a tool or live data, not just instructions. Add only when it clearly earns its place. Hooks are not GA. Note them as a follow-up if relevant; do not plan to ship them. ## Write the plan -In a few lines, state: the problem, the artifacts, the proposed shape (skill or plugin, plus which primitives), and whether anything needs decomposing. Explain the why, especially if you are proposing to split a big skill. Confirm with the user, or make the call and tell them your reasoning. +In a few lines, state: the problem, the artifacts, and the proposed shape, which primitive each part becomes (skill vs rule vs MCP), and whether anything needs decomposing. Explain the why, especially if you are proposing to split a big skill or to encode a convention as a rule. Confirm with the user, or make the call and tell them your reasoning. ## Hand off diff --git a/plugin-creator/skills/plan-composition/references/choosing-the-shape.md b/plugin-creator/skills/plan-composition/references/choosing-the-shape.md index 9e4fda4..21c474f 100644 --- a/plugin-creator/skills/plan-composition/references/choosing-the-shape.md +++ b/plugin-creator/skills/plan-composition/references/choosing-the-shape.md @@ -19,10 +19,13 @@ Do not escalate to a plugin for its own sake. If a single skill solves it, that | Use a tool or live data it does not have | MCP server | | Run something automatically at a lifecycle event | Hook (not GA, defer) | +## Skill vs rule: do not default to skills + +The most common mistake is making everything a skill. A skill fires when the agent judges a task relevant; a rule is always in context. If the user says the agent "keeps getting X wrong", "every time", or "never follows our convention", that is an always-on convention, which is a **rule** (sometimes a rule plus a skill), not a skill on its own. Weigh this explicitly for any conventions-style request before defaulting to a skill. + ## Heuristics -- A skill is instructions; an MCP server is capability. -- Rules are for the always-on minority; task-specific guidance belongs in a skill. +- A skill is instructions the agent reaches for; a rule is a convention it always obeys; an MCP server is a capability. - One responsibility per plugin; two unrelated jobs are two plugins. - Do not add primitives to look complete. From 6a8171422be8678ab4235177b73fd0ece05f2147 Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Sun, 5 Jul 2026 22:03:42 +0100 Subject: [PATCH 3/5] chore(plugin-creator): publish public for v0.1.0 Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/.tessl-plugin/plugin.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugin-creator/.tessl-plugin/plugin.json b/plugin-creator/.tessl-plugin/plugin.json index 774d383..16e6335 100644 --- a/plugin-creator/.tessl-plugin/plugin.json +++ b/plugin-creator/.tessl-plugin/plugin.json @@ -2,6 +2,6 @@ "name": "tessl/plugin-creator", "version": "0.1.0", "description": "Create a Tessl plugin: wrap your existing skills into a versioned, shareable bundle, decide what else it needs (rules, commands, MCP servers), validate, and publish.", - "private": true, + "private": false, "skills": "./skills/" } From cde9fed73fe8b2b1cc4b31946f058946edb3c9f6 Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Sun, 5 Jul 2026 22:07:56 +0100 Subject: [PATCH 4/5] chore(plugin-creator): address CodeRabbit review - create-context: clarify CLI guidance (skill new = standalone, sub-skills authored directly) - build-composition: broaden pack smoke test to rules and commands - plan-composition: add commands to the shape decision - decompose-into-skills: require verb-form sub-skill names - .tesslignore the dev tessl.json so it does not ship a second identity Skipped: the two eval-fixture SKILL.md nits (fixtures are intentionally imperfect inputs). Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/.tesslignore | 1 + plugin-creator/skills/build-composition/SKILL.md | 2 ++ plugin-creator/skills/create-context/SKILL.md | 2 +- plugin-creator/skills/decompose-into-skills/SKILL.md | 2 +- plugin-creator/skills/plan-composition/SKILL.md | 2 +- 5 files changed, 6 insertions(+), 3 deletions(-) diff --git a/plugin-creator/.tesslignore b/plugin-creator/.tesslignore index b950ba7..0c31931 100644 --- a/plugin-creator/.tesslignore +++ b/plugin-creator/.tesslignore @@ -1,2 +1,3 @@ DRAFT-NOTES.md TRY-THIS.md +tessl.json diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md index 008cc8d..1473b02 100644 --- a/plugin-creator/skills/build-composition/SKILL.md +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -53,6 +53,8 @@ Lint checks structure; pack confirms what actually ships. Because lint can pass tessl plugin lint tessl plugin pack --output /tmp/check.tgz tar tzf /tmp/check.tgz | grep SKILL.md # every intended skill must appear +tar tzf /tmp/check.tgz | grep -E '(^|/)rules/[^/]+\.md$' || true # any rules the plan called for +tar tzf /tmp/check.tgz | grep -E '(^|/)commands/[^/]+\.md$' || true # any commands the plan called for ``` Optionally install into a throwaway project (a `file:` dependency) and confirm the skills, rules, and any MCP servers materialise. diff --git a/plugin-creator/skills/create-context/SKILL.md b/plugin-creator/skills/create-context/SKILL.md index d3d7821..87cd586 100644 --- a/plugin-creator/skills/create-context/SKILL.md +++ b/plugin-creator/skills/create-context/SKILL.md @@ -27,7 +27,7 @@ Once the composition exists and everything is in the right place, the natural ne - **Honour what the user brings.** Restructure rather than rewrite. Change a user's skill content only if a review flags a real problem and they agree. - **Narrate the why.** Make the value of decomposition or packaging visible. - **Default to the smallest thing that solves the problem.** If a single skill does it, stop there. A plugin must earn its place. -- **Know the real CLI.** Scaffold with `tessl skill new` / `tessl plugin new`; inspect `tessl --help` rather than inventing commands. +- **Know the real CLI.** Use `tessl skill new` for a standalone skill and `tessl plugin new` for plugin-level scaffolding; inside a plugin, sub-skills are authored directly under `skills//SKILL.md` (see `build-composition`), not with `tessl skill new`. Inspect `tessl --help` rather than inventing commands. ## When to stop diff --git a/plugin-creator/skills/decompose-into-skills/SKILL.md b/plugin-creator/skills/decompose-into-skills/SKILL.md index 9289fb5..979e1ae 100644 --- a/plugin-creator/skills/decompose-into-skills/SKILL.md +++ b/plugin-creator/skills/decompose-into-skills/SKILL.md @@ -13,7 +13,7 @@ A skill is too big when it does several distinct jobs, has unrelated triggers, o ## 2. Propose the split -Draft a decomposition: one skill per responsibility, each with its own clear trigger. Show the user the proposed structure and the reasoning. Make the call for them, but bring them in, explain why each split makes the result more reliable and easier to prove later. This is the moment to make the benefit visible. +Draft a decomposition: one skill per responsibility, each with its own clear trigger and a verb-form skill name. Show the user the proposed structure and the reasoning. Make the call for them, but bring them in, explain why each split makes the result more reliable and easier to prove later. This is the moment to make the benefit visible. ## 3. Restructure diff --git a/plugin-creator/skills/plan-composition/SKILL.md b/plugin-creator/skills/plan-composition/SKILL.md index fdbaf96..8f3951d 100644 --- a/plugin-creator/skills/plan-composition/SKILL.md +++ b/plugin-creator/skills/plan-composition/SKILL.md @@ -1,6 +1,6 @@ --- name: plan-composition -description: Use once the problem and artifacts are understood, to decide the right shape for the context. Produces a short composition plan - a single skill for simple cases, or a plugin (with rules, MCP servers, or multiple skills) for richer ones - and confirms it with the user before anything is built. +description: Use once the problem and artifacts are understood, to decide the right shape for the context. Produces a short composition plan - a single skill for simple cases, or a plugin (with rules, commands, MCP servers, or multiple skills) for richer ones - and confirms it with the user before anything is built. --- # Plan the composition From 7996fb3eaf3012b92b9d09263e8323b749c0f2a2 Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Mon, 6 Jul 2026 08:15:23 +0100 Subject: [PATCH 5/5] feat(plugin-creator): close coverage gaps vs the old plugin-creator skill - gather-context: read .tessl/memory/preferences (plugins/agents/source-control-and-ci) for grounding - build-composition: run security review for plugins touching MCP/scripts/secrets/auth/CI - choosing-the-shape: point deterministic-check / verifier behaviours at change-verify, not skills/rules Replicates the old harness-engineering plugin-creator's remaining behaviour so this can supersede it. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/skills/build-composition/SKILL.md | 2 ++ plugin-creator/skills/gather-context/SKILL.md | 1 + .../skills/plan-composition/references/choosing-the-shape.md | 4 ++++ 3 files changed, 7 insertions(+) diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md index 1473b02..7a19e12 100644 --- a/plugin-creator/skills/build-composition/SKILL.md +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -59,6 +59,8 @@ tar tzf /tmp/check.tgz | grep -E '(^|/)commands/[^/]+\.md$' || true # any comm Optionally install into a throwaway project (a `file:` dependency) and confirm the skills, rules, and any MCP servers materialise. +If the plugin ships an MCP server, scripts, or otherwise touches secrets, auth, file or network access, or CI, also run a security review (`tessl review run security `; inspect `tessl review --help` for the current form). Expect findings and triage each, a flagged item is not automatically a blocker but should be reviewed with the user. + ## 6. Hand off The composition now exists. Point the user at the next steps: eval via `tessl/skill-optimizer` (a separate flow), and `publish-plugin` if they want to share it. Do not run evals here. diff --git a/plugin-creator/skills/gather-context/SKILL.md b/plugin-creator/skills/gather-context/SKILL.md index f766f16..1b206c7 100644 --- a/plugin-creator/skills/gather-context/SKILL.md +++ b/plugin-creator/skills/gather-context/SKILL.md @@ -19,6 +19,7 @@ Look at what they have. The material you need may already be the thing they are - **Code, configs, or docs** — the conventions to capture may be implicit in the codebase. - **A prompt, transcript, or notes** — raw intent to structure. - **A pointer** — "go find it in my PRs / recent sessions." +- **Team preferences** — if `.tessl/memory/preferences/` exists, read it for grounding: `plugins.md` (where plugins are kept), `agents.md` (which agents to set up), `source-control-and-ci.md` (existing checks and conventions). Use these so you infer rather than ask. ## 3. Fill the gaps diff --git a/plugin-creator/skills/plan-composition/references/choosing-the-shape.md b/plugin-creator/skills/plan-composition/references/choosing-the-shape.md index 21c474f..8fb3dc3 100644 --- a/plugin-creator/skills/plan-composition/references/choosing-the-shape.md +++ b/plugin-creator/skills/plan-composition/references/choosing-the-shape.md @@ -29,6 +29,10 @@ The most common mistake is making everything a skill. A skill fires when the age - One responsibility per plugin; two unrelated jobs are two plugins. - Do not add primitives to look complete. +## Some behaviours are not skills or rules at all + +Not every "the agent should X" belongs in the plugin. If a behaviour is better enforced deterministically (a lint, a test, or a hook once hooks are GA), or is a broad observable invariant better expressed as a verifier (`tessl change verify`, an LLM-judge at glob level), say so and point the user at the `change-verify` skill rather than forcing it into skill or rule prose. Context is the right tool for workflows and conventions the agent reasons about, not for deterministic guardrails. + ## Eval is downstream Evaluation is not part of choosing the shape, and not part of this plugin. Once the composition is built, skill-optimizer handles eval as a separate step. The only thing composition owes eval is a well-formed skill or plugin for it to run on.