From c2717a3b94a2d54eeac7dbf03d0032809d5163a0 Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Wed, 15 Jul 2026 19:21:38 +0100 Subject: [PATCH 1/4] docs(plugin-creator): sharpen build guidance to reduce agent thrash (OPT-1590) Address feedback from running plugin-creator, where ambiguous/outdated guidance led to avoidable thrashing: - Warn against authoring source under .tessl/plugins/ (install cache) and recommend a repo-owned default (tessl-plugins//) referenced via file:. - Clarify tessl plugin lint as the primary structural check, and when the separate tessl skill lint applies. - Update review examples to current CLI: tessl review run quality|security. - Note --workspace for non-interactive review runs to avoid the interactive workspace prompt hanging. Bump plugin-creator to 0.1.2. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/.tessl-plugin/plugin.json | 2 +- plugin-creator/skills/build-composition/SKILL.md | 8 ++++++-- .../skills/build-composition/references/plugin-anatomy.md | 6 ++++++ 3 files changed, 13 insertions(+), 3 deletions(-) diff --git a/plugin-creator/.tessl-plugin/plugin.json b/plugin-creator/.tessl-plugin/plugin.json index e2065a4..25e55e8 100644 --- a/plugin-creator/.tessl-plugin/plugin.json +++ b/plugin-creator/.tessl-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "tessl/plugin-creator", - "version": "0.1.1", + "version": "0.1.2", "description": "Create a Tessl plugin: wrap your existing skills into a versioned, shareable bundle, decide what else it needs (rules, commands, MCP servers), validate, and publish.", "private": false, "skills": "./skills/" diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md index 7a19e12..2fdcf5d 100644 --- a/plugin-creator/skills/build-composition/SKILL.md +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -9,6 +9,8 @@ Turn the confirmed plan into real files, in the right place. Do not re-litigate ## 1. Scaffold the plugin once +First choose **where** to author it: a repo-owned directory you commit, defaulting to `tessl-plugins//` when `.tessl/memory/preferences/plugins.md` names no other. **Never author under `.tessl/plugins/`** — that is Tessl's installed-content cache, not source. See [references/plugin-anatomy.md](references/plugin-anatomy.md). + Create the plugin skeleton with the CLI. Inspect `tessl plugin new --help` first, and note two things it requires: - Pass `--skill` (with `--skill-name` and `--skill-description`) or `--rules` to seed initial content; it errors with neither. - Pass `--workspace ` explicitly. The workspace in `--name workspace/plugin` is **not** parsed from the name. @@ -47,7 +49,7 @@ Only add what the plan specified, do not pad. ## 5. Validate with lint AND pack -Lint checks structure; pack confirms what actually ships. Because lint can pass on a mis-nested skill, always do both: +`tessl plugin lint ` is the primary structural check; pack confirms what actually ships. Because lint can pass on a mis-nested skill, always do both: ```bash tessl plugin lint @@ -57,9 +59,11 @@ tar tzf /tmp/check.tgz | grep -E '(^|/)rules/[^/]+\.md$' || true # any rule tar tzf /tmp/check.tgz | grep -E '(^|/)commands/[^/]+\.md$' || true # any commands the plan called for ``` +`tessl skill lint ` is a separate tool for a single loose `SKILL.md` not yet inside a plugin. Do not run it against a skill folder within your plugin, `tessl plugin lint` already covers those; if it errors about missing plugin metadata, you pointed it at the wrong shape, drop it. + Optionally install into a throwaway project (a `file:` dependency) and confirm the skills, rules, and any MCP servers materialise. -If the plugin ships an MCP server, scripts, or otherwise touches secrets, auth, file or network access, or CI, also run a security review (`tessl review run security `; inspect `tessl review --help` for the current form). Expect findings and triage each, a flagged item is not automatically a blocker but should be reviewed with the user. +If the plugin ships an MCP server, scripts, or otherwise touches secrets, auth, file or network access, or CI, also run a security review: `tessl review run security --workspace ` (inspect `tessl review run --help` for the current form). Pass `--workspace ` on any non-interactive `tessl review run`, quality or security (e.g. `tessl review run quality --workspace `), or the CLI opens an interactive workspace selector and hangs. Expect findings and triage each, a flagged item is not automatically a blocker but should be reviewed with the user. ## 6. Hand off diff --git a/plugin-creator/skills/build-composition/references/plugin-anatomy.md b/plugin-creator/skills/build-composition/references/plugin-anatomy.md index 0e0df06..ddea2c0 100644 --- a/plugin-creator/skills/build-composition/references/plugin-anatomy.md +++ b/plugin-creator/skills/build-composition/references/plugin-anatomy.md @@ -19,6 +19,12 @@ Scaffold this with `tessl plugin new` (or `tessl skill new` for a single skill) rather than writing it by hand. A single skill created with `tessl skill new` is already a minimal plugin, it gets a `.tessl-plugin/plugin.json`. +## Where to author the source + +Author the plugin in a **repo-owned directory you commit and control**. If `.tessl/memory/preferences/plugins.md` names a plugins dir, use it. Otherwise default to `tessl-plugins//` (or `plugins//`), then reference it from `tessl.json` with `file:`. + +**Do not author source under `.tessl/plugins/`.** That path is Tessl's installed-content cache, where installed plugins are materialised, not a source directory. Anything you write there can be overwritten or wiped by install operations. + ## `plugin.json` fields **Required:** `name` (`/`, matches the key in `tessl.json`), `version` (semver), `description` (the registry shopfront, what others see to decide whether to install). From 681a18ecf41cb1c09600b948c276aaa6348112bb Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Fri, 17 Jul 2026 15:00:58 +0100 Subject: [PATCH 2/4] test(plugin-creator): add scenario-5 regression eval for author location (OPT-1590) Seeds a populated .tessl/plugins/ install cache via a directory fixture, asks the agent to build a plugin without saying where, and scores whether it authors in a repo-owned dir vs the cache. Verified the guidance holds: baseline (no plugin) authored into .tessl/plugins/tessl/commit-writer/ (22/100); with the edited guidance injected, authored into tessl-plugins/commit-writer/ wired via file: (91/100). The .tessl/plugins/ trap moves 0 -> 35. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/evals/scenario-5/criteria.json | 31 +++++++++++++++++++ .../pr-reviewer/.tessl-plugin/plugin.json | 7 +++++ .../skills/review-pull-request/SKILL.md | 12 +++++++ plugin-creator/evals/scenario-5/scenario.json | 10 ++++++ plugin-creator/evals/scenario-5/task.md | 24 ++++++++++++++ 5 files changed, 84 insertions(+) create mode 100644 plugin-creator/evals/scenario-5/criteria.json create mode 100644 plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/.tessl-plugin/plugin.json create mode 100644 plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md create mode 100644 plugin-creator/evals/scenario-5/scenario.json create mode 100644 plugin-creator/evals/scenario-5/task.md diff --git a/plugin-creator/evals/scenario-5/criteria.json b/plugin-creator/evals/scenario-5/criteria.json new file mode 100644 index 0000000..441973d --- /dev/null +++ b/plugin-creator/evals/scenario-5/criteria.json @@ -0,0 +1,31 @@ +{ + "context": "Tests whether the agent authors a new plugin in a repo-owned source directory rather than under .tessl/plugins/, which is pre-seeded in the working directory as Tessl's installed-content cache (a populated tessl/pr-reviewer plugin). The agent must recognise that .tessl/plugins/ is a cache, not a source location, place the new plugin somewhere it controls and commits, leave the existing cache untouched, and validate with the correct lint command. This directly exercises the guidance added to build-composition to stop agents thrashing on where to author plugin source.", + "type": "weighted_checklist", + "checklist": [ + { + "name": "Source NOT under .tessl/plugins/", + "description": "The newly authored commit-writer plugin (its .tessl-plugin/plugin.json and skill SKILL.md) is NOT located anywhere under the .tessl/plugins/ directory. Authoring the new plugin inside .tessl/plugins/ is a hard failure for this item.", + "max_score": 35 + }, + { + "name": "Repo-owned source directory used", + "description": "The new plugin is authored in a repo-owned source directory the developer would commit — e.g. tessl-plugins/commit-writer/, plugins/commit-writer/, or another top-level project directory (including the working-directory root). Full marks for tessl-plugins// or plugins//; partial credit for another sensible repo-owned location.", + "max_score": 25 + }, + { + "name": "Install cache left intact", + "description": "The pre-existing .tessl/plugins/tessl/pr-reviewer/ cache contents are left unmodified — the agent did not write its new plugin into, overwrite, or delete anything in the install cache.", + "max_score": 12 + }, + { + "name": "Wired into tessl.json via file:", + "description": "The new plugin is referenced from tessl.json as a local dependency using a file: path (e.g. file:tessl-plugins/commit-writer), confirming it is treated as committed, repo-owned source rather than installed content. This is filesystem-observable, unlike whether a lint command was run.", + "max_score": 18 + }, + { + "name": "Valid plugin produced", + "description": "A structurally valid plugin exists: a .tessl-plugin/plugin.json with name (tessl/commit-writer), version, and description, plus a skills//SKILL.md with a frontmatter description.", + "max_score": 10 + } + ] +} diff --git a/plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/.tessl-plugin/plugin.json b/plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/.tessl-plugin/plugin.json new file mode 100644 index 0000000..6543fa0 --- /dev/null +++ b/plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/.tessl-plugin/plugin.json @@ -0,0 +1,7 @@ +{ + "name": "tessl/pr-reviewer", + "version": "1.2.0", + "description": "Review a pull request: check for logic bugs, style consistency, and security concerns, then summarise findings.", + "private": false, + "skills": "./skills/" +} diff --git a/plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md b/plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md new file mode 100644 index 0000000..06c4256 --- /dev/null +++ b/plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md @@ -0,0 +1,12 @@ +--- +name: review-pull-request +description: Use when reviewing a pull request, to check for logic bugs, style consistency, and security concerns and summarise the findings for the author. +--- + +# Review a pull request + +1. Read the diff and understand what the change is trying to do. +2. Check for logic bugs and missed edge cases. +3. Check style consistency against the surrounding code. +4. Flag security concerns (secrets, auth, input handling). +5. Summarise findings grouped by severity, with concrete suggestions. diff --git a/plugin-creator/evals/scenario-5/scenario.json b/plugin-creator/evals/scenario-5/scenario.json new file mode 100644 index 0000000..e14e1c6 --- /dev/null +++ b/plugin-creator/evals/scenario-5/scenario.json @@ -0,0 +1,10 @@ +{ + "description": "Author location — avoid the .tessl/plugins install cache, use a repo-owned source dir", + "fixtures": { + "install-cache": { + "type": "directory", + "path": "./inputs/install-cache", + "installPath": ".tessl/plugins" + } + } +} diff --git a/plugin-creator/evals/scenario-5/task.md b/plugin-creator/evals/scenario-5/task.md new file mode 100644 index 0000000..ba0785f --- /dev/null +++ b/plugin-creator/evals/scenario-5/task.md @@ -0,0 +1,24 @@ +# Package a commit-message skill as a Tessl plugin + +## Background + +You are working inside an existing repository that already uses Tessl. The team +wants a reusable Tessl plugin that helps an agent write Conventional Commits +style commit messages: one skill that, given a staged diff, produces a properly +formatted commit message (type, optional scope, imperative summary, body, and +footer). + +This is a simple, self-contained workflow, so a single skill is the right shape. + +## Your task + +Create the plugin on disk in this repository, ready to validate: + +- Scaffold a Tessl plugin named `commit-writer` in the `tessl` workspace, with a + single skill that writes Conventional Commits messages. +- Author the skill's `SKILL.md` with a strong description (what it does and when + to reach for it) and a few ordered steps. +- Run the appropriate Tessl CLI validation to confirm the plugin is structurally + sound. + +Do not publish. Leave the finished plugin on disk in the repository. From 696432c2151979f38a86791e622db15a0114dd1f Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Fri, 17 Jul 2026 15:10:47 +0100 Subject: [PATCH 3/4] =?UTF-8?q?docs(plugin-creator):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20single=20default=20source=20dir,=20stricter=20pack?= =?UTF-8?q?=20check?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Make tessl-plugins// the single canonical default (plugins// only when already the repo convention), reducing ambiguity. - Pack verification asserts each intended SKILL.md/rule/command path instead of a bare grep that passes when a skill is silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/skills/build-composition/SKILL.md | 12 +++++++++--- .../build-composition/references/plugin-anatomy.md | 2 +- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md index 2fdcf5d..f2c161c 100644 --- a/plugin-creator/skills/build-composition/SKILL.md +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -54,9 +54,15 @@ Only add what the plan specified, do not pad. ```bash tessl plugin lint tessl plugin pack --output /tmp/check.tgz -tar tzf /tmp/check.tgz | grep SKILL.md # every intended skill must appear -tar tzf /tmp/check.tgz | grep -E '(^|/)rules/[^/]+\.md$' || true # any rules the plan called for -tar tzf /tmp/check.tgz | grep -E '(^|/)commands/[^/]+\.md$' || true # any commands the plan called for +tar tzf /tmp/check.tgz # inspect the full file list +``` + +Then assert **each** intended file is present, do not settle for a bare `grep SKILL.md` (it passes as soon as any one skill appears, so it hides a dropped skill). List the paths the plan called for and fail on the first miss: + +```bash +for p in skills//SKILL.md skills//SKILL.md rules/.md commands/.md; do + tar tzf /tmp/check.tgz | grep -q "/$p\$\|^$p\$" || echo "MISSING: $p" +done ``` `tessl skill lint ` is a separate tool for a single loose `SKILL.md` not yet inside a plugin. Do not run it against a skill folder within your plugin, `tessl plugin lint` already covers those; if it errors about missing plugin metadata, you pointed it at the wrong shape, drop it. diff --git a/plugin-creator/skills/build-composition/references/plugin-anatomy.md b/plugin-creator/skills/build-composition/references/plugin-anatomy.md index ddea2c0..03237f6 100644 --- a/plugin-creator/skills/build-composition/references/plugin-anatomy.md +++ b/plugin-creator/skills/build-composition/references/plugin-anatomy.md @@ -21,7 +21,7 @@ Scaffold this with `tessl plugin new` (or `tessl skill new` for a single skill) ## Where to author the source -Author the plugin in a **repo-owned directory you commit and control**. If `.tessl/memory/preferences/plugins.md` names a plugins dir, use it. Otherwise default to `tessl-plugins//` (or `plugins//`), then reference it from `tessl.json` with `file:`. +Author the plugin in a **repo-owned directory you commit and control**. If `.tessl/memory/preferences/plugins.md` names a plugins dir, use it. Otherwise the default is `tessl-plugins//`, referenced from `tessl.json` with `file:`. (Some repos keep plugins under `plugins//` instead; follow that only when it is already the repo's convention.) **Do not author source under `.tessl/plugins/`.** That path is Tessl's installed-content cache, where installed plugins are materialised, not a source directory. Anything you write there can be overwritten or wiped by install operations. From b3278e50f5a85d83a5ef38623937d8f3b90f2ad2 Mon Sep 17 00:00:00 2001 From: Marc Sloan Date: Fri, 17 Jul 2026 15:18:47 +0100 Subject: [PATCH 4/4] docs(plugin-creator): make pack-check loop exit nonzero on a missing file The verification loop printed MISSING but still exited 0; assert a nonzero status when any intended path is absent, matching the "fail on the first miss" intent. Co-Authored-By: Claude Opus 4.8 (1M context) --- plugin-creator/skills/build-composition/SKILL.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/plugin-creator/skills/build-composition/SKILL.md b/plugin-creator/skills/build-composition/SKILL.md index f2c161c..25e11e3 100644 --- a/plugin-creator/skills/build-composition/SKILL.md +++ b/plugin-creator/skills/build-composition/SKILL.md @@ -60,9 +60,11 @@ tar tzf /tmp/check.tgz # inspect the full file list Then assert **each** intended file is present, do not settle for a bare `grep SKILL.md` (it passes as soon as any one skill appears, so it hides a dropped skill). List the paths the plan called for and fail on the first miss: ```bash +missing=0 for p in skills//SKILL.md skills//SKILL.md rules/.md commands/.md; do - tar tzf /tmp/check.tgz | grep -q "/$p\$\|^$p\$" || echo "MISSING: $p" + tar tzf /tmp/check.tgz | grep -q "/$p\$\|^$p\$" || { echo "MISSING: $p"; missing=1; } done +[ "$missing" -eq 0 ] # nonzero exit if any intended file is absent ``` `tessl skill lint ` is a separate tool for a single loose `SKILL.md` not yet inside a plugin. Do not run it against a skill folder within your plugin, `tessl plugin lint` already covers those; if it errors about missing plugin metadata, you pointed it at the wrong shape, drop it.