docs(plugin-creator): reduce agent thrash in build guidance (OPT-1590)#19
Conversation
…OPT-1590) Address feedback from running plugin-creator, where ambiguous/outdated guidance led to avoidable thrashing: - Warn against authoring source under .tessl/plugins/ (install cache) and recommend a repo-owned default (tessl-plugins/<name>/) referenced via file:. - Clarify tessl plugin lint as the primary structural check, and when the separate tessl skill lint applies. - Update review examples to current CLI: tessl review run quality|security. - Note --workspace for non-interactive review runs to avoid the interactive workspace prompt hanging. Bump plugin-creator to 0.1.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe plugin creator now documents repository-owned plugin source locations, strengthens lint-and-pack verification, clarifies standalone skill linting, and updates workspace-aware security review instructions. Scenario 5 adds an evaluation workflow for creating a local commit-writer plugin while preserving installed cache contents. ChangesPlugin creator guidance and evaluation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugin-creator/skills/build-composition/SKILL.md (1)
54-59: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMake the pack verification assert every intended file.
The current
grepchecks can pass when one skill or planned rule/command is missing, because they only search for any matching archive entry. Enumerate the expected paths from the plugin source and fail unless each one appears in the packed archive; otherwise this guidance can report success for an incomplete plugin.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugin-creator/skills/build-composition/SKILL.md` around lines 54 - 59, Update the pack verification commands in the build-composition instructions to enumerate every intended SKILL.md, rule, and command path from the plugin source, then assert each expected path appears in /tmp/check.tgz. Replace the current “any match” grep checks while preserving the existing lint and pack steps, and ensure a missing file causes verification to fail.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugin-creator/skills/build-composition/references/plugin-anatomy.md`:
- Around line 22-26: Make tessl-plugins/<plugin-name>/ the single documented
default source directory. In
plugin-creator/skills/build-composition/references/plugin-anatomy.md lines
22-26, remove or qualify plugins/<plugin-name>/ as an alternative used only when
explicitly configured; in plugin-creator/skills/build-composition/SKILL.md lines
12-13, preserve and align the same canonical authoring rule.
---
Outside diff comments:
In `@plugin-creator/skills/build-composition/SKILL.md`:
- Around line 54-59: Update the pack verification commands in the
build-composition instructions to enumerate every intended SKILL.md, rule, and
command path from the plugin source, then assert each expected path appears in
/tmp/check.tgz. Replace the current “any match” grep checks while preserving the
existing lint and pack steps, and ensure a missing file causes verification to
fail.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 85b306ca-79aa-42f3-92b6-f263fc2550f4
📒 Files selected for processing (3)
plugin-creator/.tessl-plugin/plugin.jsonplugin-creator/skills/build-composition/SKILL.mdplugin-creator/skills/build-composition/references/plugin-anatomy.md
…ion (OPT-1590) Seeds a populated .tessl/plugins/ install cache via a directory fixture, asks the agent to build a plugin without saying where, and scores whether it authors in a repo-owned dir vs the cache. Verified the guidance holds: baseline (no plugin) authored into .tessl/plugins/tessl/commit-writer/ (22/100); with the edited guidance injected, authored into tessl-plugins/commit-writer/ wired via file: (91/100). The .tessl/plugins/ trap moves 0 -> 35. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regression eval added and run (scenario-5)Added Ran it (
The baseline agent authored straight into * The recorded run scored this criterion via an earlier "was Eval run: |
…icter pack check - Make tessl-plugins/<name>/ the single canonical default (plugins/<name>/ only when already the repo convention), reducing ambiguity. - Pack verification asserts each intended SKILL.md/rule/command path instead of a bare grep that passes when a skill is silently dropped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md (1)
3-12: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueStandardise on American English spelling in Markdown documentation.
Based on learnings, Markdown documentation should consistently use American English spelling. Please consider updating "summarise" to "summarize" in both the description and the ordered steps.
♻️ Proposed fixes
-description: Use when reviewing a pull request, to check for logic bugs, style consistency, and security concerns and summarise the findings for the author. +description: Use when reviewing a pull request, to check for logic bugs, style consistency, and security concerns and summarize the findings for the author. --- # Review a pull request @@ -9,5 +9,5 @@ 2. Check for logic bugs and missed edge cases. 3. Check style consistency against the surrounding code. 4. Flag security concerns (secrets, auth, input handling). -5. Summarise findings grouped by severity, with concrete suggestions. +5. Summarize findings grouped by severity, with concrete suggestions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md` around lines 3 - 12, Update the Markdown documentation’s “summarise” occurrences in the description and ordered review steps to the American English spelling “summarize,” preserving the surrounding wording and structure.Sources: Learnings, Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugin-creator/skills/build-composition/SKILL.md`:
- Around line 60-65: Update the packed-path verification loop in the documented
tar check so a missing path causes immediate failure, rather than only printing
“MISSING”. Preserve the per-file checks for every planned path and make the loop
stop on the first missing file with a nonzero status.
---
Nitpick comments:
In
`@plugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.md`:
- Around line 3-12: Update the Markdown documentation’s “summarise” occurrences
in the description and ordered review steps to the American English spelling
“summarize,” preserving the surrounding wording and structure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 73fdf181-4967-42c2-8e7c-219afacac3ca
📒 Files selected for processing (7)
plugin-creator/evals/scenario-5/criteria.jsonplugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/.tessl-plugin/plugin.jsonplugin-creator/evals/scenario-5/inputs/install-cache/tessl/pr-reviewer/skills/review-pull-request/SKILL.mdplugin-creator/evals/scenario-5/scenario.jsonplugin-creator/evals/scenario-5/task.mdplugin-creator/skills/build-composition/SKILL.mdplugin-creator/skills/build-composition/references/plugin-anatomy.md
…file The verification loop printed MISSING but still exited 0; assert a nonzero status when any intended path is absent, matching the "fail on the first miss" intent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
@coderabbitai the packed-path check now exits nonzero on a missing file (b3278e5). On the "summarise → summarize" note: this repo uses British English spelling in its Markdown (existing plugin-creator skills use "honour", "materialise", "behaviour"). Please treat British spellings as correct for this repo and don't flag them. Leaving "summarise" as-is. Please re-review. |
|
✏️ Learnings added
If you found this review helpful, would you consider giving us a shout-out on X? Thank you for using CodeRabbit! ✅ Action performedReview finished.
|
There was a problem hiding this comment.
♻️ Duplicate comments (1)
plugin-creator/skills/build-composition/SKILL.md (1)
63-67: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMake the packed-path check fail on the first missing file.
missing=1makes the command exit nonzero eventually, but the loop continues checking later paths. This still does not implement the documented “fail on the first miss” behaviour.Suggested fix
-missing=0 for p in skills/<first>/SKILL.md skills/<second>/SKILL.md rules/<rule>.md commands/<cmd>.md; do - tar tzf /tmp/check.tgz | grep -q "/$p\$\|^$p\$" || { echo "MISSING: $p"; missing=1; } + tar tzf /tmp/check.tgz | grep -q "/$p\$\|^$p\$" || { + echo "MISSING: $p" >&2 + exit 1 + } done -[ "$missing" -eq 0 ]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugin-creator/skills/build-composition/SKILL.md` around lines 63 - 67, Update the packed-path validation loop so it exits immediately when a requested file is missing, rather than setting `missing=1` and continuing. Preserve the existing tar archive lookup and missing-file message, and ensure the overall check remains nonzero on the first miss.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@plugin-creator/skills/build-composition/SKILL.md`:
- Around line 63-67: Update the packed-path validation loop so it exits
immediately when a requested file is missing, rather than setting `missing=1`
and continuing. Preserve the existing tar archive lookup and missing-file
message, and ensure the overall check remains nonzero on the first miss.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 158fcc04-62d5-4d82-ae21-919f8ed04dcf
📒 Files selected for processing (1)
plugin-creator/skills/build-composition/SKILL.md
2 participants
Closes OPT-1590.
Sharpens
plugin-creatorbuild guidance after Alex saw the Tessl agent thrash ontesslcommands while running it. All command syntax verified against the live CLI;tessl plugin lint plugin-creatorpasses clean.Changes
.tessl/plugins/as a source dir (it's the installed-content cache) and recommend a repo-owned defaulttessl-plugins/<name>/, referenced viafile:<path>. —plugin-anatomy.md,build-composition§1tessl plugin lintis the primary structural check; clarify when the separatetessl skill lintapplies (a loose SKILL.md not yet in a plugin). —build-composition§5tessl review run quality|security <path>, replacing the staletessl review <path>/SKILL.mdexample. —build-composition§5--workspace <name>for non-interactivereview run, to avoid the interactive workspace selector hanging. —build-composition§5plugin-creatorto 0.1.2 (required byversion-checkCI; merge auto-publishes viapublish.yml).Testing
plugin lint,skill lint,review run quality|security,--workspace).tessl plugin lint plugin-creator→ valid.Evals — follow-up, not blocking
The current eval scenarios are output-artifact checks; they don't execute the CLI or inspect transcripts, so they can't assert "agent avoided
.tessl/plugins/" or "passed--workspace". Worth adding a regression scenario: a working dir with a pre-seeded.tessl/plugins/cache, scored on the agent authoring source in a repo-owned dir instead. Filing as a next step rather than blocking this doc fix.🤖 Generated with Claude Code
Summary by CodeRabbit
Documentation
New Features
Chores
0.1.1to0.1.2.