Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ export default defineConfig({
},
{ text: "Workspace Refine", link: "/refine" },
{ text: "Evals", link: "/test" },
{ text: "Optimize Description", link: "/optimize-description" },
{ text: "Settings", link: "/settings" },
{ text: "Documents", link: "/documents" },
{ text: "Settings Usage", link: "/usage" },
],
},
Expand Down
19 changes: 15 additions & 4 deletions docs/user-guide/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ When you select a skill on the dashboard route, the main area becomes a workspac
- **Overview**
- **Refine**
- **Evals**
- **Description** (currently disabled)
- **Optimize Description**

Imported skills can be viewed in the workspace shell, but builder-only actions such as Refine are not always available.

Expand Down Expand Up @@ -69,7 +69,7 @@ There is no grid/list toggle or multi-filter bar in the current dashboard UI.
- Click an in-progress builder skill to open its [workflow](workflow/overview.md).
- Click a completed skill to open its workspace shell on the dashboard route.

The workspace shell is the current home for **Overview**, **Refine**, and **Evals**.
The workspace shell is the current home for **Overview**, **Refine**, **Evals**, and **Optimize Description**.

---

Expand All @@ -89,8 +89,10 @@ The menu is grouped into sections:
**Skill**

- **Overview** — open the workspace Overview tab
- **Eval** — open the workspace Evals tab
- **Refine** — open the workspace Refine tab
- **Restore version** — restore an earlier saved version
- **Export as .skill** — save the skill as a `.skill` package

**Plugin** (hidden for marketplace skills)

Expand All @@ -114,8 +116,10 @@ These show:
Uploaded skills use the same workspace shell. Their menu can include:

- **Overview**
- **Refine** (opens the tab, but Refine itself is not available for uploaded skills)
- **Refine**
- **Eval**
- **Restore version**
- **Export as .skill**
- **Create plugin** when the skill is still in the default **Skills** plugin
- **Remove from plugin** when the skill is in a non-default plugin
- **Move to plugin** when another eligible plugin exists
Expand All @@ -126,7 +130,8 @@ Uploaded skills use the same workspace shell. Their menu can include:
Marketplace skills also use the workspace shell. Their menu can include:

- **Overview**
- **Refine** (opens the tab, but Refine itself is not available for marketplace skills)
- **Refine**
- **Eval**
- **Restore version**
- **Delete** when the skill is in the default plugin

Expand All @@ -148,6 +153,12 @@ For marketplace skills, the source appears as the marketplace URL when one is re

For uploaded skills, the source appears as **Uploaded**.

## Evals and Optimize Description tabs

Use **Evals** to define test scenarios, run selected evals, review benchmark results, and send failing results to Refine.

Use **Optimize Description** to tune the skill description that controls when Claude should trigger the skill. See [Evals](test.md) and [Optimize Description](optimize-description.md) for the full workflows.

---

## Empty and locked states
Expand Down
77 changes: 77 additions & 0 deletions docs/user-guide/documents.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Documents

Documents lives in **Settings → Documents**. Use it to add files, folders, or URLs as reference context for skill workflows.

## What's on this screen

The page shows a **Documents** heading, a short description, action buttons, and a table of added documents.

The action buttons are:

- **Upload file**
- **Add URL**
- **Add folder**

The document table has these columns:

- **Name**
- **Source**
- **Assigned to**
- **Added**

## Upload a file

1. Open **Settings → Documents**.
2. Click **Upload file**.
3. Choose a `.md`, `.txt`, or `.pdf` file.

The document is added and assigned to **All skills** by default.

## Add a URL

1. Open **Settings → Documents**.
2. Click **Add URL**.
3. In **Add document from URL**, enter **Name**.
4. Enter **URL**.
5. Use **All skills** to decide whether the document applies to every skill or selected skills.
6. Click **Fetch & Add**.

If **All skills** is off, select one or more skills from the list before adding the URL.

## Add a folder

1. Open **Settings → Documents**.
2. Click **Add folder**.
3. Choose a folder from the folder picker.

The app adds supported documents from the folder and assigns them to **All skills** by default.

## Change document assignment

1. Click the value in **Assigned to** for a document.
2. In **Assign "document name"**, use **All skills** to switch between all-skill and selected-skill assignment.
3. If assigning to selected skills, check the skills that should receive the document.
4. Click **Save**.

The assigned-to value can show **All skills**, **No skills**, a single skill name, or a count such as **3 skills**.

## Delete a document

Click the trash button on a document row to remove it.

## What you'll see

- **Loading** — **Loading documents…**
- **Empty state** — **No documents added yet. Upload a file or add a URL to get started.**
- **No skills found** — shown in the assignment picker when there are no skills to select.

## Quick reference

| Control | What it does |
|---|---|
| **Upload file** | Adds a `.md`, `.txt`, or `.pdf` file |
| **Add URL** | Fetches and adds a document from a URL |
| **Add folder** | Adds supported documents from a folder |
| **All skills** | Assigns the document to every skill when enabled |
| **Fetch & Add** | Saves a URL document |
| **Save** | Saves assignment changes |
4 changes: 3 additions & 1 deletion docs/user-guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,9 @@ Two important areas are embedded inside those surfaces rather than being standal

- [Plugins](plugins.md) explains how skills are grouped into plugins.
- [Refine](refine.md) is a workspace tab on the dashboard for builder skills.
- [Evals](test.md) is also a workspace tab, but it is currently a placeholder.
- [Evals](test.md) is a workspace tab for creating and running skill evals.
- [Optimize Description](optimize-description.md) is a workspace tab for tuning when a skill should trigger.
- [Documents](documents.md) lives in **Settings → Documents**.
- [Usage](usage.md) lives in **Settings → Usage**.

---
Expand Down
84 changes: 84 additions & 0 deletions docs/user-guide/optimize-description.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Optimize Description

The **Optimize Description** workspace tab helps tune the description that decides when Claude should trigger a skill.

Use it when a skill is triggered too often, not triggered when it should be, or has a description that is too broad or too vague.

## What's on this screen

The page has three main areas:

- **Trigger Eval Queries** — example requests split into **Should Trigger** and **Should Not Trigger**
- **Optimize Description** — runs the optimization loop
- **Results** — shows score progression, the before/after description, and actions for the best result

## Generate trigger eval queries

1. Open a skill and switch to **Optimize Description**.
2. Click **Generate** in **Trigger Eval Queries**.
3. In **Generate Eval Queries**, enter the **Number of queries**. The minimum is 10 and the recommended value is 20.
4. Click **Generate**.

While queries are generating, the dialog shows **Generating Eval Queries** and the agent output panel. Click outside the dialog or press `Esc` to open **Stop generating?**.

## Edit trigger eval queries

Each query belongs in one of two columns:

- **Should Trigger** — requests that should invoke the skill
- **Should Not Trigger** — requests that should not invoke the skill

You can:

1. Edit any query text directly.
2. Use the **Should trigger** switch to move a query between the two columns.
3. Click **Add query** to add a row.
4. Click the trash button to delete a row.

The tab saves query edits after at least one query exists.

## Run optimization

1. Make sure at least one query is in **Should Trigger**.
2. Click **Optimize**.
3. Watch **Iteration**, **Train score**, **Test score**, and **Best so far** as the run progresses.

The run can take several iterations. Click **Cancel** to stop it.

If every query is marked as not triggering, the page shows **Enable at least one query to run optimization.**

## Apply the best description

When optimization completes, **Results** shows:

- **Score Progression**
- **Description diff — original vs best**
- **Before (Original)**
- **After (Best)**

Click **Apply best description** to save the recommended description. A success message appears: **Description applied successfully.**

Click **Discard** to close the result without applying it.

## What you'll see

- **Empty query state** — **No queries yet. Generate or add them manually.**
- **Generation running** — **Generating Eval Queries** with live agent output.
- **Optimization running** — **Iteration N / 5** and **Running 3x eval queries on iteration N description…**
- **Cancel generation guard** — **Stop generating?** with **Continue generating** and **Stop**.
- **Navigation guard** — **Optimization In Progress** with **Stay** and **Leave** when you try to leave during an optimization run.

## Quick reference

| Control | What it does |
|---|---|
| **Generate** | Generates trigger eval queries |
| **Number of queries** | Sets how many queries to generate |
| **Should Trigger** | Requests that should invoke the skill |
| **Should Not Trigger** | Requests that should not invoke the skill |
| **Should trigger** | Moves a query between trigger and non-trigger groups |
| **Add query** | Adds another query row |
| **Optimize** | Starts description optimization |
| **Cancel** | Stops an optimization run |
| **Apply best description** | Saves the best generated description |
| **Discard** | Closes the optimization result without applying it |
5 changes: 5 additions & 0 deletions docs/user-guide/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The current sections are:
- **Marketplace**
- **GitHub**
- **Advanced**
- **Documents**
- **Usage**

Most changes save immediately.
Expand Down Expand Up @@ -129,6 +130,10 @@ The current Settings UI does not expose a workspace-folder reset control here.

---

## Documents

Documents is a Settings section for adding files, URLs, and folders as reference context for skill workflows. See [Documents](documents.md).

## Usage

Usage is now a Settings section, not a separate top-level route. See [Usage](usage.md).
87 changes: 78 additions & 9 deletions docs/user-guide/test.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,86 @@
# Evals

The dashboard workspace currently includes an **Evals** tab for selected skills.
The **Evals** workspace tab lets you define test scenarios for a skill, run them, review benchmark results, and use failures to start a Refine request.

At the moment, that tab is a placeholder and shows:
You can reach it from:

`Evals coming soon.`
- **More actions → Eval** on a skill in the dashboard skill list
- the **Evals** tab inside the selected skill's workspace shell
- **Eval** after Step 4 finishes in the workflow

---
## What's on this screen

## What this means
The top section is **Evals**. It lists saved evals from the selected skill and shows each eval's **Name**, **Prompt**, and assertion count.

- There is no working side-by-side evaluator flow in the current shipped UI.
- The older standalone “Test” experience described in earlier docs is not available.
- The workflow’s final **Eval** button can navigate you to this workspace tab, but the tab itself is still placeholder-only.
When evals exist, the run controls appear above the list:

Until the feature ships, use [Refine](refine.md) and the workflow review surfaces for iteration.
- selection checkbox
- **None** / **vs Baseline**
- **1×** / **3×**
- **Run selected (N)**

The lower sections appear after runs:

- live agent output while evals are running
- benchmark results after a run completes
- **Iteration History** for previous runs

## Generate an eval

1. Open the skill and switch to **Evals**.
2. Click **Generate eval**.
3. In **Generate eval**, answer **What do you want to evaluate?**
4. Click **Generate**.
5. Review the generated eval in **Review Generated Eval**.
6. Edit **Name**, **Prompt**, and **Expectations** if needed.
7. Click **Add**.

The generation dialog shows **Generating eval...** while the agent reads the skill definition and drafts the scenario. Click **Cancel** to stop before the eval is created.

## Edit or delete an eval

1. Open **Evals**.
2. Click the pencil button on a row to open **Edit Eval**.
3. Update **Name**, **Prompt**, or **Expectations**.
4. Click **Save changes**.

To remove an eval, click the trash button, then confirm **Delete** in **Delete eval?**.

## Run evals

1. Select one or more evals with the checkboxes.
2. Choose **None** for a normal run, or **vs Baseline** to compare against a no-skill baseline.
3. Choose **1×** or **3×**.
4. Click **Run selected (N)**.

While the run is active, the button changes to **Running…** and the page shows **Running evals — grading results appear below as they complete**.

## Review results and refine

After a run completes, the benchmark card summarizes the result. If assertions fail, the page shows **Refine skill**.

Click **Refine skill** to send the failing eval context to the [Refine](refine.md) tab. If the eval run is still active, the app shows **Eval Run In Progress** and asks whether to **Stay** or **Cancel eval and refine**.

## What you'll see

- **Empty state** — **No evals yet** with **Generate your first eval**.
- **Loading** — **Loading evals…** while evals are being read.
- **Generation running** — **Generating eval for "..."…** and the agent output panel.
- **Run running** — **Running evals — grading results appear below as they complete**.
- **Navigation guard** — **Eval Run In Progress** with **Stay** and **Leave** when you try to navigate away during a run.

## Quick reference

| Control | What it does |
|---|---|
| **Generate eval** | Opens the intent dialog for a new eval |
| **Generate your first eval** | Starts eval generation from the empty state |
| **Name** | Human-readable eval name |
| **Prompt** | User request the skill should handle |
| **Expectations** | Assertions the result should satisfy |
| **Add expectation** | Adds another assertion field |
| **None** | Runs selected evals without a baseline comparison |
| **vs Baseline** | Compares skill output against a no-skill baseline |
| **1×** / **3×** | Controls how many times selected evals run |
| **Run selected (N)** | Starts the selected eval run |
| **Refine skill** | Opens Refine with failing eval context |
2 changes: 1 addition & 1 deletion docs/user-guide/workflow/step-4-generate-skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,4 @@ The action bar can show:
| **Eval** | Opens the workspace **Evals** tab for this skill |
| **Done** | Returns to the [Dashboard](../dashboard.md) |

The current **Evals** tab is placeholder-only. See [Evals](../test.md).
Use **Evals** to create scenarios, run selected evals, review benchmark results, and send failures to Refine. See [Evals](../test.md).
Loading