diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index 97d0cfc2a..92e3eb731 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -28,7 +28,9 @@ export default defineConfig({ }, { text: "Workspace Refine", link: "/refine" }, { text: "Evals", link: "/test" }, + { text: "Optimize Description", link: "/optimize-description" }, { text: "Settings", link: "/settings" }, + { text: "Documents", link: "/documents" }, { text: "Settings Usage", link: "/usage" }, ], }, diff --git a/docs/user-guide/dashboard.md b/docs/user-guide/dashboard.md index e099c21ff..ab861830b 100644 --- a/docs/user-guide/dashboard.md +++ b/docs/user-guide/dashboard.md @@ -38,7 +38,7 @@ When you select a skill on the dashboard route, the main area becomes a workspac - **Overview** - **Refine** - **Evals** -- **Description** (currently disabled) +- **Optimize Description** Imported skills can be viewed in the workspace shell, but builder-only actions such as Refine are not always available. @@ -69,7 +69,7 @@ There is no grid/list toggle or multi-filter bar in the current dashboard UI. - Click an in-progress builder skill to open its [workflow](workflow/overview.md). - Click a completed skill to open its workspace shell on the dashboard route. -The workspace shell is the current home for **Overview**, **Refine**, and **Evals**. +The workspace shell is the current home for **Overview**, **Refine**, **Evals**, and **Optimize Description**. --- @@ -89,8 +89,10 @@ The menu is grouped into sections: **Skill** - **Overview** — open the workspace Overview tab +- **Eval** — open the workspace Evals tab - **Refine** — open the workspace Refine tab - **Restore version** — restore an earlier saved version +- **Export as .skill** — save the skill as a `.skill` package **Plugin** (hidden for marketplace skills) @@ -114,8 +116,10 @@ These show: Uploaded skills use the same workspace shell. Their menu can include: - **Overview** -- **Refine** (opens the tab, but Refine itself is not available for uploaded skills) +- **Refine** +- **Eval** - **Restore version** +- **Export as .skill** - **Create plugin** when the skill is still in the default **Skills** plugin - **Remove from plugin** when the skill is in a non-default plugin - **Move to plugin** when another eligible plugin exists @@ -126,7 +130,8 @@ Uploaded skills use the same workspace shell. Their menu can include: Marketplace skills also use the workspace shell. Their menu can include: - **Overview** -- **Refine** (opens the tab, but Refine itself is not available for marketplace skills) +- **Refine** +- **Eval** - **Restore version** - **Delete** when the skill is in the default plugin @@ -148,6 +153,12 @@ For marketplace skills, the source appears as the marketplace URL when one is re For uploaded skills, the source appears as **Uploaded**. +## Evals and Optimize Description tabs + +Use **Evals** to define test scenarios, run selected evals, review benchmark results, and send failing results to Refine. + +Use **Optimize Description** to tune the skill description that controls when Claude should trigger the skill. See [Evals](test.md) and [Optimize Description](optimize-description.md) for the full workflows. + --- ## Empty and locked states diff --git a/docs/user-guide/documents.md b/docs/user-guide/documents.md new file mode 100644 index 000000000..d6dd49b4b --- /dev/null +++ b/docs/user-guide/documents.md @@ -0,0 +1,77 @@ +# Documents + +Documents lives in **Settings → Documents**. Use it to add files, folders, or URLs as reference context for skill workflows. + +## What's on this screen + +The page shows a **Documents** heading, a short description, action buttons, and a table of added documents. + +The action buttons are: + +- **Upload file** +- **Add URL** +- **Add folder** + +The document table has these columns: + +- **Name** +- **Source** +- **Assigned to** +- **Added** + +## Upload a file + +1. Open **Settings → Documents**. +2. Click **Upload file**. +3. Choose a `.md`, `.txt`, or `.pdf` file. + +The document is added and assigned to **All skills** by default. + +## Add a URL + +1. Open **Settings → Documents**. +2. Click **Add URL**. +3. In **Add document from URL**, enter **Name**. +4. Enter **URL**. +5. Use **All skills** to decide whether the document applies to every skill or selected skills. +6. Click **Fetch & Add**. + +If **All skills** is off, select one or more skills from the list before adding the URL. + +## Add a folder + +1. Open **Settings → Documents**. +2. Click **Add folder**. +3. Choose a folder from the folder picker. + +The app adds supported documents from the folder and assigns them to **All skills** by default. + +## Change document assignment + +1. Click the value in **Assigned to** for a document. +2. In **Assign "document name"**, use **All skills** to switch between all-skill and selected-skill assignment. +3. If assigning to selected skills, check the skills that should receive the document. +4. Click **Save**. + +The assigned-to value can show **All skills**, **No skills**, a single skill name, or a count such as **3 skills**. + +## Delete a document + +Click the trash button on a document row to remove it. + +## What you'll see + +- **Loading** — **Loading documents…** +- **Empty state** — **No documents added yet. Upload a file or add a URL to get started.** +- **No skills found** — shown in the assignment picker when there are no skills to select. + +## Quick reference + +| Control | What it does | +|---|---| +| **Upload file** | Adds a `.md`, `.txt`, or `.pdf` file | +| **Add URL** | Fetches and adds a document from a URL | +| **Add folder** | Adds supported documents from a folder | +| **All skills** | Assigns the document to every skill when enabled | +| **Fetch & Add** | Saves a URL document | +| **Save** | Saves assignment changes | diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 237891af0..57bf38a8a 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -58,7 +58,9 @@ Two important areas are embedded inside those surfaces rather than being standal - [Plugins](plugins.md) explains how skills are grouped into plugins. - [Refine](refine.md) is a workspace tab on the dashboard for builder skills. -- [Evals](test.md) is also a workspace tab, but it is currently a placeholder. +- [Evals](test.md) is a workspace tab for creating and running skill evals. +- [Optimize Description](optimize-description.md) is a workspace tab for tuning when a skill should trigger. +- [Documents](documents.md) lives in **Settings → Documents**. - [Usage](usage.md) lives in **Settings → Usage**. --- diff --git a/docs/user-guide/optimize-description.md b/docs/user-guide/optimize-description.md new file mode 100644 index 000000000..a8812a24a --- /dev/null +++ b/docs/user-guide/optimize-description.md @@ -0,0 +1,84 @@ +# Optimize Description + +The **Optimize Description** workspace tab helps tune the description that decides when Claude should trigger a skill. + +Use it when a skill is triggered too often, not triggered when it should be, or has a description that is too broad or too vague. + +## What's on this screen + +The page has three main areas: + +- **Trigger Eval Queries** — example requests split into **Should Trigger** and **Should Not Trigger** +- **Optimize Description** — runs the optimization loop +- **Results** — shows score progression, the before/after description, and actions for the best result + +## Generate trigger eval queries + +1. Open a skill and switch to **Optimize Description**. +2. Click **Generate** in **Trigger Eval Queries**. +3. In **Generate Eval Queries**, enter the **Number of queries**. The minimum is 10 and the recommended value is 20. +4. Click **Generate**. + +While queries are generating, the dialog shows **Generating Eval Queries** and the agent output panel. Click outside the dialog or press `Esc` to open **Stop generating?**. + +## Edit trigger eval queries + +Each query belongs in one of two columns: + +- **Should Trigger** — requests that should invoke the skill +- **Should Not Trigger** — requests that should not invoke the skill + +You can: + +1. Edit any query text directly. +2. Use the **Should trigger** switch to move a query between the two columns. +3. Click **Add query** to add a row. +4. Click the trash button to delete a row. + +The tab saves query edits after at least one query exists. + +## Run optimization + +1. Make sure at least one query is in **Should Trigger**. +2. Click **Optimize**. +3. Watch **Iteration**, **Train score**, **Test score**, and **Best so far** as the run progresses. + +The run can take several iterations. Click **Cancel** to stop it. + +If every query is marked as not triggering, the page shows **Enable at least one query to run optimization.** + +## Apply the best description + +When optimization completes, **Results** shows: + +- **Score Progression** +- **Description diff — original vs best** +- **Before (Original)** +- **After (Best)** + +Click **Apply best description** to save the recommended description. A success message appears: **Description applied successfully.** + +Click **Discard** to close the result without applying it. + +## What you'll see + +- **Empty query state** — **No queries yet. Generate or add them manually.** +- **Generation running** — **Generating Eval Queries** with live agent output. +- **Optimization running** — **Iteration N / 5** and **Running 3x eval queries on iteration N description…** +- **Cancel generation guard** — **Stop generating?** with **Continue generating** and **Stop**. +- **Navigation guard** — **Optimization In Progress** with **Stay** and **Leave** when you try to leave during an optimization run. + +## Quick reference + +| Control | What it does | +|---|---| +| **Generate** | Generates trigger eval queries | +| **Number of queries** | Sets how many queries to generate | +| **Should Trigger** | Requests that should invoke the skill | +| **Should Not Trigger** | Requests that should not invoke the skill | +| **Should trigger** | Moves a query between trigger and non-trigger groups | +| **Add query** | Adds another query row | +| **Optimize** | Starts description optimization | +| **Cancel** | Stops an optimization run | +| **Apply best description** | Saves the best generated description | +| **Discard** | Closes the optimization result without applying it | diff --git a/docs/user-guide/settings.md b/docs/user-guide/settings.md index 12a89d4ed..40e07d3c7 100644 --- a/docs/user-guide/settings.md +++ b/docs/user-guide/settings.md @@ -10,6 +10,7 @@ The current sections are: - **Marketplace** - **GitHub** - **Advanced** +- **Documents** - **Usage** Most changes save immediately. @@ -129,6 +130,10 @@ The current Settings UI does not expose a workspace-folder reset control here. --- +## Documents + +Documents is a Settings section for adding files, URLs, and folders as reference context for skill workflows. See [Documents](documents.md). + ## Usage Usage is now a Settings section, not a separate top-level route. See [Usage](usage.md). diff --git a/docs/user-guide/test.md b/docs/user-guide/test.md index ad36e59f1..cdb5d4fc9 100644 --- a/docs/user-guide/test.md +++ b/docs/user-guide/test.md @@ -1,17 +1,86 @@ # Evals -The dashboard workspace currently includes an **Evals** tab for selected skills. +The **Evals** workspace tab lets you define test scenarios for a skill, run them, review benchmark results, and use failures to start a Refine request. -At the moment, that tab is a placeholder and shows: +You can reach it from: -`Evals coming soon.` +- **More actions → Eval** on a skill in the dashboard skill list +- the **Evals** tab inside the selected skill's workspace shell +- **Eval** after Step 4 finishes in the workflow ---- +## What's on this screen -## What this means +The top section is **Evals**. It lists saved evals from the selected skill and shows each eval's **Name**, **Prompt**, and assertion count. -- There is no working side-by-side evaluator flow in the current shipped UI. -- The older standalone “Test” experience described in earlier docs is not available. -- The workflow’s final **Eval** button can navigate you to this workspace tab, but the tab itself is still placeholder-only. +When evals exist, the run controls appear above the list: -Until the feature ships, use [Refine](refine.md) and the workflow review surfaces for iteration. +- selection checkbox +- **None** / **vs Baseline** +- **1×** / **3×** +- **Run selected (N)** + +The lower sections appear after runs: + +- live agent output while evals are running +- benchmark results after a run completes +- **Iteration History** for previous runs + +## Generate an eval + +1. Open the skill and switch to **Evals**. +2. Click **Generate eval**. +3. In **Generate eval**, answer **What do you want to evaluate?** +4. Click **Generate**. +5. Review the generated eval in **Review Generated Eval**. +6. Edit **Name**, **Prompt**, and **Expectations** if needed. +7. Click **Add**. + +The generation dialog shows **Generating eval...** while the agent reads the skill definition and drafts the scenario. Click **Cancel** to stop before the eval is created. + +## Edit or delete an eval + +1. Open **Evals**. +2. Click the pencil button on a row to open **Edit Eval**. +3. Update **Name**, **Prompt**, or **Expectations**. +4. Click **Save changes**. + +To remove an eval, click the trash button, then confirm **Delete** in **Delete eval?**. + +## Run evals + +1. Select one or more evals with the checkboxes. +2. Choose **None** for a normal run, or **vs Baseline** to compare against a no-skill baseline. +3. Choose **1×** or **3×**. +4. Click **Run selected (N)**. + +While the run is active, the button changes to **Running…** and the page shows **Running evals — grading results appear below as they complete**. + +## Review results and refine + +After a run completes, the benchmark card summarizes the result. If assertions fail, the page shows **Refine skill**. + +Click **Refine skill** to send the failing eval context to the [Refine](refine.md) tab. If the eval run is still active, the app shows **Eval Run In Progress** and asks whether to **Stay** or **Cancel eval and refine**. + +## What you'll see + +- **Empty state** — **No evals yet** with **Generate your first eval**. +- **Loading** — **Loading evals…** while evals are being read. +- **Generation running** — **Generating eval for "..."…** and the agent output panel. +- **Run running** — **Running evals — grading results appear below as they complete**. +- **Navigation guard** — **Eval Run In Progress** with **Stay** and **Leave** when you try to navigate away during a run. + +## Quick reference + +| Control | What it does | +|---|---| +| **Generate eval** | Opens the intent dialog for a new eval | +| **Generate your first eval** | Starts eval generation from the empty state | +| **Name** | Human-readable eval name | +| **Prompt** | User request the skill should handle | +| **Expectations** | Assertions the result should satisfy | +| **Add expectation** | Adds another assertion field | +| **None** | Runs selected evals without a baseline comparison | +| **vs Baseline** | Compares skill output against a no-skill baseline | +| **1×** / **3×** | Controls how many times selected evals run | +| **Run selected (N)** | Starts the selected eval run | +| **Refine skill** | Opens Refine with failing eval context | diff --git a/docs/user-guide/workflow/step-4-generate-skill.md b/docs/user-guide/workflow/step-4-generate-skill.md index 3dc570e60..6c858d900 100644 --- a/docs/user-guide/workflow/step-4-generate-skill.md +++ b/docs/user-guide/workflow/step-4-generate-skill.md @@ -31,4 +31,4 @@ The action bar can show: | **Eval** | Opens the workspace **Evals** tab for this skill | | **Done** | Returns to the [Dashboard](../dashboard.md) | -The current **Evals** tab is placeholder-only. See [Evals](../test.md). +Use **Evals** to create scenarios, run selected evals, review benchmark results, and send failures to Refine. See [Evals](../test.md).