diff --git a/README.md b/README.md index 0d67935..8a517a7 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ This repository contains reusable coding-agent instructions for running Quantiles evaluations from a repository. It is designed for Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other coding agents that support reusable skills or instruction files. -This skill teaches coding agents a repeatable Quantiles workflow using the `qt` CLI. It covers local initialization, benchmark and custom eval runs, sample-level inspection, run comparison, resume behavior, and regression summaries. For a concise, public, LLM-readable overview of Quantiles with links to agent guides and related documentation, see [quantiles.io/llms.txt](https://quantiles.io/llms.txt). +This skill teaches coding agents a repeatable Quantiles workflow using the `qt` CLI. It covers local initialization, built-in benchmark runs, no-code QA benchmark configuration, custom code eval runs, sample-level inspection, run comparison, resume behavior, and regression summaries. For a concise, public, LLM-readable overview of Quantiles with links to agent guides and related documentation, see [quantiles.io/llms.txt](https://quantiles.io/llms.txt). ## What the skill does @@ -10,9 +10,9 @@ This skill teaches coding agents to do the following: - Run evaluations through `qt`. - Preserve run IDs, commands, inputs, metrics, stdout, stderr, and failure context. -- Use `qt run $BENCHMARK` to run built-in benchmarks and custom eval workflows. -- Use `qt show $RUN_ID` to inspect run results. -- Use `qt compare $RUN_ID_A $RUN_ID_B` to analyze changes between runs. +- Use `qt run ` to run built-in benchmarks, no-code QA benchmarks, and custom eval workflows. +- Use `qt show ` to inspect run results. +- Use `qt compare ` to analyze changes between runs. - Use `--json` to inspect structured result outputs. - Use `qt resume` when a run is interrupted or partially completed. - Report aggregate metrics, sample-level results, failed samples, regressions, and next steps. diff --git a/SKILL.md b/SKILL.md index d54f64d..30c49fa 100644 --- a/SKILL.md +++ b/SKILL.md @@ -5,7 +5,7 @@ description: Use when writing, running, inspecting, comparing, resuming, or anal # Quantiles eval workflows -Use this skill for Quantiles AI evaluation work. The `qt` CLI is the canonical entrypoint for running benchmarks and evaluations, inspecting run results, comparing runs, resuming interrupted evals, and executing custom Python eval workflows. +Use this skill for Quantiles AI evaluation work. The `qt` CLI is the canonical entrypoint for running benchmarks and evaluations, configuring evaluations and benchmarks, inspecting run results, comparing runs, resuming interrupted evals, and executing custom Python eval workflows. Prefer `qt` CLI commands over manually reading local Quantiles storage files unless the CLI output is insufficient. Never modify local Quantiles storage files, except via `qt` CLI commands. @@ -16,6 +16,7 @@ Always pass `--json` when using `qt run`, `qt resume`, `qt list`, `qt show`, or Use this skill when the user asks to: - Run and/or configure a built-in Quantiles benchmark (e.g. `pubmedqa`, `simpleqa-verified`) +- Run and/or configure a no-code QA benchmark with `type = "custom_nocode"` - Run and/or configure a custom code Quantiles evaluation - Inspect and analyze a Quantiles eval run - Compare two Quantiles eval runs @@ -127,7 +128,7 @@ qt run --json For initial eval validation, prefer a small local smoke test: ```bash -qt run simpleqa-verified --json --input '{"samples": 10"}' +qt run simpleqa-verified --json --input '{"limit":10}' ``` If the test succeeds, and the user approves running the full dataset, remove the `--input` parameter to run the complete benchmark: @@ -140,12 +141,12 @@ The above examples use a demo model, which, as described above, just generates r ### Limiting sample count -For built-in evals, pass a JSON `samples` key through `--input`. +For built-in evals, pass a JSON `limit` key through `--input`. In `quantiles.toml`, the equivalent built-in config field is `samples`. Example: ```bash -qt run simpleqa-verified --input '{"samples":100}' --json +qt run simpleqa-verified --input '{"limit":100}' --json ``` Use small limits for smoke tests and larger limits only when the user wants a more complete benchmark run. @@ -160,6 +161,54 @@ Before running a provider-backed eval, follow the credential checks in the "Secr After running, report whether the run used the demo model or a real provider-backed model. +## No-code QA benchmarks + +Use this section when the user asks to configure a dataset-backed QA benchmark without writing custom Python or TypeScript evaluation code. + +No-code QA benchmarks are configured in `quantiles.toml` or `.quantiles.toml` with `type = "custom_nocode"` and `style = "qa"`. They run inside the `qt` CLI, render each prompt with a Jinja template file, call the configured model, and score each row with exact-match accuracy against the configured golden answer column after trimming whitespace. + +Create a config like this from the user's project root: + +```toml +[benchmarks.nocode_custom] +type = "custom_nocode" +style = "qa" +dataset = "quantiles/simpleqa-verified" +model = "random" +prompt_template_file = "prompts/qa.txt" +prompt_column = "problem" +golden_column = "answer" +limit = 10 +``` + +Create the prompt template at the path referenced by `prompt_template_file`. The template receives a `prompt` variable from `prompt_column`: + +```txt +{{ prompt }} +``` + +Run the benchmark with: + +```bash +qt run nocode_custom --json +``` + +Then inspect the run: + +```bash +qt show --json +``` + +When configuring or reviewing a no-code QA benchmark: + +- Confirm `prompt_template_file` exists before running. +- Confirm `prompt_column` and `golden_column` match columns in the dataset. +- Use `limit` for smoke tests before running a larger sample. +- Do not create a Python SDK eval for this path unless the user asks for custom code. +- Treat `model = "random"` or an omitted model as a demo run, not model-quality evidence. +- If the user provides a provider-backed model, follow the credential checks in the "Secrets and cost safety" section before running. +- Report exact-match accuracy, `correct_count`, `total_count`, run ID, model, dataset, prompt template path, and sample count when summarizing results. + ## Custom evals Use this section only when the user asks to write or modify a custom Quantiles eval. @@ -356,7 +405,7 @@ When comparing, report the following information, if available, along with any o ## Resuming interrupted runs -If an eval failed or was interrupted for any reason, resume it from where it left off instead of restarting the run from the beginning. Do so with the below command: +If an eval failed or was interrupted for any reason, resume it from where it left off instead of restarting the run from the beginning. Do so with the following command: ```bash qt resume --json