Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ When performing tasks in this repository, always do the following:
- Prefer the smallest change that satisfies the task.
- Do not run expensive, provider-backed, or full benchmark commands without explicit approval.
- Keep changes small, public-safe, and reviewable.
- Preserve Quantiles as local-first, offline system by default.
- Preserve Quantiles as a local-first, offline system by default.
- Verify commands, links, package names, benchmark and evaluation names, and release status before documenting them.
- Update public docs when CLI behavior, SDK APIs, workflows, benchmarks, schemas, setup steps, or agent guidance changes.
- Prefer concrete examples with commands, file paths, inputs, outputs, and expected behavior.
Expand All @@ -60,7 +60,7 @@ Keep root-level changes focused on public orientation, documentation, contributi

Do not silently change evaluation semantics. Changes to prompts, datasets, scorers, rubrics, metrics, sampling parameters, judge configuration, model selection, tool configuration, or step inputs can invalidate comparisons. Call out any such changes in handoff.

Update documentation in this repository (e.g. `README.md`, etc...) when any of the following change:
Update the relevant documentation in this repository, such as `README.md`, when any of the following change:

- CLI commands, flags, outputs, or setup steps.
- APIs, SDKs, imports, examples, or package names.
Expand All @@ -85,7 +85,7 @@ When editing the Python SDK subdirectory, preserve `async` behavior, stable JSON

### `typescript/` directory

>**Not released**. The TypeScript SDK is currently unsupported and unreleased.
> **Not released**. The TypeScript SDK is currently unsupported and unreleased.

The `typescript/` directory contains the Quantiles TypeScript SDK, which allows users to author custom local AI evals with the Quantiles stack using TypeScript. It exposes workflow primitives such as `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types.

Expand All @@ -105,16 +105,16 @@ Provider-backed model inputs should use provider-prefixed model names, for examp
- `anthropic:<model>`
- `gemini:<model>`

## Safety, Security And Privacy
## Safety, Security, and Privacy

Preserve Quantiles as local-first infrastructure. Follow the below guidelines to ensure the project maintains safety, security and privacy:
Preserve Quantiles as local-first infrastructure. Follow the guidelines below to ensure the project maintains safety, security, and privacy:

- The CLI and local server should store Quantiles state locally by default
- Evaluation workflows may call remote model providers, hosted judges, datasets, APIs, or external tools only when explicitly configured by the user
- Do not inspect, print, summarize, commit, or infer values from `.env` or `.envrc` files, secrets, tokens, private datasets, PHI, customer data, or local Quantiles databases unless the user explicitly asks and the data is safe to inspect.
- Never commit the `.quantiles/` directory, SQLite or metrics databases, local traces, benchmark outputs, provider credentials, or temporary run artifacts.
- Use placeholder names such as `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `QUANTILES_API_KEY` when examples need credentials.
- Keep public docs customer-safe. Avoid exposing secrets, API keys and non-public security information.
- Keep public docs customer-safe. Avoid exposing secrets, API keys, and non-public security information.

Read [`SECURITY.md`](./SECURITY.md) for more details on keeping this repository and toolchain secure.

Expand All @@ -140,7 +140,7 @@ When remote model calls, hosted judges, external tools, provider APIs, or networ

## Output Style For Coding Agents

When adding documentation to files in this repository, follow the below guidelines:
When adding documentation to files in this repository, follow the guidelines below:

- Be concise, technical, and action-oriented.
- Prefer runnable commands, real file paths, concrete inputs, and expected outputs.
Expand Down
66 changes: 22 additions & 44 deletions CONFIG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ You need a config file when you want to do one or more of the following:
- Define custom evaluations (`type = "custom_code"`)
- Resume custom evaluations later with `qt resume <run_id>`

You don't, however, need any configuration when you want to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box.
You do not need a configuration file to run built-in benchmarks. For example, `qt run pubmedqa` works out of the box.

## File location and name

If you want to make a configuration file, create either a `quantiles.toml` or `.quantiles.toml` file in the current working directory. You can only have one or the other -- if both exist, the CLI exits with an ambiguity error.
To configure Quantiles, create either `quantiles.toml` or `.quantiles.toml` in the current working directory. Use only one filename; if both files are present, the CLI exits with an ambiguity error.

## Top-level structure

Expand All @@ -23,33 +23,11 @@ Every benchmark lives under its own `[benchmarks.<eval_name>]` section. The sect
For example, if you want to override default parameters for the built-in PubMedQA benchmark, add the following section to your config file:

```toml
# This is configuration for the PubMedQA benchmark that is built into the `qt` CLI.
#
# When you run `qt run pubmedqa`, the CLI will look here for how exactly to run PubMedQA.
# Configure the model and sample limit for the built-in PubMedQA benchmark.

[benchmarks.pubmedqa]
# Limit to the first 50 samples of the benchmark
samples = 50
# Customize the model to test
model = "openai:gpt-5.4-nano"
# see below for the full reference
```

Alternatively, if you want to build your own custom evaluation using the Quantiles Python SDK, define that custom eval as follows:

```toml
# This is configuration for a custom code benchmark called my-eval.
#
# When you run `qt run my-eval`, the CLI will look here for how exactly to run your custom
# benchmark.
[benchmarks.my-eval]
# This `type` parameter defaults to "builtin". for custom evals, be sure to override
# the default with "custom_code" here.
type = "custom_code"
# This is the command that the CLI will execute to run your custom eval code. We recommend
# using the `uv` tool to configure and manage your Python evals, but this field can be any
# command, so you're free to use any tools you want.
command = ["uv", "run", "my_eval.py"]
# see below for the full reference
model = "openai:gpt-5.6"
```

## Benchmark types
Expand All @@ -60,12 +38,12 @@ Every benchmark section has a `type` field. Valid values are `"builtin"` (defaul

Built-in benchmarks run natively inside the CLI, without any custom code. Below is a list of parameters that can be customized for built-in benchmarks:

| Field | Type | Required | Description |
|--|--|--|--|
| `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. |
| `samples` | integer | no | Number of dataset rows to evaluate. |
| `model` | string or table | no | Model sampler. See [model format](#model-format). |
| `max_workers` | integer | no | Maximum concurrent workers. |
| Field | Type | Required | Description |
| ------------- | --------------- | -------- | ---------------------------------------------------------------- |
| `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. |
| `samples` | integer | no | Number of dataset rows to evaluate. |
| `model` | string or table | no | Model sampler. See [model format](#model-format). |
| `max_workers` | integer | no | Maximum concurrent workers. |

If none of these fields are customized, the built-in benchmark uses the following defaults:

Expand All @@ -76,10 +54,10 @@ If none of these fields are customized, the built-in benchmark uses the followin

#### `model` naming

The above `model` field accepts a provider-prefixed string, for example
The `model` field described above accepts a provider-prefixed string, for example:

```toml
model = "openai:gpt-5.4-nano"
model = "openai:gpt-5.6"
```

Supported provider prefixes are listed below:
Expand All @@ -92,7 +70,7 @@ Supported provider prefixes are listed below:
You can pass a TOML table instead of such a prefixed string:

```toml
model = { provider = "openai", model_id = "gpt-5.4-nano" }
model = { provider = "openai", model_id = "gpt-5.6" }
```

Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in [`cli/examples/configs`](./cli/examples/configs).
Expand All @@ -101,13 +79,13 @@ Note that models require specific configuration based on the provider. For detai

Custom evaluations are external programs built with the Quantiles Python SDK. Their config sections contain the following fields:

| Field | Type | Required | Description |
|--|--|--|--|
| `type` | string | yes | Must be `"custom_code"`. |
| `command` | array of strings | yes | Command and arguments to execute. |
| `input` | table | no | Structured input passed to the child as `QUANTILES_INPUT`. |
| Field | Type | Required | Description |
| --------- | ---------------- | -------- | ---------------------------------------------------------- |
| `type` | string | yes | Must be `"custom_code"`. |
| `command` | array of strings | yes | Command and arguments to execute. |
| `input` | table | no | Structured input passed to the child as `QUANTILES_INPUT`. |

Note that custom code evaluations can customize the model in code. See the [PubMedQA custom code example](./python/examples/pubmedqa.py) for details on customizing the model in `custom_code` benchmarks.
Note that custom code evaluations can customize the model in code. See the [PubMedQA custom code example](./python-examples/src/pubmedqa.py) for details on customizing the model in `custom_code` benchmarks.

#### The `input` table

Expand Down Expand Up @@ -178,10 +156,10 @@ When you run `qt resume <run_id>`, the CLI looks up the stored eval name and inp

```toml
[benchmarks.pubmedqa]
model = "openai:gpt-5.4-nano"
model = "openai:gpt-5.6"
```

### Built-in using demo model, and restricting samples
### Built-in using the demo model with a sample limit

```toml
[benchmarks.simpleqa-verified]
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing to Quantiles

Thank you for interest in contributing to Quantiles Open source!
Thank you for your interest in contributing to Quantiles open source!

Quantiles open source projects provide local-first evaluation infrastructure for AI systems. We welcome contributions that improve reliability, reproducibility, developer ergonomics, agent workflows, documentation, and benchmark and evaluation support.

Expand Down Expand Up @@ -39,7 +39,7 @@ Please do not report security vulnerabilities through public GitHub issues. Foll

## Code of Conduct

By participating in this project, you agree to follow our [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md)
By participating in this project, you agree to follow our [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md).

## License

Expand Down
Loading
Loading