From f314f6865c4eba8cdf2ef349cc62d19cafab09e7 Mon Sep 17 00:00:00 2001 From: phranzia <9207473+phranzia@users.noreply.github.com> Date: Fri, 10 Jul 2026 14:08:24 -0700 Subject: [PATCH 1/4] Minor doc updates --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 3fa33c9..f5f3735 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ It includes a CLI (called `qt`), SDKs, built-in benchmarks, local run history, a ## Why Quantiles -Evaluation workflows quickly outgrow one-off scripts once teams need caching, retries, dataset handling, metrics capture, and run comparison. Quantiles gives teams those primitives so they don't have to built them from scratch: +Evaluation workflows quickly outgrow one-off scripts once teams need caching, retries, dataset handling, metrics capture, and run comparison. Quantiles gives teams those primitives so they don't have to build them from scratch: - Write standard Python, with familiar developer patterns - Run workflows locally from the CLI @@ -117,12 +117,12 @@ Both the CLI and Python SDK support offline benchmark workflows, including the f ## Built-in Benchmarks -Built-in benchmarks are ready-to-run evalulations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark. +Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark. | Code | When to use | | --- | --- | -| `qt run $BENCHMARK` | Run a built-in benchmark against the demo model to inspect sample-level inputs and outputs, scoring behavior, workflow steps, and aggregate metrics | -| `qt run $BENCHMARK --input '{"model":"$MODEL_NAME"}'` | Run a built-in benchmark against your model | +| `qt run ` | Run a built-in benchmark against the demo model to inspect sample-level inputs and outputs, scoring behavior, workflow steps, and aggregate metrics | +| `qt run --input '{"model":"}'` | Run a built-in benchmark against your model | Quantiles also provides a [benchmark hub](https://quantiles.io/benchmark-hub) for discovering built-in benchmarks, understanding their evaluation setup, and reviewing common metrics used across AI evaluation workflows. From ae726cdeee87a3cfff8597a155da5509dbd0c9b5 Mon Sep 17 00:00:00 2001 From: phranzia <9207473+phranzia@users.noreply.github.com> Date: Fri, 10 Jul 2026 15:09:30 -0700 Subject: [PATCH 2/4] Minor doc updates --- AGENTS.md | 8 +-- CONFIG.md | 65 ++++++++----------- CONTRIBUTING.md | 4 +- README.md | 48 +++++++------- cli/README.md | 12 ++-- cli/examples/configs/anthropic/quantiles.toml | 2 +- .../configs/cloudflare/quantiles.toml | 2 +- cli/examples/configs/gemini/quantiles.toml | 2 +- cli/examples/configs/openai/quantiles.toml | 4 +- cli/src/builtins/mod.rs | 2 +- python/README.md | 4 +- typescript/README.md | 6 +- 12 files changed, 74 insertions(+), 85 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 197c41f..e8a0a83 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ Keep root-level changes focused on public orientation, documentation, contributi Do not silently change evaluation semantics. Changes to prompts, datasets, scorers, rubrics, metrics, sampling parameters, judge configuration, model selection, tool configuration, or step inputs can invalidate comparisons. Call out any such changes in handoff. -Update documentation in this repository (e.g. `README.md`, etc...) when any of the following change: +Update the relevant documentation in this repository, such as `README.md`, when any of the following change: - CLI commands, flags, outputs, or setup steps. - APIs, SDKs, imports, examples, or package names. @@ -85,7 +85,7 @@ When editing the Python SDK subdirectory, preserve `async` behavior, stable JSON ### `typescript/` directory ->**Not released**. The TypeScript SDK is currently unsupported and unreleased. +> **Not released**. The TypeScript SDK is currently unsupported and unreleased. The `typescript/` directory contains the Quantiles TypeScript SDK, which allows users to author custom local AI evals with the Quantiles stack using TypeScript. It exposes workflow primitives such as `QuantilesClient`, `QuantilesRun`, stable JSON utilities, and shared SDK types. @@ -107,7 +107,7 @@ Provider-backed model inputs should use provider-prefixed model names, for examp ## Safety, Security And Privacy -Preserve Quantiles as local-first infrastructure. Follow the below guidelines to ensure the project maintains safety, security and privacy: +Preserve Quantiles as local-first infrastructure. Follow the guidelines below to ensure the project maintains safety, security, and privacy: - The CLI and local server should store Quantiles state locally by default - Evaluation workflows may call remote model providers, hosted judges, datasets, APIs, or external tools only when explicitly configured by the user @@ -140,7 +140,7 @@ When remote model calls, hosted judges, external tools, provider APIs, or networ ## Output Style For Coding Agents -When adding documentation to files in this repository, follow the below guidelines: +When adding documentation to files in this repository, follow the guidelines below: - Be concise, technical, and action-oriented. - Prefer runnable commands, real file paths, concrete inputs, and expected outputs. diff --git a/CONFIG.md b/CONFIG.md index ec62c4e..235771c 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -10,11 +10,11 @@ You need a config file when you want to do one or more of the following: - Define custom evaluations (`type = "custom_code"`) - Resume custom evaluations later with `qt resume ` -You don't, however, need any configuration when you want to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box. +You do not need a configuration file to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box. ## File location and name -If you want to make a configuration file, create either a `quantiles.toml` or `.quantiles.toml` file in the current working directory. You can only have one or the other -- if both exist, the CLI exits with an ambiguity error. +To configure Quantiles, create either `quantiles.toml` or `.quantiles.toml` in the current working directory. Use only one filename; if both files are present, the CLI exits with an ambiguity error. ## Top-level structure @@ -22,34 +22,23 @@ Every benchmark lives under its own `[benchmarks.]` section. The sect For example, if you want to override default parameters for the built-in PubMedQA benchmark, add the following section to your config file: -```toml -# This is configuration for the PubMedQA benchmark that is built into the `qt` CLI. -# -# When you run `qt run pubmedqa`, the CLI will look here for how exactly to run PubMedQA. -[benchmarks.pubmedqa] -# Limit to the first 50 samples of the benchmark -samples = 50 -# Customize the model to test -model = "openai:gpt-5.4-nano" -# see below for the full reference -``` - -Alternatively, if you want to build your own custom evaluation using the Quantiles Python SDK, define that custom eval as follows: - ```toml # This is configuration for a custom code benchmark called my-eval. -# -# When you run `qt run my-eval`, the CLI will look here for how exactly to run your custom -# benchmark. +# +# When you run `qt run my-eval`, the CLI will look here for how to run +# your custom benchmark. [benchmarks.my-eval] -# This `type` parameter defaults to "builtin". for custom evals, be sure to override -# the default with "custom_code" here. + +# This parameter defaults to "builtin". For custom evaluations, override +# the default with "custom_code". type = "custom_code" -# This is the command that the CLI will execute to run your custom eval code. We recommend -# using the `uv` tool to configure and manage your Python evals, but this field can be any -# command, so you're free to use any tools you want. + +# The CLI executes this command to run your custom evaluation. We recommend +# using `uv` to configure and manage Python evaluations, but you may use +# any command or tool you prefer. command = ["uv", "run", "my_eval.py"] -# see below for the full reference + +# See below for the full reference. ``` ## Benchmark types @@ -60,12 +49,12 @@ Every benchmark section has a `type` field. Valid values are `"builtin"` (defaul Built-in benchmarks run natively inside the CLI, without any custom code. Below is a list of parameters that can be customized for built-in benchmarks: -| Field | Type | Required | Description | -|--|--|--|--| -| `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. | -| `samples` | integer | no | Number of dataset rows to evaluate. | -| `model` | string or table | no | Model sampler. See [model format](#model-format). | -| `max_workers` | integer | no | Maximum concurrent workers. | +| Field | Type | Required | Description | +| ------------- | --------------- | -------- | ---------------------------------------------------------------- | +| `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. | +| `samples` | integer | no | Number of dataset rows to evaluate. | +| `model` | string or table | no | Model sampler. See [model format](#model-format). | +| `max_workers` | integer | no | Maximum concurrent workers. | If none of these fields are customized, the built-in benchmark uses the following defaults: @@ -76,7 +65,7 @@ If none of these fields are customized, the built-in benchmark uses the followin #### `model` naming -The above `model` field accepts a provider-prefixed string, for example +The `model` field described above accepts a provider-prefixed string, for example: ```toml model = "openai:gpt-5.4-nano" @@ -101,11 +90,11 @@ Note that models require specific configuration based on the provider. For detai Custom evaluations are external programs built with the Quantiles Python SDK. Their config sections contain the following fields: -| Field | Type | Required | Description | -|--|--|--|--| -| `type` | string | yes | Must be `"custom_code"`. | -| `command` | array of strings | yes | Command and arguments to execute. | -| `input` | table | no | Structured input passed to the child as `QUANTILES_INPUT`. | +| Field | Type | Required | Description | +| --------- | ---------------- | -------- | ---------------------------------------------------------- | +| `type` | string | yes | Must be `"custom_code"`. | +| `command` | array of strings | yes | Command and arguments to execute. | +| `input` | table | no | Structured input passed to the child as `QUANTILES_INPUT`. | Note that custom code evaluations can customize the model in code. See the [PubMedQA custom code example](./python/examples/pubmedqa.py) for details on customizing the model in `custom_code` benchmarks. @@ -181,7 +170,7 @@ When you run `qt resume `, the CLI looks up the stored eval name and inp model = "openai:gpt-5.4-nano" ``` -### Built-in using demo model, and restricting samples +### Built-in using the demo model with a sample limit ```toml [benchmarks.simpleqa-verified] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 84e8d22..4e91911 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ # Contributing to Quantiles -Thank you for interest in contributing to Quantiles Open source! +Thank you for your interest in contributing to Quantiles open source! Quantiles open source projects provide local-first evaluation infrastructure for AI systems. We welcome contributions that improve reliability, reproducibility, developer ergonomics, agent workflows, documentation, and benchmark and evaluation support. @@ -39,7 +39,7 @@ Please do not report security vulnerabilities through public GitHub issues. Foll ## Code of Conduct -By participating in this project, you agree to follow our [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md) +By participating in this project, you agree to follow our [CODE_OF_CONDUCT.md](./CODE_OF_CONDUCT.md). ## License diff --git a/README.md b/README.md index f5f3735..550814a 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ -# Quantiles Open-source +# Quantiles: Local-First AI Evaluation Infrastructure -Quantiles is a local-first CLI and SDK for running durable AI evaluation workflows with fast, continuous feedback. Teams can iterate on model behavior, prompts, agent workflows, and debugging scripts while preserving the metrics and run histories needed to understand what improved, what regressed, and why. +Quantiles is a local-first CLI and SDK for running durable, reproducible AI evaluation workflows. Teams can iterate on models, prompts, agent workflows, and evaluation logic while preserving the metrics and run history needed to understand what improved, what regressed, and why. -It includes a CLI (called `qt`), SDKs, built-in benchmarks, local run history, and agent-friendly instructions for coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agentic development tools. This monorepo centralizes all the pieces of Quantiles, making it easier for engineers and coding agents to inspect, change, test, and extend the system. +It includes the `qt` CLI, a Python SDK, built-in benchmarks, local evaluation data, and agent-friendly instructions for coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. This monorepo centralizes Quantiles components, making it easier for engineers, researchers, and coding agents to inspect, modify, test, and extend the system. ## Why Quantiles @@ -59,15 +59,15 @@ Use `qt show` to inspect a single run, `qt list` to see a list of all runs, and Common commands: ```bash -qt run +qt run qt list qt show qt compare ``` ->Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output. +> Note: you can pass the `--json` flag to any of the above commands, to output machine and agent-friendly JSON instead of human-formatted output. -To learn more detail about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli). +To learn more about what you can do with the CLI, see [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli). ### Configuration file and customization @@ -110,7 +110,7 @@ The Quantiles toolchain, including the `qt` CLI, SDKs, on-disk data formats, and Both the CLI and Python SDK support offline benchmark workflows, including the following local execution and analysis features: - Benchmark code runs locally on your machine -- Measurements are computed locally, except for remote model calls, hosted judges, external tools, and LLM-as-judge evaluations that may call remote providers (e.g. OpenAI, Anthropic, cloud providers, etc.) +- Measurements are computed locally, although model calls, hosted judges, external tools, and LLM-as-judge evaluations may use remote providers such as OpenAI, Anthropic, or cloud services. - Metadata are recorded to a local, on-disk database - Metrics and evaluation outputs are recorded to local, on-disk files - `qt show`, `qt list` and `qt compare` commands access only local metadata and analytics databases @@ -119,22 +119,22 @@ Both the CLI and Python SDK support offline benchmark workflows, including the f Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark. -| Code | When to use | -| --- | --- | -| `qt run ` | Run a built-in benchmark against the demo model to inspect sample-level inputs and outputs, scoring behavior, workflow steps, and aggregate metrics | -| `qt run --input '{"model":"}'` | Run a built-in benchmark against your model | +| Code | When to use | +| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `qt run ` | Run a built-in benchmark against the demo model to inspect sample-level inputs and outputs, scoring behavior, workflow steps, and aggregate metrics | +| `qt run --input '{"model":"}'` | Run a built-in benchmark against your model | Quantiles also provides a [benchmark hub](https://quantiles.io/benchmark-hub) for discovering built-in benchmarks, understanding their evaluation setup, and reviewing common metrics used across AI evaluation workflows. -### Add a built-in benchmark +### Add a built-in benchmark If there is an open-source benchmark you would like to add as a built-in benchmark, [file an issue](https://github.com/quantiles-evals/quantiles/issues). -Helpful requests include the benchmark name, source dataset or repository, license and any reference implementation. +Helpful requests include the benchmark name, source dataset or repository, license, and any reference implementation. ## Custom Evaluations -A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](http://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons. +A custom evaluation is a [Python](https://quantiles.io/documentation/reference/python-sdk) program that is run by the `qt` CLI and uses its [local storage](https://quantiles.io/documentation/local-first-offline) and [durable workflow engine](https://quantiles.io/documentation/workflows-and-steps) to run efficiently and reliably. Your code owns the evaluation logic like loading data, calling a model or agent, scoring outputs, computing metrics, and returning a summary. Quantiles manages [durable steps, step caching, and step resume](https://quantiles.io/documentation/workflows-and-steps), metrics, inputs, outputs, and comparisons. Custom evaluations are configured in `quantiles.toml` with `type = "custom_code"`: @@ -155,9 +155,9 @@ Read more about how to build and run custom evaluations at [quantiles.io/documen ### Python SDK -Use the official Quantiles Python SDK to build your custom evaluations with primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python. The SDK integrates tightly with the `qt` CLI’s local API for running, recording and analyzing benchmarks. +Use the official Quantiles Python SDK to build your custom evaluations with primitives like durable steps, structured inputs/outputs, and metrics emission, using patterns and practices native to Python. The SDK integrates tightly with the `qt` CLI’s local API for running, recording, and analyzing benchmarks. -The code for the Python SDK is located in this repository at [`./python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk) +The code for the Python SDK is located in this repository at [`./python/`](./python). Read more about it at [quantiles.io/documentation/reference/python-sdk](https://quantiles.io/documentation/reference/python-sdk). ## Coding Agents @@ -165,21 +165,21 @@ Quantiles is designed to work well with coding agents such as Codex, Claude Code ### `SKILL.md` -The [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) repository includes a [`SKILL.md`](https://github.com/quantiles-evals/skill/blob/main/SKILL.md) file that gives agents complete instructions for running evaluations, inspecting results, comparing runs, and summarizing regressions. To use the skill with your agent, install it with the below prompt: +The [github.com/quantiles-evals/skill](https://github.com/quantiles-evals/skill) repository includes a [`SKILL.md`](https://github.com/quantiles-evals/skill/blob/main/SKILL.md) file that gives agents complete instructions for running evaluations, inspecting results, comparing runs, and summarizing regressions. To use the skill with your agent, install it with the following prompt: ```text Install the Quantiles eval skill at github.com/quantiles-evals/skill ``` -If you want your agent to run an eval, use the below prompt: +If you want your agent to run an eval, use the following prompt: ```text - Use the Quantiles eval skill to run the SimpleQA Verified benchmark and summarize the results. - ``` +Use the Quantiles eval skill to run the SimpleQA Verified benchmark and summarize the results. +``` - ### `AGENTS.md` +### `AGENTS.md` - The embedded [`AGENTS.md` file](./AGENTS.md) gives agents repository-specific instructions, such as how to add features to the CLI and SDKs, ensuring that contributors can use agents of their choice to make high-quality contributions to Quantiles open source systems. +The embedded [`AGENTS.md` file](./AGENTS.md) gives agents repository-specific instructions, such as how to add features to the CLI and SDKs, ensuring that contributors can use agents of their choice to make high-quality contributions to the Quantiles' open source components. ## Documentation @@ -193,7 +193,7 @@ Start here: ## Contributing -Quantiles exists to make AI evaluation workflows more practical, repeatable, and useful for engineering teams. We welcome contributions from the community, whether you are fixing bugs, improving documentation, adding evaluations and benchmarks, or helping make Quantiles Open Source more reliable for AI engineers and researchers. +Quantiles exists to make AI evaluation workflows more practical, repeatable, and useful for engineering teams. We welcome contributions from the community, whether you are fixing bugs, improving documentation, adding evaluations and benchmarks, or helping make Quantiles open-source more reliable for AI engineers and researchers. Please read our [contributing guide](./CONTRIBUTING.md) to get started. @@ -203,4 +203,4 @@ Please do not report security vulnerabilities through public GitHub issues. Foll ## License -Quantiles Open-source is licensed under the [Apache License 2.0](./LICENSE). Hosted, enterprise, or managed Quantiles products may be offered under separate commercial terms. +Quantiles open source is licensed under the [Apache License 2.0](./LICENSE). Hosted, enterprise, or managed Quantiles products may be offered under separate commercial terms. diff --git a/cli/README.md b/cli/README.md index eb556f5..6248a4e 100644 --- a/cli/README.md +++ b/cli/README.md @@ -1,6 +1,6 @@ # Quantiles CLI -This directory holds the source code for the `qt` CLI. It is built with [Rust](https://rust-lang.org/) to help it efficiently use the resources of the local machine, to help ensure safety, and to provide strong lints and type-system invariants for humans and agents to work with. +This directory contains the source code for the `qt` CLI. It is implemented in [Rust](https://rust-lang.org/) to use local machine resources efficiently, improve safety, and provide strong lints and type-system invariants for humans and agents. ## Install @@ -17,9 +17,9 @@ A few commands to see `qt` in action: qt init # 2. Run a built-in eval -# +# # Note that you can also build and run your own custom evals -# with the `qt` CLI. See the below "Custom evaluations" section +# with the `qt` CLI. See the following "Custom evaluations" section # for details. qt run pubmedqa @@ -28,11 +28,11 @@ qt list qt show 1 ``` ->See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. +> See [quantiles.io/documentation/reference/cli](https://quantiles.io/documentation/reference/cli) for a detailed list of `qt` commands. ### Custom evaluations -Custom evaluations are denoted in the configuration file with `type = "custom_code"`. The `command` array tells the CLI how to execute your eval, and the optional `input` table is merged with any values passed in the `qt run --input` flag, then passed to your script as `QUANTILES_INPUT`. An example is below +Custom evaluations are denoted in the configuration file with `type = "custom_code"`. The `command` array tells the CLI how to execute your eval, and the optional `input` table is merged with any values passed through the `qt run --input` flag, then passed to your script as `QUANTILES_INPUT`. An example is below: ```toml [benchmarks.my-eval] @@ -69,7 +69,7 @@ model = "openai:gpt-5.4-nano" max_workers = 100 ``` ->Note: Quantiles is designed for high-throughput execution and may issue many requests in parallel. Depending on your provider, model, and account limits, benchmark runs can quickly hit API rate limits or concurrency quotas. Consider reducing concurrency or using models/providers with higher rate limits if you encounter throttling. Example configurations illustrate how to do so. +> Note: Quantiles is designed for high-throughput execution and may issue many requests in parallel. Depending on your provider, model, and account limits, benchmark runs can quickly hit API rate limits or concurrency quotas. Consider reducing concurrency or using models/providers with higher rate limits if you encounter throttling. Example configurations illustrate how to do so. ### Custom code evals diff --git a/cli/examples/configs/anthropic/quantiles.toml b/cli/examples/configs/anthropic/quantiles.toml index 8bf60b1..7405860 100644 --- a/cli/examples/configs/anthropic/quantiles.toml +++ b/cli/examples/configs/anthropic/quantiles.toml @@ -9,7 +9,7 @@ # usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# The below model is good for demonstrating PubMedQA benchmarks without very high costs, +# The following model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. Details on the model can be found at: # # https://platform.claude.com/docs/en/about-claude/models/overview#latest-models-comparison diff --git a/cli/examples/configs/cloudflare/quantiles.toml b/cli/examples/configs/cloudflare/quantiles.toml index 9e0dcab..29bf1cd 100644 --- a/cli/examples/configs/cloudflare/quantiles.toml +++ b/cli/examples/configs/cloudflare/quantiles.toml @@ -17,7 +17,7 @@ # usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# The below model is good for demonstrating PubMedQA benchmarks without very high costs, +# The following model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap, fast, and not a reasoning model. Its details can be found at: # # https://developers.cloudflare.com/workers-ai/models/llama-3.3-70b-instruct-fp8-fast/ diff --git a/cli/examples/configs/gemini/quantiles.toml b/cli/examples/configs/gemini/quantiles.toml index 002b477..cfc348c 100644 --- a/cli/examples/configs/gemini/quantiles.toml +++ b/cli/examples/configs/gemini/quantiles.toml @@ -11,7 +11,7 @@ # usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# The below model is good for demonstrating PubMedQA benchmarks without very high costs, +# The following model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. Details on the model can be found at: # # https://ai.google.dev/gemini-api/docs/models/gemini#gemini-3.1-flash-lite diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index 8f6e164..e27d744 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -12,9 +12,9 @@ # usage, we are limiting the number of samples to 50 in this demo, to keep costs lower. samples = 50 -# The below model is good for demonstrating PubMedQA benchmarks without very high costs, +# The following model is good for demonstrating PubMedQA benchmarks without very high costs, # since it's relatively cheap and fast. You may need to ensure the project associated with -# your API key has permissions to use this model. Do so by going to the below page, and +# your API key has permissions to use this model. Do so by going to the follwing page, and # enabling the model: # # https://platform.openai.com/settings/$PROJECT_ID/limits diff --git a/cli/src/builtins/mod.rs b/cli/src/builtins/mod.rs index 0611a72..7740c57 100644 --- a/cli/src/builtins/mod.rs +++ b/cli/src/builtins/mod.rs @@ -27,7 +27,7 @@ pub struct BuiltinContext<'a> { /// Trait for builtin evals that run natively inside the CLI. #[async_trait] pub trait BuiltinWorkflow: Send + Sync { - /// Unique name of the builtin (e.g. "financebench"). + /// Unique name of the built-in (e.g. "financebench"). fn name(&self) -> &'static str; /// Execute the builtin eval. async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()>; diff --git a/python/README.md b/python/README.md index c56b698..823ca1c 100644 --- a/python/README.md +++ b/python/README.md @@ -10,7 +10,7 @@ uv add quantiles ## Usage -To build a custom eval with Python, use the below code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details. +To build a custom eval with Python, use the following code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details. ```python import asyncio @@ -31,7 +31,7 @@ if __name__ == "__main__": entrypoint(my_workflow) ``` -In local development, the SDK executes user code locally. The `qt` server deduplicates steps, triggers workflows, owns durable state, stored outputs, observability records, and metrics. +During local development, the SDK executes user code locally. The `qt` server initiates and coordinates workflows, deduplicates steps, and manages durable state, stored outputs, observability records, and metrics. ## Development diff --git a/typescript/README.md b/typescript/README.md index 6315ecf..2a63a4d 100644 --- a/typescript/README.md +++ b/typescript/README.md @@ -1,6 +1,6 @@ # `@quantiles/sdk` ->**Note**: this SDK is currently unsupported and unreleased. +> **Note**: this SDK is currently unsupported and unreleased. TypeScript client for the local Quantiles eval runner. @@ -12,7 +12,7 @@ bun install quantiles ## Usage -To build a custom eval with TypeScript, use the below code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details. +To build a custom eval with TypeScript, use the following code. To ensure this eval is runnable with `qt run`, set up a `quantiles.toml` configuration file. See [`../CONFIG.md`](../CONFIG.md) for details. ```ts import { QuantilesClient } from "@quantiles/sdk"; @@ -29,4 +29,4 @@ const output = await run.step("call-model", { prompt: "hello" }, async () => { await run.complete(); ``` -In local development, the SDK executes user code locally. The `qt` server deduplicates steps, triggers workflows, owns durable state, stored outputs, observability records, and metrics. +During local development, the SDK executes user code locally. The `qt` server initiates and coordinates workflows, deduplicates steps, and manages durable state, stored outputs, observability records, and metrics. From 1dc8e6c7a9734d66cccfd1b7e61432f7d2a13146 Mon Sep 17 00:00:00 2001 From: phranzia <9207473+phranzia@users.noreply.github.com> Date: Fri, 10 Jul 2026 15:13:27 -0700 Subject: [PATCH 3/4] Update consistent connectors --- cli/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cli/README.md b/cli/README.md index 6248a4e..96ab39b 100644 --- a/cli/README.md +++ b/cli/README.md @@ -109,14 +109,14 @@ The Quantiles CLI, `qt`, keeps execution simple: your code runs locally, while ` +-------------------+------------------+ │ │ HTTP / JSON - | + │ ▼ +--------------------------------------+ | Quantiles Server | +-------------------+------------------+ │ │ SQLite - | + │ ▼ +------------------------------------------------+ | .quantiles/quantiles.sqlite (local DB) | From 0e3f8dfc814ab08aa904cb8d6986b8b76ad95503 Mon Sep 17 00:00:00 2001 From: phranzia <9207473+phranzia@users.noreply.github.com> Date: Fri, 10 Jul 2026 15:35:52 -0700 Subject: [PATCH 4/4] Minor doc updates --- AGENTS.md | 6 ++--- CONFIG.md | 29 +++++++--------------- README.md | 6 ++--- cli/README.md | 6 ++--- cli/examples/configs/openai/quantiles.toml | 2 +- cli/src/llm/openai.rs | 2 +- cli/src/llm/sampler.rs | 18 +++++++------- python/README.md | 2 +- python/src/quantiles/concurrency.py | 2 +- typescript/AGENTS.md | 2 +- 10 files changed, 32 insertions(+), 43 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e8a0a83..c3c8592 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,7 +39,7 @@ When performing tasks in this repository, always do the following: - Prefer the smallest change that satisfies the task. - Do not run expensive, provider-backed, or full benchmark commands without explicit approval. - Keep changes small, public-safe, and reviewable. -- Preserve Quantiles as local-first, offline system by default. +- Preserve Quantiles as a local-first, offline system by default. - Verify commands, links, package names, benchmark and evaluation names, and release status before documenting them. - Update public docs when CLI behavior, SDK APIs, workflows, benchmarks, schemas, setup steps, or agent guidance changes. - Prefer concrete examples with commands, file paths, inputs, outputs, and expected behavior. @@ -105,7 +105,7 @@ Provider-backed model inputs should use provider-prefixed model names, for examp - `anthropic:` - `gemini:` -## Safety, Security And Privacy +## Safety, Security, and Privacy Preserve Quantiles as local-first infrastructure. Follow the guidelines below to ensure the project maintains safety, security, and privacy: @@ -114,7 +114,7 @@ Preserve Quantiles as local-first infrastructure. Follow the guidelines below to - Do not inspect, print, summarize, commit, or infer values from `.env` or `.envrc` files, secrets, tokens, private datasets, PHI, customer data, or local Quantiles databases unless the user explicitly asks and the data is safe to inspect. - Never commit the `.quantiles/` directory, SQLite or metrics databases, local traces, benchmark outputs, provider credentials, or temporary run artifacts. - Use placeholder names such as `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `QUANTILES_API_KEY` when examples need credentials. -- Keep public docs customer-safe. Avoid exposing secrets, API keys and non-public security information. +- Keep public docs customer-safe. Avoid exposing secrets, API keys, and non-public security information. Read [`SECURITY.md`](./SECURITY.md) for more details on keeping this repository and toolchain secure. diff --git a/CONFIG.md b/CONFIG.md index 235771c..6ed2a10 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -10,7 +10,7 @@ You need a config file when you want to do one or more of the following: - Define custom evaluations (`type = "custom_code"`) - Resume custom evaluations later with `qt resume ` -You do not need a configuration file to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box. +You do not need a configuration file to run built-in benchmarks. For example, `qt run pubmedqa` works out of the box. ## File location and name @@ -23,22 +23,11 @@ Every benchmark lives under its own `[benchmarks.]` section. The sect For example, if you want to override default parameters for the built-in PubMedQA benchmark, add the following section to your config file: ```toml -# This is configuration for a custom code benchmark called my-eval. -# -# When you run `qt run my-eval`, the CLI will look here for how to run -# your custom benchmark. -[benchmarks.my-eval] - -# This parameter defaults to "builtin". For custom evaluations, override -# the default with "custom_code". -type = "custom_code" +# Configure the model and sample limit for the built-in PubMedQA benchmark. -# The CLI executes this command to run your custom evaluation. We recommend -# using `uv` to configure and manage Python evaluations, but you may use -# any command or tool you prefer. -command = ["uv", "run", "my_eval.py"] - -# See below for the full reference. +[benchmarks.pubmedqa] +samples = 50 +model = "openai:gpt-5.6" ``` ## Benchmark types @@ -68,7 +57,7 @@ If none of these fields are customized, the built-in benchmark uses the followin The `model` field described above accepts a provider-prefixed string, for example: ```toml -model = "openai:gpt-5.4-nano" +model = "openai:gpt-5.6" ``` Supported provider prefixes are listed below: @@ -81,7 +70,7 @@ Supported provider prefixes are listed below: You can pass a TOML table instead of such a prefixed string: ```toml -model = { provider = "openai", model_id = "gpt-5.4-nano" } +model = { provider = "openai", model_id = "gpt-5.6" } ``` Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in [`cli/examples/configs`](./cli/examples/configs). @@ -96,7 +85,7 @@ Custom evaluations are external programs built with the Quantiles Python SDK. Th | `command` | array of strings | yes | Command and arguments to execute. | | `input` | table | no | Structured input passed to the child as `QUANTILES_INPUT`. | -Note that custom code evaluations can customize the model in code. See the [PubMedQA custom code example](./python/examples/pubmedqa.py) for details on customizing the model in `custom_code` benchmarks. +Note that custom code evaluations can customize the model in code. See the [PubMedQA custom code example](./python-examples/src/pubmedqa.py) for details on customizing the model in `custom_code` benchmarks. #### The `input` table @@ -167,7 +156,7 @@ When you run `qt resume `, the CLI looks up the stored eval name and inp ```toml [benchmarks.pubmedqa] -model = "openai:gpt-5.4-nano" +model = "openai:gpt-5.6" ``` ### Built-in using the demo model with a sample limit diff --git a/README.md b/README.md index 550814a..553dd56 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ For **built-in benchmarks**, configure settings like `samples`, `model`, and `ma ```toml [benchmarks.pubmedqa] samples = 50 -model = "openai:gpt-5.4-nano" +model = "openai:gpt-5.6" max_workers = 100 ``` @@ -179,7 +179,7 @@ Use the Quantiles eval skill to run the SimpleQA Verified benchmark and summariz ### `AGENTS.md` -The embedded [`AGENTS.md` file](./AGENTS.md) gives agents repository-specific instructions, such as how to add features to the CLI and SDKs, ensuring that contributors can use agents of their choice to make high-quality contributions to the Quantiles' open source components. +The embedded [`AGENTS.md` file](./AGENTS.md) gives agents repository-specific instructions, such as how to add features to the CLI and SDKs, ensuring that contributors can use agents of their choice to make high-quality contributions to the Quantiles open source components. ## Documentation @@ -193,7 +193,7 @@ Start here: ## Contributing -Quantiles exists to make AI evaluation workflows more practical, repeatable, and useful for engineering teams. We welcome contributions from the community, whether you are fixing bugs, improving documentation, adding evaluations and benchmarks, or helping make Quantiles open-source more reliable for AI engineers and researchers. +Quantiles exists to make AI evaluation workflows more practical, repeatable, and useful for engineering teams. We welcome contributions from the community, whether you are fixing bugs, improving documentation, adding evaluations and benchmarks, or helping make the open-source Quantiles project more reliable for AI engineers and researchers. Please read our [contributing guide](./CONTRIBUTING.md) to get started. diff --git a/cli/README.md b/cli/README.md index 96ab39b..42747ce 100644 --- a/cli/README.md +++ b/cli/README.md @@ -32,7 +32,7 @@ qt show 1 ### Custom evaluations -Custom evaluations are denoted in the configuration file with `type = "custom_code"`. The `command` array tells the CLI how to execute your eval, and the optional `input` table is merged with any values passed through the `qt run --input` flag, then passed to your script as `QUANTILES_INPUT`. An example is below: +Custom evaluations are denoted in the configuration file with `type = "custom_code"`. The `command` array tells the CLI how to execute your eval, and the optional `input` table is merged with any values passed through the `--input` flag, then passed to your script as `QUANTILES_INPUT`. An example is below: ```toml [benchmarks.my-eval] @@ -55,7 +55,7 @@ See [`examples/configs/custom_code/quantiles.toml`](./examples/configs/custom_co You can customize how the CLI executes built-in benchmarks and custom evaluations using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples: -- [`../CONFIG.md`](../CONFIG.md): for a guide and reference. +- [`../CONFIG.md`](../CONFIG.md) for a guide and reference. - [`./examples/configs`](./examples/configs) for complete working examples. ### Built-in benchmarks @@ -65,7 +65,7 @@ For built-in benchmarks, configure settings like `samples`, `model`, and `max_wo ```toml [benchmarks.pubmedqa] samples = 50 -model = "openai:gpt-5.4-nano" +model = "openai:gpt-5.6" max_workers = 100 ``` diff --git a/cli/examples/configs/openai/quantiles.toml b/cli/examples/configs/openai/quantiles.toml index e27d744..db46a7a 100644 --- a/cli/examples/configs/openai/quantiles.toml +++ b/cli/examples/configs/openai/quantiles.toml @@ -26,7 +26,7 @@ samples = 50 # And a list of all models can be found at: # # https://developers.openai.com/api/docs/models/all -model = "openai:gpt-5.4-nano" +model = "openai:gpt-5.6" # If you run this or other benchmarks with more samples, you may also need to increase this # `max_workers` parameter to improve throughput. This might cause you to run into rate diff --git a/cli/src/llm/openai.rs b/cli/src/llm/openai.rs index 637740b..958e25b 100644 --- a/cli/src/llm/openai.rs +++ b/cli/src/llm/openai.rs @@ -33,7 +33,7 @@ pub struct OpenAISampler { } impl OpenAISampler { - /// Create a new sampler for the given model identifier (e.g. `gpt-5.4-nano`). + /// Create a new sampler for the given model identifier (e.g. `gpt-5.6`). #[must_use] pub fn new(model_id: &str) -> Self { Self { diff --git a/cli/src/llm/sampler.rs b/cli/src/llm/sampler.rs index dfb4b24..a678f96 100644 --- a/cli/src/llm/sampler.rs +++ b/cli/src/llm/sampler.rs @@ -28,7 +28,7 @@ pub enum Sampler { /// /// TODO: support custom endpoints OpenAI { - /// `OpenAI` model identifier (e.g. `gpt-5.4-nano`). + /// `OpenAI` model identifier (e.g. `gpt-5.6`). model_id: String, }, /// Anthropic API @@ -425,22 +425,22 @@ mod tests { #[test] fn deserialize_openai_string() { - let s: Sampler = serde_json::from_str("\"openai:gpt-5.4-nano\"").unwrap(); + let s: Sampler = serde_json::from_str("\"openai:gpt-5.6\"").unwrap(); assert_eq!( s, Sampler::OpenAI { - model_id: "gpt-5.4-nano".to_string(), + model_id: "gpt-5.6".to_string(), } ); } #[test] fn deserialize_openai_table() { - let s: Sampler = serde_json::from_str(r#"{"type":"openai:gpt-5.4-nano"}"#).unwrap(); + let s: Sampler = serde_json::from_str(r#"{"type":"openai:gpt-5.6"}"#).unwrap(); assert_eq!( s, Sampler::OpenAI { - model_id: "gpt-5.4-nano".to_string(), + model_id: "gpt-5.6".to_string(), } ); } @@ -448,18 +448,18 @@ mod tests { #[test] fn serialize_openai() { let s = Sampler::OpenAI { - model_id: "gpt-5.4-nano".to_string(), + model_id: "gpt-5.6".to_string(), }; let json = serde_json::to_string(&s).unwrap(); - assert_eq!(json, "\"openai:gpt-5.4-nano\""); + assert_eq!(json, "\"openai:gpt-5.6\""); } #[test] fn display_openai() { let s = Sampler::OpenAI { - model_id: "gpt-5.4-nano".to_string(), + model_id: "gpt-5.6".to_string(), }; - assert_eq!(s.to_string(), "openai:gpt-5.4-nano"); + assert_eq!(s.to_string(), "openai:gpt-5.6"); } #[test] diff --git a/python/README.md b/python/README.md index 823ca1c..4587eb1 100644 --- a/python/README.md +++ b/python/README.md @@ -1,6 +1,6 @@ # Quantiles Python SDK -Python SDK for the [quantiles](https://quantiles.io) local AI workload observability server. +The Quantiles Python SDK for the [quantiles](https://quantiles.io) local AI workload observability server. ## Installation diff --git a/python/src/quantiles/concurrency.py b/python/src/quantiles/concurrency.py index 716fe16..8fc458d 100644 --- a/python/src/quantiles/concurrency.py +++ b/python/src/quantiles/concurrency.py @@ -106,7 +106,7 @@ async def map_dataset[ItemT: BaseModel, U]( representing the results of each function call. The order in which results are returned is specified by the given ``yield_order`` - paramter. ``"input"`` matches the order of items yielded by the given ``dataset``, + parameter. ``"input"`` matches the order of items yielded by the given ``dataset``, while ``"completion"`` matches the order in which items are computed. Args: diff --git a/typescript/AGENTS.md b/typescript/AGENTS.md index 6580205..87ea7fc 100644 --- a/typescript/AGENTS.md +++ b/typescript/AGENTS.md @@ -1,6 +1,6 @@ # AGENTS.md ->**Note**: This SDK is currently unreleased and unsupported. +> **Note**: This SDK is currently unreleased and unsupported. ## Scope