Implementing custom_nocode benchmarks for QA-style benchmarks

CONFIG.md

@@ -8,6 +8,7 @@ You need a config file when you want to do one or more of the following:
 
 - Override built-in benchmark defaults (e.g. model, sample limit)
 - Define custom evaluations (`type = "custom_code"`)
+- Define no-code QA benchmarks (`type = "custom_nocode"`, `style = "qa"`)
 - Resume custom evaluations later with `qt resume <run_id>`
 
 You don't, however, need any configuration when you want to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box.
@@ -54,7 +55,7 @@ command = ["uv", "run", "my_eval.py"]
 
 ## Benchmark types
 
-Every benchmark section has a `type` field. Valid values are `"builtin"` (default when absent) and `"custom_code"`.
+Every benchmark section has a `type` field. Valid values are `"builtin"` (default when absent), `"custom_code"`, and `"custom_nocode"`.
 
 ### `builtin`
 
@@ -64,7 +65,7 @@ Built-in benchmarks run natively inside the CLI, without any custom code. Below
 |--|--|--|--|
 | `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. |
 | `samples` | integer | no | Number of dataset rows to evaluate. |
-| `model` | string or table | no | Model sampler. See [model format](#model-format). |
+| `model` | string or table | no | Model sampler. See [model naming](#model-naming). |
 | `max_workers` | integer | no | Maximum concurrent workers. |
 
 If none of these fields are customized, the built-in benchmark uses the following defaults:
@@ -97,6 +98,42 @@ model = { provider = "openai", model_id = "gpt-5.4-nano" }
 
 Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in [`cli/examples/configs`](./cli/examples/configs).
 
+### `custom_nocode`
+
+No-code benchmarks are configured in TOML and run natively inside the CLI, without a custom Python or TypeScript evaluation program. The initial supported style is `qa`, which renders a prompt from a dataset row, calls a model, and scores the response with exact-match accuracy against the configured golden answer column.
+
+```toml
+[benchmarks.nocode_custom]
+type = "custom_nocode"
+style = "qa"
+dataset = "quantiles/simpleqa-verified"
+model = "random"
+prompt_template_file = "prompts/qa.txt"
+prompt_column = "problem"
+golden_column = "answer"
+limit = 10
+```
+
+Run it with:
+
+```bash
+qt run nocode_custom
+```
+
+For a complete minimal example, see [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml).
+
+| Field | Type | Required | Description |
+|--|--|--|--|
+| `type` | string | yes | Must be `"custom_nocode"`. |
+| `style` | string | yes | Must be `"qa"`. |
+| `dataset` | string | yes | Dataset identifier, for example `"quantiles/simpleqa-verified"`. |
+| `model` | string or table | no | Model sampler. Defaults to the demo random sampler. See [model naming](#model-naming). |
+| `prompt_template_file` | string | yes | Path to a Jinja prompt template file. The template receives `prompt` from the configured prompt column. |
+| `prompt_column` | string | yes | Dataset column containing the prompt text. |
+| `golden_column` | string | yes | Dataset column containing the golden answer. |
+| `limit` | integer | no | Number of dataset rows to evaluate. |
+| `max_workers` | integer | no | Maximum concurrent workers. |
+
 ### `custom_code`
 
 Custom evaluations are external programs built with the Quantiles Python SDK. Their config sections contain the following fields:

README.md

@@ -4,6 +4,10 @@ Quantiles is a local-first CLI and SDK for running durable AI evaluation workflo
 
 It includes a CLI (called `qt`), SDKs, built-in benchmarks, local run history, and agent-friendly instructions for coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agentic development tools. This monorepo centralizes all the pieces of Quantiles, making it easier for engineers and coding agents to inspect, change, test, and extend the system.
 
+## ![NEW!](./docs/assets/new-badge.svg) What's New
+
+**[2026.07.07]** Added `custom_nocode` QA benchmarks, which let users configure a dataset-backed question-answering benchmark in `quantiles.toml` without writing custom evaluation code. See [No-code QA benchmarks](./cli/src/builtins/custom_nocode.rs) for more details.
+
 ## Why Quantiles
 
 Evaluation workflows quickly outgrow one-off scripts once teams need caching, retries, dataset handling, metrics capture, and run comparison. Quantiles gives teams those primitives so they don't have to built them from scratch:
@@ -117,7 +121,9 @@ Both the CLI and Python SDK support offline benchmark workflows, including the f
 
 ## Built-in Benchmarks
 
-Built-in benchmarks are ready-to-run evalulations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark.
+Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark.
+
+For dataset-backed QA checks that do not need custom Python or TypeScript code, use a `custom_nocode` benchmark in `quantiles.toml`. A `custom_nocode` QA benchmark points to a dataset, a Jinja prompt template, the dataset column containing the prompt, and the dataset column containing the golden answer. See [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml) for a minimal example.
 
 | Code | When to use |
 | --- | --- |

cli/Cargo.toml

@@ -15,6 +15,7 @@ dirs = "6"
 fastembed = "5"
 futures = "0.3"
 genai = "0.6.5"
+jinja = "0.1"
 parquet = { version = "54", features = ["arrow"] }
 rand = "0.8"
 reqwest = { version = "0.13.3", features = ["json", "rustls"] }

cli/README.md

@@ -53,7 +53,7 @@ See [`examples/configs/custom_code/quantiles.toml`](./examples/configs/custom_co
 
 ## Configuration files and customization
 
-You can customize how the CLI executes built-in benchmarks and custom evaluations using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples:
+You can customize how the CLI executes built-in benchmarks, custom code evaluations, and no-code QA benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples:
 
 - [`../CONFIG.md`](../CONFIG.md): for a guide and reference.
 - [`./examples/configs`](./examples/configs) for complete working examples.
@@ -71,6 +71,28 @@ max_workers = 100
 
 >Note: Quantiles is designed for high-throughput execution and may issue many requests in parallel. Depending on your provider, model, and account limits, benchmark runs can quickly hit API rate limits or concurrency quotas. Consider reducing concurrency or using models/providers with higher rate limits if you encounter throttling. Example configurations illustrate how to do so.
 
+### No-code QA benchmarks
+
+For dataset-backed QA checks that do not need custom evaluation code, set `type = "custom_nocode"` and `style = "qa"`. The benchmark runs inside the CLI, renders each prompt with the configured Jinja template, calls the configured model, and scores each row with exact-match accuracy against the golden answer column.
+
+```toml
+[benchmarks.nocode_custom]
+type = "custom_nocode"
+style = "qa"
+dataset = "quantiles/simpleqa-verified"
+model = "random"
+prompt_template_file = "prompts/qa.txt"
+prompt_column = "problem"
+golden_column = "answer"
+limit = 10
+```
+
+```bash
+qt run nocode_custom
+```
+
+See [`../custom-nocode-examples/quantiles.toml`](../custom-nocode-examples/quantiles.toml) for a complete minimal example.
+
 ### Custom code evals
 
 For custom evaluations, set `type = "custom_code"` and provide the `command` to run. The optional `input` table is passed to your script as a JSON dictionary.

cli/src/builtins/common.rs

@@ -1,13 +1,15 @@
+use std::collections::hash_map::DefaultHasher;
+use std::hash::{Hash, Hasher};
+use std::sync::Arc;
+use std::time::Instant;
+
 use anyhow::{Context, Result};
 use sea_orm::DatabaseConnection;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
-use std::collections::hash_map::DefaultHasher;
-use std::hash::{Hash, Hasher};
-use std::time::Instant;
 
 use crate::db::steps::{self, StepDecision};
-use crate::llm::Sampler;
+use crate::llm::{LLMSampler, Sampler};
 use crate::metrics_store::MetricsStore;
 
 /// Fields shared by every builtin benchmark config. When adding a new builtin,
@@ -119,6 +121,48 @@ pub(crate) fn get_max_workers() -> usize {
     }
 }
 
+/// Resolve a model sampler, falling back to a default when none is configured.
+pub(crate) fn resolve_sampler(
+    model: Option<&Sampler>,
+    default: impl FnOnce() -> Arc<dyn LLMSampler>,
+) -> Result<Arc<dyn LLMSampler>> {
+    match model {
+        None => Ok(default()),
+        Some(sampler) => sampler.resolve(),
+    }
+}
+
+/// Emit aggregate `accuracy`, `correct_count`, and `total_count` metrics from a
+/// collection of per-sample boolean correctness values.
+#[expect(clippy::cast_precision_loss)]
+pub(crate) async fn emit_accuracy_metrics(
+    metrics_store: &MetricsStore,
+    run_id: i64,
+    results: impl IntoIterator<Item = bool>,
+) {
+    let mut correct_count = 0usize;
+    let mut total_count = 0usize;
+    for is_correct in results {
+        total_count += 1;
+        if is_correct {
+            correct_count += 1;
+        }
+    }
+
+    if total_count > 0 {
+        let accuracy = correct_count as f64 / total_count as f64;
+        metrics_store
+            .emit(run_id, None, "accuracy", accuracy, None)
+            .await;
+        metrics_store
+            .emit(run_id, None, "correct_count", correct_count as f64, None)
+            .await;
+        metrics_store
+            .emit(run_id, None, "total_count", total_count as f64, None)
+            .await;
+    }
+}
+
 /// Statistics computed from a collection of similarity scores.
 #[derive(Debug)]
 pub(crate) struct ScoreStatistics {