diff --git a/CONFIG.md b/CONFIG.md index ec62c4e..a5968f6 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -8,6 +8,7 @@ You need a config file when you want to do one or more of the following: - Override built-in benchmark defaults (e.g. model, sample limit) - Define custom evaluations (`type = "custom_code"`) +- Define no-code QA benchmarks (`type = "custom_nocode"`, `style = "qa"`) - Resume custom evaluations later with `qt resume ` You don't, however, need any configuration when you want to run built-in benchmarks. `qt run pubmedqa`, for example, works out of the box. @@ -54,7 +55,7 @@ command = ["uv", "run", "my_eval.py"] ## Benchmark types -Every benchmark section has a `type` field. Valid values are `"builtin"` (default when absent) and `"custom_code"`. +Every benchmark section has a `type` field. Valid values are `"builtin"` (default when absent), `"custom_code"`, and `"custom_nocode"`. ### `builtin` @@ -64,7 +65,7 @@ Built-in benchmarks run natively inside the CLI, without any custom code. Below |--|--|--|--| | `type` | string | no | Defaults to `"builtin"`. May be omitted for built-in benchmarks. | | `samples` | integer | no | Number of dataset rows to evaluate. | -| `model` | string or table | no | Model sampler. See [model format](#model-format). | +| `model` | string or table | no | Model sampler. See [model naming](#model-naming). | | `max_workers` | integer | no | Maximum concurrent workers. | If none of these fields are customized, the built-in benchmark uses the following defaults: @@ -97,6 +98,42 @@ model = { provider = "openai", model_id = "gpt-5.4-nano" } Note that models require specific configuration based on the provider. For details, see the `quantiles.toml` file under the provider of your choice in [`cli/examples/configs`](./cli/examples/configs). +### `custom_nocode` + +No-code benchmarks are configured in TOML and run natively inside the CLI, without a custom Python or TypeScript evaluation program. The initial supported style is `qa`, which renders a prompt from a dataset row, calls a model, and scores the response with exact-match accuracy against the configured golden answer column. + +```toml +[benchmarks.nocode_custom] +type = "custom_nocode" +style = "qa" +dataset = "quantiles/simpleqa-verified" +model = "random" +prompt_template_file = "prompts/qa.txt" +prompt_column = "problem" +golden_column = "answer" +limit = 10 +``` + +Run it with: + +```bash +qt run nocode_custom +``` + +For a complete minimal example, see [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml). + +| Field | Type | Required | Description | +|--|--|--|--| +| `type` | string | yes | Must be `"custom_nocode"`. | +| `style` | string | yes | Must be `"qa"`. | +| `dataset` | string | yes | Dataset identifier, for example `"quantiles/simpleqa-verified"`. | +| `model` | string or table | no | Model sampler. Defaults to the demo random sampler. See [model naming](#model-naming). | +| `prompt_template_file` | string | yes | Path to a Jinja prompt template file. The template receives `prompt` from the configured prompt column. | +| `prompt_column` | string | yes | Dataset column containing the prompt text. | +| `golden_column` | string | yes | Dataset column containing the golden answer. | +| `limit` | integer | no | Number of dataset rows to evaluate. | +| `max_workers` | integer | no | Maximum concurrent workers. | + ### `custom_code` Custom evaluations are external programs built with the Quantiles Python SDK. Their config sections contain the following fields: diff --git a/README.md b/README.md index de27d70..e67fab4 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,10 @@ Quantiles is a local-first CLI and SDK for running durable AI evaluation workflo It includes a CLI (called `qt`), SDKs, built-in benchmarks, local run history, and agent-friendly instructions for coding agents such as Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and other agentic development tools. This monorepo centralizes all the pieces of Quantiles, making it easier for engineers and coding agents to inspect, change, test, and extend the system. +## ![NEW!](./docs/assets/new-badge.svg) What's New + +**[2026.07.07]** Added `custom_nocode` QA benchmarks, which let users configure a dataset-backed question-answering benchmark in `quantiles.toml` without writing custom evaluation code. See [No-code QA benchmarks](./cli/src/builtins/custom_nocode.rs) for more details. + ## Why Quantiles Evaluation workflows quickly outgrow one-off scripts once teams need caching, retries, dataset handling, metrics capture, and run comparison. Quantiles gives teams those primitives so they don't have to built them from scratch: @@ -117,7 +121,9 @@ Both the CLI and Python SDK support offline benchmark workflows, including the f ## Built-in Benchmarks -Built-in benchmarks are ready-to-run evalulations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark. +Built-in benchmarks are ready-to-run evaluations with predefined datasets, scoring methodologies, and metrics. Use them when you want a standardized evaluation that provides a common reference point, a repeatable baseline, or a well-defined implementation of an industry benchmark. + +For dataset-backed QA checks that do not need custom Python or TypeScript code, use a `custom_nocode` benchmark in `quantiles.toml`. A `custom_nocode` QA benchmark points to a dataset, a Jinja prompt template, the dataset column containing the prompt, and the dataset column containing the golden answer. See [`custom-nocode-examples/quantiles.toml`](./custom-nocode-examples/quantiles.toml) for a minimal example. | Code | When to use | | --- | --- | diff --git a/cli/Cargo.lock b/cli/Cargo.lock index c34372b..3c83a5f 100644 --- a/cli/Cargo.lock +++ b/cli/Cargo.lock @@ -3074,6 +3074,19 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jinja" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4eef51ad9c68d377a1604706ae4121b1b766c1504c86cb5247f4b6946cba9f1" +dependencies = [ + "indexmap 2.14.0", + "minijinja", + "minijinja-contrib", + "serde", + "serde_json", +] + [[package]] name = "jni" version = "0.22.4" @@ -3411,6 +3424,12 @@ version = "2.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6b947ae49db0d222b1dbc6b113ce7248a3fc3a6ca21b696717bfc000ba4484d8" +[[package]] +name = "memo-map" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38d1115007560874e373613744c6fba374c17688327a71c1476d1a5954cc857b" + [[package]] name = "memoffset" version = "0.9.1" @@ -3436,6 +3455,35 @@ dependencies = [ "unicase", ] +[[package]] +name = "minijinja" +version = "2.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb3d648e68cea56d9858d535ee28f9538404e2dd8cb08ed0bd05dca379477f39" +dependencies = [ + "aho-corasick", + "indexmap 2.14.0", + "memo-map", + "percent-encoding", + "serde", + "serde_json", + "unicase", + "unicode-ident", + "v_htmlescape", +] + +[[package]] +name = "minijinja-contrib" +version = "2.21.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85342f6fac0be8ccd5bd00d9066be538f34f393f577b75d81b17c8398a6b43bb" +dependencies = [ + "minijinja", + "serde", + "textwrap", + "unicode_categories", +] + [[package]] name = "minimal-lexical" version = "0.2.1" @@ -4296,6 +4344,7 @@ dependencies = [ "fastembed", "futures", "genai", + "jinja", "parquet", "predicates", "rand 0.8.6", @@ -5399,6 +5448,12 @@ dependencies = [ "serde", ] +[[package]] +name = "smawk" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8e2fb0f499abb4d162f2bedad68f5ef91a1682b5a03596ddb67efd37768d100" + [[package]] name = "snafu" version = "0.8.9" @@ -5871,6 +5926,15 @@ version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f50febec83f5ee1df3015341d8bd429f2d1cc62bcba7ea2076759d315084683" +[[package]] +name = "textwrap" +version = "0.16.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c13547615a44dc9c452a8a534638acdf07120d4b6847c8178705da06306a3057" +dependencies = [ + "smawk", +] + [[package]] name = "thiserror" version = "2.0.18" @@ -6444,6 +6508,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "v_htmlescape" +version = "0.15.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e8257fbc510f0a46eb602c10215901938b5c2a7d5e70fc11483b1d3c9b5b18c" + [[package]] name = "value-ext" version = "0.1.3" diff --git a/cli/Cargo.toml b/cli/Cargo.toml index efd9c27..631f074 100644 --- a/cli/Cargo.toml +++ b/cli/Cargo.toml @@ -15,6 +15,7 @@ dirs = "6" fastembed = "5" futures = "0.3" genai = "0.6.5" +jinja = "0.1" parquet = { version = "54", features = ["arrow"] } rand = "0.8" reqwest = { version = "0.13.3", features = ["json", "rustls"] } diff --git a/cli/README.md b/cli/README.md index eb556f5..2ce5c51 100644 --- a/cli/README.md +++ b/cli/README.md @@ -53,7 +53,7 @@ See [`examples/configs/custom_code/quantiles.toml`](./examples/configs/custom_co ## Configuration files and customization -You can customize how the CLI executes built-in benchmarks and custom evaluations using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples: +You can customize how the CLI executes built-in benchmarks, custom code evaluations, and no-code QA benchmarks using a `quantiles.toml` or `.quantiles.toml` configuration file. See the following resources for information and examples: - [`../CONFIG.md`](../CONFIG.md): for a guide and reference. - [`./examples/configs`](./examples/configs) for complete working examples. @@ -71,6 +71,28 @@ max_workers = 100 >Note: Quantiles is designed for high-throughput execution and may issue many requests in parallel. Depending on your provider, model, and account limits, benchmark runs can quickly hit API rate limits or concurrency quotas. Consider reducing concurrency or using models/providers with higher rate limits if you encounter throttling. Example configurations illustrate how to do so. +### No-code QA benchmarks + +For dataset-backed QA checks that do not need custom evaluation code, set `type = "custom_nocode"` and `style = "qa"`. The benchmark runs inside the CLI, renders each prompt with the configured Jinja template, calls the configured model, and scores each row with exact-match accuracy against the golden answer column. + +```toml +[benchmarks.nocode_custom] +type = "custom_nocode" +style = "qa" +dataset = "quantiles/simpleqa-verified" +model = "random" +prompt_template_file = "prompts/qa.txt" +prompt_column = "problem" +golden_column = "answer" +limit = 10 +``` + +```bash +qt run nocode_custom +``` + +See [`../custom-nocode-examples/quantiles.toml`](../custom-nocode-examples/quantiles.toml) for a complete minimal example. + ### Custom code evals For custom evaluations, set `type = "custom_code"` and provide the `command` to run. The optional `input` table is passed to your script as a JSON dictionary. diff --git a/cli/src/builtins/common.rs b/cli/src/builtins/common.rs index 53ea370..e8b71e6 100644 --- a/cli/src/builtins/common.rs +++ b/cli/src/builtins/common.rs @@ -1,13 +1,15 @@ +use std::collections::hash_map::DefaultHasher; +use std::hash::{Hash, Hasher}; +use std::sync::Arc; +use std::time::Instant; + use anyhow::{Context, Result}; use sea_orm::DatabaseConnection; use serde::{Deserialize, Serialize}; use serde_json::Value; -use std::collections::hash_map::DefaultHasher; -use std::hash::{Hash, Hasher}; -use std::time::Instant; use crate::db::steps::{self, StepDecision}; -use crate::llm::Sampler; +use crate::llm::{LLMSampler, Sampler}; use crate::metrics_store::MetricsStore; /// Fields shared by every builtin benchmark config. When adding a new builtin, @@ -119,6 +121,48 @@ pub(crate) fn get_max_workers() -> usize { } } +/// Resolve a model sampler, falling back to a default when none is configured. +pub(crate) fn resolve_sampler( + model: Option<&Sampler>, + default: impl FnOnce() -> Arc, +) -> Result> { + match model { + None => Ok(default()), + Some(sampler) => sampler.resolve(), + } +} + +/// Emit aggregate `accuracy`, `correct_count`, and `total_count` metrics from a +/// collection of per-sample boolean correctness values. +#[expect(clippy::cast_precision_loss)] +pub(crate) async fn emit_accuracy_metrics( + metrics_store: &MetricsStore, + run_id: i64, + results: impl IntoIterator, +) { + let mut correct_count = 0usize; + let mut total_count = 0usize; + for is_correct in results { + total_count += 1; + if is_correct { + correct_count += 1; + } + } + + if total_count > 0 { + let accuracy = correct_count as f64 / total_count as f64; + metrics_store + .emit(run_id, None, "accuracy", accuracy, None) + .await; + metrics_store + .emit(run_id, None, "correct_count", correct_count as f64, None) + .await; + metrics_store + .emit(run_id, None, "total_count", total_count as f64, None) + .await; + } +} + /// Statistics computed from a collection of similarity scores. #[derive(Debug)] pub(crate) struct ScoreStatistics { diff --git a/cli/src/builtins/custom_nocode.rs b/cli/src/builtins/custom_nocode.rs new file mode 100644 index 0000000..3f1bf33 --- /dev/null +++ b/cli/src/builtins/custom_nocode.rs @@ -0,0 +1,503 @@ +use std::sync::Arc; + +use anyhow::{Context, Result, bail}; +use serde::{Deserialize, Serialize}; + +use crate::builtins::common::{ + emit_accuracy_metrics, extract_text, get_max_workers, hash_input, resolve_sampler, + run_timed_step, +}; +use crate::builtins::dataset_runner::DatasetRunner; +use crate::builtins::input::set_builtin_run_input; +use crate::builtins::output::set_builtin_run_output; +use crate::builtins::{BuiltinContext, BuiltinWorkflow}; +use crate::dataset::DatasetManager; +use crate::llm::random::RandomSampler; + +/// Input deserialized from the JSON assembled by `commands::run`. +#[derive(Debug, Deserialize)] +struct CustomNoCodeInput { + style: crate::config::CustomNoCodeStyle, + dataset: String, + #[serde(default)] + model: Option, + #[serde(flatten)] + qa: crate::config::CustomNoCodeQaConfig, +} + +/// Per-row step output stored as JSON in the step record. +#[derive(Debug, Serialize, Deserialize)] +struct RowOutput { + input: String, + response: String, + golden: String, + is_correct: bool, +} + +/// Arguments for the [`CustomNoCodeBuiltin::evaluate_row`] method +struct EvaluateRowArgs<'a> { + /// The row index + i: usize, + /// The row value + row: &'a serde_json::Value, + /// The name of the column in this row that has the prompt + prompt_column: &'a str, + /// The name of the column in this row that has the golden answer + golden_column: &'a str, + /// The prompt template + template_str: &'a str, + /// The pre-constructed jinja template env + env: &'a jinja::Environment<'a>, + /// The name of the model we're sampling (used for cache key hashing only) + model_name: &'a str, + /// The actual model to sample + llm: &'a std::sync::Arc, + /// The connection to the metadata DB + db: &'a sea_orm::DatabaseConnection, + /// Metrics storage + metrics_store: &'a crate::metrics_store::MetricsStore, + /// The run ID + run_id: i64, +} + +/// No-code custom benchmark builtin. +pub struct CustomNoCodeBuiltin { + name: String, +} + +impl CustomNoCodeBuiltin { + /// Create a new builtin with the workflow name from the config file. + #[must_use] + pub fn new(name: String) -> Self { + Self { name } + } + + async fn evaluate_row(&self, args: EvaluateRowArgs<'_>) -> Result { + let prompt = extract_text(args.row, args.prompt_column).with_context(|| { + format!( + "row {}: missing prompt column `{}`", + args.i, args.golden_column + ) + })?; + let golden = extract_text(args.row, args.golden_column).with_context(|| { + format!( + "row {}: missing golden column `{}`", + args.i, args.golden_column + ) + })?; + + let rendered = args + .env + .render_str(args.template_str, jinja::context!(prompt => &prompt)) + .with_context(|| format!("row {}: failed to render prompt template", args.i))?; + + let input_hash = hash_input(&format!( + "{rendered}\nmodel={}\nworkflow={}", + args.model_name, + self.name() + )); + let step_key = format!("row-{}", args.i); + + let (output, step_id) = run_timed_step( + args.db, + args.metrics_store, + args.run_id, + &step_key, + &input_hash, + async { + let model_response = args + .llm + .sample(&rendered) + .await + .with_context(|| format!("failed to sample LLM for row {}", args.i))?; + + let is_correct = is_exact_match(&model_response, &golden); + + Ok(RowOutput { + input: rendered.clone(), + response: model_response, + golden, + is_correct, + }) + }, + ) + .await?; + + if let Some(step_id) = step_id { + args.metrics_store + .emit( + args.run_id, + Some(step_id), + "is_correct", + if output.is_correct { 1.0 } else { 0.0 }, + None, + ) + .await; + } + + Ok(output.is_correct) + } +} + +#[async_trait::async_trait] +impl BuiltinWorkflow for CustomNoCodeBuiltin { + fn name(&self) -> String { + self.name.clone() + } + + async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()> { + let config = parse_input(ctx.input)?; + let (template_str, env) = load_template(&config.qa.prompt_template_file)?; + let max_workers = config.qa.max_workers.unwrap_or_else(get_max_workers); + let llm = resolve_sampler(config.model.as_ref(), || Arc::new(RandomSampler::new(80)))?; + + let (manager, info, limit) = + resolve_dataset_limit(&config.dataset, config.qa.limit).await?; + set_builtin_run_input( + ctx.db, + ctx.run_id, + config.model.as_ref(), + limit, + config.qa.max_workers, + ) + .await?; + + let db = ctx.db.clone(); + let model_name = config + .model + .as_ref() + .map_or("random".to_string(), std::string::ToString::to_string); + let run_id = ctx.run_id; + let dataset = config.dataset.clone(); + let prompt_column = config.qa.prompt_column.clone(); + let golden_column = config.qa.golden_column.clone(); + let template_str = Arc::new(template_str); + + let name = self.name(); + let results = DatasetRunner::new(&manager, &dataset, &info, limit) + .desc(&name) + .set_quiet(ctx.quiet) + .for_each_concurrent(max_workers, move |i, row| { + let llm = Arc::clone(&llm); + let db = db.clone(); + let model_name = model_name.clone(); + let template_str = Arc::clone(&template_str); + let prompt_column = prompt_column.clone(); + let golden_column = golden_column.clone(); + let env = env.clone(); + async move { + let args = EvaluateRowArgs { + i, + row: &row, + prompt_column: &prompt_column, + golden_column: &golden_column, + template_str: &template_str, + env: &env, + model_name: &model_name, + llm: &llm, + db: &db, + metrics_store: ctx.metrics_store, + run_id, + }; + self.evaluate_row(args).await + } + }) + .await?; + + let total_count = results.len(); + emit_accuracy_metrics(ctx.metrics_store, ctx.run_id, results).await; + + set_builtin_run_output(ctx.db, ctx.run_id, total_count).await?; + + Ok(()) + } +} + +fn parse_input(input: Option<&str>) -> Result { + let config: CustomNoCodeInput = input + .map(serde_json::from_str) + .transpose() + .context("invalid builtin input JSON")? + .context("custom_nocode benchmark requires input configuration")?; + + match config.style { + crate::config::CustomNoCodeStyle::Qa => {} + } + + if config.qa.limit == Some(0) { + bail!("limit must be > 0"); + } + + Ok(config) +} + +fn load_template(path: &str) -> Result<(String, jinja::Environment<'_>)> { + let template_str = std::fs::read_to_string(path) + .with_context(|| format!("failed to read prompt template file `{path}`"))?; + let env = jinja::Environment::new(); + env.render_str(&template_str, jinja::context!(prompt => "")) + .with_context(|| format!("invalid jinja syntax in prompt template file `{path}`"))?; + Ok((template_str, env)) +} + +async fn resolve_dataset_limit( + dataset: &str, + limit: Option, +) -> Result<(DatasetManager, crate::dataset::DatasetInfo, usize)> { + let manager = DatasetManager::new()?; + let info = manager.init(dataset, None, None, None).await?; + + let total = info + .total_rows + .context("could not determine dataset size; pass an explicit limit")?; + let limit = limit.unwrap_or(total).min(total); + + Ok((manager, info, limit)) +} + +/// Case-sensitive exact-match comparison after trimming whitespace. +fn is_exact_match(response: &str, golden: &str) -> bool { + response.trim() == golden.trim() +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + + #[test] + fn builtin_name_returns_configured_name() { + let builtin = CustomNoCodeBuiltin::new("my-benchmark".to_owned()); + assert_eq!(builtin.name(), "my-benchmark"); + } + + #[test] + fn render_template_with_prompt_variable() { + let template = "Answer this: {{ prompt }}"; + let env = jinja::Environment::new(); + let rendered = env + .render_str(template, jinja::context!(prompt => "what is 2+2")) + .unwrap(); + assert_eq!(rendered, "Answer this: what is 2+2"); + } + + #[test] + fn render_template_preserves_newlines() { + let template = "Question:\n{{ prompt }}\nAnswer:"; + let env = jinja::Environment::new(); + let rendered = env + .render_str(template, jinja::context!(prompt => "hello")) + .unwrap(); + assert_eq!(rendered, "Question:\nhello\nAnswer:"); + } + + #[test] + fn exact_match_case_sensitive() { + assert!(is_exact_match("hello", "hello")); + assert!(!is_exact_match("Hello", "hello")); + } + + #[test] + fn exact_match_trims_whitespace() { + assert!(is_exact_match(" hello ", "hello")); + assert!(is_exact_match("hello", " hello ")); + } + + #[tokio::test] + async fn execute_rejects_invalid_jinja_template() { + let tmpdir = tempfile::tempdir().unwrap(); + let root = tmpdir.path(); + crate::db::init_workspace(root).await.unwrap(); + let db = crate::db::open_workspace(root).await.unwrap(); + let metrics_store = + crate::metrics_store::MetricsStore::new(crate::db::metrics_dir(root)).unwrap(); + + let template_path = root.join("bad.txt"); + std::fs::write(&template_path, "{{ unclosed").unwrap(); + + let input_json = serde_json::to_string(&json!({ + "style": "qa", + "dataset": "fixture/qa", + "prompt_template_file": template_path.to_str().unwrap(), + "prompt_column": "q", + "golden_column": "a", + })) + .unwrap(); + + let run_id = crate::db::create_run(&db, "test", Some(&input_json)) + .await + .unwrap(); + + let workflow_name = "test".to_owned(); + let builtin = CustomNoCodeBuiltin::new(workflow_name.clone()); + let result = builtin + .execute(super::BuiltinContext { + db: &db, + metrics_store: &metrics_store, + run_id, + workflow_name: &workflow_name, + input: Some(&input_json), + quiet: true, + }) + .await; + + assert!(result.is_err()); + let msg = result.unwrap_err().to_string(); + assert!( + msg.contains("invalid jinja syntax"), + "unexpected error: {msg}" + ); + } + + #[expect(clippy::too_many_lines)] + #[expect(clippy::cast_possible_truncation)] + #[tokio::test] + async fn execute_records_metrics_and_steps_with_fixture() { + let server = MockServer::start().await; + let tmpdir = tempfile::tempdir().unwrap(); + let root = tmpdir.path(); + let cache_dir = root.join("cache"); + + // Save original env var values so we can restore them later. + let orig_hf = std::env::var("HF_DATASETS_SERVER").ok(); + let orig_cache = std::env::var("QUANTILES_DATASET_CACHE_DIR").ok(); + unsafe { + std::env::set_var("HF_DATASETS_SERVER", server.uri()); + std::env::set_var("QUANTILES_DATASET_CACHE_DIR", cache_dir.as_os_str()); + } + + // Mock HF dataset server endpoints used by DatasetManager::init(). + Mock::given(method("GET")) + .and(path("/splits")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "splits": [{"config": "default", "split": "train"}] + }))) + .mount(&server) + .await; + + Mock::given(method("GET")) + .and(path("/size")) + .respond_with(ResponseTemplate::new(200).set_body_json(json!({ + "size": {"splits": [{"num_rows": 2}]} + }))) + .mount(&server) + .await; + + // Initialize workspace with SQLite DB and metrics dir. + crate::db::init_workspace(root).await.unwrap(); + let db = crate::db::open_workspace(root).await.unwrap(); + let metrics_store = + crate::metrics_store::MetricsStore::new(crate::db::metrics_dir(root)).unwrap(); + + // Write a Jinja template file. + let template_path = root.join("template.txt"); + std::fs::write(&template_path, "{{ prompt }}\nAnswer:").unwrap(); + + // Pre-populate the dataset cache so no network fetch is needed for rows. + let cache = crate::dataset::cache::DatasetCache::new(cache_dir); + let rows = vec![ + json!({"question": "what is 2+2", "answer": "4"}), + json!({"question": "what is 3+3", "answer": "6"}), + ]; + let key = crate::dataset::cache::cache_key("fixture/qa", "default", "train", None); + let batch_path = cache.batch_path(&key, 0, 2); + cache.write_batch(&batch_path, &rows).await.unwrap(); + + // Assemble the input JSON that execute() expects. + let input_json = serde_json::to_string(&json!({ + "style": "qa", + "dataset": "fixture/qa", + "model": "random", + "prompt_template_file": template_path.to_str().unwrap(), + "prompt_column": "question", + "golden_column": "answer", + "limit": 2, + })) + .unwrap(); + + let run_id = crate::db::create_run(&db, "test_nocode", Some(&input_json)) + .await + .unwrap(); + + let workflow_name = "test_nocode".to_owned(); + let builtin = CustomNoCodeBuiltin::new(workflow_name.clone()); + builtin + .execute(super::BuiltinContext { + db: &db, + metrics_store: &metrics_store, + run_id, + workflow_name: &workflow_name, + input: Some(&input_json), + quiet: true, + }) + .await + .unwrap(); + + // Flush buffered metrics to Parquet so we can read them back. + metrics_store.flush(run_id).await.unwrap(); + + // Verify aggregate metrics were written. + let agg = metrics_store.list_aggregate_for_run(run_id).await.unwrap(); + let names: Vec<&str> = agg.iter().map(|m| m.metric_name.as_str()).collect(); + assert!(names.contains(&"accuracy")); + assert!(names.contains(&"correct_count")); + assert!(names.contains(&"total_count")); + + let total_metric = agg.iter().find(|m| m.metric_name == "total_count").unwrap(); + assert_eq!(total_metric.metric_value as i64, 2); + + // Random sampler responses won't match "4" or "6", so correctness is 0. + let correct_metric = agg + .iter() + .find(|m| m.metric_name == "correct_count") + .unwrap(); + assert_eq!(correct_metric.metric_value as i64, 0); + + // Verify per-step metrics were recorded for both rows. + let all_metrics = metrics_store.list_for_run(run_id).await.unwrap(); + let is_correct_count = all_metrics + .iter() + .filter(|m| m.metric_name == "is_correct") + .count(); + assert_eq!(is_correct_count, 2); + + // Verify steps were persisted in SQLite. + let steps = crate::db::list_steps_for_run(&db, run_id).await.unwrap(); + assert_eq!(steps.len(), 2); + + // Execute a second time to verify step caching reuses existing records. + let builtin2 = CustomNoCodeBuiltin::new(workflow_name.clone()); + builtin2 + .execute(super::BuiltinContext { + db: &db, + metrics_store: &metrics_store, + run_id, + workflow_name: &workflow_name, + input: Some(&input_json), + quiet: true, + }) + .await + .unwrap(); + + let steps2 = crate::db::list_steps_for_run(&db, run_id).await.unwrap(); + assert_eq!( + steps2.len(), + 2, + "second execution should reuse cached steps instead of creating new ones" + ); + + // Restore environment variables. + unsafe { + match &orig_hf { + Some(v) => std::env::set_var("HF_DATASETS_SERVER", v), + None => std::env::remove_var("HF_DATASETS_SERVER"), + } + match &orig_cache { + Some(v) => std::env::set_var("QUANTILES_DATASET_CACHE_DIR", v), + None => std::env::remove_var("QUANTILES_DATASET_CACHE_DIR"), + } + } + } +} diff --git a/cli/src/builtins/mod.rs b/cli/src/builtins/mod.rs index 0611a72..2951f7a 100644 --- a/cli/src/builtins/mod.rs +++ b/cli/src/builtins/mod.rs @@ -1,4 +1,5 @@ mod common; +mod custom_nocode; mod dataset_runner; mod financebench; mod input; @@ -7,6 +8,8 @@ mod pubmedqa; mod similarity; mod simpleqa_verified; +pub use custom_nocode::CustomNoCodeBuiltin; + use anyhow::Result; use async_trait::async_trait; use sea_orm::DatabaseConnection; @@ -27,8 +30,8 @@ pub struct BuiltinContext<'a> { /// Trait for builtin evals that run natively inside the CLI. #[async_trait] pub trait BuiltinWorkflow: Send + Sync { - /// Unique name of the builtin (e.g. "financebench"). - fn name(&self) -> &'static str; + /// Unique name of the builtin (e.g. "simpleqa-verified", "financebench"). + fn name(&self) -> String; /// Execute the builtin eval. async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()>; } diff --git a/cli/src/builtins/pubmedqa/mod.rs b/cli/src/builtins/pubmedqa/mod.rs index 7ea162c..4e0df77 100644 --- a/cli/src/builtins/pubmedqa/mod.rs +++ b/cli/src/builtins/pubmedqa/mod.rs @@ -2,13 +2,14 @@ use std::sync::Arc; use anyhow::{Context, Result, bail}; -use crate::builtins::common::{get_max_workers, hash_input, run_timed_step}; +use crate::builtins::common::{ + emit_accuracy_metrics, get_max_workers, hash_input, resolve_sampler, run_timed_step, +}; use crate::builtins::dataset_runner::DatasetRunner; use crate::builtins::input::set_builtin_run_input; use crate::builtins::output::set_builtin_run_output; use crate::builtins::{BuiltinContext, BuiltinWorkflow}; use crate::dataset::DatasetManager; -use crate::llm::LLMSampler; use crate::llm::random_label::RandomLabelSampler; use config::{PubMedQAConfig, RowOutput}; @@ -22,11 +23,10 @@ mod eval; /// `PubMedQA` builtin using the quantiles/PubMedQA dataset. pub struct PubmedqaBuiltin; -#[expect(clippy::too_many_lines)] #[async_trait::async_trait] impl BuiltinWorkflow for PubmedqaBuiltin { - fn name(&self) -> &'static str { - "pubmedqa" + fn name(&self) -> String { + "pubmedqa".to_string() } async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()> { @@ -43,10 +43,9 @@ impl BuiltinWorkflow for PubmedqaBuiltin { let max_workers = config.base.max_workers.unwrap_or_else(get_max_workers); - let llm: Arc = match config.base.model { - None => Arc::new(RandomLabelSampler::new(&["yes", "no", "maybe"])), - Some(ref sampler) => sampler.resolve()?, - }; + let llm = resolve_sampler(config.base.model.as_ref(), || { + Arc::new(RandomLabelSampler::new(&["yes", "no", "maybe"])) + })?; let manager = DatasetManager::new()?; let dataset_id = "quantiles/PubMedQA"; @@ -72,8 +71,9 @@ impl BuiltinWorkflow for PubmedqaBuiltin { let model = config.base.model.clone(); let run_id = ctx.run_id; + let name = self.name(); let results = DatasetRunner::new(&manager, dataset_id, &info, limit) - .desc(self.name()) + .desc(&name) .set_quiet(ctx.quiet) .for_each_concurrent(max_workers, move |i, row| { let llm = Arc::clone(&llm); @@ -137,34 +137,8 @@ impl BuiltinWorkflow for PubmedqaBuiltin { }) .await?; - let mut correct_count: usize = 0; let total_count = results.len(); - for is_correct in results { - if is_correct { - correct_count += 1; - } - } - - #[expect(clippy::cast_precision_loss)] - if total_count > 0 { - let accuracy = correct_count as f64 / total_count as f64; - - ctx.metrics_store - .emit(ctx.run_id, None, "accuracy", accuracy, None) - .await; - ctx.metrics_store - .emit( - ctx.run_id, - None, - "correct_count", - correct_count as f64, - None, - ) - .await; - ctx.metrics_store - .emit(ctx.run_id, None, "total_count", total_count as f64, None) - .await; - } + emit_accuracy_metrics(ctx.metrics_store, ctx.run_id, results).await; set_builtin_run_output(ctx.db, ctx.run_id, total_count).await?; diff --git a/cli/src/builtins/similarity.rs b/cli/src/builtins/similarity.rs index 051a260..1c8fe0f 100644 --- a/cli/src/builtins/similarity.rs +++ b/cli/src/builtins/similarity.rs @@ -3,14 +3,13 @@ use serde::{Deserialize, Serialize}; use std::sync::Arc; use crate::builtins::common::{ - compute_statistics, extract_text, get_max_workers, hash_input, run_timed_step, + compute_statistics, extract_text, get_max_workers, hash_input, resolve_sampler, run_timed_step, }; use crate::builtins::dataset_runner::DatasetRunner; use crate::builtins::input::set_builtin_run_input; use crate::builtins::output::set_builtin_run_output; use crate::builtins::{BuiltinContext, BuiltinWorkflow}; use crate::dataset::DatasetManager; -use crate::llm::LLMSampler; use crate::llm::random::RandomSampler; use crate::similarity::{ SimilarityMetric, SimilarityMetricName, levenshtein::LevenshteinSimilarity, @@ -69,8 +68,8 @@ pub const FINANCEBENCH: SimilarityBenchmark = SimilarityBenchmark { #[expect(clippy::too_many_lines)] #[async_trait::async_trait] impl BuiltinWorkflow for SimilarityBenchmark { - fn name(&self) -> &'static str { - self.name + fn name(&self) -> String { + self.name.to_string() } async fn execute(&self, ctx: BuiltinContext<'_>) -> Result<()> { @@ -90,10 +89,9 @@ impl BuiltinWorkflow for SimilarityBenchmark { SimilarityMetricName::Cosine => Box::new(CosineSimilarity::try_new()?), }; - let llm: Arc = match config.base.model { - None => Arc::new(RandomSampler::new(80)), - Some(ref sampler) => sampler.resolve()?, - }; + let llm = resolve_sampler(config.base.model.as_ref(), || { + Arc::new(RandomSampler::new(80)) + })?; let manager = DatasetManager::new()?; let info = manager.init(self.dataset_id, None, None, None).await?; diff --git a/cli/src/commands/compare/report.rs b/cli/src/commands/compare/report.rs index 3520e72..e2a70b3 100644 --- a/cli/src/commands/compare/report.rs +++ b/cli/src/commands/compare/report.rs @@ -341,6 +341,93 @@ mod tests { assert!(json.get("warning").is_none()); } + /// Comparing two actual `custom_nocode` runs (differing accuracy, + /// identical `total_count`) should produce a `CompareReport` that correctly + /// flags the changed metric while keeping the unchanged one as "same". + #[tokio::test] + #[expect(clippy::items_after_statements)] + async fn compare_two_custom_nocode_runs() { + let tmpdir = tempfile::tempdir().unwrap(); + let root = tmpdir.path(); + + qt::db::init_workspace(root).await.unwrap(); + let db = qt::db::open_workspace(root).await.unwrap(); + let metrics_store = + qt::metrics_store::MetricsStore::new(qt::db::metrics_dir(root)).unwrap(); + + async fn seed_run( + db: &sea_orm::DatabaseConnection, + ms: &qt::metrics_store::MetricsStore, + accuracy: f64, + total_count: f64, + ) -> i64 { + let run_id = qt::db::create_run(db, "nocode_custom", None).await.unwrap(); + ms.emit(run_id, None, "accuracy", accuracy, None).await; + ms.emit(run_id, None, "total_count", total_count, None) + .await; + ms.flush(run_id).await.unwrap(); + qt::db::complete_run(db, ms, run_id).await.unwrap(); + run_id + } + + let run_a = seed_run(&db, &metrics_store, 0.5, 10.0).await; + let run_b = seed_run(&db, &metrics_store, 0.7, 10.0).await; + + let data_a = RunData::fetch_by_id(&db, &metrics_store, run_a) + .await + .unwrap(); + let data_b = RunData::fetch_by_id(&db, &metrics_store, run_b) + .await + .unwrap(); + + let report = CompareReport::build(data_a, data_b); + + // The report should detect differences. + assert!(report.differs, "report should flag differing metrics"); + + // Both metrics should appear in the comparison. + assert_eq!(report.metrics.len(), 2, "expected 2 metrics compared"); + + // Accuracy differs between the two runs. + let accuracy_cmp = report + .metrics + .iter() + .find(|m| m.name == "accuracy") + .expect("accuracy metric should be present"); + assert!(!accuracy_cmp.same, "accuracy should differ"); + + // Total count is identical. + let total_cmp = report + .metrics + .iter() + .find(|m| m.name == "total_count") + .expect("total_count metric should be present"); + assert!(total_cmp.same, "total_count should be the same"); + + // Use JSON to verify the underlying values without touching private fields. + let json = serde_json::to_value(&report).expect("report should serialize"); + let metrics_json = json["metrics"].as_array().unwrap(); + let accuracy_json = metrics_json + .iter() + .find(|m| m["name"] == "accuracy") + .unwrap(); + assert_eq!(accuracy_json["run_a"], serde_json::json!([0.5])); + assert_eq!(accuracy_json["run_b"], serde_json::json!([0.7])); + + let total_json = metrics_json + .iter() + .find(|m| m["name"] == "total_count") + .unwrap(); + assert_eq!(total_json["run_a"], serde_json::json!([10.0])); + assert_eq!(total_json["run_b"], serde_json::json!([10.0])); + + // No benchmark-name warning since both runs share the same workflow. + assert!( + report.warning.is_none(), + "no warning expected for identical benchmark names" + ); + } + fn run_data(id: i64, workflow_name: &str) -> RunData { let now = OffsetDateTime::now_utc(); RunData { diff --git a/cli/src/commands/resume.rs b/cli/src/commands/resume.rs index 91b61af..027e287 100644 --- a/cli/src/commands/resume.rs +++ b/cli/src/commands/resume.rs @@ -1,6 +1,6 @@ use std::time::Instant; -use anyhow::{Result, bail}; +use anyhow::{Context, Result, bail}; use qt::builtins; use qt::db; @@ -36,10 +36,11 @@ pub(crate) fn plan_resume( Some(bench) => { bench.validate()?; match bench { - qt::config::BenchmarkConfig::Builtin(_) => Ok(ResumePlan::Builtin), qt::config::BenchmarkConfig::CustomCode(c) => { Ok(ResumePlan::CustomCode(c.command.clone())) } + qt::config::BenchmarkConfig::Builtin(_) + | qt::config::BenchmarkConfig::CustomNoCode(_) => Ok(ResumePlan::Builtin), } } None => { @@ -95,15 +96,23 @@ pub async fn resume(run_id: i64, json: bool, process_start: Instant) -> Result<( // wise to revisit this policy. match plan { ResumePlan::Builtin => { - super::run::execute_builtin( - &db, - &metrics_store, + let builtin: Box = match bench_config { + Some(qt::config::BenchmarkConfig::CustomNoCode(_)) => { + Box::new(builtins::CustomNoCodeBuiltin::new(workflow_name.to_owned())) + } + _ => builtins::resolve(workflow_name) + .with_context(|| format!("builtin `{workflow_name}` not found"))?, + }; + super::run::execute_builtin(super::run::ExecuteBuiltinArgs { + db: &db, + metrics_store: &metrics_store, run_id, workflow_name, - stored_input, + builtin, + input: stored_input, json, process_start, - ) + }) .await } ResumePlan::CustomCode(command) => { @@ -204,4 +213,170 @@ mod tests { let err = plan_resume("my-eval", &RunStatus::Failed, Some(&bench)).unwrap_err(); assert!(err.to_string().contains("non-empty `command`")); } + + /// A `custom_nocode` benchmark should plan to resume as a builtin so that the + /// CLI can re-run the no-code workflow natively without spawning an external command. + #[test] + fn plan_resume_custom_nocode_with_config() { + let file = tempfile::NamedTempFile::new().unwrap(); + let bench = + qt::config::BenchmarkConfig::CustomNoCode(qt::config::CustomNoCodeBenchmarkConfig { + type_: "custom_nocode".to_owned(), + style: qt::config::CustomNoCodeStyle::Qa, + dataset: "quantiles/simpleqa-verified".to_owned(), + model: Some(qt::llm::Sampler::Random {}), + qa: qt::config::CustomNoCodeQaConfig { + prompt_template_file: file.path().to_str().unwrap().to_owned(), + prompt_column: "problem".to_owned(), + golden_column: "answer".to_owned(), + limit: None, + max_workers: None, + }, + }); + let plan = plan_resume("nocode_custom", &RunStatus::Failed, Some(&bench)).unwrap(); + assert!(matches!(plan, ResumePlan::Builtin)); + } + + /// A failed `custom_nocode` run can be resumed and re-execute successfully + /// through the `CustomNoCodeBuiltin`, verifying that the resume path wires + /// the correct builtin and `ExecuteBuiltinArgs`. + #[expect(clippy::too_many_lines)] + #[tokio::test] + async fn resume_failed_custom_nocode_run_re_executes_successfully() { + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + + let server = MockServer::start().await; + let tmpdir = tempfile::tempdir().unwrap(); + let root = tmpdir.path(); + let cache_dir = root.join("cache"); + + // Save original env var values so we can restore them later. + let orig_hf = std::env::var("HF_DATASETS_SERVER").ok(); + let orig_cache = std::env::var("QUANTILES_DATASET_CACHE_DIR").ok(); + unsafe { + std::env::set_var("HF_DATASETS_SERVER", server.uri()); + std::env::set_var("QUANTILES_DATASET_CACHE_DIR", cache_dir.as_os_str()); + } + + // Mock HF dataset server endpoints used by DatasetManager::init(). + Mock::given(method("GET")) + .and(path("/splits")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "splits": [{"config": "default", "split": "train"}] + }))) + .mount(&server) + .await; + + Mock::given(method("GET")) + .and(path("/size")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "size": {"splits": [{"num_rows": 2}]} + }))) + .mount(&server) + .await; + + // Initialize workspace with SQLite DB and metrics dir. + qt::db::init_workspace(root).await.unwrap(); + let db = qt::db::open_workspace(root).await.unwrap(); + let metrics_store = + qt::metrics_store::MetricsStore::new(qt::db::metrics_dir(root)).unwrap(); + + // Write a Jinja template file. + let template_path = root.join("template.txt"); + std::fs::write(&template_path, "{{ prompt }}\nAnswer:").unwrap(); + + // Pre-populate the dataset cache so no network fetch is needed for rows. + let cache = qt::dataset::cache::DatasetCache::new(cache_dir); + let rows = vec![ + serde_json::json!({"question": "what is 2+2", "answer": "4"}), + serde_json::json!({"question": "what is 3+3", "answer": "6"}), + ]; + let key = qt::dataset::cache::cache_key("fixture/qa", "default", "train", None); + let batch_path = cache.batch_path(&key, 0, 2); + cache.write_batch(&batch_path, &rows).await.unwrap(); + + // Write a quantiles.toml so resume can load config. + std::fs::write( + root.join("quantiles.toml"), + format!( + r#" +[benchmarks.nocode_resume_test] +type = "custom_nocode" +style = "qa" +dataset = "fixture/qa" +model = "random" +prompt_template_file = "{}" +prompt_column = "question" +golden_column = "answer" +limit = 2 +"#, + template_path.to_str().unwrap() + ), + ) + .unwrap(); + + let input_json = serde_json::to_string(&serde_json::json!({ + "style": "qa", + "dataset": "fixture/qa", + "model": "random", + "prompt_template_file": template_path.to_str().unwrap(), + "prompt_column": "question", + "golden_column": "answer", + "limit": 2, + })) + .unwrap(); + + let workflow_name = "nocode_resume_test"; + let run_id = qt::db::create_run(&db, workflow_name, Some(&input_json)) + .await + .unwrap(); + + // Mark the run as failed so it can be resumed. + qt::db::fail_run(&db, &metrics_store, run_id, "simulated failure") + .await + .unwrap(); + + let run_before = qt::db::get_run(&db, run_id).await.unwrap(); + assert_eq!(run_before.status, qt::db::RunStatus::Failed); + + // Change cwd so resume finds the config file. + let orig_cwd = std::env::current_dir().unwrap(); + std::env::set_current_dir(root).unwrap(); + + resume(run_id, true, std::time::Instant::now()) + .await + .unwrap(); + + // Restore cwd. + std::env::set_current_dir(orig_cwd).unwrap(); + + // Verify the run is now completed. + let run_after = qt::db::get_run(&db, run_id).await.unwrap(); + assert_eq!(run_after.status, qt::db::RunStatus::Completed); + + // Verify aggregate metrics were written. + metrics_store.flush(run_id).await.unwrap(); + let agg = metrics_store.list_aggregate_for_run(run_id).await.unwrap(); + let names: Vec<&str> = agg.iter().map(|m| m.metric_name.as_str()).collect(); + assert!(names.contains(&"accuracy")); + assert!(names.contains(&"correct_count")); + assert!(names.contains(&"total_count")); + + // Verify steps were persisted in SQLite. + let steps = qt::db::list_steps_for_run(&db, run_id).await.unwrap(); + assert_eq!(steps.len(), 2); + + // Restore environment variables. + unsafe { + match &orig_hf { + Some(v) => std::env::set_var("HF_DATASETS_SERVER", v), + None => std::env::remove_var("HF_DATASETS_SERVER"), + } + match &orig_cache { + Some(v) => std::env::set_var("QUANTILES_DATASET_CACHE_DIR", v), + None => std::env::remove_var("QUANTILES_DATASET_CACHE_DIR"), + } + } + } } diff --git a/cli/src/commands/run.rs b/cli/src/commands/run.rs index 18fcb06..7bcd87c 100644 --- a/cli/src/commands/run.rs +++ b/cli/src/commands/run.rs @@ -75,6 +75,34 @@ pub async fn run( ) .await } + qt::config::BenchmarkConfig::CustomNoCode(c) => { + let input = assemble_custom_nocode_input(c, cli_input); + + let cwd = std::env::current_dir()?; + let root = db::resolve_workspace_root(&cwd, true).await?; + let db = db::open_workspace(&root).await?; + let metrics_store = MetricsStore::new(db::metrics_dir(&root))?; + let run_id = db::create_run(&db, workflow_name, Some(&input)).await?; + + if !json { + println!("Created run {run_id}"); + } + + let builtin = Box::new(qt::builtins::CustomNoCodeBuiltin::new( + workflow_name.to_owned(), + )); + execute_builtin(ExecuteBuiltinArgs { + db: &db, + metrics_store: &metrics_store, + run_id, + workflow_name, + builtin, + input: Some(&input), + json, + process_start, + }) + .await + } } } None => { @@ -122,6 +150,45 @@ fn assemble_builtin_input( } } +/// Config input shape for `custom_nocode` benchmarks auto-generated from +/// `quantiles.toml` `[benchmarks.*]`. +#[derive(Serialize)] +struct CustomNoCodeConfigInput<'a> { + style: &'a str, + dataset: &'a str, + #[serde(skip_serializing_if = "Option::is_none")] + model: &'a Option, + prompt_template_file: &'a str, + prompt_column: &'a str, + golden_column: &'a str, + #[serde(skip_serializing_if = "Option::is_none")] + limit: Option, + #[serde(skip_serializing_if = "Option::is_none")] + max_workers: Option, +} + +fn assemble_custom_nocode_input( + bench: &qt::config::CustomNoCodeBenchmarkConfig, + cli_input: Option<&str>, +) -> String { + if let Some(cli_str) = cli_input { + return cli_str.to_owned(); + } + + let input = CustomNoCodeConfigInput { + style: "qa", + dataset: &bench.dataset, + model: &bench.model, + prompt_template_file: &bench.qa.prompt_template_file, + prompt_column: &bench.qa.prompt_column, + golden_column: &bench.qa.golden_column, + limit: bench.qa.limit, + max_workers: bench.qa.max_workers, + }; + + serde_json::to_string(&input).expect("infallible serialization") +} + fn merge_inputs( config_input: Option<&HashMap>, cli_input: Option<&str>, @@ -162,6 +229,9 @@ async fn run_builtin_workflow( json: bool, process_start: Instant, ) -> Result<()> { + let builtin = builtins::resolve(workflow_name) + .with_context(|| format!("builtin `{workflow_name}` not found"))?; + let cwd = std::env::current_dir()?; let root = db::resolve_workspace_root(&cwd, true).await?; let db = db::open_workspace(&root).await?; @@ -172,78 +242,88 @@ async fn run_builtin_workflow( println!("Created run {run_id}"); } - execute_builtin( - &db, - &metrics_store, + execute_builtin(ExecuteBuiltinArgs { + db: &db, + metrics_store: &metrics_store, run_id, workflow_name, + builtin, input, json, process_start, - ) + }) .await } -pub async fn execute_builtin( - db: &DatabaseConnection, - metrics_store: &MetricsStore, - run_id: i64, - workflow_name: &str, - input: Option<&str>, - json: bool, - process_start: Instant, -) -> Result<()> { - let builtin = builtins::resolve(workflow_name) - .with_context(|| format!("builtin `{workflow_name}` not found"))?; +/// Arguments for the [`execute_builtin`] function. +pub struct ExecuteBuiltinArgs<'a> { + pub db: &'a DatabaseConnection, + pub metrics_store: &'a MetricsStore, + pub run_id: i64, + pub workflow_name: &'a str, + pub builtin: Box, + pub input: Option<&'a str>, + pub json: bool, + pub process_start: Instant, +} - let builtin_result = builtin +pub async fn execute_builtin(args: ExecuteBuiltinArgs<'_>) -> Result<()> { + let builtin_result = args + .builtin .execute(builtins::BuiltinContext { - db, - metrics_store, - run_id, - workflow_name, - input, - quiet: json, + db: args.db, + metrics_store: args.metrics_store, + run_id: args.run_id, + workflow_name: args.workflow_name, + input: args.input, + quiet: args.json, }) .await; - let duration = process_start.elapsed(); + let duration = args.process_start.elapsed(); match &builtin_result { Ok(()) => { - db::complete_run(db, metrics_store, run_id).await?; - if !json { + db::complete_run(args.db, args.metrics_store, args.run_id).await?; + if !args.json { println!( - "Run {run_id} completed successfully in {:.2}s", + "Run {} completed successfully in {:.2}s", + args.run_id, duration.as_secs_f64() ); } } Err(err) => { let msg = format!("{err:#}"); - db::fail_run(db, metrics_store, run_id, &msg).await?; - if !json { - println!("Run {run_id} failed: {msg}"); + db::fail_run(args.db, args.metrics_store, args.run_id, &msg).await?; + if !args.json { + println!("Run {} failed: {msg}", args.run_id); } } } - let aggregate = metrics_store.list_aggregate_for_run(run_id).await?; + let aggregate = args + .metrics_store + .list_aggregate_for_run(args.run_id) + .await?; - if json { + if args.json { let metrics_map: HashMap = aggregate .iter() .map(|m| (m.metric_name.clone(), m.metric_value)) .collect(); let output = BuiltinRunJsonOutput { aggregate_metrics: metrics_map, - run_id, + run_id: args.run_id, warning: None, }; println!("{}", serde_json::to_string(&output)?); } else { print_aggregate_metrics_table(&aggregate); - println!("\nRun `qt show {run_id} --verbose` for sample-level details."); + println!( + "\nRun `qt show {} --verbose` for sample-level details.", + args.run_id + ); } builtin_result @@ -668,4 +748,78 @@ mod tests { let (input, _) = super::assemble_builtin_input(None, None); assert!(input.is_none()); } + + /// A `custom_nocode` benchmark config with all fields should serialize into the + /// expected JSON shape, converting field names faithfully. + #[test] + fn assemble_custom_nocode_input_from_config() { + let bench = qt::config::CustomNoCodeBenchmarkConfig { + type_: "custom_nocode".to_owned(), + style: qt::config::CustomNoCodeStyle::Qa, + dataset: "quantiles/simpleqa-verified".to_owned(), + model: Some(qt::llm::Sampler::Random {}), + qa: qt::config::CustomNoCodeQaConfig { + prompt_template_file: "prompts/qa.txt".to_owned(), + prompt_column: "problem".to_owned(), + golden_column: "answer".to_owned(), + limit: Some(10), + max_workers: Some(4), + }, + }; + let input = super::assemble_custom_nocode_input(&bench, None); + let parsed: serde_json::Value = serde_json::from_str(&input).unwrap(); + assert_eq!(parsed["style"], "qa"); + assert_eq!(parsed["dataset"], "quantiles/simpleqa-verified"); + assert_eq!(parsed["model"], "random"); + assert_eq!(parsed["prompt_template_file"], "prompts/qa.txt"); + assert_eq!(parsed["prompt_column"], "problem"); + assert_eq!(parsed["golden_column"], "answer"); + assert_eq!(parsed["limit"], 10); + assert_eq!(parsed["max_workers"], 4); + } + + /// When `model`, `limit`, and `max_workers` are absent from the config, the assembled + /// JSON should omit those keys entirely rather than emit null values. + #[test] + fn assemble_custom_nocode_input_omits_none_fields() { + let bench = qt::config::CustomNoCodeBenchmarkConfig { + type_: "custom_nocode".to_owned(), + style: qt::config::CustomNoCodeStyle::Qa, + dataset: "quantiles/simpleqa-verified".to_owned(), + model: None, + qa: qt::config::CustomNoCodeQaConfig { + prompt_template_file: "prompts/qa.txt".to_owned(), + prompt_column: "problem".to_owned(), + golden_column: "answer".to_owned(), + limit: None, + max_workers: None, + }, + }; + let input = super::assemble_custom_nocode_input(&bench, None); + let parsed: serde_json::Value = serde_json::from_str(&input).unwrap(); + assert!(parsed.get("model").is_none()); + assert!(parsed.get("limit").is_none()); + assert!(parsed.get("max_workers").is_none()); + } + + /// A `--input` CLI flag should take precedence over the assembled config object, + /// returning the raw CLI string directly. + #[test] + fn assemble_custom_nocode_input_with_cli_override() { + let bench = qt::config::CustomNoCodeBenchmarkConfig { + type_: "custom_nocode".to_owned(), + style: qt::config::CustomNoCodeStyle::Qa, + dataset: "quantiles/simpleqa-verified".to_owned(), + model: None, + qa: qt::config::CustomNoCodeQaConfig { + prompt_template_file: "prompts/qa.txt".to_owned(), + prompt_column: "problem".to_owned(), + golden_column: "answer".to_owned(), + limit: None, + max_workers: None, + }, + }; + let input = super::assemble_custom_nocode_input(&bench, Some(r#"{"limit":5}"#)); + assert_eq!(input, r#"{"limit":5}"#); + } } diff --git a/cli/src/config/custom_nocode.rs b/cli/src/config/custom_nocode.rs new file mode 100644 index 0000000..7028208 --- /dev/null +++ b/cli/src/config/custom_nocode.rs @@ -0,0 +1,167 @@ +use anyhow::Result; +use serde::{Deserialize, Deserializer}; + +use crate::llm::Sampler; + +/// Style of a no-code custom benchmark. +/// +/// Future variants may include: +/// - `Judge` – evaluate responses against a rubric using a judge model. +/// - `Similarity` – score responses with a similarity metric. +/// - `Agent` – run multi-step agent loops. +#[derive(Debug, Clone)] +pub enum CustomNoCodeStyle { + /// A qa-style benchmark. These benchmarks generally have datasets with + /// a prompt and a golden answer, which the model under test must exactly + /// match (modulo minor whitespace and other cleanup) for it to "pass" + /// the sample + Qa, +} + +impl<'de> Deserialize<'de> for CustomNoCodeStyle { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let s = String::deserialize(deserializer)?; + match s.as_str() { + "qa" => Ok(CustomNoCodeStyle::Qa), + other => Err(serde::de::Error::custom(format!( + "unsupported custom_nocode style `{other}`; expected `qa`", + ))), + } + } +} + +/// QA-specific configuration for a no-code custom benchmark. +#[derive(Debug, Deserialize, Clone)] +#[serde(deny_unknown_fields)] +pub struct CustomNoCodeQaConfig { + /// Path to a Jinja template file for rendering prompts. + pub prompt_template_file: String, + /// Dataset column containing the prompt text. + pub prompt_column: String, + /// Dataset column containing the golden answer. + pub golden_column: String, + /// Number of dataset rows to evaluate. + pub limit: Option, + /// Maximum concurrent workers. + pub max_workers: Option, +} + +/// No-code custom benchmark configuration. +#[derive(Debug, Deserialize, Clone)] +#[serde(deny_unknown_fields)] +pub struct CustomNoCodeBenchmarkConfig { + #[serde(rename = "type")] + pub type_: String, + pub style: CustomNoCodeStyle, + /// Dataset identifier in `HuggingFace` (e.g. `quantiles/simpleqa-verified`). + pub dataset: String, + /// Model sampler to use. + pub model: Option, + #[serde(flatten)] + pub qa: CustomNoCodeQaConfig, +} + +#[cfg(test)] +mod tests { + use super::super::{BenchmarkConfig, WorkspaceConfig}; + use super::*; + + #[test] + fn deserialize_custom_nocode_qa() { + let toml = r#" + [benchmarks.nocode_custom] + type = "custom_nocode" + style = "qa" + dataset = "quantiles/simpleqa-verified" + model = "random" + prompt_template_file = "prompts/qa.txt" + prompt_column = "problem" + golden_column = "answer" + limit = 10 + "#; + let config: WorkspaceConfig = toml::from_str(toml).unwrap(); + let bench = config.benchmarks.get("nocode_custom").unwrap(); + assert!(matches!(bench, BenchmarkConfig::CustomNoCode(_))); + if let BenchmarkConfig::CustomNoCode(c) = bench { + assert!(matches!(c.style, CustomNoCodeStyle::Qa)); + assert_eq!(c.dataset, "quantiles/simpleqa-verified"); + assert_eq!(c.model, Some(Sampler::Random)); + assert_eq!(c.qa.prompt_template_file, "prompts/qa.txt"); + assert_eq!(c.qa.prompt_column, "problem"); + assert_eq!(c.qa.golden_column, "answer"); + assert_eq!(c.qa.limit, Some(10)); + } + } + + #[test] + fn deserialize_custom_nocode_unsupported_style_errors() { + let toml = r#" + [benchmarks.nocode_custom] + type = "custom_nocode" + style = "judge" + dataset = "quantiles/simpleqa-verified" + model = "random" + prompt_template_file = "prompts/qa.txt" + prompt_column = "problem" + golden_column = "answer" + "#; + let result: Result = toml::from_str(toml); + assert!(result.is_err()); + } + + #[test] + fn custom_nocode_missing_required_field_errors() { + let toml = r#" + [benchmarks.nocode_custom] + type = "custom_nocode" + style = "qa" + dataset = "quantiles/simpleqa-verified" + model = "random" + prompt_column = "problem" + golden_column = "answer" + "#; + let result: Result = toml::from_str(toml); + assert!(result.is_err()); + } + + #[test] + fn validate_rejects_missing_template_file() { + let bench = BenchmarkConfig::CustomNoCode(CustomNoCodeBenchmarkConfig { + type_: "custom_nocode".to_owned(), + style: CustomNoCodeStyle::Qa, + dataset: "quantiles/simpleqa-verified".to_owned(), + model: Some(Sampler::Random), + qa: CustomNoCodeQaConfig { + prompt_template_file: "does_not_exist.txt".to_owned(), + prompt_column: "problem".to_owned(), + golden_column: "answer".to_owned(), + limit: None, + max_workers: None, + }, + }); + let err = bench.validate().unwrap_err(); + assert!(err.to_string().contains("not found")); + } + + #[test] + fn validate_accepts_existing_template_file() { + let file = tempfile::NamedTempFile::new().unwrap(); + let bench = BenchmarkConfig::CustomNoCode(CustomNoCodeBenchmarkConfig { + type_: "custom_nocode".to_owned(), + style: CustomNoCodeStyle::Qa, + dataset: "quantiles/simpleqa-verified".to_owned(), + model: Some(Sampler::Random), + qa: CustomNoCodeQaConfig { + prompt_template_file: file.path().to_str().unwrap().to_owned(), + prompt_column: "problem".to_owned(), + golden_column: "answer".to_owned(), + limit: None, + max_workers: None, + }, + }); + bench.validate().unwrap(); + } +} diff --git a/cli/src/config.rs b/cli/src/config/mod.rs similarity index 91% rename from cli/src/config.rs rename to cli/src/config/mod.rs index 0da4a18..9dd9dc6 100644 --- a/cli/src/config.rs +++ b/cli/src/config/mod.rs @@ -7,12 +7,13 @@ use crate::llm::Sampler; /// Configuration for a single benchmark. /// -/// Exactly one of the two variants is deserialized based on the `type` field: -/// `builtin` (default when absent) or `custom_code`. +/// Exactly one of the variants is deserialized based on the `type` field: +/// `builtin` (default when absent), `custom_code`, or `custom_nocode`. #[derive(Debug, Clone)] pub enum BenchmarkConfig { Builtin(BuiltinBenchmarkConfig), CustomCode(CustomCodeBenchmarkConfig), + CustomNoCode(CustomNoCodeBenchmarkConfig), } impl BenchmarkConfig { @@ -30,6 +31,15 @@ impl BenchmarkConfig { } Ok(()) } + BenchmarkConfig::CustomNoCode(c) => { + if !std::path::Path::new(&c.qa.prompt_template_file).is_file() { + bail!( + "custom_nocode benchmark config `prompt_template_file` must point to an existing file. File `{}` was not found", + c.qa.prompt_template_file + ); + } + Ok(()) + } } } } @@ -53,6 +63,14 @@ impl<'de> Deserialize<'de> for BenchmarkConfig { })?; Ok(BenchmarkConfig::CustomCode(config)) } + Some("custom_nocode") => { + let config = CustomNoCodeBenchmarkConfig::deserialize(value).map_err(|e| { + serde::de::Error::custom(format!( + "failed to deserialize custom_nocode benchmark config: {e}" + )) + })?; + Ok(BenchmarkConfig::CustomNoCode(config)) + } Some("builtin") | None => { let config = BuiltinBenchmarkConfig::deserialize(value).map_err(|e| { serde::de::Error::custom(format!( @@ -62,7 +80,7 @@ impl<'de> Deserialize<'de> for BenchmarkConfig { Ok(BenchmarkConfig::Builtin(config)) } Some(other) => Err(serde::de::Error::custom(format!( - "invalid benchmark type `{other}`; expected `builtin` or `custom_code`", + "invalid benchmark type `{other}`; expected `builtin`, `custom_code`, or `custom_nocode`", ))), } } @@ -98,6 +116,9 @@ pub struct CustomCodeBenchmarkConfig { pub input: Option>, } +mod custom_nocode; +pub use custom_nocode::*; + /// Top-level workspace configuration read from `quantiles.toml` or /// `.quantiles.toml` in the current working directory. #[derive(Debug, Deserialize, Default)] diff --git a/cli/src/dataset/hf_client.rs b/cli/src/dataset/hf_client.rs index 017ea41..37356d6 100644 --- a/cli/src/dataset/hf_client.rs +++ b/cli/src/dataset/hf_client.rs @@ -3,7 +3,10 @@ use reqwest::header; use serde::Deserialize; use serde_json::Value; -static HF_DATASETS_SERVER: &str = "https://datasets-server.huggingface.co"; +fn hf_base_url() -> String { + std::env::var("HF_DATASETS_SERVER") + .unwrap_or_else(|_| "https://datasets-server.huggingface.co".to_owned()) +} /// Thin async HTTP client for the huggingface Dataset Viewer API. pub struct HuggingFaceClient { @@ -104,7 +107,10 @@ impl HuggingFaceClient { config: &str, revision: Option<&str>, ) -> Result> { - let mut url = format!("{HF_DATASETS_SERVER}/splits?dataset={dataset_id}&config={config}"); + let mut url = format!( + "{}/splits?dataset={dataset_id}&config={config}", + hf_base_url() + ); if let Some(rev) = revision { url = format!("{url}&revision={rev}"); } @@ -127,7 +133,7 @@ impl HuggingFaceClient { /// Returns an error if configs couldn't be inferred from the given dataset /// and revision. pub async fn infer_config(&self, dataset_id: &str, revision: Option<&str>) -> Result { - let mut url = format!("{HF_DATASETS_SERVER}/splits?dataset={dataset_id}"); + let mut url = format!("{}/splits?dataset={dataset_id}", hf_base_url()); if let Some(rev) = revision { url = format!("{url}&revision={rev}"); } @@ -160,7 +166,8 @@ impl HuggingFaceClient { revision: Option<&str>, ) -> Result> { let mut url = format!( - "{HF_DATASETS_SERVER}/rows?dataset={dataset_id}&config={config}&split={split}&offset={offset}&length={limit}" + "{}/rows?dataset={dataset_id}&config={config}&split={split}&offset={offset}&length={limit}", + hf_base_url() ); if let Some(rev) = revision { url = format!("{url}&revision={rev}"); @@ -189,8 +196,10 @@ impl HuggingFaceClient { split: &str, revision: Option<&str>, ) -> Result { - let mut url = - format!("{HF_DATASETS_SERVER}/size?dataset={dataset_id}&config={config}&split={split}"); + let mut url = format!( + "{}/size?dataset={dataset_id}&config={config}&split={split}", + hf_base_url() + ); if let Some(rev) = revision { url = format!("{url}&revision={rev}"); } diff --git a/cli/src/dataset/mod.rs b/cli/src/dataset/mod.rs index b740fd8..fa78209 100644 --- a/cli/src/dataset/mod.rs +++ b/cli/src/dataset/mod.rs @@ -29,11 +29,15 @@ impl DatasetManager { /// /// Returns an error if the system's cache directory cannot be determined. pub fn new() -> Result { - let cache_dir = dirs::home_dir() - .context("failed to determine home directory")? - .join(".cache") - .join("quantiles") - .join("datasets"); + let cache_dir = if let Ok(dir) = std::env::var("QUANTILES_DATASET_CACHE_DIR") { + std::path::PathBuf::from(dir) + } else { + dirs::home_dir() + .context("failed to determine home directory")? + .join(".cache") + .join("quantiles") + .join("datasets") + }; let client = HuggingFaceClient::new()?; Ok(Self { client, diff --git a/custom-nocode-examples/prompts/qa.txt b/custom-nocode-examples/prompts/qa.txt new file mode 100644 index 0000000..c4f9438 --- /dev/null +++ b/custom-nocode-examples/prompts/qa.txt @@ -0,0 +1 @@ +{{ prompt }} diff --git a/custom-nocode-examples/quantiles.toml b/custom-nocode-examples/quantiles.toml new file mode 100644 index 0000000..1372044 --- /dev/null +++ b/custom-nocode-examples/quantiles.toml @@ -0,0 +1,9 @@ +[benchmarks.nocode_custom] +type = "custom_nocode" +style = "qa" +dataset = "quantiles/simpleqa-verified" +model = "random" +prompt_template_file = "prompts/qa.txt" +prompt_column = "problem" +golden_column = "answer" +limit = 10 diff --git a/docs/assets/new-badge.svg b/docs/assets/new-badge.svg new file mode 100644 index 0000000..07e5906 --- /dev/null +++ b/docs/assets/new-badge.svg @@ -0,0 +1,5 @@ + + NEW! + + NEW! +