Evaluate AI-generated SQL with pytest.
evaldata runs text-to-SQL evals in your existing test suite.
It checks semantic equivalence of SQL queries, diffs result sets in your warehouse, and uses an LLM judge for ambiguous cases.
- Semantic equivalence. Parse both queries, normalize their ASTs, and
compare canonical forms. No execution, no LLM — when it can't confirm, it returns
unknown. - Execution in your warehouse. Run the query on DuckDB, Postgres, Databricks, or Snowflake and compare the results, accounting for row order, NULLs, float tolerance, and types.
- It's just
pytest. Every eval is a test, run in your suite and your CI on every PR. No new runner, notebook, or dashboard. - An LLM judge when you need one. For ambiguous questions, missing reference answers, or explanations to grade, use a grader model with explicit criteria.
uv add evaldata # core, includes the DuckDB adapterAn eval is a pytest test: a case (a question and its expected answer), a solver
(the system under test that writes the SQL), and a scorer (how the answer is judged).
Below, the AI's SQL is written differently from the reference query — reordered predicates,
different casing — but means the same thing. observed_equivalence() confirms the match
with AST normalization; no query runs.
from evaldata import CallableSolver, EvalCase, assert_eval, eval_case, observed_equivalence
from evaldata.platforms import duckdb_platform
platform = duckdb_platform(name="shop", path="shop.duckdb")
@eval_case(
input="Name the US customers with an id above 1.",
expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US' AND id > 1"},
platform=platform,
)
def test_us_customers(case: EvalCase) -> None:
solver = CallableSolver(lambda c: "select NAME from customers where id > 1 and country = 'US'")
assert_eval(case, solver, scorers=[observed_equivalence()])uv run pytest case result detail
──────────────────────────────────
test_us_customers PASS
1 passed, 0 failed
The full runnable version is in
examples/01_deterministic/test_showcase.py.
To test a real model instead of fixed SQL, swap the solver for
PromptSolver(model="openai/gpt-4o-mini") (needs the evaldata[litellm] extra). To judge
equivalence without a warehouse, swap the scorer for judged_equivalence(model).
- Evaluate dbt projects against gold SQL.
- Evaluate dbt Semantic Layer queries against gold MetricFlow queries.
- Evaluate Snowflake Cortex Analyst against gold SQL.
- Reproduce dbt's Semantic Layer benchmark locally on DuckDB.
uv add evaldata # core (includes the DuckDB adapter)
uv add "evaldata[postgres]" # + Postgres adapter
uv add "evaldata[databricks]" # + Databricks adapter
uv add "evaldata[snowflake]" # + Snowflake adapter
uv add "evaldata[cortex]" # + Snowflake Cortex Analyst solver
uv add "evaldata[litellm]" # + litellm, to call a model from PromptSolverDuckDB, Postgres, Databricks, and Snowflake are the adapters available today. A BigQuery adapter is planned.
Full documentation: monospaceai.github.io/evaldata
- Getting started — write and run your first eval.
- Concepts — cases, solvers, scorers, and platforms.
- Scoring guides — semantic equivalence, LLM judge, composing scorers.
- Model guides — local Ollama, hosted model.
- Platform guides — Databricks, Snowflake, Cortex Analyst.
- dbt guides — dbt project, dbt Semantic Layer, reproduce dbt's Semantic Layer benchmark.
- Run a text-to-SQL benchmark — load a Spider/BIRD dataset and measure execution accuracy.
- API reference — the public API, generated from docstrings.
Runnable examples in examples/:
| Example | Shows |
|---|---|
| Showcase | Semantic equivalence with an execution fallback — no setup |
| Deterministic | Every expected-type and scorer, with fixed SQL |
| Local AI | A self-hosted Ollama model |
| Hosted AI | A hosted model, mocked so it runs without a key |
| Databricks | The same cases on a live Databricks SQL Warehouse |
| LLM judge | Judged equivalence, mocked so it runs without a key |
| Benchmark | Load a Spider/BIRD dataset and measure execution accuracy |
| Snowflake | The same cases on a live Snowflake warehouse — live-only, needs SNOWFLAKE_* credentials |
| Cortex Analyst | Snowflake Cortex Analyst — live-only, needs SNOWFLAKE_* credentials |
| dbt | A dbt project's text-to-SQL, stubbed so it runs offline |
| dbt Semantic Layer | dbt Semantic Layer (MetricFlow) queries, scored locally on DuckDB |
See examples/README.md for details.
git clone https://github.com/monospaceai/evaldata.git
cd evaldata
uv sync # core + dev tooling
uv run pre-commit install
just check # lint + typecheck + tests with coverage (runs everything)just check runs lint, typecheck, and tests with coverage (held at 100%). See the
justfile for the full set of commands.
Adapter conformance for real platforms is marked e2e. CI provisions Postgres as a
service container and runs the suite on every push, so the Postgres adapter is exercised
against a real engine on every change.
Run it locally against Postgres with:
docker compose up -d # postgres:17 on localhost:5432
uv run --extra postgres pytest -m e2e # connection via POSTGRES_TEST_* env (defaults match compose)