golden-dataset

A retrieval-augmented generation pipeline in Python with a rigorous offline evaluation harness. Chunks and embeds documents, retrieves by vector similarity, and generates grounded answers — with pluggable LLM providers (including a deterministic local fake for tests) and metrics for retrieval quality and answer faithfulness. No API key required.

Veris: agent-agnostic local LLM evaluation and regression testing gate. Runs golden datasets against any callable, scores with DeepEval, compares rolling SQLite baselines, and returns CI-ready exit codes.

Why a passing benchmark isn't safe to ship: a free 2-stage (benchmark + replay) validation recipe for LLM model swaps & prompt changes, run on flat-rate coding-agent subagents — no eval API bill.

Turn real traffic/traces into a versioned, deduped, coverage-tracked LLM regression suite — a curation accelerator with a human-confirmed golden gate.

Catch LLM quality regressions in CI: a golden-set regression gate with calibrated graders (grounding, hallucination, tool-call, LLM-judge) that fails the PR when answer quality drops.

A Claude skill for building golden sets to test AI systems — matching, RAG, LLM-as-judge — without false greens.

A markdown-first governance framework for building trustworthy golden datasets with clear ownership, labeling, adjudication, leakage controls, versioning, retention, and release evidence.

Portfolio-grade AI quality evaluation lab with golden datasets, prompt regression, groundedness checks, hallucination tests and CI thresholds.

RAG evaluation framework: hit-rate, MRR, faithfulness scoring, and async batch evaluation with golden question datasets

Catch LLM quality regressions before they reach production — eval-driven CI/CD with LLM-as-Judge scoring, Wilson 95% CI diffing, and automatic PR alerts.

Evaluation-driven LLM harness for support-ticket intent classification: golden dataset, per-class & per-difficulty accuracy, latency/cost, and a pytest regression gate.

A hybrid RAG pipeline whose point is the evaluation gate: BM25 plus dense cosine fused by Reciprocal Rank Fusion over a deliberately tiny fixture corpus, scored against a 40-question golden set by a harness that exits non-zero below threshold. Pure Python stdlib, 0 pip dependencies, no GPU, no network. A discipline demo.

Deterministic evaluation harness for AP document-matching agents. Scores 3-way findings against a hand-audited, held-out golden dataset: per-category precision and recall, over-flagging measured on a zero-defect control, byte-reproducible scorecards, answer key structurally out of reach.

LLM classifier evaluation policy: golden sets, slice regressions, changed-case review, budgets, and accountable promotion gates.

Golden Datasets Templates for AI Evaluation

CI/CD regression detection for LLM features: golden-dataset evals, LLM-as-judge scoring, run diffing, Slack alerts, merge gating

Evaluation-driven CI for AI features: tiered golden sets, paired bootstrap significance on per-item score deltas, validated judges, and a PASS/BLOCK/INCONCLUSIVE merge gate that escalates tiers only when it cannot decide. 300 real gemma4 responses, reproducible offline.

A RAG chatbot demo: LLM + retrieval pipeline, built from scratch

Agentic diagnostic assistant for distributed-system incidents: multi-turn RAG, hypothesis updates, evidence packing, golden evals, and failure-attributed run reports.

Adaptive RAG benchmarking platform for Brazilian financial and regulatory documents - 8 retrieval strategies, adaptive router, reproducible benchmark on the RegRAG-BR golden dataset