AI Quality Evaluation Lab

Portfolio lab demonstrating AI quality evaluation with golden datasets, schema validation, groundedness checks, hallucination tests, prompt regression, and CI thresholds.

This repository shows how a QA Automation Engineer / SDET can test AI-powered systems without relying on brittle exact-string assertions or paid AI services. The default test and CI flow is fully deterministic: it uses a local FakeLLM, local JSON datasets, and local rubric scoring.

“AI systems are non-deterministic, so this project does not rely only on exact string assertions. It evaluates outputs with schema validation, rubrics, thresholds, groundedness checks, citation checks, and human-reviewed golden datasets.”

What This Repo Demonstrates

• A fictional QA Support Assistant answering questions from a local QABank Help Center knowledge base.
• RAG-style flow: user question -> retriever -> prompt builder -> FakeLLM -> schema validator -> evaluator -> report/CI gate.
• AI evaluation concepts that matter for release confidence: golden datasets, rubrics, thresholds, hallucination checks, groundedness, citations, prompt regression, and deterministic CI.
• Honest limitations: this is a portfolio lab, not a production chatbot, not real banking support, and not a complete AI safety system.

Tech Stack

• Node.js 20
• TypeScript
• Vitest
• Zod
• ESLint
• Prettier
• GitHub Actions
• JSON and Markdown evaluation reports

Architecture

flowchart LR
  User["User question"] --> Retriever["Keyword retriever"]
  Retriever --> Prompt["Prompt builder"]
  Prompt --> FakeLLM["Deterministic FakeLLM"]
  FakeLLM --> Schema["Zod schema validator"]
  Schema --> Evaluator["Rubric evaluator"]
  Evaluator --> Reports["JSON + Markdown reports"]
  Evaluator --> CI["CI quality gate"]

Loading

AI Quality Gate Matrix

Gate	What It Protects	Default
Overall score	Broad release confidence across all rubric dimensions	0.80
Groundedness	Answers must be supported by retrieved help-center context or safely refused	0.80
Schema validity	All responses must match the expected contract	100%
Case minimum score	Each golden or hallucination case must meet its own review threshold	required
Critical hallucination cases	Fake fees, URLs, phone numbers, regulatory claims, and unsupported features must be caught	required

Evaluation Methodology

Each case is scored from 0 to 1 across:

• schemaValidity
• intentMatch
• groundedness
• citationCorrectness
• noHallucination
• completeness
• escalationCorrectness
• toneHelpfulness

The evaluator uses weighted rubric scoring instead of exact output matching. Required facts and forbidden claims make each evaluation decision reviewable in GitHub.

Local datasets are also validated with Zod at load time so malformed evaluation data fails fast instead of silently weakening the gate.

Dataset

The lab includes:

• 10 fictional QABank Help Center knowledge-base documents.
• 16 golden evaluation cases.
• 5 hallucination trap cases.
• 2 prompt versions: v1 baseline and v2 safer grounded prompt.

Cases cover login help, password reset, transfer limits, transaction visibility, card freeze/unfreeze, suspicious activity, support contact, statements, notifications, unsupported international transfers, ambiguous questions, outside-scope questions, and prompt injection.

Prompt Regression

Prompt v1 is intentionally imperfect and may invent unsupported information for selected traps. Prompt v2 adds stronger grounding instructions, safer unsupported-answer behavior, and explicit refusal of fake URLs, phone numbers, fees, regulatory claims, and unavailable features.

npm run eval compares v1 and v2 in the Markdown report so prompt changes can be reviewed like any other regression risk.

This lab uses a deterministic FakeLLM so the evaluation pipeline can be tested reliably in CI. The prompt regression is a controlled simulation, not a claim that a real model would behave identically.

Local Setup

npm ci
npm run quality

No secrets, paid services, or real LLM API keys are required for the default workflow.

Commands

npm ci
npm run typecheck
npm run lint
npm run format:check
npm run test
npm run test:unit
npm run test:evaluation
npm run eval
npm run eval:ci
npm run quality

npm run quality runs type checking, linting, format checking, tests, and CI evaluation gates.

Reports

npm run eval writes:

• reports/evaluation-report.json
• reports/evaluation-report.md

The Markdown report includes:

• Overall score and pass/fail result.
• Threshold values.
• Per-metric averages.
• Failed cases table with question, score, expected intent, answer snippet, weakest metric, and metric explanation.
• Hallucination cases table.
• Prompt v1 vs v2 comparison.
• A short explanation of why failures matter.

Generated reports are ignored by git except reports/.gitkeep.

CI Thresholds

The default CI flow uses only the deterministic FakeLLM. No real LLM API keys or external AI calls are required.

Configurable environment variables:

Variable	Default	Purpose
EVAL_MIN_SCORE	0.80	Minimum overall average score
EVAL_MIN_GROUNDEDNESS	0.80	Minimum groundedness average
EVAL_REQUIRE_SCHEMA_VALIDITY	true	Require every output to match the schema

npm run eval:ci exits non-zero when a quality gate fails.

How To Add A New Evaluation Case

Add a readable case to src/data/golden-dataset.json or src/data/hallucination-cases.json with:

• A realistic userQuestion.
• The expectedIntent.
• requiredFacts that should appear in a good answer.
• forbiddenClaims that must not appear.
• expectedDocIds for citation checks.
• shouldEscalate and a case-level minimumScore.

Run npm run eval to inspect the report, then npm run quality before publishing the change.

How To Interpret A Failed Evaluation Report

Start with reports/evaluation-report.md. The failed cases table shows the case ID, question, score, minimum score, expected intent, answer snippet, weakest metric, and explanation. Use the JSON report when you need every per-metric decision for deeper debugging.

Known Limitations

• This is a portfolio lab, not a production chatbot.
• The QABank product, help center, and data are fictional.
• The evaluator is deterministic and local for CI stability.
• The retrieval implementation is simple keyword overlap, not vector search.
• The rubrics demonstrate AI quality engineering concepts but do not guarantee complete AI safety.
• Real LLM provider integration is intentionally not enabled in the default path.

Future Improvements

• Add mutation tests for deliberately corrupted knowledge-base documents.
• Add retrieval recall and precision metrics.
• Add seeded stochastic FakeLLM variants for robustness testing.
• Add a real-provider adapter behind an explicit opt-in command.
• Add richer rubric review exports for human evaluator sign-off.

Interview-Ready Explanation

This project demonstrates that AI quality work is different from traditional deterministic UI/API testing. A useful AI test strategy needs schema contracts, golden datasets, rubric scoring, thresholds, regression checks, retrieval validation, groundedness checks, hallucination traps, and CI gates that can block unsafe releases.

The repo is intentionally self-contained so reviewers can run it without secrets, paid services, or external AI APIs.