Adaptive Test Intelligence (ATI) — a seven-stage, cost-bounded LLM pipeline for autonomous test quality management, delivered as a unified modular REST microservice.
ATI is the first system to unify flaky detection, cost-bounded LLM repair, and configurable autonomy under a single modular REST microservice, evaluated on both synthetic and real flaky test corpora.
Novel contributions:
-
Autonomy dial — four-mode configuration (Manual → Gated → Supervised → Autonomous) governs LLM invocation and human oversight. Mode is a single env var; individual thresholds are independently overridable.
-
Cost-bounded LLM invocation — the Healing Success Predictor (HSP) gates every repair call using regex error classification and historical repair success rates. LLM is skipped when predicted success rate falls below a configurable threshold (default 10%), avoiding wasteful API spend on historically unrecoverable error classes.
-
End-to-end RepairValidator — every candidate repair is executed in a real pytest subprocess before being committed. No repair is accepted on LLM output alone.
-
Dual evaluation corpus — 30-scenario synthetic corpus (Phase 1) + 27 real Python flaky tests from the IDoFT benchmark (Phase 2, 5 open-source repos).
Key empirical results:
| Metric | Phase 1 (Synthetic) | Phase 2 (IDoFT Real) |
|---|---|---|
| F2 Recall | 1.000 | 1.000 |
| DHM Service F1 | — | 1.000 |
| RBTS F1 | F1@15 = 0.909 | F1@10 = 0.600 |
| ATG LLM rate | 100% (30/30) | 100% (27/27) |
| TPAD Detection rate | 54.5% | 100% |
| Unit tests passing | 173/173 | — |
Live harness results (autonomous mode, 2026-06-22):
| Metric | Gated | Autonomous |
|---|---|---|
| Repair success | 8/8 (100%) | 8/8 (100%) |
| Subprocess validation pass | 8/8 (100%) | 8/8 (100%) |
| Mean machine MTTR | 11.3 s | 11.8 s |
| Human approval required | Yes | No |
Machine MTTR is statistically equivalent across modes. The meaningful gain of autonomous mode is elimination of unbounded human review latency — in a live CI pipeline, gated MTTR = 11.3 s + human review time (minutes to hours); autonomous MTTR = 11.8 s, complete.
| Stage | Feature | Type |
|---|---|---|
| Generate | Autonomous Test Generator (ATG) + Gap Analyser (TGA) | LLM Core |
| Select | Risk-Based Test Selector (RBTS) | Deterministic |
| Triage | Failure Triage Agent (FTA) | Deterministic |
| Heal | Self-Healing Agent + Healing Success Predictor | LLM Core |
| Detect | Perf Anomaly Detector (TPAD) + Dep Health Monitor (DHM) | Deterministic |
| Diagnose | Root Cause Analyser (RCA) | LLM Core |
| Score | Repair Quality Scorer (RQS) | Deterministic |
| Module | ID | Description |
|---|---|---|
| Bayesian Flaky Detector | F2 | Beta-Binomial P(flaky) model; auto-quarantines high-risk tests |
| Visual Regression Healer | VRH | Claude Vision classifies screenshot diffs; auto-updates stale baselines |
| Cross-PR Failure Correlator | CPFC | DBSCAN clustering of failures sharing a common root cause |
| Test Smell Detector | TSD | Regex-based detection of 8 structural anti-pattern categories |
| QA Pipeline Adapter | QPA | Thin shim to trigger k11techlab full-pipeline re-runs |
k11techlab-test-intelligence/
├── api/ # FastAPI routes and Pydantic schemas (15 modules)
├── self_healing/ # Healing agent, validator, predictor, visual repair, autonomy
├── flaky_detector/ # Bayesian model, outcome tracker, quarantine, RCA
├── test_generator/ # ATG and TGA
├── triage/ # FTA and RBTS
├── performance/ # TPAD and DHM
├── intelligence/ # CPFC, RQS, TSD
├── integrations/ # QA pipeline adapter
├── tests/ # pytest test suite (173 tests)
├── docs/ # Feature documentation (16 docs)
├── results/ # Evaluation scripts, reports, and live harness
│ ├── empirical_eval.py # Phase 1: 30 synthetic scenarios
│ ├── real_data_eval.py # Phase 2: 27 IDoFT real tests
│ ├── live_harness.py # Live MTTR measurement (gated vs autonomous)
│ ├── EMPIRICAL_RESULTS.md
│ ├── REAL_DATA_RESULTS.md
│ └── LIVE_HARNESS_RESULTS.md
├── artifacts/ # Logos (ATI_logo.png, k11_logo.png)
├── .env.example # Environment variable template
└── .gitignore
All modules persist to a shared SQLite database (ti_intelligence.db), enabling cross-feature learning and feedback loops.
pip install -r requirements.txtcp .env.example .env
# Edit .env — set ANTHROPIC_API_KEY (required for LLM features)
# Set AUTONOMY_MODE (default: gated)# Default (gated mode)
uvicorn api.main:app --host 0.0.0.0 --port 8091 --reload
# Autonomous mode
AUTONOMY_MODE=autonomous uvicorn api.main:app --host 0.0.0.0 --port 8091 --reloadAPI docs: http://localhost:8091/docs
# Phase 1 — synthetic corpus (30 scenarios)
python results/empirical_eval.py
# Phase 2 — IDoFT real data (27 tests, requires gh CLI)
python results/real_data_eval.py
# Live MTTR harness (server must be running)
python results/live_harness.py
python results/live_harness.py --mode autonomouspytest # all 173 tests
pytest -v # verbose
pytest --tb=short # short tracebacks| Mode | LLM invoked | Human approval | Notification |
|---|---|---|---|
manual |
Never | N/A | Yes |
gated |
Yes (HSP gated) | Required | Yes |
supervised |
Yes (HSP gated) | Not required | Yes |
autonomous |
Yes (HSP gated) | Not required | Silent |
Set via AUTONOMY_MODE in .env. See docs/16-autonomy-modes.md for the recommended deployment trajectory and full empirical results.
All configuration is via environment variables. See .env.example for the full list.
| Variable | Default | Description |
|---|---|---|
ANTHROPIC_API_KEY |
— | Required for LLM features; others work without it |
AUTONOMY_MODE |
gated |
manual | gated | supervised | autonomous |
TI_DB_PATH |
ti_intelligence.db |
Shared SQLite path |
REPAIR_MODEL |
claude-haiku-4-5-20251001 |
Model for test repair |
VISUAL_REPAIR_MODEL |
claude-sonnet-4-6 |
Vision model for screenshot diffs |
RCA_MODEL |
claude-haiku-4-5-20251001 |
Model for root cause analysis |
PREDICTOR_SKIP_THRESHOLD |
0.10 |
HSP: skip LLM if class success rate < this |
FLAKY_QUARANTINE_THRESHOLD |
0.15 |
P(flaky) threshold to quarantine |
FLAKY_CLEAR_THRESHOLD |
0.05 |
P(flaky) threshold to release quarantine |
VALIDATOR_TIMEOUT_S |
120 |
Subprocess validation timeout |
LOG_LEVEL |
INFO |
Logging level |
| Method | Path | Description |
|---|---|---|
| GET | /health |
Health check — all 15 module statuses |
| POST | /healing/repair |
Repair a failing test (autonomy-mode aware) |
| GET | /healing/history/{test_id} |
Repair audit log |
| POST | /healing/visual-repair |
Classify and heal a visual regression |
| POST | /healing/predictor/predict |
HSP cost gate prediction |
| GET | /healing/predictor/stats |
Error class success rate statistics |
| POST | /flaky/record |
Record a test outcome |
| GET | /flaky/quarantine |
List quarantined tests |
| GET | /flaky/root-cause/{test_id} |
Root cause report |
| POST | /flaky/sweep |
Nightly maintenance sweep |
| POST | /generation/* |
ATG / TGA endpoints |
| POST | /triage/* |
FTA / RBTS endpoints |
| POST | /performance/* |
TPAD / DHM endpoints |
| POST | /intelligence/* |
CPFC / RQS / TSD endpoints |
| POST | /pipeline/run |
Trigger QA pipeline re-run |
See docs/README.md for the full index. Key docs:
| Doc | Description |
|---|---|
| Adaptive_Test_Intelligence.md | End-to-end pipeline overview — all 7 stages, LLM vs deterministic, cost model |
| 16-autonomy-modes.md | Autonomy dial — 4 modes, HSP interaction, empirical results |
| 08-test-generator.md | ATG + TGA — LLM path, template fallback, gap analysis |
| 02-flaky-detector.md | Bayesian Beta-Binomial model, quarantine lifecycle |
| 06-rest-api.md | Full REST API reference |
- API: FastAPI, Uvicorn, Pydantic
- LLM: Anthropic Claude API (
claude-haiku-4-5-20251001,claude-sonnet-4-6) - Database: SQLite (shared across all 15 modules)
- Statistics: Bayesian Beta-Binomial, EWMA, DBSCAN
- Testing: pytest (173 tests)
- Language: Python 3.11+
Copyright 2026 Kavita Jadhav / K11 Software Solutions LLC
Licensed under the Apache License, Version 2.0. See LICENSE for the full text.