test(15-5): AI schema regression test infrastructure (fixture loader + harness + seed fixtures)#114
Merged
Conversation
… + 3 synthetic seed fixtures + runbook
5 tasks
3-agent adversarial review surfaced 12 actionable findings. HIGH: - BH-1: JSON.parse error path now wraps via loadFixture() so a malformed fixture's path appears in the error message, not just `SyntaxError at position N`. - BH-2: NEW Case 6 enforces `_synthetic: true` marker on every committed fixture — privacy defense until a future operator-action `.real-fixtures.txt` manifest exists for real captures. - BH-3 + EH-10: Runbook gains REQUIRED Privacy + GDPR section before Option A. User-derived French content (writing-evaluation errors[].original) may contain names/addresses/dates/contact info. Story 9-3 GDPR scrubber does NOT run on fixtures; Story 9-4 prompt wrapping does NOT protect captures; git history is irrevocable. Section mandates field-by-field sanitization + `_redacted: true` marker convention + prefer-synthetic-when-possible guidance. - EH-9 (load-bearing): NEW Case 7 parallel strict-mode probe. The Zod schemas use default `.strip()` which silently drops unknown fields — defeating the regression-detection goal of the story. Probe runs `schema.strict().safeParse()` per fixture and reports extra-field drift as SOFT console.warn (does not fail the test — synthetic seeds pass by construction; real captures may flag drift for operator triage). Also corrected the runbook's misleading "Zod is strict() by default" claim. MED: - BH-4: Case 2 walks fs.readdirSync directly so EMPTY orphan dirs fail loud (not just dirs with files). - BH-5: `FIXTURE_SCHEMA_MAP` uses `as const satisfies` for narrow value typing — value-typo at construction caught at compile time. - BH-6: `dirent.isFile()` filter so a directory named `foo.json` doesn't crash readFileSync with EISDIR. - BH-7 / EH-1: `stripMetadata` JSDoc documents the top-level-only contract — load-bearing for nested data passthrough. - EH-6: case-insensitive `.json` match so `.JSON` / `.Json` files don't silently skip. LOW: - BH-12: Belt-and-suspenders FIXTURES_ROOT existence pin (Case 0). - EH-2: Case 4 input gains nested `_note` for symmetric coverage alongside nested `_synthetic`. Deferred (filed as follow-ups): - 15-5-followup-real-fixture-manifest (operator-action allow-list) - 15-5-followup-passage-ref-integrity (BH-8 — pre-existing schema gap) - 15-5-followup-empty-fixture-boundary (EH-3 — empty-object edge) Story file housekeeping: Status: review → done; populated File List + Completion Notes per Acceptance Auditor APPROVE_WITH_NOTES. +3 net test cases (8 → 11). All 4 quality gates green: type-check 0 / lint 0 / prettier clean / jest 2170 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/lib/schemas/__tests__/fixture-replay.test.ts(8 cases) — walks__fixtures__/<schema>/*.json, replays each through corresponding Zod parser viait.each.src/lib/schemas/__fixtures__/tree with 3 seed-schema dirs (writing-evaluation, dictation, mock-test-section) × 1 synthetic seed fixture each._bmad-output/planning-artifacts/runbooks/ai-fixture-capture.mddocumenting WHEN/HOW/WHERE to capture real fixtures._syntheticmarker, 3 schemas vs all 36).Story
15-5-ai-schema-regression-tests_bmad-output/implementation-artifacts/15-5-ai-schema-regression-tests.mdTest plan
npx jest fixture-replaypasses — +8 net cases..jsonto any__fixtures__/<schema>/dir produces a new passing test case without code changes.writingEvaluationSchemaimport name breaks TypeScript (good — catches drift).🤖 Generated with Claude Code