test(15-4): Maestro E2E skeleton flows + operator runbook#115
Merged
Conversation
5 tasks
3-agent adversarial review surfaced 4 actionable Day-1 corrections
plus 3 runbook gaps. SKELETON scope preserved.
HIGH:
- BH-1: Flow 1 asserted PRE-Story-14-1 French gate text. Per
Story 14-1 R1-Q5, EmailVerificationGate was converted to English
(verified against EmailVerificationGate.tsx:314+352). Corrected
to "Verify your email address" + "I've verified — refresh".
- BH-2: Tab labels in flows 4 + 5 did not match
app/(tabs)/_layout.tsx 'title' values. The bottom-tab labels
are "Talk" (line 61) and "TCF Test" (line 97), not
"Conversation" / "Mock test" (those are headerTitle values,
different surface). Corrected both flows.
- BH-3: Flow 1 referenced 'input-password-confirm' but signup.tsx
has ONE password field. Deleted the 3-line confirm block —
delete-don't-alias pattern; operator runbook §5 documents the
single-password contract.
MED:
- EH-4: Flow 4 mic-permission used iOS-only 'tapOn: "OK"' which
fails on Android (Android dialog is "Allow" / "While using the
app" / "Only this time"). Switched to Maestro's
'launchApp.permissions: { record_audio: allow }' directive which
bypasses the system dialog entirely AND works cross-platform.
Runbook (R1 EH-1 + EH-3 + EH-5):
- Section 5 gains a Story 12-9 email-confirmation precheck:
before running Flow 1, verify the Supabase project has the
'Confirm email' toggle ON via a curl-vs-jq one-liner. If OFF,
Flow 1 dead-ends because the gate never renders.
- Section 5 gains Flow 1 re-run cleanup guidance — 4 options
ranging from manual Studio delete to per-run email suffix
randomization via MAESTRO_RUN_ID env-var.
- Section 5 hoists password to MAESTRO_TEST_PASSWORD env-var
single source of truth — future Story 12-8 policy tightening
updates one place (runbook + seed script).
Deferred (filed as follow-ups):
- 15-4-followup-add-testids (BH-4 — needs source edit)
- 15-4-followup-yaml-vs-ts-constants-drift (EH-2 — drift detector
cross-checking hardcoded times: 15/29 vs TCF constants)
- 15-4-followup-appid-cross-check (EH-7 — config.yaml vs app.json)
- 15-4-followup-mcq-correct-incorrect-alternation (EH-8 — RNG)
Story file housekeeping: Status: review → done; populated Dev Agent
Record per Acceptance Auditor APPROVE_WITH_NOTES.
No net test count change (drift tests unchanged at 5/5). All 4
quality gates green: type-check 0 / lint 0 / prettier clean /
jest 2164 tests pass.
Simplemart17
added a commit
that referenced
this pull request
May 17, 2026
3-agent adversarial review surfaced 8 actionable findings. The
load-bearing one (BH-2/EH-4 per-directory thresholds) REVEALED a
real coverage gap that the global average had been masking.
HIGH:
- BH-2 / EH-4 (load-bearing): added per-directory coverageThreshold
scopes ('./src/lib/' + './src/hooks/') so the spec deliverable's
CONJUNCTION ("≥ 40% on src/lib AND src/hooks") is enforced, not
the average. Per-directory floors immediately revealed src/hooks/
is at 23.02% statements / 27.12% branches / 25.92% functions /
23.36% lines — well below 40%. The global metric (53.42%) was
averaging this against src/lib/ which is much higher. Per spec
AC-E ("if below 40% on ANY metric, lower threshold to
floor-minus-3% of actual"), calibrated src/hooks/ floors to
20/24/22/20. Global stays at 40 (belt-and-suspenders); src/lib/
stays at 40 (above measured). Filed
15-6-followup-lift-hooks-coverage-to-40 to raise the per-dir
floor as future stories add hook tests.
- BH-1 ordering regex anchor: pre-R1 'indexOf("- name: Tests\\n")'
was fragile to trailing-whitespace/CRLF drift. Anchored via
'/^\\s{6}- name:\\s*Tests\\s*$/m' (PR #115 EH-4 lesson reused).
- BH-3 tail-slice safer fallback: pre-R1 if coverage step ever
moved to be the last step, the silent-disable negative guard
would sweep entire file + false-positive on Expo Doctor's
legitimate 'continue-on-error: true'. Bounded slice to 1500
chars.
- EH-1 block-comment-wrap bypass defense: drift detector
deliberately doesn't comment-strip jest.config.js (glob '/**'
sequences eat the block-comment regex). New Case 0 verifies
the 200 chars preceding 'coverageThreshold:' don't contain an
unmatched opening block-comment marker.
MED:
- BH-5 gitignore anchor: 'coverage/' (no leading /) matches at
any depth. Changed to '/coverage/' so a future sub-dir like
'docs/insurance-coverage/' isn't silently dropped.
- BH-8 / EH-2 upload-artifact: AC #5 marked upload as
'recommended' but wasn't implemented. Added
'actions/upload-artifact@v4' with 'if: always()' so gate-
failure PRs still upload coverage for triage. 14-day retention.
- EH-10 sibling-test exclude: extended collectCoverageFrom with
'!**/*.test.{ts,tsx}' + '!**/*.spec.{ts,tsx}' so a future
contributor adopting sibling-test convention can't artificially
inflate coverage by running test files end-to-end.
LOW:
- EH-9 fractional tolerance: drift detector regex now tolerates
'40' OR '40.0' (Prettier reformat resilience). Per-directory
scopes use general '\\d+(?:\\.\\d+)?' since calibrated
floors differ from 40.
Documented (no code change): BH-4/EH-7 CI cost trade-off (added
CI step comment), BH-6 scope gap + follow-up filed, BH-7/EH-5
forceExit + coverage interaction, EH-8 float-vs-rounded gotcha.
Deferred (filed as follow-ups):
- 15-6-followup-extend-coverage-scope-to-components-and-app (BH-6)
- 15-6-followup-lift-hooks-coverage-to-40 (R1 BH-2/EH-4)
+2 net drift cases (6 → 8). Coverage gate passes with calibrated
floors. All 4 quality gates green: type-check 0 / lint 0 /
prettier clean / jest 2167 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.maestro/directory with config.yaml + 5 golden-flow YAML files (sign-up, onboarding, first exercise, first conversation, mock-test partial→review)._bmad-output/planning-artifacts/runbooks/maestro-e2e-setup.md(install Maestro CLI, simulator setup, selector verification viamaestro studio, test account seeding, local execution, CI wire-up procedure).src/lib/__tests__/maestro-flows-source-drift.test.ts(5 cases) pins flow file existence + structure + NEGATIVE CI-not-wired-yet contract.15-4-followup-maestro-ci-wiring(needs simulator/emulator operator-setup).Story
15-4-golden-flow-e2e_bmad-output/implementation-artifacts/15-4-golden-flow-e2e.mdTest plan
maestro studioagainst each flow.🤖 Generated with Claude Code