From f149f472d7631cad2df403b2ec858036336b1516 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 18:14:45 +0300 Subject: [PATCH 01/56] feat: add testing-expert agent for test pyramid generation --- claude/commands/testing-expert.md | 7 ++ commands/testing-expert.md | 128 ++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) create mode 100644 claude/commands/testing-expert.md create mode 100644 commands/testing-expert.md diff --git a/claude/commands/testing-expert.md b/claude/commands/testing-expert.md new file mode 100644 index 00000000..5575f173 --- /dev/null +++ b/claude/commands/testing-expert.md @@ -0,0 +1,7 @@ +--- +description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. +--- + +Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. + +Refer to the instructions located in this file: .awos/commands/testing-expert.md diff --git a/commands/testing-expert.md b/commands/testing-expert.md new file mode 100644 index 00000000..53c2fd5e --- /dev/null +++ b/commands/testing-expert.md @@ -0,0 +1,128 @@ +--- +description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. +--- + +# ROLE + +You are an expert QA Engineer and Test Automation Specialist. You operate in two distinct modes depending on who calls you and what context you receive. + +--- + +# MODE DETECTION + +Read the invocation context: + +- **Planning Mode** — called by `/awos:tasks` with a functional spec and technical spec but NO existing implementation code. Your job is to return structured test task descriptions for `tasks.md`. You do NOT write test code in this mode. +- **Execution Mode** — called by `/awos:implement` with a specific test task, the implementation code, and full spec context. Your job is to write, RED-validate, and run real test code. + +--- + +# PLANNING MODE + +## Inputs +- `functional-spec.md` from the target spec directory +- `technical-considerations.md` from the target spec directory +- The implementation sub-task description for the current slice + +## Process + +### Step 1: Discover frameworks +1. Read `context/product/architecture.md` for declared testing stack per layer (unit/integration/e2e/contract). +2. If not declared, auto-detect from dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. +3. If still not determinable, note "framework TBD — will auto-detect at execution time" in the task description. + +### Step 2: Map acceptance criteria to test layers +For the given implementation sub-task, identify which acceptance criteria it touches. For each criterion, determine which layers apply: +- **Unit** — pure logic, no external dependencies +- **Integration** — service-to-service or DB interactions +- **E2E** — full user flow through the UI or API surface +- **Contract** — API schema/interface validation (OpenAPI, Pact, etc.) + +Not every slice needs all four layers. Apply judgment. + +### Step 3: Generate test task descriptions +For each applicable layer, generate two sub-tasks: one for positive cases, one for negative cases. + +**Output format** (return this list to `/awos:tasks` for insertion into `tasks.md`): + +``` +- [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** +- [ ] Unit: [describe negative/error inputs and boundary values] — negative cases **[Agent: testing-expert]** +- [ ] Integration: [describe service interaction scenarios] **[Agent: testing-expert]** +- [ ] Contract: [describe schema/interface validations + violation cases] **[Agent: testing-expert]** +- [ ] E2E: [describe full user flow — positive] **[Agent: testing-expert]** +- [ ] E2E: [describe failure/unhappy path flow — negative] **[Agent: testing-expert]** +``` + +Omit layers that genuinely don't apply. Always include negative cases for every layer that is included. + +--- + +# EXECUTION MODE + +## Inputs +- Specific test task description (layer + positive/negative scope) +- `functional-spec.md` + `technical-considerations.md` for the target spec +- `context/product/architecture.md` +- The implementation code written by the preceding impl sub-agent +- Current `context/qa/list-of-tests.md` (if it exists) + +## Process + +### Step 1: Discover frameworks +1. Read `context/product/architecture.md` for declared testing stack per layer. +2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. + +### Step 2: Plan test cases +Map the task's acceptance criteria to concrete test cases: +- For every positive case, define at least one negative counterpart. +- Negative cases must include: invalid inputs, boundary values, error paths, permission failures, malformed data — whichever apply to this layer. + +### Step 3: Write tests with RED validation + +Write tests following this discipline (borrowed from TDD red-green-refactor): + +1. Write one test case. +2. Run it. **Confirm it FAILS** — and that the failure message matches the missing behavior, not a syntax error. + - If it passes immediately: the test is not testing new behavior. Revise it until it fails for the right reason. +3. Proceed to the next test case. + +Annotate every test file with: +``` +# @layer: unit | integration | e2e | contract +# @spec: [spec-directory-name] +# @regression ← add only for tests that should be in the permanent regression suite +``` + +### Step 4: Confirm GREEN +Run all tests written in this task. All must pass before continuing. + +### Step 5: Check for implementation gaps +If tests reveal that the implementation is incomplete: +- Do NOT modify production code. +- Report the gap to `/awos:implement` with a clear description. +- A new impl sub-task will be created; this test task stays open until that sub-task closes. + +### Step 6: Update `context/qa/list-of-tests.md` +Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: +- **Same behavior, same layer** → UPDATE the existing entry instead of adding a new one. +- **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `# @deprecated`. +- **Partial overlap** → keep both, note the relationship in the Notes column. + +Append only net-new tests. Format: + +```markdown +| path/to/test_file.py | test_function_name | unit | negative | yes | OK | +``` + +### Step 7: Mark task [x] +Signal completion to `/awos:implement`. + +--- + +# CONSTRAINTS + +- Never modify production/implementation code — only test files. +- Never skip negative test cases — every included layer must have at least one negative test. +- RED validation is non-negotiable — a test that passes immediately without implementation proves nothing. +- Co-locate test files with source or follow the existing `tests/` directory convention in the project. From cef7f15bf79c396dde2abcc09741f7ed8d3189f9 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 18:22:16 +0300 Subject: [PATCH 02/56] fix: address code quality issues in testing-expert agent --- commands/testing-expert.md | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/commands/testing-expert.md b/commands/testing-expert.md index 53c2fd5e..4d5c140a 100644 --- a/commands/testing-expert.md +++ b/commands/testing-expert.md @@ -14,6 +14,7 @@ Read the invocation context: - **Planning Mode** — called by `/awos:tasks` with a functional spec and technical spec but NO existing implementation code. Your job is to return structured test task descriptions for `tasks.md`. You do NOT write test code in this mode. - **Execution Mode** — called by `/awos:implement` with a specific test task, the implementation code, and full spec context. Your job is to write, RED-validate, and run real test code. +- **Direct invocation (no caller context)** — ask the user: "Are you planning test tasks for a spec, or executing a specific test task?" then proceed to the appropriate mode. --- @@ -48,8 +49,10 @@ For each applicable layer, generate two sub-tasks: one for positive cases, one f ``` - [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** - [ ] Unit: [describe negative/error inputs and boundary values] — negative cases **[Agent: testing-expert]** -- [ ] Integration: [describe service interaction scenarios] **[Agent: testing-expert]** -- [ ] Contract: [describe schema/interface validations + violation cases] **[Agent: testing-expert]** +- [ ] Integration: [describe service interaction scenarios — positive cases] **[Agent: testing-expert]** +- [ ] Integration: [describe downstream failures, auth failures, malformed payloads — negative cases] **[Agent: testing-expert]** +- [ ] Contract: [describe schema/interface validations — positive cases] **[Agent: testing-expert]** +- [ ] Contract: [describe schema violations and malformed payload cases — negative cases] **[Agent: testing-expert]** - [ ] E2E: [describe full user flow — positive] **[Agent: testing-expert]** - [ ] E2E: [describe failure/unhappy path flow — negative] **[Agent: testing-expert]** ``` @@ -87,7 +90,7 @@ Write tests following this discipline (borrowed from TDD red-green-refactor): - If it passes immediately: the test is not testing new behavior. Revise it until it fails for the right reason. 3. Proceed to the next test case. -Annotate every test file with: +Annotate every test file with the following (use the appropriate comment syntax for the language: `#` for Python/Ruby/Shell, `//` for JS/TS/Go/Java, `/* */` for C/C++/C#): ``` # @layer: unit | integration | e2e | contract # @spec: [spec-directory-name] @@ -100,8 +103,9 @@ Run all tests written in this task. All must pass before continuing. ### Step 5: Check for implementation gaps If tests reveal that the implementation is incomplete: - Do NOT modify production code. -- Report the gap to `/awos:implement` with a clear description. -- A new impl sub-task will be created; this test task stays open until that sub-task closes. +- Report the gap by appending a note to this task's entry in `tasks.md`: + `` +- Do NOT invoke `/awos:implement` directly. Leave this task open (`[ ]`); `/awos:implement` will detect the incomplete task on its next run and create a new impl sub-task to close the gap. ### Step 6: Update `context/qa/list-of-tests.md` Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: @@ -112,7 +116,9 @@ Before appending new entries, scan the registry for existing tests covering the Append only net-new tests. Format: ```markdown -| path/to/test_file.py | test_function_name | unit | negative | yes | OK | +| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | +|------|-----------|-------|-------------------|-------------|--------|-------| +| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | ``` ### Step 7: Mark task [x] From edddd18c619010799e96081554e2b32cee3bb206 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 18:37:50 +0300 Subject: [PATCH 03/56] feat: extend /awos:tasks to generate test pyramid tasks per vertical slice --- commands/tasks.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/commands/tasks.md b/commands/tasks.md index b4f63a6d..4821bb8f 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -67,6 +67,23 @@ Follow this process precisely. - Tech stack identified in technical-considerations.md - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table + 3b. **Generate paired test tasks for each slice (REQUIRED):** + - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. + - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. + - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. + - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. + - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. + - Example result for a slice: + ``` + - [ ] **Slice 1: User authentication** + - [ ] Implement JWT token generation **[Agent: python-expert]** + - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** + - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** + - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** + - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** + - [ ] Contract: /auth response schema validation — positive cases **[Agent: testing-expert]** + - [ ] Contract: schema violations and malformed payload cases — negative cases **[Agent: testing-expert]** + ``` 5. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 6. Create a high-level checklist item and its sub-tasks with subagent assignments. 7. Repeat this process until all requirements from the specification are covered. @@ -86,6 +103,9 @@ Follow this process precisely. - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` - `[ ] Sub-task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` + - `[ ] Unit: avatar_url column default, null handling — positive/negative **[Agent: testing-expert]**` + - `[ ] Integration: GET /user returns avatar_url when set, null when not — positive/negative **[Agent: testing-expert]**` + - `[ ] E2E: profile page shows avatar when present, placeholder when null **[Agent: testing-expert]**` ## Step 4: Present Draft and Refine From cfb2854dd378f6df4bb0bda578037b9b16be6025 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 20:49:07 +0300 Subject: [PATCH 04/56] fix: split positive/negative test examples into separate tasks in tasks.md --- commands/tasks.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index 4821bb8f..b8fa4df1 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -103,9 +103,12 @@ Follow this process precisely. - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` - `[ ] Sub-task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` - - `[ ] Unit: avatar_url column default, null handling — positive/negative **[Agent: testing-expert]**` - - `[ ] Integration: GET /user returns avatar_url when set, null when not — positive/negative **[Agent: testing-expert]**` - - `[ ] E2E: profile page shows avatar when present, placeholder when null **[Agent: testing-expert]**` + - `[ ] Unit: avatar_url column default, valid URL stored correctly — positive cases **[Agent: testing-expert]**` + - `[ ] Unit: null avatar_url, missing column, type mismatch — negative cases **[Agent: testing-expert]**` + - `[ ] Integration: GET /user returns avatar_url when set — positive cases **[Agent: testing-expert]**` + - `[ ] Integration: GET /user returns null when avatar_url not set — negative cases **[Agent: testing-expert]**` + - `[ ] E2E: profile page shows avatar when avatar_url present — positive cases **[Agent: testing-expert]**` + - `[ ] E2E: profile page shows placeholder when avatar_url is null — negative cases **[Agent: testing-expert]**` ## Step 4: Present Draft and Refine From b4e12f566dffef0fed586737c96d0dd88c84b33c Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 20:50:43 +0300 Subject: [PATCH 05/56] feat: add /awos:qa optional full-audit command --- claude/commands/qa.md | 7 ++ commands/qa.md | 166 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 173 insertions(+) create mode 100644 claude/commands/qa.md create mode 100644 commands/qa.md diff --git a/claude/commands/qa.md b/claude/commands/qa.md new file mode 100644 index 00000000..1752c836 --- /dev/null +++ b/claude/commands/qa.md @@ -0,0 +1,7 @@ +--- +description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. +--- + +Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. + +Refer to the instructions located in this file: .awos/commands/qa.md diff --git a/commands/qa.md b/commands/qa.md new file mode 100644 index 00000000..3678c3b8 --- /dev/null +++ b/commands/qa.md @@ -0,0 +1,166 @@ +--- +description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. +--- + +# ROLE + +You are a senior QA Architect running a full audit of the test suite for one or all specifications. You analyze existing tests, identify coverage gaps, generate missing tests, manage the regression suite, and produce a structured audit report. You do not modify production code. + +--- + +# TASK + +Perform a full QA audit for the target spec(s). Check existing tests for health (missing, stale, deprecated), identify gaps against spec acceptance criteria, generate missing tests, offer to run the suite, and produce an audit report in `context/qa/audit-reports/`. + +--- + +# INPUTS & OUTPUTS + +- **User Prompt (Optional):** $ARGUMENTS + - Empty = audit all specs + - Spec name/index = audit that spec only (e.g., `/awos:qa 001-user-auth`) +- **Primary Context:** + - `context/qa/list-of-tests.md` — global test registry + - `context/qa/regression-suite.md` — regression-flagged tests + - `context/product/architecture.md` — testing framework declarations + - All `functional-spec.md` files in scope + - Live codebase + existing test files +- **Output Files:** + - Updated `context/qa/list-of-tests.md` + - Updated `context/qa/regression-suite.md` + - New `context/qa/audit-reports/qa-report-YYYY-MM-DD.md` + +--- + +# PROCESS + +## Step 1: Identify scope + +1. Read ``. If it names a spec, target that directory only. +2. If empty, target all spec directories under `context/spec/`. +3. Announce: "Running QA audit for: [scope]." + +## Step 2: Discover frameworks + +1. Read `context/product/architecture.md` for declared testing stack per layer. +2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. + +## Step 3: Load test registry + +Read `context/qa/list-of-tests.md`. If it does not exist, create it from the template at `.awos/templates/qa-context-template.md` (or create an empty registry if template is absent). + +## Step 4: Audit existing tests + +For each registered test in scope: + +**Existence check:** +- Does the test file still exist in the codebase? If not → flag `MISSING`. + +**Spec linkage check:** +- Does the `@spec` annotation reference a spec directory that still exists? + - YES → OK. + - Spec directory deleted → + - Search all active `functional-spec.md` files for matching behavior/acceptance criterion. + - Match found → re-link `@spec` to the new spec, mark OK. + - No match → flag `HUMAN REVIEW` (do NOT auto-deprecate; may be intentional regression coverage). + +**Staleness check:** +- Does the test logic match the current implementation? + - Read the test file and the relevant implementation code. + - If the implementation has changed in ways that invalidate the test → flag `NEEDS UPDATE` and generate a diff suggestion. Do NOT auto-modify the test. + +**Regression tag:** +- If tagged `@regression` → carry forward to the regression-suite.md sync in Step 8. + +## Step 5: Gap analysis + +For each acceptance criterion in the functional spec(s) in scope: + +1. Check which pyramid layers have coverage (unit / integration / e2e / contract). +2. Check whether each covered layer has a negative test counterpart. +3. Record gaps: + - Layer with no tests at all → `MISSING LAYER` + - Layer with only positive tests → `MISSING NEGATIVE` + +## Step 6: Generate missing tests + +For each gap identified in Step 5: + +1. Write the missing test following RED validation discipline: + - Write test → confirm it FAILS for the right reason → confirm it PASSES. +2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations using appropriate comment syntax for the language. +3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: + - Same behavior, same layer → UPDATE existing entry instead of adding new. + - Broader test needing splitting → DEPRECATE old (annotate with `@deprecated`), add focused replacements. + - Partial overlap → keep both, annotate relationship in Notes column. + +## Step 7: Run tests (with user confirmation) + +1. Count all tests in scope. Notify user using `AskUserQuestion`: + ``` + Ready to run N tests across X specs [unit: A, integration: B, e2e: C, contract: D] + Regression suite: M tests tagged @regression + ``` +2. Ask user to choose: + - **A) Full suite** — run all N tests + - **B) Regression suite only** — run M tests tagged @regression + - **C) Skip** — do not run tests this session +3. Wait for user confirmation before executing. +4. Run the selected suite. Collect pass/fail results per layer. + +## Step 8: Update regression suite + +Sync `context/qa/regression-suite.md`: +- Scan all test files in scope for `@regression` annotations. +- Add newly tagged tests; remove entries for tests that no longer exist or have lost their `@regression` tag. +- Update "Last updated" date and total count. + +## Step 9: Produce audit report + +Save to `context/qa/audit-reports/qa-report-YYYY-MM-DD.md`: + +```markdown +# QA Audit Report — YYYY-MM-DD + +## Scope +[Spec(s) audited] + +## Coverage Summary +| Spec | Unit | Integration | E2E | Contract | ACs Covered | +|------|------|-------------|-----|----------|-------------| +| [spec] | X/Y | X/Y | X/Y | X/Y | X/Y (Z%) | + +## Flags +- MISSING: [file] — [reason] +- NEEDS UPDATE: [file]::[test] — [what changed] +- MISSING LAYER: [spec] — [AC] has no [layer] coverage +- MISSING NEGATIVE: [spec] — [AC] [layer] has no negative test +- HUMAN REVIEW: [file]::[test] — spec deleted, no active match found + +## Regression Suite Delta +- Added: N | Removed: N | Total: N + +## Run Results +[If tests were run:] +- Suite: [Full / Regression only] +- Passed: N | Failed: N | Blocked: N +- FAILED: [file]::[test] — [brief reason] + +## Recommendation +- [ ] Ready — all critical ACs covered, suite passing +- [ ] Needs attention — [N] gaps or failures require action +``` + +## Step 10: Announce + +Report summary to user and list any flags requiring human attention. + +--- + +# CONSTRAINTS + +- Never modify production code. +- Never auto-deprecate tests whose spec was deleted — always flag for human review. +- Always ask for user confirmation before running tests (use `AskUserQuestion`). +- Generate a diff suggestion for stale tests — do not auto-rewrite them. +- Never skip the overlap check when updating `list-of-tests.md`. From 3411e5d8d59b902d8d37ca7b24fc58330725a142 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 21:17:07 +0300 Subject: [PATCH 06/56] fix: restore testing-expert cross-reference and improve qa.md clarity --- commands/qa.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/commands/qa.md b/commands/qa.md index 3678c3b8..e5c46036 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -86,7 +86,7 @@ For each acceptance criterion in the functional spec(s) in scope: For each gap identified in Step 5: -1. Write the missing test following RED validation discipline: +1. Write the missing test following RED validation discipline (following `testing-expert` execution mode Step 3): - Write test → confirm it FAILS for the right reason → confirm it PASSES. 2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations using appropriate comment syntax for the language. 3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: @@ -94,9 +94,15 @@ For each gap identified in Step 5: - Broader test needing splitting → DEPRECATE old (annotate with `@deprecated`), add focused replacements. - Partial overlap → keep both, annotate relationship in Notes column. +Append net-new entries using this format: + +| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | +|------|-----------|-------|-------------------|-------------|--------|-------| +| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | + ## Step 7: Run tests (with user confirmation) -1. Count all tests in scope. Notify user using `AskUserQuestion`: +1. Count all tests in scope. Use `AskUserQuestion` with the following question and options: ``` Ready to run N tests across X specs [unit: A, integration: B, e2e: C, contract: D] Regression suite: M tests tagged @regression From be37c251ad86315263fcbbad5fbd1f3f70b1f002 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 21:18:35 +0300 Subject: [PATCH 07/56] feat: add QA context templates for test registry and regression suite --- templates/qa-context-template.md | 11 +++++++++++ templates/regression-suite-template.md | 11 +++++++++++ 2 files changed, 22 insertions(+) create mode 100644 templates/qa-context-template.md create mode 100644 templates/regression-suite-template.md diff --git a/templates/qa-context-template.md b/templates/qa-context-template.md new file mode 100644 index 00000000..68c01880 --- /dev/null +++ b/templates/qa-context-template.md @@ -0,0 +1,11 @@ +# Test Registry + +> Auto-maintained by `testing-expert` (execution mode) and `/awos:qa`. +> Each row is a registered test. Add rows by running `/awos:implement` (test tasks) or `/awos:qa`. +> Status values: OK | MISSING | NEEDS UPDATE | DEPRECATED | HUMAN REVIEW +> Layer values: unit | integration | e2e | contract + + + + + diff --git a/templates/regression-suite-template.md b/templates/regression-suite-template.md new file mode 100644 index 00000000..db356d2c --- /dev/null +++ b/templates/regression-suite-template.md @@ -0,0 +1,11 @@ +# Regression Suite + +> Auto-maintained by `/awos:qa` Step 8. Synced from `@regression` annotations in test files. +> Run with: `/awos:qa` → choose "Regression suite only". + +**Last updated:** YYYY-MM-DD +**Total:** 0 tests + + + + From d2f9c0710ce1a2f36a71d12306d9d44500329b5b Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 21:19:22 +0300 Subject: [PATCH 08/56] docs: add QA pyramid agent implementation plan --- .../plans/2026-04-07-qa-pyramid-agent.md | 575 ++++++++++++++++++ 1 file changed, 575 insertions(+) create mode 100644 docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md diff --git a/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md b/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md new file mode 100644 index 00000000..7b1cc712 --- /dev/null +++ b/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md @@ -0,0 +1,575 @@ +# QA Pyramid Agent Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Extend AWOS with a full test pyramid QA subsystem — a `testing-expert` sub-agent woven into vertical slices, a modified `/awos:tasks` that generates paired test tasks, and an optional `/awos:qa` full-audit command. + +**Architecture:** The `testing-expert` agent operates in two modes: planning mode (called by `/awos:tasks` to generate test task descriptions) and execution mode (called by `/awos:implement` to write and run actual tests). `/awos:qa` is a standalone optional audit command that reads the global `context/qa/` registry, identifies gaps, and manages the regression suite. All changes are additive — no existing AWOS commands are broken. + +**Tech Stack:** Markdown prompt files, no runtime dependencies. Agent definitions follow the AWOS `agent-template.md` convention. Command wrappers follow the existing `claude/commands/` thin-wrapper pattern. + +--- + +## Files + +| Action | File | Responsibility | +|--------|------|----------------| +| CREATE | `commands/qa.md` | `/awos:qa` full-audit command logic | +| CREATE | `claude/commands/qa.md` | Thin wrapper pointing to `commands/qa.md` | +| CREATE | `commands/testing-expert.md` | `testing-expert` agent — planning + execution modes | +| CREATE | `claude/commands/testing-expert.md` | Thin wrapper making testing-expert invocable as `/awos:testing-expert` | +| MODIFY | `commands/tasks.md` | Add Step 3b: invoke testing-expert to generate test tasks per slice | +| CREATE | `templates/qa-context-template.md` | Starter template for `context/qa/list-of-tests.md` | +| CREATE | `templates/regression-suite-template.md` | Starter template for `context/qa/regression-suite.md` | + +--- + +## Task 1: Create `testing-expert` agent + +**Files:** +- Create: `commands/testing-expert.md` + +The `testing-expert` agent has two modes. **Planning mode** produces test task descriptions for `tasks.md`. **Execution mode** writes, RED-validates, and runs actual test code. + +- [ ] **Step 1: Write `commands/testing-expert.md`** + +```markdown +--- +description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. +--- + +# ROLE + +You are an expert QA Engineer and Test Automation Specialist. You operate in two distinct modes depending on who calls you and what context you receive. + +--- + +# MODE DETECTION + +Read the invocation context: + +- **Planning Mode** — called by `/awos:tasks` with a functional spec and technical spec but NO existing implementation code. Your job is to return structured test task descriptions for `tasks.md`. You do NOT write test code in this mode. +- **Execution Mode** — called by `/awos:implement` with a specific test task, the implementation code, and full spec context. Your job is to write, RED-validate, and run real test code. + +--- + +# PLANNING MODE + +## Inputs +- `functional-spec.md` from the target spec directory +- `technical-considerations.md` from the target spec directory +- The implementation sub-task description for the current slice + +## Process + +### Step 1: Discover frameworks +1. Read `context/product/architecture.md` for declared testing stack per layer (unit/integration/e2e/contract). +2. If not declared, auto-detect from dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. +3. If still not determinable, note "framework TBD — will auto-detect at execution time" in the task description. + +### Step 2: Map acceptance criteria to test layers +For the given implementation sub-task, identify which acceptance criteria it touches. For each criterion, determine which layers apply: +- **Unit** — pure logic, no external dependencies +- **Integration** — service-to-service or DB interactions +- **E2E** — full user flow through the UI or API surface +- **Contract** — API schema/interface validation (OpenAPI, Pact, etc.) + +Not every slice needs all four layers. Apply judgment. + +### Step 3: Generate test task descriptions +For each applicable layer, generate two sub-tasks: one for positive cases, one for negative cases. + +**Output format** (return this list to `/awos:tasks` for insertion into `tasks.md`): + +``` +- [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** +- [ ] Unit: [describe negative/error inputs and boundary values] — negative cases **[Agent: testing-expert]** +- [ ] Integration: [describe service interaction scenarios] **[Agent: testing-expert]** +- [ ] Contract: [describe schema/interface validations + violation cases] **[Agent: testing-expert]** +- [ ] E2E: [describe full user flow — positive] **[Agent: testing-expert]** +- [ ] E2E: [describe failure/unhappy path flow — negative] **[Agent: testing-expert]** +``` + +Omit layers that genuinely don't apply. Always include negative cases for every layer that is included. + +--- + +# EXECUTION MODE + +## Inputs +- Specific test task description (layer + positive/negative scope) +- `functional-spec.md` + `technical-considerations.md` for the target spec +- `context/product/architecture.md` +- The implementation code written by the preceding impl sub-agent +- Current `context/qa/list-of-tests.md` (if it exists) + +## Process + +### Step 1: Discover frameworks +1. Read `context/product/architecture.md` for declared testing stack per layer. +2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. + +### Step 2: Plan test cases +Map the task's acceptance criteria to concrete test cases: +- For every positive case, define at least one negative counterpart. +- Negative cases must include: invalid inputs, boundary values, error paths, permission failures, malformed data — whichever apply to this layer. + +### Step 3: Write tests with RED validation + +Write tests following this discipline (borrowed from TDD red-green-refactor): + +1. Write one test case. +2. Run it. **Confirm it FAILS** — and that the failure message matches the missing behavior, not a syntax error. + - If it passes immediately: the test is not testing new behavior. Revise it until it fails for the right reason. +3. Proceed to the next test case. + +Annotate every test file with: +``` +# @layer: unit | integration | e2e | contract +# @spec: [spec-directory-name] +# @regression ← add only for tests that should be in the permanent regression suite +``` + +### Step 4: Confirm GREEN +Run all tests written in this task. All must pass before continuing. + +### Step 5: Check for implementation gaps +If tests reveal that the implementation is incomplete: +- Do NOT modify production code. +- Report the gap to `/awos:implement` with a clear description. +- A new impl sub-task will be created; this test task stays open until that sub-task closes. + +### Step 6: Update `context/qa/list-of-tests.md` +Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: +- **Same behavior, same layer** → UPDATE the existing entry instead of adding a new one. +- **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `# @deprecated`. +- **Partial overlap** → keep both, note the relationship in the Notes column. + +Append only net-new tests. Format: + +```markdown +| path/to/test_file.py | test_function_name | unit | negative | yes | OK | +``` + +### Step 7: Mark task [x] +Signal completion to `/awos:implement`. + +--- + +# CONSTRAINTS + +- Never modify production/implementation code — only test files. +- Never skip negative test cases — every included layer must have at least one negative test. +- RED validation is non-negotiable — a test that passes immediately without implementation proves nothing. +- Co-locate test files with source or follow the existing `tests/` directory convention in the project. +``` + +- [ ] **Step 2: Verify the file is well-formed** + +```bash +cat commands/testing-expert.md | head -5 +``` +Expected output: starts with `---` frontmatter. + +- [ ] **Step 2b: Write `claude/commands/testing-expert.md` (thin wrapper)** + +```markdown +--- +description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. +--- + +Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. + +Refer to the instructions located in this file: .awos/commands/testing-expert.md +``` + +- [ ] **Step 3: Commit** + +```bash +git add commands/testing-expert.md claude/commands/testing-expert.md +git commit -m "feat: add testing-expert agent for test pyramid generation" +``` + +--- + +## Task 2: Modify `/awos:tasks` to generate test tasks per slice + +**Files:** +- Modify: `commands/tasks.md` + +We add a new **Step 3b** after each implementation sub-task is defined. It invokes `testing-expert` in planning mode to generate paired test sub-tasks for that slice. + +- [ ] **Step 1: Read the current `commands/tasks.md`** + +```bash +cat commands/tasks.md +``` + +Identify the end of Step 3 ("Your Thought Process for Generating Tasks") — specifically after sub-step 3 where nested sub-tasks are created for a slice. + +- [ ] **Step 2: Add Step 3b — test task generation** + +After the line: +``` + - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table +``` + +Insert the following new sub-step **3b**: + +```markdown + 3b. **Generate paired test tasks for each slice (REQUIRED):** + - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. + - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. + - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. + - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. + - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. + - Example result for a slice: + ``` + - [ ] **Slice 1: User authentication** + - [ ] Implement JWT token generation **[Agent: python-expert]** + - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** + - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** + - [ ] Integration: valid/invalid credentials against /auth endpoint **[Agent: testing-expert]** + - [ ] Contract: /auth response schema validation + violation cases **[Agent: testing-expert]** + ``` +``` + +- [ ] **Step 3: Update the example in Step 3 to reflect test tasks** + +Find the existing "Good, Vertical Slices" example: +```markdown + - `[ ] **Slice 2: Display the user's actual avatar if it exists**` + - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` + ... + - `[ ] Sub-task: Run the application. Use chrome MCP... **[Agent: manual-qa-expert]**` +``` + +Append test sub-tasks to Slice 2 in the example: +```markdown + - `[ ] Unit: avatar_url column default, null handling — positive/negative **[Agent: testing-expert]**` + - `[ ] Integration: GET /user returns avatar_url when set, null when not **[Agent: testing-expert]**` + - `[ ] E2E: profile page shows avatar when present, placeholder when null **[Agent: testing-expert]**` +``` + +- [ ] **Step 4: Verify the file reads cleanly** + +```bash +cat commands/tasks.md | grep -A5 "3b" +``` +Expected: shows the new Step 3b block. + +- [ ] **Step 5: Commit** + +```bash +git add commands/tasks.md +git commit -m "feat: extend /awos:tasks to generate test pyramid tasks per vertical slice" +``` + +--- + +## Task 3: Create `/awos:qa` command + +**Files:** +- Create: `commands/qa.md` +- Create: `claude/commands/qa.md` + +- [ ] **Step 1: Write `commands/qa.md`** + +```markdown +--- +description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. +--- + +# ROLE + +You are a senior QA Architect running a full audit of the test suite for one or all specifications. You analyze existing tests, identify coverage gaps, generate missing tests, manage the regression suite, and produce a structured audit report. You do not modify production code. + +--- + +# TASK + +Perform a full QA audit for the target spec(s). Check existing tests for health (missing, stale, deprecated), identify gaps against spec acceptance criteria, generate missing tests, offer to run the suite, and produce an audit report in `context/qa/audit-reports/`. + +--- + +# INPUTS & OUTPUTS + +- **User Prompt (Optional):** $ARGUMENTS + - Empty = audit all specs + - Spec name/index = audit that spec only (e.g., `/awos:qa 001-user-auth`) +- **Primary Context:** + - `context/qa/list-of-tests.md` — global test registry + - `context/qa/regression-suite.md` — regression-flagged tests + - `context/product/architecture.md` — testing framework declarations + - All `functional-spec.md` files in scope + - Live codebase + existing test files +- **Output Files:** + - Updated `context/qa/list-of-tests.md` + - Updated `context/qa/regression-suite.md` + - New `context/qa/audit-reports/qa-report-YYYY-MM-DD.md` + +--- + +# PROCESS + +## Step 1: Identify scope + +1. Read ``. If it names a spec, target that directory only. +2. If empty, target all spec directories under `context/spec/`. +3. Announce: "Running QA audit for: [scope]." + +## Step 2: Discover frameworks + +1. Read `context/product/architecture.md` for declared testing stack per layer. +2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. + +## Step 3: Load test registry + +Read `context/qa/list-of-tests.md`. If it does not exist, create it from the template at `.awos/templates/qa-context-template.md` (or create an empty registry if template is absent). + +## Step 4: Audit existing tests + +For each registered test in scope: + +**Existence check:** +- Does the test file still exist in the codebase? If not → flag `MISSING`. + +**Spec linkage check:** +- Does the `@spec` annotation reference a spec directory that still exists? + - YES → OK. + - Spec directory deleted → + - Search all active `functional-spec.md` files for matching behavior/acceptance criterion. + - Match found → re-link `@spec` to the new spec, mark OK. + - No match → flag `HUMAN REVIEW` (do NOT auto-deprecate; may be intentional regression coverage). + +**Staleness check:** +- Does the test logic match the current implementation? + - Read the test file and the relevant implementation code. + - If the implementation has changed in ways that invalidate the test → flag `NEEDS UPDATE` and generate a diff suggestion. Do NOT auto-modify. + +**Regression tag:** +- If tagged `@regression` → carry forward to the regression-suite.md sync in Step 7. + +## Step 5: Gap analysis + +For each acceptance criterion in the functional spec(s) in scope: + +1. Check which pyramid layers have coverage (unit / integration / e2e / contract). +2. Check whether each covered layer has a negative test counterpart. +3. Record gaps: + - Layer with no tests at all → `MISSING LAYER` + - Layer with only positive tests → `MISSING NEGATIVE` + +## Step 6: Generate missing tests + +For each gap identified in Step 5: + +1. Write the missing test following RED validation discipline (from `testing-expert` execution mode): + - Write test → confirm it FAILS for the right reason → confirm it PASSES. +2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations. +3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: + - Same behavior, same layer → UPDATE existing entry. + - Broader test needing splitting → DEPRECATE old, add focused replacements. + - Partial overlap → keep both, annotate relationship. + +## Step 7: Run tests (with user confirmation) + +1. Count all tests in scope. Notify user: + ``` + Ready to run N tests across X specs [unit: A, integration: B, e2e: C, contract: D] + Regression suite: M tests tagged @regression + ``` +2. Ask user to choose: + - **A) Full suite** — run all N tests + - **B) Regression suite only** — run M tests tagged @regression + - **C) Skip** — do not run tests this session +3. Wait for user confirmation before executing. +4. Run the selected suite. Collect pass/fail results per layer. + +## Step 8: Update regression suite + +Sync `context/qa/regression-suite.md`: +- Scan all test files in scope for `@regression` annotations. +- Add newly tagged tests; remove entries for tests that no longer exist or have lost their `@regression` tag. +- Update "Last updated" date and total count. + +## Step 9: Produce audit report + +Save to `context/qa/audit-reports/qa-report-YYYY-MM-DD.md`: + +```markdown +# QA Audit Report — YYYY-MM-DD + +## Scope +[Spec(s) audited] + +## Coverage Summary +| Spec | Unit | Integration | E2E | Contract | ACs Covered | +|------|------|-------------|-----|----------|-------------| +| [spec] | X/Y | X/Y | X/Y | X/Y | X/Y (Z%) | + +## Flags +- MISSING: [file] — [reason] +- NEEDS UPDATE: [file]::[test] — [what changed] +- MISSING LAYER: [spec] — [AC] has no [layer] coverage +- MISSING NEGATIVE: [spec] — [AC] [layer] has no negative test +- HUMAN REVIEW: [file]::[test] — spec deleted, no active match found + +## Regression Suite Delta +- Added: N | Removed: N | Total: N + +## Run Results +[If tests were run:] +- Suite: [Full / Regression only] +- Passed: N | Failed: N | Blocked: N +- FAILED: [file]::[test] — [brief reason] + +## Recommendation +- [ ] Ready — all critical ACs covered, suite passing +- [ ] Needs attention — [N] gaps or failures require action +``` + +## Step 10: Announce + +Report summary to user and list any flags requiring human attention. + +--- + +# CONSTRAINTS + +- Never modify production code. +- Never auto-deprecate tests whose spec was deleted — always flag for human review. +- Always ask for user confirmation before running tests. +- Generate a diff suggestion for stale tests — do not auto-rewrite them. +``` + +- [ ] **Step 2: Write `claude/commands/qa.md` (thin wrapper)** + +```markdown +--- +description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. +--- + +Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. + +Refer to the instructions located in this file: .awos/commands/qa.md +``` + +- [ ] **Step 3: Verify both files exist** + +```bash +ls commands/qa.md claude/commands/qa.md +``` +Expected: both files listed. + +- [ ] **Step 4: Commit** + +```bash +git add commands/qa.md claude/commands/qa.md +git commit -m "feat: add /awos:qa optional full-audit command" +``` + +--- + +## Task 4: Create QA context templates + +**Files:** +- Create: `templates/qa-context-template.md` +- Create: `templates/regression-suite-template.md` + +These templates are used by `/awos:qa` when initializing a fresh `context/qa/` directory in a project. + +- [ ] **Step 1: Write `templates/qa-context-template.md`** + +```markdown +# Test Registry + +> Auto-maintained by `testing-expert` (execution mode) and `/awos:qa`. +> Each row is a registered test. Add rows by running `/awos:implement` (test tasks) or `/awos:qa`. +> Status values: OK | MISSING | NEEDS UPDATE | DEPRECATED | HUMAN REVIEW +> Layer values: unit | integration | e2e | contract + + + + + +``` + +- [ ] **Step 2: Write `templates/regression-suite-template.md`** + +```markdown +# Regression Suite + +> Auto-maintained by `/awos:qa` Step 8. Synced from `@regression` annotations in test files. +> Run with: `/awos:qa` → choose "Regression suite only". + +**Last updated:** YYYY-MM-DD +**Total:** 0 tests + + + + +``` + +- [ ] **Step 3: Verify both files exist** + +```bash +ls templates/qa-context-template.md templates/regression-suite-template.md +``` +Expected: both files listed. + +- [ ] **Step 4: Commit** + +```bash +git add templates/qa-context-template.md templates/regression-suite-template.md +git commit -m "feat: add QA context templates for test registry and regression suite" +``` + +--- + +## Task 5: Final validation + +- [ ] **Step 1: Confirm all files are present** + +```bash +git diff main --name-only +``` +Expected output: +``` +claude/commands/qa.md +claude/commands/testing-expert.md +commands/qa.md +commands/tasks.md +commands/testing-expert.md +docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md +templates/qa-context-template.md +templates/regression-suite-template.md +``` + +- [ ] **Step 2: Confirm all command wrappers reference the right paths** + +```bash +grep "Refer to" claude/commands/qa.md +``` +Expected: `Refer to the instructions located in this file: .awos/commands/qa.md` + +- [ ] **Step 3: Confirm `testing-expert` is referenced correctly in `tasks.md`** + +```bash +grep "testing-expert" commands/tasks.md | head -5 +``` +Expected: shows Step 3b referencing `testing-expert` in planning mode. + +- [ ] **Step 4: Confirm no placeholder text remains** + +```bash +grep -rn "TBD\|TODO\|PLACEHOLDER\|fill in" commands/qa.md commands/testing-expert.md commands/tasks.md templates/qa-context-template.md templates/regression-suite-template.md +``` +Expected: no matches. + +- [ ] **Step 5: Commit plan doc** + +```bash +git add docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md +git commit -m "docs: add QA pyramid agent implementation plan" +``` From 93ab40af2c6b8390781b6041ea5e738da8c2343e Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 21:27:03 +0300 Subject: [PATCH 09/56] =?UTF-8?q?chore:=20remove=20unnecessary=20testing-e?= =?UTF-8?q?xpert=20command=20wrapper=20=E2=80=94=20agent=20is=20internal-o?= =?UTF-8?q?nly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- claude/commands/testing-expert.md | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 claude/commands/testing-expert.md diff --git a/claude/commands/testing-expert.md b/claude/commands/testing-expert.md deleted file mode 100644 index 5575f173..00000000 --- a/claude/commands/testing-expert.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. ---- - -Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. - -Refer to the instructions located in this file: .awos/commands/testing-expert.md From 582c95a7ee5893481f6fc33efe3ecbdb968b6fda Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 21:27:12 +0300 Subject: [PATCH 10/56] docs: remove testing-expert wrapper entry from plan file table --- docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md b/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md index 7b1cc712..ef640ff8 100644 --- a/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md +++ b/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md @@ -17,7 +17,6 @@ | CREATE | `commands/qa.md` | `/awos:qa` full-audit command logic | | CREATE | `claude/commands/qa.md` | Thin wrapper pointing to `commands/qa.md` | | CREATE | `commands/testing-expert.md` | `testing-expert` agent — planning + execution modes | -| CREATE | `claude/commands/testing-expert.md` | Thin wrapper making testing-expert invocable as `/awos:testing-expert` | | MODIFY | `commands/tasks.md` | Add Step 3b: invoke testing-expert to generate test tasks per slice | | CREATE | `templates/qa-context-template.md` | Starter template for `context/qa/list-of-tests.md` | | CREATE | `templates/regression-suite-template.md` | Starter template for `context/qa/regression-suite.md` | From c97c91abc93677483f9b7a313283ad1a6ed87473 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 7 Apr 2026 21:33:10 +0300 Subject: [PATCH 11/56] fix: resolve contradictions in testing-expert and qa commands --- commands/qa.md | 4 +++- commands/testing-expert.md | 8 +++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/commands/qa.md b/commands/qa.md index e5c46036..a8f78ccf 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -91,14 +91,16 @@ For each gap identified in Step 5: 2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations using appropriate comment syntax for the language. 3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: - Same behavior, same layer → UPDATE existing entry instead of adding new. - - Broader test needing splitting → DEPRECATE old (annotate with `@deprecated`), add focused replacements. + - Broader test needing splitting → DEPRECATE old (annotate with `@deprecated` using appropriate comment syntax for the language), add focused replacements. - Partial overlap → keep both, annotate relationship in Notes column. Append net-new entries using this format: +```markdown | File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | |------|-----------|-------|-------------------|-------------|--------|-------| | path/to/test_file.py | test_function_name | unit | negative | yes | OK | | +``` ## Step 7: Run tests (with user confirmation) diff --git a/commands/testing-expert.md b/commands/testing-expert.md index 4d5c140a..1195d0b5 100644 --- a/commands/testing-expert.md +++ b/commands/testing-expert.md @@ -110,7 +110,7 @@ If tests reveal that the implementation is incomplete: ### Step 6: Update `context/qa/list-of-tests.md` Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: - **Same behavior, same layer** → UPDATE the existing entry instead of adding a new one. -- **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `# @deprecated`. +- **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `@deprecated` using the appropriate comment syntax for the language. - **Partial overlap** → keep both, note the relationship in the Notes column. Append only net-new tests. Format: @@ -121,8 +121,10 @@ Append only net-new tests. Format: | path/to/test_file.py | test_function_name | unit | negative | yes | OK | | ``` -### Step 7: Mark task [x] -Signal completion to `/awos:implement`. +### Step 7: Report completion status to `/awos:implement` + +- **No gaps found:** All tests pass. Your work is done — `/awos:implement` will mark this task `[x]`. +- **Gap found (Step 5 triggered):** Do NOT signal completion. Return an incomplete/blocked status so `/awos:implement` knows NOT to mark this task `[x]`. The task stays open until the gap impl sub-task is resolved and tests pass. --- From 3ff18faaec1b10b6a5067d6dbf0488347cdff61f Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Thu, 9 Apr 2026 15:32:31 +0300 Subject: [PATCH 12/56] docs(qa): add TODO section with known limitations Documents three open gaps discovered during end-to-end testing: ephemeral E2E artifacts, coverage-by-inspection vs measurement, and no regression baseline/enforcement mechanism. Co-Authored-By: Claude Sonnet 4.6 --- commands/qa.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/commands/qa.md b/commands/qa.md index a8f78ccf..2a29d775 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -165,6 +165,16 @@ Report summary to user and list any flags requiring human attention. --- +# TODO + +- **E2E tests are ephemeral — no CI artifact.** The `/awos:qa` command runs E2E tests via the `e2e-tester` agent (playwright-cli) interactively but produces no persistent test scripts. Each run is unrepeatable without a human in the loop. Future work: generate rerunnable playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as a Step 6 output for E2E gaps, so they can be committed and executed in CI without agent interaction. + +- **QA audit is coverage-by-inspection, not coverage-by-measurement.** Step 5 (gap analysis) reads source files and infers coverage from test file contents. It does not invoke actual coverage tooling (`vitest --coverage`, Istanbul, c8, pytest-cov, etc.). This means untested branches and dead-path gaps are invisible to the audit. Future work: in Step 2, detect available coverage reporters and, in Step 7, run the suite with coverage flags; parse the output to feed real line/branch metrics into the Coverage Summary table. + +- **Audit reports are snapshots with no regression baseline or enforcement.** Each `/awos:qa` run produces a dated report, but there is no mechanism to diff successive reports, track coverage trend over time, or fail a build when coverage drops below a threshold. Future work: add a `context/qa/coverage-baseline.md` file that stores the last known layer coverage counts; at the end of Step 9, compare current counts against the baseline and surface regressions explicitly in the Flags section. + +--- + # CONSTRAINTS - Never modify production code. From e44649f20a3ce2deb61ea08ef3a5058cefec8e1e Mon Sep 17 00:00:00 2001 From: Pavel Samoilov Date: Thu, 9 Apr 2026 19:12:19 +0300 Subject: [PATCH 13/56] Delete docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md --- .../plans/2026-04-07-qa-pyramid-agent.md | 574 ------------------ 1 file changed, 574 deletions(-) delete mode 100644 docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md diff --git a/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md b/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md deleted file mode 100644 index ef640ff8..00000000 --- a/docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md +++ /dev/null @@ -1,574 +0,0 @@ -# QA Pyramid Agent Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Extend AWOS with a full test pyramid QA subsystem — a `testing-expert` sub-agent woven into vertical slices, a modified `/awos:tasks` that generates paired test tasks, and an optional `/awos:qa` full-audit command. - -**Architecture:** The `testing-expert` agent operates in two modes: planning mode (called by `/awos:tasks` to generate test task descriptions) and execution mode (called by `/awos:implement` to write and run actual tests). `/awos:qa` is a standalone optional audit command that reads the global `context/qa/` registry, identifies gaps, and manages the regression suite. All changes are additive — no existing AWOS commands are broken. - -**Tech Stack:** Markdown prompt files, no runtime dependencies. Agent definitions follow the AWOS `agent-template.md` convention. Command wrappers follow the existing `claude/commands/` thin-wrapper pattern. - ---- - -## Files - -| Action | File | Responsibility | -|--------|------|----------------| -| CREATE | `commands/qa.md` | `/awos:qa` full-audit command logic | -| CREATE | `claude/commands/qa.md` | Thin wrapper pointing to `commands/qa.md` | -| CREATE | `commands/testing-expert.md` | `testing-expert` agent — planning + execution modes | -| MODIFY | `commands/tasks.md` | Add Step 3b: invoke testing-expert to generate test tasks per slice | -| CREATE | `templates/qa-context-template.md` | Starter template for `context/qa/list-of-tests.md` | -| CREATE | `templates/regression-suite-template.md` | Starter template for `context/qa/regression-suite.md` | - ---- - -## Task 1: Create `testing-expert` agent - -**Files:** -- Create: `commands/testing-expert.md` - -The `testing-expert` agent has two modes. **Planning mode** produces test task descriptions for `tasks.md`. **Execution mode** writes, RED-validates, and runs actual test code. - -- [ ] **Step 1: Write `commands/testing-expert.md`** - -```markdown ---- -description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. ---- - -# ROLE - -You are an expert QA Engineer and Test Automation Specialist. You operate in two distinct modes depending on who calls you and what context you receive. - ---- - -# MODE DETECTION - -Read the invocation context: - -- **Planning Mode** — called by `/awos:tasks` with a functional spec and technical spec but NO existing implementation code. Your job is to return structured test task descriptions for `tasks.md`. You do NOT write test code in this mode. -- **Execution Mode** — called by `/awos:implement` with a specific test task, the implementation code, and full spec context. Your job is to write, RED-validate, and run real test code. - ---- - -# PLANNING MODE - -## Inputs -- `functional-spec.md` from the target spec directory -- `technical-considerations.md` from the target spec directory -- The implementation sub-task description for the current slice - -## Process - -### Step 1: Discover frameworks -1. Read `context/product/architecture.md` for declared testing stack per layer (unit/integration/e2e/contract). -2. If not declared, auto-detect from dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. -3. If still not determinable, note "framework TBD — will auto-detect at execution time" in the task description. - -### Step 2: Map acceptance criteria to test layers -For the given implementation sub-task, identify which acceptance criteria it touches. For each criterion, determine which layers apply: -- **Unit** — pure logic, no external dependencies -- **Integration** — service-to-service or DB interactions -- **E2E** — full user flow through the UI or API surface -- **Contract** — API schema/interface validation (OpenAPI, Pact, etc.) - -Not every slice needs all four layers. Apply judgment. - -### Step 3: Generate test task descriptions -For each applicable layer, generate two sub-tasks: one for positive cases, one for negative cases. - -**Output format** (return this list to `/awos:tasks` for insertion into `tasks.md`): - -``` -- [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** -- [ ] Unit: [describe negative/error inputs and boundary values] — negative cases **[Agent: testing-expert]** -- [ ] Integration: [describe service interaction scenarios] **[Agent: testing-expert]** -- [ ] Contract: [describe schema/interface validations + violation cases] **[Agent: testing-expert]** -- [ ] E2E: [describe full user flow — positive] **[Agent: testing-expert]** -- [ ] E2E: [describe failure/unhappy path flow — negative] **[Agent: testing-expert]** -``` - -Omit layers that genuinely don't apply. Always include negative cases for every layer that is included. - ---- - -# EXECUTION MODE - -## Inputs -- Specific test task description (layer + positive/negative scope) -- `functional-spec.md` + `technical-considerations.md` for the target spec -- `context/product/architecture.md` -- The implementation code written by the preceding impl sub-agent -- Current `context/qa/list-of-tests.md` (if it exists) - -## Process - -### Step 1: Discover frameworks -1. Read `context/product/architecture.md` for declared testing stack per layer. -2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. - -### Step 2: Plan test cases -Map the task's acceptance criteria to concrete test cases: -- For every positive case, define at least one negative counterpart. -- Negative cases must include: invalid inputs, boundary values, error paths, permission failures, malformed data — whichever apply to this layer. - -### Step 3: Write tests with RED validation - -Write tests following this discipline (borrowed from TDD red-green-refactor): - -1. Write one test case. -2. Run it. **Confirm it FAILS** — and that the failure message matches the missing behavior, not a syntax error. - - If it passes immediately: the test is not testing new behavior. Revise it until it fails for the right reason. -3. Proceed to the next test case. - -Annotate every test file with: -``` -# @layer: unit | integration | e2e | contract -# @spec: [spec-directory-name] -# @regression ← add only for tests that should be in the permanent regression suite -``` - -### Step 4: Confirm GREEN -Run all tests written in this task. All must pass before continuing. - -### Step 5: Check for implementation gaps -If tests reveal that the implementation is incomplete: -- Do NOT modify production code. -- Report the gap to `/awos:implement` with a clear description. -- A new impl sub-task will be created; this test task stays open until that sub-task closes. - -### Step 6: Update `context/qa/list-of-tests.md` -Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: -- **Same behavior, same layer** → UPDATE the existing entry instead of adding a new one. -- **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `# @deprecated`. -- **Partial overlap** → keep both, note the relationship in the Notes column. - -Append only net-new tests. Format: - -```markdown -| path/to/test_file.py | test_function_name | unit | negative | yes | OK | -``` - -### Step 7: Mark task [x] -Signal completion to `/awos:implement`. - ---- - -# CONSTRAINTS - -- Never modify production/implementation code — only test files. -- Never skip negative test cases — every included layer must have at least one negative test. -- RED validation is non-negotiable — a test that passes immediately without implementation proves nothing. -- Co-locate test files with source or follow the existing `tests/` directory convention in the project. -``` - -- [ ] **Step 2: Verify the file is well-formed** - -```bash -cat commands/testing-expert.md | head -5 -``` -Expected output: starts with `---` frontmatter. - -- [ ] **Step 2b: Write `claude/commands/testing-expert.md` (thin wrapper)** - -```markdown ---- -description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. ---- - -Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. - -Refer to the instructions located in this file: .awos/commands/testing-expert.md -``` - -- [ ] **Step 3: Commit** - -```bash -git add commands/testing-expert.md claude/commands/testing-expert.md -git commit -m "feat: add testing-expert agent for test pyramid generation" -``` - ---- - -## Task 2: Modify `/awos:tasks` to generate test tasks per slice - -**Files:** -- Modify: `commands/tasks.md` - -We add a new **Step 3b** after each implementation sub-task is defined. It invokes `testing-expert` in planning mode to generate paired test sub-tasks for that slice. - -- [ ] **Step 1: Read the current `commands/tasks.md`** - -```bash -cat commands/tasks.md -``` - -Identify the end of Step 3 ("Your Thought Process for Generating Tasks") — specifically after sub-step 3 where nested sub-tasks are created for a slice. - -- [ ] **Step 2: Add Step 3b — test task generation** - -After the line: -``` - - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table -``` - -Insert the following new sub-step **3b**: - -```markdown - 3b. **Generate paired test tasks for each slice (REQUIRED):** - - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. - - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. - - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. - - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - - Example result for a slice: - ``` - - [ ] **Slice 1: User authentication** - - [ ] Implement JWT token generation **[Agent: python-expert]** - - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** - - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - - [ ] Integration: valid/invalid credentials against /auth endpoint **[Agent: testing-expert]** - - [ ] Contract: /auth response schema validation + violation cases **[Agent: testing-expert]** - ``` -``` - -- [ ] **Step 3: Update the example in Step 3 to reflect test tasks** - -Find the existing "Good, Vertical Slices" example: -```markdown - - `[ ] **Slice 2: Display the user's actual avatar if it exists**` - - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - ... - - `[ ] Sub-task: Run the application. Use chrome MCP... **[Agent: manual-qa-expert]**` -``` - -Append test sub-tasks to Slice 2 in the example: -```markdown - - `[ ] Unit: avatar_url column default, null handling — positive/negative **[Agent: testing-expert]**` - - `[ ] Integration: GET /user returns avatar_url when set, null when not **[Agent: testing-expert]**` - - `[ ] E2E: profile page shows avatar when present, placeholder when null **[Agent: testing-expert]**` -``` - -- [ ] **Step 4: Verify the file reads cleanly** - -```bash -cat commands/tasks.md | grep -A5 "3b" -``` -Expected: shows the new Step 3b block. - -- [ ] **Step 5: Commit** - -```bash -git add commands/tasks.md -git commit -m "feat: extend /awos:tasks to generate test pyramid tasks per vertical slice" -``` - ---- - -## Task 3: Create `/awos:qa` command - -**Files:** -- Create: `commands/qa.md` -- Create: `claude/commands/qa.md` - -- [ ] **Step 1: Write `commands/qa.md`** - -```markdown ---- -description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. ---- - -# ROLE - -You are a senior QA Architect running a full audit of the test suite for one or all specifications. You analyze existing tests, identify coverage gaps, generate missing tests, manage the regression suite, and produce a structured audit report. You do not modify production code. - ---- - -# TASK - -Perform a full QA audit for the target spec(s). Check existing tests for health (missing, stale, deprecated), identify gaps against spec acceptance criteria, generate missing tests, offer to run the suite, and produce an audit report in `context/qa/audit-reports/`. - ---- - -# INPUTS & OUTPUTS - -- **User Prompt (Optional):** $ARGUMENTS - - Empty = audit all specs - - Spec name/index = audit that spec only (e.g., `/awos:qa 001-user-auth`) -- **Primary Context:** - - `context/qa/list-of-tests.md` — global test registry - - `context/qa/regression-suite.md` — regression-flagged tests - - `context/product/architecture.md` — testing framework declarations - - All `functional-spec.md` files in scope - - Live codebase + existing test files -- **Output Files:** - - Updated `context/qa/list-of-tests.md` - - Updated `context/qa/regression-suite.md` - - New `context/qa/audit-reports/qa-report-YYYY-MM-DD.md` - ---- - -# PROCESS - -## Step 1: Identify scope - -1. Read ``. If it names a spec, target that directory only. -2. If empty, target all spec directories under `context/spec/`. -3. Announce: "Running QA audit for: [scope]." - -## Step 2: Discover frameworks - -1. Read `context/product/architecture.md` for declared testing stack per layer. -2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. - -## Step 3: Load test registry - -Read `context/qa/list-of-tests.md`. If it does not exist, create it from the template at `.awos/templates/qa-context-template.md` (or create an empty registry if template is absent). - -## Step 4: Audit existing tests - -For each registered test in scope: - -**Existence check:** -- Does the test file still exist in the codebase? If not → flag `MISSING`. - -**Spec linkage check:** -- Does the `@spec` annotation reference a spec directory that still exists? - - YES → OK. - - Spec directory deleted → - - Search all active `functional-spec.md` files for matching behavior/acceptance criterion. - - Match found → re-link `@spec` to the new spec, mark OK. - - No match → flag `HUMAN REVIEW` (do NOT auto-deprecate; may be intentional regression coverage). - -**Staleness check:** -- Does the test logic match the current implementation? - - Read the test file and the relevant implementation code. - - If the implementation has changed in ways that invalidate the test → flag `NEEDS UPDATE` and generate a diff suggestion. Do NOT auto-modify. - -**Regression tag:** -- If tagged `@regression` → carry forward to the regression-suite.md sync in Step 7. - -## Step 5: Gap analysis - -For each acceptance criterion in the functional spec(s) in scope: - -1. Check which pyramid layers have coverage (unit / integration / e2e / contract). -2. Check whether each covered layer has a negative test counterpart. -3. Record gaps: - - Layer with no tests at all → `MISSING LAYER` - - Layer with only positive tests → `MISSING NEGATIVE` - -## Step 6: Generate missing tests - -For each gap identified in Step 5: - -1. Write the missing test following RED validation discipline (from `testing-expert` execution mode): - - Write test → confirm it FAILS for the right reason → confirm it PASSES. -2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations. -3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: - - Same behavior, same layer → UPDATE existing entry. - - Broader test needing splitting → DEPRECATE old, add focused replacements. - - Partial overlap → keep both, annotate relationship. - -## Step 7: Run tests (with user confirmation) - -1. Count all tests in scope. Notify user: - ``` - Ready to run N tests across X specs [unit: A, integration: B, e2e: C, contract: D] - Regression suite: M tests tagged @regression - ``` -2. Ask user to choose: - - **A) Full suite** — run all N tests - - **B) Regression suite only** — run M tests tagged @regression - - **C) Skip** — do not run tests this session -3. Wait for user confirmation before executing. -4. Run the selected suite. Collect pass/fail results per layer. - -## Step 8: Update regression suite - -Sync `context/qa/regression-suite.md`: -- Scan all test files in scope for `@regression` annotations. -- Add newly tagged tests; remove entries for tests that no longer exist or have lost their `@regression` tag. -- Update "Last updated" date and total count. - -## Step 9: Produce audit report - -Save to `context/qa/audit-reports/qa-report-YYYY-MM-DD.md`: - -```markdown -# QA Audit Report — YYYY-MM-DD - -## Scope -[Spec(s) audited] - -## Coverage Summary -| Spec | Unit | Integration | E2E | Contract | ACs Covered | -|------|------|-------------|-----|----------|-------------| -| [spec] | X/Y | X/Y | X/Y | X/Y | X/Y (Z%) | - -## Flags -- MISSING: [file] — [reason] -- NEEDS UPDATE: [file]::[test] — [what changed] -- MISSING LAYER: [spec] — [AC] has no [layer] coverage -- MISSING NEGATIVE: [spec] — [AC] [layer] has no negative test -- HUMAN REVIEW: [file]::[test] — spec deleted, no active match found - -## Regression Suite Delta -- Added: N | Removed: N | Total: N - -## Run Results -[If tests were run:] -- Suite: [Full / Regression only] -- Passed: N | Failed: N | Blocked: N -- FAILED: [file]::[test] — [brief reason] - -## Recommendation -- [ ] Ready — all critical ACs covered, suite passing -- [ ] Needs attention — [N] gaps or failures require action -``` - -## Step 10: Announce - -Report summary to user and list any flags requiring human attention. - ---- - -# CONSTRAINTS - -- Never modify production code. -- Never auto-deprecate tests whose spec was deleted — always flag for human review. -- Always ask for user confirmation before running tests. -- Generate a diff suggestion for stale tests — do not auto-rewrite them. -``` - -- [ ] **Step 2: Write `claude/commands/qa.md` (thin wrapper)** - -```markdown ---- -description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. ---- - -Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. - -Refer to the instructions located in this file: .awos/commands/qa.md -``` - -- [ ] **Step 3: Verify both files exist** - -```bash -ls commands/qa.md claude/commands/qa.md -``` -Expected: both files listed. - -- [ ] **Step 4: Commit** - -```bash -git add commands/qa.md claude/commands/qa.md -git commit -m "feat: add /awos:qa optional full-audit command" -``` - ---- - -## Task 4: Create QA context templates - -**Files:** -- Create: `templates/qa-context-template.md` -- Create: `templates/regression-suite-template.md` - -These templates are used by `/awos:qa` when initializing a fresh `context/qa/` directory in a project. - -- [ ] **Step 1: Write `templates/qa-context-template.md`** - -```markdown -# Test Registry - -> Auto-maintained by `testing-expert` (execution mode) and `/awos:qa`. -> Each row is a registered test. Add rows by running `/awos:implement` (test tasks) or `/awos:qa`. -> Status values: OK | MISSING | NEEDS UPDATE | DEPRECATED | HUMAN REVIEW -> Layer values: unit | integration | e2e | contract - - - - - -``` - -- [ ] **Step 2: Write `templates/regression-suite-template.md`** - -```markdown -# Regression Suite - -> Auto-maintained by `/awos:qa` Step 8. Synced from `@regression` annotations in test files. -> Run with: `/awos:qa` → choose "Regression suite only". - -**Last updated:** YYYY-MM-DD -**Total:** 0 tests - - - - -``` - -- [ ] **Step 3: Verify both files exist** - -```bash -ls templates/qa-context-template.md templates/regression-suite-template.md -``` -Expected: both files listed. - -- [ ] **Step 4: Commit** - -```bash -git add templates/qa-context-template.md templates/regression-suite-template.md -git commit -m "feat: add QA context templates for test registry and regression suite" -``` - ---- - -## Task 5: Final validation - -- [ ] **Step 1: Confirm all files are present** - -```bash -git diff main --name-only -``` -Expected output: -``` -claude/commands/qa.md -claude/commands/testing-expert.md -commands/qa.md -commands/tasks.md -commands/testing-expert.md -docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md -templates/qa-context-template.md -templates/regression-suite-template.md -``` - -- [ ] **Step 2: Confirm all command wrappers reference the right paths** - -```bash -grep "Refer to" claude/commands/qa.md -``` -Expected: `Refer to the instructions located in this file: .awos/commands/qa.md` - -- [ ] **Step 3: Confirm `testing-expert` is referenced correctly in `tasks.md`** - -```bash -grep "testing-expert" commands/tasks.md | head -5 -``` -Expected: shows Step 3b referencing `testing-expert` in planning mode. - -- [ ] **Step 4: Confirm no placeholder text remains** - -```bash -grep -rn "TBD\|TODO\|PLACEHOLDER\|fill in" commands/qa.md commands/testing-expert.md commands/tasks.md templates/qa-context-template.md templates/regression-suite-template.md -``` -Expected: no matches. - -- [ ] **Step 5: Commit plan doc** - -```bash -git add docs/superpowers/plans/2026-04-07-qa-pyramid-agent.md -git commit -m "docs: add QA pyramid agent implementation plan" -``` From 7921ccc6850a57ac6fd9569c31599352e0ed527c Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 14 Apr 2026 12:38:33 +0300 Subject: [PATCH 14/56] chore: fix prettier formatting across markdown files Co-Authored-By: Claude Sonnet 4.6 --- commands/qa.md | 24 +- commands/tasks.md | 37 +- commands/testing-expert.md | 21 +- .../2026-04-09-quality-assurance-dimension.md | 496 ++++++++++++++++++ ...4-09-quality-assurance-dimension-design.md | 246 +++++++++ 5 files changed, 789 insertions(+), 35 deletions(-) create mode 100644 docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md create mode 100644 docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md diff --git a/commands/qa.md b/commands/qa.md index 2a29d775..c8d419a2 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -54,9 +54,11 @@ Read `context/qa/list-of-tests.md`. If it does not exist, create it from the tem For each registered test in scope: **Existence check:** + - Does the test file still exist in the codebase? If not → flag `MISSING`. **Spec linkage check:** + - Does the `@spec` annotation reference a spec directory that still exists? - YES → OK. - Spec directory deleted → @@ -65,11 +67,13 @@ For each registered test in scope: - No match → flag `HUMAN REVIEW` (do NOT auto-deprecate; may be intentional regression coverage). **Staleness check:** + - Does the test logic match the current implementation? - Read the test file and the relevant implementation code. - If the implementation has changed in ways that invalidate the test → flag `NEEDS UPDATE` and generate a diff suggestion. Do NOT auto-modify the test. **Regression tag:** + - If tagged `@regression` → carry forward to the regression-suite.md sync in Step 8. ## Step 5: Gap analysis @@ -97,9 +101,9 @@ For each gap identified in Step 5: Append net-new entries using this format: ```markdown -| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | -|------|-----------|-------|-------------------|-------------|--------|-------| -| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | +| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | +| -------------------- | ------------------ | ----- | ----------------- | ----------- | ------ | ----- | +| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | ``` ## Step 7: Run tests (with user confirmation) @@ -119,6 +123,7 @@ Append net-new entries using this format: ## Step 8: Update regression suite Sync `context/qa/regression-suite.md`: + - Scan all test files in scope for `@regression` annotations. - Add newly tagged tests; remove entries for tests that no longer exist or have lost their `@regression` tag. - Update "Last updated" date and total count. @@ -131,14 +136,17 @@ Save to `context/qa/audit-reports/qa-report-YYYY-MM-DD.md`: # QA Audit Report — YYYY-MM-DD ## Scope + [Spec(s) audited] ## Coverage Summary -| Spec | Unit | Integration | E2E | Contract | ACs Covered | -|------|------|-------------|-----|----------|-------------| -| [spec] | X/Y | X/Y | X/Y | X/Y | X/Y (Z%) | + +| Spec | Unit | Integration | E2E | Contract | ACs Covered | +| ------ | ---- | ----------- | --- | -------- | ----------- | +| [spec] | X/Y | X/Y | X/Y | X/Y | X/Y (Z%) | ## Flags + - MISSING: [file] — [reason] - NEEDS UPDATE: [file]::[test] — [what changed] - MISSING LAYER: [spec] — [AC] has no [layer] coverage @@ -146,15 +154,19 @@ Save to `context/qa/audit-reports/qa-report-YYYY-MM-DD.md`: - HUMAN REVIEW: [file]::[test] — spec deleted, no active match found ## Regression Suite Delta + - Added: N | Removed: N | Total: N ## Run Results + [If tests were run:] + - Suite: [Full / Regression only] - Passed: N | Failed: N | Blocked: N - FAILED: [file]::[test] — [brief reason] ## Recommendation + - [ ] Ready — all critical ACs covered, suite passing - [ ] Needs attention — [N] gaps or failures require action ``` diff --git a/commands/tasks.md b/commands/tasks.md index b8fa4df1..9ba92fc3 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -58,32 +58,17 @@ Follow this process precisely. 1. First, identify the absolute smallest piece of user-visible value from the spec. This is your **Slice 1**. 2. Create a high-level checklist item for that slice (e.g., `- [ ] **Slice 1: View existing avatar (or placeholder)**`). 3. Under that slice, create the nested sub-tasks (database, backend, frontend) needed to implement and verify **only that slice**. - 4. **For each sub-task, assign the appropriate subagent:** - - Analyze the sub-task description to understand what technology/domain it involves - - Analyze the Task tool definition to extract all available subagent_type values with their descriptions to understand what subagents are available for assignment. - - Match the sub-task to a subagent based on: - - Technology keywords - - Task intent - - Tech stack identified in technical-considerations.md - - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table - 3b. **Generate paired test tasks for each slice (REQUIRED):** - - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. - - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. - - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. - - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - - Example result for a slice: - ``` - - [ ] **Slice 1: User authentication** - - [ ] Implement JWT token generation **[Agent: python-expert]** - - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** - - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** - - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** - - [ ] Contract: /auth response schema validation — positive cases **[Agent: testing-expert]** - - [ ] Contract: schema violations and malformed payload cases — negative cases **[Agent: testing-expert]** - ``` + 4. **For each sub-task, assign the appropriate subagent:** - Analyze the sub-task description to understand what technology/domain it involves - Analyze the Task tool definition to extract all available subagent_type values with their descriptions to understand what subagents are available for assignment. - Match the sub-task to a subagent based on: - Technology keywords - Task intent - Tech stack identified in technical-considerations.md - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table + 3b. **Generate paired test tasks for each slice (REQUIRED):** - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - Example result for a slice: + ` - [ ] **Slice 1: User authentication** + - [ ] Implement JWT token generation **[Agent: python-expert]** + - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** + - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** + - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** + - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** + - [ ] Contract: /auth response schema validation — positive cases **[Agent: testing-expert]** + - [ ] Contract: schema violations and malformed payload cases — negative cases **[Agent: testing-expert]** + ` 5. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 6. Create a high-level checklist item and its sub-tasks with subagent assignments. 7. Repeat this process until all requirements from the specification are covered. diff --git a/commands/testing-expert.md b/commands/testing-expert.md index 1195d0b5..e3f9bd55 100644 --- a/commands/testing-expert.md +++ b/commands/testing-expert.md @@ -21,6 +21,7 @@ Read the invocation context: # PLANNING MODE ## Inputs + - `functional-spec.md` from the target spec directory - `technical-considerations.md` from the target spec directory - The implementation sub-task description for the current slice @@ -28,12 +29,15 @@ Read the invocation context: ## Process ### Step 1: Discover frameworks + 1. Read `context/product/architecture.md` for declared testing stack per layer (unit/integration/e2e/contract). 2. If not declared, auto-detect from dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. 3. If still not determinable, note "framework TBD — will auto-detect at execution time" in the task description. ### Step 2: Map acceptance criteria to test layers + For the given implementation sub-task, identify which acceptance criteria it touches. For each criterion, determine which layers apply: + - **Unit** — pure logic, no external dependencies - **Integration** — service-to-service or DB interactions - **E2E** — full user flow through the UI or API surface @@ -42,6 +46,7 @@ For the given implementation sub-task, identify which acceptance criteria it tou Not every slice needs all four layers. Apply judgment. ### Step 3: Generate test task descriptions + For each applicable layer, generate two sub-tasks: one for positive cases, one for negative cases. **Output format** (return this list to `/awos:tasks` for insertion into `tasks.md`): @@ -64,6 +69,7 @@ Omit layers that genuinely don't apply. Always include negative cases for every # EXECUTION MODE ## Inputs + - Specific test task description (layer + positive/negative scope) - `functional-spec.md` + `technical-considerations.md` for the target spec - `context/product/architecture.md` @@ -73,11 +79,14 @@ Omit layers that genuinely don't apply. Always include negative cases for every ## Process ### Step 1: Discover frameworks + 1. Read `context/product/architecture.md` for declared testing stack per layer. 2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. ### Step 2: Plan test cases + Map the task's acceptance criteria to concrete test cases: + - For every positive case, define at least one negative counterpart. - Negative cases must include: invalid inputs, boundary values, error paths, permission failures, malformed data — whichever apply to this layer. @@ -91,6 +100,7 @@ Write tests following this discipline (borrowed from TDD red-green-refactor): 3. Proceed to the next test case. Annotate every test file with the following (use the appropriate comment syntax for the language: `#` for Python/Ruby/Shell, `//` for JS/TS/Go/Java, `/* */` for C/C++/C#): + ``` # @layer: unit | integration | e2e | contract # @spec: [spec-directory-name] @@ -98,17 +108,22 @@ Annotate every test file with the following (use the appropriate comment syntax ``` ### Step 4: Confirm GREEN + Run all tests written in this task. All must pass before continuing. ### Step 5: Check for implementation gaps + If tests reveal that the implementation is incomplete: + - Do NOT modify production code. - Report the gap by appending a note to this task's entry in `tasks.md`: `` - Do NOT invoke `/awos:implement` directly. Leave this task open (`[ ]`); `/awos:implement` will detect the incomplete task on its next run and create a new impl sub-task to close the gap. ### Step 6: Update `context/qa/list-of-tests.md` + Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: + - **Same behavior, same layer** → UPDATE the existing entry instead of adding a new one. - **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `@deprecated` using the appropriate comment syntax for the language. - **Partial overlap** → keep both, note the relationship in the Notes column. @@ -116,9 +131,9 @@ Before appending new entries, scan the registry for existing tests covering the Append only net-new tests. Format: ```markdown -| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | -|------|-----------|-------|-------------------|-------------|--------|-------| -| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | +| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | +| -------------------- | ------------------ | ----- | ----------------- | ----------- | ------ | ----- | +| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | ``` ### Step 7: Report completion status to `/awos:implement` diff --git a/docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md b/docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md new file mode 100644 index 00000000..ba90ffab --- /dev/null +++ b/docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md @@ -0,0 +1,496 @@ +# Quality Assurance Dimension Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a `quality-assurance` audit dimension that evaluates testing pyramid structure, tooling maturity, and conditional contract/ML testing checks, while moving SBP-04 out of `software-best-practices`. + +**Architecture:** Three markdown edits — remove SBP-04 from `software-best-practices.md` with a lineage note, add `quality-assurance` to the `depends-on` list in `end-to-end-delivery.md`, and create the new `quality-assurance.md` dimension file with 10 checks. No code changes required; the AWOS dimension-auditor agent discovers and executes dimension files automatically. + +**Tech Stack:** Markdown, YAML frontmatter, AWOS dimension format (SKILL.md spec) + +--- + +## File Map + +| Action | File | Responsibility | +| ------ | ------------------------------------------------------------------------------ | --------------------------------------------------------------- | +| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md` | Remove SBP-04 block; add lineage note between SBP-03 and SBP-05 | +| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md` | Add `quality-assurance` to frontmatter `depends-on` list | +| Create | `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` | New dimension with QA-01 through QA-10 | + +--- + +## Task 1: Remove SBP-04 from software-best-practices and add lineage note + +**Files:** + +- Modify: `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md:42-68` + +- [ ] **Step 1: Verify SBP-04 is present** + +```bash +grep -n "SBP-04" plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md +``` + +Expected output: a line showing `### SBP-04: Test infrastructure exists with adequate coverage` + +- [ ] **Step 2: Replace SBP-04 block with lineage note** + +In `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md`, replace the entire SBP-04 block (lines 42–68) with a lineage note. + +Remove this block: + +```markdown +### SBP-04: Test infrastructure exists with adequate coverage + +- **What:** The project has meaningful tests written covering its source modules +- **How:** Detect test infrastructure based on the project's stack: + 1. **Traditional test files** (JS/TS/Kotlin/Python/Go/etc.): Glob for + `**/*.test.{ts,tsx,js,jsx}`, `**/*.spec.{ts,tsx,js,jsx}`, + `**/__tests__/**/*.{ts,tsx,js,jsx}`, `**/*_test.py`, `**/test_*.py`, + `**/*Test.kt`, `**/*Spec.kt`, `**/*_test.go`, etc. + Exclude `node_modules/`, `build/`, `dist/`, `vendor/`, `.venv/`. + Count discrete test files. + 2. **Declarative test frameworks** (dbt, Terraform, Maestro, etc.): Count + individual test _definitions_, not files. A single YAML or config file + may declare many independent test assertions, flows, or checks. Parse + the relevant format and count each discrete test unit. + + Use the metric appropriate to the stack: test _files_ for file-per-test + frameworks, test _definitions_ for declarative frameworks. Calculate the + test-coverage ratio: tested source modules / total source modules. A source + module is "tested" if at least one test (file or definition) targets it. + +- **Pass:** Test-coverage ratio >= 60% (at least 60% of source modules have + associated tests) +- **Warn:** Test-coverage ratio > 0% but < 60% (some modules lack test + coverage) +- **Fail:** No tests found +- **Severity:** critical +``` + +Replace with: + +```markdown +> _Test infrastructure and coverage are evaluated in the **Quality Assurance** dimension (`quality-assurance.md`), which provides a full testing pyramid analysis._ +``` + +- [ ] **Step 3: Verify the change** + +```bash +grep -n "SBP-04\|quality-assurance\|Quality Assurance" plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md +``` + +Expected: no `SBP-04` line, one line referencing `quality-assurance.md` + +- [ ] **Step 4: Commit** + +```bash +git add plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md +git commit -m "feat(audit): move SBP-04 to quality-assurance dimension" +``` + +--- + +## Task 2: Add quality-assurance to end-to-end-delivery depends-on + +**Files:** + +- Modify: `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md:1-16` + +- [ ] **Step 1: Verify current depends-on list** + +```bash +grep -n "depends-on\|quality-assurance" plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md +``` + +Expected: `depends-on:` present, no `quality-assurance` yet + +- [ ] **Step 2: Add quality-assurance to depends-on** + +In `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md`, replace the frontmatter `depends-on` block: + +Old: + +```yaml +depends-on: + [ + project-topology, + documentation, + security, + ai-development-tooling, + spec-driven-development, + code-architecture, + software-best-practices, + ] +``` + +New: + +```yaml +depends-on: + [ + project-topology, + documentation, + security, + ai-development-tooling, + spec-driven-development, + code-architecture, + software-best-practices, + quality-assurance, + ] +``` + +- [ ] **Step 3: Verify** + +```bash +grep -n "quality-assurance" plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md +``` + +Expected: one line with `quality-assurance,` + +- [ ] **Step 4: Commit** + +```bash +git add plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md +git commit -m "feat(audit): add quality-assurance to end-to-end-delivery depends-on" +``` + +--- + +## Task 3: Create quality-assurance.md dimension file + +**Files:** + +- Create: `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` + +- [ ] **Step 1: Create the file with full content** + +Create `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` with this exact content: + +```markdown +--- +name: quality-assurance +title: Quality Assurance +description: Evaluates testing pyramid structure, tooling maturity, and optional contract/ML testing readiness +severity: high +depends-on: [project-topology] +--- + +# Quality Assurance + +Audits the depth and structure of the project's testing approach. Checks whether tests are organized across the full testing pyramid (unit → integration → E2E), whether the pyramid shape is healthy (not inverted), and whether supporting tooling — coverage reporting, test data management, and mocking infrastructure — is in place. Conditional checks for contract testing and ML model iteration testing activate when the topology reveals multi-service or ML architectures. + +## Checks + +### QA-01: Test infrastructure exists with adequate coverage + +- **What:** The project has meaningful tests covering its source modules +- **How:** Detect test infrastructure based on the project's stack. Exclude `node_modules/`, `build/`, `dist/`, `vendor/`, `.venv/` from all globs. + + **Traditional test file globs by stack:** + - JS/TS: `**/*.test.{ts,tsx,js,jsx}`, `**/*.spec.{ts,tsx,js,jsx}`, `**/__tests__/**/*` + - Python: `**/test_*.py`, `**/*_test.py` + - Java: `**/*Test.java`, `**/*Tests.java`, `**/*IT.java` + - Kotlin: `**/*Test.kt`, `**/*Spec.kt`, `**/*IT.kt` + - Go: `**/*_test.go` + - Rust: files in `tests/` directory OR any file containing `#[cfg(test)]` + - Ruby: `**/*_spec.rb`, `**/*_test.rb`, `**/test_*.rb` + - PHP: `**/*Test.php` + - Elixir: `**/*_test.exs` + - Swift: `**/*Tests.swift`, `**/*Test.swift` + - Dart/Flutter: `**/*_test.dart` + - C#: `**/*Tests.cs`, `**/*Test.cs` + + **Declarative frameworks** (dbt, Terraform, Maestro YAML): count individual test definitions, not files. + + Calculate the test-coverage ratio: tested source modules / total source modules. A source module is "tested" if at least one test (file or definition) targets it. + +- **Pass:** Test-coverage ratio >= 60% +- **Warn:** Test-coverage ratio > 0% but < 60% +- **Fail:** No tests found +- **Severity:** critical + +--- + +### QA-02: Unit tier present + +- **What:** The project has tests that verify individual units of logic in isolation, without real I/O or external dependencies +- **How:** Look for any of the following signals: + + **File/directory naming:** + - Files: `*.unit.test.*`, `*.unit.spec.*` + - Directories: `tests/unit/`, `__tests__/unit/`, `spec/unit/` + + **Annotation/marker signals by stack:** + - Java/Kotlin (JUnit 5): `@Tag("unit")` or `@Tag("fast")` in test files + - Python (pytest): `@pytest.mark.unit` in test files, or `unit` marker registered in `pytest.ini` / `pyproject.toml` + - Ruby (RSpec): `:unit` tag on `describe` or `context` blocks + - PHP (PHPUnit): `@group unit` docblock annotation + - Go: `*_test.go` files that do NOT contain `//go:build integration` or `//go:build e2e` build constraints (absence of those tags = unit) + - Rust: `#[cfg(test)]` module co-located inside `src/` files (in-source placement = unit test pattern) + - Swift (XCTest): test target whose name ends in `UnitTests` or `Tests` (but not `UITests`) + - Elixir (ExUnit): `use ExUnit.Case` without `Phoenix.ConnTest`, `Ecto.Adapters.SQL.Sandbox`, `DataCase`, or `ConnCase` imports + - Dart/Flutter: `**/*_test.dart` located in `test/` directory (not `integration_test/`) + + **Import-based inference:** sample 5 test files — if they import only local project modules and no HTTP clients, DB clients, or external service SDKs, treat them as unit-scoped. + +- **Pass:** Unit tests detected via any signal above +- **Warn:** Test files exist but none can be clearly identified as unit-scoped +- **Fail:** No unit test signals detected +- **Severity:** high + +--- + +### QA-03: Integration tier present + +- **What:** The project has tests that verify interactions between components — across real databases, real HTTP calls, or real message queues +- **How:** Look for any of the following signals: + + **File/directory naming:** + - Files: `*.integration.test.*`, `*.integration.spec.*`, `*.int.test.*`, `*_integration_test.go`, `*IT.java`, `*IT.kt` + - Directories: `tests/integration/`, `__tests__/integration/`, `spec/integration/` + + **Annotation/framework signals by stack:** + - Spring Boot (Java/Kotlin): `@SpringBootTest`, `@DataJpaTest`, `@WebMvcTest`, `@DataMongoTest`, `@RestClientTest`, `@ServiceConnection` + - Testcontainers (any JVM): `@Testcontainers`, `@Container`, or imports from `org.testcontainers` + - Python (pytest): `@pytest.mark.integration`, or `integration` marker registered in `pytest.ini` / `pyproject.toml`, or `conftest.py` with DB/HTTP fixtures + - Ruby (RSpec): `type: :integration` metadata, or tests importing `DatabaseCleaner` or `FactoryBot` against a real database + - PHP (PHPUnit): `@group integration` annotation, or extending `AbstractIntegrationTestCase` + - Go: `//go:build integration` or legacy `// +build integration` build constraint in test files + - Rust: files inside the top-level `tests/` directory (Rust canonical location for integration tests, separate from `src/`) + - Elixir: test files that import `Phoenix.ConnTest`, `Ecto.Adapters.SQL.Sandbox`, or use `DataCase` / `ConnCase` from Phoenix + +- **Pass:** Integration tests detected via any signal above +- **Warn:** Tests appear to hit real dependencies (DB/HTTP imports in test files) but no explicit integration markers found +- **Fail:** No integration test signals detected +- **Severity:** high + +--- + +### QA-04: E2E tier present + +- **What:** The project has end-to-end tests that exercise complete user flows through a real UI, API surface, or CLI +- **How:** Look for any of the following signals: + + **Config file signals:** + - `playwright.config.{ts,js,mts,mjs}` — Playwright + - `cypress.config.{ts,js}`, `cypress.json` — Cypress + - `wdio.config.{ts,js}` — WebdriverIO + - `.detoxrc.{js,json}`, `detox.config.js` — Detox (React Native) + - `appium.config.{js,ts}`, `.appiumrc` — Appium + - `nightwatch.conf.js`, `nightwatch.config.js` — Nightwatch + - `codecept.conf.{js,ts}` — CodeceptJS + - `testcafe.config.js` or `testcafe` key in `package.json` — TestCafe + + **Directory/file signals:** + - Directories: `cypress/`, `e2e/`, `tests/e2e/`, `__tests__/e2e/` + - `maestro/` or `.maestro/` containing `*.yaml` flow files (Maestro mobile) + - `integration_test/**/*_test.dart` (Flutter E2E) + - `**/*UITests.swift`, `**/*UITest.swift` (Swift/Xcode UI tests) + - `androidTest/**/*.{java,kt}` (Android Espresso) + - Files: `**/*.e2e.test.*`, `**/*.e2e.spec.*`, `**/*.e2e-spec.*` + + **Annotation signals:** + - Python: `@pytest.mark.e2e` + - Java/Kotlin (JUnit 5): `@Tag("e2e")` + - PHP (PHPUnit): `@group e2e` + - Go: `//go:build e2e` + +- **Pass:** E2E tooling config AND test files both present +- **Warn:** E2E config present but no test files found (tooling set up but unused) +- **Fail:** No E2E signals detected +- **Skip-When:** Topology shows library (no runnable entry point) +- **Severity:** high + +--- + +### QA-05: Pyramid shape — no inversion + +- **What:** The distribution of tests across tiers follows a healthy pyramid: most tests are unit-level, fewer are integration-level, fewest are E2E +- **How:** Use findings from QA-02, QA-03, and QA-04 to estimate test counts at each tier. Count test files (or test definitions for declarative stacks) matched by each tier's signals. A healthy pyramid satisfies: unit_count >= integration_count >= e2e_count. +- **Pass:** unit_count >= integration_count >= e2e_count, or only one tier is present +- **Warn:** E2E count exceeds unit count but integration layer exists as a buffer between them +- **Fail:** E2E count > unit count (inverted pyramid), or integration count > unit count by a significant margin (2× or more) +- **Skip-When:** Fewer than 2 tiers were detected in QA-02/03/04 +- **Severity:** medium + +--- + +### QA-06: Coverage reporting configured + +- **What:** The project measures what percentage of source code is exercised by tests, optionally enforcing a minimum threshold +- **How:** Check for coverage tool configuration: + - Jest: `collectCoverage: true` or `coverageThreshold` key in `jest.config.*` + - Vitest: `coverage` section in `vitest.config.*` + - pytest-cov: `pytest-cov` in `requirements*.txt` or `pyproject.toml`; `--cov` flag in `pytest.ini` or `pyproject.toml` `addopts` + - JaCoCo: `jacoco` plugin in `build.gradle` or `build.gradle.kts` + - nyc / c8: `.nycrc`, `.nycrc.json`, or `c8` / `nyc` script in `package.json` +- **Pass:** Coverage tool configured with thresholds defined +- **Warn:** Coverage tool present but no thresholds defined, OR no coverage tooling found +- **Severity:** low + +--- + +### QA-07: Test data management + +- **What:** Tests use a structured approach to create and manage test data, rather than scattering hardcoded inline values across test files +- **How:** Check for: + - Fixture directories: `fixtures/`, `__fixtures__/`, `testdata/`, `test/fixtures/`, `spec/fixtures/` + - Factory libraries in dependencies: `factory-girl`, `fishery`, `rosie` (JS/TS); `factory_boy` (Python); `FactoryBot` / `factory_bot` (Ruby); `gomock` (Go) + - Faker libraries: `faker`, `@faker-js/faker`, `Faker` (PHP), `Bogus` (C#) + - Seed scripts: `seeds/`, `db/seeds/`, `prisma/seed.*`, `scripts/seed.*` + - Confirm usage: grep for the detected library/directory name inside test files to verify it is actually used (not just installed) +- **Pass:** Fixtures or factories present and referenced in test files +- **Warn:** One approach present but sparse (library installed but used in fewer than 3 test files) +- **Fail:** No test data strategy — tests use only hardcoded inline values +- **Severity:** low + +--- + +### QA-08: Test isolation — mocking infrastructure + +- **What:** Unit and integration tests use mocking/stubbing to isolate the code under test from external dependencies +- **How:** Check for mocking libraries in dependencies or imports: + - JS/TS: `jest.mock`, `vi.mock`, `sinon`, `nock`, `msw` (mock service worker) + - Python: `unittest.mock`, `pytest-mock`, `responses`, `httpretty` + - Java/Kotlin: `mockito-core`, `mockk`, `WireMock` + - Go: `gomock`, `testify/mock` + - Ruby: `rspec-mocks` (included in RSpec), `webmock`, `vcr` + - PHP: PHPUnit built-in mock builder (`createMock`, `getMockBuilder`) + - Rust: `mockall`, `mockito` + - Elixir: `Mox` + + Sample 5 test files and confirm mocking is actively used (grep for `mock`, `stub`, `spy`, `fake`, or library-specific calls). + +- **Pass:** Mocking library present and actively used in sampled test files +- **Warn:** Library present but used in fewer than 2 of the 5 sampled files +- **Fail:** No mocking infrastructure detected +- **Severity:** medium + +--- + +### QA-09: Contract testing + +- **What:** Service boundaries are verified through consumer-driven contract tests, ensuring producers don't break consumers +- **How:** Check for contract testing frameworks: + - Pact: `pact/` directory, `**/*.pact.json`, `@pact-foundation/pact` in dependencies, `au.com.dius.pact` in Gradle + - Spring Cloud Contract: `contracts/` directory with `*.groovy` or `*.yml` contract files + - Schemathesis / Dredd: `schemathesis` or `dredd` in dependencies with config files + - Karate: `**/*.feature` files with contract-style API tests +- **Pass:** Contract tests present covering at least one service boundary +- **Warn:** Contract tooling installed but no contract files found +- **Fail:** No contract testing detected +- **Skip-When:** Topology shows single-service repo or no inter-service communication patterns detected (TOPO-06 found no communication layer) +- **Severity:** high + +--- + +### QA-10: ML model iteration testing + +- **What:** ML models are tested for quality metrics as part of the development cycle, not just functional correctness +- **How:** Check for ML evaluation frameworks or patterns in test files: + - `great_expectations` config (`great_expectations.yml` or `great_expectations/` directory) + - `deepchecks` in dependencies + - `evidently` in dependencies + - `whylogs` in dependencies + - Test files that import ML frameworks (`sklearn`, `torch`, `tensorflow`, `xgboost`, `transformers`, `keras`) AND contain assertions on model quality metrics (accuracy, f1, precision, recall, loss thresholds) — grep for `assert.*score`, `assert.*accuracy`, `assert.*f1`, or similar patterns +- **Pass:** Model evaluation tests present with explicit quality metric assertions +- **Warn:** ML framework imports appear in test files but no metric assertions found (tests exist but don't gate on model quality) +- **Fail:** No ML model testing detected +- **Skip-When:** Topology shows no ML layer — no ML framework imports (`sklearn`, `torch`, `tensorflow`, `xgboost`, `transformers`) found in source files +- **Severity:** high +``` + +- [ ] **Step 2: Verify file exists and frontmatter is valid** + +```bash +head -8 plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md +``` + +Expected output: + +``` +--- +name: quality-assurance +title: Quality Assurance +description: Evaluates testing pyramid structure, tooling maturity, and optional contract/ML testing readiness +severity: high +depends-on: [project-topology] +--- +``` + +- [ ] **Step 3: Verify all 10 checks are present** + +```bash +grep "^### QA-" plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md +``` + +Expected output: + +``` +### QA-01: Test infrastructure exists with adequate coverage +### QA-02: Unit tier present +### QA-03: Integration tier present +### QA-04: E2E tier present +### QA-05: Pyramid shape — no inversion +### QA-06: Coverage reporting configured +### QA-07: Test data management +### QA-08: Test isolation — mocking infrastructure +### QA-09: Contract testing +### QA-10: ML model iteration testing +``` + +- [ ] **Step 4: Verify dimension is discoverable by the orchestrator** + +```bash +ls plugins/awos/skills/ai-readiness-audit/dimensions/ +``` + +Expected: `quality-assurance.md` appears in the list alongside the other 8 dimension files. + +- [ ] **Step 5: Commit** + +```bash +git add plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md +git commit -m "feat(audit): add quality-assurance dimension with testing pyramid checks" +``` + +--- + +## Task 4: Verify full DAG consistency + +- [ ] **Step 1: Check all depends-on references resolve** + +Every name in any `depends-on` list must match a `name:` field in another dimension file. Run: + +```bash +grep "^name:" plugins/awos/skills/ai-readiness-audit/dimensions/*.md +``` + +Expected names present: `project-topology`, `documentation`, `security`, `ai-development-tooling`, `spec-driven-development`, `code-architecture`, `software-best-practices`, `quality-assurance`, `end-to-end-delivery` + +- [ ] **Step 2: Confirm end-to-end-delivery depends-on list is complete** + +```bash +grep -A 12 "depends-on" plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md +``` + +Expected: both `software-best-practices` and `quality-assurance` are in the list. + +- [ ] **Step 3: Confirm SBP-04 no longer exists in software-best-practices** + +```bash +grep "SBP-04" plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md +``` + +Expected: no output (empty). + +- [ ] **Step 4: Final commit** + +```bash +git add plugins/awos/skills/ai-readiness-audit/dimensions/ +git commit -m "feat(audit): verify quality-assurance DAG integration" +``` + +Only run this step if there are uncommitted changes. If Tasks 1–3 each committed cleanly, skip this step. diff --git a/docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md b/docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md new file mode 100644 index 00000000..8b262fc2 --- /dev/null +++ b/docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md @@ -0,0 +1,246 @@ +# Quality Assurance Dimension — Design Spec + +**Date:** 2026-04-09 +**Status:** Approved for implementation + +--- + +## Summary + +Add a new `quality-assurance` dimension to the AWOS AI Readiness Audit plugin. This dimension provides deep evaluation of a project's testing maturity by analyzing testing pyramid structure, tooling, and optional conditional checks for contract and ML testing. The existing SBP-04 check is removed from `software-best-practices` and a lineage note is added there pointing to this new dimension. + +--- + +## DAG Placement + +- **New dimension:** `quality-assurance` +- **depends-on:** `[project-topology]` — Phase 2, same as other substantive dimensions +- **Impact on existing DAG:** `end-to-end-delivery` currently depends on `software-best-practices`; it must also depend on `quality-assurance` since QA-01 replaces SBP-04 +- **Frontmatter severity:** `high` + +--- + +## Changes to `software-best-practices.md` + +- Remove SBP-04 entirely +- Add after SBP-03 (before SBP-05): + + > _Test infrastructure and coverage are evaluated in the **Quality Assurance** dimension (`quality-assurance.md`), which provides a full testing pyramid analysis._ + +- No renumbering — the gap between SBP-03 and SBP-05 is bridged by the note + +--- + +## Checks + +### QA-01: Test infrastructure exists + +Full port of SBP-04. Detects test files or declarative test definitions across all supported stacks, excluding `node_modules/`, `build/`, `dist/`, `vendor/`, `.venv/`. + +**Traditional test file globs:** + +| Stack | Patterns | +| ------------ | ----------------------------------------------------------------------------- | +| JS/TS | `**/*.test.{ts,tsx,js,jsx}`, `**/*.spec.{ts,tsx,js,jsx}`, `**/__tests__/**/*` | +| Python | `**/test_*.py`, `**/*_test.py` | +| Java | `**/*Test.java`, `**/*Tests.java`, `**/*IT.java` | +| Kotlin | `**/*Test.kt`, `**/*Spec.kt`, `**/*IT.kt` | +| Go | `**/*_test.go` | +| Rust | `tests/` directory OR files containing `#[cfg(test)]` module | +| Ruby | `**/*_spec.rb`, `**/*_test.rb`, `**/test_*.rb` | +| PHP | `**/*Test.php` | +| Elixir | `**/*_test.exs` | +| Swift | `**/*Tests.swift`, `**/*Test.swift` | +| Dart/Flutter | `**/*_test.dart` | +| C# | `**/*Tests.cs`, `**/*Test.cs` | + +**Declarative frameworks** (dbt, Terraform, Maestro YAML): count individual test definitions, not files. + +Calculate test-coverage ratio: tested source modules / total source modules. + +- **Pass:** ratio ≥ 60% +- **Warn:** ratio > 0% but < 60% +- **Fail:** no tests found +- **Severity:** critical + +--- + +### QA-02: Unit tier present + +**File/directory signals:** + +- Naming: `*.unit.test.*`, `*.unit.spec.*` +- Directories: `tests/unit/`, `__tests__/unit/`, `spec/unit/` + +**Annotation/marker signals:** + +| Stack | Markers | +| --------------------- | ---------------------------------------------------------------------- | +| Java/Kotlin (JUnit 5) | `@Tag("unit")`, `@Tag("fast")` | +| Python (pytest) | `@pytest.mark.unit`, `unit` marker in `pytest.ini` or `pyproject.toml` | +| Ruby (RSpec) | `:unit` tag on `describe`/`context` blocks | +| PHP (PHPUnit) | `@group unit` docblock annotation | +| Go | `*_test.go` with no `//go:build integration` or `//go:build e2e` tag | +| Rust | `#[cfg(test)]` module co-located inside `src/` files | +| Swift (XCTest) | Test target name ending in `UnitTests` or `Tests` (not `UITests`) | +| Elixir (ExUnit) | `use ExUnit.Case` without Phoenix/Ecto integration imports | +| Dart/Flutter | `**/*_test.dart` in `test/` directory (not `integration_test/`) | + +**Import-based inference:** test file imports only local modules with no HTTP clients, DB clients, or external service SDKs. + +- **Pass:** unit tests detected via any signal above +- **Warn:** test files exist but none are clearly unit-scoped +- **Fail:** no unit test signals detected +- **Severity:** high + +--- + +### QA-03: Integration tier present + +**File/directory signals:** + +- Naming: `*.integration.test.*`, `*.integration.spec.*`, `*.int.test.*`, `*_integration_test.go`, `*IT.java`, `*IT.kt` +- Directories: `tests/integration/`, `__tests__/integration/`, `spec/integration/` + +**Annotation/framework signals:** + +| Stack | Markers | +| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| Spring Boot (Java/Kotlin) | `@SpringBootTest`, `@DataJpaTest`, `@WebMvcTest`, `@DataMongoTest`, `@RestClientTest`, `@ServiceConnection` | +| Testcontainers (any JVM) | `@Testcontainers`, `@Container`, imports from `org.testcontainers` | +| Python (pytest) | `@pytest.mark.integration`, `conftest.py` with DB/HTTP fixtures, `integration` marker in `pytest.ini`/`pyproject.toml` | +| Ruby (RSpec) | `type: :integration` metadata, or tests using `DatabaseCleaner`/`FactoryBot` with real DB | +| PHP (PHPUnit) | `@group integration`, extending `AbstractIntegrationTestCase` | +| Go | `//go:build integration` or legacy `// +build integration` tag | +| Rust | Files in top-level `tests/` directory (Rust canonical integration test location) | +| Elixir | Test files using `Phoenix.ConnTest`, `Ecto.Adapters.SQL.Sandbox`, `DataCase`, or `ConnCase` | + +- **Pass:** integration tests detected via any signal above +- **Warn:** tests appear to hit real dependencies but no explicit integration markers found +- **Fail:** no integration test signals detected +- **Severity:** high + +--- + +### QA-04: E2E tier present + +**Config file signals:** + +| Tool | Config Files | +| -------------------- | -------------------------------------------------------- | +| Playwright | `playwright.config.{ts,js,mts,mjs}` | +| Cypress | `cypress.config.{ts,js}`, `cypress.json` | +| WebdriverIO | `wdio.config.{ts,js}` | +| Detox (React Native) | `.detoxrc.{js,json}`, `detox.config.js` | +| Appium | `appium.config.{js,ts}`, `.appiumrc` | +| Nightwatch | `nightwatch.conf.js`, `nightwatch.config.js` | +| CodeceptJS | `codecept.conf.{js,ts}` | +| TestCafe | `testcafe.config.js` or `testcafe` key in `package.json` | + +**Directory/file signals:** + +- `cypress/`, `e2e/`, `tests/e2e/`, `__tests__/e2e/` +- `maestro/` or `.maestro/` with `*.yaml` flow files (Maestro mobile) +- `integration_test/**/*_test.dart` (Flutter E2E) +- `**/*UITests.swift`, `**/*UITest.swift` (Swift/Xcode UI tests) +- `androidTest/**/*.{java,kt}` (Android Espresso) +- `**/*.e2e.test.*`, `**/*.e2e.spec.*`, `**/*.e2e-spec.*` + +**Annotation signals:** + +| Stack | Markers | +| --------------------- | ------------------ | +| Python | `@pytest.mark.e2e` | +| Java/Kotlin (JUnit 5) | `@Tag("e2e")` | +| PHP (PHPUnit) | `@group e2e` | +| Go | `//go:build e2e` | + +- **Pass:** E2E tooling config and test files both present +- **Warn:** E2E config present but no test files found (tooling set up but unused) +- **Fail:** no E2E signals detected +- **Skip-When:** topology shows library (no runnable entry point) +- **Severity:** high + +--- + +### QA-05: Pyramid shape — no inversion + +Uses findings from QA-02, QA-03, QA-04 to count tests at each tier. A healthy pyramid: unit ≥ integration ≥ E2E. + +- **Pass:** unit count ≥ integration count ≥ E2E count, or only one tier exists +- **Warn:** E2E count exceeds unit count but integration layer exists as a buffer +- **Fail:** E2E count > unit count (inverted pyramid), or integration count > unit count significantly +- **Skip-When:** fewer than 2 tiers detected +- **Severity:** medium + +--- + +### QA-06: Coverage reporting configured _(optional)_ + +Check for coverage tool configuration: `jest` with `coverageThreshold` or `collectCoverage`, `vitest` coverage config, `pytest-cov` in requirements/pyproject, `jacoco` plugin in Gradle, `.nycrc`, `c8` in scripts. + +- **Pass:** coverage tool configured with thresholds defined +- **Warn:** coverage tool present but no thresholds, or no coverage tooling found +- **Severity:** low _(no FAIL state — this check is advisory only)_ + +--- + +### QA-07: Test data management + +Check for fixture directories (`fixtures/`, `__fixtures__/`, `testdata/`), factory libraries (`factory-girl`, `fishery`, `faker`, `factory_boy`, `gomock`), or seed scripts (`seeds/`, `db/seeds/`). + +- **Pass:** fixtures or factories present and referenced in test files +- **Warn:** one approach present but sparse usage +- **Fail:** no test data strategy — tests appear to use only hardcoded inline values +- **Severity:** low + +--- + +### QA-08: Test isolation — mocking infrastructure + +Check for mocking libraries: `jest.mock`, `vi.mock`, `sinon`, `nock`, `msw`, `mockito`, `gomock`, `unittest.mock`. Sample 5 test files to confirm mocks are used to isolate external dependencies in unit/integration tests. + +- **Pass:** mocking library present and actively used +- **Warn:** library present but minimal usage found in sampled files +- **Fail:** no mocking infrastructure detected +- **Severity:** medium + +--- + +### QA-09: Contract testing _(conditional)_ + +Check for consumer-driven contract frameworks: Pact files (`pact/`, `**/*.pact.json`), Spring Cloud Contract (`contracts/`), or similar. + +- **Pass:** contract tests present covering at least one service boundary +- **Warn:** contract tooling configured but no contract files found +- **Fail:** no contract tests +- **Skip-When:** topology shows single-service repo or no inter-service communication detected +- **Severity:** high + +--- + +### QA-10: ML model iteration testing _(conditional)_ + +Check for ML testing patterns: `pytest` with model evaluation assertions, `great_expectations` config, `deepchecks`, `evidently`, `whylogs`, or test files that import ML frameworks (`sklearn`, `torch`, `tensorflow`, `xgboost`, `transformers`) and assert on model quality metrics. + +- **Pass:** model evaluation tests present with quality metric assertions +- **Warn:** ML framework imports appear in test context but no metric assertions found +- **Fail:** no ML model testing +- **Skip-When:** topology shows no ML layer (no ML framework imports in source files) +- **Severity:** high + +--- + +## Scoring + +Standard AWOS scoring applies (from `scoring.md`): each check contributes to the dimension percentage. Skipped checks are excluded from the denominator. QA-06 has no FAIL state — its worst outcome is WARN, making it a low-weight advisory signal. + +--- + +## Files to Create / Modify + +| Action | File | +| ------ | -------------------------------------------------------------------------------------------------------------------- | +| Create | `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` | +| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md` — remove SBP-04, add lineage note | +| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md` — add `quality-assurance` to `depends-on` | From cc4ffb763b432f9dafb105d6c2709304e34897ed Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 14 Apr 2026 12:50:50 +0300 Subject: [PATCH 15/56] fix(tasks): sequential numbering, opt-out tests, ignore docs/superpowers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Renumber thought process steps sequentially (3b → 5, shift 5-8 → 6-9) - Make test generation opt-out instead of REQUIRED - Add docs/superpowers/ to .gitignore and untrack pushed files Co-Authored-By: Claude Sonnet 4.6 --- .gitignore | 1 + .../2026-04-09-quality-assurance-dimension.md | 496 ------------------ ...4-09-quality-assurance-dimension-design.md | 246 --------- 3 files changed, 1 insertion(+), 742 deletions(-) delete mode 100644 docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md delete mode 100644 docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md diff --git a/.gitignore b/.gitignore index c727bb73..f76b9c3e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ node_modules .idea .DS_Store +docs/superpowers/ diff --git a/docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md b/docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md deleted file mode 100644 index ba90ffab..00000000 --- a/docs/superpowers/plans/2026-04-09-quality-assurance-dimension.md +++ /dev/null @@ -1,496 +0,0 @@ -# Quality Assurance Dimension Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a `quality-assurance` audit dimension that evaluates testing pyramid structure, tooling maturity, and conditional contract/ML testing checks, while moving SBP-04 out of `software-best-practices`. - -**Architecture:** Three markdown edits — remove SBP-04 from `software-best-practices.md` with a lineage note, add `quality-assurance` to the `depends-on` list in `end-to-end-delivery.md`, and create the new `quality-assurance.md` dimension file with 10 checks. No code changes required; the AWOS dimension-auditor agent discovers and executes dimension files automatically. - -**Tech Stack:** Markdown, YAML frontmatter, AWOS dimension format (SKILL.md spec) - ---- - -## File Map - -| Action | File | Responsibility | -| ------ | ------------------------------------------------------------------------------ | --------------------------------------------------------------- | -| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md` | Remove SBP-04 block; add lineage note between SBP-03 and SBP-05 | -| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md` | Add `quality-assurance` to frontmatter `depends-on` list | -| Create | `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` | New dimension with QA-01 through QA-10 | - ---- - -## Task 1: Remove SBP-04 from software-best-practices and add lineage note - -**Files:** - -- Modify: `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md:42-68` - -- [ ] **Step 1: Verify SBP-04 is present** - -```bash -grep -n "SBP-04" plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md -``` - -Expected output: a line showing `### SBP-04: Test infrastructure exists with adequate coverage` - -- [ ] **Step 2: Replace SBP-04 block with lineage note** - -In `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md`, replace the entire SBP-04 block (lines 42–68) with a lineage note. - -Remove this block: - -```markdown -### SBP-04: Test infrastructure exists with adequate coverage - -- **What:** The project has meaningful tests written covering its source modules -- **How:** Detect test infrastructure based on the project's stack: - 1. **Traditional test files** (JS/TS/Kotlin/Python/Go/etc.): Glob for - `**/*.test.{ts,tsx,js,jsx}`, `**/*.spec.{ts,tsx,js,jsx}`, - `**/__tests__/**/*.{ts,tsx,js,jsx}`, `**/*_test.py`, `**/test_*.py`, - `**/*Test.kt`, `**/*Spec.kt`, `**/*_test.go`, etc. - Exclude `node_modules/`, `build/`, `dist/`, `vendor/`, `.venv/`. - Count discrete test files. - 2. **Declarative test frameworks** (dbt, Terraform, Maestro, etc.): Count - individual test _definitions_, not files. A single YAML or config file - may declare many independent test assertions, flows, or checks. Parse - the relevant format and count each discrete test unit. - - Use the metric appropriate to the stack: test _files_ for file-per-test - frameworks, test _definitions_ for declarative frameworks. Calculate the - test-coverage ratio: tested source modules / total source modules. A source - module is "tested" if at least one test (file or definition) targets it. - -- **Pass:** Test-coverage ratio >= 60% (at least 60% of source modules have - associated tests) -- **Warn:** Test-coverage ratio > 0% but < 60% (some modules lack test - coverage) -- **Fail:** No tests found -- **Severity:** critical -``` - -Replace with: - -```markdown -> _Test infrastructure and coverage are evaluated in the **Quality Assurance** dimension (`quality-assurance.md`), which provides a full testing pyramid analysis._ -``` - -- [ ] **Step 3: Verify the change** - -```bash -grep -n "SBP-04\|quality-assurance\|Quality Assurance" plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md -``` - -Expected: no `SBP-04` line, one line referencing `quality-assurance.md` - -- [ ] **Step 4: Commit** - -```bash -git add plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md -git commit -m "feat(audit): move SBP-04 to quality-assurance dimension" -``` - ---- - -## Task 2: Add quality-assurance to end-to-end-delivery depends-on - -**Files:** - -- Modify: `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md:1-16` - -- [ ] **Step 1: Verify current depends-on list** - -```bash -grep -n "depends-on\|quality-assurance" plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md -``` - -Expected: `depends-on:` present, no `quality-assurance` yet - -- [ ] **Step 2: Add quality-assurance to depends-on** - -In `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md`, replace the frontmatter `depends-on` block: - -Old: - -```yaml -depends-on: - [ - project-topology, - documentation, - security, - ai-development-tooling, - spec-driven-development, - code-architecture, - software-best-practices, - ] -``` - -New: - -```yaml -depends-on: - [ - project-topology, - documentation, - security, - ai-development-tooling, - spec-driven-development, - code-architecture, - software-best-practices, - quality-assurance, - ] -``` - -- [ ] **Step 3: Verify** - -```bash -grep -n "quality-assurance" plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md -``` - -Expected: one line with `quality-assurance,` - -- [ ] **Step 4: Commit** - -```bash -git add plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md -git commit -m "feat(audit): add quality-assurance to end-to-end-delivery depends-on" -``` - ---- - -## Task 3: Create quality-assurance.md dimension file - -**Files:** - -- Create: `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` - -- [ ] **Step 1: Create the file with full content** - -Create `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` with this exact content: - -```markdown ---- -name: quality-assurance -title: Quality Assurance -description: Evaluates testing pyramid structure, tooling maturity, and optional contract/ML testing readiness -severity: high -depends-on: [project-topology] ---- - -# Quality Assurance - -Audits the depth and structure of the project's testing approach. Checks whether tests are organized across the full testing pyramid (unit → integration → E2E), whether the pyramid shape is healthy (not inverted), and whether supporting tooling — coverage reporting, test data management, and mocking infrastructure — is in place. Conditional checks for contract testing and ML model iteration testing activate when the topology reveals multi-service or ML architectures. - -## Checks - -### QA-01: Test infrastructure exists with adequate coverage - -- **What:** The project has meaningful tests covering its source modules -- **How:** Detect test infrastructure based on the project's stack. Exclude `node_modules/`, `build/`, `dist/`, `vendor/`, `.venv/` from all globs. - - **Traditional test file globs by stack:** - - JS/TS: `**/*.test.{ts,tsx,js,jsx}`, `**/*.spec.{ts,tsx,js,jsx}`, `**/__tests__/**/*` - - Python: `**/test_*.py`, `**/*_test.py` - - Java: `**/*Test.java`, `**/*Tests.java`, `**/*IT.java` - - Kotlin: `**/*Test.kt`, `**/*Spec.kt`, `**/*IT.kt` - - Go: `**/*_test.go` - - Rust: files in `tests/` directory OR any file containing `#[cfg(test)]` - - Ruby: `**/*_spec.rb`, `**/*_test.rb`, `**/test_*.rb` - - PHP: `**/*Test.php` - - Elixir: `**/*_test.exs` - - Swift: `**/*Tests.swift`, `**/*Test.swift` - - Dart/Flutter: `**/*_test.dart` - - C#: `**/*Tests.cs`, `**/*Test.cs` - - **Declarative frameworks** (dbt, Terraform, Maestro YAML): count individual test definitions, not files. - - Calculate the test-coverage ratio: tested source modules / total source modules. A source module is "tested" if at least one test (file or definition) targets it. - -- **Pass:** Test-coverage ratio >= 60% -- **Warn:** Test-coverage ratio > 0% but < 60% -- **Fail:** No tests found -- **Severity:** critical - ---- - -### QA-02: Unit tier present - -- **What:** The project has tests that verify individual units of logic in isolation, without real I/O or external dependencies -- **How:** Look for any of the following signals: - - **File/directory naming:** - - Files: `*.unit.test.*`, `*.unit.spec.*` - - Directories: `tests/unit/`, `__tests__/unit/`, `spec/unit/` - - **Annotation/marker signals by stack:** - - Java/Kotlin (JUnit 5): `@Tag("unit")` or `@Tag("fast")` in test files - - Python (pytest): `@pytest.mark.unit` in test files, or `unit` marker registered in `pytest.ini` / `pyproject.toml` - - Ruby (RSpec): `:unit` tag on `describe` or `context` blocks - - PHP (PHPUnit): `@group unit` docblock annotation - - Go: `*_test.go` files that do NOT contain `//go:build integration` or `//go:build e2e` build constraints (absence of those tags = unit) - - Rust: `#[cfg(test)]` module co-located inside `src/` files (in-source placement = unit test pattern) - - Swift (XCTest): test target whose name ends in `UnitTests` or `Tests` (but not `UITests`) - - Elixir (ExUnit): `use ExUnit.Case` without `Phoenix.ConnTest`, `Ecto.Adapters.SQL.Sandbox`, `DataCase`, or `ConnCase` imports - - Dart/Flutter: `**/*_test.dart` located in `test/` directory (not `integration_test/`) - - **Import-based inference:** sample 5 test files — if they import only local project modules and no HTTP clients, DB clients, or external service SDKs, treat them as unit-scoped. - -- **Pass:** Unit tests detected via any signal above -- **Warn:** Test files exist but none can be clearly identified as unit-scoped -- **Fail:** No unit test signals detected -- **Severity:** high - ---- - -### QA-03: Integration tier present - -- **What:** The project has tests that verify interactions between components — across real databases, real HTTP calls, or real message queues -- **How:** Look for any of the following signals: - - **File/directory naming:** - - Files: `*.integration.test.*`, `*.integration.spec.*`, `*.int.test.*`, `*_integration_test.go`, `*IT.java`, `*IT.kt` - - Directories: `tests/integration/`, `__tests__/integration/`, `spec/integration/` - - **Annotation/framework signals by stack:** - - Spring Boot (Java/Kotlin): `@SpringBootTest`, `@DataJpaTest`, `@WebMvcTest`, `@DataMongoTest`, `@RestClientTest`, `@ServiceConnection` - - Testcontainers (any JVM): `@Testcontainers`, `@Container`, or imports from `org.testcontainers` - - Python (pytest): `@pytest.mark.integration`, or `integration` marker registered in `pytest.ini` / `pyproject.toml`, or `conftest.py` with DB/HTTP fixtures - - Ruby (RSpec): `type: :integration` metadata, or tests importing `DatabaseCleaner` or `FactoryBot` against a real database - - PHP (PHPUnit): `@group integration` annotation, or extending `AbstractIntegrationTestCase` - - Go: `//go:build integration` or legacy `// +build integration` build constraint in test files - - Rust: files inside the top-level `tests/` directory (Rust canonical location for integration tests, separate from `src/`) - - Elixir: test files that import `Phoenix.ConnTest`, `Ecto.Adapters.SQL.Sandbox`, or use `DataCase` / `ConnCase` from Phoenix - -- **Pass:** Integration tests detected via any signal above -- **Warn:** Tests appear to hit real dependencies (DB/HTTP imports in test files) but no explicit integration markers found -- **Fail:** No integration test signals detected -- **Severity:** high - ---- - -### QA-04: E2E tier present - -- **What:** The project has end-to-end tests that exercise complete user flows through a real UI, API surface, or CLI -- **How:** Look for any of the following signals: - - **Config file signals:** - - `playwright.config.{ts,js,mts,mjs}` — Playwright - - `cypress.config.{ts,js}`, `cypress.json` — Cypress - - `wdio.config.{ts,js}` — WebdriverIO - - `.detoxrc.{js,json}`, `detox.config.js` — Detox (React Native) - - `appium.config.{js,ts}`, `.appiumrc` — Appium - - `nightwatch.conf.js`, `nightwatch.config.js` — Nightwatch - - `codecept.conf.{js,ts}` — CodeceptJS - - `testcafe.config.js` or `testcafe` key in `package.json` — TestCafe - - **Directory/file signals:** - - Directories: `cypress/`, `e2e/`, `tests/e2e/`, `__tests__/e2e/` - - `maestro/` or `.maestro/` containing `*.yaml` flow files (Maestro mobile) - - `integration_test/**/*_test.dart` (Flutter E2E) - - `**/*UITests.swift`, `**/*UITest.swift` (Swift/Xcode UI tests) - - `androidTest/**/*.{java,kt}` (Android Espresso) - - Files: `**/*.e2e.test.*`, `**/*.e2e.spec.*`, `**/*.e2e-spec.*` - - **Annotation signals:** - - Python: `@pytest.mark.e2e` - - Java/Kotlin (JUnit 5): `@Tag("e2e")` - - PHP (PHPUnit): `@group e2e` - - Go: `//go:build e2e` - -- **Pass:** E2E tooling config AND test files both present -- **Warn:** E2E config present but no test files found (tooling set up but unused) -- **Fail:** No E2E signals detected -- **Skip-When:** Topology shows library (no runnable entry point) -- **Severity:** high - ---- - -### QA-05: Pyramid shape — no inversion - -- **What:** The distribution of tests across tiers follows a healthy pyramid: most tests are unit-level, fewer are integration-level, fewest are E2E -- **How:** Use findings from QA-02, QA-03, and QA-04 to estimate test counts at each tier. Count test files (or test definitions for declarative stacks) matched by each tier's signals. A healthy pyramid satisfies: unit_count >= integration_count >= e2e_count. -- **Pass:** unit_count >= integration_count >= e2e_count, or only one tier is present -- **Warn:** E2E count exceeds unit count but integration layer exists as a buffer between them -- **Fail:** E2E count > unit count (inverted pyramid), or integration count > unit count by a significant margin (2× or more) -- **Skip-When:** Fewer than 2 tiers were detected in QA-02/03/04 -- **Severity:** medium - ---- - -### QA-06: Coverage reporting configured - -- **What:** The project measures what percentage of source code is exercised by tests, optionally enforcing a minimum threshold -- **How:** Check for coverage tool configuration: - - Jest: `collectCoverage: true` or `coverageThreshold` key in `jest.config.*` - - Vitest: `coverage` section in `vitest.config.*` - - pytest-cov: `pytest-cov` in `requirements*.txt` or `pyproject.toml`; `--cov` flag in `pytest.ini` or `pyproject.toml` `addopts` - - JaCoCo: `jacoco` plugin in `build.gradle` or `build.gradle.kts` - - nyc / c8: `.nycrc`, `.nycrc.json`, or `c8` / `nyc` script in `package.json` -- **Pass:** Coverage tool configured with thresholds defined -- **Warn:** Coverage tool present but no thresholds defined, OR no coverage tooling found -- **Severity:** low - ---- - -### QA-07: Test data management - -- **What:** Tests use a structured approach to create and manage test data, rather than scattering hardcoded inline values across test files -- **How:** Check for: - - Fixture directories: `fixtures/`, `__fixtures__/`, `testdata/`, `test/fixtures/`, `spec/fixtures/` - - Factory libraries in dependencies: `factory-girl`, `fishery`, `rosie` (JS/TS); `factory_boy` (Python); `FactoryBot` / `factory_bot` (Ruby); `gomock` (Go) - - Faker libraries: `faker`, `@faker-js/faker`, `Faker` (PHP), `Bogus` (C#) - - Seed scripts: `seeds/`, `db/seeds/`, `prisma/seed.*`, `scripts/seed.*` - - Confirm usage: grep for the detected library/directory name inside test files to verify it is actually used (not just installed) -- **Pass:** Fixtures or factories present and referenced in test files -- **Warn:** One approach present but sparse (library installed but used in fewer than 3 test files) -- **Fail:** No test data strategy — tests use only hardcoded inline values -- **Severity:** low - ---- - -### QA-08: Test isolation — mocking infrastructure - -- **What:** Unit and integration tests use mocking/stubbing to isolate the code under test from external dependencies -- **How:** Check for mocking libraries in dependencies or imports: - - JS/TS: `jest.mock`, `vi.mock`, `sinon`, `nock`, `msw` (mock service worker) - - Python: `unittest.mock`, `pytest-mock`, `responses`, `httpretty` - - Java/Kotlin: `mockito-core`, `mockk`, `WireMock` - - Go: `gomock`, `testify/mock` - - Ruby: `rspec-mocks` (included in RSpec), `webmock`, `vcr` - - PHP: PHPUnit built-in mock builder (`createMock`, `getMockBuilder`) - - Rust: `mockall`, `mockito` - - Elixir: `Mox` - - Sample 5 test files and confirm mocking is actively used (grep for `mock`, `stub`, `spy`, `fake`, or library-specific calls). - -- **Pass:** Mocking library present and actively used in sampled test files -- **Warn:** Library present but used in fewer than 2 of the 5 sampled files -- **Fail:** No mocking infrastructure detected -- **Severity:** medium - ---- - -### QA-09: Contract testing - -- **What:** Service boundaries are verified through consumer-driven contract tests, ensuring producers don't break consumers -- **How:** Check for contract testing frameworks: - - Pact: `pact/` directory, `**/*.pact.json`, `@pact-foundation/pact` in dependencies, `au.com.dius.pact` in Gradle - - Spring Cloud Contract: `contracts/` directory with `*.groovy` or `*.yml` contract files - - Schemathesis / Dredd: `schemathesis` or `dredd` in dependencies with config files - - Karate: `**/*.feature` files with contract-style API tests -- **Pass:** Contract tests present covering at least one service boundary -- **Warn:** Contract tooling installed but no contract files found -- **Fail:** No contract testing detected -- **Skip-When:** Topology shows single-service repo or no inter-service communication patterns detected (TOPO-06 found no communication layer) -- **Severity:** high - ---- - -### QA-10: ML model iteration testing - -- **What:** ML models are tested for quality metrics as part of the development cycle, not just functional correctness -- **How:** Check for ML evaluation frameworks or patterns in test files: - - `great_expectations` config (`great_expectations.yml` or `great_expectations/` directory) - - `deepchecks` in dependencies - - `evidently` in dependencies - - `whylogs` in dependencies - - Test files that import ML frameworks (`sklearn`, `torch`, `tensorflow`, `xgboost`, `transformers`, `keras`) AND contain assertions on model quality metrics (accuracy, f1, precision, recall, loss thresholds) — grep for `assert.*score`, `assert.*accuracy`, `assert.*f1`, or similar patterns -- **Pass:** Model evaluation tests present with explicit quality metric assertions -- **Warn:** ML framework imports appear in test files but no metric assertions found (tests exist but don't gate on model quality) -- **Fail:** No ML model testing detected -- **Skip-When:** Topology shows no ML layer — no ML framework imports (`sklearn`, `torch`, `tensorflow`, `xgboost`, `transformers`) found in source files -- **Severity:** high -``` - -- [ ] **Step 2: Verify file exists and frontmatter is valid** - -```bash -head -8 plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md -``` - -Expected output: - -``` ---- -name: quality-assurance -title: Quality Assurance -description: Evaluates testing pyramid structure, tooling maturity, and optional contract/ML testing readiness -severity: high -depends-on: [project-topology] ---- -``` - -- [ ] **Step 3: Verify all 10 checks are present** - -```bash -grep "^### QA-" plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md -``` - -Expected output: - -``` -### QA-01: Test infrastructure exists with adequate coverage -### QA-02: Unit tier present -### QA-03: Integration tier present -### QA-04: E2E tier present -### QA-05: Pyramid shape — no inversion -### QA-06: Coverage reporting configured -### QA-07: Test data management -### QA-08: Test isolation — mocking infrastructure -### QA-09: Contract testing -### QA-10: ML model iteration testing -``` - -- [ ] **Step 4: Verify dimension is discoverable by the orchestrator** - -```bash -ls plugins/awos/skills/ai-readiness-audit/dimensions/ -``` - -Expected: `quality-assurance.md` appears in the list alongside the other 8 dimension files. - -- [ ] **Step 5: Commit** - -```bash -git add plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md -git commit -m "feat(audit): add quality-assurance dimension with testing pyramid checks" -``` - ---- - -## Task 4: Verify full DAG consistency - -- [ ] **Step 1: Check all depends-on references resolve** - -Every name in any `depends-on` list must match a `name:` field in another dimension file. Run: - -```bash -grep "^name:" plugins/awos/skills/ai-readiness-audit/dimensions/*.md -``` - -Expected names present: `project-topology`, `documentation`, `security`, `ai-development-tooling`, `spec-driven-development`, `code-architecture`, `software-best-practices`, `quality-assurance`, `end-to-end-delivery` - -- [ ] **Step 2: Confirm end-to-end-delivery depends-on list is complete** - -```bash -grep -A 12 "depends-on" plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md -``` - -Expected: both `software-best-practices` and `quality-assurance` are in the list. - -- [ ] **Step 3: Confirm SBP-04 no longer exists in software-best-practices** - -```bash -grep "SBP-04" plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md -``` - -Expected: no output (empty). - -- [ ] **Step 4: Final commit** - -```bash -git add plugins/awos/skills/ai-readiness-audit/dimensions/ -git commit -m "feat(audit): verify quality-assurance DAG integration" -``` - -Only run this step if there are uncommitted changes. If Tasks 1–3 each committed cleanly, skip this step. diff --git a/docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md b/docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md deleted file mode 100644 index 8b262fc2..00000000 --- a/docs/superpowers/specs/2026-04-09-quality-assurance-dimension-design.md +++ /dev/null @@ -1,246 +0,0 @@ -# Quality Assurance Dimension — Design Spec - -**Date:** 2026-04-09 -**Status:** Approved for implementation - ---- - -## Summary - -Add a new `quality-assurance` dimension to the AWOS AI Readiness Audit plugin. This dimension provides deep evaluation of a project's testing maturity by analyzing testing pyramid structure, tooling, and optional conditional checks for contract and ML testing. The existing SBP-04 check is removed from `software-best-practices` and a lineage note is added there pointing to this new dimension. - ---- - -## DAG Placement - -- **New dimension:** `quality-assurance` -- **depends-on:** `[project-topology]` — Phase 2, same as other substantive dimensions -- **Impact on existing DAG:** `end-to-end-delivery` currently depends on `software-best-practices`; it must also depend on `quality-assurance` since QA-01 replaces SBP-04 -- **Frontmatter severity:** `high` - ---- - -## Changes to `software-best-practices.md` - -- Remove SBP-04 entirely -- Add after SBP-03 (before SBP-05): - - > _Test infrastructure and coverage are evaluated in the **Quality Assurance** dimension (`quality-assurance.md`), which provides a full testing pyramid analysis._ - -- No renumbering — the gap between SBP-03 and SBP-05 is bridged by the note - ---- - -## Checks - -### QA-01: Test infrastructure exists - -Full port of SBP-04. Detects test files or declarative test definitions across all supported stacks, excluding `node_modules/`, `build/`, `dist/`, `vendor/`, `.venv/`. - -**Traditional test file globs:** - -| Stack | Patterns | -| ------------ | ----------------------------------------------------------------------------- | -| JS/TS | `**/*.test.{ts,tsx,js,jsx}`, `**/*.spec.{ts,tsx,js,jsx}`, `**/__tests__/**/*` | -| Python | `**/test_*.py`, `**/*_test.py` | -| Java | `**/*Test.java`, `**/*Tests.java`, `**/*IT.java` | -| Kotlin | `**/*Test.kt`, `**/*Spec.kt`, `**/*IT.kt` | -| Go | `**/*_test.go` | -| Rust | `tests/` directory OR files containing `#[cfg(test)]` module | -| Ruby | `**/*_spec.rb`, `**/*_test.rb`, `**/test_*.rb` | -| PHP | `**/*Test.php` | -| Elixir | `**/*_test.exs` | -| Swift | `**/*Tests.swift`, `**/*Test.swift` | -| Dart/Flutter | `**/*_test.dart` | -| C# | `**/*Tests.cs`, `**/*Test.cs` | - -**Declarative frameworks** (dbt, Terraform, Maestro YAML): count individual test definitions, not files. - -Calculate test-coverage ratio: tested source modules / total source modules. - -- **Pass:** ratio ≥ 60% -- **Warn:** ratio > 0% but < 60% -- **Fail:** no tests found -- **Severity:** critical - ---- - -### QA-02: Unit tier present - -**File/directory signals:** - -- Naming: `*.unit.test.*`, `*.unit.spec.*` -- Directories: `tests/unit/`, `__tests__/unit/`, `spec/unit/` - -**Annotation/marker signals:** - -| Stack | Markers | -| --------------------- | ---------------------------------------------------------------------- | -| Java/Kotlin (JUnit 5) | `@Tag("unit")`, `@Tag("fast")` | -| Python (pytest) | `@pytest.mark.unit`, `unit` marker in `pytest.ini` or `pyproject.toml` | -| Ruby (RSpec) | `:unit` tag on `describe`/`context` blocks | -| PHP (PHPUnit) | `@group unit` docblock annotation | -| Go | `*_test.go` with no `//go:build integration` or `//go:build e2e` tag | -| Rust | `#[cfg(test)]` module co-located inside `src/` files | -| Swift (XCTest) | Test target name ending in `UnitTests` or `Tests` (not `UITests`) | -| Elixir (ExUnit) | `use ExUnit.Case` without Phoenix/Ecto integration imports | -| Dart/Flutter | `**/*_test.dart` in `test/` directory (not `integration_test/`) | - -**Import-based inference:** test file imports only local modules with no HTTP clients, DB clients, or external service SDKs. - -- **Pass:** unit tests detected via any signal above -- **Warn:** test files exist but none are clearly unit-scoped -- **Fail:** no unit test signals detected -- **Severity:** high - ---- - -### QA-03: Integration tier present - -**File/directory signals:** - -- Naming: `*.integration.test.*`, `*.integration.spec.*`, `*.int.test.*`, `*_integration_test.go`, `*IT.java`, `*IT.kt` -- Directories: `tests/integration/`, `__tests__/integration/`, `spec/integration/` - -**Annotation/framework signals:** - -| Stack | Markers | -| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- | -| Spring Boot (Java/Kotlin) | `@SpringBootTest`, `@DataJpaTest`, `@WebMvcTest`, `@DataMongoTest`, `@RestClientTest`, `@ServiceConnection` | -| Testcontainers (any JVM) | `@Testcontainers`, `@Container`, imports from `org.testcontainers` | -| Python (pytest) | `@pytest.mark.integration`, `conftest.py` with DB/HTTP fixtures, `integration` marker in `pytest.ini`/`pyproject.toml` | -| Ruby (RSpec) | `type: :integration` metadata, or tests using `DatabaseCleaner`/`FactoryBot` with real DB | -| PHP (PHPUnit) | `@group integration`, extending `AbstractIntegrationTestCase` | -| Go | `//go:build integration` or legacy `// +build integration` tag | -| Rust | Files in top-level `tests/` directory (Rust canonical integration test location) | -| Elixir | Test files using `Phoenix.ConnTest`, `Ecto.Adapters.SQL.Sandbox`, `DataCase`, or `ConnCase` | - -- **Pass:** integration tests detected via any signal above -- **Warn:** tests appear to hit real dependencies but no explicit integration markers found -- **Fail:** no integration test signals detected -- **Severity:** high - ---- - -### QA-04: E2E tier present - -**Config file signals:** - -| Tool | Config Files | -| -------------------- | -------------------------------------------------------- | -| Playwright | `playwright.config.{ts,js,mts,mjs}` | -| Cypress | `cypress.config.{ts,js}`, `cypress.json` | -| WebdriverIO | `wdio.config.{ts,js}` | -| Detox (React Native) | `.detoxrc.{js,json}`, `detox.config.js` | -| Appium | `appium.config.{js,ts}`, `.appiumrc` | -| Nightwatch | `nightwatch.conf.js`, `nightwatch.config.js` | -| CodeceptJS | `codecept.conf.{js,ts}` | -| TestCafe | `testcafe.config.js` or `testcafe` key in `package.json` | - -**Directory/file signals:** - -- `cypress/`, `e2e/`, `tests/e2e/`, `__tests__/e2e/` -- `maestro/` or `.maestro/` with `*.yaml` flow files (Maestro mobile) -- `integration_test/**/*_test.dart` (Flutter E2E) -- `**/*UITests.swift`, `**/*UITest.swift` (Swift/Xcode UI tests) -- `androidTest/**/*.{java,kt}` (Android Espresso) -- `**/*.e2e.test.*`, `**/*.e2e.spec.*`, `**/*.e2e-spec.*` - -**Annotation signals:** - -| Stack | Markers | -| --------------------- | ------------------ | -| Python | `@pytest.mark.e2e` | -| Java/Kotlin (JUnit 5) | `@Tag("e2e")` | -| PHP (PHPUnit) | `@group e2e` | -| Go | `//go:build e2e` | - -- **Pass:** E2E tooling config and test files both present -- **Warn:** E2E config present but no test files found (tooling set up but unused) -- **Fail:** no E2E signals detected -- **Skip-When:** topology shows library (no runnable entry point) -- **Severity:** high - ---- - -### QA-05: Pyramid shape — no inversion - -Uses findings from QA-02, QA-03, QA-04 to count tests at each tier. A healthy pyramid: unit ≥ integration ≥ E2E. - -- **Pass:** unit count ≥ integration count ≥ E2E count, or only one tier exists -- **Warn:** E2E count exceeds unit count but integration layer exists as a buffer -- **Fail:** E2E count > unit count (inverted pyramid), or integration count > unit count significantly -- **Skip-When:** fewer than 2 tiers detected -- **Severity:** medium - ---- - -### QA-06: Coverage reporting configured _(optional)_ - -Check for coverage tool configuration: `jest` with `coverageThreshold` or `collectCoverage`, `vitest` coverage config, `pytest-cov` in requirements/pyproject, `jacoco` plugin in Gradle, `.nycrc`, `c8` in scripts. - -- **Pass:** coverage tool configured with thresholds defined -- **Warn:** coverage tool present but no thresholds, or no coverage tooling found -- **Severity:** low _(no FAIL state — this check is advisory only)_ - ---- - -### QA-07: Test data management - -Check for fixture directories (`fixtures/`, `__fixtures__/`, `testdata/`), factory libraries (`factory-girl`, `fishery`, `faker`, `factory_boy`, `gomock`), or seed scripts (`seeds/`, `db/seeds/`). - -- **Pass:** fixtures or factories present and referenced in test files -- **Warn:** one approach present but sparse usage -- **Fail:** no test data strategy — tests appear to use only hardcoded inline values -- **Severity:** low - ---- - -### QA-08: Test isolation — mocking infrastructure - -Check for mocking libraries: `jest.mock`, `vi.mock`, `sinon`, `nock`, `msw`, `mockito`, `gomock`, `unittest.mock`. Sample 5 test files to confirm mocks are used to isolate external dependencies in unit/integration tests. - -- **Pass:** mocking library present and actively used -- **Warn:** library present but minimal usage found in sampled files -- **Fail:** no mocking infrastructure detected -- **Severity:** medium - ---- - -### QA-09: Contract testing _(conditional)_ - -Check for consumer-driven contract frameworks: Pact files (`pact/`, `**/*.pact.json`), Spring Cloud Contract (`contracts/`), or similar. - -- **Pass:** contract tests present covering at least one service boundary -- **Warn:** contract tooling configured but no contract files found -- **Fail:** no contract tests -- **Skip-When:** topology shows single-service repo or no inter-service communication detected -- **Severity:** high - ---- - -### QA-10: ML model iteration testing _(conditional)_ - -Check for ML testing patterns: `pytest` with model evaluation assertions, `great_expectations` config, `deepchecks`, `evidently`, `whylogs`, or test files that import ML frameworks (`sklearn`, `torch`, `tensorflow`, `xgboost`, `transformers`) and assert on model quality metrics. - -- **Pass:** model evaluation tests present with quality metric assertions -- **Warn:** ML framework imports appear in test context but no metric assertions found -- **Fail:** no ML model testing -- **Skip-When:** topology shows no ML layer (no ML framework imports in source files) -- **Severity:** high - ---- - -## Scoring - -Standard AWOS scoring applies (from `scoring.md`): each check contributes to the dimension percentage. Skipped checks are excluded from the denominator. QA-06 has no FAIL state — its worst outcome is WARN, making it a low-weight advisory signal. - ---- - -## Files to Create / Modify - -| Action | File | -| ------ | -------------------------------------------------------------------------------------------------------------------- | -| Create | `plugins/awos/skills/ai-readiness-audit/dimensions/quality-assurance.md` | -| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/software-best-practices.md` — remove SBP-04, add lineage note | -| Modify | `plugins/awos/skills/ai-readiness-audit/dimensions/end-to-end-delivery.md` — add `quality-assurance` to `depends-on` | From 0a3eb4e48980f4b2094d1ecedb36a447beb95247 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Tue, 14 Apr 2026 12:57:20 +0300 Subject: [PATCH 16/56] fix(tasks): sequential numbering and opt-out test generation Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index 9ba92fc3..0defc64a 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -59,20 +59,20 @@ Follow this process precisely. 2. Create a high-level checklist item for that slice (e.g., `- [ ] **Slice 1: View existing avatar (or placeholder)**`). 3. Under that slice, create the nested sub-tasks (database, backend, frontend) needed to implement and verify **only that slice**. 4. **For each sub-task, assign the appropriate subagent:** - Analyze the sub-task description to understand what technology/domain it involves - Analyze the Task tool definition to extract all available subagent_type values with their descriptions to understand what subagents are available for assignment. - Match the sub-task to a subagent based on: - Technology keywords - Task intent - Tech stack identified in technical-considerations.md - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table - 3b. **Generate paired test tasks for each slice (REQUIRED):** - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - Example result for a slice: - ` - [ ] **Slice 1: User authentication** - - [ ] Implement JWT token generation **[Agent: python-expert]** - - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** - - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** - - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** - - [ ] Contract: /auth response schema validation — positive cases **[Agent: testing-expert]** - - [ ] Contract: schema violations and malformed payload cases — negative cases **[Agent: testing-expert]** - ` - 5. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. - 6. Create a high-level checklist item and its sub-tasks with subagent assignments. - 7. Repeat this process until all requirements from the specification are covered. - 8. For each slice's verification sub-task, identify required MCPs/services (browser MCP, curl, database access, etc.) and note any that may be missing. + 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - Example result for a slice: + ` - [ ] **Slice 1: User authentication** + - [ ] Implement JWT token generation **[Agent: python-expert]** + - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** + - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** + - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** + - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** + - [ ] Contract: /auth response schema validation — positive cases **[Agent: testing-expert]** + - [ ] Contract: schema violations and malformed payload cases — negative cases **[Agent: testing-expert]** + ` + 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. + 7. Create a high-level checklist item and its sub-tasks with subagent assignments. + 8. Repeat this process until all requirements from the specification are covered. + 9. For each slice's verification sub-task, identify required MCPs/services (browser MCP, curl, database access, etc.) and note any that may be missing. - **Example of applying the rule for "User Profile Picture Upload":** - **Bad, Horizontal Tasks (DO NOT DO THIS):** From f913646a3412cc8faf930dddf1a522166f2246f2 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 17:31:31 +0300 Subject: [PATCH 17/56] fix(testing-expert): move from commands/ to plugins/awos/agents/ --- {commands => plugins/awos/agents}/testing-expert.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) rename {commands => plugins/awos/agents}/testing-expert.md (95%) diff --git a/commands/testing-expert.md b/plugins/awos/agents/testing-expert.md similarity index 95% rename from commands/testing-expert.md rename to plugins/awos/agents/testing-expert.md index e3f9bd55..47dc5d06 100644 --- a/commands/testing-expert.md +++ b/plugins/awos/agents/testing-expert.md @@ -1,5 +1,11 @@ --- -description: Plans and executes layered test suites (unit, integration, e2e, contract) for vertical slices. Enforces negative tests and RED validation. +name: testing-expert +description: >- + QA agent for vertical slices. When called with spec files and no + implementation code, returns structured test task descriptions (planning). + When called with spec files and implementation code, writes tests with RED + validation across unit, integration, e2e, and contract layers. +tools: Read, Write, Bash, Glob, Grep --- # ROLE From 92a2d4fb939d9dda6c3155eb94d0d8b7369eee01 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 17:40:36 +0300 Subject: [PATCH 18/56] fix(tasks): replace vague 'planning mode' with explicit Agent tool invocation Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/commands/tasks.md b/commands/tasks.md index 0defc64a..22892c00 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -59,7 +59,14 @@ Follow this process precisely. 2. Create a high-level checklist item for that slice (e.g., `- [ ] **Slice 1: View existing avatar (or placeholder)**`). 3. Under that slice, create the nested sub-tasks (database, backend, frontend) needed to implement and verify **only that slice**. 4. **For each sub-task, assign the appropriate subagent:** - Analyze the sub-task description to understand what technology/domain it involves - Analyze the Task tool definition to extract all available subagent_type values with their descriptions to understand what subagents are available for assignment. - Match the sub-task to a subagent based on: - Technology keywords - Task intent - Tech stack identified in technical-considerations.md - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table - 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** - After defining the implementation sub-tasks for a slice, invoke the `testing-expert` agent in **planning mode**. - Pass it: the current slice's implementation sub-task description, `functional-spec.md`, and `technical-considerations.md`. - The `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - Example result for a slice: + 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** + - After defining the implementation sub-tasks for a slice, use the `Agent` tool with `subagent_type: "testing-expert"`, passing as context: the current slice's implementation sub-task description, the contents of `functional-spec.md`, and `technical-considerations.md`. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. + - `testing-expert` is a generic fallback agent provided by the awos plugin. If `/awos:hire` has installed a project-specific testing agent (e.g. `react-testing`, `python-testing`) for this tech stack, prefer that agent instead — it takes precedence by being in `.claude/agents/`. + - `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. Not every slice needs all four layers — the agent applies judgment based on acceptance criteria. + - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. + - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. + - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. + - Example result for a slice: ` - [ ] **Slice 1: User authentication** - [ ] Implement JWT token generation **[Agent: python-expert]** - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** From 637566b3b4028efabf83ac44cc6068fede17c30b Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 17:52:11 +0300 Subject: [PATCH 19/56] fix(tasks): clarify Agent tool context-passing and move opt-out check first --- commands/tasks.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index 22892c00..9019bd7d 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -60,11 +60,11 @@ Follow this process precisely. 3. Under that slice, create the nested sub-tasks (database, backend, frontend) needed to implement and verify **only that slice**. 4. **For each sub-task, assign the appropriate subagent:** - Analyze the sub-task description to understand what technology/domain it involves - Analyze the Task tool definition to extract all available subagent_type values with their descriptions to understand what subagents are available for assignment. - Match the sub-task to a subagent based on: - Technology keywords - Task intent - Tech stack identified in technical-considerations.md - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** - - After defining the implementation sub-tasks for a slice, use the `Agent` tool with `subagent_type: "testing-expert"`, passing as context: the current slice's implementation sub-task description, the contents of `functional-spec.md`, and `technical-considerations.md`. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. + - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. + - After defining the implementation sub-tasks for a slice, use the `Agent` tool with `subagent_type: "testing-expert"`. Read `functional-spec.md` and `technical-considerations.md` and include their contents inline in the prompt, along with the current slice's implementation sub-task description. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. - `testing-expert` is a generic fallback agent provided by the awos plugin. If `/awos:hire` has installed a project-specific testing agent (e.g. `react-testing`, `python-testing`) for this tech stack, prefer that agent instead — it takes precedence by being in `.claude/agents/`. - `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. Not every slice needs all four layers — the agent applies judgment based on acceptance criteria. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - Example result for a slice: ` - [ ] **Slice 1: User authentication** From d03f7766fbe3bb9d513f87aee2e950759deecf09 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 17:54:49 +0300 Subject: [PATCH 20/56] fix(tasks): rebalance examples to show layer judgment and consistent test coverage Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index 9019bd7d..156f829f 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -73,8 +73,6 @@ Follow this process precisely. - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** - - [ ] Contract: /auth response schema validation — positive cases **[Agent: testing-expert]** - - [ ] Contract: schema violations and malformed payload cases — negative cases **[Agent: testing-expert]** ` 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 7. Create a high-level checklist item and its sub-tasks with subagent assignments. @@ -90,6 +88,8 @@ Follow this process precisely. - `[ ] **Slice 1: Display a placeholder avatar on the profile page**` - `[ ] Sub-task: Add a non-functional 'ProfileAvatar' UI component that shows a static placeholder image. **[Agent: react-expert]**` - `[ ] Sub-task: Place the component on the profile page. **[Agent: react-expert]**` + - `[ ] Unit: ProfileAvatar renders placeholder when no src provided — positive cases **[Agent: testing-expert]**` + - `[ ] Unit: ProfileAvatar renders without crashing when src is undefined — negative cases **[Agent: testing-expert]**` - `[ ] **Slice 2: Display the user's actual avatar if it exists**` - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` @@ -99,8 +99,6 @@ Follow this process precisely. - `[ ] Unit: null avatar_url, missing column, type mismatch — negative cases **[Agent: testing-expert]**` - `[ ] Integration: GET /user returns avatar_url when set — positive cases **[Agent: testing-expert]**` - `[ ] Integration: GET /user returns null when avatar_url not set — negative cases **[Agent: testing-expert]**` - - `[ ] E2E: profile page shows avatar when avatar_url present — positive cases **[Agent: testing-expert]**` - - `[ ] E2E: profile page shows placeholder when avatar_url is null — negative cases **[Agent: testing-expert]**` ## Step 4: Present Draft and Refine From 6f110e5ab499353b366613eed925395b72e656d6 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:11:29 +0300 Subject: [PATCH 21/56] fix(testing-expert): replace caller-based mode detection with condition-based --- plugins/awos/agents/testing-expert.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plugins/awos/agents/testing-expert.md b/plugins/awos/agents/testing-expert.md index 47dc5d06..aab90916 100644 --- a/plugins/awos/agents/testing-expert.md +++ b/plugins/awos/agents/testing-expert.md @@ -18,9 +18,9 @@ You are an expert QA Engineer and Test Automation Specialist. You operate in two Read the invocation context: -- **Planning Mode** — called by `/awos:tasks` with a functional spec and technical spec but NO existing implementation code. Your job is to return structured test task descriptions for `tasks.md`. You do NOT write test code in this mode. -- **Execution Mode** — called by `/awos:implement` with a specific test task, the implementation code, and full spec context. Your job is to write, RED-validate, and run real test code. -- **Direct invocation (no caller context)** — ask the user: "Are you planning test tasks for a spec, or executing a specific test task?" then proceed to the appropriate mode. +- **No implementation code in context** — you received spec files and a slice/task description, but no existing source code. Your job is to return structured test task descriptions. You do NOT write test code in this mode. +- **Implementation code present in context** — you received spec files, a test task description, and implementation source code. Your job is to write, RED-validate, and run real test code. +- **Ambiguous context (cannot determine)** — ask: "Are you planning test tasks for a spec, or executing a specific test task?" then proceed accordingly. --- From 729d1920d4a6dad9927e7cad11711a57c2fab0a2 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:14:12 +0300 Subject: [PATCH 22/56] fix(tasks): add missing verify sub-tasks to JWT and avatar Slice 1 examples --- commands/tasks.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/commands/tasks.md b/commands/tasks.md index 156f829f..cc53c418 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -73,6 +73,7 @@ Follow this process precisely. - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** + - [ ] Verify: Start the server, call POST /auth with valid credentials, confirm JWT is returned **[Agent: manual-qa-expert]** ` 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 7. Create a high-level checklist item and its sub-tasks with subagent assignments. @@ -90,6 +91,7 @@ Follow this process precisely. - `[ ] Sub-task: Place the component on the profile page. **[Agent: react-expert]**` - `[ ] Unit: ProfileAvatar renders placeholder when no src provided — positive cases **[Agent: testing-expert]**` - `[ ] Unit: ProfileAvatar renders without crashing when src is undefined — negative cases **[Agent: testing-expert]**` + - `[ ] Verify: Start the app, open the profile page, confirm placeholder avatar is shown **[Agent: manual-qa-expert]**` - `[ ] **Slice 2: Display the user's actual avatar if it exists**` - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` From 6b15fb1b5a61dc74f50e0afb2bd432a3b33a2ada Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:14:29 +0300 Subject: [PATCH 23/56] fix(qa): remove stale testing-expert cross-reference, clarify e2e-tester, add regression-suite template init --- commands/qa.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/commands/qa.md b/commands/qa.md index c8d419a2..2a4ea611 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -90,7 +90,7 @@ For each acceptance criterion in the functional spec(s) in scope: For each gap identified in Step 5: -1. Write the missing test following RED validation discipline (following `testing-expert` execution mode Step 3): +1. Write the missing test following RED validation discipline: - Write test → confirm it FAILS for the right reason → confirm it PASSES. 2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations using appropriate comment syntax for the language. 3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: @@ -122,6 +122,8 @@ Append net-new entries using this format: ## Step 8: Update regression suite +If `context/qa/regression-suite.md` does not exist, create it from the template at `.awos/templates/regression-suite-template.md` (or create an empty file with a `# Regression Suite` header if the template is absent). + Sync `context/qa/regression-suite.md`: - Scan all test files in scope for `@regression` annotations. @@ -179,7 +181,7 @@ Report summary to user and list any flags requiring human attention. # TODO -- **E2E tests are ephemeral — no CI artifact.** The `/awos:qa` command runs E2E tests via the `e2e-tester` agent (playwright-cli) interactively but produces no persistent test scripts. Each run is unrepeatable without a human in the loop. Future work: generate rerunnable playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as a Step 6 output for E2E gaps, so they can be committed and executed in CI without agent interaction. +- **E2E tests are ephemeral — no CI artifact.** The `/awos:qa` command currently runs E2E tests inline. Future work: delegate to a dedicated `e2e-tester` agent (playwright-cli) to run E2E tests interactively but produces no persistent test scripts. Each run is unrepeatable without a human in the loop. Future work: generate rerunnable playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as a Step 6 output for E2E gaps, so they can be committed and executed in CI without agent interaction. - **QA audit is coverage-by-inspection, not coverage-by-measurement.** Step 5 (gap analysis) reads source files and infers coverage from test file contents. It does not invoke actual coverage tooling (`vitest --coverage`, Istanbul, c8, pytest-cov, etc.). This means untested branches and dead-path gaps are invisible to the audit. Future work: in Step 2, detect available coverage reporters and, in Step 7, run the suite with coverage flags; parse the output to feed real line/branch metrics into the Coverage Summary table. From de71e7cd4525e31c853ddb7118adc3f2c8999c2b Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:18:22 +0300 Subject: [PATCH 24/56] fix: clean up garbled e2e-tester TODO and rename mode headings to match condition-based detection --- commands/qa.md | 2 +- plugins/awos/agents/testing-expert.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/commands/qa.md b/commands/qa.md index 2a4ea611..5bbfae67 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -181,7 +181,7 @@ Report summary to user and list any flags requiring human attention. # TODO -- **E2E tests are ephemeral — no CI artifact.** The `/awos:qa` command currently runs E2E tests inline. Future work: delegate to a dedicated `e2e-tester` agent (playwright-cli) to run E2E tests interactively but produces no persistent test scripts. Each run is unrepeatable without a human in the loop. Future work: generate rerunnable playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as a Step 6 output for E2E gaps, so they can be committed and executed in CI without agent interaction. +- **E2E tests are ephemeral — no CI artifact.** The `/awos:qa` command currently runs E2E tests inline without a dedicated agent. Future work: delegate to a dedicated `e2e-tester` agent and generate rerunnable playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as a Step 6 output for E2E gaps, so they can be committed and executed in CI without agent interaction. - **QA audit is coverage-by-inspection, not coverage-by-measurement.** Step 5 (gap analysis) reads source files and infers coverage from test file contents. It does not invoke actual coverage tooling (`vitest --coverage`, Istanbul, c8, pytest-cov, etc.). This means untested branches and dead-path gaps are invisible to the audit. Future work: in Step 2, detect available coverage reporters and, in Step 7, run the suite with coverage flags; parse the output to feed real line/branch metrics into the Coverage Summary table. diff --git a/plugins/awos/agents/testing-expert.md b/plugins/awos/agents/testing-expert.md index aab90916..c68a8f51 100644 --- a/plugins/awos/agents/testing-expert.md +++ b/plugins/awos/agents/testing-expert.md @@ -24,7 +24,7 @@ Read the invocation context: --- -# PLANNING MODE +# TASK DESCRIPTION MODE (no implementation code in context) ## Inputs @@ -72,7 +72,7 @@ Omit layers that genuinely don't apply. Always include negative cases for every --- -# EXECUTION MODE +# TEST WRITING MODE (implementation code present in context) ## Inputs From 5d649b7fea484040aece4002d05453e535355414 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:22:51 +0300 Subject: [PATCH 25/56] fix: update stale 'execution mode' reference and restore step 4 formatting --- commands/tasks.md | 10 +++++++++- templates/qa-context-template.md | 2 +- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index cc53c418..8181c0cb 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -58,7 +58,15 @@ Follow this process precisely. 1. First, identify the absolute smallest piece of user-visible value from the spec. This is your **Slice 1**. 2. Create a high-level checklist item for that slice (e.g., `- [ ] **Slice 1: View existing avatar (or placeholder)**`). 3. Under that slice, create the nested sub-tasks (database, backend, frontend) needed to implement and verify **only that slice**. - 4. **For each sub-task, assign the appropriate subagent:** - Analyze the sub-task description to understand what technology/domain it involves - Analyze the Task tool definition to extract all available subagent_type values with their descriptions to understand what subagents are available for assignment. - Match the sub-task to a subagent based on: - Technology keywords - Task intent - Tech stack identified in technical-considerations.md - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table + 4. **For each sub-task, assign the appropriate subagent:** + - Analyze the sub-task description to understand what technology/domain it involves + - Analyze the Task tool definition to extract all available `subagent_type` values with their descriptions to understand what subagents are available for assignment + - Match the sub-task to a subagent based on: + - Technology keywords + - Task intent + - Tech stack identified in `technical-considerations.md` + - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description + - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. - After defining the implementation sub-tasks for a slice, use the `Agent` tool with `subagent_type: "testing-expert"`. Read `functional-spec.md` and `technical-considerations.md` and include their contents inline in the prompt, along with the current slice's implementation sub-task description. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. diff --git a/templates/qa-context-template.md b/templates/qa-context-template.md index 68c01880..4c554538 100644 --- a/templates/qa-context-template.md +++ b/templates/qa-context-template.md @@ -1,6 +1,6 @@ # Test Registry -> Auto-maintained by `testing-expert` (execution mode) and `/awos:qa`. +> Auto-maintained by `testing-expert` (test writing mode) and `/awos:qa`. > Each row is a registered test. Add rows by running `/awos:implement` (test tasks) or `/awos:qa`. > Status values: OK | MISSING | NEEDS UPDATE | DEPRECATED | HUMAN REVIEW > Layer values: unit | integration | e2e | contract From 902d3ed304828af60a00d39083beeef3eee188f3 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:24:38 +0300 Subject: [PATCH 26/56] fix(testing-expert): remove remaining caller-specific references from role description and output format --- plugins/awos/agents/testing-expert.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/awos/agents/testing-expert.md b/plugins/awos/agents/testing-expert.md index c68a8f51..3337ce51 100644 --- a/plugins/awos/agents/testing-expert.md +++ b/plugins/awos/agents/testing-expert.md @@ -10,7 +10,7 @@ tools: Read, Write, Bash, Glob, Grep # ROLE -You are an expert QA Engineer and Test Automation Specialist. You operate in two distinct modes depending on who calls you and what context you receive. +You are an expert QA Engineer and Test Automation Specialist. You operate in two distinct modes determined by the context you receive. --- @@ -55,7 +55,7 @@ Not every slice needs all four layers. Apply judgment. For each applicable layer, generate two sub-tasks: one for positive cases, one for negative cases. -**Output format** (return this list to `/awos:tasks` for insertion into `tasks.md`): +**Output format** (return this list to the caller for insertion into `tasks.md`): ``` - [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** From 5222fc73a6b9132e0e4fcc3d4b6f017dd3999b6b Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:40:16 +0300 Subject: [PATCH 27/56] fix(tasks): use Task tool instead of Agent tool for testing-expert invocation --- commands/tasks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/commands/tasks.md b/commands/tasks.md index 8181c0cb..c0b7c795 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -69,7 +69,7 @@ Follow this process precisely. - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. - - After defining the implementation sub-tasks for a slice, use the `Agent` tool with `subagent_type: "testing-expert"`. Read `functional-spec.md` and `technical-considerations.md` and include their contents inline in the prompt, along with the current slice's implementation sub-task description. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. + - After defining the implementation sub-tasks for a slice, use the `Task` tool with `subagent_type: "testing-expert"`. Read `functional-spec.md` and `technical-considerations.md` and include their contents inline in the prompt, along with the current slice's implementation sub-task description. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. - `testing-expert` is a generic fallback agent provided by the awos plugin. If `/awos:hire` has installed a project-specific testing agent (e.g. `react-testing`, `python-testing`) for this tech stack, prefer that agent instead — it takes precedence by being in `.claude/agents/`. - `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. Not every slice needs all four layers — the agent applies judgment based on acceptance criteria. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. From 06973be06e60415b5582d6aa6e4911c17b13e907 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:48:35 +0300 Subject: [PATCH 28/56] fix(installer): deploy testing-expert agent to .claude/agents/ during setup --- src/config/setup-config.js | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/config/setup-config.js b/src/config/setup-config.js index cf56c8f4..2b9bc9db 100644 --- a/src/config/setup-config.js +++ b/src/config/setup-config.js @@ -58,6 +58,12 @@ const copyOperations = [ patterns: ['*'], description: 'Claude Code commands', }, + { + source: 'plugins/awos/agents', + destination: '.claude/agents', + patterns: ['testing-expert.md'], + description: 'AWOS agent definitions', + }, ]; module.exports = { From cd693f75874ddb9ec016378ff8dd22ff3d32ab3c Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:51:55 +0300 Subject: [PATCH 29/56] fix(tasks): move Verify step after test sub-tasks in Slice 2 avatar example --- commands/tasks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/commands/tasks.md b/commands/tasks.md index c0b7c795..c408c164 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -104,11 +104,11 @@ Follow this process precisely. - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` - `[ ] Sub-task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` - `[ ] Unit: avatar_url column default, valid URL stored correctly — positive cases **[Agent: testing-expert]**` - `[ ] Unit: null avatar_url, missing column, type mismatch — negative cases **[Agent: testing-expert]**` - `[ ] Integration: GET /user returns avatar_url when set — positive cases **[Agent: testing-expert]**` - `[ ] Integration: GET /user returns null when avatar_url not set — negative cases **[Agent: testing-expert]**` + - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` ## Step 4: Present Draft and Refine From 669a1b06896413e940ff0cb2e8e0f54119346e5b Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:52:46 +0300 Subject: [PATCH 30/56] fix(testing-expert): normalize positive/negative suffix format across all pyramid layers --- plugins/awos/agents/testing-expert.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/plugins/awos/agents/testing-expert.md b/plugins/awos/agents/testing-expert.md index 3337ce51..4e555b94 100644 --- a/plugins/awos/agents/testing-expert.md +++ b/plugins/awos/agents/testing-expert.md @@ -60,12 +60,12 @@ For each applicable layer, generate two sub-tasks: one for positive cases, one f ``` - [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** - [ ] Unit: [describe negative/error inputs and boundary values] — negative cases **[Agent: testing-expert]** -- [ ] Integration: [describe service interaction scenarios — positive cases] **[Agent: testing-expert]** -- [ ] Integration: [describe downstream failures, auth failures, malformed payloads — negative cases] **[Agent: testing-expert]** -- [ ] Contract: [describe schema/interface validations — positive cases] **[Agent: testing-expert]** -- [ ] Contract: [describe schema violations and malformed payload cases — negative cases] **[Agent: testing-expert]** -- [ ] E2E: [describe full user flow — positive] **[Agent: testing-expert]** -- [ ] E2E: [describe failure/unhappy path flow — negative] **[Agent: testing-expert]** +- [ ] Integration: [describe service interaction scenarios] — positive cases **[Agent: testing-expert]** +- [ ] Integration: [describe downstream failures, auth failures, malformed payloads] — negative cases **[Agent: testing-expert]** +- [ ] Contract: [describe schema/interface validations] — positive cases **[Agent: testing-expert]** +- [ ] Contract: [describe schema violations and malformed payload cases] — negative cases **[Agent: testing-expert]** +- [ ] E2E: [describe full user flow] — positive cases **[Agent: testing-expert]** +- [ ] E2E: [describe failure/unhappy path flow] — negative cases **[Agent: testing-expert]** ``` Omit layers that genuinely don't apply. Always include negative cases for every layer that is included. From 8e2493b25b3d95e33c4a1fe7eabba8762f73795d Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 18:55:11 +0300 Subject: [PATCH 31/56] fix(template): clarify test registry maintenance attribution --- templates/qa-context-template.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/templates/qa-context-template.md b/templates/qa-context-template.md index 4c554538..f80f392e 100644 --- a/templates/qa-context-template.md +++ b/templates/qa-context-template.md @@ -1,7 +1,7 @@ # Test Registry -> Auto-maintained by `testing-expert` (test writing mode) and `/awos:qa`. -> Each row is a registered test. Add rows by running `/awos:implement` (test tasks) or `/awos:qa`. +> Auto-maintained by `/awos:implement` (delegates test tasks to `testing-expert`) and `/awos:qa`. +> Each row is a registered test. > Status values: OK | MISSING | NEEDS UPDATE | DEPRECATED | HUMAN REVIEW > Layer values: unit | integration | e2e | contract From 57d08f5b528e23015270bda628f12d58fa82f341 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 20 Apr 2026 22:31:36 +0300 Subject: [PATCH 32/56] fix: delegate qa gap tests to testing-expert, fix code fence, update installer docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - commands/qa.md: Step 6 now delegates to testing-expert via Task tool instead of writing tests inline; update TODO to reflect current architecture - commands/tasks.md: fix malformed single-backtick example block → proper code fence - plugins/awos/agents/testing-expert.md: Step 7 uses generic "caller" instead of hardcoded /awos:implement; add explicit completion signal on no-gap path - src/CLAUDE.md: add plugins/awos/agents/ → .claude/agents/ row to copy table Co-Authored-By: Claude Sonnet 4.6 --- commands/qa.md | 22 ++++++---------------- commands/tasks.md | 17 +++++++++-------- plugins/awos/agents/testing-expert.md | 6 +++--- src/CLAUDE.md | 13 +++++++------ 4 files changed, 25 insertions(+), 33 deletions(-) diff --git a/commands/qa.md b/commands/qa.md index 5bbfae67..b48ee8f9 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -88,23 +88,13 @@ For each acceptance criterion in the functional spec(s) in scope: ## Step 6: Generate missing tests -For each gap identified in Step 5: +For each gap identified in Step 5, invoke the `Task` tool with `subagent_type: "testing-expert"`. Pass: -1. Write the missing test following RED validation discipline: - - Write test → confirm it FAILS for the right reason → confirm it PASSES. -2. Add `@layer`, `@spec`, `@regression` (if appropriate) annotations using appropriate comment syntax for the language. -3. Update `context/qa/list-of-tests.md` with the new entry, performing the overlap check: - - Same behavior, same layer → UPDATE existing entry instead of adding new. - - Broader test needing splitting → DEPRECATE old (annotate with `@deprecated` using appropriate comment syntax for the language), add focused replacements. - - Partial overlap → keep both, annotate relationship in Notes column. +- The gap description (layer, spec, positive/negative scope) +- The contents of `functional-spec.md` and `technical-considerations.md` for the target spec +- The relevant implementation source code -Append net-new entries using this format: - -```markdown -| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | -| -------------------- | ------------------ | ----- | ----------------- | ----------- | ------ | ----- | -| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | -``` +`testing-expert` will write the tests with RED validation, add annotations (`@layer`, `@spec`, `@regression`), and update `context/qa/list-of-tests.md`. Do not write tests inline here. ## Step 7: Run tests (with user confirmation) @@ -181,7 +171,7 @@ Report summary to user and list any flags requiring human attention. # TODO -- **E2E tests are ephemeral — no CI artifact.** The `/awos:qa` command currently runs E2E tests inline without a dedicated agent. Future work: delegate to a dedicated `e2e-tester` agent and generate rerunnable playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as a Step 6 output for E2E gaps, so they can be committed and executed in CI without agent interaction. +- **E2E tests are ephemeral — no CI artifact.** Step 6 delegates E2E gap test generation to `testing-expert`, but the output is an in-session test run, not a committed rerunnable artifact. Future work: have `testing-expert` generate playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as output for E2E gaps, so they can be committed and executed in CI without agent interaction. - **QA audit is coverage-by-inspection, not coverage-by-measurement.** Step 5 (gap analysis) reads source files and infers coverage from test file contents. It does not invoke actual coverage tooling (`vitest --coverage`, Istanbul, c8, pytest-cov, etc.). This means untested branches and dead-path gaps are invisible to the audit. Future work: in Step 2, detect available coverage reporters and, in Step 7, run the suite with coverage flags; parse the output to feed real line/branch metrics into the Coverage Summary table. diff --git a/commands/tasks.md b/commands/tasks.md index c408c164..b6a8a86e 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -75,14 +75,15 @@ Follow this process precisely. - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - Example result for a slice: - ` - [ ] **Slice 1: User authentication** - - [ ] Implement JWT token generation **[Agent: python-expert]** - - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** - - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** - - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** - - [ ] Verify: Start the server, call POST /auth with valid credentials, confirm JWT is returned **[Agent: manual-qa-expert]** - ` + ``` + - [ ] **Slice 1: User authentication** + - [ ] Implement JWT token generation **[Agent: python-expert]** + - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** + - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** + - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** + - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** + - [ ] Verify: Start the server, call POST /auth with valid credentials, confirm JWT is returned **[Agent: manual-qa-expert]** + ``` 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 7. Create a high-level checklist item and its sub-tasks with subagent assignments. 8. Repeat this process until all requirements from the specification are covered. diff --git a/plugins/awos/agents/testing-expert.md b/plugins/awos/agents/testing-expert.md index 4e555b94..7eb9e9de 100644 --- a/plugins/awos/agents/testing-expert.md +++ b/plugins/awos/agents/testing-expert.md @@ -142,10 +142,10 @@ Append only net-new tests. Format: | path/to/test_file.py | test_function_name | unit | negative | yes | OK | | ``` -### Step 7: Report completion status to `/awos:implement` +### Step 7: Report completion status to the caller -- **No gaps found:** All tests pass. Your work is done — `/awos:implement` will mark this task `[x]`. -- **Gap found (Step 5 triggered):** Do NOT signal completion. Return an incomplete/blocked status so `/awos:implement` knows NOT to mark this task `[x]`. The task stays open until the gap impl sub-task is resolved and tests pass. +- **No gaps found:** All tests pass. Return a completion signal to the caller (e.g. "All tests written and passing — task complete"). The caller will mark this task `[x]`. +- **Gap found (Step 5 triggered):** Do NOT signal completion. Return an incomplete/blocked status so the caller knows NOT to mark this task `[x]`. The task stays open until the gap impl sub-task is resolved and tests pass. --- diff --git a/src/CLAUDE.md b/src/CLAUDE.md index 202ed775..5b7e5c96 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -40,12 +40,13 @@ src/ From `config/setup-config.js`: -| Source | Destination | -| ------------------ | ------------------------ | -| `commands/` | `.awos/commands/` | -| `templates/` | `.awos/templates/` | -| `scripts/` | `.awos/scripts/` | -| `claude/commands/` | `.claude/commands/awos/` | +| Source | Destination | +| ---------------------- | ------------------------ | +| `commands/` | `.awos/commands/` | +| `templates/` | `.awos/templates/` | +| `scripts/` | `.awos/scripts/` | +| `claude/commands/` | `.claude/commands/awos/` | +| `plugins/awos/agents/` | `.claude/agents/` | **Why the difference?** From 144094ca050b81f481b966674f4584ab03a25ba0 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 21:30:46 +0300 Subject: [PATCH 33/56] feat(tasks): replace per-slice testing with Feature Testing & Regression final slice --- commands/tasks.md | 64 ++++++++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 26 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index b6a8a86e..a3bcf8a2 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -67,26 +67,37 @@ Follow this process precisely. - Tech stack identified in `technical-considerations.md` - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table - 5. **Generate paired test tasks for each slice (skip only if user explicitly opts out):** - - If the user has said they do not want tests generated, skip this step entirely and omit all test sub-tasks. - - After defining the implementation sub-tasks for a slice, use the `Task` tool with `subagent_type: "testing-expert"`. Read `functional-spec.md` and `technical-considerations.md` and include their contents inline in the prompt, along with the current slice's implementation sub-task description. Do **not** pass any implementation code — `testing-expert` determines its behavior from context: no code → returns test task descriptions; with code → writes and validates real tests. - - `testing-expert` is a generic fallback agent provided by the awos plugin. If `/awos:hire` has installed a project-specific testing agent (e.g. `react-testing`, `python-testing`) for this tech stack, prefer that agent instead — it takes precedence by being in `.claude/agents/`. - - `testing-expert` will return a list of test sub-tasks covering the applicable pyramid layers (unit, integration, e2e, contract) with both positive and negative cases. Not every slice needs all four layers — the agent applies judgment based on acceptance criteria. - - Insert those test sub-tasks as children of the same slice, after the implementation sub-tasks. - - **CRITICAL — Slice Completion Rule:** A slice parent task is only `[x]` when ALL its sub-tasks — both implementation AND all test sub-tasks — are `[x]`. This is enforced by `/awos:implement`'s existing sub-task completion logic. - - Example result for a slice: - ``` - - [ ] **Slice 1: User authentication** - - [ ] Implement JWT token generation **[Agent: python-expert]** - - [ ] Unit: token payload, expiry, signing — positive cases **[Agent: testing-expert]** - - [ ] Unit: invalid secret, expired token, malformed input — negative cases **[Agent: testing-expert]** - - [ ] Integration: valid/invalid credentials against /auth endpoint — positive cases **[Agent: testing-expert]** - - [ ] Integration: downstream failures, auth failures, malformed payloads — negative cases **[Agent: testing-expert]** - - [ ] Verify: Start the server, call POST /auth with valid credentials, confirm JWT is returned **[Agent: manual-qa-expert]** - ``` - 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. - 7. Create a high-level checklist item and its sub-tasks with subagent assignments. - 8. Repeat this process until all requirements from the specification are covered. + 5. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. + 6. Create a high-level checklist item and its sub-tasks with subagent assignments. + 7. Repeat this process until all requirements from the specification are covered. + 8. **Add the Feature Testing & Regression slice (always last):** + + After all implementation slices are defined, append one final slice. This slice is generated automatically — do not ask the user about it. + + ``` + - [ ] **Slice N: Feature Testing & Regression** + > Verifies the complete feature works end-to-end as described in functional-spec.md. + > Run AFTER all implementation slices are complete. + > **Requires `testing-expert` agent.** If it is not present in `.claude/agents/`, + > stop and run `/awos:hire` before executing this slice. + + - [ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests + that verify the entire feature as a whole — not individual slices. Cover applicable + layers (unit for pure logic, integration for service interactions, e2e for user flows). + Write tests with RED validation (must fail before implementation is confirmed done). + Annotate each test with `@spec: [spec-directory]` and `@regression` if suitable + for long-term regression. **[Agent: testing-expert]** + - [ ] Run all generated tests. All must pass. Fix any failures before proceeding. + **[Agent: testing-expert]** + - [ ] Run `/awos:regression [spec-directory-name]` to review candidates for the + regression suite, resolve duplicates, and optionally execute the full + regression suite. Pass the current spec directory name as the argument + (e.g., `/awos:regression 003-user-avatar`). Do not run without an argument — + auto-detection requires all tasks to be complete, which is not yet the case. + ``` + + Replace `N` with the actual next slice number. Do not change the wording — agents + downstream depend on this exact structure. 9. For each slice's verification sub-task, identify required MCPs/services (browser MCP, curl, database access, etc.) and note any that may be missing. - **Example of applying the rule for "User Profile Picture Upload":** @@ -98,18 +109,19 @@ Follow this process precisely. - `[ ] **Slice 1: Display a placeholder avatar on the profile page**` - `[ ] Sub-task: Add a non-functional 'ProfileAvatar' UI component that shows a static placeholder image. **[Agent: react-expert]**` - `[ ] Sub-task: Place the component on the profile page. **[Agent: react-expert]**` - - `[ ] Unit: ProfileAvatar renders placeholder when no src provided — positive cases **[Agent: testing-expert]**` - - `[ ] Unit: ProfileAvatar renders without crashing when src is undefined — negative cases **[Agent: testing-expert]**` - `[ ] Verify: Start the app, open the profile page, confirm placeholder avatar is shown **[Agent: manual-qa-expert]**` - `[ ] **Slice 2: Display the user's actual avatar if it exists**` - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` - `[ ] Sub-task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - - `[ ] Unit: avatar_url column default, valid URL stored correctly — positive cases **[Agent: testing-expert]**` - - `[ ] Unit: null avatar_url, missing column, type mismatch — negative cases **[Agent: testing-expert]**` - - `[ ] Integration: GET /user returns avatar_url when set — positive cases **[Agent: testing-expert]**` - - `[ ] Integration: GET /user returns null when avatar_url not set — negative cases **[Agent: testing-expert]**` - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` + - `[ ] **Slice 3: Feature Testing & Regression**` + - `> Verifies the complete feature works end-to-end as described in functional-spec.md.` + - `> Run AFTER all implementation slices are complete.` + - `> **Requires \`testing-expert\` agent.** If it is not present in \`.claude/agents/\`, stop and run \`/awos:hire\` before executing this slice.` + - `[ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests that verify the entire feature as a whole — not individual slices. Cover applicable layers (unit for pure logic, integration for service interactions, e2e for user flows). Write tests with RED validation (must fail before implementation is confirmed done). Annotate each test with \`@spec: [spec-directory]\` and \`@regression\` if suitable for long-term regression. **[Agent: testing-expert]**` + - `[ ] Run all generated tests. All must pass. Fix any failures before proceeding. **[Agent: testing-expert]**` + - `[ ] Run \`/awos:regression [spec-directory-name]\` to review candidates for the regression suite, resolve duplicates, and optionally execute the full regression suite. Pass the current spec directory name as the argument (e.g., \`/awos:regression 003-user-avatar\`). Do not run without an argument — auto-detection requires all tasks to be complete, which is not yet the case.` ## Step 4: Present Draft and Refine From ca7c3c5810926bd671d906bf6297eccee7f3e78d Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 22:41:12 +0300 Subject: [PATCH 34/56] =?UTF-8?q?feat(qa):=20remove=20/awos:qa=20slash=20c?= =?UTF-8?q?ommand=20=E2=80=94=20preserved=20as=20future=20plugin=20referen?= =?UTF-8?q?ce?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- claude/commands/qa.md | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 claude/commands/qa.md diff --git a/claude/commands/qa.md b/claude/commands/qa.md deleted file mode 100644 index 1752c836..00000000 --- a/claude/commands/qa.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. ---- - -Use `AskUserQuestion` tool for multiple-choice questions instead of plain text or numbered lists. - -Refer to the instructions located in this file: .awos/commands/qa.md From d05f640f8274376861182e30a17e0d948c838171 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 22:47:36 +0300 Subject: [PATCH 35/56] feat(regression): add /awos:regression command with dedup, confirmation, run, and report Co-Authored-By: Claude Sonnet 4.6 --- commands/regression.md | 217 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100644 commands/regression.md diff --git a/commands/regression.md b/commands/regression.md new file mode 100644 index 00000000..94a12b0a --- /dev/null +++ b/commands/regression.md @@ -0,0 +1,217 @@ +--- +description: Regression suite manager — promotes feature tests to regression, deduplicates, runs suite, generates report. +--- + +# ROLE + +You are a Regression Suite Manager. Your job is to review tests generated during feature development, promote the right ones into the long-term regression suite (avoiding duplicates), optionally run the suite, and produce a clear report. You never write new tests — you work with tests that already exist. + +--- + +# TASK + +After a feature's "Feature Testing & Regression" slice is complete, run this command to: +1. Extract test candidates from the current feature's `tasks.md` +2. Check them against the existing `regression-suite.md` for duplicates or extendable entries +3. Ask the user to confirm the final selection +4. Update `regression-suite.md` +5. Optionally run the regression suite +6. Generate a dated report + +--- + +# INPUTS & OUTPUTS + +- **User Prompt (Optional):** $ARGUMENTS + - Empty = auto-detect the most recently completed spec (all tasks ✅, Status Completed) + - Spec name/index = target that spec (e.g., `/awos:regression 002-user-auth`) +- **Primary Inputs:** + - `context/spec/[target-spec]/tasks.md` — source of test candidates + - `context/qa/regression-suite.md` — existing suite (create from template if missing) + - `context/spec/[target-spec]/functional-spec.md` — for context when comparing tests +- **Outputs:** + - Updated `context/qa/regression-suite.md` + - New `context/qa/regression-reports/regression-YYYY-MM-DD-[spec].md` + +--- + +# PROCESS + +## Step 1: Identify target spec + +1. Read ``. If it names a spec, use that directory. +2. If empty, scan `context/spec/*/tasks.md` files. Find the spec where: + - All tasks in the "Feature Testing & Regression" slice EXCEPT the final + "Run /awos:regression" sub-task are `[x]` + - All other (implementation) slices are fully `[x]` + - `functional-spec.md` Status is `Completed` or all tasks done + - The spec is NOT already listed in `regression-suite.md` as "fully processed" +3. If multiple candidates found, use `AskUserQuestion` to let user choose. +4. If no candidate found, stop: "No completed feature specs found ready for regression. Complete all tasks in a spec first." + +## Step 2: Extract test candidates from tasks.md + +Read `context/spec/[target-spec]/tasks.md`. Find the final slice titled "Feature Testing & Regression". + +Extract every sub-task that represents a test (lines containing `**[Agent: testing-expert]**` or similar testing agents). For each test, parse: + +- **Layer** — derived from the sub-task text prefix: "Unit:", "Integration:", "E2E:", "Contract:" +- **Behavior** — the description after the layer prefix and before "— positive/negative cases" +- **Polarity** — "positive" or "negative" (from "— positive cases" / "— negative cases") +- **File** — attempt to find the actual test file path by searching the codebase for test functions matching the behavior description. If not found, mark as "pending discovery" +- **Regression candidate** — mark all as candidates by default; user will confirm + +Build a candidate table: + +``` +| # | Layer | Behavior | Polarity | File | +|---|-------------|---------------------------------------|----------|-----------------------------| +| 1 | unit | token payload, expiry, signing | positive | tests/test_auth.py | +| 2 | unit | invalid secret, expired token | negative | tests/test_auth.py | +| 3 | integration | valid credentials against /auth | positive | tests/test_auth_integration.py | +``` + +## Step 3: Load existing regression suite and detect duplicates + +Read `context/qa/regression-suite.md`. If it does not exist, create it from `templates/regression-suite-template.md`. + +For each candidate from Step 2, compare against every existing entry in `regression-suite.md`: + +**Duplicate detection rules:** +- **Exact duplicate** — same spec, same layer, same behavior description (case-insensitive) → mark `DUPLICATE — skip` +- **Extendable** — same spec, same layer, similar behavior (≥70% word overlap) but different polarity OR slightly different scope → mark `EXTEND — merge into existing entry` +- **New** — no match found → mark `NEW — add to suite` + +Build a resolution table to show the user: + +``` +| # | Candidate Behavior | Layer | Resolution | Existing Entry (if any) | +|---|----------------------------------|-------|--------------------|------------------------------------| +| 1 | token payload, expiry, signing | unit | NEW | — | +| 2 | invalid secret, expired token | unit | EXTEND | "token validation" in 001-user-auth| +| 3 | valid credentials against /auth | intg | DUPLICATE — skip | "auth endpoint happy path" | +``` + +## Step 4: User confirmation + +Present the candidate table and resolution table to the user. Use `AskUserQuestion`: + +``` +Found N test candidates from spec [spec-name]. +- X are NEW → will be added to regression suite +- Y are EXTEND → will be merged into existing entries +- Z are DUPLICATE → will be skipped + +Do you want to proceed with this plan, or adjust it? +``` + +Options: +- **Proceed as proposed** — apply all resolutions as shown +- **Review manually** — list each NEW/EXTEND one by one and ask approve/skip/modify +- **Cancel** — exit without changes + +Wait for user confirmation before modifying any files. + +## Step 5: Update `regression-suite.md` + +Apply confirmed resolutions: + +**For NEW entries** — add under the correct spec section and layer subsection: +```markdown +## [spec-directory-name] — [Feature Title from functional-spec.md] + +### Unit +| File | Test Name | Behavior | Polarity | Status | Notes | +|------|-----------|----------|----------|--------|-------| +| tests/test_auth.py | test_token_payload | token payload, expiry, signing | positive | OK | — | + +### Integration +... + +### E2E +... +``` + +Create the spec section if it doesn't exist. Create layer subsection if it doesn't exist. + +**For EXTEND entries** — append the new polarity or scope detail to the Notes column of the existing entry. Do not create a new row. + +**For DUPLICATE entries** — skip. Do not modify anything. + +Update the header: +```markdown +**Last updated:** YYYY-MM-DD +**Total:** N tests ← recalculate: count all rows across all sections +``` + +## Step 6: Run regression suite (with user confirmation) + +1. Count all tests currently in `regression-suite.md` (all rows across all sections). +2. Use `AskUserQuestion`: + + ``` + Regression suite updated: N total tests ([spec] added M new). + + Run the regression suite now? + ``` + + Options: + - **Run full suite** — run all N tests + - **Run only new tests** — run only the M tests just added + - **Skip — I'll run manually** — exit after updating the suite file + +3. If user chooses to run: + - Detect test runner: check for `docker-compose.yml`, `Makefile` (with `test` target), `package.json` (`test` script), `pytest.ini` / `pyproject.toml`, `justfile`. + - If runner found: spin up infrastructure if needed, run the selected tests, capture output. + - If NO runner found: inform user — "No test runner detected. Tests are saved in regression-suite.md. Run them manually using your project's test command." Proceed to Step 7. + +4. Collect results per test: PASS / FAIL / ERROR / SKIPPED. + +## Step 7: Generate report + +Save to `context/qa/regression-reports/regression-YYYY-MM-DD-[spec-name].md`: + +```markdown +# Regression Report — YYYY-MM-DD + +## Feature +[spec-directory-name] — [Feature Title] + +## Suite Delta +- Added: N new tests +- Extended: M existing entries +- Skipped (duplicates): K + +## Regression Suite Status +**Total tests in suite:** N + +## Run Results +[If tests were run:] +- Suite: [Full / New tests only] +- Passed: N | Failed: M | Errors: K | Skipped: J + +### Failed Tests +| File | Test Name | Layer | Error | +|------|-----------|-------|-------| +| ... | ... | ... | ... | + +[If tests were NOT run:] +- Tests were not executed this session. Run manually with [detected command or "your project's test command"]. + +## Recommendations +- [ ] [Any failing tests that need attention] +- [ ] [Suggested follow-up if infrastructure was missing] +``` + +Announce summary to user in chat. + +--- + +# CONSTRAINTS + +- Never write new tests — only promote existing ones from tasks.md. +- Never auto-delete entries from regression-suite.md — only add or extend. +- Always ask for user confirmation before modifying regression-suite.md (Step 4). +- Always ask for user confirmation before running tests (Step 6). +- If a test file path cannot be found in the codebase, mark it "pending discovery" in the suite — do not skip it. +- Do not remove a "DUPLICATE — skip" entry from the suite even if a user later asks — instead flag for human review. From f4428112707a567b82fbd84b000fc396811295b3 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 23:08:59 +0300 Subject: [PATCH 36/56] fix(regression): fix extraction logic, fully-processed check, and constraints wording Co-Authored-By: Claude Sonnet 4.6 --- commands/regression.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/commands/regression.md b/commands/regression.md index 94a12b0a..c8ca7e44 100644 --- a/commands/regression.md +++ b/commands/regression.md @@ -45,30 +45,32 @@ After a feature's "Feature Testing & Regression" slice is complete, run this com "Run /awos:regression" sub-task are `[x]` - All other (implementation) slices are fully `[x]` - `functional-spec.md` Status is `Completed` or all tasks done - - The spec is NOT already listed in `regression-suite.md` as "fully processed" + - The spec directory name does NOT appear as a section header (`## [spec-directory-name] — ...`) in `regression-suite.md` 3. If multiple candidates found, use `AskUserQuestion` to let user choose. 4. If no candidate found, stop: "No completed feature specs found ready for regression. Complete all tasks in a spec first." -## Step 2: Extract test candidates from tasks.md +## Step 2: Extract test candidates from test files -Read `context/spec/[target-spec]/tasks.md`. Find the final slice titled "Feature Testing & Regression". +Search the codebase for test files containing both `@spec: [target-spec]` and `@regression` annotations. These annotations are written by `testing-expert` during the Feature Testing & Regression slice. -Extract every sub-task that represents a test (lines containing `**[Agent: testing-expert]**` or similar testing agents). For each test, parse: +For each annotated test function found, extract: -- **Layer** — derived from the sub-task text prefix: "Unit:", "Integration:", "E2E:", "Contract:" -- **Behavior** — the description after the layer prefix and before "— positive/negative cases" -- **Polarity** — "positive" or "negative" (from "— positive cases" / "— negative cases") -- **File** — attempt to find the actual test file path by searching the codebase for test functions matching the behavior description. If not found, mark as "pending discovery" -- **Regression candidate** — mark all as candidates by default; user will confirm +- **Layer** — from `@layer: unit|integration|e2e|contract` annotation, or infer from file path/name conventions (`test_unit_*`, `*_integration_test.*`, `*_e2e_*`, etc.) +- **Behavior** — from `@behavior:` annotation, or the test function docstring, or the test function name (converted from snake_case) +- **Polarity** — from `@polarity: positive|negative` annotation, or infer from test name suffix (`_positive`, `_negative`, `_invalid`, `_missing`, `_error`, etc.) +- **File** — the test file path (already known from the search) +- **Test Name** — the test function name + +**Fallback:** If no `@spec`/`@regression` annotations are found in any test file, fall back to reading `context/spec/[target-spec]/tasks.md`. Find the "Feature Testing & Regression" slice and list each `**[Agent: testing-expert]**` sub-task as a single candidate entry, marking Layer/Behavior/Polarity as "pending discovery". Inform the user that annotations were not found. Build a candidate table: ``` -| # | Layer | Behavior | Polarity | File | -|---|-------------|---------------------------------------|----------|-----------------------------| -| 1 | unit | token payload, expiry, signing | positive | tests/test_auth.py | -| 2 | unit | invalid secret, expired token | negative | tests/test_auth.py | -| 3 | integration | valid credentials against /auth | positive | tests/test_auth_integration.py | +| # | Layer | Behavior | Polarity | File | Test Name | +|---|-------------|---------------------------------------|----------|--------------------------------|-------------------------| +| 1 | unit | token payload, expiry, signing | positive | tests/test_auth.py | test_token_payload | +| 2 | unit | invalid secret, expired token | negative | tests/test_auth.py | test_invalid_token | +| 3 | integration | valid credentials against /auth | positive | tests/test_auth_integration.py | test_auth_happy_path | ``` ## Step 3: Load existing regression suite and detect duplicates @@ -214,4 +216,4 @@ Announce summary to user in chat. - Always ask for user confirmation before modifying regression-suite.md (Step 4). - Always ask for user confirmation before running tests (Step 6). - If a test file path cannot be found in the codebase, mark it "pending discovery" in the suite — do not skip it. -- Do not remove a "DUPLICATE — skip" entry from the suite even if a user later asks — instead flag for human review. +- Never delete existing entries from `regression-suite.md` under any circumstances — if a user requests deletion, flag it for human review instead. From 9482da2f78cd15fadd39022d69653fda0bd60364 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 23:10:33 +0300 Subject: [PATCH 37/56] feat(regression): add claude/commands wrapper for /awos:regression --- claude/commands/regression.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 claude/commands/regression.md diff --git a/claude/commands/regression.md b/claude/commands/regression.md new file mode 100644 index 00000000..34c2e5db --- /dev/null +++ b/claude/commands/regression.md @@ -0,0 +1,11 @@ +--- +description: Run /awos:regression — promote feature tests to regression suite, deduplicate, run, and report. +--- + +Run the regression suite manager for the current or specified feature spec. + +$ARGUMENTS + +Use `AskUserQuestion` tool whenever the instructions say to ask the user. + +Refer to the instructions located in this file: .awos/commands/regression.md From ad53cb0c5e6a469a3eef8d7795037e63f5e0f3cc Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 23:13:21 +0300 Subject: [PATCH 38/56] =?UTF-8?q?fix(regression):=20strip=20wrapper=20to?= =?UTF-8?q?=20standard=20pattern=20=E2=80=94=20remove=20prose=20and=20doub?= =?UTF-8?q?le-tagged=20$ARGUMENTS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- claude/commands/regression.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/claude/commands/regression.md b/claude/commands/regression.md index 34c2e5db..8a885d9a 100644 --- a/claude/commands/regression.md +++ b/claude/commands/regression.md @@ -2,10 +2,6 @@ description: Run /awos:regression — promote feature tests to regression suite, deduplicate, run, and report. --- -Run the regression suite manager for the current or specified feature spec. - -$ARGUMENTS - Use `AskUserQuestion` tool whenever the instructions say to ask the user. Refer to the instructions located in this file: .awos/commands/regression.md From 033553d9172eb228552774bed81d6000b5df978c Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Fri, 24 Apr 2026 23:14:11 +0300 Subject: [PATCH 39/56] =?UTF-8?q?fix(installer):=20remove=20testing-expert?= =?UTF-8?q?=20auto-deploy=20=E2=80=94=20now=20hired=20via=20awos-recruitme?= =?UTF-8?q?nt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- src/config/setup-config.js | 6 ------ 1 file changed, 6 deletions(-) diff --git a/src/config/setup-config.js b/src/config/setup-config.js index 2b9bc9db..cf56c8f4 100644 --- a/src/config/setup-config.js +++ b/src/config/setup-config.js @@ -58,12 +58,6 @@ const copyOperations = [ patterns: ['*'], description: 'Claude Code commands', }, - { - source: 'plugins/awos/agents', - destination: '.claude/agents', - patterns: ['testing-expert.md'], - description: 'AWOS agent definitions', - }, ]; module.exports = { From f9b67089f99044ae0ad623526252acc0ad314733 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Sat, 25 Apr 2026 00:17:50 +0300 Subject: [PATCH 40/56] =?UTF-8?q?fix(verify):=20enforce=20mandatory=20Step?= =?UTF-8?q?=203=20with=20fallback=20chain=20=E2=80=94=20prevent=20silent?= =?UTF-8?q?=20skips?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- commands/verify.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/commands/verify.md b/commands/verify.md index e3cbf533..c306bb7a 100644 --- a/commands/verify.md +++ b/commands/verify.md @@ -26,6 +26,19 @@ Verify a specification's implementation against its acceptance criteria. For eac --- +# CONSTRAINTS + +- **Step 3 is MANDATORY.** Never skip verification. Never write "the user can verify this manually" + unless you have: (a) tried all available verification tools (browser MCP, curl, shell), AND + (b) received explicit user approval to skip via AskUserQuestion. +- **Verification tool fallback order:** browser MCP → curl/shell → AskUserQuestion to skip. + If browser MCP is unavailable, try curl or shell commands. Only ask to skip as a last resort. +- **Do not assume success.** If you cannot verify, STOP and inform the user. Do not mark + acceptance criteria as `[x]` without actual verification evidence. +- **Session length does not excuse skipping.** Even in long sessions, Step 3 must run. + +--- + # PROCESS ### Step 1: Identify Target Specification @@ -47,6 +60,14 @@ For each acceptance criterion in `functional-spec.md`: 2. **If met:** Mark it `[x]` 3. **If NOT met:** Report which criterion failed and what's missing, then stop +**If browser MCP is unavailable:** +1. Try `curl` or shell commands to verify API endpoints or application state. +2. Try reading log files or database state if relevant. +3. If ALL tools fail: use `AskUserQuestion` — "I cannot verify automatically because [reason]. + Do you want to verify manually and confirm, or should I stop here?" + Options: "I verified manually — mark as done" / "Stop — I'll fix the tooling first" +4. Never proceed to mark criteria `[x]` without one of the above producing evidence. + ### Step 4: Mark as Completed If all criteria verified: From c5ba8be80c3459625417e1cbf6eab45d98538606 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Sat, 25 Apr 2026 00:22:18 +0300 Subject: [PATCH 41/56] feat(template): update regression-suite with layered format; fix stale qa-context attribution Co-Authored-By: Claude Sonnet 4.6 --- templates/qa-context-template.md | 2 +- templates/regression-suite-template.md | 36 ++++++++++++++++++++++---- 2 files changed, 32 insertions(+), 6 deletions(-) diff --git a/templates/qa-context-template.md b/templates/qa-context-template.md index f80f392e..a9c6417e 100644 --- a/templates/qa-context-template.md +++ b/templates/qa-context-template.md @@ -1,6 +1,6 @@ # Test Registry -> Auto-maintained by `/awos:implement` (delegates test tasks to `testing-expert`) and `/awos:qa`. +> Auto-maintained by `testing-expert` agent (called from the Feature Testing & Regression slice). > Each row is a registered test. > Status values: OK | MISSING | NEEDS UPDATE | DEPRECATED | HUMAN REVIEW > Layer values: unit | integration | e2e | contract diff --git a/templates/regression-suite-template.md b/templates/regression-suite-template.md index db356d2c..44cb5d5c 100644 --- a/templates/regression-suite-template.md +++ b/templates/regression-suite-template.md @@ -1,11 +1,37 @@ # Regression Suite -> Auto-maintained by `/awos:qa` Step 8. Synced from `@regression` annotations in test files. -> Run with: `/awos:qa` → choose "Regression suite only". +> Auto-maintained by `/awos:regression`. +> Run with: `/awos:regression` → choose "Run full suite" or "Run only new tests". +> Each spec section is added when that feature's regression slice is processed. +> Do NOT manually add entries — use `/awos:regression` to ensure deduplication. **Last updated:** YYYY-MM-DD **Total:** 0 tests - - - +--- + + From d58b877f2f36b20cd94b9a629315f133008f2bc0 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Sat, 25 Apr 2026 23:03:20 +0300 Subject: [PATCH 42/56] =?UTF-8?q?feat(testing-expert):=20remove=20from=20a?= =?UTF-8?q?wos=20core=20=E2=80=94=20now=20lives=20in=20awos-recruitment=20?= =?UTF-8?q?registry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 --- plugins/awos/agents/testing-expert.md | 157 -------------------------- 1 file changed, 157 deletions(-) delete mode 100644 plugins/awos/agents/testing-expert.md diff --git a/plugins/awos/agents/testing-expert.md b/plugins/awos/agents/testing-expert.md deleted file mode 100644 index 7eb9e9de..00000000 --- a/plugins/awos/agents/testing-expert.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -name: testing-expert -description: >- - QA agent for vertical slices. When called with spec files and no - implementation code, returns structured test task descriptions (planning). - When called with spec files and implementation code, writes tests with RED - validation across unit, integration, e2e, and contract layers. -tools: Read, Write, Bash, Glob, Grep ---- - -# ROLE - -You are an expert QA Engineer and Test Automation Specialist. You operate in two distinct modes determined by the context you receive. - ---- - -# MODE DETECTION - -Read the invocation context: - -- **No implementation code in context** — you received spec files and a slice/task description, but no existing source code. Your job is to return structured test task descriptions. You do NOT write test code in this mode. -- **Implementation code present in context** — you received spec files, a test task description, and implementation source code. Your job is to write, RED-validate, and run real test code. -- **Ambiguous context (cannot determine)** — ask: "Are you planning test tasks for a spec, or executing a specific test task?" then proceed accordingly. - ---- - -# TASK DESCRIPTION MODE (no implementation code in context) - -## Inputs - -- `functional-spec.md` from the target spec directory -- `technical-considerations.md` from the target spec directory -- The implementation sub-task description for the current slice - -## Process - -### Step 1: Discover frameworks - -1. Read `context/product/architecture.md` for declared testing stack per layer (unit/integration/e2e/contract). -2. If not declared, auto-detect from dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. -3. If still not determinable, note "framework TBD — will auto-detect at execution time" in the task description. - -### Step 2: Map acceptance criteria to test layers - -For the given implementation sub-task, identify which acceptance criteria it touches. For each criterion, determine which layers apply: - -- **Unit** — pure logic, no external dependencies -- **Integration** — service-to-service or DB interactions -- **E2E** — full user flow through the UI or API surface -- **Contract** — API schema/interface validation (OpenAPI, Pact, etc.) - -Not every slice needs all four layers. Apply judgment. - -### Step 3: Generate test task descriptions - -For each applicable layer, generate two sub-tasks: one for positive cases, one for negative cases. - -**Output format** (return this list to the caller for insertion into `tasks.md`): - -``` -- [ ] Unit: [describe positive behaviors] — positive cases **[Agent: testing-expert]** -- [ ] Unit: [describe negative/error inputs and boundary values] — negative cases **[Agent: testing-expert]** -- [ ] Integration: [describe service interaction scenarios] — positive cases **[Agent: testing-expert]** -- [ ] Integration: [describe downstream failures, auth failures, malformed payloads] — negative cases **[Agent: testing-expert]** -- [ ] Contract: [describe schema/interface validations] — positive cases **[Agent: testing-expert]** -- [ ] Contract: [describe schema violations and malformed payload cases] — negative cases **[Agent: testing-expert]** -- [ ] E2E: [describe full user flow] — positive cases **[Agent: testing-expert]** -- [ ] E2E: [describe failure/unhappy path flow] — negative cases **[Agent: testing-expert]** -``` - -Omit layers that genuinely don't apply. Always include negative cases for every layer that is included. - ---- - -# TEST WRITING MODE (implementation code present in context) - -## Inputs - -- Specific test task description (layer + positive/negative scope) -- `functional-spec.md` + `technical-considerations.md` for the target spec -- `context/product/architecture.md` -- The implementation code written by the preceding impl sub-agent -- Current `context/qa/list-of-tests.md` (if it exists) - -## Process - -### Step 1: Discover frameworks - -1. Read `context/product/architecture.md` for declared testing stack per layer. -2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. - -### Step 2: Plan test cases - -Map the task's acceptance criteria to concrete test cases: - -- For every positive case, define at least one negative counterpart. -- Negative cases must include: invalid inputs, boundary values, error paths, permission failures, malformed data — whichever apply to this layer. - -### Step 3: Write tests with RED validation - -Write tests following this discipline (borrowed from TDD red-green-refactor): - -1. Write one test case. -2. Run it. **Confirm it FAILS** — and that the failure message matches the missing behavior, not a syntax error. - - If it passes immediately: the test is not testing new behavior. Revise it until it fails for the right reason. -3. Proceed to the next test case. - -Annotate every test file with the following (use the appropriate comment syntax for the language: `#` for Python/Ruby/Shell, `//` for JS/TS/Go/Java, `/* */` for C/C++/C#): - -``` -# @layer: unit | integration | e2e | contract -# @spec: [spec-directory-name] -# @regression ← add only for tests that should be in the permanent regression suite -``` - -### Step 4: Confirm GREEN - -Run all tests written in this task. All must pass before continuing. - -### Step 5: Check for implementation gaps - -If tests reveal that the implementation is incomplete: - -- Do NOT modify production code. -- Report the gap by appending a note to this task's entry in `tasks.md`: - `` -- Do NOT invoke `/awos:implement` directly. Leave this task open (`[ ]`); `/awos:implement` will detect the incomplete task on its next run and create a new impl sub-task to close the gap. - -### Step 6: Update `context/qa/list-of-tests.md` - -Before appending new entries, scan the registry for existing tests covering the same behavior/AC in the same layer + spec: - -- **Same behavior, same layer** → UPDATE the existing entry instead of adding a new one. -- **Broader test that needs splitting** → DEPRECATE the old entry, add focused replacements; annotate old test file with `@deprecated` using the appropriate comment syntax for the language. -- **Partial overlap** → keep both, note the relationship in the Notes column. - -Append only net-new tests. Format: - -```markdown -| File | Test Name | Layer | Positive/Negative | @regression | Status | Notes | -| -------------------- | ------------------ | ----- | ----------------- | ----------- | ------ | ----- | -| path/to/test_file.py | test_function_name | unit | negative | yes | OK | | -``` - -### Step 7: Report completion status to the caller - -- **No gaps found:** All tests pass. Return a completion signal to the caller (e.g. "All tests written and passing — task complete"). The caller will mark this task `[x]`. -- **Gap found (Step 5 triggered):** Do NOT signal completion. Return an incomplete/blocked status so the caller knows NOT to mark this task `[x]`. The task stays open until the gap impl sub-task is resolved and tests pass. - ---- - -# CONSTRAINTS - -- Never modify production/implementation code — only test files. -- Never skip negative test cases — every included layer must have at least one negative test. -- RED validation is non-negotiable — a test that passes immediately without implementation proves nothing. -- Co-locate test files with source or follow the existing `tests/` directory convention in the project. From 9925d0fa6dfd1a39110d763e93b4b5a73074d4a4 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Sat, 25 Apr 2026 23:14:38 +0300 Subject: [PATCH 43/56] =?UTF-8?q?feat(hire):=20add=20QA=20complement=20rul?= =?UTF-8?q?e=20=E2=80=94=20auto-suggest=20testing-expert=20alongside=20tec?= =?UTF-8?q?h=20agents?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- commands/hire.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/commands/hire.md b/commands/hire.md index 89d9efcd..6308a9a5 100644 --- a/commands/hire.md +++ b/commands/hire.md @@ -89,6 +89,28 @@ Follow this process precisely. | `python-backend` | `fastapi-expert` | — | — | | `aws-infra` | `terraform-pro`, `aws-deploy` | `aws-mcp` | `aws-infra-expert` | +**QA Complement Rule:** +After searching for each primary tech agent, also search for a complementary +testing agent. Use the query: "[primary technology] testing QA acceptance". + +If `testing-expert` is returned from the registry, always include it in the +proposals table with label "(Recommended for QA coverage)" alongside the +primary tech agent — unless a more specific testing agent (e.g. `react-testing`, +`rust-tester`) is also found, in which case prefer the specific one. + +Also: if the tech stack includes any frontend framework (React, Vue, Angular, +Svelte, etc.), always include the `playwright` MCP in the proposal — it enables +browser-based E2E testing via the testing-expert agent. + +Complementary pairs reference: +- React / Vue / Angular → `testing-expert` + `playwright` MCP +- FastAPI / Django / Flask → `testing-expert` + `pytest-best-practices` skill + _(if `pytest-best-practices` not found in registry — `testing-expert` covers this)_ +- Any Python backend → `testing-expert` + `pytest-best-practices` skill + _(if `pytest-best-practices` not found in registry — `testing-expert` covers this)_ +- Any TypeScript/Node backend → `testing-expert` +- Terraform / IaC → `testing-expert` (for infra validation) + ## Step 5: Install Found Components 1. **Install Skills:** For all confirmed skills, run: From 127851cf279d1b559acf91f4f652ee3c4b0a908e Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Sat, 25 Apr 2026 23:42:21 +0300 Subject: [PATCH 44/56] fix(audit): update SDD-07 to recognize Feature Testing & Regression slice model Check item 7 now handles both AWOS 2.x (single final QA slice) and the legacy per-slice QA assignment model, preventing false WARN/FAIL on projects that use the new tasks format. Co-Authored-By: Claude Sonnet 4.6 --- .../dimensions/spec-driven-development.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/plugins/awos/skills/ai-readiness-audit/dimensions/spec-driven-development.md b/plugins/awos/skills/ai-readiness-audit/dimensions/spec-driven-development.md index 272529d9..1d73198c 100644 --- a/plugins/awos/skills/ai-readiness-audit/dimensions/spec-driven-development.md +++ b/plugins/awos/skills/ai-readiness-audit/dimensions/spec-driven-development.md @@ -117,9 +117,15 @@ Audits whether the project uses AWOS for spec-driven development. AWOS provides 4. Calculate the annotation ratio: annotated sub-tasks / total sub-tasks 5. Extract all unique agent names. Occasional `general-purpose` assignments are fine for small utility tasks (commits, running linters, config tweaks) — only flag if the majority of implementation sub-tasks use `general-purpose`. 6. Check for domain mix-ups: frontend agents assigned to backend/database tasks or vice versa. Use keywords in the sub-task description to detect domain (e.g., "migration", "database", "API endpoint" → backend; "component", "UI", "page", "styling" → frontend). - 7. Check that each slice's verification/testing sub-task is assigned to a QA/tester agent (e.g., `manual-qa-expert`, `testing-expert`, or similar) — not to the same agent that implemented the slice. -- **Pass:** Majority of sub-tasks have agent assignments with no systematic domain mix-ups and verification tasks are assigned to QA/tester agents -- **Warn:** Many sub-tasks lack annotations, OR most implementation tasks use `general-purpose`, OR verification tasks lack dedicated QA agent + 7. Check for QA agent coverage using the AWOS tasks model in use: + - **New model (AWOS 2.x):** Look for a final slice titled "Feature Testing & Regression". + Verify it has sub-tasks assigned to a QA/tester agent (e.g., `testing-expert`). + Implementation slices do NOT require per-slice QA assignments in this model. + - **Legacy model:** If no "Feature Testing & Regression" slice is found, check that each + slice has at least one verification/testing sub-task assigned to a QA/tester agent + (e.g., `manual-qa-expert`, `testing-expert`) — not to the same agent that implemented it. +- **Pass:** Majority of sub-tasks have agent assignments with no systematic domain mix-ups and verification tasks are assigned to QA/tester agents; OR (new model) a "Feature Testing & Regression" final slice is present with QA agent assignments, regardless of per-slice QA coverage +- **Warn:** Many sub-tasks lack annotations, OR most implementation tasks use `general-purpose`, OR verification tasks lack dedicated QA agent; OR (new model) "Feature Testing & Regression" slice is present but has no QA agent assignments - **Fail:** No agent annotations at all, OR systematic domain mix-ups across multiple specs - **Skip-When:** SDD-01 is FAIL (AWOS not installed), or no tasks.md files exist (covered by SDD-05) - **Severity:** medium From 1608ada4dfac96327166be538338db156bc15b12 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Thu, 7 May 2026 20:16:17 +0300 Subject: [PATCH 45/56] chore(tasks): remove regression files, comment out /awos:regression sub-task MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Regression command moves to feat/regression branch. /awos:regression sub-task in Feature Testing & Regression slice is commented out with TODO — uncomment when feat/regression merges. Co-Authored-By: Claude Sonnet 4.6 --- claude/commands/regression.md | 7 - commands/regression.md | 219 ------------------------- commands/tasks.md | 4 + templates/regression-suite-template.md | 37 ----- 4 files changed, 4 insertions(+), 263 deletions(-) delete mode 100644 claude/commands/regression.md delete mode 100644 commands/regression.md delete mode 100644 templates/regression-suite-template.md diff --git a/claude/commands/regression.md b/claude/commands/regression.md deleted file mode 100644 index 8a885d9a..00000000 --- a/claude/commands/regression.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -description: Run /awos:regression — promote feature tests to regression suite, deduplicate, run, and report. ---- - -Use `AskUserQuestion` tool whenever the instructions say to ask the user. - -Refer to the instructions located in this file: .awos/commands/regression.md diff --git a/commands/regression.md b/commands/regression.md deleted file mode 100644 index c8ca7e44..00000000 --- a/commands/regression.md +++ /dev/null @@ -1,219 +0,0 @@ ---- -description: Regression suite manager — promotes feature tests to regression, deduplicates, runs suite, generates report. ---- - -# ROLE - -You are a Regression Suite Manager. Your job is to review tests generated during feature development, promote the right ones into the long-term regression suite (avoiding duplicates), optionally run the suite, and produce a clear report. You never write new tests — you work with tests that already exist. - ---- - -# TASK - -After a feature's "Feature Testing & Regression" slice is complete, run this command to: -1. Extract test candidates from the current feature's `tasks.md` -2. Check them against the existing `regression-suite.md` for duplicates or extendable entries -3. Ask the user to confirm the final selection -4. Update `regression-suite.md` -5. Optionally run the regression suite -6. Generate a dated report - ---- - -# INPUTS & OUTPUTS - -- **User Prompt (Optional):** $ARGUMENTS - - Empty = auto-detect the most recently completed spec (all tasks ✅, Status Completed) - - Spec name/index = target that spec (e.g., `/awos:regression 002-user-auth`) -- **Primary Inputs:** - - `context/spec/[target-spec]/tasks.md` — source of test candidates - - `context/qa/regression-suite.md` — existing suite (create from template if missing) - - `context/spec/[target-spec]/functional-spec.md` — for context when comparing tests -- **Outputs:** - - Updated `context/qa/regression-suite.md` - - New `context/qa/regression-reports/regression-YYYY-MM-DD-[spec].md` - ---- - -# PROCESS - -## Step 1: Identify target spec - -1. Read ``. If it names a spec, use that directory. -2. If empty, scan `context/spec/*/tasks.md` files. Find the spec where: - - All tasks in the "Feature Testing & Regression" slice EXCEPT the final - "Run /awos:regression" sub-task are `[x]` - - All other (implementation) slices are fully `[x]` - - `functional-spec.md` Status is `Completed` or all tasks done - - The spec directory name does NOT appear as a section header (`## [spec-directory-name] — ...`) in `regression-suite.md` -3. If multiple candidates found, use `AskUserQuestion` to let user choose. -4. If no candidate found, stop: "No completed feature specs found ready for regression. Complete all tasks in a spec first." - -## Step 2: Extract test candidates from test files - -Search the codebase for test files containing both `@spec: [target-spec]` and `@regression` annotations. These annotations are written by `testing-expert` during the Feature Testing & Regression slice. - -For each annotated test function found, extract: - -- **Layer** — from `@layer: unit|integration|e2e|contract` annotation, or infer from file path/name conventions (`test_unit_*`, `*_integration_test.*`, `*_e2e_*`, etc.) -- **Behavior** — from `@behavior:` annotation, or the test function docstring, or the test function name (converted from snake_case) -- **Polarity** — from `@polarity: positive|negative` annotation, or infer from test name suffix (`_positive`, `_negative`, `_invalid`, `_missing`, `_error`, etc.) -- **File** — the test file path (already known from the search) -- **Test Name** — the test function name - -**Fallback:** If no `@spec`/`@regression` annotations are found in any test file, fall back to reading `context/spec/[target-spec]/tasks.md`. Find the "Feature Testing & Regression" slice and list each `**[Agent: testing-expert]**` sub-task as a single candidate entry, marking Layer/Behavior/Polarity as "pending discovery". Inform the user that annotations were not found. - -Build a candidate table: - -``` -| # | Layer | Behavior | Polarity | File | Test Name | -|---|-------------|---------------------------------------|----------|--------------------------------|-------------------------| -| 1 | unit | token payload, expiry, signing | positive | tests/test_auth.py | test_token_payload | -| 2 | unit | invalid secret, expired token | negative | tests/test_auth.py | test_invalid_token | -| 3 | integration | valid credentials against /auth | positive | tests/test_auth_integration.py | test_auth_happy_path | -``` - -## Step 3: Load existing regression suite and detect duplicates - -Read `context/qa/regression-suite.md`. If it does not exist, create it from `templates/regression-suite-template.md`. - -For each candidate from Step 2, compare against every existing entry in `regression-suite.md`: - -**Duplicate detection rules:** -- **Exact duplicate** — same spec, same layer, same behavior description (case-insensitive) → mark `DUPLICATE — skip` -- **Extendable** — same spec, same layer, similar behavior (≥70% word overlap) but different polarity OR slightly different scope → mark `EXTEND — merge into existing entry` -- **New** — no match found → mark `NEW — add to suite` - -Build a resolution table to show the user: - -``` -| # | Candidate Behavior | Layer | Resolution | Existing Entry (if any) | -|---|----------------------------------|-------|--------------------|------------------------------------| -| 1 | token payload, expiry, signing | unit | NEW | — | -| 2 | invalid secret, expired token | unit | EXTEND | "token validation" in 001-user-auth| -| 3 | valid credentials against /auth | intg | DUPLICATE — skip | "auth endpoint happy path" | -``` - -## Step 4: User confirmation - -Present the candidate table and resolution table to the user. Use `AskUserQuestion`: - -``` -Found N test candidates from spec [spec-name]. -- X are NEW → will be added to regression suite -- Y are EXTEND → will be merged into existing entries -- Z are DUPLICATE → will be skipped - -Do you want to proceed with this plan, or adjust it? -``` - -Options: -- **Proceed as proposed** — apply all resolutions as shown -- **Review manually** — list each NEW/EXTEND one by one and ask approve/skip/modify -- **Cancel** — exit without changes - -Wait for user confirmation before modifying any files. - -## Step 5: Update `regression-suite.md` - -Apply confirmed resolutions: - -**For NEW entries** — add under the correct spec section and layer subsection: -```markdown -## [spec-directory-name] — [Feature Title from functional-spec.md] - -### Unit -| File | Test Name | Behavior | Polarity | Status | Notes | -|------|-----------|----------|----------|--------|-------| -| tests/test_auth.py | test_token_payload | token payload, expiry, signing | positive | OK | — | - -### Integration -... - -### E2E -... -``` - -Create the spec section if it doesn't exist. Create layer subsection if it doesn't exist. - -**For EXTEND entries** — append the new polarity or scope detail to the Notes column of the existing entry. Do not create a new row. - -**For DUPLICATE entries** — skip. Do not modify anything. - -Update the header: -```markdown -**Last updated:** YYYY-MM-DD -**Total:** N tests ← recalculate: count all rows across all sections -``` - -## Step 6: Run regression suite (with user confirmation) - -1. Count all tests currently in `regression-suite.md` (all rows across all sections). -2. Use `AskUserQuestion`: - - ``` - Regression suite updated: N total tests ([spec] added M new). - - Run the regression suite now? - ``` - - Options: - - **Run full suite** — run all N tests - - **Run only new tests** — run only the M tests just added - - **Skip — I'll run manually** — exit after updating the suite file - -3. If user chooses to run: - - Detect test runner: check for `docker-compose.yml`, `Makefile` (with `test` target), `package.json` (`test` script), `pytest.ini` / `pyproject.toml`, `justfile`. - - If runner found: spin up infrastructure if needed, run the selected tests, capture output. - - If NO runner found: inform user — "No test runner detected. Tests are saved in regression-suite.md. Run them manually using your project's test command." Proceed to Step 7. - -4. Collect results per test: PASS / FAIL / ERROR / SKIPPED. - -## Step 7: Generate report - -Save to `context/qa/regression-reports/regression-YYYY-MM-DD-[spec-name].md`: - -```markdown -# Regression Report — YYYY-MM-DD - -## Feature -[spec-directory-name] — [Feature Title] - -## Suite Delta -- Added: N new tests -- Extended: M existing entries -- Skipped (duplicates): K - -## Regression Suite Status -**Total tests in suite:** N - -## Run Results -[If tests were run:] -- Suite: [Full / New tests only] -- Passed: N | Failed: M | Errors: K | Skipped: J - -### Failed Tests -| File | Test Name | Layer | Error | -|------|-----------|-------|-------| -| ... | ... | ... | ... | - -[If tests were NOT run:] -- Tests were not executed this session. Run manually with [detected command or "your project's test command"]. - -## Recommendations -- [ ] [Any failing tests that need attention] -- [ ] [Suggested follow-up if infrastructure was missing] -``` - -Announce summary to user in chat. - ---- - -# CONSTRAINTS - -- Never write new tests — only promote existing ones from tasks.md. -- Never auto-delete entries from regression-suite.md — only add or extend. -- Always ask for user confirmation before modifying regression-suite.md (Step 4). -- Always ask for user confirmation before running tests (Step 6). -- If a test file path cannot be found in the codebase, mark it "pending discovery" in the suite — do not skip it. -- Never delete existing entries from `regression-suite.md` under any circumstances — if a user requests deletion, flag it for human review instead. diff --git a/commands/tasks.md b/commands/tasks.md index a3bcf8a2..f0e485be 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -89,11 +89,13 @@ Follow this process precisely. for long-term regression. **[Agent: testing-expert]** - [ ] Run all generated tests. All must pass. Fix any failures before proceeding. **[Agent: testing-expert]** + ``` Replace `N` with the actual next slice number. Do not change the wording — agents @@ -121,7 +123,9 @@ Follow this process precisely. - `> **Requires \`testing-expert\` agent.** If it is not present in \`.claude/agents/\`, stop and run \`/awos:hire\` before executing this slice.` - `[ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests that verify the entire feature as a whole — not individual slices. Cover applicable layers (unit for pure logic, integration for service interactions, e2e for user flows). Write tests with RED validation (must fail before implementation is confirmed done). Annotate each test with \`@spec: [spec-directory]\` and \`@regression\` if suitable for long-term regression. **[Agent: testing-expert]**` - `[ ] Run all generated tests. All must pass. Fix any failures before proceeding. **[Agent: testing-expert]**` + ## Step 4: Present Draft and Refine diff --git a/templates/regression-suite-template.md b/templates/regression-suite-template.md deleted file mode 100644 index 44cb5d5c..00000000 --- a/templates/regression-suite-template.md +++ /dev/null @@ -1,37 +0,0 @@ -# Regression Suite - -> Auto-maintained by `/awos:regression`. -> Run with: `/awos:regression` → choose "Run full suite" or "Run only new tests". -> Each spec section is added when that feature's regression slice is processed. -> Do NOT manually add entries — use `/awos:regression` to ensure deduplication. - -**Last updated:** YYYY-MM-DD -**Total:** 0 tests - ---- - - From 019e678f4b255cb93a869004cab9c81ee82478f5 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Thu, 7 May 2026 20:29:02 +0300 Subject: [PATCH 46/56] feat(tasks): add artifact cleanup sub-task after each implementation slice MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After a slice's verification, agents now delete temporary artifacts (screenshots, videos, e2e scripts) generated by e2e-tester or browser MCP. The Feature Testing & Regression slice is explicitly excluded — its artifacts are retained for the regression suite. Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index f0e485be..c5003b36 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -53,6 +53,7 @@ Follow this process precisely. - You must **check and require all needed MCPs, services, and dependencies** for testing. If something is missing, instruct the user to install it. - If a slice **cannot be tested**, explain why and **get user approval** before proceeding. - A slice **is not complete** unless it is tested or explicitly approved to skip testing. + - After a slice is verified and marked complete, **delete all temporary artifacts** generated during that slice's verification — screenshots, recorded videos, generated e2e test scripts, and any other ephemeral files produced by the e2e-tester or browser MCP. Exception: do **not** delete artifacts from the **Feature Testing & Regression** slice — those are intentionally kept for the regression suite. - **Your Thought Process for Generating Tasks:** 1. First, identify the absolute smallest piece of user-visible value from the spec. This is your **Slice 1**. @@ -67,10 +68,15 @@ Follow this process precisely. - Tech stack identified in `technical-considerations.md` - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table - 5. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. - 6. Create a high-level checklist item and its sub-tasks with subagent assignments. - 7. Repeat this process until all requirements from the specification are covered. - 8. **Add the Feature Testing & Regression slice (always last):** + 5. After the verification sub-task, add a cleanup sub-task as the last item of the slice: + ``` + - [ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]** + ``` + Skip this sub-task for the **Feature Testing & Regression** slice — its artifacts are kept intentionally. + 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. + 7. Create a high-level checklist item and its sub-tasks with subagent assignments. + 8. Repeat this process until all requirements from the specification are covered. + 9. **Add the Feature Testing & Regression slice (always last):** After all implementation slices are defined, append one final slice. This slice is generated automatically — do not ask the user about it. @@ -100,7 +106,7 @@ Follow this process precisely. Replace `N` with the actual next slice number. Do not change the wording — agents downstream depend on this exact structure. - 9. For each slice's verification sub-task, identify required MCPs/services (browser MCP, curl, database access, etc.) and note any that may be missing. + 10. For each slice's verification sub-task, identify required MCPs/services (browser MCP, curl, database access, etc.) and note any that may be missing. - **Example of applying the rule for "User Profile Picture Upload":** - **Bad, Horizontal Tasks (DO NOT DO THIS):** @@ -112,11 +118,13 @@ Follow this process precisely. - `[ ] Sub-task: Add a non-functional 'ProfileAvatar' UI component that shows a static placeholder image. **[Agent: react-expert]**` - `[ ] Sub-task: Place the component on the profile page. **[Agent: react-expert]**` - `[ ] Verify: Start the app, open the profile page, confirm placeholder avatar is shown **[Agent: manual-qa-expert]**` + - `[ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]**` - `[ ] **Slice 2: Display the user's actual avatar if it exists**` - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` - `[ ] Sub-task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` + - `[ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]**` - `[ ] **Slice 3: Feature Testing & Regression**` - `> Verifies the complete feature works end-to-end as described in functional-spec.md.` - `> Run AFTER all implementation slices are complete.` From 65157bdd430e923e1c574a4001ae954059110bfc Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Thu, 7 May 2026 21:27:11 +0300 Subject: [PATCH 47/56] chore: fix prettier formatting Co-Authored-By: Claude Sonnet 4.6 --- commands/hire.md | 1 + commands/tasks.md | 5 +++-- commands/verify.md | 1 + 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/commands/hire.md b/commands/hire.md index 6308a9a5..792ca944 100644 --- a/commands/hire.md +++ b/commands/hire.md @@ -103,6 +103,7 @@ Svelte, etc.), always include the `playwright` MCP in the proposal — it enable browser-based E2E testing via the testing-expert agent. Complementary pairs reference: + - React / Vue / Angular → `testing-expert` + `playwright` MCP - FastAPI / Django / Flask → `testing-expert` + `pytest-best-practices` skill _(if `pytest-best-practices` not found in registry — `testing-expert` covers this)_ diff --git a/commands/tasks.md b/commands/tasks.md index c5003b36..6526e7f5 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -106,6 +106,7 @@ Follow this process precisely. Replace `N` with the actual next slice number. Do not change the wording — agents downstream depend on this exact structure. + 10. For each slice's verification sub-task, identify required MCPs/services (browser MCP, curl, database access, etc.) and note any that may be missing. - **Example of applying the rule for "User Profile Picture Upload":** @@ -128,12 +129,12 @@ Follow this process precisely. - `[ ] **Slice 3: Feature Testing & Regression**` - `> Verifies the complete feature works end-to-end as described in functional-spec.md.` - `> Run AFTER all implementation slices are complete.` - - `> **Requires \`testing-expert\` agent.** If it is not present in \`.claude/agents/\`, stop and run \`/awos:hire\` before executing this slice.` + - `> **Requires \`testing-expert\` agent.\*\* If it is not present in \`.claude/agents/\`, stop and run \`/awos:hire\` before executing this slice.` - `[ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests that verify the entire feature as a whole — not individual slices. Cover applicable layers (unit for pure logic, integration for service interactions, e2e for user flows). Write tests with RED validation (must fail before implementation is confirmed done). Annotate each test with \`@spec: [spec-directory]\` and \`@regression\` if suitable for long-term regression. **[Agent: testing-expert]**` - `[ ] Run all generated tests. All must pass. Fix any failures before proceeding. **[Agent: testing-expert]**` + --> ## Step 4: Present Draft and Refine diff --git a/commands/verify.md b/commands/verify.md index c306bb7a..b7594284 100644 --- a/commands/verify.md +++ b/commands/verify.md @@ -61,6 +61,7 @@ For each acceptance criterion in `functional-spec.md`: 3. **If NOT met:** Report which criterion failed and what's missing, then stop **If browser MCP is unavailable:** + 1. Try `curl` or shell commands to verify API endpoints or application state. 2. Try reading log files or database state if relevant. 3. If ALL tools fail: use `AskUserQuestion` — "I cannot verify automatically because [reason]. From 2ef8817ff23dd440f2c956fdc832dd4cd04cfead Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 11 May 2026 11:17:10 +0300 Subject: [PATCH 48/56] fix: address CodeRabbit review comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add language tags to fenced code blocks in tasks.md and qa.md (MD040) - Add plugins/awos/agents → .claude/agents copy operation to setup-config.js to match src/CLAUDE.md documentation Co-Authored-By: Claude Sonnet 4.6 --- commands/qa.md | 2 +- commands/tasks.md | 4 ++-- src/config/setup-config.js | 6 ++++++ 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/commands/qa.md b/commands/qa.md index b48ee8f9..0a3cc32a 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -99,7 +99,7 @@ For each gap identified in Step 5, invoke the `Task` tool with `subagent_type: " ## Step 7: Run tests (with user confirmation) 1. Count all tests in scope. Use `AskUserQuestion` with the following question and options: - ``` + ```text Ready to run N tests across X specs [unit: A, integration: B, e2e: C, contract: D] Regression suite: M tests tagged @regression ``` diff --git a/commands/tasks.md b/commands/tasks.md index 6526e7f5..8997adb1 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -69,7 +69,7 @@ Follow this process precisely. - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table 5. After the verification sub-task, add a cleanup sub-task as the last item of the slice: - ``` + ```md - [ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]** ``` Skip this sub-task for the **Feature Testing & Regression** slice — its artifacts are kept intentionally. @@ -80,7 +80,7 @@ Follow this process precisely. After all implementation slices are defined, append one final slice. This slice is generated automatically — do not ask the user about it. - ``` + ```md - [ ] **Slice N: Feature Testing & Regression** > Verifies the complete feature works end-to-end as described in functional-spec.md. > Run AFTER all implementation slices are complete. diff --git a/src/config/setup-config.js b/src/config/setup-config.js index cf56c8f4..4de809dd 100644 --- a/src/config/setup-config.js +++ b/src/config/setup-config.js @@ -58,6 +58,12 @@ const copyOperations = [ patterns: ['*'], description: 'Claude Code commands', }, + { + source: 'plugins/awos/agents', + destination: '.claude/agents', + patterns: ['*'], + description: 'AWOS agents', + }, ]; module.exports = { From d0b1f791b9c34403f8340cf3b88c18642ec37926 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 11 May 2026 11:20:17 +0300 Subject: [PATCH 49/56] chore: fix prettier formatting Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index 8997adb1..21e52698 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -82,11 +82,11 @@ Follow this process precisely. ```md - [ ] **Slice N: Feature Testing & Regression** + > Verifies the complete feature works end-to-end as described in functional-spec.md. > Run AFTER all implementation slices are complete. > **Requires `testing-expert` agent.** If it is not present in `.claude/agents/`, > stop and run `/awos:hire` before executing this slice. - - [ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests that verify the entire feature as a whole — not individual slices. Cover applicable layers (unit for pure logic, integration for service interactions, e2e for user flows). @@ -94,14 +94,14 @@ Follow this process precisely. Annotate each test with `@spec: [spec-directory]` and `@regression` if suitable for long-term regression. **[Agent: testing-expert]** - [ ] Run all generated tests. All must pass. Fix any failures before proceeding. - **[Agent: testing-expert]** + **[Agent: testing-expert]** + --> ``` Replace `N` with the actual next slice number. Do not change the wording — agents From 191a9b26d274aceb7e88efaf2f6031c3a338e5ac Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 25 May 2026 17:00:23 +0300 Subject: [PATCH 50/56] fix: address review comments on hire.md and qa.md - hire.md: restructure complementary pairs to search-first pattern with testing-expert as fallback; remove duplicate Python entry; replace playwright MCP with playwright CLI - qa.md: update description to "QA health check"; add user confirmation for full-scope audit; make architecture.md required with warning; rewrite Step 3 to reflect list-of-tests.md is maintained by testing-expert, not created by /awos:qa; implement risk-based gap analysis in Step 5 (two-pass: project-level + per-AC); implement coordinator pattern in Step 6 (specialist agent writes, testing-expert validates and updates registry); add guard for missing functional-spec; remove regression suite steps (moved to PR #114); add staleness and delta-coverage notes to TODO Co-Authored-By: Claude Sonnet 4.6 --- commands/hire.md | 16 ++++++------ commands/qa.md | 66 +++++++++++++++++++++++++++++++++++++----------- 2 files changed, 59 insertions(+), 23 deletions(-) diff --git a/commands/hire.md b/commands/hire.md index 792ca944..dc1997cd 100644 --- a/commands/hire.md +++ b/commands/hire.md @@ -103,14 +103,14 @@ Svelte, etc.), always include the `playwright` MCP in the proposal — it enable browser-based E2E testing via the testing-expert agent. Complementary pairs reference: - -- React / Vue / Angular → `testing-expert` + `playwright` MCP -- FastAPI / Django / Flask → `testing-expert` + `pytest-best-practices` skill - _(if `pytest-best-practices` not found in registry — `testing-expert` covers this)_ -- Any Python backend → `testing-expert` + `pytest-best-practices` skill - _(if `pytest-best-practices` not found in registry — `testing-expert` covers this)_ -- Any TypeScript/Node backend → `testing-expert` -- Terraform / IaC → `testing-expert` (for infra validation) +For each stack, search the registry for a technology-specific testing agent. +If none found, fall back to `testing-expert`. + +- React / Vue / Angular → + `playwright` CLI for E2E +- FastAPI / Django / Flask → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ +- Other Python backends → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ +- TypeScript / Node backend → (no additional skill needed) +- Terraform / IaC → (for infra validation) ## Step 5: Install Found Components diff --git a/commands/qa.md b/commands/qa.md index 0a3cc32a..4802afd9 100644 --- a/commands/qa.md +++ b/commands/qa.md @@ -1,5 +1,5 @@ --- -description: Optional full-audit QA command — coverage analysis, gap detection, regression suite management. +description: QA health check — inspects existing tests against spec acceptance criteria, detects coverage gaps by applicable pyramid layer, generates missing tests via specialist agents, and produces a structured report. --- # ROLE @@ -37,17 +37,28 @@ Perform a full QA audit for the target spec(s). Check existing tests for health ## Step 1: Identify scope 1. Read ``. If it names a spec, target that directory only. -2. If empty, target all spec directories under `context/spec/`. +2. If empty, list all spec directories under `context/spec/` that contain + `functional-spec.md`. Announce the list and ask: "Found N specs for a full + audit: [list]. This will scan all test files and may generate missing tests. + Proceed with full audit, or specify a single spec to limit scope?" + Wait for user confirmation before proceeding. 3. Announce: "Running QA audit for: [scope]." ## Step 2: Discover frameworks 1. Read `context/product/architecture.md` for declared testing stack per layer. + - If absent: warn — "architecture.md not found. Without it, gap analysis may + falsely flag intentionally excluded layers. Consider running `/awos:arch` + first. Continue anyway? (y/n)" + - Wait for user confirmation before proceeding without it. 2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. ## Step 3: Load test registry -Read `context/qa/list-of-tests.md`. If it does not exist, create it from the template at `.awos/templates/qa-context-template.md` (or create an empty registry if template is absent). +Read `context/qa/list-of-tests.md` if it exists. This file is auto-maintained by +`testing-expert` during the Feature Testing & Regression slice of `/awos:tasks` — it is +absent until at least one spec has completed that slice. If absent, proceed without the +registered test list; Step 4 will scan live test files directly. ## Step 4: Audit existing tests @@ -78,23 +89,46 @@ For each registered test in scope: ## Step 5: Gap analysis -For each acceptance criterion in the functional spec(s) in scope: +**Before checking any AC, determine applicable layers in two passes:** -1. Check which pyramid layers have coverage (unit / integration / e2e / contract). -2. Check whether each covered layer has a negative test counterpart. -3. Record gaps: - - Layer with no tests at all → `MISSING LAYER` - - Layer with only positive tests → `MISSING NEGATIVE` +1. **Project-level gate:** Read `architecture.md` (or use auto-detected stack from Step 2) + to identify which layers the project actively supports. Exclude unsupported layers from + all gap checks. Example: if `architecture.md` documents "no e2e infrastructure" → + never report `MISSING LAYER: e2e`. + +2. **Per-AC gate:** For each acceptance criterion, determine applicable layers based on + its nature: + - User-facing flow / UI interaction → e2e applicable (if supported by project) + - Service boundary / external integration → integration applicable + - Pure business logic / utility → unit applicable + - Public API contract → contract applicable (if supported by project) + +3. Check coverage only within determined applicable layers: + - Applicable layer with no tests → `MISSING LAYER` + - Applicable layer with only positive tests → `MISSING NEGATIVE` ## Step 6: Generate missing tests -For each gap identified in Step 5, invoke the `Task` tool with `subagent_type: "testing-expert"`. Pass: +Before invoking any agent: verify that `functional-spec.md` exists for the target spec. +If missing — skip gap generation for that spec and log in the audit report: +`SPEC INCOMPLETE: [spec-dir] — functional-spec.md missing, gap analysis skipped.` -- The gap description (layer, spec, positive/negative scope) -- The contents of `functional-spec.md` and `technical-considerations.md` for the target spec -- The relevant implementation source code +For each gap identified in Step 5: -`testing-expert` will write the tests with RED validation, add annotations (`@layer`, `@spec`, `@regression`), and update `context/qa/list-of-tests.md`. Do not write tests inline here. +1. Check `.claude/agents/` for a technology-specific testing agent suited to the gap + (e.g. a layer-specific or stack-specific agent). +2. If found — invoke it via the `Task` tool to **write the test code only**. Pass: + - The gap description (layer, spec, positive/negative scope) + - The relevant implementation source code +3. Then invoke `testing-expert` as **coordinator**: pass it the written test and ask it to + validate and apply annotations (`@layer`, `@spec`, `@regression`), run RED validation, + and update `context/qa/list-of-tests.md`. +4. If no specialist found — invoke `testing-expert` directly to handle everything. Pass: + - The gap description (layer, spec, positive/negative scope) + - The contents of `functional-spec.md` and `technical-considerations.md` for the target spec + - The relevant implementation source code + +Do not write tests inline here. ## Step 7: Run tests (with user confirmation) @@ -171,11 +205,13 @@ Report summary to user and list any flags requiring human attention. # TODO +- **Test registry (`list-of-tests.md`) has staleness risk.** The central registry is a practical choice — one file for the agent to read vs. scanning every test file on each run. Staleness is mitigated by the existence check in Step 4. Future work: generate the registry automatically from inline annotations (`@spec`, `@regression`) in test files, making them the source of truth. + - **E2E tests are ephemeral — no CI artifact.** Step 6 delegates E2E gap test generation to `testing-expert`, but the output is an in-session test run, not a committed rerunnable artifact. Future work: have `testing-expert` generate playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as output for E2E gaps, so they can be committed and executed in CI without agent interaction. - **QA audit is coverage-by-inspection, not coverage-by-measurement.** Step 5 (gap analysis) reads source files and infers coverage from test file contents. It does not invoke actual coverage tooling (`vitest --coverage`, Istanbul, c8, pytest-cov, etc.). This means untested branches and dead-path gaps are invisible to the audit. Future work: in Step 2, detect available coverage reporters and, in Step 7, run the suite with coverage flags; parse the output to feed real line/branch metrics into the Coverage Summary table. -- **Audit reports are snapshots with no regression baseline or enforcement.** Each `/awos:qa` run produces a dated report, but there is no mechanism to diff successive reports, track coverage trend over time, or fail a build when coverage drops below a threshold. Future work: add a `context/qa/coverage-baseline.md` file that stores the last known layer coverage counts; at the end of Step 9, compare current counts against the baseline and surface regressions explicitly in the Flags section. +- **Audit reports are snapshots with no regression baseline or enforcement.** Each `/awos:qa` run produces a dated report, but there is no mechanism to diff successive reports, track coverage trend over time, or fail a build when coverage drops below a threshold. Future work: add a `context/qa/coverage-baseline.md` file that stores the last known layer coverage counts; at the end of Step 9, compare current counts against the baseline and surface drops explicitly in the Flags section. Goal is delta detection (did coverage drop?), not enforcing absolute thresholds — absolute numbers are meaningless in isolation. --- From 5245b664b991086386b64094325178f031e8d45b Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 25 May 2026 18:43:17 +0300 Subject: [PATCH 51/56] remove: delete commands/qa.md The command had no active Claude Code wrapper and its functionality is covered by two dedicated commands: - testing-expert agent (proactive test writing via /awos:tasks) - /awos:regression (regression suite management, PR #114) Retroactive staleness/gap auditing can be revisited as a plugin if needed in the future. Co-Authored-By: Claude Sonnet 4.6 --- commands/qa.md | 224 ------------------------------------------------- 1 file changed, 224 deletions(-) delete mode 100644 commands/qa.md diff --git a/commands/qa.md b/commands/qa.md deleted file mode 100644 index 4802afd9..00000000 --- a/commands/qa.md +++ /dev/null @@ -1,224 +0,0 @@ ---- -description: QA health check — inspects existing tests against spec acceptance criteria, detects coverage gaps by applicable pyramid layer, generates missing tests via specialist agents, and produces a structured report. ---- - -# ROLE - -You are a senior QA Architect running a full audit of the test suite for one or all specifications. You analyze existing tests, identify coverage gaps, generate missing tests, manage the regression suite, and produce a structured audit report. You do not modify production code. - ---- - -# TASK - -Perform a full QA audit for the target spec(s). Check existing tests for health (missing, stale, deprecated), identify gaps against spec acceptance criteria, generate missing tests, offer to run the suite, and produce an audit report in `context/qa/audit-reports/`. - ---- - -# INPUTS & OUTPUTS - -- **User Prompt (Optional):** $ARGUMENTS - - Empty = audit all specs - - Spec name/index = audit that spec only (e.g., `/awos:qa 001-user-auth`) -- **Primary Context:** - - `context/qa/list-of-tests.md` — global test registry - - `context/qa/regression-suite.md` — regression-flagged tests - - `context/product/architecture.md` — testing framework declarations - - All `functional-spec.md` files in scope - - Live codebase + existing test files -- **Output Files:** - - Updated `context/qa/list-of-tests.md` - - Updated `context/qa/regression-suite.md` - - New `context/qa/audit-reports/qa-report-YYYY-MM-DD.md` - ---- - -# PROCESS - -## Step 1: Identify scope - -1. Read ``. If it names a spec, target that directory only. -2. If empty, list all spec directories under `context/spec/` that contain - `functional-spec.md`. Announce the list and ask: "Found N specs for a full - audit: [list]. This will scan all test files and may generate missing tests. - Proceed with full audit, or specify a single spec to limit scope?" - Wait for user confirmation before proceeding. -3. Announce: "Running QA audit for: [scope]." - -## Step 2: Discover frameworks - -1. Read `context/product/architecture.md` for declared testing stack per layer. - - If absent: warn — "architecture.md not found. Without it, gap analysis may - falsely flag intentionally excluded layers. Consider running `/awos:arch` - first. Continue anyway? (y/n)" - - Wait for user confirmation before proceeding without it. -2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. - -## Step 3: Load test registry - -Read `context/qa/list-of-tests.md` if it exists. This file is auto-maintained by -`testing-expert` during the Feature Testing & Regression slice of `/awos:tasks` — it is -absent until at least one spec has completed that slice. If absent, proceed without the -registered test list; Step 4 will scan live test files directly. - -## Step 4: Audit existing tests - -For each registered test in scope: - -**Existence check:** - -- Does the test file still exist in the codebase? If not → flag `MISSING`. - -**Spec linkage check:** - -- Does the `@spec` annotation reference a spec directory that still exists? - - YES → OK. - - Spec directory deleted → - - Search all active `functional-spec.md` files for matching behavior/acceptance criterion. - - Match found → re-link `@spec` to the new spec, mark OK. - - No match → flag `HUMAN REVIEW` (do NOT auto-deprecate; may be intentional regression coverage). - -**Staleness check:** - -- Does the test logic match the current implementation? - - Read the test file and the relevant implementation code. - - If the implementation has changed in ways that invalidate the test → flag `NEEDS UPDATE` and generate a diff suggestion. Do NOT auto-modify the test. - -**Regression tag:** - -- If tagged `@regression` → carry forward to the regression-suite.md sync in Step 8. - -## Step 5: Gap analysis - -**Before checking any AC, determine applicable layers in two passes:** - -1. **Project-level gate:** Read `architecture.md` (or use auto-detected stack from Step 2) - to identify which layers the project actively supports. Exclude unsupported layers from - all gap checks. Example: if `architecture.md` documents "no e2e infrastructure" → - never report `MISSING LAYER: e2e`. - -2. **Per-AC gate:** For each acceptance criterion, determine applicable layers based on - its nature: - - User-facing flow / UI interaction → e2e applicable (if supported by project) - - Service boundary / external integration → integration applicable - - Pure business logic / utility → unit applicable - - Public API contract → contract applicable (if supported by project) - -3. Check coverage only within determined applicable layers: - - Applicable layer with no tests → `MISSING LAYER` - - Applicable layer with only positive tests → `MISSING NEGATIVE` - -## Step 6: Generate missing tests - -Before invoking any agent: verify that `functional-spec.md` exists for the target spec. -If missing — skip gap generation for that spec and log in the audit report: -`SPEC INCOMPLETE: [spec-dir] — functional-spec.md missing, gap analysis skipped.` - -For each gap identified in Step 5: - -1. Check `.claude/agents/` for a technology-specific testing agent suited to the gap - (e.g. a layer-specific or stack-specific agent). -2. If found — invoke it via the `Task` tool to **write the test code only**. Pass: - - The gap description (layer, spec, positive/negative scope) - - The relevant implementation source code -3. Then invoke `testing-expert` as **coordinator**: pass it the written test and ask it to - validate and apply annotations (`@layer`, `@spec`, `@regression`), run RED validation, - and update `context/qa/list-of-tests.md`. -4. If no specialist found — invoke `testing-expert` directly to handle everything. Pass: - - The gap description (layer, spec, positive/negative scope) - - The contents of `functional-spec.md` and `technical-considerations.md` for the target spec - - The relevant implementation source code - -Do not write tests inline here. - -## Step 7: Run tests (with user confirmation) - -1. Count all tests in scope. Use `AskUserQuestion` with the following question and options: - ```text - Ready to run N tests across X specs [unit: A, integration: B, e2e: C, contract: D] - Regression suite: M tests tagged @regression - ``` -2. Ask user to choose: - - **A) Full suite** — run all N tests - - **B) Regression suite only** — run M tests tagged @regression - - **C) Skip** — do not run tests this session -3. Wait for user confirmation before executing. -4. Run the selected suite. Collect pass/fail results per layer. - -## Step 8: Update regression suite - -If `context/qa/regression-suite.md` does not exist, create it from the template at `.awos/templates/regression-suite-template.md` (or create an empty file with a `# Regression Suite` header if the template is absent). - -Sync `context/qa/regression-suite.md`: - -- Scan all test files in scope for `@regression` annotations. -- Add newly tagged tests; remove entries for tests that no longer exist or have lost their `@regression` tag. -- Update "Last updated" date and total count. - -## Step 9: Produce audit report - -Save to `context/qa/audit-reports/qa-report-YYYY-MM-DD.md`: - -```markdown -# QA Audit Report — YYYY-MM-DD - -## Scope - -[Spec(s) audited] - -## Coverage Summary - -| Spec | Unit | Integration | E2E | Contract | ACs Covered | -| ------ | ---- | ----------- | --- | -------- | ----------- | -| [spec] | X/Y | X/Y | X/Y | X/Y | X/Y (Z%) | - -## Flags - -- MISSING: [file] — [reason] -- NEEDS UPDATE: [file]::[test] — [what changed] -- MISSING LAYER: [spec] — [AC] has no [layer] coverage -- MISSING NEGATIVE: [spec] — [AC] [layer] has no negative test -- HUMAN REVIEW: [file]::[test] — spec deleted, no active match found - -## Regression Suite Delta - -- Added: N | Removed: N | Total: N - -## Run Results - -[If tests were run:] - -- Suite: [Full / Regression only] -- Passed: N | Failed: N | Blocked: N -- FAILED: [file]::[test] — [brief reason] - -## Recommendation - -- [ ] Ready — all critical ACs covered, suite passing -- [ ] Needs attention — [N] gaps or failures require action -``` - -## Step 10: Announce - -Report summary to user and list any flags requiring human attention. - ---- - -# TODO - -- **Test registry (`list-of-tests.md`) has staleness risk.** The central registry is a practical choice — one file for the agent to read vs. scanning every test file on each run. Staleness is mitigated by the existence check in Step 4. Future work: generate the registry automatically from inline annotations (`@spec`, `@regression`) in test files, making them the source of truth. - -- **E2E tests are ephemeral — no CI artifact.** Step 6 delegates E2E gap test generation to `testing-expert`, but the output is an in-session test run, not a committed rerunnable artifact. Future work: have `testing-expert` generate playwright-cli script files (e.g., `tests/e2e/*.sh` or `.ts`) as output for E2E gaps, so they can be committed and executed in CI without agent interaction. - -- **QA audit is coverage-by-inspection, not coverage-by-measurement.** Step 5 (gap analysis) reads source files and infers coverage from test file contents. It does not invoke actual coverage tooling (`vitest --coverage`, Istanbul, c8, pytest-cov, etc.). This means untested branches and dead-path gaps are invisible to the audit. Future work: in Step 2, detect available coverage reporters and, in Step 7, run the suite with coverage flags; parse the output to feed real line/branch metrics into the Coverage Summary table. - -- **Audit reports are snapshots with no regression baseline or enforcement.** Each `/awos:qa` run produces a dated report, but there is no mechanism to diff successive reports, track coverage trend over time, or fail a build when coverage drops below a threshold. Future work: add a `context/qa/coverage-baseline.md` file that stores the last known layer coverage counts; at the end of Step 9, compare current counts against the baseline and surface drops explicitly in the Flags section. Goal is delta detection (did coverage drop?), not enforcing absolute thresholds — absolute numbers are meaningless in isolation. - ---- - -# CONSTRAINTS - -- Never modify production code. -- Never auto-deprecate tests whose spec was deleted — always flag for human review. -- Always ask for user confirmation before running tests (use `AskUserQuestion`). -- Generate a diff suggestion for stale tests — do not auto-rewrite them. -- Never skip the overlap check when updating `list-of-tests.md`. From 395caa40c2288b84d2042cd157cd5bfe1e68c463 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 25 May 2026 18:57:09 +0300 Subject: [PATCH 52/56] feat(tasks): add --no-tests / skip tests flag to suppress verification and testing slice Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index 21e52698..9338ba78 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -30,8 +30,9 @@ Follow this process precisely. ## Step 1: Identify the Target Specification -1. **Analyze User Prompt:** Analyze the ``. If it clearly references a spec by name or index, identify the corresponding directory in `context/spec/`. -2. **Ask for Clarification:** If the `` is **empty or ambiguous**, you MUST ask the user to choose. +1. **Detect `--no-tests` flag:** Before anything else, check whether the `` contains `skip tests` or `--no-tests` (case-insensitive). If found, set an internal flag `SKIP_TESTS = true`. This flag suppresses all verification sub-tasks inside slices and omits the Feature Testing & Regression slice entirely. Strip the flag from the prompt before spec identification. +2. **Analyze User Prompt:** Analyze the ``. If it clearly references a spec by name or index, identify the corresponding directory in `context/spec/`. +3. **Ask for Clarification:** If the `` is **empty or ambiguous**, you MUST ask the user to choose. - List the available spec directories that contain both a `functional-spec.md` and `technical-considerations.md`. - Example: "Which specification would you like to break down into tasks? Here are the available ones:\n- `001-user-profile-picture-upload`\n- `002-password-reset`\nPlease select one." - Do not proceed until the user has selected a valid spec. @@ -68,7 +69,9 @@ Follow this process precisely. - Tech stack identified in `technical-considerations.md` - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table - 5. After the verification sub-task, add a cleanup sub-task as the last item of the slice: + 5. **If `SKIP_TESTS = true`**, omit the verification sub-task and the cleanup sub-task for this slice entirely. Skip to generating the next slice. + + **If `SKIP_TESTS = false`**, after the verification sub-task, add a cleanup sub-task as the last item of the slice: ```md - [ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]** ``` @@ -76,9 +79,11 @@ Follow this process precisely. 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 7. Create a high-level checklist item and its sub-tasks with subagent assignments. 8. Repeat this process until all requirements from the specification are covered. - 9. **Add the Feature Testing & Regression slice (always last):** + 9. **Add the Feature Testing & Regression slice (always last, unless `SKIP_TESTS = true`):** + + If `SKIP_TESTS = true`, skip this entire step — do not add the Feature Testing & Regression slice. - After all implementation slices are defined, append one final slice. This slice is generated automatically — do not ask the user about it. + Otherwise, after all implementation slices are defined, append one final slice. This slice is generated automatically — do not ask the user about it. ```md - [ ] **Slice N: Feature Testing & Regression** From 4145b62d645cc9825e27c11f72eb31c7e8496dd5 Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 25 May 2026 19:38:40 +0300 Subject: [PATCH 53/56] chore: fix prettier formatting in tasks.md Co-Authored-By: Claude Sonnet 4.6 --- commands/tasks.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/commands/tasks.md b/commands/tasks.md index 0ec8deca..b0b2128a 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -74,10 +74,13 @@ Follow this process precisely. 5. **If `SKIP_TESTS = true`**, omit the verification sub-task and the cleanup sub-task for this slice entirely. Skip to generating the next slice. **If `SKIP_TESTS = false`**, after the verification sub-task, add a cleanup sub-task as the last item of the slice: + ```md - [ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]** ``` + Skip this sub-task for the **Feature Testing & Regression** slice — its artifacts are kept intentionally. + 6. Next, identify the second-smallest piece of value that builds on the first. This is **Slice 2**. 7. Create a high-level checklist item and its sub-tasks with subagent assignments. 8. Repeat this process until all requirements from the specification are covered. From ba3a38b194ab93a2597339cd077091f9592c7b8f Mon Sep 17 00:00:00 2001 From: Pavlo Samoilov Date: Mon, 25 May 2026 20:48:05 +0300 Subject: [PATCH 54/56] fix: resolve 3 consistency issues found in PR review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - hire.md line 109: playwright MCP → playwright CLI (matches line 116 and global CLAUDE.md rule; MCP vs CLI was a contradiction) - hire.md line 120: complete the Terraform/IaC entry — was truncated with no agent reference - tasks.md example: replace "chrome MCP" with playwright-cli phrasing (legacy example text, inconsistent with playwright-cli convention) Co-Authored-By: Claude Sonnet 4.6 --- commands/hire.md | 4 ++-- commands/tasks.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/commands/hire.md b/commands/hire.md index 14d22be0..5b95ce4b 100644 --- a/commands/hire.md +++ b/commands/hire.md @@ -106,7 +106,7 @@ primary tech agent — unless a more specific testing agent (e.g. `react-testing `rust-tester`) is also found, in which case prefer the specific one. Also: if the tech stack includes any frontend framework (React, Vue, Angular, -Svelte, etc.), always include the `playwright` MCP in the proposal — it enables +Svelte, etc.), always include `playwright` CLI in the proposal — it enables browser-based E2E testing via the testing-expert agent. Complementary pairs reference: @@ -117,7 +117,7 @@ If none found, fall back to `testing-expert`. - FastAPI / Django / Flask → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ - Other Python backends → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ - TypeScript / Node backend → (no additional skill needed) -- Terraform / IaC → (for infra validation) +- Terraform / IaC → search for an infra-validation agent _(if not in registry, `testing-expert` covers plan/output validation)_ ## Step 5: Install Found Components diff --git a/commands/tasks.md b/commands/tasks.md index b0b2128a..87d3a4c2 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -134,7 +134,7 @@ Follow this process precisely. - `[ ] Sub-task: Add avatar_url column to the users table via a migration. **[Agent: python-expert]**` - `[ ] Sub-task: Update the user API endpoint to return the avatar_url. **[Agent: python-expert]**` - `[ ] Sub-task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - - `[ ] Sub-task: Run the application. Use chrome MCP to connect the page in Browser. Verify that the profile page shows the correct avatar or placeholder. **[Agent: manual-qa-expert]**` + - `[ ] Verify: Run the application, use playwright-cli to open the profile page in a browser, confirm correct avatar or placeholder is shown. **[Agent: manual-qa-expert]**` - `[ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]**` - `[ ] **Slice 3: Feature Testing & Regression**` - `> Verifies the complete feature works end-to-end as described in functional-spec.md.` From 3d9c043f4c035b127a6f1547dec2e382ac3186fb Mon Sep 17 00:00:00 2001 From: Aleksandr Makarov Date: Fri, 29 May 2026 14:23:33 +0400 Subject: [PATCH 55/56] fix(tasks): trim Feature Testing & Regression slice noise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two cleanups to the emitted slice, from testing the branch on a real spec: - Collapse the three blockquote note lines into one description. The "QA agent for this slice: `{agent}` (selected from .claude/agents/ and the Agent tool's description block)" line leaked the internal selection rationale into the generated artifact — the `**[Agent: ...]**` markers already say who runs the slice. - Drop the `` block. Its trigger lives in a different repo (the feat/regression PR), so it can't be tracked from here and just sits as dead commentary in every user's tasks.md. The `/awos:regression` wiring belongs in PR #114, which owns that command. --- commands/tasks.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/commands/tasks.md b/commands/tasks.md index ae2a4b16..8f6e3df4 100644 --- a/commands/tasks.md +++ b/commands/tasks.md @@ -92,14 +92,9 @@ Skip this step if `SKIP_TESTS = true`. ```md - [ ] **Slice N: Feature Testing & Regression** - > Verifies the complete feature works end-to-end as described in functional-spec.md. - > Run AFTER all implementation slices are complete. - > QA agent for this slice: `{qa-agent}` (selected from `.claude/agents/` and the Agent tool's description block). + > Verifies the whole feature end-to-end against functional-spec.md, run after all implementation slices are complete. - [ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests that verify the entire feature as a whole — not individual slices. Cover applicable layers (unit for pure logic, integration for service interactions, e2e for user flows) based on the project's testing stack. Write tests with RED validation (must fail before implementation is confirmed done). Annotate each test with `@spec: [spec-directory]` and `@regression` if suitable for long-term regression. **[Agent: {qa-agent}]** - [ ] Run all generated tests. All must pass. Fix any failures before proceeding. **[Agent: {qa-agent}]** - ``` - **Example of applying the rule for "User Profile Picture Upload":** @@ -118,9 +113,7 @@ Skip this step if `SKIP_TESTS = true`. - `[ ] Task: Update the 'ProfileAvatar' component to fetch and display the user's avatar_url, falling back to the placeholder if null. **[Agent: react-expert]**` - `[ ] Verify: Run the application, drive the profile page through the available browser-automation tool (whichever the project ships — playwright-cli, cypress, the chrome MCP, etc.), confirm the correct avatar or placeholder is shown, and delete any screenshots or recordings produced during the check. **[Agent: manual-qa-expert]**` - `[ ] **Slice 3: Feature Testing & Regression**` - > Verifies the complete feature works end-to-end as described in functional-spec.md. - > Run AFTER all implementation slices are complete. - > QA agent for this slice: `testing-expert` (selected from `.claude/agents/` and the Agent tool's description block). + > Verifies the whole feature end-to-end against functional-spec.md, run after all implementation slices are complete. - `[ ] Read functional-spec.md acceptance criteria in full. Generate acceptance-level tests that verify the entire feature as a whole — not individual slices. Cover applicable layers (unit for pure logic, integration for service interactions, e2e for user flows) based on the project's testing stack. Write tests with RED validation (must fail before implementation is confirmed done). Annotate each test with @spec: [spec-directory] and @regression if suitable for long-term regression. **[Agent: testing-expert]**` - `[ ] Run all generated tests. All must pass. Fix any failures before proceeding. **[Agent: testing-expert]**` From ba6e46c242ba6f3bd2bfa168d2faa4e9f601157b Mon Sep 17 00:00:00 2001 From: Aleksandr Makarov Date: Fri, 29 May 2026 16:48:56 +0400 Subject: [PATCH 56/56] fix(verify): require real UI rendering + screenshots in docs/screenshots/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reframing /awos:verify as "look-and-feel" wasn't enough — on a UI-heavy spec the agent satisfied it with in-process component tests (NiceGUI test-client / pytest) and never rendered the UI, so no visual evidence was produced. - Visual/UI acceptance criteria now MUST be verified by driving the actual running UI through the project's browser-automation tool (Playwright MCP/CLI, Cypress, chrome MCP, …). A passing component or test-client test confirms logic, not look-and-feel, and no longer counts as evidence for a visual criterion. Non-visual criteria keep the pick-by-fit freedom. - Screenshots are saved to `docs/screenshots/` — the same evidence folder the testing-expert agent (awos-recruitment) writes E2E captures to — named `-.png` so they sort by spec. The browser tool creates the folder on first write; verify does NOT edit .gitignore (git-ignoring docs/screenshots/ is one-time project setup), matching testing-expert's scope guarantee. The report lists the paths. - skip-tests still suppresses test suites but not look-and-feel. --- commands/verify.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/commands/verify.md b/commands/verify.md index 8de3a65f..223a018d 100644 --- a/commands/verify.md +++ b/commands/verify.md @@ -36,8 +36,9 @@ Verify a specification's implementation against its acceptance criteria. For eac - **`/awos:verify` is a look-and-feel + spec-freshness check, not a test runner.** Generated test suites belong in the Feature Testing & Regression slice produced by `/awos:tasks` and executed by `/awos:implement`. This command verifies that the implementation matches the spec's acceptance criteria from a user's perspective and that downstream documentation has not drifted. - **Step 3 still requires evidence.** Do not mark an acceptance criterion `[x]` without confirming it — either by driving the UI/API yourself or by getting an explicit user confirmation via `AskUserQuestion`. "The user can verify this manually" without a check is not acceptable. -- **Pick the verification tool by fit, not by a fixed priority.** Browser-automation MCPs/CLIs, `curl`, shell scripts, log/database inspection, and direct user confirmation are all valid — choose whichever proves the criterion fastest and most reliably for the project's stack. Do not assume any specific tool is available; if none of the obvious options work, fall back to `AskUserQuestion` and let the user confirm. -- **Honour the `skip-tests` mode.** If the spec's `tasks.md` carries the `` marker (set by `/awos:tasks` when the user opted out of test generation), perform Step 3 as a look-and-feel walk-through only — do not attempt to run or generate test suites, and treat missing test tooling as expected rather than as a verification failure. +- **For non-visual criteria, pick the tool by fit.** APIs, data, CLI, and business-logic criteria can be confirmed with whatever proves them fastest and most reliably — `curl`, shell, log/database inspection, a browser-automation tool, or direct user confirmation. Do not assume any specific tool is available; if none work, fall back to `AskUserQuestion`. The one exception is look-and-feel, covered next. +- **Look-and-feel means real rendering, with screenshots kept as evidence.** For any acceptance criterion describing something a user sees or does in a UI — a rendered page, a control, a state change, a redirect — in-process or component tests are **not** sufficient: they confirm logic, not look-and-feel. Start the app if it isn't running, drive the actual UI through whatever browser-automation tool the project ships (Playwright MCP/CLI, Cypress, the chrome MCP, etc.), and capture screenshots of the states the criteria describe. Save them to `docs/screenshots/` — the same evidence folder the `testing-expert` agent writes E2E captures to — naming each file so it sorts by spec: `docs/screenshots/-.png` (e.g. `docs/screenshots/011-scheduled-tasks-amber-pill.png`). The browser tool creates the folder on first write; do not edit `.gitignore` — git-ignoring `docs/screenshots/` is a one-time project setup, not part of verification. Reference the saved paths in the report — they are what the human reviews. +- **Honour the `skip-tests` mode.** If the spec's `tasks.md` carries the `` marker (set by `/awos:tasks` when the user opted out of test generation), perform Step 3 as a look-and-feel walk-through only — do not run or generate test suites, and treat missing test tooling as expected rather than as a verification failure. Visual criteria are still verified by rendering the UI and capturing screenshots; skip-tests suppresses test suites, not look-and-feel. - **Session length does not excuse skipping.** Even in long sessions, Step 3 must run. --- @@ -59,8 +60,10 @@ Verify a specification's implementation against its acceptance criteria. For eac For each acceptance criterion in `functional-spec.md`: -1. **Verify:** confirm the implementation satisfies the criterion using whichever tool fits the criterion best. Examples (pick by fit, not in this order): drive the UI through a browser-automation MCP or CLI the project already configures; hit an HTTP endpoint with `curl`; run a shell command; inspect logs or the database; ask the user via `AskUserQuestion` when no automated check is possible. -2. **If met:** mark it `[x]` and briefly record how you confirmed it (e.g. "verified via curl /api/health", "user confirmed in browser"). +1. **Verify:** confirm the implementation satisfies the criterion. + - **Non-visual criterion** (API, data, CLI, logic): use whatever check fits best — `curl`, a shell command, log/database inspection. + - **Visual / UI criterion** (anything a user sees or does in a browser): start the app if needed (per `technical-considerations.md`), drive the running UI through the project's browser-automation tool, observe the actual rendered behavior, and save a screenshot of the verified state to `docs/screenshots/-.png` (the shared screenshot folder; see CONSTRAINTS). A passing component/test-client test does not satisfy a visual criterion — render it for real. +2. **If met:** mark it `[x]` and record the evidence — the command output for non-visual criteria, or the screenshot path for visual ones (e.g. "verified via curl /api/health", "see docs/screenshots/011-scheduled-tasks-amber-pill.png"). 3. **If NOT met:** report which criterion failed and what's missing, then stop. 4. **If no tool can verify the criterion in this environment:** ask the user via `AskUserQuestion` — "I can't verify [criterion] automatically because [reason]. Verify manually and confirm, or stop here?" Options: "I verified manually — mark as done" / "Stop — I'll fix the tooling first". Never mark criteria `[x]` without evidence from one of the paths above. @@ -95,3 +98,4 @@ Check if `context/product/` documents need updates based on what was learned dur - Success: spec verified and marked complete; report the verified criteria count. - Failure: list the unmet criteria with the command output that demonstrated the failure. - Verification disabled: list criteria marked `[?]` so the user knows what still needs manual confirmation. +- **Visual evidence:** for any UI criteria verified, list the retained screenshot paths under `docs/screenshots/` so the user can review the look-and-feel without re-running.