feat: Feature Testing & Regression slice, QA audit command, and verification hardening#109
Conversation
Documents three open gaps discovered during end-to-end testing: ephemeral E2E artifacts, coverage-by-inspection vs measurement, and no regression baseline/enforcement mechanism. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| - Tech stack identified in technical-considerations.md | ||
| - Append the subagent assignment using format: `**[Agent: agent-name]**` at the end of the sub-task description | ||
| - Use `general-purpose` agent when no specialist clearly matches the task — but **track these assignments** for the Recommendations table | ||
| 3b. **Generate paired test tasks for each slice (REQUIRED):** |
There was a problem hiding this comment.
Weird numbering mixing thought process syncing
There was a problem hiding this comment.
What if I don't want to generate any tests?
There was a problem hiding this comment.
Weird numbering mixing thought process syncing
Fixed. Thanks.
There was a problem hiding this comment.
What if I don't want to generate any tests?
Added as an option to skip test creation if the user asks about it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Renumber thought process steps sequentially (3b → 5, shift 5-8 → 6-9) - Make test generation opt-out instead of REQUIRED - Add docs/superpowers/ to .gitignore and untrack pushed files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Just to sum-up what was changed in the current flow: Added
Changed
|
| Context received | Behavior |
|---|---|
| spec files, no implementation code | returns a list of test task descriptions — no code written |
| spec files + real implementation code | writes actual test code, runs RED validation, confirms GREEN |
testing-expertis not a slash command — it lives inplugins/awos/agents/and is invoked exclusively via theTasktool withsubagent_type: "testing-expert".testing-expertreturns a completion signal (no gaps) or incomplete/blocked status (gap found) — the caller decides what to do with it, not the agent.
Test layer selection
testing-expert selects test layers per slice based on the slice description and acceptance criteria from functional-spec.md, plus technical-considerations.md.
Slice implementation sub-task description
+ acceptance criteria from functional-spec.md
│
├──► pure logic, no external deps? → Unit
├──► service-to-service / DB calls? → Integration
├──► full user flow via UI or API? → E2E
└──► API schema / interface check? → Contract
Not every slice needs all four layers. Apply judgment.
Notes
- Test layer selection is a judgment call, not a fixed algorithm.
- If
/awos:hirehas installed a project-specific testing agent (e.g.react-testing,python-testing), it takes precedence overtesting-expert. - The previous
tasks.mdexample was misleading because it implied every slice always needs all four layers. In practice, a slice may only require unit + integration coverage.
…vocation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…test coverage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ter, add regression-suite template init
…ch condition-based detection
- hire.md: restructure complementary pairs to search-first pattern with testing-expert as fallback; remove duplicate Python entry; replace playwright MCP with playwright CLI - qa.md: update description to "QA health check"; add user confirmation for full-scope audit; make architecture.md required with warning; rewrite Step 3 to reflect list-of-tests.md is maintained by testing-expert, not created by /awos:qa; implement risk-based gap analysis in Step 5 (two-pass: project-level + per-AC); implement coordinator pattern in Step 6 (specialist agent writes, testing-expert validates and updates registry); add guard for missing functional-spec; remove regression suite steps (moved to PR #114); add staleness and delta-coverage notes to TODO Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
commands/qa.md (1)
11-35:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAdd explicit Creation/Update modes and enforce single-document output contract.
This command does not define Creation Mode vs Update Mode behavior, and it currently specifies multiple mutable outputs (
list-of-tests.md,regression-suite.md, plus a report) instead of a single output document contract.As per coding guidelines, commands in
commands/**/*.mdmust support both Creation Mode and Update Mode and generate a single output document per command.Also applies to: 37-62, 147-203
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@commands/qa.md` around lines 11 - 35, The QA command must explicitly implement Creation Mode vs Update Mode and enforce a single-document output contract: update commands/qa.md to add a mode flag/parameter (e.g., mode=create|update or auto-detect by presence of target file) and implement logic in the command flow (the section referencing context/qa/list-of-tests.md, context/qa/regression-suite.md, and output file generation) so that when mode=create it generates a new single audit report (context/qa/audit-reports/qa-report-YYYY-MM-DD.md) and when mode=update it merges required changes into that same single report rather than emitting multiple top-level files; ensure any mutations to list-of-tests.md and regression-suite.md are represented as sections/patches inside the single audit-report file (or as apply-instructions within it) so the command emits only one document; update the command documentation and usage text in commands/qa.md to describe the two modes and the single-file output behavior and reference the functions/process that perform the merge/update.
🧹 Nitpick comments (1)
commands/qa.md (1)
133-145: ⚡ Quick winMove
AskUserQuestionusage into an# INTERACTIONsection.
AskUserQuestionis used in PROCESS/CONSTRAINTS, but this file is missing a dedicated# INTERACTIONsection for that contract.As per coding guidelines,
AskUserQuestionbelongs in corecommands/*.mdfiles under an# INTERACTIONsection.Also applies to: 218-225
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@commands/qa.md` around lines 133 - 145, The AskUserQuestion invocation currently appears under PROCESS/CONSTRAINTS in Step 7 (the "Run tests" flow) but must be relocated to a dedicated # INTERACTION section; create an # INTERACTION header in commands/qa.md and move the AskUserQuestion prompt and options (the Ready to run N tests... and choices A/B/C, and the wait-for-confirmation step) into that section, removing it from PROCESS/CONSTRAINTS, and apply the same relocation for the other occurrence noted around lines 218-225 so all AskUserQuestion uses live only under # INTERACTION.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@commands/qa.md`:
- Around line 118-124: Replace the delegation tool reference "Task" with "Agent"
in the guidance list so the prompt uses the correct framework name; specifically
update the step that currently says "invoke it via the `Task` tool" to say
"invoke it via the `Agent` tool" (keep the rest of the parameters and subsequent
mention of `testing-expert` unchanged).
---
Outside diff comments:
In `@commands/qa.md`:
- Around line 11-35: The QA command must explicitly implement Creation Mode vs
Update Mode and enforce a single-document output contract: update commands/qa.md
to add a mode flag/parameter (e.g., mode=create|update or auto-detect by
presence of target file) and implement logic in the command flow (the section
referencing context/qa/list-of-tests.md, context/qa/regression-suite.md, and
output file generation) so that when mode=create it generates a new single audit
report (context/qa/audit-reports/qa-report-YYYY-MM-DD.md) and when mode=update
it merges required changes into that same single report rather than emitting
multiple top-level files; ensure any mutations to list-of-tests.md and
regression-suite.md are represented as sections/patches inside the single
audit-report file (or as apply-instructions within it) so the command emits only
one document; update the command documentation and usage text in commands/qa.md
to describe the two modes and the single-file output behavior and reference the
functions/process that perform the merge/update.
---
Nitpick comments:
In `@commands/qa.md`:
- Around line 133-145: The AskUserQuestion invocation currently appears under
PROCESS/CONSTRAINTS in Step 7 (the "Run tests" flow) but must be relocated to a
dedicated # INTERACTION section; create an # INTERACTION header in
commands/qa.md and move the AskUserQuestion prompt and options (the Ready to run
N tests... and choices A/B/C, and the wait-for-confirmation step) into that
section, removing it from PROCESS/CONSTRAINTS, and apply the same relocation for
the other occurrence noted around lines 218-225 so all AskUserQuestion uses live
only under # INTERACTION.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 52d67e6a-4201-4bed-b76f-20e9efc84c25
📒 Files selected for processing (2)
commands/hire.mdcommands/qa.md
✅ Files skipped from review due to trivial changes (1)
- commands/hire.md
|
|
||
| # TASK | ||
|
|
||
| Perform a full QA audit for the target spec(s). Check existing tests for health (missing, stale, deprecated), identify gaps against spec acceptance criteria, generate missing tests, offer to run the suite, and produce an audit report in `context/qa/audit-reports/`. |
There was a problem hiding this comment.
Check existing tests for health (missing, stale, deprecated)
For big codebase and after AWOS update it could trigger quite heavy task. Better to scope it somehow.
| # INPUTS & OUTPUTS | ||
|
|
||
| - **User Prompt (Optional):** <user_prompt>$ARGUMENTS</user_prompt> | ||
| - Empty = audit all specs |
There was a problem hiding this comment.
- After AWOS update it could trigger quite heavy task.
- All specs != all code base (if AWOS wasn't used for all tasks or was added to project midterm), so it could make gaps in tests coverage anyway.
I would suggest to always scope to one spec, if not provided - ask user for the scope. Only if user explicitly asked for the whole project - run for the whole project.
| ## Step 2: Discover frameworks | ||
|
|
||
| 1. Read `context/product/architecture.md` for declared testing stack per layer. | ||
| 2. Fall back to auto-detection via dependency files: `package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pyproject.toml`, `pom.xml`. |
There was a problem hiding this comment.
I agree with concern but let me add few facts here as well:
package.json,requirements.txt,go.mod,Gemfile,pyproject.toml,pom.xml
won't provide the full architecture - they usually provide information only about the main language or repo. Also they may be placed in folders/subfolders and CC may miss them. Or it could be one "BUILD" file orchestrating big monorepo.
We can't rely on CI files as well, because repos may don't have CI at all, have it partial or have CI configured in a separate CI server.
Therefore I would rather fail if context/product/architecture.md is absent and don't try to provide unclear quality of tests for projects without required AWOS artifacts.
|
|
||
| - Does the `@spec` annotation reference a spec directory that still exists? | ||
| - YES → OK. | ||
| - Spec directory deleted → |
There was a problem hiding this comment.
Let me note that
codebase and existing tests are the source of truth
is not the optimal solution because of tokens usage and quality overheads. We may mitigate this but fundamentally index of tests is more efficient. If it is up-to-date.
While we should handle absence of specs gently, because it could happen with:
- AWOS was added to repo midterm;
- Specs were archived/outdated (AWOS doesn't have such feature yet, but competitors do);
- Not all tests could be added in scope of spec of with AWOS participation at all.
TL;DR: managing of tests is a separate feature which requires careful design. I would rather split it into separate PR.
|
|
||
| For each acceptance criterion in the functional spec(s) in scope: | ||
|
|
||
| 1. Check which pyramid layers have coverage (unit / integration / e2e / contract). |
There was a problem hiding this comment.
"React component test" it is unit test (when component doesn't interact with other services directly, which is a bad design). Please look at https://martinfowler.com/articles/practical-test-pyramid.html
So I would agree with the current layers.
| **Staleness check:** | ||
|
|
||
| - Does the test logic match the current implementation? | ||
| - Read the test file and the relevant implementation code. |
There was a problem hiding this comment.
Weak metrics. Consider to use some code coverage metrics instead.
|
|
||
| ## Step 7: Run tests (with user confirmation) | ||
|
|
||
| 1. Count all tests in scope. Use `AskUserQuestion` with the following question and options: |
There was a problem hiding this comment.
(nit) I would rather ask LLM to decide (basing on current spec/code changes) - interrupting user is much longer and usually provides the same quality of decision nowadays.
|
|
||
| --- | ||
|
|
||
| # TODO |
There was a problem hiding this comment.
LLM would read this section as context like any other section. So mentioning future work here just dilutes context. Please remove or put into supported by CC <!-- TODO: ... --> comments.
|
|
||
| - Never modify production code. | ||
| - Never auto-deprecate tests whose spec was deleted — always flag for human review. | ||
| - Always ask for user confirmation before running tests (use `AskUserQuestion`). |
There was a problem hiding this comment.
Why we need to bother user for a few seconds background task?
|
|
||
| # CONSTRAINTS | ||
|
|
||
| - Never modify production code. |
There was a problem hiding this comment.
Not agree. Tests allows to find bad design patterns and I time to time change production code after trying to cover it with tests without good results.
The command had no active Claude Code wrapper and its functionality is covered by two dedicated commands: - testing-expert agent (proactive test writing via /awos:tasks) - /awos:regression (regression suite management, PR #114) Retroactive staleness/gap auditing can be revisited as a plugin if needed in the future. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Update: After reviewing all the feedback on this file and mapping it against the broader command set, we concluded that
This makes most of the open comments on |
…n and testing slice Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- commands/tasks.md: adopt main's simplified prose + preserve Feature Testing & Regression slice, --no-tests flag, cleanup sub-tasks, and artifact-deletion rule - commands/verify.md: adopt main's INTERACTION section alongside our CONSTRAINTS section - src/CLAUDE.md: adopt main's preserveOnUpdate column + our agents row - tests/lint-prompts.test.js: strip HTML comments before scanning for /awos:* references so TODO-gated regression reference doesn't fail the cross-reference lint Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- hire.md line 109: playwright MCP → playwright CLI (matches line 116 and global CLAUDE.md rule; MCP vs CLI was a contradiction) - hire.md line 120: complete the Terraform/IaC entry — was truncated with no agent reference - tasks.md example: replace "chrome MCP" with playwright-cli phrasing (legacy example text, inconsistent with playwright-cli convention) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
||
| - React / Vue / Angular → + `playwright` CLI for E2E | ||
| - FastAPI / Django / Flask → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ | ||
| - Other Python backends → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ |
There was a problem hiding this comment.
(nit) We may merge two lines above together. Not fixed from previous review.
| - FastAPI / Django / Flask → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ | ||
| - Other Python backends → + `pytest-best-practices` skill _(if not in registry, `testing-expert` covers it)_ | ||
| - TypeScript / Node backend → (no additional skill needed) | ||
| - Terraform / IaC → search for an infra-validation agent _(if not in registry, `testing-expert` covers plan/output validation)_ |
There was a problem hiding this comment.
for an infra-validation agent
Why "infra-validation" here without backticks formatting as other agents above? Actually I can't find such agent in AWOS-recruitment, so probably better to explain agent in some common words without using kebab case (to match future agents), i.e. "infra-validation" -> "technology specific or dedicated infrastructure validation agent".
|
|
||
| 1. Analyze `<user_prompt>`. If it clearly references a spec by name or index, identify the corresponding directory in `context/spec/`. | ||
| 2. If the prompt is empty or ambiguous, list the spec directories that contain both `functional-spec.md` and `technical-considerations.md` and ask the user to choose. Do not proceed until a valid spec is selected. | ||
| 1. **Detect `--no-tests` flag:** Before anything else, check whether the `<user_prompt>` contains `skip tests` or `--no-tests` (case-insensitive). If found, set an internal flag `SKIP_TESTS = true`. This flag suppresses all verification sub-tasks inside slices and omits the Feature Testing & Regression slice entirely. Strip the flag from the prompt before spec identification. |
There was a problem hiding this comment.
Detect
--no-testsflag: Before anything else, check whether the<user_prompt>containsskip testsor--no-tests(case-insensitive). If found, set an internal flagSKIP_TESTS = true.
I would propose to don't shape it as a flag. Because it may fail on human's wording - i.e. with "don't generate qa stuff now" phrase would probably be not detected here as "--no-tests flag" while I may bet that user wanted exactly this.
Like:
| 1. **Detect `--no-tests` flag:** Before anything else, check whether the `<user_prompt>` contains `skip tests` or `--no-tests` (case-insensitive). If found, set an internal flag `SKIP_TESTS = true`. This flag suppresses all verification sub-tasks inside slices and omits the Feature Testing & Regression slice entirely. Strip the flag from the prompt before spec identification. | |
| 1. If `<user_prompt>` explicitly mentions that tests are not expected then skip all verification sub-tasks inside slices and omits the Feature Testing & Regression slice entirely. |
| - Check that the project has the MCPs, services, and dependencies needed for testing each slice. If something is missing, instruct the user to install it. | ||
| - If a slice cannot be tested, explain why and get user approval before proceeding. | ||
| - A slice is not complete unless it is tested or the user has explicitly approved skipping the test. | ||
| - After a slice is verified and marked complete, **delete all temporary artifacts** generated during that slice's verification — screenshots, recorded videos, generated e2e test scripts, and any other ephemeral files produced by the e2e-tester or browser MCP. Exception: do **not** delete artifacts from the **Feature Testing & Regression** slice — those are intentionally kept for the regression suite. |
There was a problem hiding this comment.
e2e-tester
What it is?
| { | ||
| source: 'plugins/awos/agents', | ||
| destination: '.claude/agents', | ||
| patterns: ['*'], | ||
| description: 'AWOS agents', | ||
| }, |
There was a problem hiding this comment.
.claude/agents/ is the user's customization area. This copy operation has no preserveOnUpdate: true, so every npx @provectusinc/awos will silently clobber any user agent file whose name collides with an AWOS-shipped one. Same UX bug PR #121 just fixed for .claude/commands/awos/.
So please don't ship agents through the installer at all — per https://provectus.slack.com/archives/C09GCR80NC8/p1779723615299239?thread_ts=1778491156.726109&cid=C09GCR80NC8 testing-expert is to be hired from awos-recruitment, which contradicts this copy op.
| Also: if the tech stack includes any frontend framework (React, Vue, Angular, | ||
| Svelte, etc.), always include `playwright` CLI in the proposal — it enables | ||
| browser-based E2E testing via the testing-expert agent. |
There was a problem hiding this comment.
Hardcoding playwright CLI for every frontend stack contradicts the search-first principle stated a few lines below (line 113: "search the registry for a technology-specific testing agent"). Cypress, WebdriverIO, Vitest-browser, Storybook test-runner all exist and may be a better fit per project.
Same anti-hardcoding concern as commands/verify.md line 40. The bullet at line 116 (React / Vue / Angular → + playwright CLI for E2E) repeats the same hardcoding.
| **QA Complement Rule:** | ||
| After searching for each primary tech agent, also search for a complementary | ||
| testing agent. Use the query: "[primary technology] testing QA acceptance". | ||
|
|
||
| If `testing-expert` is returned from the registry, always include it in the | ||
| proposals table with label "(Recommended for QA coverage)" alongside the | ||
| primary tech agent — unless a more specific testing agent (e.g. `react-testing`, | ||
| `rust-tester`) is also found, in which case prefer the specific one. | ||
|
|
||
| Also: if the tech stack includes any frontend framework (React, Vue, Angular, | ||
| Svelte, etc.), always include `playwright` CLI in the proposal — it enables | ||
| browser-based E2E testing via the testing-expert agent. | ||
|
|
||
| Complementary pairs reference: | ||
| For each stack, search the registry for a technology-specific testing agent. | ||
| If none found, fall back to `testing-expert`. |
There was a problem hiding this comment.
Two paragraphs cover the same testing-expert fallback logic with conflicting defaults:
- Lines 99-106 (QA Complement Rule): "always include
testing-expertin proposals … unless a more specific testing agent is also found." - Lines 112-114 (Complementary pairs reference): "search the registry for a technology-specific testing agent. If none found, fall back to
testing-expert."
First frames testing-expert as always-include-with-caveat, the second as search-first-with-fallback. Both rules currently apply → ambiguous behavior. Pick one framing and remove the other.
| - `[ ] **Slice 3: Feature Testing & Regression**` | ||
| - `> Verifies the complete feature works end-to-end as described in functional-spec.md.` | ||
| - `> Run AFTER all implementation slices are complete.` | ||
| - `> **Requires \`testing-expert\` agent.\*\* If it is not present in \`.claude/agents/\`, stop and run \`/awos:hire\` before executing this slice.` |
There was a problem hiding this comment.
Markdown escape artifacts. The \*\* and backslash-escaped backticks look unintentional — likely from copying the template at line 98 (which renders fine) into this example with extra escaping. Compare to template:
> **Requires `testing-expert` agent.** If it is not present in `.claude/agents/`
The example line will render with literal \*\* instead of bold.
BTW a good place to use XML tags.
| **If `SKIP_TESTS = false`**, after the verification sub-task, add a cleanup sub-task as the last item of the slice: | ||
|
|
||
| ```md | ||
| - [ ] Cleanup: Delete any screenshots, videos, or e2e scripts generated during this slice's verification. **[Agent: general-purpose]** |
There was a problem hiding this comment.
(nit) [Agent: general-purpose] for the cleanup step — rm of known-named artifacts is a one-line shell command. SDD-07 audit dimension flags "majority of implementation tasks use general-purpose" as a warn condition; cleanup sub-tasks add to that count without adding decision value. Two options:
- Inline the
rminto the Verify step (no separate sub-task). - Exempt cleanup sub-tasks from the SDD-07 count, and document that exemption in
plugins/awos/skills/ai-readiness-audit/dimensions/spec-driven-development.md.
| @@ -0,0 +1,11 @@ | |||
| # Test Registry | |||
|
|
|||
| > Auto-maintained by `testing-expert` agent (called from the Feature Testing & Regression slice). | |||
There was a problem hiding this comment.
Future-debt note: this template ships, but the contract for keeping it in sync with the real codebase is informal (just "auto-maintained by testing-expert"). It is just not used in this repo.
The same concerns raised in the now-deleted commands/qa.md threads (registry staleness, merge-conflict hotspots) apply here.
Consider to move it into provectus/awos-recruitment#60 entirely or into future PR-s.
| > Verifies the complete feature works end-to-end as described in functional-spec.md. | ||
| > Run AFTER all implementation slices are complete. | ||
| > **Requires `testing-expert` agent.** If it is not present in `.claude/agents/`, | ||
| > stop and run `/awos:hire` before executing this slice. |
There was a problem hiding this comment.
This would cause interruption for /awos:implement command while we may catch absence of testing-expert just here and form request to user with 3 options:
- Install agent,
- Skip slice with tests.
- Add slice but without guarantees for quality (BTW not sure it is possible at all).
… search-first model Merges main (#126, #127, #129) into the branch and replaces the hard `testing-expert` dependency with a search-first, registry-driven QA agent selection. Addresses the round-2 reviewer feedback on PR #109. Code changes - commands/tasks.md - Step 1.3 detects test-skip intent via natural language, not a literal flag substring (avoids false-positives like "don't skip tests"). - Step 3a: new dedicated step that searches the Agent tool's description block for a QA-coded subagent, falls back to testing-expert / general-purpose, and uses AskUserQuestion to offer a 3-option fallback (hire, generate-with-general-purpose, skip). Replaces the prior "Requires testing-expert" hard gate. - Inlines verification-artifact cleanup into each Verify task instead of emitting a separate Cleanup task with [Agent: general-purpose] (which would inflate SDD-07's general-purpose-task count). - Drops references to a phantom "e2e-tester" agent and to a hardcoded `playwright-cli` runner in the example. - Persists the skip-tests decision as a top-of-file `<!-- skip-tests: true -->` marker for /awos:verify to detect. - commands/hire.md - Rewrites the QA Complement Rule as a single search-first rule with an explicit "do not hardcode tool names or runners" note. - Drops the always-include-testing-expert and always-include-playwright mandates, the duplicate "Other Python backends" bullet, and the bare "infra-validation" agent reference. - commands/verify.md - CONSTRAINTS reframed: /awos:verify is a look-and-feel + spec- freshness check, not a test runner. Test suites live in the Feature Testing & Regression slice produced by /awos:tasks. - Honours the `<!-- skip-tests: true -->` marker — look-and-feel walk-through only when the spec opted out of test generation. - Drops the "browser MCP → curl/shell → AskUserQuestion" hardcoded fallback ladder; tool choice is by fit + wall-clock time. - src/config/setup-config.js + src/CLAUDE.md - Removes the `plugins/awos/agents` → `.claude/agents` copy operation. `.claude/agents/` is the user's customization area and must not be silently overwritten on update. - templates/qa-context-template.md - Deleted. The template was unused inside this repo; the testing-expert agent that would populate it lives in awos-recruitment, where the template can ship alongside if desired. - plugins/awos/skills/ai-readiness-audit/dimensions/spec-driven-development.md - SDD-07 recognizes both the new (Feature Testing & Regression final slice) and legacy (per-slice Verify-task) QA coverage models. Pass criteria broadened to accept any QA-coded agent, not just testing-expert. - CLAUDE.md - Adds a "do not hard-wrap markdown prose at 80 columns" rule under Editing Prompts, capturing the long-standing local convention. Tests - tests/lint-prompts.test.js - New contract tests: Feature Testing & Regression slice marker is present; tasks.md uses a search-first QA selection and does not hard-require testing-expert; the `<!-- skip-tests: true -->` marker shape is locked; verify.md acknowledges skip-tests and drops the hardcoded tool ladder; hire.md QA Complement Rule has no "always include" mandates; setup-config never targets `.claude/agents/`; qa-context-template.md does not exist; SDD-07 recognizes the dual-model coverage. All three test layers + prettier are green (78 tests across 8 files).
Two cleanups to the emitted slice, from testing the branch on a real spec:
- Collapse the three blockquote note lines into one description. The
"QA agent for this slice: `{agent}` (selected from .claude/agents/ and
the Agent tool's description block)" line leaked the internal selection
rationale into the generated artifact — the `**[Agent: ...]**` markers
already say who runs the slice.
- Drop the `<!-- TODO: enable when feat/regression merges -->` block.
Its trigger lives in a different repo (the feat/regression PR), so it
can't be tracked from here and just sits as dead commentary in every
user's tasks.md. The `/awos:regression` wiring belongs in PR #114,
which owns that command.
…ots/ Reframing /awos:verify as "look-and-feel" wasn't enough — on a UI-heavy spec the agent satisfied it with in-process component tests (NiceGUI test-client / pytest) and never rendered the UI, so no visual evidence was produced. - Visual/UI acceptance criteria now MUST be verified by driving the actual running UI through the project's browser-automation tool (Playwright MCP/CLI, Cypress, chrome MCP, …). A passing component or test-client test confirms logic, not look-and-feel, and no longer counts as evidence for a visual criterion. Non-visual criteria keep the pick-by-fit freedom. - Screenshots are saved to `docs/screenshots/` — the same evidence folder the testing-expert agent (awos-recruitment) writes E2E captures to — named `<spec-directory>-<state>.png` so they sort by spec. The browser tool creates the folder on first write; verify does NOT edit .gitignore (git-ignoring docs/screenshots/ is one-time project setup), matching testing-expert's scope guarantee. The report lists the paths. - skip-tests still suppresses test suites but not look-and-feel.
| > Verifies the complete feature works end-to-end as described in functional-spec.md. | ||
| > Run AFTER all implementation slices are complete. | ||
| > **Requires `testing-expert` agent.** If it is not present in `.claude/agents/`, | ||
| > stop and run `/awos:hire` before executing this slice. |
There was a problem hiding this comment.
These > note lines leak meta-instructions into the artifact. Testing this branch on a real spec, the rendered slice carried a > QA agent for this slice: testing-expert (selected from .claude/agents/ and the Agent tool's description block) line — the agent-selection rationale ended up in the user's file. Recommend collapsing to a single one-line description (matching how every other slice has one > description) and dropping the selection note — the **[Agent: ...]** markers already state who runs the slice. The > **Requires testing-expert** … run /awos:hire gate is likewise superseded by the search-first QA-agent selection.
| <!-- TODO: enable when feat/regression merges | ||
| - [ ] Run `/awos:regression [spec-directory-name]` to review candidates for the | ||
| regression suite, resolve duplicates, and optionally execute the full | ||
| regression suite. Pass the current spec directory name as the argument | ||
| (e.g., `/awos:regression 003-user-avatar`). Do not run without an argument — | ||
| auto-detection requires all tasks to be complete, which is not yet the case. | ||
| --> |
There was a problem hiding this comment.
Better to drop this <!-- TODO: enable when feat/regression merges --> block. Its trigger lives in a different PR (#114) in this repo, so it can't be tracked from a generated tasks.md sitting in a user's project — it just ships as dead commentary in every user's file. Let PR #114, which owns /awos:regression, add the call to this slice when it lands. The duplicate block in the example below (~line 145) should go too.
| - **Verification tool fallback order:** browser MCP → curl/shell → AskUserQuestion to skip. | ||
| If browser MCP is unavailable, try curl or shell commands. Only ask to skip as a last resort. |
There was a problem hiding this comment.
(found this verifying a real UI-heavy spec) Even with browser-MCP listed first, /awos:verify produced no visual verification and no screenshots. On an already-implemented (Status: Completed) spec the agent took the fastest-evidence path — ran the existing pytest/component suite and read the code — and never rendered the UI. NiceGUI / React-Testing-Library (in my case) test-client tests confirm logic, not look-and-feel, so the run passed with zero rendered-pixel evidence.
Two gaps:
- Nothing forces real rendering for UI/visual acceptance criteria — in-process component tests are accepted as sufficient. For look-and-feel this is not enough - the actual UI must be visible to a human.
- No screenshots are captured or retained, so a human can't review the look-and-feel afterward.
Suggest aligning with the testing-expert agent (awos-recruitment): it already writes E2E captures to docs/screenshots/ and treats git-ignoring that folder as one-time project setup (it never edits .gitignore). /awos:verify should (a) require driving the real UI for visual ACs — component tests don't count, (b) save screenshots to that same docs/screenshots/ folder, named per-spec so they sort by spec, and (c) reference the paths in its report. Reuse testing-expert's folder + no-.gitignore-edit convention rather than inventing a new evidence location.
|
@kmakarychev-dev, @workshur, @FlySpot - please note that https://github.com/provectus/awos-qa/pull/10 adds 2 new e2e tests specifically for this functionality (see "INSTRUCTIONS.md" files for the checks they do), could be executed with something like |
6 participants
Summary
/awos:tasks— adds a mandatory Feature Testing & Regression final slice to every task list. The slice runstesting-expertto generate acceptance-level tests againstfunctional-spec.mdacceptance criteria after all implementation slices are complete. Also adds a Cleanup sub-task after each implementation slice to delete ephemeral artifacts (screenshots, videos, e2e scripts) produced during verification. Artifact cleanup is skipped for the Feature Testing & Regression slice — those are retained for the regression suite./awos:hire— adds a QA Complement Rule: after hiring a primary tech agent, always proposetesting-expertalongside it. Frontend stacks also getplaywrightMCP suggested./awos:verify— adds a CONSTRAINTS section enforcing that Step 3 verification is mandatory. Defines a fallback chain: browser MCP → curl/shell → logs/database → explicit user approval viaAskUserQuestion. Agents can no longer silently skip verification.templates/qa-context-template.md— starter template for the test registry atcontext/qa/list-of-tests.md.spec-driven-development.md(SDD-07) updated to recognize the Feature Testing & Regression final-slice model alongside the legacy per-slice model.src/config/setup-config.jsnow copiesplugins/awos/agents/→.claude/agents/duringnpx @provectusinc/awos, matching the documentation insrc/CLAUDE.md.What is NOT in this PR
/awos:regressioncommand — lives infeat/regression(PR Draft: feat: add /awos:regression command #114). The regression sub-task inside the Feature Testing & Regression slice is commented out pending that merge.testing-expertagent — hired viaawos-recruitmentregistry, not bundled in the core package./awos:qaslash command — the Claude Code wrapper was removed;commands/qa.mdis preserved as a future plugin reference only.File map
Test plan
/awos:taskson a spec — confirm the Feature Testing & Regression slice appears as the last slice with correct wording and agent assignments/awos:hirewith a frontend tech — confirmtesting-expertandplaywrightMCP appear in proposals/awos:verifywithout browser MCP — confirm fallback chain is followed and agent cannot silently skip🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation