From cc97b16035815a5e8d39cf11130d03c5cb431dbe Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 2 Jul 2026 15:34:38 +0200 Subject: [PATCH] feat(plan-feature): add testing task generation from readiness template MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add auto-discovery of docs/testing-readiness.md in target repositories during Step 3, and generate one cross-cutting testing task per test category in the template during Step 5. Testing tasks depend on all implementation tasks and omit code-change sections. Includes 3 new constraints (§1.85–1.87) with traceability, eval fixture with 2 test categories, and positive/negative eval assertions. Implements TC-4878 Co-Authored-By: Claude Opus 4.6 (1M context) Assisted-by: Claude Code --- docs/constraints.md | 5 +- evals/plan-feature/evals.json | 19 ++++---- evals/plan-feature/files/testing-readiness.md | 15 ++++++ .../skills/plan-feature/SKILL.md | 48 +++++++++++++++++++ 4 files changed, 78 insertions(+), 9 deletions(-) create mode 100644 evals/plan-feature/files/testing-readiness.md diff --git a/docs/constraints.md b/docs/constraints.md index ad2b691c2..e0fe45c1a 100644 --- a/docs/constraints.md +++ b/docs/constraints.md @@ -94,6 +94,9 @@ existing instruction in a SKILL.md or CLAUDE.md file. | 1.82 | `plan-feature` parent issue linking MUST validate that the parent issue's `hierarchyLevel` is higher than the Feature's `hierarchyLevel` before setting the link. | `plan-feature/SKILL.md` — Step 1.5 | | 1.83 | `plan-feature` MUST generate a documentation task when the Feature description's "Documentation Considerations" section indicates doc impact (New Content, Updates, or Release Notes). | `plan-feature/SKILL.md` — Step 5 (Documentation task generation) | | 1.84 | `plan-feature` MUST NOT generate a documentation task when the Feature description has no "Documentation Considerations" section or states "No Doc Impact". | `plan-feature/SKILL.md` — Step 5 (Documentation task generation) | +| 1.85 | `plan-feature` MUST generate testing tasks when a testing readiness template exists at `docs/testing-readiness.md` in the target repository, with one task per test category in the template. | `plan-feature/SKILL.md` — Step 5 (Testing task generation) | +| 1.86 | `plan-feature` MUST NOT generate testing tasks when no testing readiness template exists in the target repository. | `plan-feature/SKILL.md` — Step 5 (Testing task generation) | +| 1.87 | `plan-feature` testing tasks MUST depend on all implementation tasks. | `plan-feature/SKILL.md` — Step 5 (Testing task generation) | ### Prior Art — Cross-phase integrity (§1.33–1.35) @@ -174,7 +177,7 @@ existing instruction in a SKILL.md or CLAUDE.md file. Each constraint above references its source. The full source files are: -- `plugins/sdlc-workflow/skills/plan-feature/SKILL.md` — Guardrails (§1.1–1.3), Step 1 Priority/fixVersion extraction (§1.74, §1.76), Step 1.5 Parent linking (§1.82), Step 2.5 Discover Project Issue Types (§1.62–1.64), Step 4.5 Determine Workflow Mode (§1.27), Step 5 Epic grouping (§1.79, §1.81), Step 5 Convention-aware task enrichment (§4.11, §4.13), Step 5 Target Branch assignment (§4.12), Step 5 Bookend task generation (§3.4), Step 5 Documentation task generation (§1.83, §1.84), Step 6 sub-step 6a.0 Epic creation (§1.79), Step 6 graceful degradation (§1.80), Step 6 Priority/fixVersion inheritance (§1.74, §1.75, §1.76), Step 6a Digest posting (§1.33), Task Description Template (§4.1–4.10) +- `plugins/sdlc-workflow/skills/plan-feature/SKILL.md` — Guardrails (§1.1–1.3), Step 1 Priority/fixVersion extraction (§1.74, §1.76), Step 1.5 Parent linking (§1.82), Step 2.5 Discover Project Issue Types (§1.62–1.64), Step 4.5 Determine Workflow Mode (§1.27), Step 5 Epic grouping (§1.79, §1.81), Step 5 Convention-aware task enrichment (§4.11, §4.13), Step 5 Target Branch assignment (§4.12), Step 5 Bookend task generation (§3.4), Step 5 Documentation task generation (§1.83, §1.84), Step 5 Testing task generation (§1.85–1.87), Step 6 sub-step 6a.0 Epic creation (§1.79), Step 6 graceful degradation (§1.80), Step 6 Priority/fixVersion inheritance (§1.74, §1.75, §1.76), Step 6a Digest posting (§1.33), Task Description Template (§4.1–4.10) - `plugins/sdlc-workflow/skills/implement-task/SKILL.md` — Important Rules (§1.4–1.6, §5.1–5.3), Step 1 (§1.6), Step 1.5 Digest verification (§1.34, §1.35), Step 4/6/9 (§5.4), Step 5 (§1.15, §3.1, §3.4), Step 7 (§5.9–5.13), Step 9 (§2.1–2.3, §5.6–5.8), Step 10 (§3.2, §3.3) - `plugins/sdlc-workflow/shared/task-description-template.md` — Rules (§4.12) - `plugins/sdlc-workflow/shared/description-digest-protocol.md` — Digest format and verification procedure (§1.33, §1.34, §1.35) diff --git a/evals/plan-feature/evals.json b/evals/plan-feature/evals.json index 9083c3c98..93798ed6d 100644 --- a/evals/plan-feature/evals.json +++ b/evals/plan-feature/evals.json @@ -3,12 +3,12 @@ "evals": [ { "id": 1, - "prompt": "Plan the implementation for feature TC-9001. The feature description is in feature-standard.md and the target repository structure is in repo-backend.md. Write all outputs to the workspace outputs/ directory. For each task you would create, write a markdown file named task-N-slug.md with the full task description. Write the impact map to impact-map.md.", - "expected_output": "A structured implementation plan with task decomposition, each task following task-description-template.md format, with clear dependency ordering. Tasks should reference real file paths from the repo manifest.", - "files": ["files/feature-standard.md", "files/repo-backend.md"], + "prompt": "Plan the implementation for feature TC-9001. The feature description is in feature-standard.md and the target repository structure is in repo-backend.md. The target repository has a testing readiness template at docs/testing-readiness.md (provided as testing-readiness.md). Write all outputs to the workspace outputs/ directory. For each task you would create, write a markdown file named task-N-slug.md with the full task description. Write the impact map to impact-map.md.", + "expected_output": "A structured implementation plan with task decomposition, each task following task-description-template.md format, with clear dependency ordering. Tasks should reference real file paths from the repo manifest. Testing tasks should be generated from the testing readiness template.", + "files": ["files/feature-standard.md", "files/repo-backend.md", "files/testing-readiness.md"], "assertions": [ - "Each non-documentation task file contains all required template sections: Repository, Target Branch, Description, at least one of Files to Modify or Files to Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation tasks (tasks whose filename or description indicates doc-only scope) are exempt from requiring Files to Modify, Files to Create, and Implementation Notes — they must still include Repository, Target Branch, Description, Acceptance Criteria, and Test Requirements", - "Task count is between 3 and 10 inclusive", + "Each non-documentation, non-testing task file contains all required template sections: Repository, Target Branch, Description, at least one of Files to Modify or Files to Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation tasks (tasks whose filename or description indicates doc-only scope) and testing tasks (tasks whose filename or description indicates cross-cutting testing scope) are exempt from requiring Files to Modify, Files to Create, and Implementation Notes — they must still include Repository, Target Branch, Description, Acceptance Criteria, and Test Requirements", + "Task count is between 3 and 12 inclusive", "All tasks reference repository 'trustify-backend'", "Task dependencies form a DAG with no circular dependencies", "Every task's Files to Modify and Files to Create reference paths plausible within the repo-backend.md structure", @@ -16,14 +16,16 @@ "Every generated task description contains a Repository section with a single repository name (not multiple repos per task)", "Every generated task description contains Target Branch, Description, Acceptance Criteria, and Test Requirements sections as required by the handoff contract in task-description-template.md", "File paths in Files to Modify and Files to Create reference paths from the repo-backend.md mock repository structure manifest, not invented paths", - "Implementation Notes in every non-documentation task reference specific file paths and code patterns from the repository, not abstract or generic guidance. Documentation tasks are exempt from this requirement as they intentionally omit Implementation Notes", + "Implementation Notes in every non-documentation, non-testing task reference specific file paths and code patterns from the repository, not abstract or generic guidance. Documentation tasks and testing tasks are exempt from this requirement as they intentionally omit Implementation Notes", "Every generated task description contains a Target Branch section set to 'main'", "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py", "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies matching the convention's '), not free-form prose", "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')", "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields", "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why", - "A documentation task is generated because the feature description's Documentation Considerations section indicates doc impact (Updates — add endpoint to REST API reference). The documentation task includes Repository, Target Branch, Description, Acceptance Criteria, Test Requirements, and Dependencies sections, and omits Files to Modify, Files to Create, API Changes, Implementation Notes, Reuse Candidates, and Verification Commands" + "A documentation task is generated because the feature description's Documentation Considerations section indicates doc impact (Updates — add endpoint to REST API reference). The documentation task includes Repository, Target Branch, Description, Acceptance Criteria, Test Requirements, and Dependencies sections, and omits Files to Modify, Files to Create, API Changes, Implementation Notes, Reuse Candidates, and Verification Commands", + "Testing tasks are generated from the testing readiness template — exactly 2 testing tasks are created (one per test category: Smoke Tests and Performance Benchmarks). Each testing task includes Repository, Target Branch, Description, Acceptance Criteria, Test Requirements, and Dependencies sections, and omits Files to Modify, Files to Create, API Changes, Implementation Notes, Reuse Candidates, and Verification Commands", + "Each testing task depends on all implementation tasks (not on documentation or other testing tasks)" ] }, { @@ -46,7 +48,8 @@ "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')", "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields", "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why", - "No documentation task is generated because the feature description does not contain a Documentation Considerations section" + "No documentation task is generated because the feature description does not contain a Documentation Considerations section", + "No testing tasks are generated because no testing readiness template is present in the eval fixture files" ] }, { diff --git a/evals/plan-feature/files/testing-readiness.md b/evals/plan-feature/files/testing-readiness.md new file mode 100644 index 000000000..c3a8487dd --- /dev/null +++ b/evals/plan-feature/files/testing-readiness.md @@ -0,0 +1,15 @@ +# Testing Readiness Template + +Cross-cutting test categories for feature-level validation. + +## Smoke Tests + +- [ ] All new API endpoints return successful responses with valid inputs +- [ ] All modified API endpoints maintain backward compatibility +- [ ] End-to-end workflow completes without errors + +## Performance Benchmarks + +- [ ] API response time is within acceptable thresholds under load +- [ ] No memory leaks detected during sustained usage +- [ ] Database query performance does not degrade with increased data volume diff --git a/plugins/sdlc-workflow/skills/plan-feature/SKILL.md b/plugins/sdlc-workflow/skills/plan-feature/SKILL.md index b5cdb0d5b..3ca5d1c35 100644 --- a/plugins/sdlc-workflow/skills/plan-feature/SKILL.md +++ b/plugins/sdlc-workflow/skills/plan-feature/SKILL.md @@ -484,6 +484,20 @@ For each target repository, identify existing documentation files that may be im Use Glob, `list_dir`, or `search_for_pattern` to locate these files. Record them for use in Step 5 when determining whether tasks need a **Documentation Updates** section. +### Testing readiness template discovery + +For each target repository, check for a testing readiness template at the +conventional path `docs/testing-readiness.md`: + +1. Determine the repository root path from the **Repository Registry** in CLAUDE.md. +2. Check for `docs/testing-readiness.md` at the repository root using Glob or Read + (Serena's `list_dir` if available). +3. If found, read and parse the template. The template format uses `## ` + headings followed by acceptance criteria as bullet points. Each category represents + a cross-cutting test type (e.g., "Smoke Tests", "Performance Benchmarks", "E2E Tests"). +4. Store the parsed categories and their acceptance criteria for use in Step 5. +5. If not found, log silently and proceed — no testing tasks will be generated. + ### Backend API discovery for manual REST calls When a feature involves a frontend repository making REST calls to a backend API — and @@ -703,6 +717,40 @@ which doc files to update based on the description): - Reuse Candidates - Verification Commands +### Testing task generation + +Generate cross-cutting testing tasks when a testing readiness template was +discovered in Step 3. + +1. **Check for testing readiness template**: if no testing readiness template was + discovered in Step 3 for any target repository, skip testing task generation + silently. +2. **Generate testing tasks**: if a template was found, generate one testing task + per test category (`## ` heading) listed in the template. +3. Each testing task follows the `task-description-template.md` format with: + + - `## Repository` — same as the implementation tasks + - `## Target Branch` — follows the same Target Branch assignment rules as other + tasks (main for direct-to-main, feature issue ID for feature-branch) + - `## Description` — what to test, referencing the Feature issue and the test + category's acceptance criteria from the readiness template + - `## Acceptance Criteria` — the acceptance criteria from the readiness template + for that category + - `## Test Requirements` — specific test scenarios derived from the readiness + template + - `## Dependencies` — depends on all implementation tasks (testing tasks run + after implementation is complete) + +4. Testing tasks MUST omit the following sections (these are validation activities, + not code changes): + + - Files to Modify + - Files to Create + - API Changes + - Implementation Notes + - Reuse Candidates + - Verification Commands + ### Convention-aware task enrichment After drafting each task's Implementation Notes, cross-reference the conventions discovered