Conversation
Add auto-discovery of docs/testing-readiness.md in target repositories during Step 3, and generate one cross-cutting testing task per test category in the template during Step 5. Testing tasks depend on all implementation tasks and omit code-change sections. Includes 3 new constraints (§1.85–1.87) with traceability, eval fixture with 2 test categories, and positive/negative eval assertions. Implements TC-4878 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Assisted-by: Claude Code
Reviewer's GuideAdds testing readiness template discovery to plan-feature Step 3, introduces testing task generation to Step 5 backed by new constraints, and wires in an eval fixture and traceability updates to validate positive/negative behavior. Flow diagram for testing readiness discovery and testing task generationflowchart LR
subgraph Step3_Testing_readiness_template_discovery
A[Get repository_root from Repository_Registry]
B{docs/testing-readiness.md exists?}
A --> B
B -->|yes| C[Read testing-readiness.md]
C --> D[Parse headings as test_categories]
D --> E[Store test_categories for Step5]
B -->|no| F[Log silently and continue]
end
subgraph Step5_Task_generation
G[Generate implementation_tasks]
H{test_categories available?}
G --> H
H -->|no| I[Skip testing_task generation]
H -->|yes| J[For each test_category generate testing_task]
J --> K[Set testing_task.dependencies = all implementation_tasks]
end
Step3_Testing_readiness_template_discovery --> Step5_Task_generation
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've left some high level feedback:
- The testing readiness template discovery and task generation logic could better specify behavior when some target repositories have
docs/testing-readiness.mdand others do not (e.g., per-repository task generation vs. all-or-nothing), to avoid ambiguity with constraints §1.85–1.86. - Consider defining how to handle edge cases in
docs/testing-readiness.mdparsing, such as missing##headings, empty categories, or duplicate category names, so the skill’s behavior remains predictable with imperfect templates. - The requirement that testing tasks omit Verification Commands may make it harder to indicate how tests should actually be run; you might want to clarify where execution details should live for these tasks to keep the workflow consistent.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The testing readiness template discovery and task generation logic could better specify behavior when some target repositories have `docs/testing-readiness.md` and others do not (e.g., per-repository task generation vs. all-or-nothing), to avoid ambiguity with constraints §1.85–1.86.
- Consider defining how to handle edge cases in `docs/testing-readiness.md` parsing, such as missing `##` headings, empty categories, or duplicate category names, so the skill’s behavior remains predictable with imperfect templates.
- The requirement that testing tasks omit Verification Commands may make it harder to indicate how tests should actually be run; you might want to clarify where execution details should live for these tasks to keep the workflow consistent.Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
There was a problem hiding this comment.
Eval Results
Eval Results: plan-feature
| Eval | Passed | Failed | Pass Rate |
|---|---|---|---|
| eval-1 | 14/19 | 5 | 74% |
| eval-2 | 12/16 | 4 | 75% |
| eval-3 | 11/15 | 4 | 73% |
| eval-4 | 6/11 | 5 | 55% |
| eval-5 | 10/15 | 5 | 67% |
| eval-6 | 5/14 | 9 | 36% |
Failed Assertions
eval-1: 5 failing assertions
-
Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
Evidence: "Searched all 9 output files for 'sha256', 'digest', and 'sha-256' — no matches found. No evidence of description digest comments in any output file. The outputs contain only the task description files and impact map with no digest markers or comment records." -
Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "Searched all output files for 'convention', 'CONVENTIONS', and 'Applies:' — found only incidental uses of the word 'conventions' in task-1 ('field conventions') and task-3 ('repo conventions'), neither of which are convention applicability rationales in the prescribed format. No task contains 'Per CONVENTIONS.md' references or 'Applies: task modifies <file> matching the convention's <scope>' rationales as required by convention-applicability-rules.md. The repo-backend.md manifest lists a CONVENTIONS.md file, so conventions should have been evaluated for applicability." -
Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
Evidence: "Feature fixture (feature-standard.md) has 'Priority: Major'. Searched all output files for 'priority', 'Major', and 'additional_fields' — no matches found. No output file contains evidence that priority was propagated to tasks via additional_fields." -
Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
Evidence: "Feature fixture has 'Fix Versions: RHTPA 1.5.0'. CLAUDE.md has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning fixVersions should be propagated to tasks. Searched all output files for 'fixVersion', 'RHTPA', and 'additional_fields' — no matches found. No output file contains evidence that fixVersions was propagated to tasks." -
Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
Evidence: "No summary comment file exists in the outputs directory. The outputs contain only: impact-map.md and task-1 through task-8 description files. Searched for 'summary comment', 'propagat', 'step 6c' — no matches found. No evidence of a summary comment being generated or posted."
eval-2: 4 failing assertions
-
Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
Evidence: "No sha256 digest, hash, or '[sdlc-workflow] Description digest' marker appears anywhere in the 5 output files (impact-map.md, task-1 through task-4). grep for 'sha256', 'digest', and 'hash' across all output files returned no matches. No separate digest or comment files exist in the outputs directory." -
Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
Evidence: "The feature has Priority: Normal. No output file contains any reference to 'priority', 'additional_fields', or 'Normal'. grep for 'priority' across all output files returned no matches. There is no evidence that priority was included in any task creation parameters." -
Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
Evidence: "The feature has Fix Versions: RHTPA 1.6.0, and CLAUDE.md has no Jira Field Defaults section (defaulting to 'both'), so fixVersions should be propagated. No output file contains any reference to 'fixVersion', 'RHTPA 1.6.0', or 'additional_fields'. grep for 'fixversion' (case-insensitive) across all output files returned no matches." -
Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
Evidence: "No summary comment file exists in the outputs directory. The only files are impact-map.md and task-1 through task-4.md. None of these files contain a summary comment referencing priority inheritance, fixVersion propagation, or any Step 6c content."
eval-3: 4 failing assertions
-
Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
Evidence: "No output file contains any SHA-256 digest, '[sdlc-workflow] Description digest:' marker, 'sha256-md:', or 'sha256-adf:' string. The outputs directory contains only 9 files (impact-map.md, task-1 through task-8 markdown files) — none of which include evidence of digest comments being posted after task creation." -
Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
Evidence: "The Feature issue has Priority: Critical, so every task should inherit priority: Critical in additional_fields. No output file contains any mention of 'additional_fields', 'priority: Critical', or any evidence that the priority field was propagated to the created tasks. The task description files contain only markdown descriptions without Jira API field metadata." -
Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
Evidence: "The Feature issue has Fix Versions: RHTPA 1.5.0, and CLAUDE.md has no Jira Field Defaults section (defaulting scope to 'both'), so every task should include fixVersions: [RHTPA 1.5.0] in additional_fields. No output file contains any mention of 'fixVersions', 'RHTPA 1.5.0', 'additional_fields', or any evidence that the fix version was propagated to the created tasks." -
Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
Evidence: "No output file contains a summary comment for the feature issue. There is no file referencing Step 6c, no summary of propagated priority/fixVersion values, and no explanation for their omission. The outputs directory contains only the impact map and 8 task description files — no summary comment file exists."
eval-4: 5 failing assertions
-
Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
Evidence: "grep for 'sha256', 'digest', and 'Description digest' across all output files returned NO MATCHES. None of the 5 output files (impact-map.md, task-1 through task-4) contain any SHA-256 hash, digest comment, or '[sdlc-workflow] Description digest:' marker. No separate comment files exist in the outputs directory." -
Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "No convention-enrichment entries using the prescribed format appear in any task description. Tasks 1-3 reference repository patterns with free-form prose (task-1 line 30: 'Per the Key Conventions in the repository: all error handling should use...'; task-2 line 61: 'Per repository conventions: all handlers return...'; task-3 line 36: 'Per repository conventions: integration tests hit...') but none use the prescribed 'Applies: task modifies <file> matching the convention's <scope>' rationale format. No 'Per CONVENTIONS.md §...' references appear anywhere. While no inapplicable conventions are annotated with 'Not applicable' (satisfying that part), the convention references that do exist use free-form prose instead of the prescribed format, and no evidence of formal convention-applicability validation exists." -
Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
Evidence: "The feature fixture (feature-adversarial.md line 6) sets Priority: Major. grep for 'priority' (case-insensitive) across all task-*.md files returned NO MATCHES. None of the 4 task descriptions contain an 'additional_fields' section or any reference to priority inheritance. The priority 'Major' from the feature was not propagated to any task." -
Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
Evidence: "The feature fixture (feature-adversarial.md line 7) sets Fix Versions: RHTPA 1.5.0. The CLAUDE.md project config has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning tasks should inherit fixVersions. grep for 'fixversion' and 'fix.version' (case-insensitive) across all task-*.md files returned NO MATCHES. None of the 4 task descriptions contain an 'additional_fields' section or any reference to fixVersion inheritance. The fixVersion 'RHTPA 1.5.0' from the feature was not propagated to any task." -
Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
Evidence: "No summary comment file exists in the outputs directory. The only files present are impact-map.md and task-1 through task-4. grep for 'priority' and 'fixversion' (case-insensitive) returned no matches in any output file. There is no evidence of a summary comment being generated for the feature issue, let alone one that documents priority (Major) and fixVersion (RHTPA 1.5.0) propagation."
eval-5: 5 failing assertions
-
Assertion: "Each non-documentation task file contains all required template sections: Repository, Target Branch, Description, at least one of Files to Modify or Files to Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation tasks are exempt from requiring Files to Modify, Files to Create, and Implementation Notes — they must still include Repository, Target Branch, Description, Acceptance Criteria, and Test Requirements"
Evidence: "Intermediate non-documentation tasks (2-6) all contain the required sections: Repository, Target Branch, Description, Files to Modify/Create, Implementation Notes, Acceptance Criteria, Test Requirements. Documentation task (Task 7) correctly includes Repository, Target Branch, Description, Acceptance Criteria, Test Requirements (exempt from Files to Modify/Create and Implementation Notes). However, the bookend tasks (Task 1 and Task 8) are non-documentation tasks that lack 'Files to Modify'/'Files to Create' and 'Implementation Notes' sections. Task 1 has: Repository, Target Branch, Bookend Type, Description, Acceptance Criteria, Test Requirements, Dependencies — missing Files to Modify/Create and Implementation Notes. Task 8 similarly lacks these sections. The assertion only exempts documentation tasks, not bookend tasks." -
Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
Evidence: "No evidence of description digest comments in any output file. The outputs consist of 8 task files and 1 impact-map.md. None contain '[sdlc-workflow] Description digest: sha256-' markers or any SHA-256 hashes. No log files, comment files, or other artifacts showing digest comments were posted after task creation." -
Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
Evidence: "The feature issue has 'Priority: High' (feature-feature-branch.md line 8). However, none of the 8 task files contain an 'additional_fields' section or any mention of 'priority: High' being inherited. No task file includes the word 'additional_fields'. The priority value 'High' is not propagated to any task." -
Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
Evidence: "The feature issue has 'Fix Versions: RHTPA 2.0.0' (feature-feature-branch.md line 9). CLAUDE.md has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning fixVersions should be propagated to all tasks. However, none of the 8 task files contain an 'additional_fields' section or any mention of 'fixVersions' or 'RHTPA 2.0.0'. The fixVersion value is not propagated to any task." -
Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
Evidence: "No summary comment file exists in the outputs directory. The outputs contain only 8 task files and 1 impact-map.md. The impact-map.md does not include any mention of priority ('High') or fixVersion ('RHTPA 2.0.0') propagation. No file in the outputs directory contains a summary comment addressing priority or fixVersion inheritance."
eval-6: 8 failing assertions
-
Assertion: "Each task specifies its parent Epic key in the output — tasks targeting trustify-backend are assigned to the backend Epic, tasks targeting trustify-ui are assigned to the frontend Epic"
Evidence: "No task file (task-1 through task-8) contains any reference to a parent Epic key. The tasks specify a 'Repository' section and 'Dependencies' section, but none mention an Epic key (e.g., TC-XXXX). The impact-map.md assigns tasks to Epics conceptually ('Tasks 1, 2, 3' to Epic 1, 'Tasks 4, 5, 6, 7' to Epic 2), but the individual task files themselves do not specify their parent Epic key." -
Assertion: "Incorporates links are created from the Feature to each Epic (not from Feature to individual Tasks)"
Evidence: "No output file contains any reference to 'Incorporates' links, link types, or link creation from the Feature (TC-9006) to any Epic. Searched all output files for 'Incorporates', 'incorporates', and 'link type' with no results." -
Assertion: "Epics are created with the level-1 issue type name ('Epic') and parent set to the feature issue key"
Evidence: "No output file contains evidence that Epics were created with issue type 'Epic' or with parent set to TC-9006. The impact-map.md lists 'Epic 1' and 'Epic 2' in a table but does not specify the Jira issue type name or parent field configuration. No Jira API creation details or Epic issue keys are present in any output file." -
Assertion: "After each task is created, a description digest comment is posted with a format-tagged SHA-256 hash — exactly 64 lowercase hex characters prefixed by 'sha256-md:' or 'sha256-adf:', not a placeholder, abbreviated value, or example string. Marker format: '[sdlc-workflow] Description digest: sha256-md:<64-char-hex>' (or sha256-adf). The digest is computed by re-fetching the description from the API and running scripts/sha256-digest.py"
Evidence: "No output file contains any reference to 'sha256', 'digest', 'Description digest', or '[sdlc-workflow]'. Searched all files in /tmp/plan-feature-eval-pr/eval-6/outputs/ with no matches. No digest comments were posted for any of the 8 tasks." -
Assertion: "Convention-aware enrichment validates file-type applicability per shared/convention-applicability-rules.md before including a convention — inapplicable conventions are excluded entirely (not listed with 'Not applicable' annotations), and applicable ones include a rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'), not free-form prose"
Evidence: "No task file contains convention applicability rationale in the prescribed format ('Applies: task modifies <file> matching the convention's <scope>'). Searched all output files for 'Applies:' and 'convention' — only references found are mentions of 'Conventional Commits format' in Implementation Notes and 'error handling convention', which are free-form prose references to coding patterns, not convention-applicability-rules.md formatted rationales." -
Assertion: "When the Feature issue has a priority set (not 'Undefined'), every created task's additional_fields includes 'priority' with the inherited priority name. When the Feature's priority is 'Undefined', the priority key is omitted entirely from additional_fields (not set to null or 'Undefined')"
Evidence: "The Feature TC-9006 has priority 'Major' (from feature-epic-hierarchy.md). However, no task file contains any reference to 'priority', 'Major', or 'additional_fields'. Searched all output files for these terms with no relevant matches. The priority was not inherited or propagated to any task." -
Assertion: "When the Feature issue has a non-empty fixVersions array and the fixVersion scope config (from ### Jira Field Defaults in CLAUDE.md) is 'task' or 'both' (or absent, defaulting to 'both'), every created task's additional_fields includes 'fixVersions' with the inherited version(s). When fixVersion scope is 'feature' or the Feature has no fixVersions, the fixVersions key is omitted entirely from additional_fields"
Evidence: "The Feature TC-9006 has fixVersions 'RHTPA 1.5.0' (from feature-epic-hierarchy.md). The CLAUDE.md has no '### Jira Field Defaults' section, so fixVersion scope defaults to 'both', meaning tasks should inherit fixVersions. However, no task file contains any reference to 'fixVersions', 'RHTPA 1.5.0', or 'additional_fields'. The fixVersions were not inherited or propagated to any task." -
Assertion: "The summary comment on the feature issue (Step 6c) includes the inherited priority and fixVersion values that were propagated to tasks, or states they were omitted and why"
Evidence: "No output file contains a summary comment for the feature issue. Searched all output files for 'summary comment', 'Step 6', 'priority', 'fixVersion', and 'RHTPA' with no relevant matches. No summary comment was generated or posted on the feature issue."
Pass rate: 63% · Tokens: 68,231 · Duration: 277s
Baseline (be2d4411): 87% · 82,759 tokens · 389s
Generated by sdlc-workflow/run-evals v0.12.0
|
[sdlc-workflow/verify-pr] Re: @sourcery-ai[bot] review — Classified as suggestion — these propose specification improvements (edge case handling, multi-repo behavior, execution detail placement) that are not backed by documented conventions or established codebase patterns. No sub-task created. |
Verification Report for TC-4878 (commit cc97b16)
Overall: PASSAll checks pass. Test Quality WARN is informational and does not affect the overall result. Eval Quality AnalysisAll 5 new assertions introduced by this PR pass (2 testing-task positive assertions in eval-1, 1 testing-task dependency assertion in eval-1, 1 negative assertion in eval-2, plus the TC-4877 documentation assertion in eval-2). Overall eval pass rate dropped from 87% (baseline
Regression vs non-determinism: Evals 3–6 were not modified by this PR yet show significant variance — eval-6 dropped from 100% to 36% and eval-3 from 100% to 73% with zero prompt or assertion changes. The same assertion types fail across all evals (digest comments, convention enrichment format, priority/fixVersion inheritance, summary comment). This pattern indicates non-deterministic LLM eval variance rather than regressions from this PR's code changes. Eval-1 caveat: The prompt change (adding testing readiness template context) may contribute to eval-1's 5 new failures on previously-passing assertions, as the LLM may allocate effort differently with additional instructions. However, these same assertion types also regress in unchanged evals, suggesting the primary cause is variance. This comment was AI-generated by sdlc-workflow/verify-pr v0.12.0. |
Summary
docs/testing-readiness.md) to plan-feature Step 3docs/constraints.mdwith traceability index updateImplements TC-4878
Test plan
🤖 Generated with Claude Code
Summary by Sourcery
Add support in the plan-feature skill for generating testing tasks based on a repository testing readiness template and document the corresponding constraints and traceability.
New Features:
Documentation:
Tests: