feat(evals): add full-workflow idempotency eval for triage-security re-runs

[ruromero]

Summary

• Adds eval ID 18 to evals/triage-security/evals.json — tests that re-running triage on an already-triaged CVE produces no duplicate mutations
• Creates fixture vuln-issue-standard-already-triaged.md mirroring the standard CVE (TC-8001) with post-triage state: ai-cve-triaged label, In Progress status, existing remediation task links, digest comment, and summary comment
• Five assertions cover idempotency of task creation, label addition, status transition, digest comments, and summary comments

Resolves TC-4904

Test plan

• Eval 18 passes when run against the triage-security skill
• Existing evals (1-17, 19-24) are unaffected
• JSON is valid (verified via python3 -c "import json; json.load(open('evals/triage-security/evals.json'))")

🤖 Generated with Claude Code

Summary by Sourcery

Add a new triage-security evaluation to verify idempotent re-runs on already-triaged vulnerability issues.

New Features:

• Introduce eval ID 18 for triage-security to ensure rerunning triage on an already-triaged CVE does not create duplicate artifacts.
• Add an already-triaged vulnerability issue fixture capturing post-triage labels, status, links, and comments for idempotency testing.

[sourcery-ai]

Reviewer's Guide

Adds a new idempotency eval (ID 18) for triage-security by introducing a pre-triaged CVE fixture and wiring it into the triage-security evals JSON configuration.

File-Level Changes

Change	Details	Files
Add eval 18 definition to triage-security eval suite to verify idempotent re-runs on already-triaged CVEs.	Register new eval entry with ID 18 in evals.json for triage-security. Configure eval 18 to use an already-triaged vulnerability issue fixture as input. Document eval purpose as checking for absence of duplicate mutations across tasks, labels, status, digest comments, and summary comments.	evals/triage-security/evals.json
Introduce synthetic already-triaged vulnerability issue fixture used by the new idempotency eval.	Create Markdown fixture modeling TC-8001 in post-triage state, including labels, status, custom fields, links, and comments. Embed existing remediation task issue links to represent pre-existing upstream and downstream tasks. Include prior digest and post-triage summary comments to ensure eval checks that re-runs do not duplicate them.	evals/triage-security/files/vuln-issue-standard-already-triaged.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've reviewed your changes and they look great!

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[github-actions]

Eval Results

Eval Results: triage-security

Eval	Passed	Failed	Pass Rate
eval-1	11/11	0	100%
eval-10	5/5	0	100%
eval-11	5/5	0	100%
eval-12	5/5	0	100%
eval-13	5/5	0	100%
eval-14	5/5	0	100%
eval-15	5/5	0	100%
eval-16	7/7	0	100%
eval-17	5/5	0	100%
eval-18	5/5	0	100%
eval-19	5/5	0	100%
eval-2	5/5	0	100%
eval-20	4/4	0	100%
eval-21	4/4	0	100%
eval-22	4/4	0	100%
eval-23	4/4	0	100%
eval-24	4/4	0	100%
eval-25	4/4	0	100%
eval-3	5/5	0	100%
eval-4	5/5	0	100%
eval-5	6/6	0	100%
eval-6	6/6	0	100%
eval-7	5/5	0	100%
eval-8	8/8	0	100%
eval-9	5/5	0	100%

Pass rate: 100% · Tokens: 49,377 · Duration: 137s

Baseline (40c31ac8): 98% · 54,987 tokens · 132s

---

Generated by sdlc-workflow/run-evals v0.11.1

[ruromero]

Verification Report for TC-4904 (commit 9442582)

Check	Result	Details
Review Feedback	N/A	No review comments to classify
Root-Cause Investigation	N/A	No sub-tasks created
Scope Containment	PASS	PR files match task spec exactly (2/2 files)
Diff Size	PASS	94 lines across 2 files — proportionate for 1 eval entry + 1 fixture
Commit Traceability	FAIL	Commit 9442582 does not reference TC-4904 in headline or body
Sensitive Patterns	PASS	No secrets or credentials in added lines
CI Status	PASS	All 4 CI checks pass
Acceptance Criteria	PASS	3/3 criteria met — eval 18 covers full re-run idempotency
Test Quality	PASS	Eval Quality: 100% (25/25 evals, up from 98% baseline)
Test Change Classification	N/A	No test files in diff
Verification Commands	N/A	No verification commands specified; no eval infrastructure changes

Overall: FAIL

Commit traceability failure: commit 9442582e uses headline feat(evals): add full-workflow idempotency eval for triage-security re-runs without referencing the Jira task ID TC-4904. The branch name provides implicit traceability but does not satisfy commit-level traceability.

All other checks pass. The new eval-18 passes 5/5 assertions at 100%, and the overall eval suite improved from 98% to 100% pass rate vs baseline.

---

This comment was AI-generated by sdlc-workflow/verify-pr v0.11.0.

[ruromero]

Verification Report for TC-4904 (commit 680a5c8)

Check	Result	Details
Review Feedback	N/A	No review comments to classify
Root-Cause Investigation	N/A	No sub-tasks created
Scope Containment	PASS	PR files match task spec exactly (2/2 files)
Diff Size	PASS	94 lines across 2 files — proportionate for 1 eval entry + 1 fixture
Commit Traceability	PASS	Commit 680a5c8 references TC-4904 in headline
Sensitive Patterns	PASS	No secrets or credentials in added lines
CI Status	PASS	All 4 CI checks pass
Acceptance Criteria	PASS	3/3 criteria met — eval 18 covers full re-run idempotency
Test Quality	PASS	Eval Quality: 100% (25/25 evals, up from 98% baseline)
Test Change Classification	N/A	No test files in diff
Verification Commands	N/A	No verification commands specified; no eval infrastructure changes

Overall: PASS

All checks pass. The new eval-18 passes 5/5 assertions at 100%, and the overall eval suite improved from 98% to 100% pass rate vs baseline.

Updated from prior FAIL report — commit traceability resolved by amending commit to include TC-4904 in the headline.

---

This comment was AI-generated by sdlc-workflow/verify-pr v0.11.0.