Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions evals/triage-security/evals.json
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,32 @@
"Automated triage stops after the unsupported ecosystem notification — no version impact analysis or remediation task creation is performed for the unsupported ecosystem",
"The notification includes both the ecosystem name and the instruction that manual assessment is required"
]
},
{
"id": 26,
"prompt": "Triage Vulnerability issue TC-8050. The issue details are in vuln-issue-dev-dependency.md, the security matrix is in security-matrix-mock.md, and the project CLAUDE.md is in claude-md-security-config.md. The issue has stream suffix [rhtpa-2.2] so triage is scoped to the 2.2.x stream. The vulnerable library (criterion) is a dev-only dependency — the mock dependency chain data in the fixture shows it is declared in [dev-dependencies] and is NOT present in production builds. Do NOT actually call Jira MCP, git show, or any external tools. Instead, write your triage analysis to the workspace outputs/ directory: write outputs/data-extraction.md with the parsed CVE data table from Step 1, write outputs/version-impact.md with the version impact table from Step 2 including the dependency chain context from Step 2.3.5, and write outputs/remediation.md with the remediation task descriptions showing the dev-dependency label and Normal priority override.",
"expected_output": "A triage analysis where Step 2.3.5 identifies criterion as a dev-only dependency (in [dev-dependencies], not shipped in production). The dependency scope decision tree is applied: remediation tasks are still created (supply chain risk) but with the dev-dependency label and Normal priority instead of inheriting the CVE's priority. The remediation task description includes a note that the dependency is dev/build-only.",
"files": ["files/vuln-issue-dev-dependency.md", "files/security-matrix-mock.md", "files/claude-md-security-config.md"],
"assertions": [
"Step 2.3.5 dependency chain context identifies criterion as a dev-only dependency based on its presence in [dev-dependencies] in the manifest",
"The dependency scope decision tree classifies criterion as 'dev-only — not shipped in production' and applies the dev-dependency remediation handling",
"Remediation tasks include the dev-dependency label in the labels array: labels contain 'dev-dependency' alongside 'ai-generated-jira', 'Security', and the CVE ID",
"Remediation task priority is set to Normal regardless of the CVE severity (CVSS 5.3 Medium) — the priority override is applied per the decision tree",
"Remediation task description includes a note indicating the dependency is dev/build-only and not shipped in production"
]
},
{
"id": 27,
"prompt": "Triage Vulnerability issue TC-8051. The issue details are in vuln-issue-feature-gated.md, the security matrix is in security-matrix-mock.md, and the project CLAUDE.md is in claude-md-security-config.md. The issue has stream suffix [rhtpa-2.2] so triage is scoped to the 2.2.x stream. The vulnerable library (rustls) is an optional dependency gated behind the non-default 'tls-rustls' feature flag — the mock dependency chain data in the fixture shows the default features do NOT include tls-rustls. Do NOT actually call Jira MCP, git show, or any external tools. Instead, write your triage analysis to the workspace outputs/ directory: write outputs/data-extraction.md with the parsed CVE data table from Step 1, write outputs/version-impact.md with the version impact table from Step 2 including the dependency chain context from Step 2.3.5, and write outputs/feature-gate-prompt.md with the VEX justification prompt presented to the user for the feature-gated dependency.",
"expected_output": "A triage analysis where Step 2.3.5 identifies rustls as a feature-gated optional dependency behind the non-default 'tls-rustls' feature. The dependency scope decision tree presents the user with a VEX justification prompt: option 1 to skip remediation with 'Vulnerable Code not in Execute Path' justification, or option 2 to proceed with standard remediation. The prompt includes the library name, feature flag name, and the recommended VEX justification.",
"files": ["files/vuln-issue-feature-gated.md", "files/security-matrix-mock.md", "files/claude-md-security-config.md"],
"assertions": [
"Step 2.3.5 dependency chain context identifies rustls as a feature-gated optional dependency behind the non-default 'tls-rustls' feature flag",
"The dependency scope decision tree presents a VEX justification prompt to the user with two options: skip remediation or proceed with standard remediation",
"The VEX justification prompt recommends 'Vulnerable Code not in Execute Path' as the justification for skipping remediation",
"The VEX justification prompt includes the library name (rustls) and the feature flag name (tls-rustls) so the user has context for the decision",
"If the user chooses to skip remediation, the version is closed as not affected with the VEX justification — no remediation tasks are created for that version"
]
}
]
}
24 changes: 24 additions & 0 deletions evals/triage-security/files/security-matrix-mock.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,30 @@ eval, use this data as the simulated output.
| `v0.4.11` | 0.4.9 |
| `v0.4.12` | 0.4.9 |

## criterion versions by tag

| Tag | criterion version |
|-----|-------------------|
| `v0.3.8` | 0.5.1 |
| `v0.3.12` | 0.5.1 |
| `v0.4.5` | 0.5.1 |
| `v0.4.8` | 0.5.1 |
| `v0.4.9` | _(retag of v0.4.8)_ |
| `v0.4.11` | 0.5.1 |
| `v0.4.12` | 0.5.1 |

## rustls versions by tag

| Tag | rustls version |
|-----|----------------|
| `v0.3.8` | _(not present)_ |
| `v0.3.12` | _(not present)_ |
| `v0.4.5` | 0.23.4 |
| `v0.4.8` | 0.23.4 |
| `v0.4.9` | _(retag of v0.4.8)_ |
| `v0.4.11` | 0.23.4 |
| `v0.4.12` | 0.23.4 |

## openssl-libs versions by tag (rpms.lock.yaml)

The following sections provide the RPM package versions that would be extracted
Expand Down
62 changes: 62 additions & 0 deletions evals/triage-security/files/vuln-issue-dev-dependency.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
<!-- SYNTHETIC TEST DATA — Vulnerability issue for a dev-only dependency (criterion) for triage-security eval testing -->

# Mock Jira Vulnerability Issue

**Key**: TC-8050
**Summary**: CVE-2026-99001 criterion - Path traversal in benchmark output [rhtpa-2.2]
**Issue Type**: Vulnerability
**Status**: New
**Labels**: CVE-2026-99001, pscomponent:org/rhtpa-server
**Affects Versions**: RHTPA 2.2.0
**Due Date**: 2026-08-01
**Assignee**: Unassigned

## Remote Links

- [CVE-2026-99001](https://www.cve.org/CVERecord?id=CVE-2026-99001) — CVE Record

## Comments

_(no comments)_

---

## Description

A vulnerability was found in criterion. The criterion crate before version 0.5.2 allows an attacker to write benchmark output files to arbitrary paths via crafted benchmark names containing path separators.

**Affected package**: criterion
**Affected versions**: versions before 0.5.2
**Fixed version**: 0.5.2
**CVSS**: 5.3 (Medium)

### References

- https://www.cve.org/CVERecord?id=CVE-2026-99001

---

## Mock Dependency Chain Data

The following data simulates what Step 2.3.5 would discover when inspecting
the Cargo.toml manifest files. In a real triage, the skill reads manifests
via `git show`; in this eval, use this data as the simulated output.

### criterion dependency chain for backend

```
Dependency chain for criterion:
backend (workspace) → criterion (direct dev-dependency)
Profile: dev-only ([dev-dependencies] in backend/Cargo.toml)
NOT present in production builds — used for benchmarks only

First appeared: 2.1.0 (initial project setup)
Present in all versions
```

**Manifest evidence:**
```toml
# backend/Cargo.toml (all versions)
[dev-dependencies]
criterion = "0.5.1"
```
75 changes: 75 additions & 0 deletions evals/triage-security/files/vuln-issue-feature-gated.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
<!-- SYNTHETIC TEST DATA — Vulnerability issue for a feature-gated optional dependency for triage-security eval testing -->

# Mock Jira Vulnerability Issue

**Key**: TC-8051
**Summary**: CVE-2026-99002 rustls - Certificate validation bypass [rhtpa-2.2]
**Issue Type**: Vulnerability
**Status**: New
**Labels**: CVE-2026-99002, pscomponent:org/rhtpa-server
**Affects Versions**: RHTPA 2.2.0
**Due Date**: 2026-08-01
**Assignee**: Unassigned

## Remote Links

- [CVE-2026-99002](https://www.cve.org/CVERecord?id=CVE-2026-99002) — CVE Record
- [rustls/rustls#2100](https://github.com/rustls/rustls/pull/2100) — Upstream fix PR

## Comments

_(no comments)_

---

## Description

A vulnerability was found in rustls. The rustls crate before version 0.23.5 improperly validates server certificates when using custom certificate verifiers, allowing a man-in-the-middle attacker to present an invalid certificate chain.

**Affected package**: rustls
**Affected versions**: versions before 0.23.5
**Fixed version**: 0.23.5
**CVSS**: 8.1 (High)

### References

- https://www.cve.org/CVERecord?id=CVE-2026-99002

---

## Mock Dependency Chain Data

The following data simulates what Step 2.3.5 would discover when inspecting
the Cargo.toml manifest files. In a real triage, the skill reads manifests
via `git show`; in this eval, use this data as the simulated output.

### rustls dependency chain for backend

```
Dependency chain for rustls:
backend (workspace) → rustls (direct optional dependency)
Profile: feature-gated (optional = true, behind non-default feature "tls-rustls")
Default features do NOT include "tls-rustls" — the product ships with
the "tls-native" feature enabled by default

Feature declaration:
[features]
default = ["tls-native"]
tls-native = ["dep:native-tls"]
tls-rustls = ["dep:rustls"]

First appeared: 2.2.0 (added as alternative TLS backend)
Not present in: 2.1.x (only native-tls was available)
```

**Manifest evidence:**
```toml
# backend/Cargo.toml (v0.4.5+)
[dependencies]
rustls = { version = "0.23.4", optional = true }

[features]
default = ["tls-native"]
tls-native = ["dep:native-tls"]
tls-rustls = ["dep:rustls"]
```
5 changes: 4 additions & 1 deletion plugins/sdlc-workflow/skills/triage-security/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -799,7 +799,10 @@ MUST include the Comment Footnote (see above).
The upstream task covers the source repo fix; the downstream subtask covers
the Konflux release repo update and is blocked by the upstream task. System
package ecosystems produce a single task. A single Task spanning multiple
streams would be unimplementable by `/implement-task`.
streams would be unimplementable by `/implement-task`. For dev-only or
build-only dependencies (identified in Step 2.3.5), add the `dev-dependency`
label and override priority to Normal — see the dependency scope decision
tree in `version-impact-analysis.md`.
9. **Follow `task-description-template.md` for generated tasks.** The remediation
Task description must be parseable by `/implement-task`. Do not invent custom
sections or deviate from the template format.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ Advisory: [advisory URL from remote links]
that might prevent the bump
- If a direct bump introduces breaking changes, assess whether a
code-level workaround is viable (see upstream changelog)
- If the vulnerable dependency is dev-only or build-only (identified
in Step 2.3.5), the remediation priority is Normal regardless of CVE
severity. Add `dev-dependency` label to the task.

## Acceptance Criteria

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,8 @@ Inspect the lock file and manifest files to determine:
- **npm**: `devDependencies` (build/test only), `optionalDependencies`

If the dependency is only present in a non-production profile (e.g., dev-only),
note this — it changes the risk assessment.
note this — it changes the risk assessment and remediation handling per the
decision tree below.
4. **Introduction point** — if a dependency is present in one version but not
another, note when it was introduced (helps identify which upgrade or feature
addition brought it in)
Expand All @@ -204,6 +205,56 @@ First appeared: 2.2.0 (commit 05a3af91 added reqwest http3 feature)
Not present in: 2.1.x (reqwest used without http3 feature)
```

#### Dependency scope decision tree

When the profile/scope analysis (item 3 above) identifies a non-production
dependency, apply the following decision tree to determine remediation handling:

**Dev-only or build-only dependencies (not shipped in production):**

These are dependencies that do not appear in the production binary or container
image:

- **Cargo** `[dev-dependencies]` — used for tests and benchmarks only; NOT shipped
- **Cargo** `[build-dependencies]` — used in build scripts only; NOT shipped
- **npm** `devDependencies` — used for build/test only; NOT shipped (unless
bundled by the build tool — verify bundler configuration if uncertain)

Even though dev-only dependencies are not shipped, they still represent a supply
chain risk (compromised dev deps can inject malicious code during builds). Still
create remediation tasks, but with these modifications:

- Add the `dev-dependency` label to the remediation task
- Set priority to **Normal** regardless of the CVE severity — do not inherit
the CVE's priority
- Include a note in the remediation task description: "This dependency is
dev/build-only and is not shipped in production. Remediation priority is
Normal (supply chain risk only)."

**Feature-gated optional dependencies:**

These are dependencies declared in `[dependencies]` with `optional = true` and
gated behind a non-default feature flag in Cargo, or `optionalDependencies`
in npm that are not enabled by default.

Before creating remediation tasks, present the user with a VEX justification
option:

> "The vulnerable dependency `[library]` is gated behind the `[feature-flag]`
> feature, which is not enabled by default. Recommended VEX justification:
> **Vulnerable Code not in Execute Path**.
>
> Options:
> 1. Skip remediation — apply VEX justification and close as not affected
> 2. Proceed with remediation — create tasks despite the feature gate
>
> Choose (1/2):"

If the user chooses option 1, close the version as not affected with VEX
justification "Vulnerable Code not in Execute Path" (the overall issue closure
depends on whether other versions are affected). If the user chooses option 2,
create standard remediation tasks without label or priority modifications.

#### Container-level dependencies (RPM, system packages)

For RPM and system-level packages, classify origin to determine the remediation
Expand Down
Loading