Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions .opencode/agents/validator.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ Before validating a finding, read:
- `.opencode/skills/finding-format/SKILL.md`
- `.opencode/skills/exploit-validation/SKILL.md`
- `.opencode/skills/sandbox-validation/SKILL.md`
- If present, `itemdb/notes/threat-model.md` — the operational threat model from Phase 1b. Use it to ensure validation planning respects documented attacker capabilities, non-capabilities, trust boundaries, existing controls, and open assumptions.
Comment thread
pruiz marked this conversation as resolved.
- relevant files under `itemdb/notes/`
- the assigned finding
- relevant source files under `src/`
Expand Down Expand Up @@ -117,13 +118,14 @@ Multiple methods may be combined.
2. Extract the exact vulnerability claim.
3. Review the counter-analysis.
4. Identify what evidence would confirm or reject the claim.
5. Inspect the relevant source code.
6. Prepare the sandbox.
7. Execute the validation plan or improve it.
8. Record commands and outputs.
9. Store evidence under `itemdb/evidence/<finding-id>/`.
10. Update the finding.
11. Move the finding to the correct status directory if needed.
5. When the threat model exists, review how the finding's validation plan aligns with documented attacker capabilities, trust boundaries, existing controls, and open assumptions. Refine the plan if it assumes capabilities contradicted by the threat model.
6. Inspect the relevant source code.
7. Prepare the sandbox.
8. Execute the validation plan or improve it.
9. Record commands and outputs.
10. Store evidence under `itemdb/evidence/<finding-id>/`.
11. Update the finding.
12. Move the finding to the correct status directory if needed.

## Evidence directory

Expand Down
16 changes: 9 additions & 7 deletions .opencode/skills/exploit-validation/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Read:
- the assigned finding
- relevant source files under `src/`
- sandbox documentation under `sandbox/`
- If present, `itemdb/notes/threat-model.md` — the operational threat model from Phase 1b
Comment thread
pruiz marked this conversation as resolved.

## Outputs

Expand Down Expand Up @@ -104,13 +105,14 @@ For the assigned finding:
1. Read the finding completely.
2. Understand the hypothesis and counter-analysis.
3. Identify the minimal proof needed.
4. Inspect relevant source code.
5. Determine the best validation method.
6. Prepare the sandbox.
7. Execute validation steps.
8. Capture commands, inputs, outputs, logs, and observations.
9. Decide whether the finding is confirmed, rejected, or unresolved.
10. Update the finding and evidence directory.
4. When the threat model is present, review how the finding's claimed attacker profile, trust boundary, and impact align with the threat model's documented capabilities, non-capabilities, and existing controls.
5. Inspect relevant source code.
6. Determine the best validation method.
7. Prepare the sandbox.
8. Execute validation steps.
9. Capture commands, inputs, outputs, logs, and observations.
10. Decide whether the finding is confirmed, rejected, or unresolved.
11. Update the finding and evidence directory.

## Evidence directory

Expand Down
22 changes: 15 additions & 7 deletions prompts/phase-4-validate.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Read the following files (all paths are relative to the project/workspace root):
- the assigned finding
- relevant source files under `src/`
- sandbox documentation under `sandbox/`
- If present, `itemdb/notes/threat-model.md` — operational threat model from Phase 1b: assets, attacker capabilities and non-capabilities, trust-boundary summary, existing controls, abuse-path themes, risk calibration, and open assumptions. Use it to plan realistic validation attempts.
Comment thread
pruiz marked this conversation as resolved.

Use additional target-specific skills only if they clearly apply.

Expand All @@ -59,13 +60,19 @@ Do not create unrelated findings.
2. Extract the exact vulnerability claim.
3. Review the existing counter-analysis.
4. Identify what evidence would confirm or reject the claim.
5. Inspect relevant source files.
6. Prepare the sandbox under `sandbox/`.
7. Execute the validation plan or improve it.
8. Capture commands, inputs, outputs, logs, and observations.
9. Store evidence under `itemdb/evidence/<finding-id>/`.
10. Update the finding.
11. Move the finding to the correct status directory if needed.
5. When available, consult `itemdb/notes/threat-model.md` before finalizing the validation approach:
- Attacker capabilities: validate against the described attacker profile. Do not design a validation that assumes capabilities explicitly excluded as non-capabilities.
- Trust boundaries: ensure the validation crosses documented trust boundaries. A finding that does not cross a documented boundary may still be valid but requires additional justification.
- Existing controls: identify controls that may block or narrow the validation path and factor them into the plan.
- Affected assets: map the finding's claimed impact to documented assets and security objectives.
- Open assumptions: note any threat-model assumptions that change exploitability or validation feasibility, and record them in the evidence.
6. Inspect relevant source files.
7. Prepare the sandbox under `sandbox/`.
8. Execute the validation plan or improve it.
9. Capture commands, inputs, outputs, logs, and observations.
10. Store evidence under `itemdb/evidence/<finding-id>/`.
11. Update the finding.
12. Move the finding to the correct status directory if needed.

## Evidence requirements

Expand Down Expand Up @@ -191,6 +198,7 @@ At the end, summarize:
- result: CONFIRMED / REJECTED / UNRESOLVED,
- evidence files created,
- finding file moved or updated,
- threat-model assumptions that materially affected validation strategy or evidence interpretation,
- open questions for the user (same content as in the run summary),
- re-run prompt hints (same content as in the run summary; use `PROMPT_EXTRA` / `PROMPT_EXTRA_FILE` snippets),
- remaining limitations.
Expand Down
13 changes: 13 additions & 0 deletions templates/evidence-readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,19 @@ Include:
- relevant configuration,
- date of validation.

# Threat-model assumptions (if applicable)

When the validation result was materially affected by assumptions from the threat model, document them here.

Examples:

- Attacker capability constrained by threat-model non-capabilities
- Trust boundary documented in `itemdb/notes/threat-model.md` that shaped the validation path
- Existing control from the threat model that blocked or narrowed validation
- Open assumption from the threat model that affected exploitability assessment

If the threat model did not affect the result, this section may be omitted.

# Commands executed

List the exact commands executed.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@ def _read_prompt(name: str) -> str:
return path.read_text(encoding="utf-8")


def _read_opencode(path_from_root: str) -> str:
path = ROOT / path_from_root
assert path.is_file(), f"{path} does not exist"
return path.read_text(encoding="utf-8")


def test_phase_1c_recon_prompt_exists() -> None:
path = ROOT / "prompts" / "phase-1c-recon.md"
assert path.is_file(), f"{path} does not exist"
Expand Down Expand Up @@ -100,3 +106,71 @@ def test_phase_1_recon_md_removed() -> None:
def test_phase_1b_codeql_recon_md_removed() -> None:
path = ROOT / "prompts" / "phase-1b-codeql-recon.md"
assert not path.exists(), f"{path} should have been renamed"



# ---------------------------------------------------------------------------
# Phase 4 validator agent and skill — threat-model.md integration
# ---------------------------------------------------------------------------

def test_validator_agent_references_threat_model() -> None:
content = _read_opencode(".opencode/agents/validator.md")
assert "itemdb/notes/threat-model.md" in content


def test_validator_agent_uses_conditional_language() -> None:
content = _read_opencode(".opencode/agents/validator.md")
content_lower = content.lower()
assert (
"when available" in content_lower
or "when present" in content_lower
or "if present" in content_lower
)


Comment thread
pruiz marked this conversation as resolved.
Comment thread
pruiz marked this conversation as resolved.
def test_exploit_validation_skill_references_threat_model() -> None:
content = _read_opencode(".opencode/skills/exploit-validation/SKILL.md")
assert "itemdb/notes/threat-model.md" in content


def test_exploit_validation_skill_mentions_attacker_capabilities() -> None:
content = _read_opencode(".opencode/skills/exploit-validation/SKILL.md")
content_lower = content.lower()
assert "attacker" in content_lower
assert "non-capabilities" in content_lower

Comment thread
pruiz marked this conversation as resolved.
Comment thread
pruiz marked this conversation as resolved.

# ---------------------------------------------------------------------------
# Phase 4 — threat-model.md integration
# ---------------------------------------------------------------------------

def test_phase_4_explicitly_references_threat_model_when_present() -> None:
content = _read_prompt("phase-4-validate.md")
assert "itemdb/notes/threat-model.md" in content


def test_phase_4_uses_conditional_when_available_language() -> None:
content = _read_prompt("phase-4-validate.md")
content_lower = content.lower()
assert (
"when this file is available" in content_lower
or "when available" in content_lower
or "when present" in content_lower
or "if present" in content_lower
)


def test_phase_4_mentions_attacker_capabilities_and_non_capabilities() -> None:
content = _read_prompt("phase-4-validate.md")
assert "attacker capabilit" in content.lower()
assert "non-capabilities" in content.lower()


def test_phase_4_mentions_trust_boundaries_in_validation_context() -> None:
content = _read_prompt("phase-4-validate.md")
assert "trust boundar" in content.lower()


def test_phase_4_mentions_existing_controls() -> None:
content = _read_prompt("phase-4-validate.md")
assert "existing controls" in content.lower()
Comment thread
pruiz marked this conversation as resolved.
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Loading