Feedback loop: user verdicts on findings for false-positive learning

[andthattoo]

Summary

Findings currently flow one direction: agent -> user. Users have no way to signal whether a finding was useful, a false positive, or irrelevant. Adding a feedback mechanism is the highest priority learning feature — without user signal, cross-run memory and style preferences have nothing meaningful to learn from.

Goals

• Let users mark each finding with a verdict: false positive, accepted, or won't fix
• Aggregate verdicts into a per-org FeedbackProfile keyed by (org, vulnerability_class)
• Use the profile to suppress known noise, calibrate severity, and track precision over time

Proposed Changes

1. user_verdict field on ExploitRecord

Add an enum field to the existing ExploitRecord schema:

class UserVerdict(str, Enum):
    PENDING = "pending"
    FALSE_POSITIVE = "false_positive"
    ACCEPTED = "accepted"
    WONT_FIX = "wont_fix"

2. FeedbackProfile model

A new model that aggregates verdicts across runs for an org:

• Per vulnerability class: total count, FP count, average user-assigned severity
• Suppression rules derived from repeated FP verdicts (e.g. "stop reporting informational gas optimizations for this org")
• Precision tracking: FP rate over sliding window of N runs

3. Profile-aware prompt injection

Feed the FeedbackProfile into analyzer/verifier prompts as few-shot examples of what to skip or emphasize:

• Analyzer prompt: "These vulnerability classes have been consistently marked as false positives for this org: ..."
• Verifier prompt: "Severity calibration — this team typically downgrades reentrancy from critical to medium"

4. Precision metrics

Surface to the user: "Your false positive rate dropped from 40% to 15% over 10 runs" — this is what makes users feel the product learning.

Context

• State infrastructure already in place: LocalStateManager, ExploitRecord, RunRecord
• This is the recommended first step from the product learning roadmap (docs/product-learning.md)
• Cross-run memory ( feat: Some big changes #2) and fix style preferences (feat: proper packaging #3) depend on this signal

[aktasbatuhan]

We can wire this to the existing "userLabel" we have for each exploit to manage the Kanban view. Currently, users can update an exploit status as "To be Fixed“, "Fixed”, or "Won't Do". We can map these as:
Won't Do --> false positive / won't fix
To be Fixed / Fixed --> accepted

We also plan to implement GitHub/Jira/Linear webhooks for 3rd-party status updates, so we can use external status as user verdicts in the future.

In addition to these, we can even use GitHub webhooks to fetch PRs where an exploit was fixed, and use that as another piece of information for Profile-aware prompt injection. This way, beyond verdicts, we can even inject the user's development pattern during fixes. However, this is dependent on webhook implementation anyway.