Store LLM Patch Outcomes in the Database

[arpit2006]

Summary

PatchPilot can generate AI-assisted patches using Ollama, but there is currently no mechanism to persist information about generated patches or whether they successfully passed verification.

Persisting these outcomes is essential for analytics, future reinforcement learning workflows, model evaluation, and contributor insights. Every generated patch should be recorded together with its verification status so the system can measure the effectiveness of AI-generated fixes over time.

---

Affected Files

Implementation will primarily involve:

• backend/app/database.py
• backend/app/main.py
• backend/app/models/
• backend/app/ml/
• backend/tests/
• CONTRIBUTING.md

---

Root Cause

Issue A — Generated Patches Are Not Persisted

After an LLM generates a patch, the result exists only for the duration of the request.

Once the response is returned, there is no record of:

• which model generated the patch,
• which finding it addressed,
• or whether it was eventually successful.

---

Issue B — Verification Results Are Lost

The /verify pipeline determines whether a generated patch successfully resolves a vulnerability.

However, these verification results are not associated with the original LLM-generated patch, making it impossible to evaluate model performance over time.

---

Issue C — No Training Signal

Future Tier 3 improvements depend on historical verification outcomes.

Without storing successful and failed patch attempts, there is no reliable dataset for:

• model evaluation,
• reward modeling,
• benchmarking,
• future fine-tuning,
• or reporting AI effectiveness.

---

Issue D — Missing Documentation

The new database table should be documented so contributors understand its purpose, schema, and how it fits into the Tier 3 roadmap.

---

Expected Behavior

Create the following table during application startup:

CREATE TABLE IF NOT EXISTS patch_outcomes (
    id              TEXT PRIMARY KEY,
    job_id          TEXT NOT NULL,
    finding_id      TEXT NOT NULL,
    model           TEXT NOT NULL,
    passed          INTEGER,
    prompt_tokens   INTEGER,
    created_at      TEXT DEFAULT (datetime('now')),
    verified_at     TEXT
);

Whenever /fix-llm generates a patch:

• Insert a new record.
• Set:

passed = NULL

until verification completes.

After /verify finishes:

• Update the corresponding row.

• Store:

  • verification result
  • verification timestamp

---

Suggested Approach

• Create the patch_outcomes table during database initialization.

• Generate a unique identifier for every patch.

• Insert one row for every generated patch.

• Record:

  • Job ID
  • Finding ID
  • Model name
  • Prompt token count (if available)
  • Creation timestamp

• After verification completes:

  • Update the passed field.
  • Set verified_at.

• Document the table structure and its purpose in CONTRIBUTING.md.

---

Acceptance Criteria

• patch_outcomes table is created automatically during application startup.
• One database row is inserted for every generated LLM patch.
• passed is initially stored as NULL.
• Verification updates the passed field correctly.
• verified_at is populated after verification.
• Model name is stored.
• Job ID and Finding ID are stored correctly.
• Prompt token count is recorded when available.
• Existing database functionality remains unaffected.
• CONTRIBUTING.md documents the new table and its purpose.
• Unit tests verify insertion and update behavior.

[github-actions]

👋 Welcome @arpit2006!

Thank you for opening an issue and contributing to Patchpilot. 🚀

Next Steps

• Please ensure you've provided all the required details.
• A maintainer will review your issue shortly.
• Feel free to ask questions if anything is unclear.

⚡ Want faster assignment and quicker responses?

Join our Discord community:

🔗 https://discord.gg/FcXuyw2Rs

We actively discuss issues, features, roadmap updates, and contributor support there.

Happy Contributing! 🎉

[mithaliphadtare]

@arpit2006 I would like to work on this issue under SSoC 26. Could you please assign this issue to me?

[Codewithpabitra]

/assign ssoc'26