Skip to content

Add LLM-powered PR classification#698

Open
pavsoss wants to merge 1 commit into
Coder-s-OG-s:mainfrom
pavsoss:feat/llm-pr-classification
Open

Add LLM-powered PR classification#698
pavsoss wants to merge 1 commit into
Coder-s-OG-s:mainfrom
pavsoss:feat/llm-pr-classification

Conversation

@pavsoss

@pavsoss pavsoss commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces heuristic-only PR classification with LLM-powered structured classification using the existing llmCall abstraction.

The classifier now analyzes PR title and body context through the shared LLM provider pipeline while preserving the existing heuristic logic as a graceful fallback.

Changes Made

LLM-Powered Classification

  • Converted classifyPrAsAi to an async function.
  • Added structured LLM classification using llmCall.
  • Validates classification output with a Zod schema containing:
    • isAiGenerated
    • confidence
    • reason
  • Uses the validated isAiGenerated result as the classifier decision.

Heuristic Fallback

  • Preserved the existing classification logic in classifyPrHeuristic.
  • Automatically falls back to the heuristic when:
    • all LLM providers fail
    • the LLM request returns an error
    • structured output cannot be validated

This preserves the existing deterministic behavior when AI classification is unavailable.

Async Pipeline Updates

Updated all functional callers of classifyPrAsAi to await the new async classifier:

  • pr-backfill.ts
  • process-pr-event.ts

Tests

  • Updated existing heuristic tests to explicitly exercise the fallback path.
  • Added coverage for AI-generated LLM classifications.
  • Added coverage for genuine PR classifications.
  • Verified heuristic fallback when llmCall returns an error.

Closes #696

@vercel

vercel Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

@pavsoss is attempting to deploy a commit to the codersogs-3057's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] -Improve PR classification using LLM-powered structured classification

1 participant