Skip to content
This repository was archived by the owner on Nov 15, 2025. It is now read-only.
This repository was archived by the owner on Nov 15, 2025. It is now read-only.

[Feature] Optimize Prompts for Direct LLM Calls in Non-Agentic Workflows #16

@christso

Description

@christso

Description:

Problem: Users need to optimize system prompts for simple, direct LLM calls (e.g., via OpenAI API) for tasks like SQL code reviews or schema change validation, incorporating company-specific guidelines (e.g., attached PDF/YAML files for style rules).

Proposed Solution: Extend bbeval opt to support direct LLM optimization:

  • Input: Base system prompt + attached guidelines file (e.g., --guidelines coding_rules.pdf).
  • Use: BootstrapRS (≤5 trials) to tune prompt for accuracy on test tasks (e.g., "Review this SQL: [code]" → scored against expected feedback).
  • Output: JSON prompt with enforced rules (e.g., required placeholders like {{guidelines}}, banned phrases).
  • Integration: --mode direct-llm flag; validate via code_execution tool for SQL syntax.

Benefits: Enables quick tuning for non-agentic reviews; reduces manual prompt engineering by 80% on guideline adherence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions