GEPA prompt optimization framework#5
Merged
Conversation
- Add GEPA module with dataset loading, rule-based evaluator, and module wrapper - Create evaluation datasets: trainset.json (23 examples), valset.json (10 examples) - Add gepa-optimize CLI binary with --eval-demo and --eval-live modes - Add GEPA documentation with architecture, implementation plan, and examples Refactor AgentResponse signature for GEPA optimization: - Split conversation_context into separate fields: - current_time, persona_block, human_block, memory_metadata - previous_context_summary, recent_conversation - is_first_time_user (boolean flag instead of injected text) - Each field can be independently optimized by GEPA Update AGENT_INSTRUCTION with clearer memory distinction: - Core Memory: always visible, for essential frequently-needed info - Archival Memory: searchable storage for details worth remembering later - Clear rules: 'every conversation?' -> Core, 'recall someday?' -> Archival Add justfile commands: gepa-optimize-dev, gepa-optimize, gepa-compare, gepa-history Baseline evaluation score: 0.97 (examples may be too easy) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Add rule: field markers must be on their own line (fixes DSRs parsing) - Use Claude as GEPA judge via Anthropic API - Refactor GEPA optimizer to use pure DSRs patterns - Add rich feedback evaluator with 5 checks - Update AGENT_INSTRUCTION with MEMORY PROTOCOLS, SEARCH SELECTION RULES - Add CASUAL CHAT RULE for multiple messages - Add FIRST-TIME USERS rule - Add AFTER TOOL RESULTS rules for silent memory operations - Create lib.rs + tools.rs for shared code between binaries - Store optimized instruction in optimized_instructions/latest.txt Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- user_introduces_name: User tells name + asks question, Sage stores and responds - memory_append_done_silent: After memory op, return done with no messages Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Add GEPA Prompt Optimization section to README with usage examples - Update justfile: gepa-eval, gepa-optimize, gepa-show, gepa-examples - Remove non-existent --mode flags from justfile - Update 'Built for Prompt Optimization' to reflect working status Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Kimi K2.5 is a thinking model that puts reasoning in reasoning_content. Having a separate reasoning output field caused double-thinking and </think> tags leaking into the output, breaking DSRs parsing. Changes: - Remove reasoning field from AgentResponse and CorrectionResponse - Update AGENT_INSTRUCTION to show only 2 output fields (messages, tool_calls) - Update GEPA evaluator to not check reasoning - Add training example for context-aware actions (don't re-ask for confirmed requests) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
When user mentions location, proactively set their timezone silently. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- First-time greeting: Focus on friendship, keep it short - Casual greeting: Show genuine interest in them - Job announcement: Ask how they're feeling, not just celebrate - Thanks: Just 'Anytime!' - no corporate 'let me know if you need anything' - Remove duplicate pet example, consolidate into companion_curious version - Add bad patterns for robotic/transactional behavior Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Updated load_trainset() to dynamically read from examples/gepa/trainset.json - GEPA now uses all 15 examples instead of 7 hardcoded ones - Fixed Austin/timezone example (now calls set_preference) - Added Common Storage Patterns section for location, jobs, pets - Clarified life events protocol: respond first, then store in next turn Remaining failures are companionship examples (asking follow-ups) - need to add soul/personality to instruction. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Changed from 'helpful AI assistant' to 'companion and friend' - Added WHO YOU ARE section emphasizing curiosity, warmth, genuine care - Added BE A FRIEND, NOT A SERVICE communication guidelines - Explicit instructions to ask follow-up questions, avoid corporate-speak - Added CASUAL MENTIONS protocol for being curious about new info - Examples: 'NO WAY!!' not 'That's wonderful news!' GEPA score: 0.967 (14.5/15 examples passing) Remaining issues on 2 companionship examples (follow-up questions) Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Updated AGENT_INSTRUCTION with: - Companion identity instead of assistant - WHO YOU ARE section with personality traits - BE A FRIEND, NOT A SERVICE communication style - Curiosity prompts for casual mentions - Common storage patterns for location, pets, etc. All 30 tests passing. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…bad response - Added 'user_introduces_name' as GOOD example of curious companionship behavior - Warm greeting, asks 'what makes you *you*?', stores silently - Added second bad_response to first_time_greeting for pattern that doesn't ask name Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
… Sage
Perfect example of soulful behavior:
- Genuine surprise at meta revelation ('YOU work on *me*?? 🤯')
- Self-aware humor ('terrified you're running a debugger on me')
- Authentic self-reflection about what makes Sage 'Sage'
- Reciprocal curiosity with follow-up questions
- Rich archival storage capturing personality context
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Fix rustfmt formatting across multiple files - Collapse nested if into single condition (clippy::collapsible_if) - Move evaluate_response above test module (clippy::items_after_test_module) - Prefix unused variable with underscore (unused_variables) - Allow dead_code on GEPACandidate.generation field - Remove unused imports Tool and ToolResult from main.rs Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
The gepa-optimize binary has its own self-contained optimization loop and never imported from the library modules (gepa/mod.rs, module.rs, evaluator.rs, dataset.rs). Also syncs latest.txt with current AGENT_INSTRUCTION. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds GEPA (Genetic-Pareto) optimization system for automatically improving Sage's agent instruction based on test cases.
Changes:
</think>tag leakage13 commits on this branch.