Intelligent Planning System with Model-Aware Execution

[braden-seaborn]

Implement an intelligent planning system that creates transparent, user-visible execution plans while leveraging the Dynamic Model Escalation System for optimal cost and accuracy during execution.

Enhanced Planning Requirements

Current State:

"Before the agent interacts with the webpage, but after it navigates to it, it should create a preliminary, high-level plan that the user can see. It should be able to mark items complete with a tool call. The updated plan should be included with every agent turn."

Improved Implementation:

1. Multi-Phase Planning Architecture

   • Initial Assessment Phase: GPT-4.1-mini analyzes page complexity and creates detailed plan
   • Execution Phase: GPT-4.1-nano executes individual plan steps
   • Dynamic Adjustment: Plan updates based on model escalation triggers

2. Model-Aware Planning Intelligence

   • Complexity Analysis: Automatically categorize tasks as simple/medium/complex
   • Model Assignment: Assign appropriate models to different plan components
   • Escalation Integration: Built-in fallback strategies when execution models fail

3. User Transparency & Control

   • Real-time Plan Updates: Live progress tracking with completion status
   • Model Usage Visibility: Show which model is handling each step
   • Cost Estimation: Display estimated vs actual costs for plan execution

Detailed Implementation

Planning Phase (GPT-4.1-Mini)

plan_structure = {
"session_id": "unique_identifier",
"page_analysis": {
"complexity": "simple|medium|complex",
"detected_elements": [...],
"estimated_steps": 5,
"recommended_model": "nano|mini|full"
},
"execution_plan": [
{
"step_id": 1,
"description": "Fill email field",
"complexity": "simple",
"assigned_model": "nano",
"status": "pending|in_progress|completed|failed",
"fallback_strategy": "escalate_to_mini",
"estimated_cost": 0.0007
}
],
"total_estimated_cost": 0.0035,
"success_probability": 0.95
}

Dynamic Execution Integration

• Step-by-Step Execution: Each plan step executed with assigned model
• Real-time Escalation: Automatic model upgrades when steps fail
• Plan Adaptation: Update remaining steps based on escalation outcomes
• Cost Tracking: Live cost vs estimate comparison

User Interface Enhancements

• Progressive Plan Display: Show plan creation, then step-by-step execution
• Model Indicator: Visual badges showing which model is active
• Cost Monitor: Real-time cost tracking with savings metrics
• Manual Override: Allow users to force model escalation for critical steps

Technical Requirements

Core Planning System

• Implement PlanGenerator class using GPT-4.1-mini
• Create PlanStep model with complexity analysis
• Add mark_step_complete() tool for execution tracking
• Build plan persistence across agent turns
• Integrate with existing Dynamic Model Escalation System

Model Integration

• Planning: Always use GPT-4.1-mini for initial plan creation
• Execution: Start with nano, escalate as needed per existing logic
• Adaptation: Modify remaining plan steps based on execution outcomes
• Context Passing: Maintain plan context during model switches

User Experience

• Plan Visualization: Rich, collapsible plan display in UI
• Progress Tracking: Real-time step completion with timestamps
• Model Transparency: Show which model handled each step
• Cost Dashboard: Live cost tracking and savings metrics
• Manual Controls: Pause, skip, or force-escalate individual steps

Error Handling & Recovery

• Plan Validation: Verify plan feasibility before execution
• Step Failure Recovery: Automatic plan adaptation when steps fail
• Rollback Capability: Undo completed steps if needed
• Alternative Strategies: Multiple approaches for critical steps

Implementation Details

Plan Creation Flow:

1. Page Navigation: Agent navigates to target page
2. Complexity Analysis: GPT-4.1-mini analyzes page structure and requirements
3. Plan Generation: Create detailed, step-by-step execution plan
4. User Approval: Display plan for user review/approval
5. Model-Aware Execution: Execute with nano, escalate as needed
6. Live Updates: Update plan status and costs in real-time

Integration with Dynamic Escalation:

class IntelligentPlanner:
def init(self, escalation_system):
self.escalation_system = escalation_system
self.planning_model = "gpt-4.1-mini" # Always use mini for planning

def create_plan(self, page_context):
    # Generate plan with mini
    plan = self.generate_with_mini(page_context)
    
    # Assign models based on complexity
    for step in plan.steps:
        step.assigned_model = self.determine_model(step.complexity)
        
    return plan
    
def execute_plan(self, plan):
    for step in plan.steps:
        result = self.escalation_system.execute_with_escalation(step)
        self.update_plan_progress(plan, step, result)
        self.adapt_remaining_steps(plan, result)

Cost Optimization Integration:

• Planning Cost: Fixed ~$0.001 per plan (mini model)
• Execution Savings: 70-80% cost reduction through nano execution
• Total ROI: Planning cost pays for itself through execution optimization
• User Visibility: Show cost breakdown and savings in real-time

Acceptance Criteria

• Plans generated before any page interaction begins
• All plans visible to users with clear step descriptions
• Step completion tracking with mark_step_complete() tool
• Plan updates included in every agent response
• Integration with Dynamic Model Escalation System
• Real-time cost tracking and model usage display
• Plan adaptation when escalation occurs
• Manual user controls for critical interventions
• Error recovery and alternative strategy handling

Success Metrics

• User Satisfaction: 95% of users find plans helpful and clear
• Execution Accuracy: 98% of planned steps complete successfully
• Cost Efficiency: 75% cost reduction vs. always using mini
• Planning Speed: Plans generated in <5 seconds
• Adaptation Rate: 90% of failed steps recovered through plan updates

Integration Points

Dependencies:

• Dynamic Model Escalation System (parent issue)
• Weave browser automation core
• User interface components

Related Components:

• WeaveAgent class for plan execution
• Cost tracking and analytics
• User dashboard and progress display
• Error handling and recovery systems

User Experience Flow

1. Agent navigates to page
2. "Analyzing page and creating plan..." indicator
3. Plan displays with estimated steps, models, and costs
4. User can review, modify, or approve plan
5. Step-by-step execution with live progress updates
6. Model escalation notifications when they occur
7. Plan adaptations shown in real-time
8. Final completion summary with cost analysis

This enhanced planning system transforms browser automation from a "black box" into a transparent, intelligent process that users can understand, monitor, and control while optimizing costs through the Dynamic Model Escalation System.