feat: Integrate Browser Streaming Service with AI Orchestrator Service

[braden-seaborn]

Description

Integrate the existing browser streaming service with the AI Orchestrator service to enable real-time visualization and user control of Playwright-controlled Chromium browsers during automation sessions.

Background

The streaming/input service logic is already implemented. Now we need to connect it to the Playwright browser running in Weave, allowing users to watch automation in real-time and take manual control when needed.

Requirements

Integration Tasks

• Connect existing streaming service to the AI Orchestrator's Playwright browser instances
• Route browser viewport data from AI Orchestrator to the streaming service
• Pipe user input from stream back to Playwright in AI Orchestrator
• Handle session handoff between AI automation and manual user control

Multi-tab Support

• Stream multiple browser tabs/contexts simultaneously
• Provide tab switching UI in the frontend
• Maintain correct input routing to active tab

Performance & Monitoring

• Implement configurable FPS caps (15fps, 30fps, 60fps)
• Add adjustable resolution settings
• Stream health metrics dashboard:
  • Latency measurements
  • Frame drop rate
  • Connection status
  • Bandwidth usage

Acceptance Criteria

• Streaming service connects to Weave's Playwright browser
• Users can view live browser viewport during automation
• Users can take manual control and interact with the page
• Multi-tab navigation works seamlessly
• FPS/resolution settings are configurable
• Health metrics are visible in real-time

Later TODOs

• When the user interacts with the web-browser, it pauses the LangGraph agents and any deterministic automation.

[KedarPanchal]

Merged the prompt-input branch to add frontend prompt sending hooks and a skeleton component that calls these hooks.

[KedarPanchal]

Created Agent interface that can be subclassed to implement customized agents. Created documentation on adding new agents and the overall organization of the AI Orchestrator microservice. Added debugging logs to the AI Orchestrator code and resolved import bugs. Fixed server-side bugs related to receiving client actions and added test cases for testing client actions. Streamlined browser streaming code on the server-side (yet to test) and added shutdown and cleanup functionality to StageHandler.

[KedarPanchal]

Finalized the entirety of the backend portion of the streaming service with client actions. Now pivoting to testing the frontend and fixing any bugs. The commit that finalized the backend is tagged with backend-streaming, so utilize that tag when branching off the stagehand-streaming branch if you want to test other things.