Agentic Browser

An AI-powered web automation tool that lets you control websites using natural language commands. The AI analyzes screenshots of web pages and performs actions like clicking buttons, filling forms, and navigating through web interfaces.

Features

• Natural Language Control: Use plain English to interact with web pages
• Visual Analysis: AI sees the page via screenshots and identifies elements by bounding box coordinates
• Live Browser View: Real-time screenshot streaming (every 2 seconds) from a headless Playwright browser
• WebSocket Communication: Real-time bidirectional communication between frontend and backend
• Configurable AI Models: Switch models via the UI dropdown — default is nvidia/nemotron-nano-12b-v2-vl:free
• In-Browser API Key Setup: Set your OpenRouter API key from the UI, saved to backend/config.json

Architecture

Backend (Python)

• FastAPI with WebSocket support
• Playwright for headless browser automation and screenshot capture
• OpenAI SDK pointed at OpenRouter API for AI model requests
• Bounding box pipeline: Vision model returns element coordinates → a second model cleans/validates them → backend generates JavaScript click code → executes in browser

Frontend (React)

• React 18 with hooks
• WebSocket for real-time communication
• Split layout: Browser view (2/3) + Chat panel (1/3)
• Model selector dropdown and API key/system message modals

Workflow

1. User sends a natural language request via chat
2. Backend captures a screenshot of the current browser page
3. Screenshot + request sent to the vision AI model via OpenRouter
4. AI returns bounding box coordinates for the target element
5. A second model (alibaba/tongyi-deepresearch-30b-a3b:free) cleans/validates the coordinates
6. Backend generates JavaScript to click at the bounding box center
7. JavaScript executes in the Playwright browser via page.evaluate()
8. Updated screenshot sent back to the frontend

Prerequisites

• Python 3.8+
• Node.js 16+ and npm
• OpenRouter API Key (openrouter.ai)

Setup

1. Install Backend Dependencies

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
playwright install chromium

2. Install Frontend Dependencies

cd frontend
npm install
cd ..

3. Configure API Key

You have two options:

Option A — Via the UI (recommended): Start the app and click the "🔑 API Key" button in the header to enter your key. It gets saved to backend/config.json.

Option B — Via environment variable:

cp .env.example .env
# Edit .env and set OPENROUTER_API_KEY

Note: The UI-saved key in config.json takes priority over the .env file.

Running

Start Backend (Terminal 1)

source venv/bin/activate
cd backend
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Or use the startup script:

./start_backend.sh

Start Frontend (Terminal 2)

cd frontend
npm start

Or use the startup script:

./start_frontend.sh

The frontend runs on http://localhost:3000, backend on http://localhost:8000.

Usage

1. Enter a URL in the address bar (e.g., https://google.com)
2. Ask the AI to interact: "Click the search box", "What do you see?"
3. The AI analyzes the screenshot, finds the element, and clicks it

Example Commands

• "Go to google.com" (via URL bar)
• "What do you see on this page?"
• "Click the login button"
• "Click the search box"

Configuration

Switching AI Models

Use the dropdown in the header (visible after setting an API key). Built-in options:

• nvidia/nemotron-nano-12b-v2-vl:free (default, free)
• openai/gpt-4o
• openai/gpt-4o-mini
• anthropic/claude-3-haiku
• meta-llama/llama-3.1-8b-instruct:free
• Custom model via the "🔧 Custom Model..." option

The selected model must support vision/image inputs.

System Message

Click "💬 System" in the header to override the default system prompt.

Docker Deployment

export OPENROUTER_API_KEY=your_key_here
docker-compose up --build

Frontend at http://localhost:3000, backend at http://localhost:8000.

Project Structure

agentic-browser/
├── backend/
│   ├── main.py                 # FastAPI app, WebSocket handler, bounding box → JS pipeline
│   ├── browser_manager.py      # Playwright browser control, screenshot streaming
│   ├── openrouter_client.py    # OpenAI SDK client for OpenRouter, bounding box processor
│   ├── config.example.json     # Example config (API key + model)
│   └── __init__.py
├── frontend/
│   ├── src/
│   │   ├── App.js              # Main app, WebSocket message routing, modals
│   │   ├── components/
│   │   │   ├── BrowserView.js  # Screenshot display, URL bar, bounding box overlay
│   │   │   └── ChatPanel.js    # Chat interface
│   │   └── hooks/
│   │       └── useWebSocket.js # WebSocket connection with auto-reconnect
│   └── package.json
├── requirements.txt
├── docker-compose.yml
├── Dockerfile.backend
├── Dockerfile.frontend
├── start_backend.sh
├── start_frontend.sh
├── start_backend_headless.py
├── .env.example
└── .gitignore

Security Notes

• API key stored in plaintext in backend/config.json when set via the UI. Do not commit this file. It is already in .gitignore.
• No authentication or rate limiting — not suitable for public deployment without additional security.
• The backend executes AI-generated JavaScript in the browser context. Use caution in production.

Troubleshooting

• Browser doesn't start: Run playwright install chromium and ensure dependencies are installed (playwright install-deps chromium on Linux).
• WebSocket won't connect: Verify backend is running on port 8000. Check for port conflicts with lsof -i :8000.
• API errors: Verify your API key is correct. Ensure the selected model supports vision inputs.
• Frontend won't start: Delete frontend/node_modules and run npm install again.

License

MIT