Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
ef813ed
refactored codebase, structured with folders, added pytest and tests
Thambolo Dec 13, 2025
20e5f86
changed tests to headless mode
Thambolo Dec 13, 2025
9ffcf95
added CI pipeline
Thambolo Dec 13, 2025
410e27b
fixed indentation for scroll strategies
tangkenzee Dec 15, 2025
1cdc88f
cleaned up tests and updated ci.yml to handle secrets and env variables
Thambolo Dec 16, 2025
34c5aa6
updated README.md
Thambolo Dec 16, 2025
751165d
Merge branch 'main' into deep-research-feature
Thambolo Dec 16, 2025
926e871
Merge branch 'main' of https://github.com/openonion/browser-agent int…
tangkenzee Dec 17, 2025
0289ee8
Merge branch 'deep-research-feature' of https://github.com/openonion/…
tangkenzee Dec 17, 2025
aa97ba5
updated requirements and setup.sh print
Thambolo Dec 17, 2025
ae55740
moved agent state into the class
Thambolo Dec 17, 2025
c9e9660
Merge branch 'deep-research-feature' of https://github.com/openonion/…
Thambolo Dec 17, 2025
db0a6ba
removed .co/config.toml from repo and removed static configs in web_a…
Thambolo Dec 17, 2025
6ce9907
Merge branch 'deep-research-feature' of https://github.com/openonion/…
tangkenzee Dec 17, 2025
3b2d035
added test for chromium login
tangkenzee Dec 17, 2025
4b8af27
fixed merge conflicts for tests
tangkenzee Dec 17, 2025
0b121ed
removed perrsistent profile, login fresh each time, and bypasses the …
tangkenzee Dec 17, 2025
cd4d01c
removed debugging test
tangkenzee Dec 17, 2025
39a710f
collapsed entrypoint.py into agent.py and uses typer+rich. headless f…
Thambolo Dec 17, 2025
2ab354b
fixed PR review
Thambolo Dec 18, 2025
0efbabc
merged
Thambolo Dec 18, 2025
949b432
fixed all merge issues
Thambolo Dec 18, 2025
2f145c6
added pytest-xdist to run tests in parallel
Thambolo Dec 18, 2025
31e50fe
feat: Implement user-confirmed deep research workflow
tangkenzee Dec 19, 2025
9f939fe
feat: agent upgrade with deep research, iframe-aware finding, and sim…
tangkenzee Jan 2, 2026
c2173cb
refactor: simplify error handling in element finder and web automation
tangkenzee Jan 2, 2026
dee67c0
persistent profiles are working
tangkenzee Jan 3, 2026
93083d0
Merge origin/main, resolving conflicts in favor of current branch str…
tangkenzee Jan 5, 2026
717f8b4
Update .gitignore with modern Python tooling ignore patterns
tangkenzee Jan 5, 2026
f8f1149
resolved internal error lol
tangkenzee Jan 5, 2026
f5f7d12
file writing results of deep research implemented
tangkenzee Jan 5, 2026
86a9713
fixed tests to remove the handle popup
tangkenzee Jan 5, 2026
3635a2f
Fix CI parallel execution by using unique Chrome profiles per test
tangkenzee Jan 5, 2026
0f263f9
added more linient matching for finding elements
tangkenzee Jan 5, 2026
affc722
added the review comments
tangkenzee Jan 6, 2026
96deaaf
resolved the review comments
tangkenzee Jan 7, 2026
af82dcc
tests are running on model gemini 2.5
tangkenzee Jan 7, 2026
c26372a
removed the create_agent function and initialized browser agent norma…
tangkenzee Jan 7, 2026
699d065
new readme file to reflect changes
tangkenzee Jan 7, 2026
4804800
split the deep research agent into agent and tools
tangkenzee Jan 7, 2026
5478cdb
removed overengineered code, functionality retains
tangkenzee Jan 7, 2026
5a582e7
removed the form filling functions, agent shall use find_element_by_d…
tangkenzee Jan 8, 2026
8ceed80
removed redunent scroll functions and renamed to get_search_results t…
tangkenzee Jan 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -504,11 +504,11 @@ agent = Agent(
)

# Manual session (no LLM) — call tools directly
print(agent.tool_map["start_browser"](headless=True))
title = agent.tool_map["goto"]("https://example.com")
print(agent.tool_map["format_title"](title=title))
print(agent.tool_map["screenshot"](filename="example.png"))
print(agent.tool_map["close"]())
print(agent.tools.start_browser.run(headless=True))
title = agent.tools.goto.run("https://example.com")
print(agent.tools.format_title.run(title=title))
print(agent.tools.screenshot.run(filename="example.png"))
print(agent.tools.close.run())
```

**Example output:**
Expand Down Expand Up @@ -1501,6 +1501,305 @@ def my_tool(natural_input: str) -> str:

---

## Plugin System - Reusable Event Bundles

Plugins are reusable event lists that package capabilities like reflection and reasoning for use across multiple agents.

### Quick Start (60 seconds)

```python
from connectonion import Agent
from connectonion.useful_plugins import reflection, react

# Add built-in plugins to any agent
agent = Agent(
name="assistant",
tools=[search, calculate],
plugins=[reflection, react] # One line adds both!
)

agent.input("Search for Python and calculate 15 * 8")

# After each tool execution:
# 💭 We learned that Python is a popular programming language...
# 🤔 We should next calculate 15 * 8 to complete the request.
```

### What is a Plugin?

**A plugin is an event list** - just like `on_events`, but reusable across agents:

```python
from connectonion import after_tools, after_each_tool, after_llm

# This is a plugin (one event list)
reflection = [after_tools(add_reflection)] # after_tools for message injection

# This is also a plugin (multiple events in one list)
logger = [after_llm(log_llm), after_each_tool(log_tool)] # after_each_tool for per-tool logging

# Use them (plugins takes a list of plugins)
agent = Agent("assistant", tools=[search], plugins=[reflection, logger])
```

**Just like tools:**
- Tools: `Agent(tools=[search, calculate])`
- Plugins: `Agent(plugins=[reflection, logger])`

### Plugin vs on_events

The difference:
- **on_events**: Takes one event list (custom for this agent)
- **plugins**: Takes a list of event lists (reusable across agents)

```python
# Reusable plugin (an event list)
logger = [after_llm(log_llm)]

# Use both together
agent = Agent(
name="assistant",
tools=[search],
plugins=[logger], # List of event lists
on_events=[after_llm(add_timestamp), after_each_tool(log_tool)] # One event list
)
```

### Built-in Plugins (useful_plugins)

ConnectOnion provides ready-to-use plugins:

**Reflection Plugin** - Generates insights after each tool execution:

```python
from connectonion import Agent
from connectonion.useful_plugins import reflection

agent = Agent("assistant", tools=[search], plugins=[reflection])

agent.input("Search for Python")
# 💭 We learned that Python is a popular high-level programming language known for simplicity
```

**ReAct Plugin** - Uses ReAct-style reasoning to plan next steps:

```python
from connectonion import Agent
from connectonion.useful_plugins import react

agent = Agent("assistant", tools=[search], plugins=[react])

agent.input("Search for Python and explain it")
# 🤔 We learned Python is widely used. We should next explain its key features and use cases.
```

**Image Result Formatter Plugin** - Converts base64 image results to proper image messages for vision models:

```python
from connectonion import Agent
from connectonion.useful_plugins import image_result_formatter

agent = Agent("assistant", tools=[take_screenshot], plugins=[image_result_formatter])

agent.input("Take a screenshot of the homepage and describe what you see")
# 🖼️ Formatted tool result as image (image/png)
# Agent can now see and analyze the actual image, not just base64 text!
```

**When to use:** Tools that return screenshots, generated images, or any visual data as base64.
**Supported formats:** PNG, JPEG, WebP, GIF
**What it does:** Detects base64 images in tool results and converts them to OpenAI vision API format, allowing multimodal LLMs to see images visually instead of as text.

**Using Multiple Plugins Together:**

```python
from connectonion import Agent
from connectonion.useful_plugins import reflection, react, image_result_formatter

# Combine plugins for powerful agents
agent = Agent(
name="visual_researcher",
tools=[take_screenshot, search, analyze],
plugins=[image_result_formatter, reflection, react]
)

# Now you get:
# 🖼️ Image formatting for screenshots
# 💭 Reflection: What we learned
# 🤔 ReAct: What to do next
```

### Writing Custom Plugins

Learn by example - here's how the reflection plugin works:

**Step 1: Message Compression Helper**

```python
from typing import List, Dict

def _compress_messages(messages: List[Dict], tool_result_limit: int = 150) -> str:
"""
Compress conversation messages with structure:
- USER messages → Keep FULL
- ASSISTANT tool_calls → Keep parameters FULL
- ASSISTANT text → Keep FULL
- TOOL results → Truncate to tool_result_limit chars
"""
lines = []

for msg in messages:
role = msg['role']

if role == 'user':
lines.append(f"USER: {msg['content']}")

elif role == 'assistant':
if 'tool_calls' in msg:
tools = [f"{tc['function']['name']}({tc['function']['arguments']})"
for tc in msg['tool_calls']]
lines.append(f"ASSISTANT: {', '.join(tools)}")
else:
lines.append(f"ASSISTANT: {msg['content']}")

elif role == 'tool':
result = msg['content']
if len(result) > tool_result_limit:
result = result[:tool_result_limit] + '...'
lines.append(f"TOOL: {result}")

return "\n".join(lines)
```

**Why this works:**
- Keep user messages FULL (need to know what they asked)
- Keep tool parameters FULL (exactly what actions were taken)
- Keep assistant text FULL (reasoning/responses)
- Truncate tool results (save tokens while maintaining overview)

**Step 2: Event Handler Function**

```python
from connectonion.events import after_tool
from connectonion.llm_do import llm_do

def _add_reflection(agent) -> None:
"""Reflect on tool execution result"""
trace = agent.current_session['trace'][-1]

if trace['type'] == 'tool_execution' and trace['status'] == 'success':
# Extract current tool execution
user_prompt = agent.current_session.get('user_prompt', '')
tool_name = trace['tool_name']
tool_args = trace['arguments']
tool_result = trace['result']

# Compress conversation messages
conversation = _compress_messages(agent.current_session['messages'])

# Build prompt with conversation context + current execution
prompt = f"""CONVERSATION:
{conversation}

CURRENT EXECUTION:
User asked: {user_prompt}
Tool: {tool_name}({tool_args})
Result: {tool_result}

Reflect in 1-2 sentences on what we learned:"""

reflection_text = llm_do(
prompt,
model="co/gpt-4o",
temperature=0.3,
system_prompt="You reflect on tool execution results to generate insights."
)

# Add reflection as assistant message
agent.current_session['messages'].append({
'role': 'assistant',
'content': f"💭 {reflection_text}"
})

agent.console.print(f"[dim]💭 {reflection_text}[/dim]")
```

**Key insights:**
- Access agent state via `agent.current_session`
- Use `llm_do()` for AI-powered analysis
- Add results back to conversation messages
- Print to console for user feedback

**Step 3: Create Plugin (Event List)**

```python
# Plugin is an event list
reflection = [after_tools(_add_reflection)] # after_tools for message injection
```

**That's it!** A plugin is just an event list.

**Step 4: Use Your Plugin**

```python
agent = Agent("assistant", tools=[search], plugins=[reflection])
```

### Quick Custom Plugin Example

Build a simple plugin in 3 lines:

```python
from connectonion import Agent, after_tool

def log_tool(agent):
trace = agent.current_session['trace'][-1]
print(f"✓ {trace['tool_name']} completed in {trace['timing']}ms")

# Plugin is an event list
logger = [after_each_tool(log_tool)] # after_each_tool for per-tool logging

# Use it
agent = Agent("assistant", tools=[search], plugins=[logger])
```

### Reusing Plugins

Use the same plugin across multiple agents:

```python
# Define once
reflection = [after_tools(add_reflection)] # after_tools for message injection
logger = [after_llm(log_llm), after_each_tool(log_tool)] # after_each_tool for per-tool logging

# Use in multiple agents
researcher = Agent("researcher", tools=[search], plugins=[reflection, logger])
writer = Agent("writer", tools=[generate], plugins=[reflection])
analyst = Agent("analyst", tools=[calculate], plugins=[logger])
```

### Summary

**A plugin is an event list:**

```python
# Define a plugin (an event list)
my_plugin = [after_llm(handler1), after_tools(handler2)] # after_tools for message injection

# Use it (plugins takes a list of event lists)
agent = Agent("assistant", tools=[search], plugins=[my_plugin])
```

**on_events vs plugins:**
- `on_events=[after_llm(h1), after_each_tool(h2)]` → one event list
- `plugins=[plugin1, plugin2]` → list of event lists

**Event naming:**
- `after_each_tool` → fires for EACH tool (per-tool logging/monitoring)
- `after_tools` → fires ONCE after all tools (safe for message injection)

---

## Best Practices

### Principles: Avoid over‑engineering with agents
Expand Down
44 changes: 23 additions & 21 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,26 +12,28 @@ jobs:

env:
OPENONION_API_KEY: ${{ secrets.OPENONION_API_KEY }}
BROWSER_AGENT_MODEL: gemini-2.5-flash
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

steps:
- uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: 'pip'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest

- name: Install Playwright browsers
run: playwright install chromium

- name: Run Tests
run: |
# Skip manual/slow tests, run main integration suite
pytest tests/test_all.py tests/test_direct.py tests/test_image_plugin.py -v
- uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install pytest

- name: Install Playwright browsers
run: playwright install chromium

- name: Run Tests
run: |
# Run all tests except manual ones in parallel
pytest -n auto -m "not manual" -v -p no:anyio
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ ENV/
.pytest_cache/
.tox/
htmlcov/
.mypy_cache/
.ruff_cache/

# Jupyter Notebook
.ipynb_checkpoints
Expand All @@ -64,4 +66,6 @@ firefox_automation_profile/

# Generated data and screenshots
data/

research_results.md
screenshots/
Loading