Deep research feature: Added base deep-research functionality by Thambolo · Pull Request #2 · openonion/browser-agent

Thambolo · 2025-12-16T07:49:37Z

This pull request introduces several significant improvements and updates to the browser-agent project, focusing on documentation, configuration, CI/CD setup, and code organization. The most important changes include a major expansion of a new GitHub Actions workflow for automated testing, and updates to project structure and configuration files to reflect recent best practices and enhancements.

Documentation and Developer Experience:

Renamed CLAUDE.md to GEMINI.md and updated its content to reflect the switch from Claude to Gemini as the default LLM, along with improvements to setup, testing, and development instructions. [1] [2] [3] [4] [5] [6]
Updated README.md to clarify the project structure, usage of the new browser_agent package, plugin integration, and the Chrome profile handling mechanism. [1] [2] [3] [4] [5] [6]

Automation and Configuration:

Added a new GitHub Actions workflow (.github/workflows/ci.yml) to automate testing on push and pull requests, including environment setup, dependency installation, Playwright browser setup, and running tests (excluding manual ones).
Updated .co/config.toml to use a newer framework version, set the default model to co/gemini-2.5-pro, and improved secret management by referencing a .env file.

Project Organization:

Refactored project structure in README.md to reflect the introduction of the browser_agent package and its submodules, and clarified the location of key files and scripts for easier navigation and maintenance. [1] [2] [3]

These changes collectively modernize the project, improve developer onboarding, and lay the groundwork for more robust and reusable agent functionality.

References:
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

…o deep-research-feature

…browser-agent into deep-research-feature

wu-changxing

Thanks for adding CI and deep research feature, but the file structure needs rework.

Problem

Creating browser_agent/ folder with entrypoint.py is over-engineered:

agent.py → browser_agent/entrypoint.py → creates Agent (extra layer for no benefit)
Folder name browser_agent/ doesn't match the feature being added (deep research)
Moving existing files breaks imports unnecessarily

Proposed Structure

browser-agent/
├── agent.py                 # Main entry point (keep in root)
├── tools/
│   ├── __init__.py
│   ├── web_automation.py    # Shared browser tool
│   └── scroll_strategies.py
├── agents/
│   ├── __init__.py
│   └── deep_research.py     # Sub-agent
├── prompts/
│   ├── browser_agent.md     # Main agent prompt
│   └── deep_research.md     # Sub-agent prompt
├── tests/
├── .github/workflows/ci.yml  # Keep this - CI is good
├── CLAUDE.md                 # Keep - don't rename to GEMINI.md
└── README.md

Why This Structure

Folder	Purpose
`tools/`	Shared capabilities (web_automation used by ALL agents)
`agents/`	Sub-agents only (deep_research, future sub-agents)
`prompts/`	All system prompts in one place
`agent.py` in root	Entry point, not a sub-agent

What to Change

Delete browser_agent/ folder and entrypoint.py
Create tools/, agents/, prompts/ folders instead
Keep agent.py in root as entry point
Keep CLAUDE.md (don't rename to GEMINI.md)
Keep CI pipeline - that's useful

This separates concerns (tools vs agents vs prompts) without unnecessary abstraction layers.

wu-changxing · 2025-12-17T04:04:10Z

+    This helps in breaking down complex research goals into manageable sub-tasks.
+    """
+
+    def __init__(self, web_automation: WebAutomation):


Bug: self.research_agent is created here but NEVER USED. In perform_deep_research() you create a NEW Agent called researcher. This is wasted code.

Either:

Remove this and just create Agent in perform_deep_research(), OR

Use self.research_agent.input(topic) instead of creating new one

@wu-changxing I don't see an issue here, there is no self.research_agent

i have now created it to make the class manage the agent state

wu-changxing · 2025-12-17T04:04:11Z

+"""
+
+from typing import Optional
+import os


Over-engineered: Why use a class? This could be a simple function:

def create_research_agent(web): return Agent( name='deep_researcher', tools=web, ... )

No need for class wrapper when there's no state to manage.

I'm keeping the class because I want to initialize the sub-agent once (in init) and reuse it across multiple calls to avoid overhead. This makes the class a stateful container for the sub-agent.

wu-changxing · 2025-12-17T04:04:11Z

 from connectonion import xray, llm_do
 from playwright.sync_api import sync_playwright, Page, Browser, Playwright
 import base64
-import json


Unnecessary instance variables for static config:

self.SCREENSHOTS_DIR = 'screenshots' self.CHROMIUM_PROFILE_DIR = 'chromium_automation_profile' self.DEFAULT_TIMEOUT = 30000

If only used once, just inline them. No need for 'configurability' that nobody will use.

removed them now

wu-changxing · 2025-12-17T04:04:11Z

 [cli]
 version = "1.0.0"
 command = "co init"
 template = "none"


Should not commit personal agent address/email changes. These should stay as they were or be gitignored.

wu-changxing · 2025-12-17T04:10:39Z

More Over-Engineering Issues Found

1. Delete `entrypoint.py` - vague name

entrypoint.py hidden in a subfolder is confusing. We already have agent.py at root.

If you want CLI functionality, either:

Keep it in agent.py (simplest)
Create cli.py at root (clear name)

Consider using typer + rich instead of argparse - much cleaner:

import typer
from rich import print

app = typer.Typer()

@app.command()
def run(prompt: str, headless: bool = False, deep_research: bool = False):
    print(f'[bold]Running:[/bold] {prompt}')
    # ...

if __name__ == '__main__':
    app()

2. `--headless` flag is parsed but NEVER USED

Dead code - either use it or remove it.

3. `import os` inside functions

Multiple files have import os scattered inside functions. Move to top of file.

4. `os.getenv('BROWSER_AGENT_MODEL')` can be None

Add fallback: os.getenv('BROWSER_AGENT_MODEL', 'co/gpt-4o')

…browser-agent into deep-research-feature

…utomation.py

…browser-agent into deep-research-feature

…google signin catcher for automated browserr

…lag is now used

This commit introduces a robust, user-confirmed deep research workflow, enhancing the agent's ability to handle complex information gathering tasks. Key features and improvements include: - **Two-Track Task System:** The main agent now intelligently classifies user requests as either "simple tasks" or "deep research tasks." - **User Confirmation for Deep Research:** For complex research queries, the agent proactively asks the user for explicit confirmation (Y/N) before proceeding, preventing unintended long-running operations. - **Dedicated Deep Research Workflow:** A structured, step-by-step process for deep research has been implemented, covering: - Initial deletion of `research_results.md` to ensure a clean slate for each task. - Web search capabilities to find relevant sources (`get_search_results`). - Systematic exploration of web pages and extraction of relevant information (`explore`, `analyze_html`). - Recording of findings to `research_results.md` (`append_to_file`, `read_file`). - Synthesis of a comprehensive final report. - **Proactive Popup Handling:** The `handle_popups()` tool was implemented and integrated into the workflow to automatically dismiss cookie banners and other intrusive popups, improving research reliability. - **Flexible Browser Closure:** The `close()` tool was enhanced with a `keep_browser_open` option, allowing the agent to leave the browser open for user interaction on simple tasks and close it for research tasks. - **New Core Tools:** The `WebAutomation` suite in `tools/web_automation.py` was extended with: - `analyze_html()`: LLM-driven HTML content analysis. - `explore()`: Combines navigation and HTML analysis. - `get_search_results()`: Heuristic Playwright-based search result extraction (note: this remains a challenging area due to target site dynamism). - `append_to_file()`: Appends content to a specified file. - `read_file()`: Reads content from a specified file. - `ask_user_confirmation()`: Prompts the user for Y/N input. - `delete_file()`: Safely deletes a file. - **Prompt Refinements:** `prompts/browser_agent.md` and `prompts/deep_research.md` were thoroughly updated to guide the agent through this new, intelligent workflow. - **Test Suite Enhancements:** Integration tests for `handle_popups()` were added and verified. - **Gitignore Update:** `research_results.md` and `screenshots/` added to `.gitignore` to prevent tracking of temporary research artifacts. This work significantly improves the agent's autonomy, user experience, and capability for complex tasks.

…plified architecture Core Architecture: - **Index-Based Element Finder**: Migrated from LLM-generated CSS selectors to a robust system where JS injects unique IDs into interactive elements. The LLM now SELECTS from an indexed list, ensuring 100% selector accuracy. - **Iframe Awareness**: Updated extraction and click logic to traverse all iframes on a page. This allows the agent to 'see' and interact with content inside sub-documents (like cookie banners and login modals). - **Deep Research Delegation**: Registered the 'DeepResearch' sub-agent as a tool for the main 'browser_agent'. This enables a Manager/Specialist workflow where complex research is delegated to a sub-agent while sharing the same browser session. Tool & Prompt Improvements: - **Simplified Toolset**: Removed the specialized 'handle_popups' tool. Updated 'agent.md' to instruct the agent to handle popups naturally using generic clicks. - **Robust Search**: Refactored 'get_search_results' to use reliable DOM traversal for organic results, providing the agent with a high-quality menu of sources. - **Full-Page Vision**: Removed character limits on 'analyze_html' and 'analyze_page' to allow the LLM to process entire documents for summarization. - **Prompt Consolidation**: Replaced legacy prompts with a modernized set: 'agent.md', 'deep_research.md', 'element_matcher.md', and 'form_filler.md'. - **Modular File Tools**: Extracted 'append_to_file', 'read_file', and 'delete_file' into a standalone 'FileTools' class, registering them exclusively with the deep research sub-agent to maintain separation of concerns. Verification: - Added 'tests/test_element_finder.py' for direct logic validation. - Verified successful end-to-end Deep Research runs on competitive analysis tasks. - Confirmed all automated tests in the project suite pass.

- Removed unnecessary try-except blocks to follow 'let it break' principles. - Errors will now bubble up naturally, providing clearer debugging information. - Maintained a specific guard for cross-origin frame offset calculation as it is a known browser constraint.

…ucture

wu-changxing

Some files need be modified

wu-changxing · 2026-01-06T03:40:37Z

this one is only for define the agent, and for cli normally we need to create another file which is cli.py

wu-changxing · 2026-01-06T03:42:28Z

keep this one for now

wu-changxing · 2026-01-06T03:43:46Z

+For complex questions that require reading multiple sources and synthesizing a detailed report (e.g., "What are the best marketing strategies for AI tools?"), use the **`perform_deep_research(topic)`** tool.
+- This will spawn a specialized sub-agent to handle the deep exploration.
+- **Pass the FULL user request** as the `topic` argument. Do not summarize it.
+  - ✅ Correct: `perform_deep_research("Find the history of the mouse and save it to mouse_history.txt")`
+  - ❌ Incorrect: `perform_deep_research("history of the mouse")`
+- The sub-agent has tools to save files and format reports, so it needs the specific instructions.
+- Use it when a task is too big for a single sequential browsing session.


this may need some example of when to use the deep research

wu-changxing · 2026-01-06T03:44:14Z

+
+## Output Format
+
+Your final response must be the **Comprehensive Research Report** itself. Do not say "I have finished research." Just provide the report.


lack of examples

wu-changxing · 2026-01-06T03:45:16Z

need create the test for init create profile and resue

wu-changxing · 2026-01-06T03:56:12Z

+class FileTools:
+    """Tools for reading and writing files."""
+
+    @xray
+    def append_to_file(self, filepath: str, content: str) -> str:
+        """
+        Appends content to a specified file.
+        Useful for logging research notes or findings.
+        """
+        with open(filepath, "a", encoding="utf-8") as f:
+            f.write(content + "\n")
+        return f"Successfully appended to {filepath}"
+
+    @xray
+    def write_file(self, filepath: str, content: str) -> str:
+        """
+        Writes (or overwrites) content to a specified file.
+        Useful for saving final reports.
+        """
+        with open(filepath, "w", encoding="utf-8") as f:
+            f.write(content)
+        return f"Successfully wrote to {filepath}"
+
+    @xray
+    def read_file(self, filepath: str) -> str:
+        """
+        Reads the entire content of a specified file.
+        Useful for reviewing notes.
+        """
+        if not os.path.exists(filepath):
+            return f"File not found: {filepath}"
+        with open(filepath, "r", encoding="utf-8") as f:
+            return f.read()
+
+    @xray
+    def delete_file(self, filepath: str) -> str:
+        """
+        Deletes a specified file.
+        Useful for cleaning up temporary notes.
+        """
+        if os.path.exists(filepath):
+            os.remove(filepath)
+            return f"Successfully deleted file: {filepath}"
+        return f"File not found: {filepath}"


name it for the function, reflect the features , Report class, then create report file append to report file etcs

wu-changxing · 2026-01-06T03:57:59Z

+            print(f"  ❌ {strategy_name} failed: {e}")
+            continue
+


over engineering

wu-changxing · 2026-01-06T04:02:03Z

+def page_scroll_strategy(page, times: int):
+    """Scroll the page window."""
+    for i in range(times):
+        page.evaluate("window.scrollBy(0, 1000)")
+        time.sleep(1)


over engineering

wu-changxing · 2026-01-06T04:02:20Z

+    if not page:
+        return "Browser not open"
+


over enginering

wu-changxing · 2026-01-06T04:03:08Z

use gmail or such kind of env to test this one and make the code simple and elegant

…lly, passing instance to cli.py

…escription and the click to perform form filling operations

…o google_search

wu-changxing

LGTM

wu-changxing · 2026-01-07T03:27:08Z

+def create_agent(headless: bool = False) -> Tuple[Agent, WebAutomation]:
+    """
+    Initialize and return the browser agent and web automation instance.
+
+    Args:
+        headless (bool): Whether to run the browser in headless mode.

-# Create the agent with browser tools and streaming plugins
-# image_result_formatter converts base64 screenshots to vision format for LLM to see
-# ui_stream sends real-time activity events to connected UI clients via WebSocket
-agent = Agent(
-    name="browser_agent",
-    model="co/gemini-3-flash-preview",
-    system_prompt=Path(__file__).parent / "prompts" / "agent.md",
-    tools=web,
-    plugins=[image_result_formatter, ui_stream],  # Vision + real-time streaming
-    max_iterations=50  # Increased for scrolling through all emails
-)
+    Returns:
+        Tuple[Agent, WebAutomation]: The configured agent and web instance.
+    """


wu-changxing · 2026-01-08T03:26:03Z

+    def submit_form(self) -> str:
+        """Submit the current form."""
+        # Try common submit buttons
+        for selector in [
+            "button[type='submit']",
+            "input[type='submit']",
+            "button:has-text('Submit')",
+            "button:has-text('Send')",
+            "button:has-text('Continue')",
+            "button:has-text('Next')"
+        ]:
+            if self.page.locator(selector).count() > 0:
+                self.page.click(selector)
+                return "Form submitted"
+
+        # Try pressing Enter in the last form field
+        if self.form_data:
+            last_field = list(self.form_data.keys())[-1]
+            self.page.press(f"input[name='{last_field}']", "Enter")
+            return "Form submitted with Enter key"
+
+        return "Could not find submit button"


find_ele_by_descrition.

wu-changxing · 2026-01-08T03:26:48Z

+    def find_forms(self) -> List[FormField]:
+        """Find all form fields on the current page."""
+        fields_data = self.page.evaluate(
+            """
+            () => {
+                const fields = [];
+                const inputs = document.querySelectorAll('input, textarea, select');
+
+                inputs.forEach(input => {
+                    const label = input.labels?.[0]?.textContent ||
+                                input.placeholder ||
+                                input.name ||
+                                input.id ||
+                                'Unknown';
+
+                    fields.push({
+                        name: input.name || input.id || label,
+                        label: label.trim(),
+                        type: input.type || input.tagName.toLowerCase(),
+                        value: input.value || '',
+                        required: input.required || false,
+                        options: input.tagName === 'SELECT' ?
+                                Array.from(input.options).map(o => o.text) : []
+                    });
+                });
+
+                return fields;
+            }
+        """
+        )
+
+        return [FormField(**field) for field in fields_data]
+
+    def fill_form(self, data: Dict[str, str]) -> str:
+        """Fill multiple form fields at once."""
+        results = []
+        for field_name, value in data.items():
+            result = self.type_text(field_name, value)
+            results.append(f"{field_name}: {result}")
+
+        return "\n".join(results)


wu-changxing · 2026-01-08T03:27:20Z

+# Simple, clear data models
+class FormField(BaseModel):
+    """A form field on a web page."""
+    name: str = Field(..., description="Field name or identifier")
+    label: str = Field(..., description="User-facing label")
+    type: str = Field(..., description="Input type (text, email, select, etc.)")
+    value: Optional[str] = Field(None, description="Current value")
+    required: bool = Field(False, description="Is this field required?")
+    options: List[str] = Field(default_factory=list, description="Available options for select/radio")
+


wu-changxing · 2026-01-08T03:27:55Z

+    def smart_fill_form(self, user_info: str) -> str:
+        """Generate smart form values and fill the form."""
+        fields = self.find_forms()
+        class FormData(BaseModel):
+            values: Dict[str, str]
+
+        field_descriptions = "\n".join([f"- {f.name}: {f.label}" for f in fields])
+        result = llm_do(
+            f"Generate form values based on: {user_info}\n\nFields:\n{field_descriptions}",
+            output=FormData,
+            model=self.DEFAULT_AI_MODEL,
+            temperature=0.7
+        )
+        return self.fill_form(result.values)


wu-changxing · 2026-01-08T04:06:24Z

+class FormField(BaseModel):
+    """A form field on a web page."""
+    name: str = Field(..., description="Field name or identifier")
+    label: str = Field(..., description="User-facing label")
+    type: str = Field(..., description="Input type (text, email, select, etc.)")
+    value: Optional[str] = Field(None, description="Current value")
+    required: bool = Field(False, description="Is this field required?")
+    options: List[str] = Field(default_factory=list, description="Available options for select/radio")
+


wu-changxing · 2026-01-08T04:09:01Z

+        return f"User confirmed login to {site_name}"
+
+    @xray
+    def get_search_results(self, query: str, max_results: int = 10) -> List[Dict[str, str]]:


wu-changxing · 2026-01-08T04:10:36Z

+    def scroll_page(self, direction: str = "down", amount: int = 1000) -> str:
+        """Scroll the page up or down."""
+        return scroll_strategies.scroll_page(self.page, direction, amount)
+
+    def scroll_element(self, selector: str, amount: int = 1000) -> str:
+        """Scroll a specific element."""
+        return scroll_strategies.scroll_element(self.page, selector, amount)


Thambolo and others added 6 commits December 13, 2025 23:21

refactored codebase, structured with folders, added pytest and tests

ef813ed

changed tests to headless mode

20e5f86

added CI pipeline

9ffcf95

fixed indentation for scroll strategies

410e27b

cleaned up tests and updated ci.yml to handle secrets and env variables

1cdc88f

updated README.md

34c5aa6

Thambolo self-assigned this Dec 16, 2025

Thambolo and others added 3 commits December 16, 2025 18:50

Merge branch 'main' into deep-research-feature

751165d

Merge branch 'main' of https://github.com/openonion/browser-agent int…

926e871

…o deep-research-feature

Merge branch 'deep-research-feature' of https://github.com/openonion/…

0289ee8

…browser-agent into deep-research-feature

wu-changxing requested changes Dec 17, 2025

View reviewed changes

wu-changxing reviewed Dec 17, 2025

View reviewed changes

Thambolo and others added 12 commits December 17, 2025 12:55

updated requirements and setup.sh print

aa97ba5

moved agent state into the class

ae55740

Merge branch 'deep-research-feature' of https://github.com/openonion/…

c9e9660

…browser-agent into deep-research-feature

removed .co/config.toml from repo and removed static configs in web_a…

db0a6ba

…utomation.py

Merge branch 'deep-research-feature' of https://github.com/openonion/…

6ce9907

…browser-agent into deep-research-feature

added test for chromium login

3b2d035

fixed merge conflicts for tests

4b8af27

removed perrsistent profile, login fresh each time, and bypasses the …

0b121ed

…google signin catcher for automated browserr

removed debugging test

cd4d01c

collapsed entrypoint.py into agent.py and uses typer+rich. headless f…

39a710f

…lag is now used

fixed PR review

2ab354b

merged

0efbabc

Thambolo requested a review from wu-changxing December 18, 2025 05:08

Thambolo and others added 4 commits December 18, 2025 15:42

fixed all merge issues

949b432

added pytest-xdist to run tests in parallel

2f145c6

tangkenzee added 7 commits January 2, 2026 18:19

persistent profiles are working

dee67c0

Merge origin/main, resolving conflicts in favor of current branch str…

93083d0

…ucture

Update .gitignore with modern Python tooling ignore patterns

717f8b4

resolved internal error lol

f8f1149

file writing results of deep research implemented

f5f7d12

fixed tests to remove the handle popup

86a9713

tangkenzee self-assigned this Jan 5, 2026

tangkenzee added 2 commits January 5, 2026 16:28

Fix CI parallel execution by using unique Chrome profiles per test

3635a2f

added more linient matching for finding elements

0f263f9

wu-changxing requested changes Jan 6, 2026

View reviewed changes

tangkenzee added 9 commits January 6, 2026 18:01

added the review comments

affc722

resolved the review comments

96deaaf

tests are running on model gemini 2.5

af82dcc

removed the create_agent function and initialized browser agent norma…

c26372a

…lly, passing instance to cli.py

new readme file to reflect changes

699d065

split the deep research agent into agent and tools

4804800

removed overengineered code, functionality retains

5478cdb

removed the form filling functions, agent shall use find_element_by_d…

5a582e7

…escription and the click to perform form filling operations

removed redunent scroll functions and renamed to get_search_results t…

8ceed80

…o google_search

wu-changxing approved these changes Jan 8, 2026

View reviewed changes

tangkenzee merged commit ec1416d into main Jan 8, 2026
1 check passed

tangkenzee deleted the deep-research-feature branch January 8, 2026 04:21

This was referenced Feb 24, 2026

Sync browser agent features from ConnectOnion CLI #4

Open

Sync advanced browser agent features from standalone browser-agent #5

Open


		## Output Format

		Your final response must be the Comprehensive Research Report itself. Do not say "I have finished research." Just provide the report.

Uh oh!

Conversation

Thambolo commented Dec 16, 2025

Uh oh!

wu-changxing left a comment

Choose a reason for hiding this comment

Problem

Proposed Structure

Why This Structure

What to Change

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wu-changxing commented Dec 17, 2025

More Over-Engineering Issues Found

1. Delete entrypoint.py - vague name

2. --headless flag is parsed but NEVER USED

3. import os inside functions

4. os.getenv('BROWSER_AGENT_MODEL') can be None

Uh oh!

wu-changxing left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wu-changxing left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

1. Delete `entrypoint.py` - vague name

2. `--headless` flag is parsed but NEVER USED

3. `import os` inside functions

4. `os.getenv('BROWSER_AGENT_MODEL')` can be None