[Feature]: Allow the browser to read images in web pages

[Miguel0312]

🚀 Background Description

Currently, the mobile and the browser agents use two different strategies: while the former uses exclusively screenshots of the UI, the latter relies entirely on the HTML and the DOM. This has a clear limitation for the browser agent: it cannot read images in web pages even when the underlying LLM can interpret them.

Design Ideas

The solution to the problem would be to add a tool to the browser-agent. This has to be done in two parts: the tool itself to allow the agent to take screenshots as well as a probe that extract relevant regions in the web page where screenshots may be useful and adds this data to the agent's context.

Involved Public APIs

The browser-agent is modified, but its public API is not changed at all.

Description of Relevance to Other Modules

No response

Test Design and Test Plan

A simple workflow that can be tested is a script similar to the following:

import asyncio
import os
import uuid

from openjiuwen.core.foundation.llm import init_model
from openjiuwen.core.runner.runner import Runner
from openjiuwen.harness.subagents.browser_agent import create_browser_agent

os.environ.setdefault("API_BASE", "ApiBase")
os.environ.setdefault("API_KEY", "ApiKey")
os.environ.setdefault("MODEL_PROVIDER", "OpenAI")
os.environ.setdefault("MODEL_NAME", "Gemma4-26B")
os.environ.setdefault("LLM_SSL_VERIFY", "false")

os.environ.setdefault("PLAYWRIGHT_MCP_ARGS", "-y @playwright/mcp@latest --browser chromium")

CHART_URL = "https://www.chartjs.org/docs/latest/samples/bar/vertical.html"

TASK = (
    f"Open {CHART_URL} and report the index values in the x-axis as well as the colors of the bars"
)

# Create the model the browser agent drives. The browser runtime reuses this
# client config (provider / key / base / model name) for its own settings.
model = init_model(
    provider=os.getenv("MODEL_PROVIDER"),
    model_name=os.getenv("MODEL_NAME"),
    api_key=os.getenv("API_KEY"),
    api_base=os.getenv("API_BASE"),
    verify_ssl=os.getenv("LLM_SSL_VERIFY").lower() == "true",
)

# Create the browser agent. It mounts the Playwright MCP tools plus the runtime
# helpers, including browser_vision for canvas-only content like this chart.
browser_agent = create_browser_agent(
    model=model,
    workspace=None,
    max_iterations=25,
    language="en",
    browser_key="vision-check",
)


# Run Agent
async def main():
    await Runner.start()
    try:
        await browser_agent.ensure_initialized()
        await browser_agent.prepare_task_resources()
        try:
            invoke_result = await Runner.run_agent(
                browser_agent,
                {
                    "query": TASK,
                    "conversation_id": f"browser_vision_{uuid.uuid4().hex[:12]}",
                },
            )
        finally:
            await browser_agent.cleanup_task_resources()
    finally:
        await Runner.stop()

    print(f"result_type >>> {invoke_result.get('result_type')}")
    print(f"BrowserAgent output result >>> {invoke_result.get('output')}")


asyncio.run(main())

Additional Information

No response

[openjiuwen-collaboration-bot]

注意

此 Issue 需要负责人。 如果没有指派时， iamcandiceguo 是默认的负责人