From 836294fe23290179abeba013580cb2feea726247 Mon Sep 17 00:00:00 2001 From: bowen628 Date: Sun, 26 Jul 2026 17:44:56 +0800 Subject: [PATCH 1/3] docs: add computer-use refactor plan and audit findings --- docs/plans/computer-use-refactor-plan.md | 229 +++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100644 docs/plans/computer-use-refactor-plan.md diff --git a/docs/plans/computer-use-refactor-plan.md b/docs/plans/computer-use-refactor-plan.md new file mode 100644 index 0000000000..b342634ac2 --- /dev/null +++ b/docs/plans/computer-use-refactor-plan.md @@ -0,0 +1,229 @@ +# BitFun Computer Use / 浏览器控制能力重构方案 + +> 范围：`src/crates/assembly/core` 的 ComputerUse / ControlHub / browser_control / web 工具族、`src/apps/desktop/src/computer_use/*` 执行层、`src/crates/execution` 契约层、web-ui 配置与模式面。 +> 依据：三份内部代码排查 + cua / Codex CLI / Anthropic computer-use-demo / browser-use / playwright-mcp & stagehand 五份标杆调研。所有文件路径均来自排查实证。 + +--- + +## 1. 现状诊断 + +### 1.1 重叠控制路径盘点 + +**"控制/使用浏览器"共 7 条并存路径：** + +| # | 路径 | 入口 | 状态 | +|---|------|------|------| +| 1 | ControlHub `domain=browser`（自研 Rust CDP，~40 action，~4900 行） | `control_hub_tool.rs` + `browser_control/*` | 最深实现，恒开 | +| 2 | agent-browser 内置技能（vercel-labs npm CLI，自带完整 CDP 栈 + 同名 `@eN` ref） | `builtin_skills/agent-browser/SKILL.md` | agentic/Claw/coding 默认开启 | +| 3 | ComputerUse 桌面视觉/AX 路径（可物理操作浏览器窗口） | `computer_use_tool.rs` | guard 是死代码，实际不设防 | +| 4 | ComputerUse `open_url`/`open_file`（OS 默认浏览器，不可控） | `computer_use_actions.rs:1512` | 活 | +| 5 | ControlHub `browser.open_builtin`（内置面板，纯展示、agent 读不回） | `EventHandlerModule.ts:2174` | 单向 | +| 6 | 内容读取族：WebFetch / browser.fetch / read_article / get_html / get_text / evaluate | `web/fetch.rs`、`browser_control/actions.rs` | 4-5 条重叠，无路由指导 | +| 7 | 用户自配 MCP（如 Playwright MCP） | `mcp_tools.rs` | 潜在 | + +**ComputerUse 内部"点击一个目标"共 7 种方言：** `locate→mouse_move→click`、`click_element`、`click_target`、`move_to_text→click`、`app_click{6 种寻址}`、`interactive_click(i)`、`visual_click(i)`；叠加 **4 套坐标系**（image px / native px / global logical / normalized 0-1000）。 + +### 1.2 核心耦合点 + +1. **`computer_use_tool.rs` ↔ `computer_use_actions.rs` 互相调用成环**（open_app 三跳绕环；未来漏配 dispatcher 列表即无限递归），且 ComputerUse 借用 `control_hub/errors.rs` 的封套，`domain='desktop'` 泄漏——两套错误形状在同一工具内混用。 +2. **ComputerUseHost 是 60+ 方法胖 trait**（`tools/computer_use_host.rs`，560 行），把截图裁剪、OCR 信任、点击守卫状态机、Set-of-Mark、Codex 风格 app_* 全部塞进一个接口，执行层（`src/apps/desktop/src/computer_use/*`，~1.9 万行）与工具层强耦合。 +3. **桌面 host 是进程级单例**（`lib.rs:1615`）+ `APP_LOOP_TRACKER` 全局静态，会话状态跨 session 污染。 +4. **三层能力开关互不咬合**：cargo feature 是装饰（`tool-provider-groups` 的 `enabled_feature_groups()` 无消费者）、`core.integration` 把 7 组能力绑死、ControlHub `is_enabled()` 硬编码 true。 +5. **三份 LOCAL_ONLY deny 表手工同步已漂移**（FE `peer-device-adapter.ts` / desktop `peer_host_invoke.rs` / cli `peer_host/deny.rs`），与 `remote_workspace_policy.rs` 的 LocalOnly 声明是两套独立"本地性"语义。 +6. **概念身份不收敛**：ComputerUse 同时是模式（FE CORE_AGENT_IDS）、restricted SubAgent（`agent-runtime/src/agents.rs:96`）、工具名（`toolGroups.ts:78`），文案三种称呼。 + +### 1.3 问题清单（合并去重，按 severity 排序） + +#### Critical + +| # | 问题 | 关键文件 | +|---|------|---------| +| C1 | **双浏览器自动化栈并存，prompt 指导互相矛盾**：SKILL.md 说优先 agent-browser，claw_mode.md/computer_use_mode.md 说用 ControlHub；两套 `@eN` ref 空间、两个浏览器实例、登录态互不可见 | `builtin_skills/agent-browser/SKILL.md`、`control_hub_tool.rs`、`agents/prompts/claw_mode.md`、`skills/policy.rs` | +| C2 | **浏览器/桌面边界 guard 是不可达死代码**：`desktop_action_targets_browser` 只在 `handle_desktop` 落空分支被调，而它想拦的 click/type_text 等在 `call_impl` 内联处理，边界只剩提示词 | `computer_use_actions.rs`、`computer_use_tool.rs` | +| C3 | **单一 god-tool：40 action、7 条点击方言、4 套坐标系、5KB 手写描述 ×2 份**，参数命名互不一致（`text_contains`/`target_text`/`text_query`/`ocr_text.needle`） | `computer_use_tool.rs`（2181 行）、`computer_use_actions.rs`（1648 行） | +| C4 | **headless 模式是假的且危险**：无 headless 启动实现，与 default 共用 9222 端口；可能附着用户真实登录浏览器却标为 "Headless test browser" | `control_hub_tool.rs:539-575,603`、`services-integrations/src/browser_control/launcher.rs` | +| C5 | **关闭 Computer use 后 ControlHub 仍可完全控制用户真实浏览器**：`is_enabled()` 恒 true、不申报 permission intent、无全局规则可 deny，文案却承诺"关闭后任何模式都不启用" | `control_hub_tool.rs:1925`、`GlobalPermissionRulesDialog.tsx`、`locales/en-US/settings/session-config.json` | +| C6 | **Firefox/Safari 用户被双向锁死**：guard 拒绝桌面输入并指向 ControlHub，而 BrowserKind 只支持 Chromium 系，CDP connect 必败 | `computer_use_actions.rs:121-150`、`launcher.rs:25-32`、`computer_use_mode.md` | + +#### Major + +| # | 问题 | 关键文件 | +|---|------|---------| +| M1 | 模型可见文案引用已删除的幽灵工具（ComputerUseMouseStep/MousePrecise/MouseClick），诱导调用不存在的工具 | `tool-contracts/src/computer_use.rs:1214,1584`、`tool-contracts/src/framework.rs:2269-2279`、`computer_use_tool.rs:992`、`computer_use_host.rs` | +| M2 | 桌面 host 会话状态进程级单例 + `APP_LOOP_TRACKER` 全局静态，跨 session 污染守卫/截图缓存/循环检测 | `desktop/src/lib.rs:1615`、`desktop_host/mod.rs`、`computer_use_actions.rs:30` | +| M3 | 大量死代码：input/result shim、verification/RetryStrategy 未接线、`handle_system('open_app')` 不可达、`enabled_feature_groups()` 无消费者 | `computer_use_input.rs`、`computer_use_result.rs`、`computer_use_verification.rs`、`tool-execution/src/computer_use.rs`、`tool-provider-groups/src/lib.rs` | +| M4 | tool ↔ actions 调用环 + ControlHub 错误封套泄漏，模型看到两种失败形状 | `computer_use_tool.rs`、`computer_use_actions.rs`、`control_hub/errors.rs` | +| M5 | 每次截图无条件写入 `/.bitfun/computer_use_debug/`，无门控、无轮转，隐私风险 | `computer_use_tool.rs`（try_save_screenshot_for_debug） | +| M6 | 未对接任何 provider 原生 computer-use 形态（Anthropic computer_20250124/OpenAI computer-use-preview），模型无法复用训练先验；多模态回传仅限两类 converter | `computer_use_tool.rs`、`tool-execution/src/context.rs` | +| M7 | text-only 门控不一致：schema 仍暴露 Set-of-Mark 纯视觉 action，`handle_desktop_ax` 附图不检查多模态能力 | `computer_use_tool.rs:249`、`computer_use_actions.rs:648-838` | +| M8 | `frame`/`frame_main` 是死功能（active_frame 无读者）；同源 iframe 内点击坐标缺 offset 修正，点错位置 | `control_hub_tool.rs:1657-1698`、`browser_control/actions.rs:552-568` | +| M9 | `control_hub_tool.rs` 2668 行 god file，错误分类靠 `to_lowercase().contains(...)` 字符串猜测反推 ErrorCode | `control_hub_tool.rs`、`browser_control/actions.rs` | +| M10 | 内容读取 4-5 条路径无选择指导，`browser.fetch` 带用户登录态发任意请求却与 WebFetch 无差异化约束 | `web/fetch.rs`、`agentic_mode.md` | +| M11 | TerminalControl 与 ControlHub terminal 域双入口，session id 发现口径各说各话 | `terminal_control_tool.rs`、`control_hub_tool.rs:1717-1815` | +| M12 | `open_builtin` 内置面板单向不可观察，模型易误以为可继续 snapshot | `EventHandlerModule.ts:2174`、`control_hub_tool.rs` | +| M13 | Peer 模式混合机器语义：SessionConfig 裸调 Tauri invoke 打本机，configManager 写远端——权限弹窗弹在控制端、工具跑在 peer 端 | `SessionConfig.tsx:165-205,593,644-695`、`peer-device-adapter.ts`、`peer_host_invoke.rs` | +| M14 | 三份 deny 表漂移（speech_* 只在 FE、CLI 缺项），`browser_control_*` 不在任何 deny 表——控制器可静默在 peer 主机启动浏览器 | `peer-device-adapter.ts`、`peer_host_invoke.rs`、`cli/src/peer_host/deny.rs`、`remote_workspace_policy.rs` | +| M15 | ComputerUse 模式禁用门禁只在 FE 下拉生效，slash 命令 `/ComputerUse` 不拦截，后端不校验，工具静默缺失无解释 | `ChatInput.tsx:2440,4130,4977`、`AgentsScene.tsx:704` | + +#### Minor（合并列举） + +- 结果 JSON 三重字段别名（image_jpeg_width/image_width/display_width_px…）；scroll 的 `scroll_x/scroll_y` 绕过 `ensure_global_xy_on_display` 边界守卫；num_clicks 循环模拟双击不用 CGEvent click_state（`computer_use_tool.rs:1581`、`tool-contracts/src/computer_use.rs`）。 +- `analyze_image_tool.rs` / `view_image_tool.rs` 整段复制 ResolvedImagePath/读取逻辑，三条图片链路无统一选择指引。 +- 陈旧注释指向不存在的 `claw_mode.md`（应为 computer_use_mode.md）、ControlHub 域口吻残留、loop 警告用已删除的 `desktop.screenshot` 语法；Linux 后端仅 141 行空壳但 schema 不裁剪。 +- cdp 方法白名单是摆设（evaluate 全权可绕）；`--remote-debugging-port=9222` hint 教用户裸暴露登录态。 +- `ai.computer_use_enabled` 订阅逻辑三处复制且初始默认值矛盾（true vs false，首帧误导）；`SessionConfig.tsx` 1775 行双 variant 互相污染；`AIFeaturesConfig.tsx` 死组件；`computer_use_open_system_settings` Windows 分支 UI 不可达；FE AIConfig 类型漂移；模型无视觉能力时无降级提示。 + +--- + +## 2. 标杆对比 + +| 维度 | BitFun 现状 | cua | Codex CLI | Anthropic demo | browser-use | playwright-mcp / stagehand | 差距结论 | +|------|------------|-----|-----------|----------------|-------------|---------------------------|---------| +| **动作空间** | 40 action 自造方言，7 条点击路径 | OpenAI+Anthropic 动作并集，按 tag 分发 | 工具极少（shell/apply_patch/view_image），GUI 委托 MCP | 日期版本化 enum，10-17 个动作，服务端定义 schema | ~20 个结构化动作，index 为句柄 | 每域一文件的声明式小工具 | **决策面失控**：应收敛到标准动作集 + 版本化 enum | +| **provider 原生形态** | 无，自造 5KB 描述现学 | 模型 regex 注册表，边缘转换到原生 computer_20251124 / computer-use-preview | — | 原生 Anthropic-defined tool，客户端零 schema | 能力门控换 schema | — | **放弃训练先验是执行质量差的直接原因** | +| **坐标处理** | 4 套坐标系并存，scroll 绕过校验 | per-screenshot scale factor 追踪 + reset | — | `scale_coordinates(source,x,y)` 单函数双向 | 截图尺寸→viewport 换算，坐标是门控降级 | ref 免坐标 | **需要唯一的双向缩放模块** | +| **浏览器交互范式** | CDP JS 注入 + `@eN` 属性写入，两套 ref 栈打架 | pixel + BrowserTool 页级动词 | 委托 MCP | — | **a11y 三树合并 + index 句柄**（成功率来源） | **aria snapshot + ref**，坐标隔离在 vision capability | **语义引用优先，坐标降级** 是业界共识 | +| **观察闭环** | 动作后需另调 screenshot；augment_result 附零散字段 | 执行器烘焙 post-action 截图 | — | 动作后 2s settle + 自动截图 | 动作即回灌新状态 + diff `*` 标注 | Response 聚合器：快照+tab diff+事件一并回传 | **"动作即观察"缺失，回合数浪费** | +| **工具契约/注册** | 手写双份 JSON schema + 测试防漂移 | Protocol + 注册表 | **spec 与 runtime 同 trait 对象**、ToolExposure 四态、每回合 spec_plan 组装 | ToolGroup{version,tools,beta_flag} 注册表 | 装饰器 + schema 自动派生 + 域名过滤 | Tool{capability,kind,zod,handle} + filteredTools | **schema/实现分离导致漂移；应 spec-runtime 同体** | +| **分层** | 工具层直连 60+ 方法胖 trait，执行层在 Tauri 进程内 | Provider ⊥ Interface ⊥ Handler 三层正交 | 契约 crate ⊥ 编排 ⊥ 风险编排 ⊥ 沙箱 crate | UI/loop/dispatch/tool/executor 五层 | Agent/Registry/Session/DOM 四层 | tools/mcp/backend 三层 | **BitFun 缺清晰层界，横切关注点全内联** | +| **会话状态** | 进程级单例 Mutex + 全局静态 | per-Computer 实例 | per-turn 组装 + 会话级审批缓存 | per-session 对象 | per-BrowserSession | per-context | **必须 per-session 键控** | +| **错误模型** | 两套封套混用 + 字符串猜 ErrorCode | 结构化 tool-error item，永不 abort | 稳定 ErrorCode + 审批 key | ToolError→is_error tool_result 唯一转换点 | 一切异常→ActionResult(error) 回灌 | 可恢复错误 + 恢复指令（"Try new snapshot"） | **需要唯一异常边界 + 稳定 code** | +| **安全/审批** | guard 死代码、ControlHub 恒开无 intent、9222 裸端口 hint | safety_checks 透传（含 TODO） | 审批 key 化缓存 + 沙箱升级阶梯 + 网络审批独立流 | Docker 沙箱 + prompt injection 分类器承接 | 敏感数据 `` 占位 + 域名白名单 | allowed/blockedOrigins 网络层强制 + element 描述供审批 UI | **安全边界应在 Rust 核心强制，不在 prompt** | +| **截图/上下文管理** | JPEG 无条件落盘 + 全量回传 | ImageRetentionCallback、trajectory 落盘可 replay | 输出截断一等策略 | 按块修剪保护 prompt cache | 干净截图 + 人用高亮分离 | 大输出写文件 + outputMaxSize | **无 retention 策略，落盘无门控** | + +--- + +## 3. 目标架构 + +### 3.1 分层设计 + +``` +┌────────────────────────────────────────────────────────────────┐ +│ L3 模式与配置面 │ +│ · 单一开关面: ai.computer_use_enabled ⊇ browser_control │ +│ · permission intents: computer_use + browser_control │ +│ · 每回合工具组装 (仿 codex spec_plan): 按模型能力/平台/远程裁剪 │ +│ · deny 表单一真源 (Rust 导出 + contract test 三端对齐) │ +├────────────────────────────────────────────────────────────────┤ +│ L2 工具面 (模型可见) │ +│ · Desktop: Anthropic 原生 computer 形态 (Claude) / │ +│ 标准化自定义 fallback (其他模型)；辅助定位工具独立 │ +│ · Browser: 单一栈, snapshot@ref 交互, 坐标为门控降级 │ +│ · Response 聚合器: 动作即观察 │ +│ · 统一 ToolResult / 稳定 ErrorCode / 唯一异常边界 │ +├────────────────────────────────────────────────────────────────┤ +│ L1 执行后端层 (Surface traits) │ +│ · DesktopSurface: screenshot/click/type/key/scroll/drag/ │ +│ ax_snapshot/window_ops (per-session 状态) │ +│ · BrowserSurface: CDP snapshot/resolve_ref/click/fill/ │ +│ navigate/fetch/events (session registry 保留) │ +│ · 坐标策略唯一模块: scale(Api↔Physical) + DPI 折算 │ +├────────────────────────────────────────────────────────────────┤ +│ L0 契约层 (独立 crate, 不依赖 Session) │ +│ · Action enum (serde tag, 版本化, 对齐 Anthropic 动作集) │ +│ · ToolResult{output,error,image,system} / TargetRef │ +│ · ToolSpec 与执行绑定同一 trait 对象 (spec-runtime 同体) │ +└────────────────────────────────────────────────────────────────┘ +``` + +### 3.2 关键决策及理由 + +**决策 1：桌面控制走"视觉坐标为主干 + AX 为辅助定位"，对 Claude 映射 Anthropic 原生 computer 工具形态。** +- 理由：桌面没有普适的 DOM；Anthropic computer_20250124/20251124 动作集（screenshot/left_click/type/key/scroll/zoom…）是模型训练过的先验，cua 与 Anthropic demo 证明按原生形态声明（display_width_px = 实际发送截图尺寸）可直接消除"从 5KB 描述现学"的质量损失（对应 M6）。 +- 现有的 AX（`windows_ax_ui`/`macos_ax_ui`）、OCR、Set-of-Mark 能力**不删除，降级为补充**：合并为一个 `desktop_snapshot`（UIA/AX 树序列化为带 ref 的文本，服务 text-only 模型与精确定位）+ 一个 `desktop_find`（文本检索），不再作为并列的 7 条点击方言。40 action 收敛为：观察（screenshot/snapshot/get_app_state）、定位（单一 `target` 对象语法：`{ref} | {text} | {x,y}`，内部按 AX→OCR→coords 阶梯解析，复用现有 click_target 解析器）、动作（click/type/key/scroll/drag/wait）、系统（open_app/open_url/clipboard），约 15 个。 +- 坐标系收敛为 2 套：模型空间（= 发送截图尺寸）与物理空间，唯一双向函数（仿 demo 的 `scale_coordinates`），Windows per-monitor DPI 折算进同一变换；删除 normalized 0-1000 与 "Ignored…host rejects" 参数。 + +**决策 2：浏览器控制走 accessibility-tree/DOM 引用（snapshot@ref），视觉坐标仅作能力门控降级。** +- 理由：browser-use（69k stars）与 playwright-mcp 的一致结论——有可枚举语义树就用索引：确定性、可校验、不受截图缩放/DPI 影响；两家的成功率投资都在语义树提取（三树合并/paint-order 过滤），不在视觉 grounding。现有 ControlHub snapshot 已有 `@eN` ref 机制，方向正确，需修 iframe 坐标（M8）并把交互参数统一为 `{element: 人类可读描述, target: ref|selector}` 双通道（描述供审批 UI）。 + +**决策 3：浏览器自动化栈二选一——保留 ControlHub Rust CDP 栈为唯一路径，agent-browser 技能降为默认关闭的 opt-in（对应 C1）。** +- 理由：ControlHub 栈在自己进程内、可被权限系统/审批/deny 表统一管辖、与 web-ui 事件打通；agent-browser 是外部 npm CLI + 独立 Chromium + 独立 auth vault，无法纳入 BitFun 的权限与 Peer 策略，且两套 `@eN` ref 并存是模型出错最大源头。`skills/policy.rs` 的 `resolve_builtin_default_enabled` 全模式改 false。 + +**决策 4：浏览器/桌面边界 guard 真正执行，且区分 CDP 可控与不可控浏览器（对应 C2/C6）。** +- 把 `desktop_action_targets_browser` 移到统一 dispatcher 的输入类动作入口前；前台是 **Chromium 系（CDP 可控）** 时拒绝并指向 browser 工具；前台是 **Firefox/Safari** 时放行桌面控制（走视觉坐标路径），消除双向锁死。 + +**决策 5：会话状态 per-session 键控。** `DesktopComputerUseHost` 的 `ComputerUseSessionMutableState` 改为 `HashMap`（或经 ToolUseContext 注入 per-session 包装），删除 `APP_LOOP_TRACKER` 全局静态（M2）。 + +**决策 6：唯一异常边界 + 稳定错误形状。** 定义 `ComputerToolError`（带稳定 code + 给模型的恢复指令，如 "ref stale, take a new snapshot"），dispatcher 是唯一 catch 点，转 `is_error` tool result；ComputerUse 停用 ControlHub 封套；`browser_control/actions.rs` 直接产生带 code 的错误，删除 `map_dispatch_error` 字符串猜测（M4/M9）。 + +**决策 7：动作即观察。** 执行器在每个 mutating 动作后：settle 延迟（桌面固定/浏览器等 network idle）→ 自动截图或快照 diff → 打包进同一 result（仿 playwright-mcp Response 聚合器 + cua post-action screenshot）。配套截图 retention（只留最近 N 张，按块修剪保护 prompt cache）。 + +**决策 8：能力开关收敛为两级真实门控。** 删除装饰性 cargo feature 层；ControlHub 实现真实 `is_enabled()`（服从 `ai.computer_use_enabled` 或新增 `ai.browser_control_enabled`，按 DeliveryProfile/远程会话裁剪）；新增 `browser_control` permission intent 进后端枚举与 `GlobalPermissionRulesDialog.tsx`（对应 C5）。 + +--- + +## 4. 重构路线图 + +每阶段可独立合并、可编译可测；前 3 阶段不改变模型可见行为面（除删除幽灵引用），从阶段 4 起改变工具面需 A/B 验证。 + +### 阶段 0：止血（~1 周，即速赢清单，见 §5） + +### 阶段 1：死代码清理 + 打断环 + 错误统一（~1 周） + +- **删除**：`computer_use_input.rs`、`computer_use_result.rs`、`computer_use_verification.rs`、`tool-execution/src/computer_use.rs` 中未接线的 `RetryStrategy`/`detect_visual_change`、`handle_system('open_app')` 分支、`ComputerUseHost::get_action_history`、`tool-provider-groups` 的 `enabled_feature_groups()` 装饰层、`AIFeaturesConfig.tsx`、`computer_use_actions.rs:1277` 空注释段。 +- **打断环**：新建 `computer_use/dispatch.rs`，`call_impl` 与 `handle_desktop`/`handle_system` 全部单向汇入；删除 `handle_desktop` 尾部反向 new `ComputerUseTool` 的 fallback。 +- **错误统一**：ComputerUse 全面切换到自有错误类型（稳定 code），移除对 `control_hub/errors.rs` 的依赖；把 `ComputerUseActions` 的 system_* 测试从 `control_hub_tests`（`control_hub_tool.rs:2487-2648`）移回所属文件。 +- 风险：低（删的都是零引用代码）。验证：`cargo build` 全工作区 + 现有 4 个 schema 防漂移单测 + grep 确认零引用。 + +### 阶段 2：动作空间收敛（~2 周） + +- **文件**：`computer_use_tool.rs`、`computer_use_actions.rs`、`computer_use_locate.rs`、`tool-contracts/src/computer_use.rs`。 +- 40 action → ~15：`click_element`/`move_to_text`/`locate` 退化为 `click_target` 统一解析器的内部实现并从 schema 移除；`app_click` 六种寻址与 `interactive_click`/`visual_click` 合并进单一 `target` 语法；`delta_x`/`dx` 等双收参数、三重结果别名（保留 `image_*` 与 `native_*` 各一组）清理。 +- **坐标模块**：新建 `tool-contracts/src/computer_use/coords.rs`，唯一 `scale(source, x, y)` 双向函数 + DPI；scroll 的 `scroll_x/scroll_y` 补 `ensure_global_xy_on_display`；删除 normalized 0-1000。 +- **text-only 门控统一**：text-only schema 移除 interactive/visual view action；`handle_desktop_ax` 附件统一走 `require_multimodal_tool_output_for_screenshot` 同款检查。 +- **边界 guard 落地**（决策 4）：guard 移入 dispatcher 输入动作入口，Firefox/Safari 放行。 +- 风险：中——模型行为面变化。验证：保留旧 action 名为 alias 一个版本期（deserialize 兼容 + deprecation 警告）；用现有 ComputerUse 子代理跑固定任务集（打开 app、点击、输入、滚动）录 trajectory 对比回合数与成功率。 + +### 阶段 3：会话状态 per-session + host trait 瘦身（~1-2 周） + +- **文件**：`src/apps/desktop/src/lib.rs:1615`、`desktop_host/mod.rs`、`computer_use_actions.rs:30`、`computer_use_host.rs`、`api/computer_use_api.rs`。 +- `DesktopComputerUseHost` 状态按 session key 键控；删除 `APP_LOOP_TRACKER` 静态（循环检测并入 per-session optimizer）；Tauri 命令与管线共享同一实例边界定义。 +- `ComputerUseHost` 60+ 方法按 §3.1 拆为 `DesktopSurface`（输入/截图/窗口）+ `AxProvider`（快照/定位）+ `OcrProvider`，工具层只依赖窄接口。 +- 风险：中（并发路径）。验证：新增两会话并发单测（守卫/循环检测互不干扰）；macOS/Windows 手测。 + +### 阶段 4：provider 原生形态映射（~2 周） + +- **文件**：`computer_use_tool.rs`、`tool-execution/src/context.rs`、provider converter 层。 +- 新建版本注册表（仿 demo `groups.py`）：Claude 模型 → `computer_20250124`/`computer_20251124` + beta header，工具声明 `display_width_px/height` = 实际截图尺寸，默认 `enable_zoom`；非 Claude 模型沿用阶段 2 收敛后的自定义 schema；describe_screen 文本降级保留。 +- 截图 retention（最近 N 张、按块修剪）进上下文管理器。 +- 风险：中高——converter 改动影响所有多模态回传。验证：Anthropic 直连 + OpenAI 兼容两条链路的集成测试；同任务集对比原生形态 vs 自定义形态成功率（预期显著提升）。 + +### 阶段 5：浏览器栈收敛（~2-3 周） + +- **文件**：`skills/policy.rs`（agent-browser 全模式默认 false）、`control_hub_tool.rs`、`browser_control/actions.rs`、`launcher.rs`、`claw_mode.md`/`computer_use_mode.md`/`agentic_mode.md`。 +- 修 iframe：`element_center` 累加 `frameElement.getBoundingClientRect()` 偏移；删除死功能 `frame`/`frame_main`。 +- headless 修复：`launch_with_cdp_opts` 实现真 headless 启动（独立端口 9223+、独立 user-data-dir），connect 校验 `/json/version` Headless 标识；绝不与 default 共用 9222。 +- 结构化错误：`actions.rs` 直接返回 ErrorCode，删 `map_dispatch_error`；`control_hub_tool.rs` 按域拆文件。 +- 合并 TerminalControl 双入口（保留 ControlHub terminal 域，注销独立注册）；`open_builtin` 返回值明示"面板不可观察"或补 URL/标题回传事件。 +- **路由指导集中成一份**注入所有相关 prompt：WebFetch（无登录态读文）→ browser.read_article/fetch（登录态读）→ browser connect/snapshot（交互）→ ComputerUse（非 CDP 浏览器/桌面）→ open_builtin（给用户看）。 +- 风险：中——agent-browser 用户回退路径需公告；headless 改动影响现有连接流程。验证：Chromium/Edge/Brave 连接矩阵测试 + iframe 点击回归页面 + prompt 一致性 grep 测试。 + +### 阶段 6：配置/权限/Peer 面（~2 周） + +- **文件**：`control_hub_tool.rs`（真实 `is_enabled`）、`GlobalPermissionRulesDialog.tsx` + 后端 intent 枚举（新增 `browser_control`）、`session-config.json` 文案修正、`SessionConfig.tsx`（拆 personalization/permissions 两组件、状态命令走传输适配层或标注本机/远端）、`peer-device-adapter.ts`/`peer_host_invoke.rs`/`cli/peer_host/deny.rs`（Rust 单一真源导出 + contract test，`browser_control_*` 补 deny）、`ChatInput.tsx`/`AgentsScene.tsx`（抽 `useComputerUseEnabled()` hook，slash 路径补门禁，门禁移后端 `get_available_modes`）、`agents.rs`/`agentVisibility.ts`（统一 ComputerUse 身份与命名）。 +- 风险：低中。验证：deny 表 contract test 三端对齐；Peer 场景手测开关/权限弹窗归属；关闭 computer use 后确认 ControlHub browser 域同步禁用。 + +--- + +## 5. 速赢清单（一周内，高价值小改动） + +1. **清除幽灵工具名**（M1，半天）：`tool-contracts/src/computer_use.rs:1214,1584`、`framework.rs:2269-2279`、`computer_use_tool.rs:992`、`computer_use_host.rs` doc 中的 ComputerUseMouseStep/MousePrecise/MouseClick 全替换为现行 action 名。直接消除"模型调用不存在工具"的失败循环。 +2. **截图落盘加门控**（M5，半天）：`try_save_screenshot_for_debug` 改为 debug 配置开关（默认关）+ 数量/天数轮转，`.bitfun/computer_use_debug` 进默认 gitignore。 +3. **headless 误标止血**（C4，1 天）：在真 headless 实现前，`control_hub_tool.rs:539-603` 的 headless connect 至少校验 `/json/version` 是否含 Headless，否则报错而非标 "Headless test browser"；hint 改为引导 BitFun 托管 profile（`launch_with_cdp_opts` 已支持 `managed_profile_root`）而非教用户裸开 9222。 +4. **边界 guard 最小落地**（C2/C6，1 天）：`desktop_action_targets_browser` 调用点移入 `call_impl` 的 click/type/key/scroll/drag 分发前；`is_probably_browser_app` 关键词表移除 firefox/safari。 +5. **slash 门禁补齐**（M15，半天）：`ChatInput.tsx` 的 `selectSlashCommandMode`（L4130-4137）与 SlashModeItem 列表复用 `modeDisabled` 检查。 +6. **文案矛盾统一**（C1 部分，半天）：`claw_mode.md`/`computer_use_mode.md`/agent-browser SKILL.md local_patch 三处路由指令统一为一个口径（过渡期先统一说 ControlHub）；修正 `computer_use_actions.rs:26` 的 `claw_mode.md` 错误引用。 +7. **text-only schema 裁剪**（M7，半天）：`input_schema_text_only`（`computer_use_tool.rs:249`）移除 build_interactive_view/interactive_click/build_visual_mark_view/visual_click。 +8. **scroll 坐标守卫**（半天）：`computer_use_tool.rs:1581-1586` 的 `scroll_x/scroll_y` 补 `ensure_global_xy_on_display` 校验。 +9. **删除四个零引用死文件**（半天）：`computer_use_input.rs`、`computer_use_result.rs`、`computer_use_verification.rs`、`AIFeaturesConfig.tsx`。 +10. **`useComputerUseEnabled()` hook**（半天）：统一 `ChatInput.tsx:872`/`AgentsScene.tsx:245`/`SessionConfig.tsx` 三处复制的订阅逻辑，初始值统一 false，消除首帧误导。 +11. **文案过度承诺修正**（C5 部分，半天）：`session-config.json` 的 enableDesc 在 ControlHub 真实门控落地前，先如实说明"浏览器控制（ControlHub）不受此开关约束"。 + +--- + +### 附：预期收益 + +- 模型决策面从 40 action / 7 点击方言 → ~15 action / 1 条定位语法；Claude 直接吃训练先验（阶段 4 是执行质量的最大单点收益）。 +- 浏览器控制从 7 条路径 → 1 条主路径（snapshot@ref）+ 明确降级阶梯，两套 `@eN` 冲突消失。 +- 安全面从"提示词约束 + 恒开工具"→ 双 intent 权限 + 真实开关 + deny 表单一真源。 +- 代码量预计净删 6-8 千行（死代码 + 重复方言 + 双份 schema），`control_hub_tool.rs` 与 `computer_use_tool.rs` 两个 2000+ 行 god file 拆解为按域模块。 \ No newline at end of file From 57d9cc7500d7b22b7faec48caddf4d47dbe06996 Mon Sep 17 00:00:00 2001 From: bowen628 Date: Sun, 26 Jul 2026 17:44:56 +0800 Subject: [PATCH 2/3] refactor(computer-use): land phase-0 quick wins from audit - Enforce the browser/desktop boundary guard in ComputerUse dispatch (previously unreachable dead code); narrow browser detection to Chromium-family so Firefox/Safari users are not locked out - Remove ghost tool references (ComputerUseMouseStep/MousePrecise/ MouseClick) from model-visible prompts, errors and MiniApp deny list - Validate CDP /json/version before labelling a headless session; stop hinting users to expose a debug port on their everyday browser - Trim visual-only actions from the text-only ComputerUse schema and reject them at runtime with a clear error - Gate debug screenshot persistence behind an env flag with retention - Guard scroll_x/scroll_y with the display-bounds check like other pointer actions - Delete dead compat shims (computer_use_input/result/verification, AIFeaturesConfig) and their boundary-check rules - Unify ai.computer_use_enabled subscription into a shared hook with a consistent initial value; close the slash-command bypass of the ComputerUse mode gate; correct over-promising toggle copy - Align browser routing guidance across mode prompts and the agent-browser skill (ControlHub-first ladder) --- .../rules/source/forbidden-rules.mjs | 20 -- .../core/builtin-skills-upstreams.json | 2 +- .../builtin_skills/agent-browser/SKILL.md | 4 +- .../agentic/agents/prompts/agentic_mode.md | 1 + .../src/agentic/agents/prompts/claw_mode.md | 6 + .../agents/prompts/computer_use_mode.md | 8 +- .../src/agentic/tools/computer_use_host.rs | 8 +- .../tools/computer_use_verification.rs | 9 - .../implementations/computer_use_actions.rs | 89 +++++-- .../implementations/computer_use_input.rs | 69 ------ .../implementations/computer_use_result.rs | 110 --------- .../implementations/computer_use_tool.rs | 199 +++++++++++++++- .../tools/implementations/control_hub_tool.rs | 101 +++++++- .../src/agentic/tools/implementations/mod.rs | 2 - .../assembly/core/src/agentic/tools/mod.rs | 1 - .../tool-contracts/src/computer_use.rs | 12 +- .../execution/tool-contracts/src/framework.rs | 12 - .../src/app/scenes/agents/AgentsScene.tsx | 21 +- .../src/flow_chat/components/ChatInput.tsx | 27 +-- .../config/components/AIFeaturesConfig.tsx | 224 ------------------ .../config/components/SessionConfig.tsx | 3 +- .../config/hooks/useComputerUseEnabled.ts | 41 ++++ .../en-US/settings/session-config.json | 2 +- .../zh-CN/settings/session-config.json | 2 +- .../zh-TW/settings/session-config.json | 2 +- 25 files changed, 439 insertions(+), 536 deletions(-) delete mode 100644 src/crates/assembly/core/src/agentic/tools/computer_use_verification.rs delete mode 100644 src/crates/assembly/core/src/agentic/tools/implementations/computer_use_input.rs delete mode 100644 src/crates/assembly/core/src/agentic/tools/implementations/computer_use_result.rs delete mode 100644 src/web-ui/src/infrastructure/config/components/AIFeaturesConfig.tsx create mode 100644 src/web-ui/src/infrastructure/config/hooks/useComputerUseEnabled.ts diff --git a/scripts/core-boundaries/rules/source/forbidden-rules.mjs b/scripts/core-boundaries/rules/source/forbidden-rules.mjs index 38211ac172..3c4ded44a0 100644 --- a/scripts/core-boundaries/rules/source/forbidden-rules.mjs +++ b/scripts/core-boundaries/rules/source/forbidden-rules.mjs @@ -2149,26 +2149,6 @@ export const forbiddenContentRules = [ }, ], }, - { - path: 'src/crates/assembly/core/src/agentic/tools/computer_use_verification.rs', - patterns: [ - { - regex: /\bpub struct VerificationResult\b/, - message: - 'core Computer Use verification facade must not own verification contracts; use tool-runtime computer_use', - }, - { - regex: /\bpub struct RetryStrategy\b/, - message: - 'core Computer Use verification facade must not own retry strategy state; use tool-runtime computer_use', - }, - { - regex: /\bpub fn detect_visual_change\b/, - message: - 'core Computer Use verification facade must not own visual-change logic; use tool-runtime computer_use', - }, - ], - }, { path: 'src/crates/assembly/core/src/agentic/session/turn_skill_agent_snapshot_store.rs', patterns: [ diff --git a/src/crates/assembly/core/builtin-skills-upstreams.json b/src/crates/assembly/core/builtin-skills-upstreams.json index 54dd13fc1b..d15e4a66f6 100644 --- a/src/crates/assembly/core/builtin-skills-upstreams.json +++ b/src/crates/assembly/core/builtin-skills-upstreams.json @@ -20,7 +20,7 @@ "revision": "81c336c1c20b80ac648e0416a7b6e0c0ae7878bb", "package_version": "0.32.3", "local_patches": [ - "route web and supported Electron work to agent-browser and native desktop work to BitFun ComputerUse", + "prefer BitFun ControlHub browser domain when available, use agent-browser only when ControlHub is unavailable or for supported Electron work, and route native desktop work to BitFun ComputerUse", "pin the documented install version and require user approval", "preserve explicit missing-prerequisite and no-silent-fallback behavior" ] diff --git a/src/crates/assembly/core/builtin_skills/agent-browser/SKILL.md b/src/crates/assembly/core/builtin_skills/agent-browser/SKILL.md index a0ece3da22..409b975866 100644 --- a/src/crates/assembly/core/builtin_skills/agent-browser/SKILL.md +++ b/src/crates/assembly/core/builtin_skills/agent-browser/SKILL.md @@ -1,6 +1,6 @@ --- name: agent-browser -description: Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser for web and supported Electron automation; use BitFun ComputerUse for native desktop UI that agent-browser cannot reach. +description: Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer BitFun's ControlHub browser domain for web automation when it is available; use agent-browser only when ControlHub is unavailable or for supported Electron automation, and use BitFun ComputerUse for native desktop UI that agent-browser cannot reach. allowed-tools: Bash(agent-browser:*), Bash(npx agent-browser:*) hidden: true --- @@ -9,6 +9,8 @@ hidden: true Fast browser automation CLI for AI agents. Chrome/Chromium via CDP with accessibility-tree snapshots and compact `@eN` element refs. +Prefer BitFun's `ControlHub` browser domain when it is available; use this skill only when `ControlHub` is unavailable. The two stacks use separate browser instances, element refs, and login state, so do not mix them within one task. + Install only after user approval: `npm i -g agent-browser@0.32.3 && agent-browser install` If the CLI is unavailable and the user declines installation, explain the missing prerequisite and offer a non-browser fallback; do not silently switch tools. diff --git a/src/crates/assembly/core/src/agentic/agents/prompts/agentic_mode.md b/src/crates/assembly/core/src/agentic/agents/prompts/agentic_mode.md index 57a0a2aefb..623b66ceea 100644 --- a/src/crates/assembly/core/src/agentic/agents/prompts/agentic_mode.md +++ b/src/crates/assembly/core/src/agentic/agents/prompts/agentic_mode.md @@ -89,6 +89,7 @@ The user will primarily request you perform software engineering tasks. This inc - When the user explicitly asks to complete work and review it carefully, finish the implementation first, then dispatch at most one independent read-only `CodeReview` Task. Do not fan out `CodeReview` into architecture, performance, security, product, or other invented dimensions: broader coverage belongs to the unified `/review` path, which selects bounded review lenses and owns cost confirmation. Do not launch review by default for every task. - Treat reviewer output as adversarial evidence. The reviewer never fixes its own findings. Apply accepted fixes in the implementation agent, then request a fresh independent review only when the change or risk warrants it. - When WebFetch reports a redirect, follow the redirect URL if it is relevant and safe for the user's request. +- For browser and web-page work, route in this order: (1) reading page content that does not require the user's login state: use WebFetch; (2) pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs); (3) non-Chromium browsers (Firefox/Safari) or native desktop apps: use `ComputerUse` desktop actions. Prefer `ControlHub` over browser-automation skills such as `agent-browser`; use those skills only when `ControlHub` is unavailable. - When multiple tool calls are independent, run them in parallel. Keep dependent operations sequential, and never use placeholders or guess missing parameters. - Use specialized tools for file reads, edits, searches, and deletions because they preserve workspace context and permissions. Use ExecCommand for commands that genuinely need a shell. Do not use shell commands only to communicate with the user. - For security-sensitive tasks, support defensive analysis and remediation only. Refuse malicious code, exploit workflows, credential harvesting, or instructions that would facilitate abuse. diff --git a/src/crates/assembly/core/src/agentic/agents/prompts/claw_mode.md b/src/crates/assembly/core/src/agentic/agents/prompts/claw_mode.md index 5794d5c4d6..72a803daa6 100644 --- a/src/crates/assembly/core/src/agentic/agents/prompts/claw_mode.md +++ b/src/crates/assembly/core/src/agentic/agents/prompts/claw_mode.md @@ -20,6 +20,12 @@ Use `ControlHub` for browser automation, terminal signalling, and routing/capabi - `domain: "terminal"` for signalling existing terminal sessions, such as interrupting or killing them. - `domain: "meta"` for capability and route checks. +For browser and web-page work, route in this order: + +1. Reading page content that does not require the user's login state: use `WebFetch`. +2. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs). +3. Non-Chromium browsers (Firefox/Safari) or native desktop apps: delegate to a `ComputerUse` session as described below. + Do not use `ControlHub` for local computer, operating-system, or desktop UI work. Desktop and system actions have moved to the dedicated `ComputerUse` tool/agent. This includes screenshots, OCR, mouse, keyboard, app state, app launching, opening files or URLs through the OS, clipboard access, OS facts, and local scripts. If the user asks you to operate or inspect the local computer, delegate the task to a `ComputerUse` session via SessionControl/SessionMessage only when both tools appear in your current tool list. Include the user's goal, target app/window/site, safety constraints, and expected verification in the handoff. If delegation is unavailable, explain that the task needs the Computer Use mode. diff --git a/src/crates/assembly/core/src/agentic/agents/prompts/computer_use_mode.md b/src/crates/assembly/core/src/agentic/agents/prompts/computer_use_mode.md index 950d9f7ab1..de82fa4362 100644 --- a/src/crates/assembly/core/src/agentic/agents/prompts/computer_use_mode.md +++ b/src/crates/assembly/core/src/agentic/agents/prompts/computer_use_mode.md @@ -67,7 +67,13 @@ When Runtime Context indicates the primary model does not support image understa # Browser Work -For websites and web apps, prefer `ControlHub` with `domain: "browser"` when it is available so cookies, login state, and extensions are preserved. If `ControlHub` is unavailable, do not claim browser-domain automation; use `ComputerUse` only for browser chrome or OS-level interaction that it can actually observe and verify. +For websites and web apps, route in this order: + +1. Reading page content that does not require the user's login state: use `WebFetch` when it is available. +2. Pages that require the user's login state or JavaScript interaction: use `ControlHub` with `domain: "browser"` (connect, snapshot, then act through `@eN` refs) so cookies, login state, and extensions are preserved. +3. Non-Chromium browsers (Firefox/Safari) or native desktop apps: use `ComputerUse` desktop actions. + +If `ControlHub` is unavailable, do not claim browser-domain automation; use `ComputerUse` only for browser chrome or OS-level interaction that it can actually observe and verify. Use desktop-domain controls only for browser chrome, OS dialogs, permission prompts, file pickers, or when browser-domain capabilities are unavailable. diff --git a/src/crates/assembly/core/src/agentic/tools/computer_use_host.rs b/src/crates/assembly/core/src/agentic/tools/computer_use_host.rs index 08ec04a533..88b02c6496 100644 --- a/src/crates/assembly/core/src/agentic/tools/computer_use_host.rs +++ b/src/crates/assembly/core/src/agentic/tools/computer_use_host.rs @@ -88,7 +88,7 @@ pub trait ComputerUseHost: Send + Sync + std::fmt::Debug { /// Fails if no screenshot was taken in this process since startup (or since last host reset). fn map_image_coords_to_pointer(&self, x: i32, y: i32) -> BitFunResult<(i32, i32)>; - /// Same as `map_image_coords_to_pointer` but **sub-point** precision (macOS: use for `ComputerUseMousePrecise`). + /// Same as `map_image_coords_to_pointer` but **sub-point** precision (macOS: use for the `mouse_move` action). fn map_image_coords_to_pointer_f64(&self, x: i32, y: i32) -> BitFunResult<(f64, f64)> { let (a, b) = self.map_image_coords_to_pointer(x, y)?; Ok((a as f64, b as f64)) @@ -110,10 +110,10 @@ pub trait ComputerUseHost: Send + Sync + std::fmt::Debug { async fn mouse_move(&self, x: i32, y: i32) -> BitFunResult<()>; - /// Move the pointer by `(dx, dy)` in **global screen pixels** (same space as `ComputerUseMousePrecise` absolute). + /// Move the pointer by `(dx, dy)` in **global screen pixels** (same space as absolute `mouse_move` globals). async fn pointer_move_relative(&self, dx: i32, dy: i32) -> BitFunResult<()>; - /// Click at the **current** pointer position only (does not move). Use `ComputerUseMousePrecise` / `ComputerUseMouseStep` / `pointer_move_rel` first. + /// Click at the **current** pointer position only (does not move). Use `mouse_move` / `move_to_text` / `pointer_move_rel` first. /// `button`: "left" | "right" | "middle" /// On desktop, enforces the vision fine-screenshot guard (unlike [`mouse_click_authoritative`](Self::mouse_click_authoritative)). async fn mouse_click(&self, button: &str) -> BitFunResult<()>; @@ -187,7 +187,7 @@ pub trait ComputerUseHost: Send + Sync + std::fmt::Debug { /// After a successful `screenshot_display`, the model may `mouse_click` (until the pointer moves again). fn computer_use_after_screenshot(&self) {} - /// After `ComputerUseMousePrecise` / `ComputerUseMouseStep` / relative pointer moves: the next `mouse_click` must be preceded by a new screenshot. + /// After `mouse_move` / `pointer_move_rel` pointer moves: the next `mouse_click` must be preceded by a new screenshot. fn computer_use_after_pointer_mutation(&self) {} /// After `mouse_click`, require a fresh screenshot before the next click (unless pointer moved, which also invalidates). diff --git a/src/crates/assembly/core/src/agentic/tools/computer_use_verification.rs b/src/crates/assembly/core/src/agentic/tools/computer_use_verification.rs deleted file mode 100644 index 0f45644db0..0000000000 --- a/src/crates/assembly/core/src/agentic/tools/computer_use_verification.rs +++ /dev/null @@ -1,9 +0,0 @@ -pub use tool_runtime::computer_use::{ - detect_visual_change, generate_retry_suggestion, RetryStrategy, VerificationResult, -}; - -use crate::util::errors::BitFunError; - -pub fn should_retry_action(error: &BitFunError, action_type: &str) -> bool { - tool_runtime::computer_use::should_retry_action_message(&error.to_string(), action_type) -} diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_actions.rs b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_actions.rs index 7c9c963327..1407669ca9 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_actions.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_actions.rs @@ -22,8 +22,8 @@ use super::control_hub::{coded_tool_error, err_response, ControlHubError, ErrorC /// Key = target PID, value = `(target_signature, before_digest, count)`. /// When the same `(action,target)` lands on an unchanged digest twice in a /// row the dispatcher injects an `app_state.loop_warning` so the model is -/// forced off the failing path on its **next** turn (`/Screenshot policy/ -/// Mandatory screenshot moments` in `claw_mode.md`). +/// forced off the failing path on its **next** turn (see the observe → act → +/// verify guidance in `computer_use_mode.md`). type AppLoopTracker = std::sync::OnceLock>>; @@ -107,7 +107,7 @@ impl ComputerUseActions { ControlHubError::new( ErrorCode::GuardRejected, format!( - "desktop.{} is blocked while {} is frontmost. Use ControlHub domain=\"browser\" for all browser interaction; desktop mouse/keyboard browser control is forbidden.", + "ComputerUse `{}` is blocked while {} is frontmost. Use ControlHub domain=\"browser\" for all browser interaction; desktop mouse/keyboard browser control is forbidden.", action, app_name ), ) @@ -130,26 +130,21 @@ impl ComputerUseActions { .unwrap_or("") .to_ascii_lowercase(); - const NAME_HINTS: &[&str] = &[ - "chrome", - "chromium", - "edge", - "brave", - "arc", - "firefox", - "safari", - "browser", - "浏览器", - ]; - const BUNDLE_HINTS: &[&str] = &[ - "chrome", "chromium", "edge", "brave", "arc", "firefox", "safari", "browser", - ]; + // Only Chromium-family browsers are guarded: they are the only ones the + // ControlHub browser domain can drive over CDP. Firefox/Safari (and other + // non-Chromium browsers) have no CDP path, so desktop control must stay + // allowed for them — blocking both surfaces would leave no control path. + const NAME_HINTS: &[&str] = &["chrome", "chromium", "edge", "brave", "arc"]; + const BUNDLE_HINTS: &[&str] = &["chrome", "chromium", "edge", "brave", "arc"]; NAME_HINTS.iter().any(|hint| name.contains(hint)) || BUNDLE_HINTS.iter().any(|hint| bundle.contains(hint)) } - async fn desktop_action_targets_browser( + /// Rejects physical input actions while a CDP-drivable browser is frontmost. + /// Read-only observation actions (`screenshot`, `locate`, `describe_screen`, …) + /// stay allowed. Called by `ComputerUseTool::call_impl` before dispatch. + pub(crate) async fn desktop_action_targets_browser( &self, action: &str, context: &ToolUseContext, @@ -166,7 +161,6 @@ impl ComputerUseActions { "key_chord", "type_text", "paste", - "locate", "move_to_text", ]; if !guarded_actions.contains(&action) { @@ -356,10 +350,6 @@ impl ComputerUseActions { _ => {} } - if let Some(err) = self.desktop_action_targets_browser(action, context).await { - return Ok(err_response("desktop", action, err)); - } - // UX shortcut: every screen-coordinate action accepts an optional // `display_id`. If present (and different from the currently pinned // display), pin it BEFORE forwarding so the model doesn't need a @@ -837,6 +827,26 @@ impl ComputerUseActions { result_with_optional_screenshot(data, summary, shot_opt) } + // These actions only make sense with a marked-up screenshot the model + // can look at; text-only models are steered to the AX/OCR text paths. + if text_only + && matches!( + action, + "build_interactive_view" + | "interactive_click" + | "build_visual_mark_view" + | "visual_click" + ) + { + return Err(coded_tool_error( + ErrorCode::NotAvailable, + format!( + "`{}` requires a vision-capable primary model (its result is a marked-up screenshot). Use `describe_screen` or `get_app_state` to observe as text, then act with `app_click`, `click_target`, `move_to_text`, or `key_chord`.", + action + ), + )); + } + let bg = host.supports_background_input(); let ax = host.supports_ax_tree(); @@ -1591,6 +1601,8 @@ fn error_code_from_local(code: &str) -> ErrorCode { #[cfg(test)] mod tests { use super::loop_tracker_observe; + use super::ComputerUseActions; + use crate::agentic::tools::computer_use_host::ComputerUseForegroundApplication; // A unique PID avoids interference with the shared APP_LOOP_TRACKER state // across tests in the same process. @@ -1636,6 +1648,37 @@ mod tests { ); } + fn foreground(name: &str, bundle_id: &str) -> ComputerUseForegroundApplication { + ComputerUseForegroundApplication { + name: Some(name.to_string()), + bundle_id: Some(bundle_id.to_string()), + process_id: Some(1), + } + } + + /// Only Chromium-family browsers are CDP-drivable via the ControlHub + /// browser domain. Firefox/Safari must NOT trip the desktop browser guard + /// or the user would have no control path at all. + #[test] + fn browser_guard_matches_only_chromium_family() { + assert!(ComputerUseActions::is_probably_browser_app(&foreground( + "Google Chrome", + "com.google.Chrome" + ))); + assert!(ComputerUseActions::is_probably_browser_app(&foreground( + "Microsoft Edge", + "com.microsoft.edgemac" + ))); + assert!(!ComputerUseActions::is_probably_browser_app(&foreground( + "Firefox", + "org.mozilla.firefox" + ))); + assert!(!ComputerUseActions::is_probably_browser_app(&foreground( + "Safari", + "com.apple.Safari" + ))); + } + /// A genuine tree mutation (digest changes) must NOT trigger the warning, /// even on the same target — progress resets the streak. #[test] diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_input.rs b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_input.rs deleted file mode 100644 index b48a33a8b3..0000000000 --- a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_input.rs +++ /dev/null @@ -1,69 +0,0 @@ -//! Compatibility shims for the former core-owned Computer Use input helpers. -//! -//! The provider-neutral implementation now lives in `bitfun-agent-tools`. - -use crate::agentic::tools::computer_use_host::{ - ComputerUseImplicitScreenshotCenter, ComputerUseNavigateQuadrant, ComputerUseScreenshotParams, - ScreenshotCropCenter, -}; -use crate::util::errors::BitFunResult; -use serde_json::Value; - -pub use bitfun_agent_tools::computer_use::{ - coordinate_mode, input_has_screenshot_crop_fields, parse_screenshot_window_flag, - use_screen_coordinates, -}; - -pub fn ensure_pointer_move_uses_screen_coordinates_only(input: &Value) -> BitFunResult<()> { - bitfun_agent_tools::computer_use::ensure_pointer_move_uses_screen_coordinates_only(input) - .map_err(Into::into) -} - -pub fn parse_screenshot_crop_center(input: &Value) -> BitFunResult> { - bitfun_agent_tools::computer_use::parse_screenshot_crop_center(input).map_err(Into::into) -} - -pub fn parse_screenshot_crop_half_extent_native(input: &Value) -> BitFunResult> { - bitfun_agent_tools::computer_use::parse_screenshot_crop_half_extent_native(input) - .map_err(Into::into) -} - -pub fn parse_screenshot_implicit_center( - input: &Value, -) -> BitFunResult> { - bitfun_agent_tools::computer_use::parse_screenshot_implicit_center(input).map_err(Into::into) -} - -pub fn parse_screenshot_navigate_quadrant( - input: &Value, -) -> BitFunResult> { - bitfun_agent_tools::computer_use::parse_screenshot_navigate_quadrant(input).map_err(Into::into) -} - -pub fn parse_screenshot_params(input: &Value) -> BitFunResult<(ComputerUseScreenshotParams, bool)> { - bitfun_agent_tools::computer_use::parse_screenshot_params(input).map_err(Into::into) -} - -#[cfg(test)] -mod tests { - use super::*; - use serde_json::json; - - #[test] - fn compatibility_parser_keeps_old_result_type_and_behavior() { - let input = json!({ - "screenshot_navigate_quadrant": "top_left", - "screenshot_crop_center_x": 120, - "screenshot_crop_center_y": 340, - "screenshot_reset_navigation": true, - }); - - let (params, ignored_crop) = - parse_screenshot_params(&input).expect("parse screenshot params"); - - assert_eq!(params.navigate_quadrant, None); - assert_eq!(params.crop_center, None); - assert!(!params.reset_navigation); - assert!(!ignored_crop); - } -} diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_result.rs b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_result.rs deleted file mode 100644 index b0a62c6ffc..0000000000 --- a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_result.rs +++ /dev/null @@ -1,110 +0,0 @@ -//! Compatibility shims for the former core-owned Computer Use result helpers. -//! -//! The active screenshot tool body builder now lives in `bitfun-agent-tools`. - -use crate::agentic::tools::computer_use_host::{ComputerScreenshot, ComputerUseInteractionState}; -use serde_json::{json, Value}; - -pub fn append_interaction_state(body: &mut Value, interaction: &ComputerUseInteractionState) { - if let Value::Object(map) = body { - map.insert("interaction_state".to_string(), json!(interaction)); - } -} - -pub fn build_screenshot_body( - shot: &ComputerScreenshot, - debug_rel: Option, - interaction: &ComputerUseInteractionState, -) -> Value { - let mut data = json!({ - "success": true, - "mime_type": shot.mime_type, - "image_jpeg_width": shot.image_width, - "image_jpeg_height": shot.image_height, - "display_native_width": shot.native_width, - "display_native_height": shot.native_height, - "display_native_origin_x": shot.display_origin_x, - "display_native_origin_y": shot.display_origin_y, - "image_width": shot.image_width, - "image_height": shot.image_height, - "display_width_px": shot.image_width, - "display_height_px": shot.image_height, - "native_width": shot.native_width, - "native_height": shot.native_height, - "display_origin_x": shot.display_origin_x, - "display_origin_y": shot.display_origin_y, - "vision_scale": shot.vision_scale, - "pointer_image_x": shot.pointer_image_x, - "pointer_image_y": shot.pointer_image_y, - "screenshot_crop_center": shot.screenshot_crop_center, - "point_crop_half_extent_native": shot.point_crop_half_extent_native, - "navigation_native_rect": shot.navigation_native_rect, - "quadrant_navigation_click_ready": shot.quadrant_navigation_click_ready, - "implicit_confirmation_crop_applied": shot.implicit_confirmation_crop_applied, - "debug_screenshot_path": debug_rel, - }); - append_interaction_state(&mut data, interaction); - data -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::agentic::tools::computer_use_host::{ - ComputerUseImageContentRect, ComputerUseInteractionScreenshotKind, - }; - - #[test] - fn compatibility_body_keeps_explicit_dimension_aliases() { - let shot = ComputerScreenshot { - screenshot_id: Some("test-shot".to_string()), - bytes: vec![1, 2, 3], - mime_type: "image/jpeg".to_string(), - image_width: 100, - image_height: 80, - native_width: 100, - native_height: 80, - display_origin_x: 0, - display_origin_y: 0, - vision_scale: 1.0, - pointer_image_x: Some(10), - pointer_image_y: Some(11), - screenshot_crop_center: None, - point_crop_half_extent_native: None, - navigation_native_rect: None, - quadrant_navigation_click_ready: false, - image_content_rect: Some(ComputerUseImageContentRect { - left: 1, - top: 2, - width: 98, - height: 76, - }), - image_global_bounds: None, - implicit_confirmation_crop_applied: false, - ui_tree_text: None, - }; - let interaction = ComputerUseInteractionState { - click_ready: false, - enter_ready: true, - requires_fresh_screenshot_before_click: true, - requires_fresh_screenshot_before_enter: false, - recommend_screenshot_to_verify_last_action: false, - last_screenshot_kind: Some(ComputerUseInteractionScreenshotKind::FullDisplay), - last_mutation: None, - recommended_next_action: Some("screenshot_navigate_quadrant".to_string()), - displays: vec![], - active_display_id: None, - }; - - let body = build_screenshot_body(&shot, None, &interaction); - - assert_eq!(body["image_jpeg_width"], json!(100)); - assert_eq!(body["display_native_width"], json!(100)); - assert_eq!(body["image_width"], body["image_jpeg_width"]); - assert_eq!(body["native_width"], body["display_native_width"]); - assert_eq!( - body["interaction_state"]["last_screenshot_kind"], - json!("full_display") - ); - } -} diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs index e8eb1c2ac7..4134fa70ff 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/computer_use_tool.rs @@ -1,7 +1,7 @@ //! Desktop automation (Computer use). use super::computer_use_locate::execute_computer_use_locate; -use super::control_hub::{coded_tool_error, ErrorCode}; +use super::control_hub::{coded_tool_error, err_response, ErrorCode}; use crate::agentic::tools::computer_use_capability::computer_use_desktop_available; use crate::agentic::tools::computer_use_host::{ AppSelector, ComputerScreenshot, ComputerUseHost, ComputerUseNavigateQuadrant, OcrRegionNative, @@ -110,8 +110,14 @@ pub(crate) async fn computer_use_augment_result_json( } /// On-disk copy of each Computer use screenshot (pointer overlay included) for debugging. +/// Opt-in: only written when [`COMPUTER_USE_DEBUG_SCREENSHOTS_ENV`] is set to `1`; +/// the directory is pruned to the newest [`COMPUTER_USE_DEBUG_MAX_FILES`] files after each write. /// Filenames: `cu__full.jpg` (whole display) or `cu__crop__.jpg` when a point crop was requested. const COMPUTER_USE_DEBUG_SUBDIR: &str = ".bitfun/computer_use_debug"; +/// Set to `1` to enable on-disk debug copies of Computer use screenshots. +const COMPUTER_USE_DEBUG_SCREENSHOTS_ENV: &str = "BITFUN_COMPUTER_USE_DEBUG_SCREENSHOTS"; +/// Newest debug screenshots retained in [`COMPUTER_USE_DEBUG_SUBDIR`]; older files are deleted. +const COMPUTER_USE_DEBUG_MAX_FILES: usize = 20; pub struct ComputerUseTool; @@ -213,7 +219,6 @@ The **primary model cannot consume images** in tool results — **do not** use * "target": { "type": "object", "description": "For `app_click`: click target such as `{ \"node_idx\": 3 }`, image/screen coordinates, or OCR text." }, "focus": { "type": ["object", "null"], "description": "For app-scoped text/scroll actions: optional focus target." }, "predicate": { "type": "object", "description": "For `app_wait_for`: wait predicate." }, - "opts": { "type": "object", "description": "For `build_interactive_view` / `build_visual_mark_view`: optional view options." }, "i": { "type": ["integer", "null"], "description": "For interactive/visual actions: element or mark index from the latest view." }, "dx": { "type": "integer", "description": "For app/interactive scroll actions: horizontal delta." }, "dy": { "type": "integer", "description": "For app/interactive scroll actions: vertical delta." }, @@ -246,7 +251,7 @@ The **primary model cannot consume images** in tool results — **do not** use * let properties = Self::merge_with_shared_properties(json!({ "action": { "type": "string", - "enum": ["click_target", "move_to_target", "click_element", "move_to_text", "click", "mouse_move", "scroll", "drag", "locate", "key_chord", "type_text", "pointer_move_rel", "wait", "list_displays", "focus_display", "paste", "list_apps", "get_app_state", "get_app_shortcuts", "describe_screen", "app_click", "app_type_text", "app_scroll", "app_key_chord", "app_wait_for", "build_interactive_view", "interactive_click", "interactive_type_text", "interactive_scroll", "build_visual_mark_view", "visual_click", "open_app", "open_url", "open_file", "clipboard_get", "clipboard_set", "run_script", "run_apple_script", "get_os_info"], + "enum": ["click_target", "move_to_target", "click_element", "move_to_text", "click", "mouse_move", "scroll", "drag", "locate", "key_chord", "type_text", "pointer_move_rel", "wait", "list_displays", "focus_display", "paste", "list_apps", "get_app_state", "get_app_shortcuts", "describe_screen", "app_click", "app_type_text", "app_scroll", "app_key_chord", "app_wait_for", "interactive_type_text", "interactive_scroll", "open_app", "open_url", "open_file", "clipboard_get", "clipboard_set", "run_script", "run_apple_script", "get_os_info"], "description": "The action to perform. **Primary model is text-only — no `screenshot`.** **ACTION PRIORITY:** 1) Use Bash tool for CLI/terminal/system commands first. 2) **`open_app`** to launch apps. **`run_apple_script`** for AppleScript (macOS). 3) Prefer `key_chord` for shortcuts/navigation. Before guessing a shortcut, call **`get_app_shortcuts`** to look up what a target app actually has registered (e.g. \"what triggers Save in this app?\"), then fire it with `key_chord` / `app_key_chord` — avoids trial-and-error mouse clicks. 4) Only when above fail: `click_target` / `move_to_target` (AX → OCR → screen coords in one call), then lower-level `click_element`, `move_to_text`, or `mouse_move` + `click`. Never guess coordinates. **`describe_screen`** is the text-only equivalent of `screenshot`: it returns a structured text snapshot (frontmost app + AX tree + UI tree text + pointer + window geometry) with NO image — use it to observe and verify state when the primary model cannot view screenshots." }, "use_screen_coordinates": { "type": "boolean", "description": "For `mouse_move`, `drag`: **must be true** — global display coordinates from `move_to_text`, `locate`, AX, or `pointer_global`. **Not** for `click`." }, @@ -768,12 +773,16 @@ The **primary model cannot consume images** in tool results — **do not** use * } /// Writes the exact JPEG sent to the model (including pointer overlay) under the workspace for debugging. + /// No-op unless [`COMPUTER_USE_DEBUG_SCREENSHOTS_ENV`] is set to `1`. async fn try_save_screenshot_for_debug( bytes: &[u8], context: &ToolUseContext, crop: Option, nav_label: Option<&str>, ) -> Option { + if std::env::var(COMPUTER_USE_DEBUG_SCREENSHOTS_ENV).as_deref() != Ok("1") { + return None; + } let root = context.workspace_root()?; let dir = root.join(COMPUTER_USE_DEBUG_SUBDIR); if let Err(e) = tokio::fs::create_dir_all(&dir).await { @@ -815,6 +824,7 @@ The **primary model cannot consume images** in tool results — **do not** use * path.display() ), } + Self::prune_debug_screenshots(&dir).await; Some(format!( "{}/{}", COMPUTER_USE_DEBUG_SUBDIR.replace('\\', "/"), @@ -822,6 +832,37 @@ The **primary model cannot consume images** in tool results — **do not** use * )) } + /// Keeps only the newest [`COMPUTER_USE_DEBUG_MAX_FILES`] files (by mtime) in the debug dir. + async fn prune_debug_screenshots(dir: &std::path::Path) { + let Ok(mut entries) = tokio::fs::read_dir(dir).await else { + return; + }; + let mut files: Vec<(std::time::SystemTime, std::path::PathBuf)> = Vec::new(); + while let Ok(Some(entry)) = entries.next_entry().await { + let Ok(meta) = entry.metadata().await else { + continue; + }; + if !meta.is_file() { + continue; + } + let modified = meta.modified().unwrap_or(std::time::UNIX_EPOCH); + files.push((modified, entry.path())); + } + if files.len() <= COMPUTER_USE_DEBUG_MAX_FILES { + return; + } + files.sort_by(|a, b| b.0.cmp(&a.0)); + for (_, path) in files.into_iter().skip(COMPUTER_USE_DEBUG_MAX_FILES) { + if let Err(e) = tokio::fs::remove_file(&path).await { + warn!( + "computer_use debug screenshot prune {}: {}", + path.display(), + e + ); + } + } + } + /// Build tool JSON + one JPEG attachment + assistant hint from an already-captured [`ComputerScreenshot`]. async fn pack_screenshot_tool_output( shot: &ComputerScreenshot, @@ -989,7 +1030,7 @@ impl Tool for ComputerUseTool { **`move_to_text`:** OCR-match visible text (`text_query`) and **move the pointer** to it (no click, no keys); **no prior `screenshot` required for targeting** (host captures **raw** pixels for Vision — no agent screenshot overlays; on macOS defaults to the **frontmost window** unless **`ocr_region_native`** overrides). Matching **strips whitespace** between CJK glyphs and allows **small edit distance** when Vision mis-reads one character. The host **trusts** the resulting globals — **next `click`** does **not** require an extra `screenshot` (same as AX). If **several** hits match, the host returns **preview JPEGs + accessibility** per candidate — pick **`move_to_text_match_index`** (1-based) and call **`move_to_text` again** with the same query/region, or narrow with **`ocr_region_native`**. Use **`click`** afterward if you need a mouse press. Prefer after `click_element` misses when text is visible. \ **`click`:** Press at **current pointer only** — **never** pass `x`, `y`, `coordinate_mode`, or `use_screen_coordinates`. Position first with **`move_to_text`**, **`mouse_move`** (**globals only**), or **`click_element`**. After pointer moves, **`screenshot`** again before the next guarded **`click`** when the host requires it. \ **`mouse_move` / `drag`:** **`use_screen_coordinates`: true** required — global coordinates from **`move_to_text`**, **`locate`**, AX, or **`pointer_global`**; never JPEG pixel guesses. \ -**`scroll` / `type_text` / `pointer_move_rel` / `wait` / `locate`:** No mandatory pre-screenshot by themselves. **`pointer_move_rel`** (and **ComputerUseMouseStep**) are **blocked immediately after `screenshot`** until **`move_to_text`**, **`mouse_move`** (globals), or **`click_element`** — do not nudge from the JPEG. \ +**`scroll` / `type_text` / `pointer_move_rel` / `wait` / `locate`:** No mandatory pre-screenshot by themselves. **`pointer_move_rel`** is **blocked immediately after `screenshot`** until **`move_to_text`**, **`mouse_move`** (globals), or **`click_element`** — do not nudge from the JPEG. \ **`key_chord`:** Press key combination; prefer over **`click`** when shortcuts or **Enter**/**Escape**/**Tab** suffice. **Mandatory fresh screenshot only** when chord includes Return/Enter. \ **`screenshot`:** JPEG for **confirmation** (optional pointer overlay). When the host requires a fresh capture before **`click`** or Enter **`key_chord`**, a bare `screenshot` is **~500×500** around the **mouse** or **caret** (also during quadrant drill). Use **`screenshot_reset_navigation`**: true to force **full-screen** for wide context. \ **`type_text`:** Type text; prefer clipboard for long content. Does **not** move the pointer — **Enter** **`key_chord`** may follow without a mandatory `screenshot` unless you moved the pointer since the last capture. If **`screenshot`** shows the correct chat is already open and the input may be focused, **try `type_text` first** before spending steps on `click_element` / `move_to_text`.", @@ -1054,6 +1095,7 @@ impl Tool for ComputerUseTool { "screenshot_implicit_center": { "type": "string", "enum": ["mouse", "text_caret"], "description": "For `screenshot` when `requires_fresh_screenshot_before_click` / `requires_fresh_screenshot_before_enter` is true: center the implicit ~500×500 on the mouse (`mouse`, default) or on the focused text control (`text_caret`, macOS AX; falls back to mouse). Applies to the **first** confirmation capture too. Ignored when you set `screenshot_crop_center_*` / `screenshot_navigate_quadrant` / `screenshot_reset_navigation`." }, "app_name": { "type": "string", "description": "For `open_app`: the application name to launch (e.g. \"Safari\", \"WeChat\", \"Visual Studio Code\")." }, "script": { "type": "string", "description": "For `run_apple_script`: the AppleScript code to execute via `osascript`. macOS only." }, + "opts": { "type": "object", "description": "For `build_interactive_view` / `build_visual_mark_view`: optional view options." }, "scroll_x": { "type": "integer", "description": "For `scroll`: optional global X coordinate to move pointer before scrolling. Use with `scroll_y`. Requires `use_screen_coordinates`: true." }, "scroll_y": { "type": "integer", "description": "For `scroll`: optional global Y coordinate to move pointer before scrolling. Use with `scroll_x`. Requires `use_screen_coordinates`: true." } })); @@ -1130,6 +1172,17 @@ impl Tool for ComputerUseTool { .and_then(|v| v.as_str()) .ok_or_else(|| BitFunError::tool("action is required".to_string()))?; + // Browser-boundary guard: physical input actions (click/type/scroll/…) + // must not drive a CDP-drivable (Chromium-family) browser from the + // desktop side — the ControlHub browser domain owns that surface. + // Read-only observation actions pass through. + if let Some(err) = super::computer_use_actions::ComputerUseActions::new() + .desktop_action_targets_browser(action, context) + .await + { + return Ok(err_response("computer_use", action, err)); + } + match action { "open_url" | "open_file" | "clipboard_get" | "clipboard_set" | "run_script" | "get_os_info" => { @@ -1529,7 +1582,7 @@ impl Tool for ComputerUseTool { Ok(vec![ToolResult::ok(body, Some(summary))]) } - // ---- NEW: mouse_move (absolute pointer move, consolidated from ComputerUseMousePrecise) ---- + // ---- mouse_move (absolute pointer move in global screen coordinates) ---- "mouse_move" => { ensure_pointer_move_uses_screen_coordinates_only(input)?; let x = req_i32(input, "x")?; @@ -1568,7 +1621,7 @@ impl Tool for ComputerUseTool { Ok(vec![ToolResult::ok(body, Some(summary))]) } - // ---- NEW: scroll (consolidated from ComputerUseMouseClick wheel action) ---- + // ---- scroll (mouse wheel; optional scroll_x/scroll_y move the pointer first) ---- "scroll" => { let dx = input.get("delta_x").and_then(|v| v.as_i64()).unwrap_or(0) as i32; let dy = input.get("delta_y").and_then(|v| v.as_i64()).unwrap_or(0) as i32; @@ -1581,7 +1634,11 @@ impl Tool for ComputerUseTool { let scroll_pos_x = input.get("scroll_x").and_then(|v| v.as_i64()); let scroll_pos_y = input.get("scroll_y").and_then(|v| v.as_i64()); if let (Some(sx), Some(sy)) = (scroll_pos_x, scroll_pos_y) { - host_ref.mouse_move_global_f64(sx as f64, sy as f64).await?; + let (gx, gy) = (sx as f64, sy as f64); + // Same display-bounds guard as mouse_move/drag: reject + // image-pixel coordinates passed as globals. + ensure_global_xy_on_display(host_ref, gx, gy).await?; + host_ref.mouse_move_global_f64(gx, gy).await?; host_ref.wait_ms(30).await?; } host_ref.scroll(dx, dy).await?; @@ -2018,7 +2075,12 @@ fn req_i32(input: &Value, key: &str) -> BitFunResult { #[cfg(test)] mod tests { use super::ComputerUseTool; + use crate::agentic::tools::computer_use_host::{ + ComputerScreenshot, ComputerUseForegroundApplication, ComputerUseHost, + ComputerUsePermissionSnapshot, ComputerUseScreenshotParams, ComputerUseSessionSnapshot, + }; use crate::agentic::tools::framework::{Tool, ToolUseContext}; + use crate::util::errors::{BitFunError, BitFunResult}; use serde_json::{json, Value}; #[test] @@ -2164,6 +2226,129 @@ mod tests { } } + /// Visual-only actions (their results are marked-up screenshots) must not + /// be advertised to text-only models; their view-options parameter goes + /// with them. + #[test] + fn visual_only_actions_are_absent_from_text_only_schema() { + let full_actions = action_enum(&ComputerUseTool::new().input_schema()); + let text_only_actions = action_enum(&ComputerUseTool::input_schema_text_only()); + for action in [ + "build_interactive_view", + "interactive_click", + "build_visual_mark_view", + "visual_click", + ] { + assert!( + full_actions.iter().any(|a| a == action), + "full schema should list `{action}`" + ); + assert!( + !text_only_actions.iter().any(|a| a == action), + "text-only schema should NOT list `{action}`" + ); + } + let text_only_keys = property_keys(&ComputerUseTool::input_schema_text_only()); + assert!( + !text_only_keys.contains("opts"), + "`opts` only configures the removed view-building actions" + ); + assert!(property_keys(&ComputerUseTool::new().input_schema()).contains("opts")); + } + + /// Minimal host whose only signal is a Chromium-family frontmost app; + /// every input primitive fails loudly so the test proves the browser + /// guard rejects `click` before any physical input is attempted. + #[derive(Debug)] + struct ChromeForegroundHost; + + fn not_expected() -> BitFunResult { + Err(BitFunError::tool( + "not expected to be called in this test".to_string(), + )) + } + + #[async_trait::async_trait] + impl ComputerUseHost for ChromeForegroundHost { + async fn permission_snapshot(&self) -> BitFunResult { + not_expected() + } + async fn request_accessibility_permission(&self) -> BitFunResult<()> { + not_expected() + } + async fn request_screen_capture_permission(&self) -> BitFunResult<()> { + not_expected() + } + async fn screenshot_display( + &self, + _params: ComputerUseScreenshotParams, + ) -> BitFunResult { + not_expected() + } + fn map_image_coords_to_pointer(&self, _x: i32, _y: i32) -> BitFunResult<(i32, i32)> { + not_expected() + } + fn map_normalized_coords_to_pointer(&self, _x: i32, _y: i32) -> BitFunResult<(i32, i32)> { + not_expected() + } + async fn mouse_move(&self, _x: i32, _y: i32) -> BitFunResult<()> { + not_expected() + } + async fn pointer_move_relative(&self, _dx: i32, _dy: i32) -> BitFunResult<()> { + not_expected() + } + async fn mouse_click(&self, _button: &str) -> BitFunResult<()> { + not_expected() + } + async fn scroll(&self, _delta_x: i32, _delta_y: i32) -> BitFunResult<()> { + not_expected() + } + async fn key_chord(&self, _keys: Vec) -> BitFunResult<()> { + not_expected() + } + async fn type_text(&self, _text: &str) -> BitFunResult<()> { + not_expected() + } + async fn wait_ms(&self, _ms: u64) -> BitFunResult<()> { + not_expected() + } + async fn computer_use_session_snapshot(&self) -> ComputerUseSessionSnapshot { + ComputerUseSessionSnapshot { + foreground_application: Some(ComputerUseForegroundApplication { + name: Some("Google Chrome".to_string()), + bundle_id: Some("com.google.Chrome".to_string()), + process_id: Some(4242), + }), + pointer_global: None, + } + } + } + + /// The browser-boundary guard must be reachable from `call_impl`: a + /// physical input action while a Chromium-family browser is frontmost is + /// rejected with the ControlHub browser-domain redirect instead of + /// clicking into the page. + #[tokio::test] + async fn click_is_rejected_while_chromium_browser_is_frontmost() { + let mut context = ToolUseContext::for_tool_listing(None, None); + context.computer_use_host = Some(std::sync::Arc::new(ChromeForegroundHost)); + let results = ComputerUseTool::new() + .call_impl(&json!({ "action": "click" }), &context) + .await + .expect("guard rejection is a structured envelope, not a hard error"); + let body = results[0].content(); + assert_eq!( + body.get("ok").and_then(Value::as_bool), + Some(false), + "guarded click should return an error envelope: {body}" + ); + let error_text = body.get("error").map(Value::to_string).unwrap_or_default(); + assert!( + error_text.contains("browser"), + "guard error should redirect to the ControlHub browser domain: {error_text}" + ); + } + /// The `action` enum, description, and a handful of other fields are /// deliberately different (richer guidance) between the two schemas. This /// test documents that the shared/override split does not silently diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs b/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs index 73bb2fce8d..e5b26f531b 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/control_hub_tool.rs @@ -12,7 +12,7 @@ use crate::agentic::tools::browser_control::actions::BrowserActions; use crate::agentic::tools::browser_control::browser_launcher::{ BrowserKind, BrowserLauncher, LaunchResult, DEFAULT_CDP_PORT, }; -use crate::agentic::tools::browser_control::cdp_client::CdpClient; +use crate::agentic::tools::browser_control::cdp_client::{CdpClient, CdpVersionInfo}; use crate::agentic::tools::browser_control::session_registry::{ BrowserSession, BrowserSessionRegistry, BrowserSessionState, DialogHandler, }; @@ -71,8 +71,8 @@ impl ControlHubTool { vec![ "For login/cookies/extensions, use the user's default browser via CDP — never fall back to desktop mouse/keyboard automation.".to_string(), format!( - "If CDP is not ready, restart the browser with the test port enabled: \"{}\" --remote-debugging-port={}", - exe, port + "If CDP is not ready on test port {}, retry browser.connect — it starts \"{}\" against BitFun's managed profile with CDP enabled. Do not ask the user to enable a debug port on their everyday browser profile.", + port, exe ), "After the browser is listening on the test port, use browser.connect / snapshot / click / fill to drive the DOM directly.".to_string(), ] @@ -89,6 +89,35 @@ impl ControlHubTool { ] } + /// `connect { mode: "headless" }` cannot launch a headless browser yet — + /// it only attaches to whatever is already listening on the CDP port, + /// which may be the user's real logged-in browser. Verify from the + /// /json/version handshake that the endpoint reports itself as headless + /// (Chromium reports e.g. "HeadlessChrome/126.0") before the session is + /// labelled a disposable test browser; mislabelling the user's default + /// browser invites destructive automation on their real profile. + fn verify_headless_cdp_browser( + version: &CdpVersionInfo, + port: u16, + ) -> Result<(), ControlHubError> { + let browser = version.browser.as_deref().unwrap_or(""); + if browser.to_ascii_lowercase().contains("headless") { + return Ok(()); + } + let reported = if browser.is_empty() { "unknown" } else { browser }; + Err(ControlHubError::new( + ErrorCode::NotAvailable, + format!( + "The browser on test port {} reports as '{}', which is not a headless browser. Headless mode requires a real headless browser instance and must never attach to the user's default browser.", + port, reported + ), + ) + .with_hints(Self::headless_browser_connect_hints(port)) + .with_hint( + "Use connect { mode: \"default\" } to drive the BitFun-managed browser profile instead.", + )) + } + fn normalize_builtin_browser_url(raw_url: &str) -> Result { let trimmed = raw_url.trim(); if trimmed.is_empty() { @@ -599,6 +628,12 @@ Branch on `ok` and `error.code`, not on English messages. match &launch_result { LaunchResult::AlreadyConnected | LaunchResult::Launched => { + let version = CdpClient::get_version(port).await?; + if mode == "headless" { + if let Err(error) = Self::verify_headless_cdp_browser(&version, port) { + return Ok(err_response("browser", "connect", error)); + } + } let pages = CdpClient::list_pages(port).await?; let connected_browser = if mode == "headless" { "Headless test browser".to_string() @@ -664,7 +699,6 @@ Branch on `ok` and `error.code`, not on English messages. BitFunError::tool("Page has no WebSocket debugger URL".to_string()) })?; let client = CdpClient::connect(ws_url).await?; - let version = CdpClient::get_version(port).await?; let session = BrowserSession { session_id: page.id.clone(), port, @@ -2455,6 +2489,65 @@ mod control_hub_tests { ); } + #[test] + fn headless_connect_rejects_non_headless_cdp_endpoint() { + // The port may be occupied by the user's real logged-in browser; + // labelling that session "Headless test browser" would invite + // destructive automation on their real profile. + for reported in [Some("Chrome/126.0.6478.127"), None] { + let version = CdpVersionInfo { + browser: reported.map(str::to_string), + protocol_version: None, + web_socket_debugger_url: None, + }; + let err = ControlHubTool::verify_headless_cdp_browser(&version, 9222) + .expect_err("non-headless endpoint must be rejected in headless mode"); + assert!(matches!(err.code, ErrorCode::NotAvailable)); + assert!( + err.message + .contains("must never attach to the user's default browser"), + "error must forbid attaching the user's default browser: {}", + err.message + ); + assert!( + err.hints.iter().any(|h| h.contains("mode: \"default\"")), + "hints must offer the default managed-profile mode: {:?}", + err.hints + ); + } + } + + #[test] + fn headless_connect_accepts_headless_cdp_endpoint() { + // "Headless" matching must be case-insensitive across Chromium's + // historical spellings. + for reported in ["HeadlessChrome/126.0.6478.127", "headlesschrome/1.0"] { + let version = CdpVersionInfo { + browser: Some(reported.to_string()), + protocol_version: None, + web_socket_debugger_url: None, + }; + assert!( + ControlHubTool::verify_headless_cdp_browser(&version, 9222).is_ok(), + "endpoint '{reported}' must be accepted as headless" + ); + } + } + + #[test] + fn default_connect_hints_point_to_managed_profile_not_user_debug_port() { + let hints = ControlHubTool::default_browser_connect_hints(&BrowserKind::Chrome, 9222); + let joined = hints.join(" | "); + assert!( + joined.contains("managed profile"), + "hints must guide toward BitFun's managed profile launch: {joined}" + ); + assert!( + !joined.contains("--remote-debugging-port"), + "hints must not teach enabling a raw debug port on the user's everyday browser: {joined}" + ); + } + #[test] fn browser_open_builtin_normalizes_domain_url() { assert_eq!( diff --git a/src/crates/assembly/core/src/agentic/tools/implementations/mod.rs b/src/crates/assembly/core/src/agentic/tools/implementations/mod.rs index 89a6dd0214..9e0c9bd7c5 100644 --- a/src/crates/assembly/core/src/agentic/tools/implementations/mod.rs +++ b/src/crates/assembly/core/src/agentic/tools/implementations/mod.rs @@ -8,9 +8,7 @@ pub mod bash_tool; pub mod canvas_tools; pub mod code_review_tool; pub mod computer_use_actions; -pub mod computer_use_input; pub mod computer_use_locate; -pub mod computer_use_result; pub mod computer_use_tool; pub mod control_hub; pub mod control_hub_tool; diff --git a/src/crates/assembly/core/src/agentic/tools/mod.rs b/src/crates/assembly/core/src/agentic/tools/mod.rs index a1fdd41fb0..fad9e9a27d 100644 --- a/src/crates/assembly/core/src/agentic/tools/mod.rs +++ b/src/crates/assembly/core/src/agentic/tools/mod.rs @@ -5,7 +5,6 @@ pub mod browser_control; pub mod computer_use_capability; pub mod computer_use_host; pub mod computer_use_optimizer; -pub mod computer_use_verification; pub(crate) mod file_permissions; pub mod file_read_state_runtime; pub mod file_tool_guidance; diff --git a/src/crates/execution/tool-contracts/src/computer_use.rs b/src/crates/execution/tool-contracts/src/computer_use.rs index 3a6c298b95..5ce62a5427 100644 --- a/src/crates/execution/tool-contracts/src/computer_use.rs +++ b/src/crates/execution/tool-contracts/src/computer_use.rs @@ -227,7 +227,7 @@ pub struct ComputerScreenshot { /// When true (desktop), `click` is allowed on this frame without an extra ~500×500 point crop — region is small enough for pointer positioning + `click`. #[serde(default, skip_serializing_if = "is_false")] pub quadrant_navigation_click_ready: bool, - /// Screen capture rectangle in JPEG pixel coordinates (offset zero when there is no frame padding); `ComputerUseMousePrecise` maps this rect to the display. + /// Screen capture rectangle in JPEG pixel coordinates (offset zero when there is no frame padding); the host maps this rect to the display. #[serde(default, skip_serializing_if = "Option::is_none")] pub image_content_rect: Option, /// Approximate global screen rectangle represented by the screenshot. Use @@ -1211,7 +1211,7 @@ pub fn ensure_pointer_move_uses_screen_coordinates_only( return Ok(()); } Err(ComputerUseContractError::tool( - "Positioning from screenshot pixels (coordinate_mode image/normalized) is disabled: do not guess coordinates from vision. Set use_screen_coordinates: true with global display coordinates from move_to_text (global_center_x/y), locate, click_element, or pointer_image_x/y from the last screenshot JSON; or use move_to_text, click_element, pointer_move_rel, ComputerUseMouseStep. Screenshots are for confirmation only.".to_string(), + "Positioning from screenshot pixels (coordinate_mode image/normalized) is disabled: do not guess coordinates from vision. Set use_screen_coordinates: true with global display coordinates from move_to_text (global_center_x/y), locate, click_element, or pointer_image_x/y from the last screenshot JSON; or use move_to_text, click_element, pointer_move_rel. Screenshots are for confirmation only.".to_string(), )) } @@ -1476,7 +1476,7 @@ pub fn build_screenshot_tool_body_and_hint( }); let shortcut_policy = format!( "**Verify step:** after **`click`**, **`key_chord`**, **`type_text`**, **`scroll`**, or **`drag`**, check **`interaction_state.recommend_screenshot_to_verify_last_action`** — when true, call **`screenshot`** next to confirm UI state (Cowork-style). \ -**Targeting priority:** `click_element` → **`move_to_text`** (OCR + move; no prior `screenshot` for targeting) → **`screenshot`** (confirm / drill) + **`mouse_move`** (**`use_screen_coordinates`: true only**) + **`click`** last. **Screenshots are for confirmation and navigation — do not guess move targets from JPEG pixels.** **`click`** never moves the pointer. **Host-only mandatory screenshot:** before **`click`** or Enter **`key_chord`** when the pointer changed since the last capture — **not** before `mouse_move`, `scroll`, `type_text`, `locate`, `wait`, or non-Enter `key_chord`. **Valid basis for a guarded `click`:** `FullDisplay`, `quadrant_navigation_click_ready`, or point crop; or bare **`screenshot`** after a pointer-changing action (**~500×500** implicit confirmation around mouse/caret). **`mouse_move`** must use **global** coordinates (from `move_to_text` global_center_*, `locate`, AX, or `pointer_global`). **Bare confirmation `screenshot`:** whenever the host still requires a capture before **`click`** or Enter **`key_chord`** (`requires_fresh_screenshot_*`), a bare `screenshot` (no crop / no reset) is **~500×500** centered on **mouse** (`screenshot_implicit_center` default `mouse`) — **including during quadrant drill** and the **first** such capture in a session. Before Enter in a text field, set **`screenshot_implicit_center`: `text_caret`**. Use **`screenshot_reset_navigation`**: true for a **full-screen** capture instead. **If AX failed:** try **`move_to_text`** before a long screenshot drill. **Optional refinement** for tiny targets: `screenshot_navigate_quadrant` until `quadrant_navigation_click_ready` (long edge < {} px) or point crop. Small moves: **ComputerUseMouseStep** over tiny **ComputerUseMousePrecise** (screen globals only).", +**Targeting priority:** `click_element` → **`move_to_text`** (OCR + move; no prior `screenshot` for targeting) → **`screenshot`** (confirm / drill) + **`mouse_move`** (**`use_screen_coordinates`: true only**) + **`click`** last. **Screenshots are for confirmation and navigation — do not guess move targets from JPEG pixels.** **`click`** never moves the pointer. **Host-only mandatory screenshot:** before **`click`** or Enter **`key_chord`** when the pointer changed since the last capture — **not** before `mouse_move`, `scroll`, `type_text`, `locate`, `wait`, or non-Enter `key_chord`. **Valid basis for a guarded `click`:** `FullDisplay`, `quadrant_navigation_click_ready`, or point crop; or bare **`screenshot`** after a pointer-changing action (**~500×500** implicit confirmation around mouse/caret). **`mouse_move`** must use **global** coordinates (from `move_to_text` global_center_*, `locate`, AX, or `pointer_global`). **Bare confirmation `screenshot`:** whenever the host still requires a capture before **`click`** or Enter **`key_chord`** (`requires_fresh_screenshot_*`), a bare `screenshot` (no crop / no reset) is **~500×500** centered on **mouse** (`screenshot_implicit_center` default `mouse`) — **including during quadrant drill** and the **first** such capture in a session. Before Enter in a text field, set **`screenshot_implicit_center`: `text_caret`**. Use **`screenshot_reset_navigation`**: true for a **full-screen** capture instead. **If AX failed:** try **`move_to_text`** before a long screenshot drill. **Optional refinement** for tiny targets: `screenshot_navigate_quadrant` until `quadrant_navigation_click_ready` (long edge < {} px) or point crop. Small moves: prefer **`pointer_move_rel`** over tiny **`mouse_move`** adjustments (screen globals only).", COMPUTER_USE_QUADRANT_CLICK_READY_MAX_LONG_EDGE ); let region_crop_size_note = shot @@ -1504,7 +1504,7 @@ pub fn build_screenshot_tool_body_and_hint( "phase": "quadrant_terminal", "image_is_crop_only": true, "shortcut_policy": shortcut_policy, - "instruction": "Region is small enough for precise pointer: **`quadrant_navigation_click_ready`** is true. **Do not** use **`ComputerUseMouseStep`** / **`pointer_move_rel`** immediately after a **`screenshot`** (host blocks — vision nudges are wrong). First **`move_to_text`**, **`mouse_move`** (`use_screen_coordinates`: true), or **`click_element`**, then optional **`ComputerUseMouseStep`** / **`ComputerUseMousePrecise`**. Then **`ComputerUseMouseClick`** (`action`: click). Host requires a **fresh** screenshot before the next **`click`** or Enter **`key_chord`** if pointer state changed since last capture (see shortcut_policy)." + "instruction": "Region is small enough for precise pointer: **`quadrant_navigation_click_ready`** is true. **Do not** use **`pointer_move_rel`** immediately after a **`screenshot`** (host blocks — vision nudges are wrong). First **`move_to_text`**, **`mouse_move`** (`use_screen_coordinates`: true), or **`click_element`**, then optional **`pointer_move_rel`**. Then **`click`**. Host requires a **fresh** screenshot before the next **`click`** or Enter **`key_chord`** if pointer state changed since last capture (see shortcut_policy)." }) } else if !screenshot_covers_full_display(shot) { json!({ @@ -1554,7 +1554,7 @@ pub fn build_screenshot_tool_body_and_hint( } let pointer_line = match (shot.pointer_image_x, shot.pointer_image_y) { (Some(px), Some(py)) => format!( - " TRUE POINTER: **red cursor with gray border** (tip = hotspot) in the JPEG at image x={}, y={} — **confirmation only**; use **`mouse_move`** with **`use_screen_coordinates`: true** using globals from tool JSON (`pointer_global`, `move_to_text`, `locate`), then **`click`**. **Do not** use **`pointer_move_rel`** / **ComputerUseMouseStep** as the next action after this **`screenshot`** (host blocks). Prior screenshot is stale after **ComputerUseMousePrecise** / **ComputerUseMouseStep** / `pointer_move_rel` until you screenshot again.", + " TRUE POINTER: **red cursor with gray border** (tip = hotspot) in the JPEG at image x={}, y={} — **confirmation only**; use **`mouse_move`** with **`use_screen_coordinates`: true** using globals from tool JSON (`pointer_global`, `move_to_text`, `locate`), then **`click`**. **Do not** use **`pointer_move_rel`** as the next action after this **`screenshot`** (host blocks). Prior screenshot is stale after **`mouse_move`** / **`pointer_move_rel`** until you screenshot again.", px, py ), _ => " TRUE POINTER: not on this capture (pointer_image_x/y null). No red synthetic cursor — OS mouse may be on another display; use use_screen_coordinates with global coords or bring the pointer here and re-screenshot." @@ -1581,7 +1581,7 @@ pub fn build_screenshot_tool_body_and_hint( ) } else if shot.quadrant_navigation_click_ready { format!( - "Quadrant terminal {}x{} (native region {:?}). **`quadrant_navigation_click_ready`**: align with **ComputerUseMouseStep** / **`mouse_move`** (**`use_screen_coordinates`: true** only) / **ComputerUseMousePrecise**, then **`ComputerUseMouseClick`** (`action`: click) — **`click`** has no coordinates.{}.{}", + "Quadrant terminal {}x{} (native region {:?}). **`quadrant_navigation_click_ready`**: align with **`mouse_move`** (**`use_screen_coordinates`: true** only) or **`pointer_move_rel`**, then **`click`** — **`click`** has no coordinates.{}.{}", shot.image_width, shot.image_height, shot.navigation_native_rect, diff --git a/src/crates/execution/tool-contracts/src/framework.rs b/src/crates/execution/tool-contracts/src/framework.rs index 3de63aad3c..b17c4933e9 100644 --- a/src/crates/execution/tool-contracts/src/framework.rs +++ b/src/crates/execution/tool-contracts/src/framework.rs @@ -2265,18 +2265,6 @@ pub fn miniapp_headless_agent_tool_restrictions() -> ToolRuntimeRestrictions { "ComputerUse", "ComputerUse is unavailable in MiniApp headless agent runs.", ), - ( - "ComputerUseMouseClick", - "ComputerUseMouseClick is unavailable in MiniApp headless agent runs.", - ), - ( - "ComputerUseMouseStep", - "ComputerUseMouseStep is unavailable in MiniApp headless agent runs.", - ), - ( - "ComputerUseMousePrecise", - "ComputerUseMousePrecise is unavailable in MiniApp headless agent runs.", - ), ( "ReviewPlatform", "ReviewPlatform is unavailable in MiniApp headless agent runs.", diff --git a/src/web-ui/src/app/scenes/agents/AgentsScene.tsx b/src/web-ui/src/app/scenes/agents/AgentsScene.tsx index 8ad36bb302..1d9c1df7be 100644 --- a/src/web-ui/src/app/scenes/agents/AgentsScene.tsx +++ b/src/web-ui/src/app/scenes/agents/AgentsScene.tsx @@ -53,7 +53,7 @@ import { isLocallyManageableSubagent, } from './agentVisibility'; import { CustomAgentAPI } from '@/infrastructure/api/service-api/CustomAgentAPI'; -import { configManager } from '@/infrastructure/config/services/ConfigManager'; +import { useComputerUseEnabled } from '@/infrastructure/config/hooks/useComputerUseEnabled'; import type { ModeSkillInfo, SubagentModelSelection } from '@/infrastructure/config/types'; import { buildSkillCoverageSourceMap, @@ -197,7 +197,7 @@ const AgentsHomeView: React.FC = () => { const [savingSkills, setSavingSkills] = React.useState(false); const [savingSubagents, setSavingSubagents] = React.useState(false); const [savingSubagentModel, setSavingSubagentModel] = React.useState(false); - const [computerUseEnabled, setComputerUseEnabled] = useState(true); + const { computerUseEnabled } = useComputerUseEnabled(); const { buildModelOption, renderModelOption, renderModelValue } = useModelSelectPresentation(); const { groups: userToolGroups, @@ -242,23 +242,6 @@ const AgentsHomeView: React.FC = () => { }, }); - useEffect(() => { - let cancelled = false; - const loadComputerUseEnabled = () => { - void configManager.getConfig('ai.computer_use_enabled').then((enabled) => { - if (!cancelled) setComputerUseEnabled(enabled ?? false); - }); - }; - loadComputerUseEnabled(); - const unsubscribe = configManager.onConfigChange((path) => { - if (path === 'ai.computer_use_enabled' || path === 'ai') loadComputerUseEnabled(); - }); - return () => { - cancelled = true; - unsubscribe(); - }; - }, []); - const coreAgentMeta = useMemo((): Record => ({ agentic: { role: t('coreAgentsZone.modes.agentic.role'), diff --git a/src/web-ui/src/flow_chat/components/ChatInput.tsx b/src/web-ui/src/flow_chat/components/ChatInput.tsx index 0feed2e0bd..70d9356f31 100644 --- a/src/web-ui/src/flow_chat/components/ChatInput.tsx +++ b/src/web-ui/src/flow_chat/components/ChatInput.tsx @@ -107,6 +107,7 @@ import { normalizeToolPermissionConfig, permissionConfigService, } from '@/infrastructure/config'; +import { useComputerUseEnabled } from '@/infrastructure/config/hooks/useComputerUseEnabled'; import type { ToolPermissionConfig } from '@/infrastructure/config/types'; import type { ModeSkillInfo } from '@/infrastructure/config/types'; import { SubagentAPI, type SubagentInfo } from '@/infrastructure/api/service-api/SubagentAPI'; @@ -869,24 +870,7 @@ export const ChatInput: React.FC = ({ const [targetModeEnabledTools, setTargetModeEnabledTools] = useState(null); const [userDefaultModeId, setUserDefaultModeId] = useState(null); const [defaultModeSavingId, setDefaultModeSavingId] = useState(null); - const [computerUseEnabled, setComputerUseEnabled] = useState(true); - - useEffect(() => { - let cancelled = false; - const loadComputerUseEnabled = () => { - void configManager.getConfig('ai.computer_use_enabled').then((enabled) => { - if (!cancelled) setComputerUseEnabled(enabled ?? false); - }); - }; - loadComputerUseEnabled(); - const unsubscribe = configManager.onConfigChange((path) => { - if (path === 'ai.computer_use_enabled' || path === 'ai') loadComputerUseEnabled(); - }); - return () => { - cancelled = true; - unsubscribe(); - }; - }, []); + const { computerUseEnabled } = useComputerUseEnabled(); const [skillsFlyoutOpen, setSkillsFlyoutOpen] = useState(false); const [skillsFlyoutLeft, setSkillsFlyoutLeft] = useState(false); @@ -3779,6 +3763,11 @@ export const ChatInput: React.FC = ({ }, [effectiveTargetSessionId, externalPromptCommandsIssue, t, workspacePath]); const selectSlashCommandMode = useCallback((modeId: string) => { + // Same gating as the mode dropdown; slash commands must not bypass it. + if (modeId === 'ComputerUse' && !computerUseEnabled) { + notificationService.warning(t('chatInput.computerUseDisabled')); + return; + } const operationGeneration = ++nativePromptModeSelectionGenerationRef.current; const operationIsCurrent = () => ( nativePromptModeSelectionGenerationRef.current === operationGeneration @@ -3832,7 +3821,7 @@ export const ChatInput: React.FC = ({ selectedIndex: 0, }); })(); - }, [dispatchInput, getSlashPickerItems, inlineTriggerState, persistExplicitNativePromptCommandChoice, requestModeChange]); + }, [computerUseEnabled, dispatchInput, getSlashPickerItems, inlineTriggerState, persistExplicitNativePromptCommandChoice, requestModeChange, t]); const selectSlashCommandAction = useCallback((actionId: SlashActionId) => { const raw = inputState.value || ''; diff --git a/src/web-ui/src/infrastructure/config/components/AIFeaturesConfig.tsx b/src/web-ui/src/infrastructure/config/components/AIFeaturesConfig.tsx deleted file mode 100644 index 5f6b560ef9..0000000000 --- a/src/web-ui/src/infrastructure/config/components/AIFeaturesConfig.tsx +++ /dev/null @@ -1,224 +0,0 @@ - - -import React, { useState, useEffect, useCallback } from 'react'; -import { useTranslation } from 'react-i18next'; -import { Switch, ConfigPageLoading } from '@/component-library'; -import { ConfigPageHeader, ConfigPageLayout, ConfigPageContent, ConfigPageSection, ConfigPageRow } from './common'; -import { aiExperienceConfigService, type AIExperienceSettings } from '../services/AIExperienceConfigService'; -import { configManager } from '../services/ConfigManager'; -import { useNotification, notificationService } from '@/shared/notification-system'; -import type { AIModelConfig } from '../types'; -import { ModelSelectionRadio } from './ModelSelectionRadio'; -import { createLogger } from '@/shared/utils/logger'; -import './AIFeaturesConfig.scss'; - -const log = createLogger('AIFeaturesConfig'); - -interface FeatureConfig { - id: string; - settingKey?: keyof AIExperienceSettings; - agentName?: string; -} - - -const FEATURE_CONFIGS: FeatureConfig[] = [ - { - id: 'sessionTitle', - settingKey: 'enable_session_title_generation', - agentName: 'startchat-func-agent', - }, -]; - -const AIFeaturesConfig: React.FC = () => { - const { t } = useTranslation('settings/ai-features'); - const notification = useNotification(); - - - const [settings, setSettings] = useState(() => - aiExperienceConfigService.getSettings() - ); - const [isLoading, setIsLoading] = useState(true); - - - const [models, setModels] = useState([]); - const [funcAgentModels, setFuncAgentModels] = useState>({}); - - const loadAllData = useCallback(async () => { - setIsLoading(true); - try { - - const [ - loadedSettings, - allModels, - funcAgentModelsData, - ] = await Promise.all([ - aiExperienceConfigService.getSettingsAsync(), - configManager.getConfig('ai.models') || [], - configManager.getConfig>('ai.func_agent_models') || {}, - ]); - - setSettings(loadedSettings); - setModels(allModels); - setFuncAgentModels(funcAgentModelsData); - } catch (error) { - log.error('Failed to load data', error); - setSettings(aiExperienceConfigService.getSettings()); - } finally { - setIsLoading(false); - } - }, []); - - useEffect(() => { - void loadAllData(); - }, [loadAllData]); - - - const getModelName = useCallback((modelId: string | null | undefined): string | undefined => { - if (!modelId) return undefined; - return models.find(m => m.id === modelId)?.name; - }, [models]); - - const updateSetting = async ( - key: K, - value: AIExperienceSettings[K] - ) => { - - const newSettings = { ...settings, [key]: value }; - setSettings(newSettings); - - - try { - await aiExperienceConfigService.saveSettings(newSettings); - notification.success(t('messages.saveSuccess')); - } catch (error) { - log.error('Failed to save AI features settings', error); - notification.error(`${t('messages.saveFailed')}: ` + (error instanceof Error ? error.message : String(error))); - - setSettings(settings); - } - }; - - - function getFeatureIdByAgent(agentName: string): string { - const feature = FEATURE_CONFIGS.find(f => f.agentName === agentName); - return feature?.id || agentName; - } - - const handleAgentSelectionChange = async ( - agentName: string, - modelId: string - ) => { - try { - const currentFuncAgentModels = await configManager.getConfig>('ai.func_agent_models') || {}; - - const updatedFuncAgentModels = { - ...currentFuncAgentModels, - [agentName]: modelId, - }; - await configManager.setConfig('ai.func_agent_models', updatedFuncAgentModels); - - setFuncAgentModels(updatedFuncAgentModels); - - - let modelDesc = ''; - if (modelId === 'primary') { - modelDesc = t('model.primary'); - } else if (modelId === 'fast') { - modelDesc = t('model.fast'); - } else { - modelDesc = getModelName(modelId) || modelId || ''; - } - - notificationService.success( - t('models.updateSuccess', { agentName: t(`features.${getFeatureIdByAgent(agentName)}.title`), modelName: modelDesc }), - { duration: 2000 } - ); - } catch (error) { - log.error('Failed to update agent model', { agentName, modelId, error }); - notificationService.error(t('messages.updateFailed'), { duration: 3000 }); - } - }; - - - - const enabledModels = models.filter(m => m.enabled); - - if (isLoading) { - return ( - - - - - - - ); - } - - return ( - - - - - {FEATURE_CONFIGS.map((feature) => { - const hasSwitch = !!feature.settingKey; - const hasModel = !!feature.agentName; - const isEnabled = hasSwitch ? Boolean(settings[feature.settingKey!]) : true; - const configuredModelId = hasModel ? (funcAgentModels[feature.agentName!] || 'fast') : 'fast'; - const warning = t(`features.${feature.id}.warning`, { defaultValue: '' }); - - return ( - - {hasSwitch && ( - -
- updateSetting(feature.settingKey!, e.target.checked)} - size="small" - /> -
- - )} - - {hasModel && ( - -
- handleAgentSelectionChange(feature.agentName!, modelId)} - layout="horizontal" - size="small" - /> -
- - )} - - ); - })} - - - - ); -}; - -export default AIFeaturesConfig; diff --git a/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx b/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx index 32119a30bc..a82b0041ce 100644 --- a/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx +++ b/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx @@ -28,6 +28,7 @@ import { type AgentCompanionPetPackage, } from '../services/AgentCompanionPetService'; import { configManager } from '../services/ConfigManager'; +import { useComputerUseEnabled } from '../hooks/useComputerUseEnabled'; import { DEFAULT_TOOL_PERMISSION_CONFIG, normalizeToolPermissionConfig, @@ -132,7 +133,7 @@ const SessionSettingsPanels: React.FC = ({ variant } const [permissionModeControlVisibilitySaving, setPermissionModeControlVisibilitySaving] = useState(false); const [isGlobalPermissionRulesDialogOpen, setIsGlobalPermissionRulesDialogOpen] = useState(false); - const [computerUseEnabled, setComputerUseEnabled] = useState(false); + const { computerUseEnabled, setComputerUseEnabled } = useComputerUseEnabled(); const [computerUseAccess, setComputerUseAccess] = useState(false); const [computerUseScreen, setComputerUseScreen] = useState(false); const [computerUseBusy, setComputerUseBusy] = useState(false); diff --git a/src/web-ui/src/infrastructure/config/hooks/useComputerUseEnabled.ts b/src/web-ui/src/infrastructure/config/hooks/useComputerUseEnabled.ts new file mode 100644 index 0000000000..09d05b32cd --- /dev/null +++ b/src/web-ui/src/infrastructure/config/hooks/useComputerUseEnabled.ts @@ -0,0 +1,41 @@ +import { useEffect, useState, type Dispatch, type SetStateAction } from 'react'; +import { configManager } from '../services/ConfigManager'; + +const COMPUTER_USE_ENABLED_CONFIG_PATH = 'ai.computer_use_enabled'; + +export interface UseComputerUseEnabledResult { + computerUseEnabled: boolean; + /** + * Local override for optimistic UI (e.g. the settings toggle). Config + * change events re-sync the value once the write lands or fails. + */ + setComputerUseEnabled: Dispatch>; +} + +/** + * Tracks the `ai.computer_use_enabled` config value. Starts as `false` until + * the config loads so ComputerUse affordances never flash as available on + * first render. + */ +export function useComputerUseEnabled(): UseComputerUseEnabledResult { + const [computerUseEnabled, setComputerUseEnabled] = useState(false); + + useEffect(() => { + let cancelled = false; + const load = () => { + void configManager.getConfig(COMPUTER_USE_ENABLED_CONFIG_PATH).then((enabled) => { + if (!cancelled) setComputerUseEnabled(enabled ?? false); + }); + }; + load(); + const unsubscribe = configManager.onConfigChange((path) => { + if (path === COMPUTER_USE_ENABLED_CONFIG_PATH || path === 'ai') load(); + }); + return () => { + cancelled = true; + unsubscribe(); + }; + }, []); + + return { computerUseEnabled, setComputerUseEnabled }; +} diff --git a/src/web-ui/src/locales/en-US/settings/session-config.json b/src/web-ui/src/locales/en-US/settings/session-config.json index 82b6d20136..2583345f87 100644 --- a/src/web-ui/src/locales/en-US/settings/session-config.json +++ b/src/web-ui/src/locales/en-US/settings/session-config.json @@ -125,7 +125,7 @@ "sectionTitle": "Computer use (desktop)", "sectionDescription": "In the BitFun desktop app, the assistant can capture the screen and control the mouse and keyboard; requires a multimodal model for vision.", "enable": "Enable Computer use", - "enableDesc": "When off, the ComputerUse tool stays disabled in every session mode.", + "enableDesc": "When off, the ComputerUse tool stays disabled in every session mode. Browser control (ControlHub) is not yet gated by this switch.", "accessibility": "Accessibility", "accessibilityDesc": "macOS: lets BitFun read on-screen UI elements and send mouse/keyboard input to other apps. Not applicable on Windows; on Linux this requires an X11 session.", "screenCapture": "Screen recording", diff --git a/src/web-ui/src/locales/zh-CN/settings/session-config.json b/src/web-ui/src/locales/zh-CN/settings/session-config.json index 8b30f8a815..b44001982e 100644 --- a/src/web-ui/src/locales/zh-CN/settings/session-config.json +++ b/src/web-ui/src/locales/zh-CN/settings/session-config.json @@ -125,7 +125,7 @@ "sectionTitle": "Computer use（桌面自动化）", "sectionDescription": "在 BitFun 桌面端允许助理截取屏幕并控制键鼠；需多模态模型理解画面。", "enable": "启用 Computer use", - "enableDesc": "关闭时，任何会话模式都不会启用 ComputerUse 工具。", + "enableDesc": "关闭时，任何会话模式都不会启用 ComputerUse 工具；浏览器控制（ControlHub）目前不受此开关约束。", "accessibility": "辅助功能", "accessibilityDesc": "macOS 上用于让 BitFun 读取屏幕上的界面元素并向其他应用发送鼠标/键盘操作；Windows 无需单独授权，Linux 需要 X11 会话。", "screenCapture": "屏幕录制", diff --git a/src/web-ui/src/locales/zh-TW/settings/session-config.json b/src/web-ui/src/locales/zh-TW/settings/session-config.json index dcc9d95b85..9a95a0ca5a 100644 --- a/src/web-ui/src/locales/zh-TW/settings/session-config.json +++ b/src/web-ui/src/locales/zh-TW/settings/session-config.json @@ -125,7 +125,7 @@ "sectionTitle": "Computer use（桌面自動化）", "sectionDescription": "在 BitFun 桌面端允許助理截取屏幕並控制鍵鼠；需多模態模型理解畫面。", "enable": "啟用 Computer use", - "enableDesc": "關閉時，任何會話模式都不會啟用 ComputerUse 工具。", + "enableDesc": "關閉時，任何會話模式都不會啟用 ComputerUse 工具；瀏覽器控制（ControlHub）目前不受此開關約束。", "accessibility": "輔助功能", "accessibilityDesc": "macOS 上用於讓 BitFun 讀取畫面上的介面元素並向其他應用傳送滑鼠/鍵盤操作；Windows 無需單獨授權，Linux 需要 X11 工作階段。", "screenCapture": "屏幕錄製", From 7edda57e67d769459c3f3dd1091e62ceaa5b3288 Mon Sep 17 00:00:00 2001 From: bowen628 Date: Sun, 26 Jul 2026 17:51:30 +0800 Subject: [PATCH 3/3] fix(web-ui): add missing useCallback deps for computer-use setter --- .../src/infrastructure/config/components/SessionConfig.tsx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx b/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx index a82b0041ce..077dd709a7 100644 --- a/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx +++ b/src/web-ui/src/infrastructure/config/components/SessionConfig.tsx @@ -176,7 +176,7 @@ const SessionSettingsPanels: React.FC = ({ variant } } finally { setComputerUseStatusLoading(false); } - }, []); + }, [setComputerUseEnabled]); const refreshBrowserControlStatus = useCallback(async () => { if (!IS_TAURI_DESKTOP) return; @@ -220,7 +220,7 @@ const SessionSettingsPanels: React.FC = ({ variant } void systemAPI.getSystemInfo() .then((info) => setPlatform(info.platform || '')) .catch((error) => log.warn('getSystemInfo failed', error)); - }, [refreshComputerUseStatus, refreshBrowserControlStatus]); + }, [refreshComputerUseStatus, refreshBrowserControlStatus, setComputerUseEnabled]); const loadAllData = useCallback(async () => { setIsLoading(true);