Feature/screenshot

AGENTS.md

@@ -0,0 +1,26 @@
+# Agent Notes
+
+## Beta Release Flow
+
+When asked to publish a beta version:
+
+1. Do not change `package.json` version on the current branch.
+2. Determine `BASE_VERSION` from the latest npm beta dist-tag, unless the user explicitly pins a different base version. Do not derive it from the current branch `package.json`.
+   - npm beta: `npm_config_cache=/private/tmp/web-cap-npm-cache npm view web-capability@beta version`
+   - Example: if npm beta is `0.0.7-beta.3`, then `BASE_VERSION=0.0.7` and the next beta is `0.0.7-beta.4`.
+3. Check existing beta versions from both npm and the private git remote, then choose the next beta number for that `BASE_VERSION`.
+   - npm: `npm_config_cache=/private/tmp/web-cap-npm-cache npm view web-capability versions --json`
+   - private tags: `git ls-remote --tags private "v${BASE_VERSION}-beta.*"`
+4. Commit any requested code changes on the current branch first, without changing the package version.
+5. Create a temporary release worktree from the current `HEAD`.
+   - Example: `git worktree add /private/tmp/web-cap-tag-${BASE_VERSION}-beta.N HEAD`
+6. In that temporary worktree only, update `package.json` to the beta version.
+7. Commit the release version change in the temporary worktree.
+   - Example: `chore: release ${BASE_VERSION}-beta.N`
+8. Create `v${BASE_VERSION}-beta.N` on that temporary release commit.
+9. Push only the tag to the `private` remote.
+   - Example: `git push private v${BASE_VERSION}-beta.N`
+10. Verify:
+   - `git show v${BASE_VERSION}-beta.N:package.json` reports the beta version.
+   - The current branch `package.json` still has its original version.
+   - The current branch worktree is clean.

README.md

@@ -41,7 +41,7 @@ Compared with action-first browser tools, Web Cap focuses on:
 
 - In-page execution, so scripts can work directly with the DOM and page state.
 - Reusable capabilities, so successful scripts can be searched, inspected, and called again.
-- Composable scripts, so one script can call another through `cap.call(...)`.
+- Deprecated: composable scripts, where one script calls another through `cap.call(...)`.
 - Optional post-execution observation, so script runs can return evidence about what changed on the page when evidence collection is enabled.
 - Local persistence, so agent-learned workflows can survive beyond a single run.
 - CLI access, so agents can use the same browser capabilities from normal command-line workflows.
@@ -135,7 +135,7 @@ A typical agent flow is:
 
 Execute script code in the selected browser tab. Scripts receive one object argument and return one JSON object.
 
-`script-execute` accepts optional execution settings such as `--timeout-ms`, `--script-file`, `--input-file`, and `--register`. During execution, scripts can call other scripts through `cap.call(scriptId, input)`. `--register` saves the inline script only after execution succeeds with `ok: true`.
+`script-execute` accepts optional execution settings such as `--timeout-ms`, `--script-file`, `--input-file`, and `--register`. `--register` saves the inline script only after execution succeeds with `ok: true`.
 
 ### Browser commands
 
@@ -147,21 +147,22 @@ Scripts are JavaScript functions with JSON-compatible inputs and outputs:
 
 ```js
 export default async function (input) {
-  const page = await cap.call('builtin.page.inspect', {});
+  const heading = await page.locator('h1').first().textContent().catch(() => '');
 
   return {
     ok: true,
-    title: page.title,
+    title: document.title,
+    heading,
     selector: input.selector,
   };
 }
 ```
 
-The runtime injects `cap` while the script executes.
+The runtime injects `page` and `cap.page` while the script executes.
 
 Available runtime helpers:
 
-- `cap.call(scriptId, input)` - call a built-in or registered script.
+- Deprecated: `cap.call(scriptId, input)` - call a built-in or registered script.
 - `cap.get(scriptId)` - read one script schema summary.
 - `cap.list()` - list callable script schema summaries.
 

README.zh-CN.md

@@ -41,7 +41,7 @@ Agent 可以在页面内运行 JavaScript，组合内置能力，并把有用脚
 
 - 页面内执行，脚本可以直接访问 DOM 和页面状态。
 - 能力复用，成功脚本可以被搜索、查看并再次调用。
-- 脚本组合，一个脚本可以通过 `cap.call(...)` 调用另一个脚本。
+- 已废弃：脚本组合，即一个脚本通过 `cap.call(...)` 调用另一个脚本。
 - 可选的执行后观察，在启用证据采集时脚本运行可以返回页面变化证据。
 - 本地持久化，让 agent 学到的工作流不只存在于单次运行中。
 - 命令行访问，让 agent 可以在普通 CLI 工作流中使用同一套浏览器能力。
@@ -135,7 +135,7 @@ pnpm cli session-status
 
 在选定的浏览器标签页中执行脚本。脚本接收一个对象参数，并返回一个 JSON 对象。
 
-`script-execute` 支持 `--timeout-ms`、`--script-file`、`--input-file`、`--register` 等可选执行配置。脚本执行期间，可以通过 `cap.call(scriptId, input)` 调用其他脚本。`--register` 只会在执行成功且结果包含 `ok: true` 时保存内联脚本。
+`script-execute` 支持 `--timeout-ms`、`--script-file`、`--input-file`、`--register` 等可选执行配置。`--register` 只会在执行成功且结果包含 `ok: true` 时保存内联脚本。
 
 ### 浏览器命令
 
@@ -147,21 +147,22 @@ Web Cap 还包括 `browser-new-tab`、`session-status`、`wait-events` 等命令
 
 ```js
 export default async function (input) {
-  const page = await cap.call('builtin.page.inspect', {});
+  const heading = await page.locator('h1').first().textContent().catch(() => '');
 
   return {
     ok: true,
-    title: page.title,
+    title: document.title,
+    heading,
     selector: input.selector,
   };
 }
 ```
 
-运行时会在脚本执行期间注入 `cap`。
+运行时会在脚本执行期间注入 `page` 和 `cap.page`。
 
 可用 runtime helper：
 
-- `cap.call(scriptId, input)` - 调用内置脚本或已注册脚本。
+- 已废弃：`cap.call(scriptId, input)` - 调用内置脚本或已注册脚本。
 - `cap.get(scriptId)` - 读取某个脚本的 schema 摘要。
 - `cap.list()` - 列出当前可调用脚本的 schema 摘要。
 

extension/entrypoints/background.ts

@@ -8,6 +8,7 @@ import {
   type ExecutionEvidenceEvent,
   type ExecutionEvidenceOption,
   type RuntimeEnvelope,
+  type RuntimeScreenshotArtifactPayload,
   type ScriptExecutionHistoryEntry,
   type RuntimeTabSnapshot,
 } from '@shared/protocol';
@@ -20,6 +21,7 @@ import {
 import {
   isDebuggerFallbackEligibleError,
   isExecutionInterruptedByNavigationError,
+  type ScriptScreenshotArtifact,
   type ScriptExecutionResponse,
 } from '../runtime/execution-helpers';
 import { BrowserCommandHandler } from '../runtime/browser-command-handler';
@@ -95,7 +97,7 @@ class RuntimeClient {
     },
     sendTabSnapshot: () => this.sendTabSnapshot(),
     toTabSnapshot: (tab) => this.toTabSnapshot(tab),
-  });
+  }, this.debuggerExecutor.getDebuggerClient());
 
   start(): void {
     this.connect();
@@ -174,6 +176,7 @@ class RuntimeClient {
           envelope.payload.tabId,
           envelope.payload.activateTab,
           envelope.payload.evidence ?? [],
+          envelope.payload.screenshotArtifactBasePath,
         );
         break;
       case 'browser_command':
@@ -203,8 +206,20 @@ class RuntimeClient {
     tabId?: number,
     activateTab?: boolean,
     evidenceOptions: ExecutionEvidenceOption[] = ['common'],
+    screenshotArtifactBasePath?: string,
   ): Promise<void> {
-    const selectedTab = tabId ? await browser.tabs.get(tabId) : await this.getActiveTab();
+    let selectedTab: BrowserTabLike | undefined;
+    try {
+      selectedTab = tabId ? await browser.tabs.get(tabId) : await this.getActiveTab();
+    } catch (error) {
+      this.sendError(
+        requestId,
+        'EXECUTION_FAILED',
+        error instanceof Error ? error.message : String(error),
+        { scriptId: scriptDefinition.id, tabId },
+      );
+      return;
+    }
     if (!selectedTab?.id || !selectedTab.url) {
       this.sendError(requestId, 'TAB_NOT_FOUND', 'No active browser tab is available.', {
         scriptId: scriptDefinition.id,
@@ -249,6 +264,7 @@ class RuntimeClient {
           input,
           scriptRegistry,
           evidenceOptions,
+          screenshotArtifactBasePath,
         );
       } catch (error) {
         if (!isExecutionInterruptedByNavigationError(error)) {
@@ -317,6 +333,10 @@ class RuntimeClient {
             result,
             evidence,
             status: response.status ?? 'succeeded',
+            screenshotArtifacts: this.sendBinaryScreenshotArtifacts(
+              response.screenshotArtifacts ?? [],
+              requestId,
+            ),
           },
           { sessionId: this.sessionId, requestId },
         ),
@@ -359,6 +379,7 @@ class RuntimeClient {
     input: Record<string, unknown>,
     scriptRegistry: ScriptDefinition[],
     evidence: ExecutionEvidenceOption[],
+    screenshotArtifactBasePath?: string,
   ): Promise<ScriptExecutionResponse> {
     const requiresBrowserLevelClick = scriptRequiresBrowserLevelClick(
       scriptDefinition,
@@ -386,6 +407,7 @@ class RuntimeClient {
         input,
         scriptRegistry,
         evidence,
+        screenshotArtifactBasePath,
       );
     }
 
@@ -400,6 +422,7 @@ class RuntimeClient {
         input,
         scriptRegistry,
         evidence,
+        screenshotArtifactBasePath,
       );
     } catch (error) {
       if (
@@ -415,6 +438,7 @@ class RuntimeClient {
         input,
         scriptRegistry,
         evidence,
+        screenshotArtifactBasePath,
       );
     }
   }
@@ -425,7 +449,18 @@ class RuntimeClient {
     input: Record<string, unknown>,
     tabId?: number,
   ): Promise<void> {
-    const activeTab = tabId ? await browser.tabs.get(tabId) : await this.getActiveTab();
+    let activeTab: BrowserTabLike | undefined;
+    try {
+      activeTab = tabId ? await browser.tabs.get(tabId) : await this.getActiveTab();
+    } catch (error) {
+      this.sendError(
+        requestId,
+        'EXECUTION_FAILED',
+        error instanceof Error ? error.message : String(error),
+        { command, tabId },
+      );
+      return;
+    }
     if (!activeTab?.id || !activeTab.url) {
       this.sendError(requestId, 'TAB_NOT_FOUND', 'No active browser tab is available.', {
         command,
@@ -458,7 +493,11 @@ class RuntimeClient {
         createRuntimeEnvelope(
           'browser_command_result',
           {
-            result: response.result,
+            result: this.extractBinaryScreenshotArtifacts(
+              response.result,
+              requestId,
+              'metadata',
+            ) as Record<string, unknown>,
           },
           { sessionId: this.sessionId, requestId },
         ),
@@ -886,11 +925,137 @@ class RuntimeClient {
     );
   }
 
+  private extractBinaryScreenshotArtifacts(
+    value: unknown,
+    requestId: string,
+    resultShape: 'path' | 'metadata',
+  ): unknown {
+    if (isScreenshotArtifact(value)) {
+      const transferId = crypto.randomUUID();
+      const bytes = decodeBase64(value.data);
+      const type = value.type === 'jpeg' ? 'jpeg' : 'png';
+      const mimeType = typeof value.mimeType === 'string'
+        ? value.mimeType
+        : type === 'jpeg'
+          ? 'image/jpeg'
+          : 'image/png';
+
+      this.send(
+        createRuntimeEnvelope(
+          'binary_payload_start',
+          {
+            transferId,
+            kind: 'screenshot',
+            mimeType,
+            type,
+            byteLength: bytes.byteLength,
+            resultShape,
+          },
+          { sessionId: this.sessionId, requestId },
+        ),
+      );
+      this.sendBinary(bytes);
+
+      return {
+        __webCapType: 'screenshot_transfer',
+        transferId,
+        resultShape,
+      };
+    }
+
+    if (Array.isArray(value)) {
+      return value.map((item) => this.extractBinaryScreenshotArtifacts(item, requestId, resultShape));
+    }
+
+    if (isRecord(value)) {
+      return Object.fromEntries(
+        Object.entries(value).map(([key, item]) => [
+          key,
+          this.extractBinaryScreenshotArtifacts(item, requestId, resultShape),
+        ]),
+      );
+    }
+
+    return value;
+  }
+
+  private sendBinaryScreenshotArtifacts(
+    artifacts: ScriptScreenshotArtifact[],
+    requestId: string,
+  ): RuntimeScreenshotArtifactPayload[] {
+    return artifacts.map((artifact) => {
+      const transferId = crypto.randomUUID();
+      const bytes = decodeBase64(artifact.data);
+      const type = artifact.type === 'jpeg' ? 'jpeg' : 'png';
+      const mimeType = typeof artifact.mimeType === 'string'
+        ? artifact.mimeType
+        : type === 'jpeg'
+          ? 'image/jpeg'
+          : 'image/png';
+
+      this.send(
+        createRuntimeEnvelope(
+          'binary_payload_start',
+          {
+            transferId,
+            kind: 'screenshot',
+            mimeType,
+            type,
+            byteLength: bytes.byteLength,
+            resultShape: 'metadata',
+            path: artifact.path,
+          },
+          { sessionId: this.sessionId, requestId },
+        ),
+      );
+      this.sendBinary(bytes);
+
+      return {
+        kind: 'screenshot',
+        path: artifact.path,
+        transferId,
+        mimeType,
+        type,
+      };
+    });
+  }
+
   private send(envelope: RuntimeEnvelope): void {
     if (this.socket?.readyState === WebSocket.OPEN) {
       this.socket.send(JSON.stringify(envelope));
     }
   }
+
+  private sendBinary(bytes: Uint8Array): void {
+    if (this.socket?.readyState === WebSocket.OPEN) {
+      this.socket.send(bytes);
+    }
+  }
+}
+
+function isScreenshotArtifact(value: unknown): value is {
+  data: string;
+  mimeType?: string;
+  type?: string;
+} {
+  return (
+    isRecord(value) &&
+    value.__webCapType === 'screenshot' &&
+    typeof value.data === 'string'
+  );
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+  return typeof value === 'object' && value !== null && !Array.isArray(value);
+}
+
+function decodeBase64(value: string): Uint8Array {
+  const binary = atob(value);
+  const bytes = new Uint8Array(binary.length);
+  for (let index = 0; index < binary.length; index += 1) {
+    bytes[index] = binary.charCodeAt(index);
+  }
+  return bytes;
 }
 
 function applyExecutionTabIndicatorScript(titlePrefix: string): void {