feat(chrome): image budget controls for screenshot quality + per-turn capture (#311)#349
Open
Ayush7614 wants to merge 8 commits into
Open
feat(chrome): image budget controls for screenshot quality + per-turn capture (#311)#349Ayush7614 wants to merge 8 commits into
Ayush7614 wants to merge 8 commits into
Conversation
… capture (issue webbrain-one#311) Add an 'Image budget' section under Settings → Multimodal with three persisted controls applied to screenshot capture and vision request formatting: - imageDetail (high/low/auto): sent as the OpenAI-style image_url 'detail' field; 'auto' (default) is a no-op so providers keep their default. - maxScreenshotsPerTurn (0-5, 0=unlimited): caps auto-screenshots per turn via a per-turn counter reset at the start of each turn. - maxImageDimension (512-2048px): caps every vision image's largest side and the token budget derived from it. Defaults preserve pre-webbrain-one#311 behavior: auto detail, unlimited screenshots, and a 1568px cap that maps to the legacy IMAGE_BUDGET byte-for-byte (near-square viewports still downscale to ~1109px). Budget is applied to all vision capture paths (auto-screenshot, /screenshot, full_page_screenshot, verify_form, crop, user attachments). UI shows a cost/latency explanation.
|
@Ayush7614 is attempting to deploy a commit to the esokullu's projects Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Author
|
cc: @esokullu |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Chrome image-budget controls for vision captures and requests.
Changes:
- Adds persisted image detail, screenshot-count, and dimension controls.
- Applies image detail and resizing across several vision paths.
- Adds settings UI and locale fallbacks.
Reviewed changes
Copilot reviewed 19 out of 20 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/chrome/src/agent/agent.js |
Implements image budgets and capture limits. |
src/chrome/src/background.js |
Hydrates persisted budget settings. |
src/chrome/src/ui/settings.js |
Loads and saves the controls. |
src/chrome/src/ui/settings.html |
Adds the Image budget settings card. |
src/chrome/src/ui/locales/en.js |
Adds English labels and guidance. |
src/chrome/src/ui/locales/ar.js |
Adds Arabic-locale fallbacks. |
src/chrome/src/ui/locales/es.js |
Adds Spanish-locale fallbacks. |
src/chrome/src/ui/locales/fr.js |
Adds French-locale fallbacks. |
src/chrome/src/ui/locales/he.js |
Adds Hebrew-locale fallbacks. |
src/chrome/src/ui/locales/id.js |
Adds Indonesian-locale fallbacks. |
src/chrome/src/ui/locales/ja.js |
Adds Japanese-locale fallbacks. |
src/chrome/src/ui/locales/ko.js |
Adds Korean-locale fallbacks. |
src/chrome/src/ui/locales/ms.js |
Adds Malay-locale fallbacks. |
src/chrome/src/ui/locales/pl.js |
Adds Polish-locale fallbacks. |
src/chrome/src/ui/locales/ru.js |
Adds Russian-locale fallbacks. |
src/chrome/src/ui/locales/th.js |
Adds Thai-locale fallbacks. |
src/chrome/src/ui/locales/tl.js |
Adds Tagalog-locale fallbacks. |
src/chrome/src/ui/locales/tr.js |
Adds Turkish-locale fallbacks. |
src/chrome/src/ui/locales/uk.js |
Adds Ukrainian-locale fallbacks. |
src/chrome/src/ui/locales/zh.js |
Adds Chinese-locale fallbacks. |
Files not reviewed (1)
- src/chrome/src/ui/locales/th.js: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+2608
to
+2613
| const _screenshotCap = Number(this.maxScreenshotsPerTurn) || 0; | ||
| const _overBudget = _screenshotCap > 0 && (this.autoScreenshotCount.get(tabId) || 0) >= _screenshotCap; | ||
| await new Promise(r => setTimeout(r, 250)); | ||
| const shot = await this._captureAutoScreenshot(tabId); | ||
| const shot = _overBudget ? null : await this._captureAutoScreenshot(tabId); | ||
| if (shot) { | ||
| if (_screenshotCap > 0) this.autoScreenshotCount.set(tabId, (this.autoScreenshotCount.get(tabId) || 0) + 1); |
| content: [ | ||
| { type: 'text', text: prompt }, | ||
| { type: 'image_url', image_url: { url: screenshot.dataUrl } }, | ||
| { type: 'image_url', image_url: this._withImageDetail({ url: screenshot.dataUrl }) }, |
| const rawUrl = `data:image/png;base64,${screenshot.data}`; | ||
| return { | ||
| dataUrl: await this._compressJpegToByteCeiling(rawUrl), | ||
| dataUrl: await this._compressJpegToByteCeiling(rawUrl, this._budgetForCapture()), |
| }; | ||
| } | ||
| blocks.push({ type: 'image_url', image_url: { url: att.dataUrl } }); | ||
| blocks.push({ type: 'image_url', image_url: this._withImageDetail({ url: att.dataUrl }) }); |
| if (stored.maxScreenshotsPerTurn != null) agent.maxScreenshotsPerTurn = stored.maxScreenshotsPerTurn; | ||
| if (stored.maxImageDimension != null) agent.maxImageDimension = stored.maxImageDimension; | ||
| } | ||
| loadImageBudget(); |
Collaborator
|
@Ayush7614 thank you. Pls confirm these are all addressed |
Resolve the settings.js conflict with main, then close the Copilot review items on webbrain-one#349: - Count initial viewport auto-screenshots toward maxScreenshotsPerTurn via a shared check/capture/increment helper - Apply maxImageDimension to localization, coord-aligned captures (with CSS scale transforms), and user image attachments - Await imageBudgetReady in handleMessage so cold SW starts hydrate budget settings before the first turn Tests: 759 passed.
Collaborator
Progress update — review + conflicts addressedPushed Merge conflicts
Copilot review items
Verification
@Ayush7614 / @esokullu — ready for re-review / merge. |
Collaborator
Code review:
|
Mirror Chrome image-budget (webbrain-one#311) to Firefox (agent helpers, settings, storage hydration, locales), skip re-shrink when vision dims are already set, and add unit tests for budget mapping, detail, and per-turn counters.
Collaborator
Follow-up: review items (1), (4), (5)Addressed on commit Done
Still open from the original review
Note on PR headThis PR’s head is on fork
|
…ip signal Coord-aligned captures honor maxImageDimension per side without the vision token budget so common viewports stay 1:1 when under the cap. Exhausted per-turn auto-screenshot limits emit a warning, trusted note, and trace; settings copy no longer promises state-change captures after the budget.
…ss budget) Align settings const indentation, mark 1568px as the selected default, and pass the active capture budget into non-coord JPEG byte-ceiling compress.
Normalize imageDetail / maxScreenshotsPerTurn / maxImageDimension on load and settings change so corrupt storage cannot reach providers. Viewport screenshot save:true keeps a pre-budget CSS-resolution data URL for Downloads (matching full_page_screenshot) while the model still gets the budgeted, optionally redacted copy.
Replace English stubs for st.imageBudget.* with real translations in Chrome and Firefox locale files (review nit webbrain-one#10).
Collaborator
Image budget review — status of all findingsFollow-up commits through Bugs
Suggestions
Nits
Tests
Notable commits (this review follow-up)
Manual smoke ideas
|
Collaborator
Pre-merge notes: vision risk / defaults vs mainReference notes before deciding to merge. Defaults were designed to stay close to pre-#311 behavior; residual risks are narrower than “vision is off,” but worth knowing. What was already true on
|
| Path | Behavior on main |
|---|---|
| Normal auto / non-coord capture | Already fit to static IMAGE_BUDGET (~1568px side + ~1568 vision tokens + JPEG byte ceiling) |
| Coord-aligned capture | Full CSS size — byte-ceiling compress only, no dimension/token resize |
| User image attachments | Sent as-is (no budget shrink) |
| Auto-shots per turn | Unlimited |
OpenAI-style detail |
Not sent |
Everyday vision was already “budget so providers don’t choke,” not “always native resolution.”
Defaults on this PR
- Image detail =
auto→ still nodetailfield (same as main) - Max screenshots per turn = unlimited (
0) → same as main - Max image dimension = 1568 → same numeric side cap as old
IMAGE_BUDGET.maxTargetPx
At default, _budgetForCapture() returns the legacy budget byte-for-byte when the cap is 1568.
Low risk at defaults (most agent vision)
- Normal auto-screenshots / non-coord
/screenshot: same budget math as main at 1568. - Unlimited auto shots: agent will not go blind from a counter.
detail: auto: no change to provider payloads for detail.
Residual default risks (honest)
-
Coord-aligned / 1:1 pixel-click path
- Main: never resized for dimension.
- This PR: resizes only if a side exceeds
maxImageDimension(1568). Token budget does not force shrink (review issue 2 fix). - 1440×900, 1366×768, etc.: stay 1:1.
- 1920×1080 and wider: can still downscale (1920 > 1568) → possible coord click misses on large monitors even at defaults.
- Mitigations: raise max dimension to 1920/2048, or prefer selectors / AX over pure image-pixel clicks.
-
User-attached images
Now resized for the model under the budget; original kept for UI/history. Slight detail loss possible on huge attachments; unlikely to break operation. -
Visible-media localization
Model-facing crop is budgeted; raw crop for local download stays full. Double-shrink removed (quality improvement vs early PR). -
Firefox auto path
Side-only budget on CSSscale:1path — mid-size viewports may keep more pixels than a full token-budget shrink (usually better for vision).
When settings are changed (by design)
| Change | Effect | Risk |
|---|---|---|
| Max screenshots = 1–5 | Stops auto-shots after N; warning + trusted note + trace | Real semi-blind mid-turn after limit |
| Max dimension → 512/768 | Much smaller vision images | Quality risk on dense UI / small text |
| Max dimension → 2048 | Bigger images | More fidelity + cost; possible provider size/latency pressure |
| Detail = high / low | OpenAI-style endpoints may honor; others ignore | Cost/quality tradeoff where supported |
Verdict
| Scenario | Assessment |
|---|---|
| Ship with defaults, normal Act/Ask + auto-screenshots | Unlikely to break core vision loop vs main’s existing budget |
| Users set low max shots / low dimension | Can degrade vision usefulness — intentional tradeoff |
| Heavy coord-aligned pixel clicks at 1080p+ | Possible regression vs main at default 1568 side cap |
Bottom line: defaults ≈ previous sensible budgeting for the common path. Main surprise surface is coord-aligned full-CSS on wide screens, not “vision turned off.” Sharp settings (max shots, low dimension) can hurt vision deliberately.
Suggested manual smoke before merge
- Dense UI page at defaults — model still “sees” labels after auto-shot.
- 1920×1080 window — if using coord clicks from screenshots, check landing accuracy.
- Attach a large PNG once — model still uses it.
- Max shots = 1, multi-step act — skip warning + trusted note appear; agent doesn’t thrash silently.
Related review follow-ups (already landed on this branch)
Issues 1–11 from the earlier review were addressed in subsequent commits (Firefox parity, coord side-only budget, skip signals, tests, storage validation, full-res save, i18n, nits). See earlier review comments on this PR for the checklist.
Contributor
Author
|
@esokullu Confirmed — all Copilot review items are addressed in the follow-up commits (shared auto-screenshot counter, localization/coord-aligned/attachment budget paths, cold-start hydration, Firefox parity, tests, storage validation, and related fixes). Thanks for the thorough review and follow-up work. |
Collaborator
|
thanks @Ayush7614 -- this is a super sensitive part of the code, it's the eyes of the agent. that's why i'm taking time to fully understand it and making sure it won't break things before deciding on a merger or modifications. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements issue #311 — "Image budget" controls for screenshot quality and capture budget.
Adds a new Image budget section under Settings → Multimodal with three persisted controls, applied to both screenshot capture and vision request formatting (Chrome + Firefox):
high/low/auto): sent as the OpenAI-styleimage_url.detailfield.auto(default) is a no-op so providers keep their own default.0–5,0= unlimited): caps auto-screenshots per turn via a per-turn counter that resets at the start of each turn/run.512–2048px): caps the largest side of every vision image and the derived token budget.Acceptance criteria
chrome.storage.local/browser.storage.local, hydrated inbackground.js, and applied inagent.jsacross all vision capture paths (auto-screenshot,/screenshot,full_page_screenshot,verify_form, crop/localization, user attachments).autodetail, unlimited screenshots,1568px cap. The default cap maps to the legacyIMAGE_BUDGETbyte-for-byte, so near-square viewports still downscale to ~1109px exactly as before.detail, caps shrink images before sending, manually saved full-resolution images are unaffected).Verification
node --checkon edited JS files: pass.npm test(node test/run.js): 789 passed, 0 failed.Notes / scope