Windows port of the computer-use helper#1
Open
silfralabs wants to merge 5 commits into
Open
Conversation
…ort) Relax the darwin-only guards in computer-use-helper.ts (ensure/available/ relaunchForGrant) and add win32 install()/launch() that spawn the native blitz-cu-helper.exe with the socket path (no LaunchServices, no TCC). Add bundledHelperExe()/installedHelperExe() and a winProc handle killed on shutdown. Transport fix: node:net IPC is a NAMED PIPE on Windows, not AF_UNIX (WINDOWS-INTEGRATION.md section 4 was wrong; verified: a filesystem path gives listen EACCES). sockPath is now \\.\pipe\blitzcu-<pid> on win32; the helper picks pipe-vs-AF_UNIX from the path shape. JSON framing stays byte-identical. index.ts: hoist droppedBrowser() once and force it undefined off-darwin so every Windows drop is a window-connect (HWND to connectionConnectWindow), never the Apple-Events tab path. connection-window-link.ts: send windowId alongside pid on ax_read/ax_act/activate so Windows resolves the exact dropped window via UIA FromHandle (pid to MainWindowHandle is wrong for UWP/WinUI apps). macOS reads pid and ignores it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…uild scripts) Add the C# helper project in-repo at native/computer-use-helper/win/ (Program.cs, Picker.cs, app.manifest, blitz-cu-helper.csproj), the Windows analog of main.swift. build-win.ps1 publishes it to a single self-contained exe at native/computer-use-helper/build/blitz-cu-helper.exe (the .app analog). dist-win.ps1 orchestrates helper build, electron-vite build, then electron-builder --win, mirroring dist-mac.sh; wired as npm run dist:win. electron-builder.yml: win nsis target (x64) + icon + extraResources shipping the exe to resourcesPath; perMachine install into Program Files (the trusted path the future uiAccess="true" + Authenticode signing needs to drive elevated windows). gitignore the helper build outputs and the .NET intermediates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Top-level extraResources (arm64 tmux, the two macOS helper .app bundles, notch-geometry, activity-logging) were inherited by every target, so a Windows pack would skip the missing macOS build outputs with warnings and wrongly ship the arm64 tmux. Move all macOS runtime resources under mac.extraResources; keep only the win exe + activity-logging under win.extraResources. No top-level extraResources remain, so the win target inherits none of mac's. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
electron-builder renames win-unpacked.tmp -> win-unpacked during Electron extraction; on a OneDrive/Dropbox-synced checkout the sync client locks the fresh files and the rename fails with EPERM. Document pointing directories.output outside the synced tree (verified: the full nsis pack succeeds that way). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The whole UI is the macOS notch overlay: it opens via notch hover or the ⌥Space chord, both unavailable on Windows (no notch; Alt+Space is the Win32 window menu). Main's notchGated is false off-darwin, so the os:notch-geometry push that sets notchOn + carries `synthetic` never fires, and NotchHost (gated on notchOn) never mounts. Result: a blank window with no entry point. Fix: on first mount on a non-macOS platform, pin the island OPEN exactly like the VM/synthetic path (notchOn=true, hasNotch=false, pinned panel, interactive). The chat + "+" attach panel (computer-use / window picker) are then always visible in the normal window. macOS is untouched (gated on platform). Verified via CDP on a real Windows run: NotchHost mounts, the Blitz chat + attach button render. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
Does this work? Are you using it right now? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the macOS computer-use helper (
native/computer-use-helper/main.swift) to a C#/.NET 8 Windows helper that BlitzOS drives over the same newline-JSON IPC: AX read/act -> UI Automation (FlaUI), CGEvent -> SendInput, CGWindowList -> EnumWindows, ScreenCaptureKit -> PrintWindow, plus the drag-to-attach window picker (a WH_MOUSE_LL hook + layered click-through overlays). The TS side gets theWINDOWS-INTEGRATION.mdedits plus one transport correction.Transport correction (important)
WINDOWS-INTEGRATION.mdsection 4 assumed Node'snetserver speaks AF_UNIX on Windows. It does not:node:netIPC is a named pipe on Windows (a plain filesystem path giveslisten EACCES; confirmed against the Node v22 docs). So the helper now selects AF_UNIX vs named pipe from the socket-path shape (\\.\pipe\...-> pipe), and BlitzOS setssockPath = \\.\pipe\blitzcu-<pid>on win32. The newline-JSON framing on top is byte-identical, so this is the only change beyond the documented diffs.Changes
TS integration
computer-use-helper.ts: relax the darwin-only guards (ensure/available/relaunchForGrant); win32install()(no-op) +launch()(spawn the exe, no LaunchServices, no TCC); addbundledHelperExe()/installedHelperExe(); pipesockPath;winProchandle killed on shutdown.index.ts: hoistdroppedBrowser()once and force itundefinedoff-darwin so every Windows drop is a window-connect (HWND ->connectionConnectWindow), never the Apple-Events tab path.connection-window-link.ts: sendwindowIdalongsidepidonax_read/ax_act/activateso Windows resolves the exact dropped window via UIAFromHandle(pid -> MainWindowHandleis wrong for UWP/WinUI apps whose window is owned byApplicationFrameHost.exe). macOS readspidand ignores the extra field.Packaging
native/computer-use-helper/win/(the analog ofmain.swift).build-win.ps1builds a single self-contained exe atnative/computer-use-helper/build/blitz-cu-helper.exe;dist-win.ps1mirrorsdist-mac.sh(helper build -> electron-vite build ->electron-builder --win); wired asnpm run dist:win.electron-builder.yml: winnsistarget (x64) + icon +extraResourcesshipping the exe toresourcesPath;perMachineinstall into Program Files.Verified
hellofirst, thenping/list_windows/ax_tree/cg_click/activate;ax_actdrives a real button (charmap "Select"); a fullpick_start-> drag ->pick_dropemits the exact HWND.tsc --noEmitclean.Not done / follow-ups
uiAccess="true"+ Program Files install to drive ELEVATED windows (helper staysasInvokerfor now).window_screenshot-> Windows.Graphics.Capture for occluded/GPU windows.electron-builderpack was not run in this environment.🤖 Generated with Claude Code