[3/5] Port the signed Swift helper background-click route

[cm2435-hcomp]

Why

The first Rust point-click implementation was based on an earlier Chromium experiment. It refused native Calculator before dispatch and diverged from the currently installed signed Codex Computer Use helper.

We inspected and disassembled the signed macOS helper shipped inside ChatGPT. The binary exposes concrete components for per-window capture, AX targeting, synthetic application focus, focus-steal prevention, a virtual cursor, and per-PID event delivery. Its current generic pointer route is not gated on a guessed AppKit/Chromium classification.

Recovered Swift control flow

The relevant helper path establishes this sequence:

1. Resolve an exact application, window, AX target, PID, and target window ID.
2. Maintain controller-owned synthetic focus belief and prevent the target from stealing real foreground focus.
3. Construct mouse events through AppKit NSEvent.mouseEvent(...).
4. Stamp the target window and use fresh event timestamps.
5. Post one down/up pair once to the target PID with CGEventPostToPid.
6. Capture and settle the target window without moving the hardware cursor.

The helper also contains explicit local exceptions, including Catalyst preparation. It does not use a broad framework classification as the generic route selector.

What changed

• Replaces the Chromium-specific primer and dual-post pointer path with one canonical AppKit-derived down/up pair.
• Ports the CPS process-notification focus mechanism and durable three-state synthetic-focus coordinator.
• Adds exact focus acknowledgement, focus-steal containment, event-tap teardown barriers, and synthesized-action markers.
• Selects generic point delivery from exact PID, process generation, CGWindowID, AX-window, geometry, current-Space, and capture facts.
• Keeps Catalyst and menu-required routes refused until their explicit helper recipes are ported.
• Tightens lifecycle identity, cold-launch registry publication, ScreenCaptureKit freshness, and observation bracketing discovered during live QA.
• Allows semantic text insertion for Unknown framework windows when exact AX facts prove the route.

Live-host result

Calculator passed five consecutive background repetitions:

• Chrome remained foreground.
• The driver clicked AC, 1, 2, +, 3, 0, and = using captured-surface point targets.
• Calculator displayed 42, confirmed through an independent AX witness.
• Every point used route targeted_pointer and one AppKit-derived CGEventPostToPid down/up pair.
• The hardware cursor did not move and no input leaked to the foreground app.

TextEdit also accepted exact semantic Unicode insertion with AX readback. Later TextEdit actions exposed the concurrent-user posture issue listed below.

Validation

• cargo test -p cua-driver-core: 148 tests passed, 0 failed.
• cargo test -p platform-macos: 197 tests passed, 0 failed, 1 doc test ignored.
• git diff --check: passed.
• Calculator targeted-pointer workflow: 5/5 complete live-host repetitions.

Repository-wide formatting remains red on unrelated 0.7.1 files and is not rewritten here.

Outstanding work

• Port the helper’s generic keyboard route: effective focused-element PID resolution, focus preparation, synthesized key events, and direct CGEventPostToPid. press_key currently refuses before dispatch.
• Separate target containment from ambient user activity. The current posture witness treats any unrelated foreground, key-window, or hardware-cursor change as failure, even when the target effect was verified.
• Keep framework labels diagnostic. Use positive framework evidence only for local exceptions such as Catalyst or Chromium AX preparation.
• Complete live workflows for Spotify semantic scroll/navigation, Chromium edge pointer/scroll/menu behavior, and Finder lifecycle/secondary actions.
• Forward-port the reviewed stack from CUA 0.7.1 to current upstream 0.12.2.

Stack

Depends on #2. This is the final native checkpoint PR before the Python client and QA/demo handoff PRs in hcompai/cua-drivers.

[github-actions]

Linux visual regression artifacts

Matrix jobs now run independently. Download visual artifacts from this workflow run.
Each background-GUI job uploads a .gif of the interaction plus two annotated PNGs (<app>.png raw, <app>-atspi.png with AT-SPI element boxes); the cua-driver-linux-som-overlays artifact adds <app>-som.png cua Set-of-Marks overlays:

• cua-driver-linux-cursor-click-gif
• cua-driver-linux-background-terminal-gif
• cua-driver-linux-parallel-drag-xserver
• cua-driver-linux-background-gui-chromium
• cua-driver-linux-background-gui-tk
• cua-driver-linux-background-gui-gtk3-gedit
• cua-driver-linux-background-gui-gtk3-mousepad
• cua-driver-linux-background-gui-gtk3-scite
• cua-driver-linux-background-gui-gtk4-characters
• cua-driver-linux-background-gui-qt5-manuskript
• cua-driver-linux-background-gui-qt5-klog
• cua-driver-linux-background-gui-qt5-openambit
• cua-driver-linux-background-gui-qt6-kate
• cua-driver-linux-background-gui-qt6-kcalc
• cua-driver-linux-background-gui-qt6-okular
• cua-driver-linux-background-gui-qt6-qownnotes
• cua-driver-linux-background-gui-electron-zettlr
• cua-driver-linux-background-gui-electron-joplin
• cua-driver-linux-background-gui-electron-logseq
• cua-driver-linux-som-overlays

Open workflow run and download artifacts