diff --git a/HANDOFF.md b/HANDOFF.md new file mode 100644 index 0000000..02808fb --- /dev/null +++ b/HANDOFF.md @@ -0,0 +1,303 @@ +# OpenParsec — Trackpad / Keyboard / Input Overhaul — Engineering Handoff + +> Hand-off document for a reviewing AI agent. Self-contained. Covers every change on +> branch `fix/trackpad-input`, the rationale, the current state, user-reported issues, +> outstanding bugs, and host-side limitations that cannot be fixed client-side. +> Last updated at commit `fc6776c`. + +--- + +## 0. TL;DR for the reviewing agent + +- **Repo:** `2extndd/OpenParsec` (fork of `hugeBlack/OpenParsec`). iPad app, Swift/SwiftUI + UIKit, Parsec SDK (binary `ParsecSDK.framework`, vendored, no headers in tree). Deployment target **iOS 14.0**. +- **Branch:** `fix/trackpad-input`, 17 commits ahead of `cd155f8` (fork's `main`). HEAD = `fc6776c`. +- **What it does:** fixes Magic Keyboard trackpad input (cursor lag + choppy scroll), adds Mac↔iPad keyboard-layout sync, local cursor overlay, mouse acceleration, Windows-host key remap, in-session resolution change, display persistence, a Low-Latency mode, and a crash reporter. +- **Build/CI:** GitHub Actions (`.github/workflows/build.yml`) on `macos-latest`, Xcode 26.2, produces `OpenParsec.ipa`. CI is currently green at HEAD. +- **The user tests on a real iPad M4 + Magic Keyboard, streaming to an M3 MacBook Pro (and sometimes a Windows host).** No Mac/Xcode locally → all builds go through CI; .ipa is sideloaded via Scarlet/eSign (unsigned variant) or AltStore (ad-hoc). +- **Your job (suggested):** verify correctness of the input pipeline, concurrency safety, and the host-protocol assumptions. Highest-risk areas are flagged in §6. + +--- + +## 1. Project & repo context + +OpenParsec is an open-source Parsec client for iPad. Parsec streams a host desktop (Mac/Windows/Linux) to the client over a low-latency UDP protocol (BUD). The client sends input (mouse/keyboard/gamepad) via `ParsecClientSendMessage` and receives video frames + a separately-streamed cursor image + user-data events. + +Key architectural facts the reviewer must hold: + +- **`CParsec`** (`CParsec.swift`) is a static facade over **`ParsecSDKBridge`** (`ParsecSDKBridge.swift`), a singleton (`CParsec.parsecImpl`) that lives for the whole process. State on the bridge (`backgroundTaskRunning`, `didSetResolution`, `didRestoreSavedDisplay`, `mouseInfo`) persists across connect/disconnect cycles. +- **`ParsecViewController`** (`ParsecViewController.swift`, ~1500 lines now) owns the GLKView render surface, all gesture recognizers, the keyboard pipeline, the local cursor overlay, and language sync. It is **persisted across SwiftUI updates** by `ParsecSession` (an `ObservableObject` holding the VC) — see the comment in `ParsecView.swift`. Do not assume it is recreated per-connection. +- **`ParsecView`** (`ParsecView.swift`) is the SwiftUI layer: the in-stream overlay menu (Resolution / Bitrate / Display / Mute / Keyboard / Zoom), the status bar poller (`ParsecStatusBar`), and the connection lifecycle. +- **Input only flows on the main thread** for touch/gesture paths. The **GCMouse / GCController** path (`GameController.swift`) fires on GameController's *private background queue* — this is the source of a class of threading bugs (see §3.8, §6). +- Parsec's iOS SDK keyboard input is **scancode-only** (`MESSAGE_KEYBOARD` + `ParsecKeycode`). There is **no Unicode/text message**. This drove the design of language sync (§3.3) and Windows remap (§3.5). + +--- + +## 2. Branch state & commit timeline + +`git log --oneline cd155f8..HEAD` (oldest → newest): + +| SHA | Title | Theme | +|-----|-------|-------| +| `8277e0d` | Fix trackpad input lag and choppy 2-finger scroll | Bug 1 + Bug 2 baseline | +| `c6f9eb2` | Add Mac↔iPad keyboard layout sync via host hotkey | Language sync | +| `96ae81e` | Trackpad polish: natural scroll, inertia, sensitivity + key capture | Scroll polish + UIKeyCommand | +| `5d2bf2d` | Fix keyboard toolbar regression + scroll inertia + Mac-first labels | Regression fix | +| `5464629` | Persist last-used display + in-session resolution change | Display + resolution | +| `54a96d0` | ci: retrigger workflow | empty commit | +| `de86645` | Apply audit P0/P1/P2/P3 | Big audit batch | +| `4ad1d42` | Fix build: ParsecStatusBar init + local cursor overlay | Build fix + overlay | +| `0bc675d` | Windows host key remap + 4 audit-found bugfixes | Windows remap | +| `2b57e24` | v4 — real bugfixes from user testing | Scroll dir, cursor, reconnect, display | +| `af5261b` | v4 follow-up: loosen language-sync gates + lower inertia threshold | Tuning | +| `aca422b` | Add Mac Ctrl+Shift hotkey + adjustable mouse acceleration | Features (introduced a build break) | +| `fc8357f` | Fix scroll inertia tail — threshold + decay + touchesBegan gate | Inertia tuning | +| `5d31035` | Fix Resolution-menu crash, display persistence, display-switch debounce | Crash + display | +| `041c789` | Fix build: actually add @AppStorage mouseAcceleration | Build fix | +| `e924473` | GCMouse: move local cursor, fix wheel direction, fix x/y wheel swap | External mouse | +| `fc6776c` | Fix GCMouse off-main crash + crash reporter + CADisplayLink deinit | Crash fixes | + +**8 files changed, ~1316 insertions / 72 deletions.** Heaviest: `ParsecViewController.swift` (~827 lines added). + +> Note for the reviewer: several commits fixed build breaks introduced by earlier commits in the same series (`aca422b`→`041c789`, `5d2bf2d`'s `inputAccessoryView` read-only override→`4ad1d42`). The series was authored without a local compiler — only CI validated. Treat the *final* state at `fc6776c` as the truth; intermediate commits may contain code that was later corrected. + +--- + +## 3. Every change, by subsystem + +### 3.1 Trackpad cursor lag — Bug 1 (issue #47) + +**Symptom:** cursor in the stream lagged/juddered while moving a finger on the Magic Keyboard trackpad. + +**Root cause:** the main `panGestureRecognizer` had no `allowedTouchTypes` filter, so it ingested `.indirectPointer` UITouches (iPad trackpad/pointer, raw type 3). `UIPanGestureRecognizer` imposes a small movement threshold before `.began` and re-arms its state machine between strokes; at the per-frame trackpad event rate that produces visible stickiness. + +**Fix (`ParsecViewController.swift`, `viewDidLoad`):** +- `panGestureRecognizer.allowedTouchTypes = [NSNumber(.direct.rawValue), NSNumber(.pencil.rawValue)]` — excludes `.indirectPointer`. +- Added `override func touchesMoved(...)` handling `.indirectPointer` touches directly via `preciseLocation(in:) - precisePreviousLocation(in:)`, sub-pixel accumulation through `accumulatedDeltaX/Y`. `cursorMode == .direct` → `sendMousePosition`; otherwise `sendMouseDelta`. +- `touchesBegan` resets accumulators on `.indirectPointer`; `touchesEnded`/`touchesCancelled` too. +- `prefersPointerLocked = true` (pre-existing) is what makes iPad deliver trackpad motion as `.indirectPointer` touches. + +### 3.2 Trackpad 2-finger scroll + inertia — Bug 2 + +**Symptom:** choppy, stepped 2-finger scroll; later "no inertia at all". + +**Root cause(s):** +- Original 2-finger branch used `velocity(in:)/20` → large irregular wheel deltas. +- Later inertia attempt had a stop threshold (`0.5` pts/frame) that killed the glide in ~270 ms. +- Scroll accumulator used `Int32()` truncation → sub-pixel ticks swallowed. +- `touchesBegan` killed momentum on every touch incl. `.indirectPointer`. + +**Fix (`ParsecViewController.swift`):** +- Dedicated `UIPanGestureRecognizer` with `allowedScrollTypesMask = .all`, `maximumNumberOfTouches = 0` → only scroll-wheel/trackpad-scroll events. Handler `handleTrackpadScroll` uses `translation(in:)` deltas. +- Scroll accumulator now `.rounded(.toNearestOrAwayFromZero)`. +- Peak-velocity tracking during `.changed` (the recognizer's `velocity` is decayed to ~0 by iPad's own deceleration before `.ended`); peak reset only after a >1.0 s gap. +- Momentum via `CADisplayLink`: stop threshold `0.05` pts/frame, decay `0.90 + 0.095*strength` (strength 0..1 from `scrollMomentumStrength`). +- `touchesBegan` kills momentum only on `.direct`/`.pencil`. +- `naturalScrolling ? +1 : -1` direction sign (ON = no client-side invert, matching macOS default Natural Scrolling). + +> **Reviewer caution (§6):** the "best-practices" research concluded iPadOS auto-synthesizes momentum phases on the pan recognizer, and apps like Moonlight do **not** implement client-side inertia. The current code does client-side `CADisplayLink` inertia. This works but is non-canonical; a future refactor may remove it. Verify the current decay parameters don't double-apply on top of iPad's own deceleration tail. + +### 3.3 Mac↔iPad keyboard layout sync + +**Goal:** when the user toggles the iPad's hardware-keyboard input language (Caps Lock / Ctrl+Space), the host's input source should follow. + +**Constraint:** Parsec iOS SDK is scancode-only — no way to send composed Unicode. So we cannot bypass host layout; instead we fire a configurable hotkey at the host to make *it* switch. + +**Mechanics (`ParsecViewController.swift`, `LanguageSyncCoordinator` + `LanguageSyncTextField`):** +- A 1×1, alpha-0 `LanguageSyncTextField` (UITextField subclass) is added to the view and made first responder. Needed because `UITextInputMode.currentInputModeDidChangeNotification` only fires when a text-input first responder exists. +- The field installs an empty `inputView` (suppress soft keyboard) and an empty non-nil `inputAccessoryView` (halt the responder-chain walk so the VC's keyboard toolbar does NOT appear by default — this was a regression fixed in `5d2bf2d`). +- The field forwards `pressesBegan/Ended/Changed/Cancelled` to the VC **without calling `super`** — hardware-keyboard scancodes keep flowing through the existing `pressesBegan` pipeline (same trick Moonlight uses). +- On a real language change, `sendLayoutSyncHotkey()` fires the configured chord via `CParsec.sendVirtualKeyboardInput`. Options: Ctrl+Space (default, macOS), Ctrl+Shift, Cmd+Space, Opt+Space, Alt+Shift (Windows), Off. +- Coordinator yields FR before the VC becomes FR (soft keyboard via 3-finger tap / button) and reclaims after. + +> **Reviewer caution (§6):** the host only switches if it has the matching shortcut bound (macOS Sequoia default for "Select previous input source" is NOT Ctrl+Space). This is the dominant reason the user perceives "sync doesn't work" — it is a host-config issue, not necessarily a client bug. Also verify: (a) hidden field reliably reclaims FR after the soft keyboard is dismissed via OS routes that bypass `setKeyboardVisible(false)`; (b) the initial-seed logic doesn't fire a spurious hotkey at session start for users with 3+ layouts. + +### 3.4 System-shortcut capture (UIKeyCommand registry) + +**Goal:** let Cmd+letter shortcuts (Cmd+A/C/V/Z/S…) reach the host instead of being eaten by the iPad shell. + +**Mechanics:** `override var keyCommands` returns a cached (`static var _cachedKeyCommands`) list of ~286 `UIKeyCommand`s — `(a–z, 0–9, punctuation) × (Cmd, Cmd+Shift, Cmd+Opt, Cmd+Ctrl, Opt, Opt+Shift)` + Cmd+(Tab/Space/Enter/`). On iOS 15+, `wantsPriorityOverSystemBehavior = true`. `handleCapturedKey` translates to a modifier-press / key / release scancode sequence (synchronous in Low-Latency mode, async +20/+60 ms otherwise). + +**Hard limit:** Cmd+Space (Spotlight), Cmd+H, Cmd+Tab, Globe key, swipe-up — wired below the responder chain in SpringBoard; **no sandboxed app can intercept them**. + +### 3.5 Windows host key remap + +`SettingsHandler.windowsHostKeyboardRemap` (default off). When on, `ParsecSDKBridge.remapKeyForHostIfNeeded` swaps scancodes at the lowest layer (so every input path inherits it): `227 LGUI ↔ 224 LCTRL`, `231 RGUI ↔ 228 RCTRL`. Opt (226/230) and Shift (225/229) untouched. So Cmd+C on the iPad arrives as Ctrl+C on Windows. + +### 3.6 Local cursor overlay + +`SettingsHandler.localCursorOverlay` (default off). Draws a 13 pt iPadOS-style gray dot (`UIView` with cornerRadius/border/shadow) on `contentView`, tracked client-side from input deltas (no host RTT). When on, `updateImage` hides the host-streamed cursor (`u`). Seeded at `contentView.center` in `viewDidLayoutSubviews` (one-shot via `hasSeededLocalCursor`, because `viewDidLoad` sees zero bounds). Also useful as a workaround when a Windows host doesn't stream a cursor image at all. + +### 3.7 Mouse acceleration + +`SettingsHandler.mouseAcceleration` (0…1.5, default 0 = linear). `effectiveDeltaScale(rawDX:rawDY:)` returns `sensitivity + accel × (|delta|/5)` — fast flicks travel further. Applied in both `touchesMoved` (.indirectPointer) and `handlePanGesture` (touchscreen) touchpad branches. + +> **Reviewer note:** best-practices research recommends linear + let the host apply its own curve. The acceleration here is a client-side curve stacked on top of the host's. Default 0 keeps it off; only opt-in users get the stacked curve. + +### 3.8 External mouse (GCMouse) — `GameController.swift` + +- `mouseMovedHandler` sends `sendMouseDelta`; **the local-cursor overlay update is dispatched to `DispatchQueue.main`** because GCMouse handlers run on GC's private background queue (touching `UIView.center` off-main traps — this was the crash fixed in `fc6776c`). +- Scroll: `yAxis`→y, `xAxis`→x (fixed a pre-existing x/y swap), with `naturalScrolling` sign + `scrollSensitivity`. +- Magic Keyboard trackpad does **not** enumerate as GCMouse — this path is only for external USB/BT mice. + +### 3.9 In-session resolution change — `ParsecView.changeResolution` + +**Constraint discovered:** Parsec host honours bitrate / FPS / output via `setVideoConfig` user-data, but **not resolution** — resolution is only read at `ParsecClientConnect`. So `changeResolution` does a clean disconnect + 600 ms gap + reconnect with new `ParsecClientConfig`. +- `isReconfiguring` guard prevents re-entry (spam-tap). +- Suppresses the status-bar disconnect alert during the gap; shows a "Switching resolution…" overlay; pauses GLKViewController (`isPaused = true`) so the last frame stays on screen instead of going black. +- Branches on `connect()` status — surfaces a real "Reconnect failed" alert if the host went away. + +> **Reviewer caution (§7):** even at connect, a macOS host with a single physical display ignores small resolution requests (no virtual display). The user reported 1920×1080 in Settings is ignored — this is host behavior, not a client bug. Bitrate/H.265 are the working bandwidth levers. + +### 3.10 Display selection persistence — `ParsecView.changeDisplay` + `ParsecSDKBridge` case 12 + +- `SettingsHandler.savedDisplayOutput` (id) + `savedDisplayName` (name+adapter fallback, because Parsec regenerates display ids across sessions). +- `handleUserDataEvent case 12` restores once per session (`didRestoreSavedDisplay`, reset in **both** `connect()` and `disconnect()` — the connect() reset was the key fix because reconnect paths bypass disconnect). +- `updateHostVideoConfig` resends the payload at +250 ms and a `getVideoConfig` at +450 ms — the host can drop a `setVideoConfig` mid-encoder-reset, which is why display switches needed multiple taps. + +### 3.11 Low Latency Mode + latency reductions + +`SettingsHandler.lowLatencyMode` toggle flips `preferredFramesPerSecond = 0` (device max), `decoder = h265`, `noOverlay = true`, and gates the 20/60 ms captured-key holds. Independently: +- `preferredFramesPerSecond` default changed 60 → **0** (was capping 120 Hz iPads at 60 → doubled present latency). +- `startBackgroundTask` poll timeout scales with FPS; QoS raised to `.userInteractive`. +- `ParsecGLKRenderer` gates PiP `captureFrame` on `isPiPActive || isStarting`. +- Mouse accumulator rounding (no event coalescing on slow drags). + +### 3.12 Reconnect UX + send gating + +- All `ParsecSDKBridge` send methods early-return on `!backgroundTaskRunning` (the gate is set true in `connect()`, false in `disconnect()`). Prevents `ParsecClientSendMessage` into a torn-down client during the reconnect gap. +- `disconnect()` sleeps 20 ms to drain the two poll loops before a fast reconnect spawns new ones. + +### 3.13 Crash reporter — `AppDelegate.swift` + +`CrashReporter.install()` sets `NSSetUncaughtExceptionHandler` + signal handlers (SIGABRT/SEGV/BUS/ILL/FPE/TRAP). Writes `Documents/last_crash.log` with a backtrace. On next launch: copies the log to `UIPasteboard` (syncs to a Mac via Universal Clipboard) and leaves the file in Documents (browsable via Files app — `UIFileSharingEnabled` + `LSSupportsOpeningDocumentsInPlace` added to Info.plist). + +### 3.14 Concurrency / lifecycle hardening + +- `ParsecViewController.deinit` invalidates `momentumDisplayLink` and stops `languageSync`. +- `CADisplayLink` always invalidated before a new one is created in `startScrollMomentum`. +- `keyCommands` cached (was rebuilding 286 objects per query). + +--- + +## 4. New settings (`SettingsHandler.swift` @AppStorage keys) + +| Key | Type | Default | Surfaced in SettingsView | +|-----|------|---------|--------------------------| +| `mouseAcceleration` | Double | 0.0 | Interactivity | +| `localCursorOverlay` | Bool | false | Interactivity | +| `scrollSensitivity` | Double | 1.0 | Interactivity | +| `naturalScrolling` | Bool | true | Interactivity | +| `scrollMomentum` | Bool | true | Interactivity | +| `scrollMomentumStrength` | Double | 0.5 | Interactivity | +| `captureSystemKeys` | Bool | true | Keyboard | +| `windowsHostKeyboardRemap` | Bool | false | Keyboard | +| `syncKeyboardLayout` | Bool | true | Keyboard | +| `layoutSyncHotkey` | LayoutSyncHotkey | .ctrlSpace | Keyboard | +| `lowLatencyMode` | Bool | false | Graphics | +| `preferredFramesPerSecond` | Int | **0** (was 60) | Graphics | +| `savedDisplayOutput` | String | "" | (internal) | +| `savedDisplayName` | String | "" | (internal) | + +> **Pre-existing bug, NOT introduced here, but worth flagging:** `SettingsHandler.swift` reuses the key `"cursorScale"` for both `cursorScale: Double` AND `hideStatusBar: Bool`. They collide in UserDefaults. Left untouched to keep the diff scoped, but a reviewer may want to fix it (`hideStatusBar` should have its own key). + +--- + +## 5. User feedback log (chronological, paraphrased) + +1. ✅ Trackpad cursor lag fixed (confirmed by user). +2. "Screen moves when scrolling" → was the keyboard accessory toolbar showing by default (language-sync FR regression). Fixed (`5d2bf2d`). +3. Couldn't install .ipa ("integrity") → sideloader cert / bundle-id; solved by unsigned + unique-bundle-id variants. +4. Natural-scroll toggle inverted / didn't work → sign flipped (`2b57e24`); scroll accumulator rounding (`fc8357f`). +5. Local cursor "crooked, doesn't move" → SF-Symbol arrow → UIView dot, seed in `viewDidLayoutSubviews` (`2b57e24`). +6. Display selection not remembered → `didRestoreSavedDisplay` reset in connect (`5d31035`). +7. Resolution change → "Disconnected 20" → 600 ms reconnect gap (`2b57e24`/`5d31035`). +8. Resolution in Settings ignored → **host-side limitation** (§7), not fixed client-side. +9. Language switch doesn't work → loosened gates (`af5261b`); likely also host-shortcut config (§7). +10. Scroll "no inertia at all", "ragged with fingers on trackpad" → threshold/decay/touchesBegan fixes (`fc8357f`). +11. Resolution menu **crashes** → iOS 14.0–14.4 UIMenu `_ConditionalContent` bug; fixed with HStack (`5d31035`). +12. Add Ctrl+Shift hotkey ✅ (`aca422b`). Add mouse acceleration ✅ (`aca422b`/`041c789`). +13. External mouse → cursor disappears / wrong wheel direction / no cursor on Windows → GCMouse fixes (`e924473`), off-main crash fix (`fc6776c`). +14. "App crashes during scroll / in general" → most likely the GCMouse off-main UIView mutation (`fc6776c`) and/or the iOS-14 Resolution-menu crash (`5d31035`). **Awaiting a crash log** from the new crash reporter to confirm. + +--- + +## 6. Outstanding bugs / open code-review findings + +A 5-angle code review was started but the finder sub-agents hit a session limit before completing. The reviewer-author's own confirmed/plausible findings: + +1. **(FIXED `fc6776c`) [HIGH]** GCMouse `moveLocalCursor` off-main → UIView mutation crash. +2. **[MED, OPEN]** `updateHostVideoConfig` schedules 2 async resends per call. Rapid bitrate-slider drags stack many resends + `getVideoConfig` echoes; the case-11 echo writes `DataManager.model.bitrate` back to the host's reported value, which could snap the slider mid-drag. *Suggested fix:* debounce `updateHostVideoConfig` with a token/timestamp; skip resend if superseded. +3. **[MED, OPEN]** Language-sync hidden field may not reclaim first responder if the soft keyboard is dismissed via an OS route that doesn't call `setKeyboardVisible(false)`. *Verify:* does `keyboardWillHide` → `onKeyboardVisibilityChanged` → `setKeyboardVisible(false)` cover swipe-to-dismiss and hardware Esc? +4. **[LOW, OPEN]** Two identical external monitors collide on the name fallback (`changeDisplay`/case 12 name match picks first). Id-match runs first, so only matters when ids roll AND monitors are identical. +5. **[LOW, OPEN]** `scrollMomentumTick` and `handleTrackpadScroll` both touch `momentumVelocity*` — both on main, so not a true race, but confirm no path schedules the tick off-main. +6. **[LOW, OPEN]** Spurious Ctrl+Space at session start possible for 3+ layout users (initial-seed nil → first notification fires). Accepted trade-off; revisit if reported. +7. **[INFO]** Client-side scroll inertia is non-canonical (iPadOS provides momentum phases). Consider removing in favor of forwarding native momentum events (Moonlight approach). Would also remove the `scrollMomentumStrength` tuning surface. +8. **[INFO]** Mouse acceleration stacks a client curve on the host's curve. Default off mitigates. + +--- + +## 7. Hard constraints — cannot be fixed client-side + +1. **Resolution downscale on macOS hosts.** Parsec captures the physical display at native resolution; `parsecClientCfg.video.0.resolutionX/Y` is advisory and ignored without a virtual-display driver (BetterDummy etc.). Bitrate + H.265 are the real bandwidth levers. +2. **Hiding the cursor on the host's own physical display.** No client→host cursor message exists in the Parsec SDK; macOS draws the cursor before Parsec captures the frame. Requires a host-side helper (e.g. Hammerspoon `CGDisplayHideCursor` bound to a hotkey the iPad sends). +3. **System shortcuts** Cmd+Space / Cmd+H / Cmd+Tab / Globe / swipe-up — SpringBoard-level, not interceptable. Workaround: Windows-host remap, or Opt-based combos. +4. **Layout sync requires host config.** The host must have the chosen hotkey bound to "Select previous input source". macOS Sequoia does not bind Ctrl+Space by default. + +--- + +## 8. Files reference map + +| File | What changed | +|------|--------------| +| `OpenParsec/ParsecViewController.swift` | Trackpad cursor (`touchesMoved`), scroll + inertia (`handleTrackpadScroll`, momentum CADisplayLink), language sync (`LanguageSyncCoordinator`, `LanguageSyncTextField`), key capture (`keyCommands`, `handleCapturedKey`), local cursor overlay, mouse acceleration (`effectiveDeltaScale`), layout-sync hotkey sender, deinit. | +| `OpenParsec/ParsecSDKBridge.swift` | Send gates (`backgroundTaskRunning`), `remapKeyForHostIfNeeded`, `didRestoreSavedDisplay`, case-12 restore, `updateHostVideoConfig` resend, `connect`/`disconnect` state resets, poll-thread QoS/timeout, `applyConfig` resolution. | +| `OpenParsec/ParsecView.swift` | Resolution/Bitrate menu HStack fix, `changeResolution` reconnect flow, `changeDisplay` persist, `ParsecStatusBar` `isReconfiguring` gating + overlay. | +| `OpenParsec/SettingsHandler.swift` | All new @AppStorage keys. | +| `OpenParsec/SettingsView.swift` | UI for all new settings. | +| `OpenParsec/CParsec.swift` | `lastConnectedPeerID` lifecycle. | +| `OpenParsec/GameController.swift` | GCMouse: local-cursor (main-dispatched), wheel direction/sensitivity, x/y swap fix. | +| `OpenParsec/ParsecGLKRenderer.swift` | PiP captureFrame gate. | +| `OpenParsec/AppDelegate.swift` | `CrashReporter`. | +| `OpenParsec/Info.plist` | `UIFileSharingEnabled`, `LSSupportsOpeningDocumentsInPlace`. | + +--- + +## 9. Build, CI, signing, release + +- **CI:** `.github/workflows/build.yml`, `macos-latest`, Xcode 26.2 → `xcodebuild archive ... CODE_SIGNING_ALLOWED=NO` → fake-sign with `ldid` on Ubuntu → `OpenParsec.ipa` artifact. (Fork required Actions to be manually enabled once.) +- **Releases** are produced manually by the author: download the CI `.ipa`, then for sideloader compatibility produce four variants per release tag: + - `OpenParsec-vN.ipa` — CI original (Linux `ldid` fake-sign, bundle `com.aigch.OpenParsec`). + - `OpenParsec-vN-unsigned.ipa` — signature stripped, original bundle id (Scarlet/eSign re-sign cleanly). + - `OpenParsec-vN-trackpadfix.ipa` — unique bundle id `com.2extndd.openparsec.trackpadfix`, macOS ad-hoc signed (AltStore/SideStore). + - `OpenParsec-vN-trackpadfix-unsigned.ipa` — unique bundle id, unsigned (parallel install). + - `CFBundleVersion` is bumped per release to defeat sideloader caches. +- **Upstream PR:** `hugeBlack/OpenParsec#70` tracks the branch. + +--- + +## 10. Test plan (manual QA, per feature) + +**Trackpad:** cursor smooth on slow + fast finger moves; no judder at gesture start. 2-finger scroll smooth; lift → visible inertia glide (~0.5 s default); toggle Inertia off → stops dead. Natural Scrolling on = swipe-down moves content down. + +**Local cursor:** enable overlay → gray dot appears centered, follows finger with no RTT; host cursor hidden. Plug external mouse → dot follows mouse (no crash). Zoom in (pinch) → scroll pans locally, dot scales with content. + +**Keyboard:** type normally (no double chars, no stuck toolbar). Cmd+A/C/V/Z in a host text app. Caps-Lock layout toggle → host switches (requires host shortcut bound). Windows remap on → Cmd+C = copy on Windows. 3-finger tap → soft keyboard + toolbar; Done → toolbar gone, language sync still live. + +**Resolution:** open overlay → Resolution menu (must NOT crash on iOS 14.x). Pick a value → "Switching resolution…" overlay, last frame frozen (no black), reconnect ≤ ~1 s, no false Disconnected alert. + +**Display:** multi-display host → pick display (switches on first tap). Disconnect, reconnect → same display restored. Kill app, relaunch, reconnect → still restored. + +**Latency:** Low Latency Mode on → metrics overlay shows device-max FPS (120 on ProMotion). Cmd shortcuts feel instant. + +**Crash reporter:** force a crash (if repro known) → relaunch → crash log in clipboard + in Files app (On My iPad → OpenParsec → last_crash.log). + +--- + +## 11. Recommended next steps for the reviewing agent + +1. **Resolve the finder-agent code review** (§6) — especially #2 (bitrate resend pile-up) and #3 (FR reclaim). These are the most likely remaining real bugs. +2. **Confirm GCMouse fix** (`fc6776c`) actually serializes all `localCursorImageView` access to main (also check the scroll x/y handlers — they only call `CParsec.sendWheelMsg`, which is SDK-thread-safe, so they're fine off-main). +3. **Decide on the inertia architecture** — keep client-side `CADisplayLink` (current) or switch to forwarding iPadOS momentum phases (canonical). The user has repeatedly reported inertia feel issues; the canonical approach may end the back-and-forth. +4. **Verify language-sync reliability** end-to-end with a host that has the shortcut bound — separate "client didn't fire" from "host didn't act". +5. **Consider fixing the `cursorScale`/`hideStatusBar` UserDefaults key collision** (§4) — pre-existing but trivial. +6. **Once a real crash log arrives**, symbolicate and confirm whether the remaining crashes are the GCMouse path (now fixed) or something else. diff --git a/OpenParsec/AppDelegate.swift b/OpenParsec/AppDelegate.swift index 2eb006f..21f2c0a 100644 --- a/OpenParsec/AppDelegate.swift +++ b/OpenParsec/AppDelegate.swift @@ -1,10 +1,179 @@ import UIKit +import Darwin + +// Lightweight crash reporter. Writes the last uncaught exception / fatal +// signal (with a backtrace) to Documents/last_crash.log. On the next launch +// the log is copied to the system pasteboard (so it can be pasted straight +// into a chat, or synced to a Mac via Universal Clipboard) and left in the +// app's Documents folder, which is exposed in the Files app via +// UIFileSharingEnabled. Zero infrastructure required. +enum CrashReporter { + static var logURL: URL { + let docs = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] + return docs.appendingPathComponent("last_crash.log") + } + + // ---- Async-signal-safe state, all pre-allocated in install() ---- + // The signal handler must never call malloc, Foundation, or Swift String: + // if the crashing thread already holds the malloc lock (the common case for + // SIGSEGV/SIGABRT), any allocation inside the handler deadlocks and the log + // never gets written — exactly the failure that made crashes undiagnosable. + // So everything the handler touches is allocated up front here. + private static var logPathC: UnsafeMutablePointer? = nil // strdup'd log path + private static var headerPrefixC: UnsafeMutablePointer? = nil // "=== OpenParsec fatal signal " + private static var headerSuffixC: UnsafeMutablePointer? = nil // " ===\nBacktrace:\n" + private static let backtraceCapacity: Int32 = 128 + private static var backtraceBuffer: UnsafeMutablePointer? = nil + // Set to 1 by the NSException handler so a following SIGABRT (from the + // uncaught-exception abort) appends instead of clobbering the richer log. + private static var exceptionRecorded: sig_atomic_t = 0 + + // Rich recorder for the NSException path ONLY. Runs in normal context + // (not a signal trampoline), so Foundation/allocation is safe here. + static func record(_ header: String) { + let stamp = ISO8601DateFormatter().string(from: Date()) + var text = "=== OpenParsec crash @ \(stamp) ===\n" + text += header + "\n\nBacktrace:\n" + text += Thread.callStackSymbols.joined(separator: "\n") + text += "\n" + try? text.write(to: logURL, atomically: true, encoding: .utf8) + exceptionRecorded = 1 + } + + // Async-signal-safe write of a decimal Int32 using a stack buffer only — + // no heap, so it is safe to call from the signal handler. + private static func writeInt(_ fd: Int32, _ value: Int32) { + withUnsafeTemporaryAllocation(of: CChar.self, capacity: 12) { p in + var i = 12 + var n = value + let negative = n < 0 + if negative { n = -n } + if n == 0 { i -= 1; p[i] = CChar(48) } // '0' + while n > 0 { i -= 1; p[i] = CChar(48 + Int(n % 10)); n /= 10 } + if negative { i -= 1; p[i] = CChar(45) } // '-' + _ = write(fd, p.baseAddress! + i, 12 - i) + } + } + + // THE SIGNAL HANDLER. Async-signal-safe primitives only: open, write, + // backtrace, backtrace_symbols_fd (writes straight to the fd, no malloc), + // close, signal, raise. + private static func handleSignal(_ s: Int32) { + if let path = logPathC { + // Append (don't truncate) if the exception handler already wrote a + // richer log before abort()ing into this signal. + let flags = exceptionRecorded != 0 + ? (O_WRONLY | O_CREAT | O_APPEND) + : (O_WRONLY | O_CREAT | O_TRUNC) + let fd = open(path, flags, 0o644) + if fd >= 0 { + if let h = headerPrefixC { _ = write(fd, h, strlen(h)) } + writeInt(fd, s) + if let h = headerSuffixC { _ = write(fd, h, strlen(h)) } + if let buf = backtraceBuffer { + let frames = backtrace(buf, backtraceCapacity) + backtrace_symbols_fd(buf, frames, fd) + } + _ = fsync(fd) + close(fd) + } + } + signal(s, SIG_DFL) + raise(s) + } + + static func install() { + // Pre-allocate everything the signal handler will touch. + logPathC = strdup(logURL.path) + headerPrefixC = strdup("=== OpenParsec fatal signal ") + headerSuffixC = strdup(" ===\nBacktrace:\n") + backtraceBuffer = UnsafeMutablePointer.allocate(capacity: Int(backtraceCapacity)) + + NSSetUncaughtExceptionHandler { exception in + CrashReporter.record( + "Uncaught NSException: \(exception.name.rawValue)\n" + + "Reason: \(exception.reason ?? "(nil)")\n" + + "User stack:\n" + exception.callStackSymbols.joined(separator: "\n") + ) + } + for sig in [SIGABRT, SIGSEGV, SIGBUS, SIGILL, SIGFPE, SIGTRAP] { + signal(sig) { s in + CrashReporter.handleSignal(s) + } + } + } + + // Non-destructive read so a Settings "Copy Last Crash Log" action can + // retrieve it at any time. The file is overwritten on the next crash and + // can be cleared explicitly via clear(). + static func peek() -> String? { + guard let text = try? String(contentsOf: logURL, encoding: .utf8), !text.isEmpty else { return nil } + return text + } + + static func clear() { + try? FileManager.default.removeItem(at: logURL) + } +} + +// Lightweight append-only diagnostics channel, sibling to CrashReporter. +// Used for empirical discovery that has no local console (e.g. the host-OS +// int encoding in S04, or netProtocol/mediaContainer in S08). Entries persist +// to Documents/diagnostics.log and can be copied out via Settings → +// "Copy Diagnostics" or pulled from the Files app. Called only from normal +// (non-signal) contexts, so Foundation use is fine here. +enum Diagnostics { + static var logURL: URL { + let docs = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] + return docs.appendingPathComponent("diagnostics.log") + } + + static func note(_ line: String) { + let stamp = ISO8601DateFormatter().string(from: Date()) + let entry = "[\(stamp)] \(line)\n" + print("[OpenParsec][diag] \(line)") + guard let data = entry.data(using: .utf8) else { return } + if let handle = try? FileHandle(forWritingTo: logURL) { + defer { try? handle.close() } + handle.seekToEndOfFile() + handle.write(data) + } else { + // File doesn't exist yet — create it with this first entry. + try? data.write(to: logURL, options: .atomic) + } + } + + static func peek() -> String? { + guard let text = try? String(contentsOf: logURL, encoding: .utf8), !text.isEmpty else { return nil } + return text + } + + static func clear() { + try? FileManager.default.removeItem(at: logURL) + } +} @main class AppDelegate: UIResponder, UIApplicationDelegate { func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions:[UIApplication.LaunchOptionsKey: Any]?) -> Bool { + // Install the crash reporter as early as possible so it catches + // failures during the rest of launch too. + CrashReporter.install() + // Q1: repair the hideStatusBar / cursorScale shared-key corruption + // before any UI reads these settings. + SettingsHandler.migrateLegacyStatusBarKeyIfNeeded() + if let crash = CrashReporter.peek() { + // Make the previous crash trivially retrievable: copy to the + // pasteboard (syncs to a Mac on the same Apple ID via Universal + // Clipboard) and print it for any attached console. The file is + // kept (not consumed) so the Settings "Copy Last Crash Log" row + // can re-surface it later. + UIPasteboard.general.string = crash + print("[OpenParsec] Recovered crash log from previous run:\n\(crash)") + } + // Override point for customization after application launch. UTMViewControllerPatches.patchAll() return true diff --git a/OpenParsec/CParsec.swift b/OpenParsec/CParsec.swift index d1fc736..6fa982c 100644 --- a/OpenParsec/CParsec.swift +++ b/OpenParsec/CParsec.swift @@ -118,6 +118,30 @@ enum ParsecResolution: String, CaseIterable, Hashable { } +// Host operating system, derived from the case-11 video-config `hostOS` Int. +// The Int→OS encoding is NOT documented (the ParsecSDK framework headers are +// not vendored), so the mapping below is intentionally empty until the values +// are confirmed empirically. Connect to a known Mac host and a known Windows +// host and read the `hostOS=` lines written by Diagnostics.note (retrievable +// via Settings → "Copy Diagnostics" or the Files app), then fill in the cases. +// Until then `from` returns .unknown and every OS-gated feature falls back to +// its manual toggle — so a wrong guess can never mis-gate behaviour. +enum HostOS { + case unknown + case macos + case windows + case linux + + static func from(_ raw: Int) -> HostOS { + switch raw { + // TODO(discovery): map the observed raw values here, e.g. + // case 1: return .windows + // case 2: return .macos + default: return .unknown + } + } +} + struct MouseInfo { var pngCursor: Bool = false var mouseX:Int32 = 1 @@ -137,6 +161,9 @@ protocol ParsecService { var hostWidth: Float { get } var hostHeight: Float { get } var mouseInfo: MouseInfo { get } + // Raw host-OS int from the case-11 video config (-1 until received). Read + // lock-free from input threads; a plain Int load is atomic on the device. + var hostOSValue: Int { get } func connect(_ peerID: String) -> ParsecStatus func disconnect() @@ -180,11 +207,23 @@ class CParsec public static var mouseInfo: MouseInfo { return parsecImpl.mouseInfo } + + // Resolved host OS for feature gating. .unknown until the case-11 value + // arrives AND the HostOS mapping is filled in from discovery. + public static var hostOS: HostOS { + return HostOS.from(parsecImpl.hostOSValue) + } static var parsecImpl: ParsecService! + // Remember the last peer we connected to so callers like changeResolution + // can disconnect + reconnect with new ParsecClientConfig (resolution can + // only be changed via a fresh ParsecClientConnect — the in-session + // setVideoConfig user-data event does not move resolution on the host). + public static var lastConnectedPeerID: String? + static func initialize() { parsecImpl = ParsecSDKBridge() @@ -192,16 +231,22 @@ class CParsec static func destroy() { - + } static func connect(_ peerID: String) -> ParsecStatus { - parsecImpl.connect(peerID) + lastConnectedPeerID = peerID + return parsecImpl.connect(peerID) } static func disconnect() { + // Clear the peer-ID memo on any disconnect path; changeResolution sets + // it again right before calling connect(), so the reconnect dance is + // unaffected — but a user-initiated disconnect (close stream button, + // app background, etc.) should leave us with no stale peer. + lastConnectedPeerID = nil parsecImpl.disconnect() } diff --git a/OpenParsec/GameController.swift b/OpenParsec/GameController.swift index d8e44a7..f1ea060 100644 --- a/OpenParsec/GameController.swift +++ b/OpenParsec/GameController.swift @@ -91,8 +91,9 @@ class GamepadController { func registerMouseHandler() { for mouse in GCMouse.mice() { mice.insert(mouse) - mouse.mouseInput?.leftButton.pressedChangedHandler = {(input: GCControllerButtonInput, v: Float, pressed: Bool) in + mouse.mouseInput?.leftButton.pressedChangedHandler = {[weak self] (input: GCControllerButtonInput, v: Float, pressed: Bool) in CParsec.sendMouseClickMessage(MOUSE_L, pressed) + _ = self // keep reference path symmetrical } mouse.mouseInput?.rightButton?.pressedChangedHandler = {(input: GCControllerButtonInput, v: Float, pressed: Bool) in CParsec.sendMouseClickMessage(MOUSE_R, pressed) @@ -100,14 +101,38 @@ class GamepadController { mouse.mouseInput?.middleButton?.pressedChangedHandler = {(input: GCControllerButtonInput, v: Float, pressed: Bool) in CParsec.sendMouseClickMessage(MOUSE_MIDDLE, pressed) } - mouse.mouseInput?.mouseMovedHandler={(input: GCMouseInput, v: Float, v2: Float) in - CParsec.sendMouseDelta(Int32(v/1.25 * Float(SettingsHandler.mouseSensitivity)), Int32(-v2/1.25 * Float(SettingsHandler.mouseSensitivity))) + mouse.mouseInput?.mouseMovedHandler = {[weak self] (input: GCMouseInput, v: Float, v2: Float) in + let sens = Float(SettingsHandler.mouseSensitivity) + let dx = v / 1.25 * sens + let dy = -v2 / 1.25 * sens // GCMouse Y is inverted vs screen + // GCMouse handlers fire on GameController's private background + // queue (no handlerQueue is set to main). CParsec sends are + // thread-safe, but the local-cursor overlay touches UIView + // geometry — that MUST happen on the main thread or UIKit + // traps. Dispatch the overlay update to main; keep the send + // inline so input latency isn't affected. + CParsec.sendMouseDelta(Int32(dx), Int32(dy)) + if SettingsHandler.localCursorOverlay { + DispatchQueue.main.async { + if let vc = self?.viewController as? ParsecViewController { + vc.moveLocalCursor(byX: CGFloat(dx), y: CGFloat(dy)) + } + } } + } + // Scroll wheel: yAxis = vertical (send as y), xAxis = horizontal + // (send as x). Previously these were swapped — pre-existing bug. + // Also apply naturalScrolling + scrollSensitivity so the wheel + // matches the trackpad's direction setting. mouse.mouseInput?.scroll.yAxis.valueChangedHandler = {(axis: GCControllerAxisInput, value: Float) in - CParsec.sendWheelMsg(x: Int32(value), y: 0) + let dir: Float = SettingsHandler.naturalScrolling ? 1.0 : -1.0 + let sens = Float(SettingsHandler.scrollSensitivity) + CParsec.sendWheelMsg(x: 0, y: Int32(value * sens * dir)) } mouse.mouseInput?.scroll.xAxis.valueChangedHandler = {(axis: GCControllerAxisInput, value: Float) in - CParsec.sendWheelMsg(x: 0, y: Int32(value)) + let dir: Float = SettingsHandler.naturalScrolling ? 1.0 : -1.0 + let sens = Float(SettingsHandler.scrollSensitivity) + CParsec.sendWheelMsg(x: Int32(value * sens * dir), y: 0) } } } diff --git a/OpenParsec/Info.plist b/OpenParsec/Info.plist index bc661cd..b7144a1 100644 --- a/OpenParsec/Info.plist +++ b/OpenParsec/Info.plist @@ -39,6 +39,10 @@ LSRequiresIPhoneOS + UIFileSharingEnabled + + LSSupportsOpeningDocumentsInPlace + UIApplicationSceneManifest UIApplicationSupportsMultipleScenes diff --git a/OpenParsec/LoginView.swift b/OpenParsec/LoginView.swift index 30bb609..7c2f5ec 100644 --- a/OpenParsec/LoginView.swift +++ b/OpenParsec/LoginView.swift @@ -192,15 +192,11 @@ struct LoginView:View { (data, response, error) in DispatchQueue.main.async { isLoading = false - if let data = data + if let data = data, let http = response as? HTTPURLResponse { - let statusCode:Int = (response as! HTTPURLResponse).statusCode + let statusCode:Int = http.statusCode let decoder = JSONDecoder() - print("Login Information:") - print(statusCode) - print(String(data:data, encoding:.utf8)!) - if statusCode == 201 // 201 Created { // store it and recover it from the next app opening, so people won't swear @@ -216,7 +212,12 @@ struct LoginView:View } else if statusCode >= 400 // 4XX client errors { - let info:ErrorInfo = try! decoder.decode(ErrorInfo.self, from:data) + guard let info = try? decoder.decode(ErrorInfo.self, from:data) else + { + alertText = "Login failed (HTTP \(statusCode))." + showAlert = true + return + } do { @@ -244,6 +245,11 @@ struct LoginView:View } } } + else + { + alertText = "Network error. Check your connection and try again." + showAlert = true + } } } task.resume() diff --git a/OpenParsec/MainView.swift b/OpenParsec/MainView.swift index 785ff38..9c7015b 100644 --- a/OpenParsec/MainView.swift +++ b/OpenParsec/MainView.swift @@ -433,43 +433,51 @@ struct MainView: View let task = URLSession.shared.dataTask(with:request) { (data, response, error) in DispatchQueue.main.async { - if let data = data + if let data = data, let http = response as? HTTPURLResponse { - let statusCode:Int = (response as! HTTPURLResponse).statusCode + let statusCode:Int = http.statusCode let decoder = JSONDecoder() if statusCode == 200 // 200 OK { - let info:HostInfoList = try! decoder.decode(HostInfoList.self, from:data) - hosts.removeAll() - if let datas = info.data + if let info = try? decoder.decode(HostInfoList.self, from:data) { - datas.forEach - { h in - hosts.append(IdentifiableHostInfo(id:h.peer_id, hostname:h.name, user:h.user, connections:h.players)) + hosts.removeAll() + if let datas = info.data + { + datas.forEach + { h in + hosts.append(IdentifiableHostInfo(id:h.peer_id, hostname:h.name, user:h.user, connections:h.players)) + } } - } - var grammar: String = "hosts" - if hosts.count == 1 - { - grammar = "host" - } + var grammar: String = "hosts" + if hosts.count == 1 + { + grammar = "host" + } - hostCountStr = "\(hosts.count) \(grammar)" + hostCountStr = "\(hosts.count) \(grammar)" - let formatter = DateFormatter() - formatter.dateFormat = "M/d/yyyy h:mm a" - refreshTime = "Last refreshed at \(formatter.string(from:Date()))" + let formatter = DateFormatter() + formatter.dateFormat = "M/d/yyyy h:mm a" + refreshTime = "Last refreshed at \(formatter.string(from:Date()))" + } } else if statusCode == 403 // 403 Forbidden { - let info:ErrorInfo = try! decoder.decode(ErrorInfo.self, from:data) - - baseAlertText = "Error gathering hosts: \(info.error)" - showBaseAlert = true + if let info = try? decoder.decode(ErrorInfo.self, from:data) + { + baseAlertText = "Error gathering hosts: \(info.error)" + showBaseAlert = true + } } } + else if error != nil + { + baseAlertText = "Network error gathering hosts. Check your connection and try again." + showBaseAlert = true + } isRefreshing = false } @@ -499,22 +507,25 @@ struct MainView: View let task = URLSession.shared.dataTask(with:request) { (data, response, error) in DispatchQueue.main.async { - if let data = data + if let data = data, let http = response as? HTTPURLResponse { - let statusCode:Int = (response as! HTTPURLResponse).statusCode + let statusCode:Int = http.statusCode let decoder = JSONDecoder() if statusCode == 200 // 200 OK { - let data: SelfInfoData = try! decoder.decode(SelfInfo.self, from:data).data - userInfo = IdentifiableUserInfo(id:data.id, username:data.name) + if let selfData = try? decoder.decode(SelfInfo.self, from:data).data + { + userInfo = IdentifiableUserInfo(id:selfData.id, username:selfData.name) + } } else { - let info:ErrorInfo = try! decoder.decode(ErrorInfo.self, from:data) - - baseAlertText = "Error gathering user info: \(info.error)" - showBaseAlert = true + if let info = try? decoder.decode(ErrorInfo.self, from:data) + { + baseAlertText = "Error gathering user info: \(info.error)" + showBaseAlert = true + } } } } @@ -544,40 +555,40 @@ struct MainView: View let task = URLSession.shared.dataTask(with:request) { (data, response, error) in DispatchQueue.main.async { - if let data = data + if let data = data, let http = response as? HTTPURLResponse { - let statusCode:Int = (response as! HTTPURLResponse).statusCode + let statusCode:Int = http.statusCode let decoder = JSONDecoder() - print("/friendships: \(statusCode)") - print(String(data:data, encoding:.utf8)!) - if statusCode == 200 // 200 OK { - let info:FriendInfoList = try! decoder.decode(FriendInfoList.self, from:data) - friends.removeAll() - if let datas = info.data + if let info = try? decoder.decode(FriendInfoList.self, from:data) { - datas.forEach - { f in - friends.append(IdentifiableUserInfo(id:f.user_id, username:f.user_name)) + friends.removeAll() + if let datas = info.data + { + datas.forEach + { f in + friends.append(IdentifiableUserInfo(id:f.user_id, username:f.user_name)) + } } - } - var grammar: String = "friends" - if friends.count == 1 - { - grammar = "friend" - } + var grammar: String = "friends" + if friends.count == 1 + { + grammar = "friend" + } - friendCountStr = "\(friends.count) \(grammar)" + friendCountStr = "\(friends.count) \(grammar)" + } } else { - let info:ErrorInfo = try! decoder.decode(ErrorInfo.self, from:data) - - baseAlertText = "Error gathering friends: \(info.error)" - showBaseAlert = true + if let info = try? decoder.decode(ErrorInfo.self, from:data) + { + baseAlertText = "Error gathering friends: \(info.error)" + showBaseAlert = true + } } } @@ -596,6 +607,13 @@ struct MainView: View var status = CParsec.connect(who.id) + // Invalidate any in-flight poll timer before scheduling a new one. A + // background-reconnect (onShouldReconnect -> connectTo) or a rapid + // re-tap during the connecting phase can call connectTo while a prior + // timer is still live; without this the old repeating timer is orphaned + // and both fire, racing setView(.parsec)/showBaseAlert. + pollTimer?.invalidate() + // Polling status pollTimer = Timer.scheduledTimer(withTimeInterval:1, repeats: true) { timer in @@ -628,7 +646,10 @@ struct MainView: View CParsec.disconnect() - pollTimer!.invalidate() + // Q2: pollTimer is nil if Cancel is tapped before the connect poll + // timer schedules — the old force-unwrap crashed on that race. + pollTimer?.invalidate() + pollTimer = nil } func logout() diff --git a/OpenParsec/ParsecBackgroundManager.swift b/OpenParsec/ParsecBackgroundManager.swift index 6b27361..926a621 100644 --- a/OpenParsec/ParsecBackgroundManager.swift +++ b/OpenParsec/ParsecBackgroundManager.swift @@ -9,6 +9,19 @@ class ParsecBackgroundManager { private var didDisconnectDueToBackground = false private(set) var isReconnecting = false + // Keep-alive grace window: when the app is backgrounded without PiP we + // hold a finite UIBackgroundTask instead of dropping the stream at once. + // A quick app-switch that returns inside the window resumes the live + // connection instantly; staying away past it (or the OS reclaiming the + // time via the expiration handler) falls back to the disconnect+reconnect + // path. beginBackgroundTask needs no background-mode entitlement. + private var backgroundTaskId: UIBackgroundTaskIdentifier = .invalid + private var graceTimer: Timer? + // Held under iOS's typical ~30 s background allowance so our timer — not a + // hard OS suspension — usually drives the disconnect; the expiration + // handler is the backstop when the OS grants less time. + private let backgroundGracePeriod: TimeInterval = 20 + var onShouldReconnect: ((String) -> Void)? var onShouldDisconnect: (() -> Void)? @@ -41,6 +54,14 @@ class ParsecBackgroundManager { } func sceneDidBecomeActive() { + // Returned inside the keep-alive window: the connection was never torn + // down, so cancel the pending disconnect and resume instantly. No + // reconnect — didDisconnectDueToBackground was never set. + if graceTimer != nil { + endBackgroundGrace() + return + } + // Takes priority over isPiPActive check because stopPiP() is async if didDisconnectDueToBackground, let peerId = lastPeerId { didDisconnectDueToBackground = false @@ -52,15 +73,68 @@ class ParsecBackgroundManager { } } - func sceneDidEnterBackground() { - if hasActiveConnection { - var pipAttempted = false - if #available(iOS 15.0, *) { - pipAttempted = isPiPActive || PictureInPictureManager.shared.isStarting - } - if !pipAttempted { - didDisconnectDueToBackground = true - } + // Backgrounded without PiP: open a finite keep-alive window rather than + // dropping the stream immediately. Called from SceneDelegate only when a + // connection is active and PiP was not attempted. + func beginBackgroundGrace() { + guard hasActiveConnection else { return } + // Re-entrancy: a window is already open, leave it running. + guard backgroundTaskId == .invalid else { return } + + backgroundTaskId = UIApplication.shared.beginBackgroundTask(withName: "ParsecKeepAlive") { [weak self] in + // OS is about to suspend us — disconnect now while we still can. + self?.expireBackgroundGrace() + } + + // No background time granted: fall back to the old immediate drop. + if backgroundTaskId == .invalid { + triggerBackgroundDisconnect() + return + } + + graceTimer?.invalidate() + graceTimer = Timer.scheduledTimer(withTimeInterval: backgroundGracePeriod, repeats: false) { [weak self] _ in + self?.expireBackgroundGrace() + } + } + + // Window survived to its end (timer fired or OS reclaimed the time): + // disconnect and let the normal return-to-foreground reconnect take over. + private func expireBackgroundGrace() { + guard backgroundTaskId != .invalid else { return } + graceTimer?.invalidate() + graceTimer = nil + triggerBackgroundDisconnect() + endBackgroundTaskAssertion() + } + + // Window cancelled because the app came back: keep the live connection. + private func endBackgroundGrace() { + graceTimer?.invalidate() + graceTimer = nil + endBackgroundTaskAssertion() + } + + private func triggerBackgroundDisconnect() { + didDisconnectDueToBackground = true + // Tear the SDK session down synchronously *here*. When this runs from + // the UIBackgroundTask expiration handler, iOS may suspend us the + // instant endBackgroundTaskAssertion() releases the assertion, so we + // cannot rely on the async UI-notification path to reach + // CParsec.disconnect() in time — the same suspend hazard the PiP-stop + // path guards against (ParsecView.post). disconnect() also flips + // hasActiveConnection to false via connectionDidEnd(), so a second + // background event can't reopen a window over an already-dropped + // session. The notification below then does the UI teardown (return to + // main view, GL cleanup), which is safe to finish on the next runloop. + CParsec.disconnect() + onShouldDisconnect?() + } + + private func endBackgroundTaskAssertion() { + if backgroundTaskId != .invalid { + UIApplication.shared.endBackgroundTask(backgroundTaskId) + backgroundTaskId = .invalid } } @@ -73,5 +147,7 @@ class ParsecBackgroundManager { didDisconnectDueToBackground = false isReconnecting = false lastPeerId = nil + // An explicit disconnect cancels any open keep-alive window. + endBackgroundGrace() } } diff --git a/OpenParsec/ParsecGLKRenderer.swift b/OpenParsec/ParsecGLKRenderer.swift index 5a76fb6..4d1ea69 100644 --- a/OpenParsec/ParsecGLKRenderer.swift +++ b/OpenParsec/ParsecGLKRenderer.swift @@ -48,7 +48,11 @@ class ParsecGLKRenderer:NSObject, GLKViewDelegate, GLKViewControllerDelegate CParsec.renderGLFrame(timeout: timeout) - if #available(iOS 15.0, *) { + // Only run PiP frame capture while PiP is actually active (or about + // to be). Previously this ran on every frame even with PiP off, + // wasting a glReadPixels stall on the critical render path. + if #available(iOS 15.0, *), + PictureInPictureManager.shared.isPiPActive || PictureInPictureManager.shared.isStarting { PictureInPictureManager.shared.captureFrame( viewWidth: GLsizei(view.drawableWidth), viewHeight: GLsizei(view.drawableHeight), diff --git a/OpenParsec/ParsecGLKViewController.swift b/OpenParsec/ParsecGLKViewController.swift index f610a7c..d6fdcec 100644 --- a/OpenParsec/ParsecGLKViewController.swift +++ b/OpenParsec/ParsecGLKViewController.swift @@ -51,6 +51,11 @@ class ParsecGLKViewController : ParsecPlayground { private func setupGLKViewController() { glkView.context = EAGLContext(api: .openGLES3)! + // Track the superview's bounds so the drawable can't desync / go + // zero-size when the view is moved between parents or the layout + // changes on screen return (R4). updateSize still drives explicit + // resolution changes; this just keeps the surface pinned otherwise. + glkView.autoresizingMask = [.flexibleWidth, .flexibleHeight] glkViewController.view = glkView // Use configured FPS or device max (for ProMotion displays) @@ -71,10 +76,28 @@ class ParsecGLKViewController : ParsecPlayground { return glkView?.context } + // Symmetric stop: pause the CADisplayLink-driven render loop so + // glkView(_:drawIn:) stops being called. Used when the surface is going + // off screen; resume() reverses it. func cleanUp() { + glkViewController.isPaused = true + } + // Idempotent render resume. Safe to call repeatedly. Makes the EAGL + // context current on the main thread (where GLKViewController renders), + // unpauses the loop LAST (Apple's ordering rule), then forces one frame + // so a stale/blank framebuffer repaints immediately instead of waiting + // for the next streamed frame. This is the core fix for the black screen + // on screen return: any path that left isPaused == true (changeResolution, + // PiP, background) is self-healed here. + func resume() { + if let ctx = glkView?.context { + _ = EAGLContext.setCurrent(ctx) + } + glkViewController.isPaused = false + glkView?.setNeedsDisplay() } - + func updateSize(width: CGFloat, height: CGFloat) { glkView.frame.size.width = width glkView.frame.size.height = height diff --git a/OpenParsec/ParsecSDKBridge.swift b/OpenParsec/ParsecSDKBridge.swift index cc0f23b..8d57b83 100644 --- a/OpenParsec/ParsecSDKBridge.swift +++ b/OpenParsec/ParsecSDKBridge.swift @@ -1,6 +1,7 @@ import ParsecSDK import MetalKit import UIKit +import os enum RendererType: Int { @@ -20,6 +21,21 @@ enum CursorMode: Int case direct } +// Which hotkey to fire at the host when the iPad's hardware-keyboard input +// language changes. Default: Ctrl+Space, the macOS built-in "select previous +// input source" shortcut — works if the user has the same two layouts on the +// host as on the iPad, in the same order. For other layout sets / OSes the +// user can pick a different combination in Settings. +enum LayoutSyncHotkey: Int, CaseIterable +{ + case none = 0 + case ctrlSpace = 1 + case cmdSpace = 2 + case altSpace = 3 + case altShift = 4 + case ctrlShift = 5 // ⌃⇧ — another common macOS layout-toggle binding +} + enum RightClickPosition: Int { case firstFinger @@ -46,18 +62,88 @@ class ParsecSDKBridge: ParsecService private let _audioPtr: UnsafeRawPointer private var isVirtualShiftOn = false - + public var clientWidth: Float = 1920 public var clientHeight: Float = 1080 - + public var netProtocol: Int32 = 1 public var mediaContainer: Int32 = 0 public var pngCursor: Bool = false + // Doubles as a gate for outgoing input messages: while false (between an + // explicit disconnect and the next connect, including the brief gap in + // changeResolution), every send* method below early-returns so we don't + // fire ParsecClientSendMessage into a disconnected client. var backgroundTaskRunning = true + // C3 fix: monotonic token bumped on every startBackgroundTask(). Each poll + // loop captures the value at spawn and exits the instant the token moves, + // so a fast disconnect→reconnect (which the 0.02 s drain in disconnect() + // can't guarantee has fully drained) cannot leave two generations of + // audio/event loops running against one client and double-polling + // ParsecGetBuffer / ParsecFree. + private var pollGeneration: Int = 0 var didSetResolution = false - - public var mouseInfo = MouseInfo() - + // Restored once per session in handleUserDataEvent case 12 so display + // hot-plug / sleep-wake echoes don't keep re-firing updateHostVideoConfig + // and causing momentary re-encode flicker. + var didRestoreSavedDisplay = false + + // C1 fix: `mouseInfo` is written on the poll thread (handleCursorEvent), + // on the input paths (sendMousePosition / setFrame), and read on the main + // thread (updateImage). It holds a `CGImage?` (`cursorImg`) — an ARC + // reference. A struct copy in the getter retains `cursorImg` while a + // concurrent write releases it; that non-atomic retain/release races and + // over-releases the CGImage → use-after-free crash that fires constantly + // during cursor motion. All access now goes through `os_unfair_lock`: + // readers take an atomic snapshot via the `mouseInfo` getter, writers + // mutate under the same lock via `withMouseInfo`. + private var _mouseInfo = MouseInfo() + private var mouseInfoLock = os_unfair_lock_s() + + // S04: raw host-OS int mirror, written from the case-11 main block and read + // lock-free from input threads (plain Int load is atomic on-device). Reset + // to -1 on connect AND disconnect so a stale value can't bleed across a + // host switch. -1 = unknown / not yet received. + var hostOSValue: Int = -1 + + // S06: monotonic token bumped per updateHostVideoConfig() so a superseded + // resend/echo-request from an earlier call drops instead of stacking timers + // and re-confirming a stale value. Plus the user-owned fields awaiting host + // confirmation: while set, a case-11 echo CONFIRMS (clears) on match but + // must never clobber the user's in-flight bitrate/fps selection with a + // stale echoed value — that was the "checkmark jumps back" bug. Touched on + // the main thread (all callers are main); the resend closures only read the + // token (benign plain-Int read, same pattern as backgroundTaskRunning). + private var configRevision: Int = 0 + private var pendingUserConfig: (bitrate: Int, constantFps: Bool)? = nil + // The display id a switch is still trying to land. A single fire-and-forget + // setVideoConfig can be dropped while the host encoder is mid-reset from the + // switch — the reason a display change historically needed two or three taps + // (and why "select the current monitor, then the target" worked: it spaced + // two sends by a human delay). We keep re-asserting this output on a widening + // schedule until the case-11 echo reports it as the host's current output, or + // we hit the attempt cap. Cleared on the main thread; the retry closures read + // it lock-free (same benign-plain-read pattern as pendingUserConfig). + private var pendingOutput: String? = nil + + // Atomic snapshot for cross-thread readers. Returns a consistent copy of + // the whole struct under the lock so `cursorImg`'s retain happens while no + // writer can release it. + var mouseInfo: MouseInfo { + os_unfair_lock_lock(&mouseInfoLock) + defer { os_unfair_lock_unlock(&mouseInfoLock) } + return _mouseInfo + } + + // Serialize every mutation under the same lock the snapshot getter uses. + // Keep the body short — never do heavy work (e.g. CGImage construction) + // while holding the lock; build first, then assign inside. + @discardableResult + private func withMouseInfo(_ body: (inout MouseInfo) -> T) -> T { + os_unfair_lock_lock(&mouseInfoLock) + defer { os_unfair_lock_unlock(&mouseInfoLock) } + return body(&_mouseInfo) + } + init() { print("Parsec SDK Version: " + String(ParsecSDKBridge.PARSEC_VER)) @@ -92,6 +178,24 @@ class ParsecSDKBridge: ParsecService } func connect(_ peerID: String) -> ParsecStatus { + // CRITICAL: disconnect() set this to false to drain the poll loops. + // If we don't flip it back to true here, the new poll loops spawned + // by startBackgroundTask() below will read false on their first + // iteration and exit immediately — leaving the session with no audio + // callbacks, no cursor updates, and no user-data events for the rest + // of its lifetime. Also acts as the "sending allowed" gate for input + // messages, so flip it before any input could possibly fire. + backgroundTaskRunning = true + // Every fresh connect() should attempt to restore the saved display + // when case-12 arrives. Resetting here (not only in disconnect()) is + // load-bearing — common reconnect paths (alert dismiss → reconnect, + // background → resume) skip disconnect() entirely, so without this + // the flag stayed `true` from a prior session and the restore was + // silently never re-run. + didRestoreSavedDisplay = false + didSetResolution = false + hostOSValue = -1 + DispatchQueue.main.async { DataManager.model.hostOS = -1 } var parsecClientCfg = ParsecClientConfig() parsecClientCfg.video.0.decoderIndex = 1 @@ -124,11 +228,26 @@ class ParsecSDKBridge: ParsecService } func disconnect() { - + audio_clear(&_audio) ParsecClientDisconnect(_parsec) backgroundTaskRunning = false - + // Reset so the next case-11 echo after a reconnect re-pushes our + // desired resolution instead of clobbering it with whatever the host + // happens to advertise. + didSetResolution = false + didRestoreSavedDisplay = false + hostOSValue = -1 + DispatchQueue.main.async { DataManager.model.hostOS = -1 } + + // Give the two `while backgroundTaskRunning` loops in + // startBackgroundTask() one full poll-timeout to notice the flag + // flip and exit. Without this drain, a fast reconnect() can spawn + // fresh loops while the old ones are still inside ParsecClientPollAudio + // / ParsecClientPollEvents, briefly doubling the poll rate and + // causing audio glitches. + Thread.sleep(forTimeInterval: 0.02) + ParsecBackgroundManager.shared.connectionDidEnd() } @@ -151,8 +270,10 @@ class ParsecSDKBridge: ParsecService clientWidth = Float(width) clientHeight = Float(height) - mouseInfo.mouseX = Int32(width / 2) - mouseInfo.mouseY = Int32(height / 2) + withMouseInfo { + $0.mouseX = Int32(width / 2) + $0.mouseY = Int32(height / 2) + } } // timeout in ms, 16 == 60 FPS, 8 == 120 FPS, etc. @@ -204,14 +325,56 @@ class ParsecSDKBridge: ParsecService let videoConfig = config.video[0] DispatchQueue.main.async { + // resolutionX/Y are host-authoritative — always adopt. DataManager.model.resolutionX = videoConfig.resolutionX DataManager.model.resolutionY = videoConfig.resolutionY - DataManager.model.bitrate = videoConfig.encoderMaxBitrate - DataManager.model.constantFps = videoConfig.fullFPS + // S06: bitrate/constantFps are user-owned. If a user change + // is in flight, the echo only CONFIRMS (clears pending on a + // match) — it must not overwrite the user's selection with a + // stale value, which made the menu checkmark jump back. With + // nothing pending, adopt the host's reported values. + if let pending = self.pendingUserConfig { + if videoConfig.encoderMaxBitrate == pending.bitrate + && videoConfig.fullFPS == pending.constantFps { + self.pendingUserConfig = nil + } + } else { + DataManager.model.bitrate = videoConfig.encoderMaxBitrate + DataManager.model.constantFps = videoConfig.fullFPS + } + // Confirm a display switch: once the host echoes the output we + // asked for, stop re-asserting it. We only READ the echoed + // output here (never write it back into model.output) so a + // host that omits/normalizes the field can't clobber the + // user's selection — same caution as the bitrate guard above. + if let pendingOut = self.pendingOutput, videoConfig.output == pendingOut { + self.pendingOutput = nil + } + // S04: capture the host-OS int and log it once per session + // change so its undocumented encoding can be discovered + // against known Mac/Windows hosts. + if self.hostOSValue != videoConfig.hostOS { + self.hostOSValue = videoConfig.hostOS + DataManager.model.hostOS = videoConfig.hostOS + Diagnostics.note("hostOS=\(videoConfig.hostOS) (resolution=\(videoConfig.resolutionX)x\(videoConfig.resolutionY), bitrate=\(videoConfig.encoderMaxBitrate))") + } if !self.didSetResolution { self.didSetResolution = true DataManager.model.resolutionX = SettingsHandler.resolution.width DataManager.model.resolutionY = SettingsHandler.resolution.height + // S08: apply the poor-network bitrate cap on the same + // first echo that pushes the saved resolution. Bitrate + // only reaches the host via updateHostVideoConfig (the + // connect-time SDK config carries no encoderMaxBitrate), + // and the case-11 branch above otherwise adopts the + // host's bitrate — so without this the profile's cap + // would never take effect. Riding the existing one-shot + // push avoids a separate timer/race and leaves `output` + // untouched, so it can't clobber the restored display. + if SettingsHandler.lowLatencyMode && SettingsHandler.bitrate > 0 { + DataManager.model.bitrate = SettingsHandler.bitrate + Diagnostics.note("S08 low-latency: capping bitrate to \(SettingsHandler.bitrate) Mbps (host reported \(videoConfig.encoderMaxBitrate))") + } self.updateHostVideoConfig() } } @@ -225,6 +388,29 @@ class ParsecSDKBridge: ParsecService let config = try decoder.decode(Array.self, from: Data(bytesNoCopy: pointer!, count: strlen(pointer!), deallocator: .none)) DispatchQueue.main.async { DataManager.model.displayConfigs = config + // Restore the saved display ONCE per session. The host + // can re-advertise its display list multiple times mid- + // stream (sleep/wake, display hot-plug); without this + // gate, every echo would re-fire updateHostVideoConfig + // and cause a brief re-encode flicker. + if !self.didRestoreSavedDisplay { + self.didRestoreSavedDisplay = true + let savedId = SettingsHandler.savedDisplayOutput + let savedName = SettingsHandler.savedDisplayName + guard !savedId.isEmpty || !savedName.isEmpty else { return } + // Match by id first (stable when the host reports + // consistent display ids across sessions). Fall + // back to name+adapter match so a display that + // changed id between connects (Parsec sometimes + // regenerates them) is still found. + let match = config.first(where: { $0.id == savedId }) + ?? config.first(where: { !savedName.isEmpty && "\($0.name) \($0.adapterName)" == savedName }) + if let match = match { + DataManager.model.output = match.id + SettingsHandler.savedDisplayOutput = match.id // re-sync if id rolled + self.updateHostVideoConfig() + } + } } } catch { print("error while parsing user data: \(error.localizedDescription)") @@ -238,41 +424,49 @@ class ParsecSDKBridge: ParsecService } func handleCursorEvent(event: ParsecClientCursorEvent) { - let prevHidden = mouseInfo.cursorHidden - mouseInfo.cursorHidden = event.cursor.hidden - mouseInfo.mousePositionRelative = event.cursor.relative - - if event.cursor.imageUpdate || !getFirstCursor{ - getFirstCursor = true - let imgKey = event.key - let pointer = ParsecGetBuffer(_parsec, imgKey) - if pointer == nil{ - return - } - let size = event.cursor.size - let width = event.cursor.width - let height = event.cursor.height - mouseInfo.cursorWidth = Int(width) - mouseInfo.cursorHeight = Int(height) - + // hidden / relative always track the latest event; capture the prior + // hidden state in the same locked section to decide the reposition. + let prevHidden = withMouseInfo { info -> Bool in + let prev = info.cursorHidden + info.cursorHidden = event.cursor.hidden + info.mousePositionRelative = event.cursor.relative + return prev + } + + guard event.cursor.imageUpdate || !getFirstCursor else { return } + getFirstCursor = true + + let pointer = ParsecGetBuffer(_parsec, event.key) + if pointer == nil { + return + } + + let size = event.cursor.size + let width = event.cursor.width + let height = event.cursor.height + + // Build the CGImage BEFORE taking the lock — image construction is the + // expensive part and must not run while the snapshot getter is blocked. + let elmentLength: Int = 4 + let render: CGColorRenderingIntent = CGColorRenderingIntent.defaultIntent + let rgbColorSpace = CGColorSpaceCreateDeviceRGB() + let bitmapInfo: CGBitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue) + let providerRef: CGDataProvider? = CGDataProvider(data: NSData(bytes: pointer, length: Int(size))) + let cgimage: CGImage? = CGImage(width: Int(width), height: Int(height), bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: Int(width) * elmentLength, space: rgbColorSpace, bitmapInfo: bitmapInfo, provider: providerRef!, decode: nil, shouldInterpolate: true, intent: render) + ParsecFree(pointer) + + withMouseInfo { info in + info.cursorWidth = Int(width) + info.cursorHeight = Int(height) if prevHidden && !event.cursor.hidden { - mouseInfo.mouseX = Int32(event.cursor.positionX) - mouseInfo.mouseY = Int32(event.cursor.positionY) + info.mouseX = Int32(event.cursor.positionX) + info.mouseY = Int32(event.cursor.positionY) } - - mouseInfo.cursorHotX = Int(event.cursor.hotX) - mouseInfo.cursorHotY = Int(event.cursor.hotY) - - let elmentLength: Int = 4 - let render: CGColorRenderingIntent = CGColorRenderingIntent.defaultIntent - let rgbColorSpace = CGColorSpaceCreateDeviceRGB() - let bitmapInfo: CGBitmapInfo = CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue) - let providerRef: CGDataProvider? = CGDataProvider(data: NSData(bytes: pointer, length: Int(size))) - let cgimage: CGImage? = CGImage(width: Int(width), height: Int(height), bitsPerComponent: 8, bitsPerPixel: 32, bytesPerRow: Int(width) * elmentLength, space: rgbColorSpace, bitmapInfo: bitmapInfo, provider: providerRef!, decode: nil, shouldInterpolate: true, intent: render) - if cgimage != nil { - mouseInfo.cursorImg = cgimage + info.cursorHotX = Int(event.cursor.hotX) + info.cursorHotY = Int(event.cursor.hotY) + if let cgimage = cgimage { + info.cursorImg = cgimage } - ParsecFree(pointer) } } @@ -284,15 +478,19 @@ class ParsecSDKBridge: ParsecService var parsecClientCfg = ParsecClientConfig() + // Preserve the user's chosen resolution. The previous code hardcoded + // 0/0 (= "use host default"), which silently overwrote whatever + // connect() had set, making it look like the in-overlay Resolution + // picker did nothing. parsecClientCfg.video.0.decoderIndex = 1 - parsecClientCfg.video.0.resolutionX = 0 - parsecClientCfg.video.0.resolutionY = 0 + parsecClientCfg.video.0.resolutionX = Int32(SettingsHandler.resolution.width) + parsecClientCfg.video.0.resolutionY = Int32(SettingsHandler.resolution.height) parsecClientCfg.video.0.decoderCompatibility = SettingsHandler.decoderCompatibility parsecClientCfg.video.0.decoderH265 = SettingsHandler.decoder == .h265 parsecClientCfg.video.1.decoderIndex = 1 - parsecClientCfg.video.1.resolutionX = 0 - parsecClientCfg.video.1.resolutionY = 0 + parsecClientCfg.video.1.resolutionX = Int32(SettingsHandler.resolution.width) + parsecClientCfg.video.1.resolutionY = Int32(SettingsHandler.resolution.height) parsecClientCfg.video.1.decoderCompatibility = SettingsHandler.decoderCompatibility parsecClientCfg.video.1.decoderH265 = SettingsHandler.decoder == .h265 @@ -304,11 +502,18 @@ class ParsecSDKBridge: ParsecService ParsecClientSetConfig(_parsec, &parsecClientCfg); } + // All outgoing-input methods gate on backgroundTaskRunning. False means + // we're between an explicit disconnect and the next connect (including + // the gap inside changeResolution's reconnect dance) — sending into a + // disconnected SDK is at best a wasted message and at worst a NULL deref + // inside ParsecClientSendMessage. + func sendMouseMessage(_ button:ParsecMouseButton, _ x:Int32, _ y:Int32, _ pressed: Bool) { + guard backgroundTaskRunning else { return } // Send the mouse position sendMousePosition(x, y) - + // Send the mouse button state var buttonMessage = ParsecMessage() buttonMessage.type = MESSAGE_MOUSE_BUTTON @@ -316,40 +521,69 @@ class ParsecSDKBridge: ParsecService buttonMessage.mouseButton.pressed = pressed ParsecClientSendMessage(_parsec, &buttonMessage) } - + func sendMouseClickMessage(_ button:ParsecMouseButton, _ pressed: Bool) { + guard backgroundTaskRunning else { return } var buttonMessage = ParsecMessage() buttonMessage.type = MESSAGE_MOUSE_BUTTON buttonMessage.mouseButton.button = button buttonMessage.mouseButton.pressed = pressed ParsecClientSendMessage(_parsec, &buttonMessage) } - + func sendMouseDelta(_ dx: Int32, _ dy: Int32) { - if mouseInfo.mousePositionRelative { + guard backgroundTaskRunning else { return } + // One atomic snapshot, then act on it — avoids two separate locked + // reads racing a concurrent position write. + let info = mouseInfo + if info.mousePositionRelative { sendMouseRelativeMove(dx, dy) } else { - sendMousePosition(mouseInfo.mouseX + dx, mouseInfo.mouseY + dy) + sendMousePosition(info.mouseX + dx, info.mouseY + dy) } - + } static func clamp(_ value: T, minValue: T, maxValue: T) -> T where T : Comparable { return min(max(value, minValue), maxValue) } + + // Swap GUI ↔ Ctrl scan codes when the user has flagged the host as + // Windows. Mac-keyboard layout calls the modifier-row keys, left-to-right, + // Control / Option / Cmd. On a Windows host the equivalents are + // Ctrl / Alt / Win — but Win+C does nothing useful and Ctrl+C is copy. + // Remapping at the scan-code layer means every consumer (UIKey path, + // virtual keyboard, UIKeyCommand captured shortcuts) gets the swap with + // no per-caller awareness. + static func remapKeyForHostIfNeeded(_ code: ParsecKeycode) -> ParsecKeycode { + guard SettingsHandler.windowsHostKeyboardRemap else { return code } + switch code.rawValue { + case 227: return ParsecKeycode(224) // LGUI → LCTRL (Cmd → Ctrl) + case 224: return ParsecKeycode(227) // LCTRL → LGUI (Ctrl → Win) + case 231: return ParsecKeycode(228) // RGUI → RCTRL + case 228: return ParsecKeycode(231) // RCTRL → RGUI + default: return code // Shift / Alt / printable keys unchanged + } + } func sendMousePosition(_ x:Int32, _ y:Int32) { - mouseInfo.mouseX = ParsecSDKBridge.clamp(x, minValue: 0, maxValue: Int32(self.clientWidth)) - mouseInfo.mouseY = ParsecSDKBridge.clamp(y, minValue: 0, maxValue: Int32(self.clientHeight)) + guard backgroundTaskRunning else { return } + let cx = ParsecSDKBridge.clamp(x, minValue: 0, maxValue: Int32(self.clientWidth)) + let cy = ParsecSDKBridge.clamp(y, minValue: 0, maxValue: Int32(self.clientHeight)) + withMouseInfo { + $0.mouseX = cx + $0.mouseY = cy + } var motionMessage = ParsecMessage() motionMessage.type = MESSAGE_MOUSE_MOTION motionMessage.mouseMotion.x = x motionMessage.mouseMotion.y = y ParsecClientSendMessage(_parsec, &motionMessage) } - + func sendMouseRelativeMove(_ dx:Int32, _ dy:Int32) { + guard backgroundTaskRunning else { return } var motionMessage = ParsecMessage() motionMessage.type = MESSAGE_MOUSE_MOTION motionMessage.mouseMotion.x = dx @@ -389,23 +623,30 @@ class ParsecSDKBridge: ParsecService } func sendVirtualKeyboardInput(text: String) { + guard backgroundTaskRunning else { return } let (keyCode, useShift) = getKeyCodeByText(text: text) - + guard let keyCode else { return } + let remapped = ParsecSDKBridge.remapKeyForHostIfNeeded(keyCode) var keyboardMessagePress = ParsecMessage() keyboardMessagePress.type = MESSAGE_KEYBOARD if !isVirtualShiftOn && useShift { keyboardMessagePress.keyboard = ParsecKeyboardMessage(code: KEY_LSHIFT, mod: MOD_NONE, pressed: true, __pad: (0,0,0)) ParsecClientSendMessage(_parsec, &keyboardMessagePress) } - keyboardMessagePress.keyboard = ParsecKeyboardMessage(code: keyCode, mod: MOD_NONE, pressed: true, __pad: (0,0,0)) + keyboardMessagePress.keyboard = ParsecKeyboardMessage(code: remapped, mod: MOD_NONE, pressed: true, __pad: (0,0,0)) ParsecClientSendMessage(_parsec, &keyboardMessagePress) - + // add release delay in case some games ignore instant key release DispatchQueue.global().asyncAfter(deadline: .now() + 0.02) { - keyboardMessagePress.keyboard = ParsecKeyboardMessage(code: keyCode, mod: MOD_NONE, pressed: false, __pad: (0,0,0)) + // Re-check the gate inside the closure: a disconnect can land in + // these 20 ms (matches the drain sleep in disconnect() exactly), + // in which case we would otherwise fire ParsecClientSendMessage + // against a torn-down client. + guard self.backgroundTaskRunning else { return } + keyboardMessagePress.keyboard = ParsecKeyboardMessage(code: remapped, mod: MOD_NONE, pressed: false, __pad: (0,0,0)) if !self.isVirtualShiftOn && useShift { keyboardMessagePress.keyboard = ParsecKeyboardMessage(code: KEY_LSHIFT, mod: MOD_NONE, pressed: false, __pad: (0,0,0)) } @@ -414,39 +655,44 @@ class ParsecSDKBridge: ParsecService } func sendVirtualKeyboardInput(text: String, isOn: Bool) { + guard backgroundTaskRunning else { return } let (keyCode, _) = getKeyCodeByText(text: text) - + guard let keyCode else { return } - + if keyCode.rawValue == 225 { isVirtualShiftOn = isOn } - + + let remapped = ParsecSDKBridge.remapKeyForHostIfNeeded(keyCode) var keyboardMessagePress = ParsecMessage() keyboardMessagePress.type = MESSAGE_KEYBOARD keyboardMessagePress.keyboard.pressed = isOn - keyboardMessagePress.keyboard.code = keyCode + keyboardMessagePress.keyboard.code = remapped ParsecClientSendMessage(_parsec, &keyboardMessagePress) - + } func sendKeyboardMessage(event:KeyBoardKeyEvent) { + guard backgroundTaskRunning else { return } if event.input == nil { return } - + + let rawCode = ParsecKeycode(UInt32(KeyCodeTranslators.uiKeyCodeToInt(key: event.input?.keyCode ?? UIKeyboardHIDUsage.keyboardErrorUndefined))) var keyboardMessagePress = ParsecMessage() keyboardMessagePress.type = MESSAGE_KEYBOARD - keyboardMessagePress.keyboard.code = ParsecKeycode(UInt32(KeyCodeTranslators.uiKeyCodeToInt(key: event.input?.keyCode ?? UIKeyboardHIDUsage.keyboardErrorUndefined))) + keyboardMessagePress.keyboard.code = ParsecSDKBridge.remapKeyForHostIfNeeded(rawCode) keyboardMessagePress.keyboard.pressed = event.isPressBegin ParsecClientSendMessage(_parsec, &keyboardMessagePress) } func sendGameControllerButtonMessage(controllerId: UInt32, _ button:ParsecGamepadButton, pressed: Bool) { + guard backgroundTaskRunning else { return } var pmsg = ParsecMessage() pmsg.type = MESSAGE_GAMEPAD_BUTTON pmsg.gamepadButton.id = controllerId @@ -467,6 +713,7 @@ class ParsecSDKBridge: ParsecService func sendGameControllerAxisMessage(controllerId: UInt32, _ button:ParsecGamepadAxis, _ value: Int16) { + guard backgroundTaskRunning else { return } var pmsg = ParsecMessage() pmsg.type = MESSAGE_GAMEPAD_AXIS pmsg.gamepadAxis.id = controllerId @@ -477,6 +724,7 @@ class ParsecSDKBridge: ParsecService func sendGameControllerUnplugMessage(controllerId: UInt32) { + guard backgroundTaskRunning else { return } var pmsg = ParsecMessage() pmsg.type = MESSAGE_GAMEPAD_UNPLUG; pmsg.gamepadUnplug.id = controllerId; @@ -484,6 +732,7 @@ class ParsecSDKBridge: ParsecService } func sendWheelMsg(x: Int32, y: Int32) { + guard backgroundTaskRunning else { return } var pmsg = ParsecMessage() pmsg.type = MESSAGE_MOUSE_WHEEL; pmsg.mouseWheel.x = x @@ -492,29 +741,44 @@ class ParsecSDKBridge: ParsecService } func startBackgroundTask(){ - - + // Scale poll timeout to the configured render fps so the audio and + // event threads don't sit blocked inside the SDK longer than a + // frame budget. On 120 Hz iPads that's 8 ms; on 60 Hz, 16 ms. + let fps = SettingsHandler.preferredFramesPerSecond == 0 + ? UIScreen.main.maximumFramesPerSecond + : SettingsHandler.preferredFramesPerSecond + let pollTimeout = UInt32(max(1000 / fps, 8)) + + // Advance the generation and capture it for this pair of loops. Any + // previously-spawned loop sees the bumped value and exits. + pollGeneration &+= 1 + let generation = pollGeneration + let item1 = DispatchWorkItem { - while self.backgroundTaskRunning { - self.pollAudio() + while self.backgroundTaskRunning && self.pollGeneration == generation { + self.pollAudio(timeout: pollTimeout) } - } let item2 = DispatchWorkItem { - while self.backgroundTaskRunning { - self.pollEvent() - - + while self.backgroundTaskRunning && self.pollGeneration == generation { + self.pollEvent(timeout: pollTimeout) } - } - let mainQueue = DispatchQueue.global() - mainQueue.async(execute: item1) - mainQueue.async(execute: item2) + // .userInteractive is the right QoS for remote-desktop input/event + // dispatch — these threads gate audio callbacks and cursor updates. + // Previously used unspecified (.default) which sometimes coalesces + // under system load. + let pollQueue = DispatchQueue.global(qos: .userInteractive) + pollQueue.async(execute: item1) + pollQueue.async(execute: item2) } func sendUserData(type: ParsecUserDataType, message: Data) { + // V2: the only send* method that was missing this gate — protects + // against a NULL deref inside ParsecClientSendUserData during the + // disconnect→reconnect gap, consistent with every other send path. + guard backgroundTaskRunning else { return } var nullTerminatedMessage = message nullTerminatedMessage.append(0) nullTerminatedMessage.withUnsafeBytes { ptr in @@ -532,6 +796,84 @@ class ParsecSDKBridge: ParsecService videoConfig.video[0].output = DataManager.model.output let encoder = JSONEncoder() let data = try! encoder.encode(videoConfig) + + // S06: bump the revision so any in-flight resend from a previous call + // drops, and record the user-owned fields so the case-11 echo confirms + // (rather than reverts) this selection. + configRevision &+= 1 + let revision = configRevision + pendingUserConfig = (DataManager.model.bitrate, DataManager.model.constantFps) + // Track a concrete display switch so the confirm-and-retry loop below can + // keep re-asserting it. "none" (Auto) has no stable echoed id to confirm + // against, so it just rides the base send + the existing 250 ms resend. + let switchTarget = DataManager.model.output + if switchTarget != "none" && !switchTarget.isEmpty { + pendingOutput = switchTarget + scheduleDisplaySwitchRetry(revision: revision, data: data, attempt: 0) + } else { + pendingOutput = nil + } + CParsec.sendUserData(type: .setVideoConfig, message: data) + // User reports: display switches needed two or three taps to actually + // take effect. setVideoConfig is fire-and-forget; the host can drop + // the message if its encoder is in the middle of a reset triggered + // by a previous request. Re-send the same payload after 250 ms + // (idempotent — same output reapplied is a no-op on the host), but only + // if this call is still the latest (a newer tap supersedes it). + // Then ask the host to echo back its current config so case-11 + // confirms the switch landed. + DispatchQueue.global().asyncAfter(deadline: .now() + 0.25) { [weak self] in + guard let self = self, self.backgroundTaskRunning, self.configRevision == revision else { return } + CParsec.sendUserData(type: .setVideoConfig, message: data) + } + DispatchQueue.global().asyncAfter(deadline: .now() + 0.45) { [weak self] in + guard let self = self, self.backgroundTaskRunning, self.configRevision == revision else { return } + let empty = "".data(using: .utf8)! + CParsec.sendUserData(type: .getVideoConfig, message: empty) + } + // Self-heal: drop the pending guard after a confirmation window so a + // host that never echoes our exact value can't permanently block future + // echo adoption of bitrate/fps. Only clears if still the latest call. + DispatchQueue.main.asyncAfter(deadline: .now() + 1.5) { [weak self] in + guard let self = self, self.configRevision == revision else { return } + self.pendingUserConfig = nil + } + } + + // Re-assert a display switch on a widening schedule until the host's case-11 + // echo confirms it (which clears pendingOutput) or we exhaust the attempts. + // A display change re-inits the host encoder for the new output; a request + // that lands during that reset is dropped, so one fire-and-forget send (or a + // single 250 ms resend, which fires while the host is often still busy) was + // unreliable — hence the historical "needs two or three taps". Each resend is + // idempotent (same output reapplied is a no-op once it's the active display), + // so over-asserting is harmless; under-asserting was the bug. + private func scheduleDisplaySwitchRetry(revision: Int, data: Data, attempt: Int) { + let delays: [Double] = [0.35, 0.8, 1.5, 2.5] + guard attempt < delays.count else { + // Give up re-asserting; drop the guard so a host that never echoes a + // matching output can't leave it pending forever. + DispatchQueue.main.async { [weak self] in + guard let self = self, self.configRevision == revision else { return } + self.pendingOutput = nil + } + return + } + DispatchQueue.global().asyncAfter(deadline: .now() + delays[attempt]) { [weak self] in + guard let self = self, self.backgroundTaskRunning, self.configRevision == revision else { return } + // pendingOutput is mutated on the main queue (set in updateHostVideoConfig, + // cleared by the case-11 confirmation echo). It's a String?, not an atomic + // scalar, so read it on main to avoid racing that clear. Re-assert there if + // the switch is still unconfirmed — sending on main is fine (the base send + // at the top of updateHostVideoConfig already does). + DispatchQueue.main.async { + guard self.backgroundTaskRunning, self.configRevision == revision, self.pendingOutput != nil else { return } + CParsec.sendUserData(type: .setVideoConfig, message: data) + let empty = "".data(using: .utf8)! + CParsec.sendUserData(type: .getVideoConfig, message: empty) + self.scheduleDisplaySwitchRetry(revision: revision, data: data, attempt: attempt + 1) + } + } } } diff --git a/OpenParsec/ParsecView.swift b/OpenParsec/ParsecView.swift index ed17bb9..40365a1 100644 --- a/OpenParsec/ParsecView.swift +++ b/OpenParsec/ParsecView.swift @@ -4,15 +4,40 @@ import Foundation import AVFoundation struct ParsecStatusBar : View { + @Binding var isReconfiguring : Bool @Binding var showMenu : Bool @State var metricInfo: String = "Loading..." @Binding var showDCAlert: Bool @Binding var DCAlertText: String @State var parsecViewController: ParsecViewController? @State var wasDisconnected: Bool = true + // Consecutive non-OK polls before we believe the session is really gone. + // At a 0.2s poll interval, 5 ≈ 1s of grace — long enough to ride out a + // transient loss/RTT spike on a jittery link (BUD recovers on its own), + // short enough that a genuine drop still surfaces promptly. + @State var consecutiveFailures: Int = 0 + private let disconnectFailureThreshold = 5 let timer = Timer.publish(every: 0.2, on: .main, in: .common).autoconnect() - init(showMenu: Binding, showDCAlert: Binding, DCAlertText: Binding, parsecViewController: ParsecViewController) { + // Translate a raw ParsecStatus error into something a user can act on. The + // SDK returns bare integers; the worst case is a session that drops with an + // opaque "-6023" the user can't interpret. Known codes get a plain-language + // reason; the raw code is always appended so we can still triage anything new. + static func disconnectMessage(forCode code: Int) -> String { + switch code { + case -6023, -6024: + // Parsec: "Unable To Negotiate A Successful Connection" — a network/NAT + // problem, not an app bug. The iPad and host couldn't open a P2P tunnel. + // Only these two are documented; everything else shows the raw code so + // we never present an unverified reason as fact. + return "Disconnected: couldn't reach the host (network/NAT). Check Wi-Fi, firewall, and the host's UPnP / port-forwarding.\n(code \(code))" + default: + return "Disconnected (code \(code))" + } + } + + init(isReconfiguring: Binding, showMenu: Binding, showDCAlert: Binding, DCAlertText: Binding, parsecViewController: ParsecViewController) { + _isReconfiguring = isReconfiguring _showMenu = showMenu _showDCAlert = showDCAlert _DCAlertText = DCAlertText @@ -43,10 +68,42 @@ struct ParsecStatusBar : View { .onReceive(timer) { p in poll() } + + // Reconnecting overlay — only visible during changeResolution's + // disconnect→reconnect dance. Mirrors MainView's connecting overlay. + if isReconfiguring { + ZStack { + Rectangle() + .fill(Color.black.opacity(0.45)) + .edgesIgnoringSafeArea(.all) + VStack(spacing: 12) { + ProgressView() + .scaleEffect(1.4) + .progressViewStyle(CircularProgressViewStyle(tint: .white)) + Text("Switching resolution…") + .foregroundColor(.white) + .font(.system(size: 16, weight: .medium)) + } + .padding(24) + .background( + RoundedRectangle(cornerRadius: 12) + .fill(Color("BackgroundPrompt").opacity(0.85)) + ) + } + .zIndex(3) + } } - + func poll() { + // While we're deliberately disconnecting/reconnecting for a resolution + // change, getStatusEx will briefly report a non-OK status. Don't pop + // the "Disconnected" alert during that window — it's an intentional + // gap with an overlay in front of it. + if isReconfiguring + { + return + } if showDCAlert { return // no need to poll if we aren't connected anymore @@ -76,12 +133,22 @@ struct ParsecStatusBar : View { return } - wasDisconnected = true - DCAlertText = "Disconnected (code \(status.rawValue))" - showDCAlert = true + // Debounce transient loss: a single bad poll on a jittery link is + // usually the BUD transport mid-recovery, not a real disconnect. + // Only alert after the status has been non-OK for several polls in + // a row; a genuine drop persists and still surfaces within ~1s. + consecutiveFailures += 1 + if consecutiveFailures >= disconnectFailureThreshold { + wasDisconnected = true + DCAlertText = Self.disconnectMessage(forCode: Int(status.rawValue)) + showDCAlert = true + } return } - + + // Status is OK — clear any in-flight failure streak. + consecutiveFailures = 0 + if showMenu { let str = String.fromBuffer(&pcs.decoder.0.name.0, length:16) @@ -113,6 +180,10 @@ struct ParsecView: View @State var showKeyboard: Bool = false @State var zoomEnabled: Bool = false + // True while changeResolution is in its disconnect→reconnect dance. + // Suppresses the status-bar disconnect alert and shows a small + // "Switching resolution…" overlay so the user knows what's happening. + @State var isReconfiguring: Bool = false @State var muted: Bool = false @State var preferH265: Bool = true @@ -162,7 +233,7 @@ struct ParsecView: View .zIndex(1) .prefersPersistentSystemOverlaysHidden() - ParsecStatusBar(showMenu: $showMenu, showDCAlert: $showDCAlert, DCAlertText: $DCAlertText, parsecViewController: parsecViewController) + ParsecStatusBar(isReconfiguring: $isReconfiguring, showMenu: $showMenu, showDCAlert: $showDCAlert, DCAlertText: $DCAlertText, parsecViewController: parsecViewController) VStack() { @@ -234,13 +305,21 @@ struct ParsecView: View } Menu() { ForEach(resolutions, id: \.self) { resolution in + let isCurrent = resolution.width == dataModel.resolutionX && resolution.height == dataModel.resolutionY Button(action: { changeResolution(res: resolution) }) { - if resolution.width == dataModel.resolutionX && resolution.height == dataModel.resolutionY { - Label(resolution.desc, systemImage: "checkmark") - } else { + // iOS 14.0–14.4's UIMenu bridge crashes when SwiftUI lowers a + // `if Label else Text` body inside a Menu's Button to + // `_ConditionalContent`. A single homogeneous + // HStack with a conditional Image is safe and renders the + // same checkmark visual. + HStack { Text(resolution.desc) + if isCurrent { + Spacer(minLength: 8) + Image(systemName: "checkmark") + } } } } @@ -252,14 +331,17 @@ struct ParsecView: View } Menu() { ForEach(bitrates, id: \.self) { bitrate in + let isCurrent = bitrate == dataModel.bitrate Button(action: { changeBitRate(bitrate: bitrate) }) { - if bitrate == dataModel.bitrate { - Label("\(bitrate) Mbps", systemImage: "checkmark") - } else { - Text("\(bitrate) Mbps") - } + HStack { + Text("\(bitrate) Mbps") + if isCurrent { + Spacer(minLength: 8) + Image(systemName: "checkmark") + } + } } } } label: { @@ -471,11 +553,73 @@ struct ParsecView: View } func changeResolution(res: ParsecResolution) { - SettingsHandler.resolution = res + // Guard against spam-tap: if a previous changeResolution is still in + // its disconnect→reconnect dance, ignore further selections until it + // finishes. Otherwise two overlapping connect/disconnect cycles can + // race the SDK state machine. + if isReconfiguring { return } + + SettingsHandler.resolution = res DispatchQueue.main.async { DataManager.model.resolutionX = res.width DataManager.model.resolutionY = res.height - CParsec.updateHostVideoConfig() + } + + // Parsec's host honours bitrate / FPS / output via setVideoConfig + // user-data, but NOT resolution — that field is only read at + // ParsecClientConnect time. To actually change the streaming + // resolution we have to disconnect + reconnect with the new + // ParsecClientConfig. If we don't have a peer to reconnect to + // (shouldn't happen mid-session), fall back to just pushing the + // user-data update. + guard let peerID = CParsec.lastConnectedPeerID else { + DispatchQueue.main.async { + CParsec.updateHostVideoConfig() + } + return + } + + DispatchQueue.main.async { + // Suppress disconnect alert + show the "Switching resolution…" + // overlay during the gap. + self.isReconfiguring = true + // Freeze the last decoded frame on screen instead of going black: + // pausing the GLKViewController stops `glkView(_:drawIn:)` from + // being called, so the framebuffer keeps its current contents. + if let parsecGLK = self.parsecViewController.glkView as? ParsecGLKViewController { + parsecGLK.glkViewController.isPaused = true + } + CParsec.disconnect() + } + // 600 ms gives the SDK enough time to fully tear down the session + // (audio queue, poll loops, internal state) before we reconnect. + // The user previously saw "Disconnected 20" with a 100 ms gap — + // turns out the SDK isn't truly ready for connect() that fast. + DispatchQueue.main.asyncAfter(deadline: .now() + 0.6) { + let status = CParsec.connect(peerID) + // connect() already installs the fresh ParsecClientConfig with the + // new resolution; calling applyConfig() right after would issue a + // redundant ParsecClientSetConfig against a just-negotiated + // session (sometimes racy). Skip it. + if let parsecGLK = self.parsecViewController.glkView as? ParsecGLKViewController { + parsecGLK.glkViewController.isPaused = false + } + + // If the reconnect didn't take (host offline, network blip), + // surface a real error instead of leaving the user staring at + // a frozen frame behind the spinner. + if status != PARSEC_OK && status != PARSEC_CONNECTING { + self.isReconfiguring = false + self.DCAlertText = "Reconnect failed (code \(status.rawValue))" + self.showDCAlert = true + return + } + + // Drop the overlay a beat later — give the first new frame time + // to arrive so the user sees content, not the spinner-over-stale. + DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) { + self.isReconfiguring = false + } } } @@ -515,7 +659,42 @@ struct ParsecView: View func changeDisplay(displayId: String) { DispatchQueue.main.async { DataManager.model.output = displayId + // Persist so the next connect can auto-restore this choice once + // the host enumerates displays (user-data event 12). "none" is + // the "Auto" pseudo-id and isn't worth remembering. + if displayId == "none" { + SettingsHandler.savedDisplayOutput = "" + SettingsHandler.savedDisplayName = "" + } else { + SettingsHandler.savedDisplayOutput = displayId + // Also persist the human-readable name so a regenerated id + // next session can be matched by name. + if let cfg = DataManager.model.displayConfigs.first(where: { $0.id == displayId }) { + SettingsHandler.savedDisplayName = "\(cfg.name) \(cfg.adapterName)" + } + } + + // S10: switching the streamed monitor makes the host re-init its + // encoder for the new display (usually a different resolution), + // forcing a client-side decoder reset. During that window + // getStatusEx briefly returns non-OK, and the poll loop pops a + // spurious "Disconnected" alert — suppressed only while + // isReconfiguring. changeResolution raises that flag; changeDisplay + // historically never did (D1). Bracket the switch the same way. + self.isReconfiguring = true CParsec.updateHostVideoConfig() + + // No reconnect happens here, so there's no completion callback to + // clear the flag — use a fixed window covering + // updateHostVideoConfig's 250/450 ms resend + echo cycle plus + // headroom. Resume the GL surface once it settles in case the + // decoder reset blanked it (D2 — same failure class as S01, on a + // path viewDidAppear/foreground hooks don't cover since no screen + // transition occurs here). + DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) { + self.parsecViewController.glkView?.resume() + self.isReconfiguring = false + } } } diff --git a/OpenParsec/ParsecViewController.swift b/OpenParsec/ParsecViewController.swift index 00c8106..f0920ab 100644 --- a/OpenParsec/ParsecViewController.swift +++ b/OpenParsec/ParsecViewController.swift @@ -7,6 +7,10 @@ protocol ParsecPlayground { init(viewController: UIViewController, updateImage: @escaping () -> Void) func viewDidLoad() func cleanUp() + // Idempotent render resume — re-currents the GL context and unpauses the + // render loop. Called on screen return / app foreground to recover from a + // paused or stale-drawable state (the black-screen-on-return bug). + func resume() func updateSize(width: CGFloat, height: CGFloat) } @@ -16,6 +20,16 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { var gamePadController: GamepadController! var touchController: TouchController! var u: UIImageView? + // Locally-rendered iPadOS-style pointer dot that follows input immediately, + // skipping the host RTT that the host-streamed cursor (`u`) inherits. + // Visibility is toggled in updateImage based on + // SettingsHandler.localCursorOverlay. + var localCursorImageView: UIView? + var localCursorPosition: CGPoint = .zero + // viewDidLoad runs before the contentView has a non-zero size, so seeding + // the cursor there reads midX/Y as 0 and parks it at the corner. Re-seed + // once layout produces real bounds. One-shot via this flag. + var hasSeededLocalCursor: Bool = false var lastImg: CGImage? var lastMouseX: Int32 = -1 var lastMouseY: Int32 = -1 @@ -27,10 +41,65 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { var accumulatedDeltaY: Float = 0.0 var lastPanLocation: CGPoint = .zero var lastPanTranslation: CGPoint = .zero + + // Trackpad / mouse-wheel scroll accumulators (separate from the touchscreen + // 2-finger pan path, which keeps using velocity-based wheel messages for + // direct-touch swipes). + var accumulatedScrollX: Float = 0.0 + var accumulatedScrollY: Float = 0.0 + var lastScrollTranslation: CGPoint = .zero + + // Client-side scroll inertia. The trackpad scroll path is driven by a raw + // UIPanGestureRecognizer (allowedScrollTypesMask = .all, max 0 touches), NOT + // a UIScrollView — and a raw recognizer gets NO OS-provided momentum tail + // after the fingers lift (only UIScrollView synthesizes deceleration). So a + // flick died instantly and the host saw no glide. S03 removed the old inertia + // on the wrong assumption that "the OS momentum rides in for free"; it does + // not on this recognizer, which is why inertia disappeared entirely. + // + // We re-seed a decaying glide from the PEAK velocity sampled during `.changed` + // (gestureRecognizer.velocity at `.ended` is already near-zero because iPadOS + // pre-decelerates its scroll-event stream) and drive it with a CADisplayLink. + // BOTH the per-tick advance and the decay are scaled by the real frame + // duration, so the glide distance is identical at 60 Hz and 120 Hz — fixing + // the original bug (a hardcoded `peak/60` seed that doubled the glide on the + // 120 Hz M4 iPad) without throwing away the feature. + var momentumDisplayLink: CADisplayLink? + var momentumVelocityX: Float = 0.0 // scaled host-wheel units / second + var momentumVelocityY: Float = 0.0 + private var peakScrollVelocityX: CGFloat = 0 // raw points / second + private var peakScrollVelocityY: CGFloat = 0 + // Glide tuning. Decay is expressed per-second and raised to dt each tick, so + // it is frame-rate independent. Start/stop are in scaled units/sec (same + // scale as momentumVelocity). These are deliberately conservative; the real + // feel can only be judged on-device, so they are easy to nudge. + private let scrollMomentumDecayPerSecond: Float = 0.02 // → ~2% of speed remains after 1 s + private let scrollMomentumStartSpeed: Float = 60.0 // below this a release doesn't glide + private let scrollMomentumStopSpeed: Float = 24.0 // below this the glide ends + + // Layout sync — fires a hotkey at the host when the iPad's hardware-keyboard + // input language changes (e.g. Caps Lock toggle on Magic Keyboard). + var languageSync: LanguageSyncCoordinator? var mouseSensitivity: Float = Float(SettingsHandler.mouseSensitivity) var activatedPanFingerNumber: Int = 0 + // Ctrl+Shift→Cmd+Space chord state machine (S05). `chordArmed` is true while + // both Ctrl and Shift are held with nothing else pressed; `chordSawOtherKey` + // trips the moment any non-modifier key joins the combo, so Ctrl+Shift+Arrow + // (host selection) does NOT emulate Cmd+Space. We track held modifiers as a + // small set of HID usages rather than reading event.modifierFlags so left/ + // right Ctrl and Shift are both honored and key-up bookkeeping is exact. + private var heldModifierKeyCodes: Set = [] + private var chordArmed: Bool = false + private var chordSawOtherKey: Bool = false + + // Key-downs we deliberately swallowed (currently only the backtick→Cmd+Space + // remap). Their matching key-up must be swallowed too, otherwise the host + // receives a lone grave release and — worse — if the user holds the key, the + // auto-repeat key-downs we drop but a single trailing key-up could desync. + private var suppressedKeyUps: Set = [] + var keyboardAccessoriesView : UIView? var keyboardHeight : CGFloat = 0.0 var keyboardVisible : Bool = false @@ -45,7 +114,18 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { override var prefersHomeIndicatorAutoHidden : Bool { return true } - + + // With the pointer locked (prefersPointerLocked), iPadOS still owns a strip at + // the bottom edge for the home-indicator / Control-Center swipe. Trackpad motion + // landing in that strip is partly eaten by the system gesture and never reaches + // our pointer handling, which the user feels as a dead-zone — and only inside + // Parsec, because the pointer lock is unique to this screen. Deferring the system + // edge gestures hands that strip back to the app. A second deliberate swipe still + // reaches the system gesture, so Control-Center access is preserved. + override var preferredScreenEdgesDeferringSystemGestures: UIRectEdge { + return .all + } + init() { super.init(nibName: nil, bundle: nil) @@ -58,14 +138,37 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { required init?(coder: NSCoder) { fatalError("init(coder:) has not been implemented") } - + + deinit { + languageSync?.stop() + stopScrollMomentum() + } + func updateImage() { - // Optimization: Snap current valus - let currentMouseX = CParsec.mouseInfo.mouseX - let currentMouseY = CParsec.mouseInfo.mouseY - let currentHidden = CParsec.mouseInfo.cursorHidden - let currentImg = CParsec.mouseInfo.cursorImg - + // Take ONE atomic snapshot of the cross-thread mouse state. Re-reading + // CParsec.mouseInfo per field would each lock-and-copy separately and + // could tear across a concurrent poll-thread write; the snapshot also + // pins cursorImg's lifetime for the whole frame (C1 fix). + let info = CParsec.mouseInfo + let currentMouseX = info.mouseX + let currentMouseY = info.mouseY + let currentHidden = info.cursorHidden + let currentImg = info.cursorImg + + // Toggle host vs local cursor visibility from the same place so they + // stay mutually exclusive without scattered if-statements. + let useLocalOverlay = SettingsHandler.localCursorOverlay + localCursorImageView?.isHidden = !useLocalOverlay + if useLocalOverlay { + // Suppress the host-rendered cursor image — we're drawing our + // own. Early-return to skip the position math below since `u` + // isn't being displayed. + u?.isHidden = true + return + } else { + u?.isHidden = false + } + // Skip if nothing changed if currentMouseX == lastMouseX && currentMouseY == lastMouseY && @@ -85,10 +188,10 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { } // Using tracked values for bounds - u?.frame = CGRect(x: Int(currentMouseX) - Int(Double(CParsec.mouseInfo.cursorHotX) * SettingsHandler.cursorScale), - y: Int(currentMouseY) - Int(Double(CParsec.mouseInfo.cursorHotY) * SettingsHandler.cursorScale), - width: Int(Double(CParsec.mouseInfo.cursorWidth) * SettingsHandler.cursorScale), - height: Int(Double(CParsec.mouseInfo.cursorHeight) * SettingsHandler.cursorScale)) + u?.frame = CGRect(x: Int(currentMouseX) - Int(Double(info.cursorHotX) * SettingsHandler.cursorScale), + y: Int(currentMouseY) - Int(Double(info.cursorHotY) * SettingsHandler.cursorScale), + width: Int(Double(info.cursorWidth) * SettingsHandler.cursorScale), + height: Int(Double(info.cursorHeight) * SettingsHandler.cursorScale)) // Check bounds and pan if needed // Only pan if we are zoomed in OR if the keyboard is visible (to allow scrolling up) @@ -208,9 +311,35 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { u = UIImageView(frame: CGRect(x: 0,y: 0,width: 100, height: 100)) contentView.addSubview(u!) // Add Cursor to ContentView + + // Local overlay cursor — iPadOS-style pointer dot drawn programmatically + // so it actually looks native (the SF Symbol cursorarrow we tried + // before was a Mac-style arrow that didn't match iPad muscle memory). + // Light gray with a darker border for contrast on any background, + // soft shadow for legibility. Same parent (contentView) so it scrolls + // with the streamed content when the user is zoomed in. + let dotSize: CGFloat = 13 + let localCursor = UIView(frame: CGRect(x: 0, y: 0, width: dotSize, height: dotSize)) + localCursor.backgroundColor = UIColor(white: 0.92, alpha: 0.85) + localCursor.layer.cornerRadius = dotSize / 2 + localCursor.layer.borderWidth = 0.5 + localCursor.layer.borderColor = UIColor(white: 0.15, alpha: 0.45).cgColor + localCursor.layer.shadowColor = UIColor.black.cgColor + localCursor.layer.shadowOpacity = 0.35 + localCursor.layer.shadowRadius = 2.5 + localCursor.layer.shadowOffset = CGSize(width: 0, height: 1) + localCursor.isUserInteractionEnabled = false + localCursor.isHidden = !SettingsHandler.localCursorOverlay + contentView.addSubview(localCursor) + localCursorImageView = localCursor + // The real seeding happens in viewDidLayoutSubviews — contentView's + // bounds aren't valid until layout has run, so reading midX/Y here + // would give (0, 0). Park at (0, 0) for now; the first layout pass + // will move it to the centre. setNeedsUpdateOfPrefersPointerLocked() - + setNeedsUpdateOfScreenEdgesDeferringSystemGestures() + let pointerInteraction = UIPointerInteraction(delegate: self) view.addInteraction(pointerInteraction) @@ -222,8 +351,26 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { // Important: Allow our pan gesture to work alongside scrollview's? // No, we want 1 finger for this pan, 2 fingers for scrollview. // So they are distinct by touch count. + // Exclude .indirectPointer (Magic Keyboard trackpad / iPad pointer) — those + // events are handled directly via touchesMoved below to avoid the latency + // of the gesture-recognizer state machine (issue #47: sticky cursor). + panGestureRecognizer.allowedTouchTypes = [ + NSNumber(value: UITouch.TouchType.direct.rawValue), + NSNumber(value: UITouch.TouchType.pencil.rawValue) + ] view.addGestureRecognizer(panGestureRecognizer) + // Dedicated recognizer for trackpad / mouse-wheel scroll events + // (iPadOS 13.4+, available unconditionally at deployment target 14). + // maximumNumberOfTouches = 0 makes it respond ONLY to scroll-wheel events, + // not to fingers — so touchscreen 2-finger swipes still flow through the + // main pan recognizer with allowedTouchTypes = direct + pencil. + let trackpadScrollRecognizer = UIPanGestureRecognizer(target: self, action: #selector(self.handleTrackpadScroll(_:))) + trackpadScrollRecognizer.delegate = self + trackpadScrollRecognizer.allowedScrollTypesMask = .all + trackpadScrollRecognizer.maximumNumberOfTouches = 0 + view.addGestureRecognizer(trackpadScrollRecognizer) + // Remove custom Pinch logic, ScrollView handles it. // But we might want to know isPinching status? // Let's rely on ScrollView delegate for updates. @@ -265,9 +412,28 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { name: UIResponder.keyboardWillHideNotification, object: nil ) - + + NotificationCenter.default.addObserver( + self, + selector: #selector(appWillEnterForeground), + name: UIApplication.willEnterForegroundNotification, + object: nil + ) + } + override func viewDidLayoutSubviews() { + super.viewDidLayoutSubviews() + // One-shot seed of the local cursor at the content view's centre + // after layout has produced real bounds. viewDidLoad reads zero + // bounds and would park the cursor at the corner. + if !hasSeededLocalCursor && contentView.bounds.width > 0 && contentView.bounds.height > 0 { + localCursorPosition = CGPoint(x: contentView.bounds.midX, y: contentView.bounds.midY) + localCursorImageView?.center = localCursorPosition + hasSeededLocalCursor = true + } + } + override func viewWillTransition(to size: CGSize, with coordinator: UIViewControllerTransitionCoordinator) { super.viewWillTransition(to: size, with: coordinator) @@ -299,6 +465,18 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { becomeFirstResponder() } scrollView.pinchGestureRecognizer?.isEnabled = zoomEnabled + startLanguageSyncIfNeeded() + // Recover the render loop on any return to this screen (SwiftUI + // re-render, PiP exit, interrupted resolution change). Without this + // a left-over isPaused == true keeps the surface frozen/black. + glkView?.resume() + } + + @objc private func appWillEnterForeground() { + // Background→foreground invalidates GL drawable currency; re-current + // the context and unpause so the next frame renders instead of going + // black. Scene plumbing doesn't reach this VC, so observe directly. + glkView?.resume() } override func viewWillDisappear(_ animated: Bool) { @@ -309,23 +487,288 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { } NotificationCenter.default.removeObserver(self, name: UIResponder.keyboardWillShowNotification, object: nil) NotificationCenter.default.removeObserver(self, name: UIResponder.keyboardWillHideNotification, object: nil) + stopLanguageSync() + stopScrollMomentum() } + // Direct trackpad pointer handling (issue #47). With prefersPointerLocked = true, + // iPadOS delivers Magic Keyboard trackpad motion as UITouches with + // type == .indirectPointer. Routing those through a UIPanGestureRecognizer + // imposes a recognition threshold and a state-machine churn between strokes, + // which is what users experience as the "sticky / juddery" cursor. + // + // The main pan recognizer's allowedTouchTypes excludes .indirectPointer + // (see viewDidLoad), so those touches reach this override unobstructed. + override func touchesBegan(_ touches: Set, with event: UIEvent?) { + super.touchesBegan(touches, with: event) + // Reset the trackpad-cursor delta accumulators when an .indirectPointer + // touch begins so a new stroke doesn't inherit leftover sub-pixel motion. + for touch in touches where touch.type == .indirectPointer { + accumulatedDeltaX = 0.0 + accumulatedDeltaY = 0.0 + } + } + + override func touchesMoved(_ touches: Set, with event: UIEvent?) { + super.touchesMoved(touches, with: event) + for touch in touches where touch.type == .indirectPointer { + if SettingsHandler.cursorMode == .direct { + let pos = touch.preciseLocation(in: view) + let adjusted = contentView.convert(pos, from: view) + CParsec.sendMousePosition(Int32(adjusted.x), Int32(adjusted.y)) + moveLocalCursor(to: adjusted) + } else { + // Iterate coalesced samples so fast trackpad motion is sent at + // the hardware sample rate (up to 120 Hz) rather than being + // sub-sampled to the slower UIKit event rate. Each sample's + // precisePreviousLocation chains to the prior sample, so summing + // per-sample deltas reconstructs the full path — the cursor + // tracks fast flicks smoothly instead of jumping in big steps. + let samples = event?.coalescedTouches(for: touch) ?? [touch] + for sample in samples { + let prev = sample.precisePreviousLocation(in: view) + let cur = sample.preciseLocation(in: view) + let rawDX = cur.x - prev.x + let rawDY = cur.y - prev.y + let scale = effectiveDeltaScale(rawDX: rawDX, rawDY: rawDY) + let preciseDX = rawDX * scale + let preciseDY = rawDY * scale + + // Move the LOCAL cursor at full sub-pixel precision so it + // glides smoothly even when the rounded host-delta is zero. + moveLocalCursor(byX: preciseDX, y: preciseDY) + + accumulatedDeltaX += Float(preciseDX) + accumulatedDeltaY += Float(preciseDY) + // Round-half-away-from-zero (not Int32 truncation) so that + // sub-pixel ticks like 0.4 still emit a 1-pixel send to host. + let dx = Int32(accumulatedDeltaX.rounded(.toNearestOrAwayFromZero)) + let dy = Int32(accumulatedDeltaY.rounded(.toNearestOrAwayFromZero)) + if dx != 0 || dy != 0 { + CParsec.sendMouseDelta(dx, dy) + accumulatedDeltaX -= Float(dx) + accumulatedDeltaY -= Float(dy) + } + } + } + } + } + + // Combined sensitivity × acceleration. Acceleration adds a per-event + // gain proportional to the per-event speed: bigger movements travel + // proportionally further, mimicking macOS pointer acceleration. Falls + // back to pure linear sensitivity when mouseAcceleration == 0. + private func effectiveDeltaScale(rawDX: CGFloat, rawDY: CGFloat) -> CGFloat { + let sens = CGFloat(SettingsHandler.mouseSensitivity) + let accel = CGFloat(SettingsHandler.mouseAcceleration) + guard accel > 0 else { return sens } + // |raw_delta| is in points per input event (typically 0..15). + // Dividing by 5 normalizes so 5 px/event ≈ 1× boost at accel=1. + let speed = sqrt(rawDX * rawDX + rawDY * rawDY) / 5.0 + return sens + accel * speed + } + + // Move the local cursor overlay (no-op if it's hidden — we just keep + // the tracker fresh so toggling the setting mid-session lands on a + // sensible position). + func moveLocalCursor(to position: CGPoint) { + localCursorPosition = position + clampAndApplyLocalCursor() + } + + func moveLocalCursor(byX dx: CGFloat, y dy: CGFloat) { + localCursorPosition.x += dx + localCursorPosition.y += dy + clampAndApplyLocalCursor() + } + + private func clampAndApplyLocalCursor() { + let bounds = contentView.bounds + localCursorPosition.x = max(0, min(localCursorPosition.x, bounds.width)) + localCursorPosition.y = max(0, min(localCursorPosition.y, bounds.height)) + localCursorImageView?.center = localCursorPosition + } + + // Cleared sub-pixel residue so the next gesture starts from zero. Without + // this the residue carries across strokes at low sensitivity and creates + // systematic drift. + override func touchesEnded(_ touches: Set, with event: UIEvent?) { + super.touchesEnded(touches, with: event) + for touch in touches where touch.type == .indirectPointer { + accumulatedDeltaX = 0.0 + accumulatedDeltaY = 0.0 + break + } + } + + override func touchesCancelled(_ touches: Set, with event: UIEvent?) { + super.touchesCancelled(touches, with: event) + for touch in touches where touch.type == .indirectPointer { + accumulatedDeltaX = 0.0 + accumulatedDeltaY = 0.0 + break + } + } + override func pressesBegan(_ presses: Set, with event: UIPressesEvent?) { - + for press in presses { + chordTrackPressBegan(press.key) + // Backtick→Cmd+Space remap: a bare ` becomes a manual language switch. + // Swallow the grave key-down and synthesize Cmd+Space at the host; + // record the suppression so the matching key-up is swallowed too. + if backtickWillEmulateCmdSpace(press.key), let usage = press.key?.keyCode { + // Auto-repeat delivers repeated pressesBegan with no intervening + // key-up; fire Cmd+Space only on the first down so holding ` does + // not spam input-source switches at the host. + if suppressedKeyUps.insert(usage).inserted { + fireCmdSpace() + } + continue + } + // Always forward the raw scancode unchanged — the Cmd+Space emulation + // is additive and fires only on a clean Ctrl+Shift release, so host + // shortcuts like Ctrl+Shift+X keep working. CParsec.sendKeyboardMessage(event:KeyBoardKeyEvent(input: press.key, isPressBegin: true) ) } - + } - + override func pressesEnded (_ presses: Set, with event: UIPressesEvent?) { - + for press in presses { + // If we swallowed this key's key-down (backtick→Cmd+Space), swallow the + // key-up too — the host already saw a complete Cmd+Space tap. + if let usage = press.key?.keyCode, suppressedKeyUps.remove(usage) != nil { + chordTrackPressEnded(press.key) + continue + } CParsec.sendKeyboardMessage(event:KeyBoardKeyEvent(input: press.key, isPressBegin: false) ) + chordTrackPressEnded(press.key) + } + + } + + override func pressesCancelled(_ presses: Set, with event: UIPressesEvent?) { + super.pressesCancelled(presses, with: event) + // A cancelled press never delivers pressesEnded, so its key-up would be + // lost — both for the host (stuck modifier) and for our chord bookkeeping. + for press in presses { + if let usage = press.key?.keyCode, suppressedKeyUps.remove(usage) != nil { + chordTrackPressEnded(press.key) + continue + } + CParsec.sendKeyboardMessage(event: KeyBoardKeyEvent(input: press.key, isPressBegin: false)) + chordTrackPressEnded(press.key) + } + // Any interruption invalidates an in-flight chord. + chordArmed = false + chordSawOtherKey = false + } + + // True when the backtick→Cmd+Space remap should fire for this press: the + // feature is enabled, the key is the grave/backtick, and NO Cmd/Ctrl/Alt/ + // Shift is held (so Shift+` = tilde and Cmd+` still reach the host as normal + // keys). Caps Lock and other non-blocking flags are ignored. + private func backtickWillEmulateCmdSpace(_ key: UIKey?) -> Bool { + guard SettingsHandler.backtickEmulatesCmdSpace, let key = key else { return false } + guard key.keyCode == .keyboardGraveAccentAndTilde else { return false } + return key.modifierFlags.isDisjoint(with: [.command, .control, .alternate, .shift]) + } + + // MARK: Ctrl+Shift → Cmd+Space chord machine (S05) + + private func isModifierUsage(_ usage: UIKeyboardHIDUsage) -> Bool { + switch usage { + case .keyboardLeftControl, .keyboardRightControl, + .keyboardLeftShift, .keyboardRightShift, + .keyboardLeftAlt, .keyboardRightAlt, + .keyboardLeftGUI, .keyboardRightGUI: + return true + default: + return false + } + } + + private func chordHasControl() -> Bool { + return heldModifierKeyCodes.contains(.keyboardLeftControl) + || heldModifierKeyCodes.contains(.keyboardRightControl) + } + + private func chordHasShift() -> Bool { + return heldModifierKeyCodes.contains(.keyboardLeftShift) + || heldModifierKeyCodes.contains(.keyboardRightShift) + } + + // Any modifier outside Ctrl/Shift (Alt or Cmd) disqualifies the chord — we + // only want a *clean* Ctrl+Shift, not Ctrl+Shift+Alt etc. + private func chordHasForeignModifier() -> Bool { + return heldModifierKeyCodes.contains(.keyboardLeftAlt) + || heldModifierKeyCodes.contains(.keyboardRightAlt) + || heldModifierKeyCodes.contains(.keyboardLeftGUI) + || heldModifierKeyCodes.contains(.keyboardRightGUI) + } + + private func chordTrackPressBegan(_ key: UIKey?) { + guard SettingsHandler.ctrlShiftEmulatesCmdSpace, let usage = key?.keyCode else { return } + if isModifierUsage(usage) { + heldModifierKeyCodes.insert(usage) + // Arm only on a clean Ctrl+Shift with no foreign modifier. + if chordHasControl() && chordHasShift() && !chordHasForeignModifier() { + chordArmed = true + chordSawOtherKey = false + } else if chordHasForeignModifier() { + // Alt/Cmd joined → this is a different combo, not our chord. + chordSawOtherKey = true + } + } else { + // A real key was pressed while modifiers are held → not a bare chord. + chordSawOtherKey = true + } + } + + private func chordTrackPressEnded(_ key: UIKey?) { + guard let usage = key?.keyCode else { return } + guard SettingsHandler.ctrlShiftEmulatesCmdSpace else { + // Setting may have flipped off mid-hold; keep the held-set honest. + if isModifierUsage(usage) { heldModifierKeyCodes.remove(usage) } + return + } + guard isModifierUsage(usage) else { return } + + let wasArmed = chordArmed + let cleanRelease = !chordSawOtherKey + // Fire when the last of the Ctrl/Shift pair lifts on a clean chord. + let isCtrlOrShift = (usage == .keyboardLeftControl || usage == .keyboardRightControl + || usage == .keyboardLeftShift || usage == .keyboardRightShift) + + heldModifierKeyCodes.remove(usage) + + // Fire only once the pair has *fully* lifted (neither Ctrl nor Shift + // still held). Keying on "not both-held" instead would fire on the + // FIRST release while the other modifier is still down — so dropping + // Ctrl to continue with Shift+ would spuriously emit Cmd+Space. + if wasArmed && cleanRelease && isCtrlOrShift && !chordHasControl() && !chordHasShift() { + fireCmdSpace() + } + // Once neither Ctrl nor Shift remains, fully reset. + if !chordHasControl() && !chordHasShift() { + chordArmed = false + chordSawOtherKey = false + } + } + + private func fireCmdSpace() { + // LGUI = left Cmd. Press Cmd → tap Space → release Cmd, with the same + // ~50 ms modifier-hold the layout-sync chord uses so the host registers + // the combo intact. Note: iPadOS owns the *physical* Cmd+Space, but here + // we synthesize it as host scancodes, which the host honors. + CParsec.sendVirtualKeyboardInput(text: "LGUI", isOn: true) + CParsec.sendVirtualKeyboardInput(text: "SPACE") + DispatchQueue.main.asyncAfter(deadline: .now() + 0.05) { + CParsec.sendVirtualKeyboardInput(text: "LGUI", isOn: false) } - } @objc func keyboardWillShow(notification: NSNotification) { @@ -371,8 +814,24 @@ class ParsecViewController: UIViewController, UIScrollViewDelegate { scrollView.setContentOffset(CGPoint(x: scrollView.contentOffset.x, y: newOffsetY), animated: true) } onKeyboardVisibilityChanged?(false) + + // Reclaim first responder for the hidden language-sync field after ANY + // soft-keyboard dismissal. Previously only setKeyboardVisible(false) + // reclaimed it, so dismissing via the toolbar Done button, a swipe-down, + // the Globe key, or any system-driven hide left the hidden field resigned + // — at which point currentInputModeDidChangeNotification stops firing and + // keyboard-layout sync silently dies until the next setKeyboardVisible(false). + // keyboardWillHide is the common funnel for every dismiss path, so reclaim + // here. Deferred to the next runloop tick so we don't fight the in-progress + // hide; guarded on !keyboardVisible so a show that raced in keeps the VC as + // first responder. The field has an empty inputView, so this never re-shows + // the keyboard. + DispatchQueue.main.async { [weak self] in + guard let self = self, !self.keyboardVisible else { return } + self.languageSync?.reclaimFirstResponder() + } } - + } extension ParsecViewController : UIGestureRecognizerDelegate { @@ -430,6 +889,7 @@ extension ParsecViewController : UIGestureRecognizerDelegate { // Convert to content coordinates let adjustedPosition = contentView.convert(position, from: view) CParsec.sendMousePosition(Int32(adjustedPosition.x), Int32(adjustedPosition.y)) + moveLocalCursor(to: adjustedPosition) } else { // Simple translation-based movement with sub-pixel accumulation let currentTranslation = gestureRecognizer.translation(in: gestureRecognizer.view) @@ -440,19 +900,28 @@ extension ParsecViewController : UIGestureRecognizerDelegate { accumulatedDeltaY = 0.0 } - // Calculate delta since last update - let deltaX = Float(currentTranslation.x - lastPanTranslation.x) * mouseSensitivity - let deltaY = Float(currentTranslation.y - lastPanTranslation.y) * mouseSensitivity + // Calculate delta since last update at full precision, with + // the combined sensitivity × acceleration curve. + let rawDX = currentTranslation.x - lastPanTranslation.x + let rawDY = currentTranslation.y - lastPanTranslation.y + let scale = effectiveDeltaScale(rawDX: rawDX, rawDY: rawDY) + let preciseDX = rawDX * scale + let preciseDY = rawDY * scale lastPanTranslation = currentTranslation - // Accumulate for sub-pixel precision - accumulatedDeltaX += deltaX - accumulatedDeltaY += deltaY + // Move the LOCAL cursor at full sub-pixel precision so its + // motion stays smooth even between whole-pixel host deltas. + moveLocalCursor(byX: preciseDX, y: preciseDY) - // Send movement when we have at least 1 pixel - let intDeltaX = Int32(accumulatedDeltaX) - let intDeltaY = Int32(accumulatedDeltaY) + // Accumulate for sub-pixel precision, then round-half-away- + // from-zero so 0.4-px ticks still emit a 1-pixel message + // (Int32 truncation made slow drags feel sticky). + accumulatedDeltaX += Float(preciseDX) + accumulatedDeltaY += Float(preciseDY) + + let intDeltaX = Int32(accumulatedDeltaX.rounded(.toNearestOrAwayFromZero)) + let intDeltaY = Int32(accumulatedDeltaY.rounded(.toNearestOrAwayFromZero)) if intDeltaX != 0 || intDeltaY != 0 { CParsec.sendMouseDelta(intDeltaX, intDeltaY) @@ -469,8 +938,140 @@ extension ParsecViewController : UIGestureRecognizerDelegate { } } + // Trackpad / mouse-wheel scroll handler, separated from handlePanGesture so it + // can use translation deltas (smooth) instead of velocity (rough wheel + // messages). Hooked up by a UIPanGestureRecognizer with + // allowedScrollTypesMask = .all and maximumNumberOfTouches = 0 in viewDidLoad, + // so it only sees scroll-wheel / trackpad-scroll events, never finger touches. + @objc func handleTrackpadScroll(_ gestureRecognizer: UIPanGestureRecognizer) { + // "Natural scrolling" toggle ON = swipe-direction follows content, which + // is the macOS default ("Natural" in System Settings). With the user's + // Mac on its default Natural Scrolling = ON, we want to forward + // translation deltas without flipping — host applies its own + // inversion. Flip only when toggle is OFF (= user wants classic + // mouse-wheel feel). + let direction: Float = SettingsHandler.naturalScrolling ? 1.0 : -1.0 + let sensitivity = Float(SettingsHandler.scrollSensitivity) + + switch gestureRecognizer.state { + case .began: + // A fresh scroll cancels any glide still in progress. + stopScrollMomentum() + lastScrollTranslation = .zero + accumulatedScrollX = 0.0 + accumulatedScrollY = 0.0 + peakScrollVelocityX = 0 + peakScrollVelocityY = 0 + case .changed: + let translation = gestureRecognizer.translation(in: gestureRecognizer.view) + // When the user is zoomed in, let UIScrollView's own pan handle the + // scroll locally — otherwise we'd both scroll the local view AND + // send wheel messages to the host (double-pan). Keep + // lastScrollTranslation in sync before returning (SC5): otherwise + // the next forwarded `.changed` after un-zoom computes a delta + // against a stale anchor and jumps the host. + if scrollView.zoomScale > 1.0 { + lastScrollTranslation = translation + return + } + + let deltaX = Float(translation.x - lastScrollTranslation.x) * sensitivity * direction + let deltaY = Float(translation.y - lastScrollTranslation.y) * sensitivity * direction + lastScrollTranslation = translation + + // Track the peak velocity over the gesture. `velocity(in:)` at + // `.ended` is already near-zero because iPadOS pre-decelerates its + // scroll-event stream, so we seed the post-lift glide from the peak + // seen mid-gesture instead. + let v = gestureRecognizer.velocity(in: gestureRecognizer.view) + if abs(v.x) > abs(peakScrollVelocityX) { peakScrollVelocityX = v.x } + if abs(v.y) > abs(peakScrollVelocityY) { peakScrollVelocityY = v.y } + + accumulatedScrollX += deltaX + accumulatedScrollY += deltaY + // Rounding accumulator: Int32 truncation previously swallowed + // sub-pixel ticks so slow trackpad scrolls felt dead until enough + // whole pixels piled up. + let intX = Int32(accumulatedScrollX.rounded(.toNearestOrAwayFromZero)) + let intY = Int32(accumulatedScrollY.rounded(.toNearestOrAwayFromZero)) + if intX != 0 || intY != 0 { + CParsec.sendWheelMsg(x: intX, y: intY) + accumulatedScrollX -= Float(intX) + accumulatedScrollY -= Float(intY) + } + case .ended: + // Seed a decaying glide from the peak velocity (scaled the same way + // the live deltas were). The accumulator carries over so sub-pixel + // remainder isn't lost between the live phase and the glide. + lastScrollTranslation = .zero + if scrollView.zoomScale <= 1.0 { + startScrollMomentum( + velX: Float(peakScrollVelocityX) * sensitivity * direction, + velY: Float(peakScrollVelocityY) * sensitivity * direction + ) + } + peakScrollVelocityX = 0 + peakScrollVelocityY = 0 + case .cancelled, .failed: + stopScrollMomentum() + lastScrollTranslation = .zero + accumulatedScrollX = 0.0 + accumulatedScrollY = 0.0 + peakScrollVelocityX = 0 + peakScrollVelocityY = 0 + default: + break + } + } + + // MARK: - Trackpad scroll momentum (client-side inertia) + + private func startScrollMomentum(velX: Float, velY: Float) { + stopScrollMomentum() + // Don't bother gliding for a near-stationary release. + guard hypotf(velX, velY) >= scrollMomentumStartSpeed else { return } + momentumVelocityX = velX + momentumVelocityY = velY + let link = CADisplayLink(target: self, selector: #selector(stepScrollMomentum(_:))) + link.add(to: .main, forMode: .common) + momentumDisplayLink = link + } + + @objc private func stepScrollMomentum(_ link: CADisplayLink) { + // dt is the real duration of the upcoming frame, so the glide travels the + // same distance per second at 60 Hz and 120 Hz (the bug that doubled the + // old hardcoded `peak/60` seed on the 120 Hz iPad). Clamp out the absurd + // first-tick / stall values. + let dt = Float(link.targetTimestamp - link.timestamp) + guard dt > 0, dt < 0.1 else { return } + + accumulatedScrollX += momentumVelocityX * dt + accumulatedScrollY += momentumVelocityY * dt + let intX = Int32(accumulatedScrollX.rounded(.toNearestOrAwayFromZero)) + let intY = Int32(accumulatedScrollY.rounded(.toNearestOrAwayFromZero)) + if intX != 0 || intY != 0 { + CParsec.sendWheelMsg(x: intX, y: intY) + accumulatedScrollX -= Float(intX) + accumulatedScrollY -= Float(intY) + } + + let decay = powf(scrollMomentumDecayPerSecond, dt) + momentumVelocityX *= decay + momentumVelocityY *= decay + if hypotf(momentumVelocityX, momentumVelocityY) < scrollMomentumStopSpeed { + stopScrollMomentum() + } + } + + private func stopScrollMomentum() { + momentumDisplayLink?.invalidate() + momentumDisplayLink = nil + momentumVelocityX = 0 + momentumVelocityY = 0 + } + @objc func handleSingleFingerTap(_ gestureRecognizer: UITapGestureRecognizer) { - + let location = gestureRecognizer.location(in:gestureRecognizer.view) let adjustedLocation = contentView.convert(location, from: view) touchController.onTap(typeOfTap: 1, location: adjustedLocation) @@ -755,6 +1356,9 @@ extension ParsecViewController : UIKeyInput, UITextInputTraits { } @objc func showKeyboard() { + // Yield FR to the VC so the soft keyboard can attach. Symmetric to the + // path in setKeyboardVisible. + languageSync?.yieldFirstResponder() becomeFirstResponder() } @@ -768,6 +1372,10 @@ extension ParsecViewController : UIKeyInput, UITextInputTraits { if visible { DispatchQueue.main.async { self.reloadInputViews() + // Cede first-responder so the soft keyboard can attach to the + // view controller (the hidden language-sync field holds it + // otherwise). + self.languageSync?.yieldFirstResponder() let success = self.becomeFirstResponder() if !success { // Fallback: try again? or just log (can't log). @@ -776,7 +1384,403 @@ extension ParsecViewController : UIKeyInput, UITextInputTraits { } } else { resignFirstResponder() + // Reclaim so we keep getting currentInputModeDidChangeNotification + // while the soft keyboard is hidden. + languageSync?.reclaimFirstResponder() } } - + +} + +// MARK: - System-shortcut capture +// +// iPadOS shell normally swallows Cmd+letter combinations (and many Cmd+modifier +// chords) before they reach any app — that's why Cmd+A inside a Parsec session +// doesn't "select all" on the host, Cmd+S doesn't save, etc. +// +// Registering an explicit UIKeyCommand for each combination tells the responder +// chain to deliver them to us instead. On iOS 15+, setting +// `wantsPriorityOverSystemBehavior = true` further suppresses the system's own +// text-input handling for the same chord. +// +// What we CANNOT capture on iPadOS via public APIs (no app can): +// • Cmd+Space (Spotlight) +// • Cmd+H (Home) +// • Cmd+Tab (app switcher) +// • Globe key shortcut layer +// • Swipe-up-from-bottom (Home gesture) +// +// Those are wired below the responder chain in SpringBoard. The PR #64 +// approach (remap Opt → Cmd on the host) is the workaround the iPad +// ecosystem has settled on for the most common case; we don't duplicate it +// here because that PR is in flight. +extension ParsecViewController { + // Cached because keyCommands is queried each event and we register ~280 + // commands. Rebuilds itself the first time captureSystemKeys flips on + // during a session and the user re-enters the streaming view. + private static let _capturableCharset: String = "abcdefghijklmnopqrstuvwxyz0123456789-=[];',./`\\" + private static let _capturableModifierCombos: [UIKeyModifierFlags] = [ + .command, + [.command, .shift], + [.command, .alternate], + [.command, .control], + .alternate, + [.alternate, .shift] + ] + + // iOS queries `keyCommands` on every key event and on every first-responder + // change. Rebuilding 286 UIKeyCommand objects each time would jank typing + // on older iPads. Cache once per type (UIKeyCommand objects are stateless + // modulo their action selector, so sharing across instances is safe — the + // system dispatches the action to whichever responder is first). + // If `captureSystemKeys` flips at runtime the user has to leave + re-enter + // the streaming view to pick up the change. + private static var _cachedKeyCommands: [UIKeyCommand]? + + func buildSystemCaptureKeyCommands() -> [UIKeyCommand] { + if let cached = Self._cachedKeyCommands { + return cached + } + var commands: [UIKeyCommand] = [] + for char in Self._capturableCharset { + for mods in Self._capturableModifierCombos { + let cmd = UIKeyCommand( + input: String(char), + modifierFlags: mods, + action: #selector(handleCapturedKey(_:)) + ) + if #available(iOS 15.0, *) { + cmd.wantsPriorityOverSystemBehavior = true + } + commands.append(cmd) + } + } + // Cmd + special keys. Cmd+Space is registered but iPadOS still wins; + // the host won't see it. Leaving it in so behaviour matches if Apple + // ever loosens this. + for special in ["\t", " ", "\r", "`"] { + let cmd = UIKeyCommand( + input: special, + modifierFlags: .command, + action: #selector(handleCapturedKey(_:)) + ) + if #available(iOS 15.0, *) { + cmd.wantsPriorityOverSystemBehavior = true + } + commands.append(cmd) + } + Self._cachedKeyCommands = commands + return commands + } + + override var keyCommands: [UIKeyCommand]? { + guard SettingsHandler.captureSystemKeys else { return super.keyCommands } + return buildSystemCaptureKeyCommands() + } + + // Translates a UIKeyCommand into a modifier-aware scancode sequence for + // the host. We deliberately use sendVirtualKeyboardInput(text:, isOn:) + // instead of (text:) so the wrapper doesn't auto-add Shift — we're + // managing modifiers explicitly. + @objc func handleCapturedKey(_ cmd: UIKeyCommand) { + let mods = cmd.modifierFlags + + if mods.contains(.command) { CParsec.sendVirtualKeyboardInput(text: "LGUI", isOn: true) } + if mods.contains(.shift) { CParsec.sendVirtualKeyboardInput(text: "SHIFT", isOn: true) } + if mods.contains(.control) { CParsec.sendVirtualKeyboardInput(text: "CONTROL", isOn: true) } + if mods.contains(.alternate) { CParsec.sendVirtualKeyboardInput(text: "LALT", isOn: true) } + + guard let raw = cmd.input, !raw.isEmpty else { + releaseHeldModifiers(mods) + return + } + + let keyText: String + switch raw { + case "\t": keyText = "TAB" + case " ": keyText = "SPACE" + case "\r", "\n": keyText = "ENTER" + default: keyText = raw.uppercased() + } + + CParsec.sendVirtualKeyboardInput(text: keyText, isOn: true) + // In Low Latency Mode we send keyup + modifier-release synchronously, + // avoiding ~80 ms of artificial hold time on every Cmd-shortcut. The + // old asyncAfter path was originally there as a defensive hold-time + // for games that misread instant releases — not needed on macOS. + if SettingsHandler.lowLatencyMode { + CParsec.sendVirtualKeyboardInput(text: keyText, isOn: false) + releaseHeldModifiers(mods) + } else { + DispatchQueue.main.asyncAfter(deadline: .now() + 0.02) { + CParsec.sendVirtualKeyboardInput(text: keyText, isOn: false) + } + DispatchQueue.main.asyncAfter(deadline: .now() + 0.06) { [weak self] in + self?.releaseHeldModifiers(mods) + } + } + } + + private func releaseHeldModifiers(_ mods: UIKeyModifierFlags) { + if mods.contains(.command) { CParsec.sendVirtualKeyboardInput(text: "LGUI", isOn: false) } + if mods.contains(.shift) { CParsec.sendVirtualKeyboardInput(text: "SHIFT", isOn: false) } + if mods.contains(.control) { CParsec.sendVirtualKeyboardInput(text: "CONTROL", isOn: false) } + if mods.contains(.alternate) { CParsec.sendVirtualKeyboardInput(text: "LALT", isOn: false) } + } +} + +// MARK: - Language sync (Mac ↔ iPad keyboard layout) +// +// Goal: when the user switches the iPad's hardware-keyboard input language +// (Caps Lock toggle on Magic Keyboard / Ctrl+Space / Globe key), fire a hotkey +// at the host so its input source switches in lock-step. +// +// Why a hotkey and not a "real" Unicode-text path: Parsec's iOS SDK only sends +// MESSAGE_KEYBOARD with HID scancodes (see ParsecSDKBridge.sendKeyboardMessage). +// There is no documented MESSAGE_CHAR / UTF-8 path that would let us bypass +// the host layout. So we ask the host to switch its own layout. Default +// hotkey is Ctrl+Space (macOS built-in "select previous input source"); user +// can pick something else if their host config differs. +// +// Detecting the language change requires a UIResponder that accepts text +// input to be first responder (Apple's `currentInputModeDidChangeNotification` +// only fires in that case). We use a 1×1 alpha-0 UITextField with an empty +// inputView so the soft keyboard never appears; the field forwards +// pressesBegan/Ended to the view controller so hardware-keyboard scancodes +// continue to flow through the existing pipeline. +extension ParsecViewController { + func startLanguageSyncIfNeeded() { + guard SettingsHandler.syncKeyboardLayout, languageSync == nil else { return } + let coordinator = LanguageSyncCoordinator(host: self, keyForwardTarget: self) + coordinator.onLanguageChange = { [weak self] _ in + self?.sendLayoutSyncHotkey() + } + coordinator.start() + languageSync = coordinator + } + + func stopLanguageSync() { + languageSync?.stop() + languageSync = nil + } + + func sendLayoutSyncHotkey() { + let hotkey = SettingsHandler.layoutSyncHotkey + switch hotkey { + case .none: + return + case .ctrlSpace: + tapKey(modifierKey: "CONTROL", normalKey: "SPACE") + case .cmdSpace: + tapKey(modifierKey: "LGUI", normalKey: "SPACE") + case .altSpace: + tapKey(modifierKey: "LALT", normalKey: "SPACE") + case .altShift: + tapModifierChord(firstModifier: "LALT", secondModifier: "SHIFT") + case .ctrlShift: + tapModifierChord(firstModifier: "CONTROL", secondModifier: "SHIFT") + } + } + + // Press modifier → press+release normal key → release modifier. The + // release of the normal key is async (+20ms) inside CParsec, so we delay + // the modifier release by ~50ms to keep the chord intact on the host. + private func tapKey(modifierKey: String, normalKey: String) { + CParsec.sendVirtualKeyboardInput(text: modifierKey, isOn: true) + CParsec.sendVirtualKeyboardInput(text: normalKey) + DispatchQueue.main.asyncAfter(deadline: .now() + 0.05) { + CParsec.sendVirtualKeyboardInput(text: modifierKey, isOn: false) + } + } + + // Two modifiers held + released as a chord (e.g. Alt+Shift on Windows). + private func tapModifierChord(firstModifier: String, secondModifier: String) { + CParsec.sendVirtualKeyboardInput(text: firstModifier, isOn: true) + CParsec.sendVirtualKeyboardInput(text: secondModifier, isOn: true) + DispatchQueue.main.asyncAfter(deadline: .now() + 0.04) { + CParsec.sendVirtualKeyboardInput(text: secondModifier, isOn: false) + } + DispatchQueue.main.asyncAfter(deadline: .now() + 0.07) { + CParsec.sendVirtualKeyboardInput(text: firstModifier, isOn: false) + } + } +} + +// Hidden field that owns first-responder status so that +// UITextInputMode.currentInputModeDidChangeNotification keeps firing. It +// forwards all UIPress events to the host view controller without consuming +// them — that way hardware-keyboard scancodes continue to flow through +// ParsecViewController.pressesBegan/Ended unchanged. +final class LanguageSyncTextField: UITextField { + weak var keyForwardTarget: UIResponder? + + override var canBecomeFirstResponder: Bool { return true } + + // Returning an empty UIView for inputView suppresses the on-screen keyboard + // while the field is first responder, even without a connected hardware + // keyboard. + private let _emptyInputView = UIView() + override var inputView: UIView? { + get { return _emptyInputView } + set { /* ignore — we want the soft kb suppressed unconditionally */ } + } + + // Critical: when this hidden field is first responder, iOS walks the + // responder chain looking for an inputAccessoryView. The chain leads up + // to ParsecViewController, which provides the OpenParsec keyboard + // toolbar (⌘ ⌃ ⌥ ⇧ F1-F12 etc.). Result: the toolbar would appear on + // every connection without any user action. Returning our own non-nil + // (empty) accessory view halts the chain walk and keeps the toolbar + // hidden until the user explicitly invokes showKeyboard(). + private let _emptyAccessoryView = UIView(frame: .zero) + override var inputAccessoryView: UIView? { + // Explicit getter + no-op setter because UIResponder.inputAccessoryView + // is a mutable property — overriding with read-only fails compile. + get { return _emptyAccessoryView } + set { /* ignore */ } + } + + override init(frame: CGRect) { + super.init(frame: frame) + // Suppress the iPad shortcuts bar (predictive text / cut/copy/paste + // chevrons that hardware-keyboard text fields show by default). + inputAssistantItem.leadingBarButtonGroups = [] + inputAssistantItem.trailingBarButtonGroups = [] + } + + required init?(coder: NSCoder) { + fatalError("init(coder:) has not been implemented") + } + + // By NOT calling super, we prevent UITextField's legacy text-input path + // from consuming printable keys via UIKeyInput.insertText. Same trick + // Moonlight uses in StreamView.m pressesBegan/pressesEnded. + override func pressesBegan(_ presses: Set, with event: UIPressesEvent?) { + keyForwardTarget?.pressesBegan(presses, with: event) + } + override func pressesEnded(_ presses: Set, with event: UIPressesEvent?) { + keyForwardTarget?.pressesEnded(presses, with: event) + } + override func pressesChanged(_ presses: Set, with event: UIPressesEvent?) { + keyForwardTarget?.pressesChanged(presses, with: event) + } + override func pressesCancelled(_ presses: Set, with event: UIPressesEvent?) { + keyForwardTarget?.pressesCancelled(presses, with: event) + } +} + +// Owns the hidden field, the change-notification observer, and the +// last-seen-language memo. Cooperates with the view controller's +// becomeFirstResponder / resignFirstResponder lifecycle via yield/reclaim. +final class LanguageSyncCoordinator { + private weak var host: UIViewController? + private weak var keyForwardTarget: UIResponder? + private var hiddenField: LanguageSyncTextField? + private var observer: NSObjectProtocol? + private var lastLanguage: String? + // True once we've observed at least one notification (or seeded a + // non-nil value asynchronously). Until then, treat any change as the + // initial state and don't fire the host hotkey — otherwise every + // session start spuriously sends Ctrl+Space because `lastLanguage` + // reads as nil immediately after becomeFirstResponder and the first + // real notification looks like "nil → en-US". + private var hasSeenInitialLanguage = false + // Debounce: rapid Caps-Lock taps can fire the notification twice on + // some iPadOS versions, which would interleave modifier press/release + // chords on the host. 150 ms is comfortably below typical typing + // rhythm but above iPadOS double-fire window. + private var lastHotkeyAt: CFTimeInterval = 0 + var onLanguageChange: ((String?) -> Void)? + + init(host: UIViewController, keyForwardTarget: UIResponder) { + self.host = host + self.keyForwardTarget = keyForwardTarget + } + + func start() { + guard hiddenField == nil, let hostView = host?.view else { return } + + let field = LanguageSyncTextField(frame: CGRect(x: -100, y: -100, width: 1, height: 1)) + field.alpha = 0 + field.autocorrectionType = .no + field.autocapitalizationType = .none + field.spellCheckingType = .no + field.smartDashesType = .no + field.smartQuotesType = .no + field.smartInsertDeleteType = .no + field.keyForwardTarget = keyForwardTarget + hostView.addSubview(field) + field.becomeFirstResponder() + hiddenField = field + + // Defer seeding: textInputMode is sometimes still nil right after + // becomeFirstResponder. Asynchronously read on the next run-loop + // tick, by which point iOS has updated it. + DispatchQueue.main.async { [weak self] in + guard let self = self else { return } + if !self.hasSeenInitialLanguage { + self.lastLanguage = self.currentLanguage(from: nil) + self.hasSeenInitialLanguage = true + } + } + + observer = NotificationCenter.default.addObserver( + forName: UITextInputMode.currentInputModeDidChangeNotification, + object: nil, + queue: .main + ) { [weak self] note in + self?.handleChange(note: note) + } + } + + func stop() { + if let obs = observer { + NotificationCenter.default.removeObserver(obs) + observer = nil + } + hiddenField?.resignFirstResponder() + hiddenField?.removeFromSuperview() + hiddenField = nil + } + + // Step aside so the host view controller can become first responder + // (e.g. when the soft keyboard is shown via 3-finger tap). + func yieldFirstResponder() { + hiddenField?.resignFirstResponder() + } + + // Re-take first-responder status when the host VC has resigned (typically + // after the soft keyboard is dismissed). + func reclaimFirstResponder() { + hiddenField?.becomeFirstResponder() + } + + private func handleChange(note: Notification) { + let lang = currentLanguage(from: note) + // On user reports the previous initial-seed + 150 ms debounce gates + // were swallowing legitimate hotkey fires (the very first iPad layout + // switch after a session start in particular). Loosened: only skip if + // the language string actually hasn't changed. A stray duplicate + // Ctrl+Space at session start is harmless; missing every real switch + // is not. + guard lang != lastLanguage else { return } + lastLanguage = lang + hasSeenInitialLanguage = true + // Short debounce ONLY to coalesce iPadOS double-notification glitches + // — 50 ms is well under any human switching cadence. + let now = CACurrentMediaTime() + guard now - lastHotkeyAt > 0.05 else { return } + lastHotkeyAt = now + onLanguageChange?(lang) + } + + // Prefer the mode advertised by the notification, fall back to whatever + // the hidden field currently reports. Either may be nil if no responder + // accepts text input at the moment. + private func currentLanguage(from note: Notification?) -> String? { + if let mode = note?.object as? UITextInputMode, let lang = mode.primaryLanguage { + return lang + } + return hiddenField?.textInputMode?.primaryLanguage + } } diff --git a/OpenParsec/SceneDelegate.swift b/OpenParsec/SceneDelegate.swift index 6534d8f..50f6e49 100644 --- a/OpenParsec/SceneDelegate.swift +++ b/OpenParsec/SceneDelegate.swift @@ -41,18 +41,20 @@ class SceneDelegate: UIResponder, UIWindowSceneDelegate func sceneDidEnterBackground(_ scene: UIScene) { + guard ParsecBackgroundManager.shared.hasActiveConnection else { return } + var pipAttempted = false if #available(iOS 15.0, *) { - if ParsecBackgroundManager.shared.hasActiveConnection { - PictureInPictureManager.shared.startPiP() - pipAttempted = PictureInPictureManager.shared.isPiPActive || PictureInPictureManager.shared.isStarting - } + PictureInPictureManager.shared.startPiP() + pipAttempted = PictureInPictureManager.shared.isPiPActive || PictureInPictureManager.shared.isStarting } - if !pipAttempted && ParsecBackgroundManager.shared.hasActiveConnection { - ParsecBackgroundManager.shared.onShouldDisconnect?() + // PiP keeps the stream alive in the background; its own stop/fail + // callbacks own the disconnect from there. Without PiP, hold a short + // keep-alive window so a quick app-switch resumes instantly instead of + // forcing a full reconnect. + if !pipAttempted { + ParsecBackgroundManager.shared.beginBackgroundGrace() } - - ParsecBackgroundManager.shared.sceneDidEnterBackground() } } diff --git a/OpenParsec/SettingsHandler.swift b/OpenParsec/SettingsHandler.swift index fdfb5ea..9ac044b 100644 --- a/OpenParsec/SettingsHandler.swift +++ b/OpenParsec/SettingsHandler.swift @@ -9,16 +9,110 @@ struct SettingsHandler { @AppStorage("cursorMode") public static var cursorMode: CursorMode = .touchpad @AppStorage("cursorScale") public static var cursorScale: Double = 0.5 @AppStorage("mouseSensitivity") public static var mouseSensitivity: Double = 1.0 + // Non-linear acceleration applied on top of mouseSensitivity. 0 = pure + // linear (fastest gestures travel the same per-pixel as slow ones); up + // to 1.5 = strong macOS-style acceleration where fast flicks travel + // further. Surfaced as a slider in Settings → Interactivity. + @AppStorage("mouseAcceleration") public static var mouseAcceleration: Double = 0.0 + // Draw a local arrow cursor on top of the streamed video and skip + // rendering the host's own cursor — the local one tracks input + // immediately while the host's cursor visually lags by the network RTT. + @AppStorage("localCursorOverlay") public static var localCursorOverlay: Bool = false + @AppStorage("scrollSensitivity") public static var scrollSensitivity: Double = 1.0 + // When true (default), trackpad scroll direction matches what iPad/macOS + // call "natural scrolling" — swipe down moves content down. Flip to false + // if you want classic mouse-wheel direction. + @AppStorage("naturalScrolling") public static var naturalScrolling: Bool = true + // S03: client-side scroll inertia was removed (iPadOS provides native + // scroll-deceleration events; the client tail double-applied them). The + // scrollMomentum / scrollMomentumStrength settings are gone with it. + // Best-effort iPadOS system-shortcut capture via UIKeyCommand registry + // with .wantsPriorityOverSystemBehavior (iOS 15+). Catches Cmd+letter + // combinations the iPad shell would otherwise eat. Cmd+Space, Cmd+H, + // Globe key, and swipe-up gestures stay system-level — those cannot be + // intercepted from a sandboxed iPad app. + @AppStorage("captureSystemKeys") public static var captureSystemKeys: Bool = true + // When streaming TO a Windows host from a Mac-style iPad keyboard, swap + // the GUI ↔ Ctrl scan codes so Cmd+C (the iPad user's expectation) + // arrives as Ctrl+C on the host, Ctrl+anything arrives as Win+anything, + // and Opt stays Alt (it's the same physical key + same Windows mapping). + @AppStorage("windowsHostKeyboardRemap") public static var windowsHostKeyboardRemap: Bool = false @AppStorage("noOverlay") public static var noOverlay: Bool = false - @AppStorage("cursorScale") public static var hideStatusBar: Bool = true + // Q1: previously shared the "cursorScale" key with the Double cursorScale + // above — a Bool write coerced cursorScale (false → 0 = invisible cursor) + // and vice versa. Own key now; migrateLegacyStatusBarKeyIfNeeded() seeds it + // and clamps any corrupted cursorScale on first launch after the upgrade. + @AppStorage("hideStatusBar") public static var hideStatusBar: Bool = true @AppStorage("rightClickPosition") public static var rightClickPosition: RightClickPosition = .firstFinger - @AppStorage("preferredFramesPerSecond") public static var preferredFramesPerSecond: Int = 60 // 0 = use device max (ProMotion) + @AppStorage("preferredFramesPerSecond") public static var preferredFramesPerSecond: Int = 0 // 0 = use device max (ProMotion). Default was 60 — that capped 120 Hz iPads at half their refresh, doubling glass-to-glass present latency. @AppStorage("decoderCompatibility") public static var decoderCompatibility: Bool = false // Enable for stutter issues on some devices + // Umbrella switch — when true, suppresses the artificial 20 ms / 60 ms + // holds on the captured-key path and the unconditional PiP capture in + // the render loop. Surfaced as a single Settings toggle that also flips + // FPS / decoder / overlay / momentum defaults via onChange. + @AppStorage("lowLatencyMode") public static var lowLatencyMode: Bool = false @AppStorage("showKeyboardButton") public static var showKeyboardButton: Bool = true + // When the iPad's hardware keyboard layout changes (Caps Lock / Ctrl+Space + // on Magic Keyboard), fire a hotkey at the host so the host's input source + // switches in lock-step. Eliminates the "wrong characters after switching + // language" problem people hit on iPad ↔ Mac Parsec sessions. + @AppStorage("syncKeyboardLayout") public static var syncKeyboardLayout: Bool = true + @AppStorage("layoutSyncHotkey") public static var layoutSyncHotkey: LayoutSyncHotkey = .ctrlSpace + + // When true, pressing Ctrl+Shift *alone* (no other key in between) fires + // Cmd+Space at the host — the macOS "switch input source" / Spotlight chord + // the iPad shell otherwise swallows. Holding Ctrl+Shift and then pressing + // any other key (e.g. Ctrl+Shift+Arrow to extend a selection) is forwarded + // normally and does NOT fire the emulation. Off by default; intended for Mac + // hosts. Not auto-gated on host OS yet because host-OS detection (S04) still + // resolves to .unknown until the Int→OS mapping is discovered empirically — + // so this stays a deliberate manual opt-in. + @AppStorage("ctrlShiftEmulatesCmdSpace") public static var ctrlShiftEmulatesCmdSpace: Bool = false + + // When true, pressing a *bare* backtick/grave (`) — no Cmd/Ctrl/Alt/Shift — + // fires Cmd+Space at the host instead of sending the grave scancode. This is + // a manual language-switch macro: the physical Cmd+Space is swallowed by + // iPadOS (Spotlight) and never reaches the host, but a backtick is an + // ordinary key we can intercept and re-emit as host scancodes. Shift+` + // (tilde) and Cmd+` are untouched because any modifier disqualifies the + // remap. Off by default; turning it on means you can no longer type a literal + // backtick into the host while connected. Intended for Mac hosts whose + // "Select previous input source" / Spotlight is bound to ⌘Space. + @AppStorage("backtickEmulatesCmdSpace") public static var backtickEmulatesCmdSpace: Bool = false + @AppStorage("saveSessionSettings") public static var saveSessionSettings: Bool = true @AppStorage("savedZoomEnabled") public static var savedZoomEnabled: Bool = false @AppStorage("savedConstantFps") public static var savedConstantFps: Bool = false @AppStorage("savedMuted") public static var savedMuted: Bool = false + // Remember which display the user picked last; restored on the next + // connect once the host enumerates its displays (user-data event 12). + // The id is the primary key; the name (e.g. "Built-in Retina Display ...") + // is a fallback because Parsec sometimes regenerates display ids between + // connects for the same physical display. + @AppStorage("savedDisplayOutput") public static var savedDisplayOutput: String = "" + @AppStorage("savedDisplayName") public static var savedDisplayName: String = "" + + // Q1 one-time migration. hideStatusBar used to (incorrectly) share the + // "cursorScale" UserDefaults key, so the two settings corrupted each other. + // On the first launch after the fix, seed the new "hideStatusBar" key with + // its original default and clamp any cursorScale value that a Bool write + // may have driven out of range (notably 0 = invisible cursor). Call once at + // app launch, before any UI reads these values. + static func migrateLegacyStatusBarKeyIfNeeded() { + let defaults = UserDefaults.standard + // Absence of the new key == migration not yet run. + guard defaults.object(forKey: "hideStatusBar") == nil else { return } + // The shared value can't be split back into two settings, so restore + // the original hideStatusBar default (true) and sanitize cursorScale. + defaults.set(true, forKey: "hideStatusBar") + if defaults.object(forKey: "cursorScale") != nil { + let raw = defaults.double(forKey: "cursorScale") + let clamped = min(max(raw, 0.1), 4.0) + if clamped != raw { + defaults.set(clamped, forKey: "cursorScale") + } + } + } } diff --git a/OpenParsec/SettingsView.swift b/OpenParsec/SettingsView.swift index 48d0017..129d247 100644 --- a/OpenParsec/SettingsView.swift +++ b/OpenParsec/SettingsView.swift @@ -11,13 +11,26 @@ struct SettingsView:View @AppStorage("cursorMode") var cursorMode: CursorMode = .touchpad @AppStorage("cursorScale") var cursorScale: Double = 0.5 @AppStorage("mouseSensitivity") var mouseSensitivity: Double = 1.0 + @AppStorage("mouseAcceleration") var mouseAcceleration: Double = 0.0 + @AppStorage("localCursorOverlay") var localCursorOverlay: Bool = false + @AppStorage("scrollSensitivity") var scrollSensitivity: Double = 1.0 + @AppStorage("naturalScrolling") var naturalScrolling: Bool = true + @AppStorage("captureSystemKeys") var captureSystemKeys: Bool = true + @AppStorage("windowsHostKeyboardRemap") var windowsHostKeyboardRemap: Bool = false @AppStorage("noOverlay") var noOverlay: Bool = false - @AppStorage("cursorScale") var hideStatusBar: Bool = true + @AppStorage("hideStatusBar") var hideStatusBar: Bool = true @AppStorage("rightClickPosition") var rightClickPosition: RightClickPosition = .firstFinger - @AppStorage("preferredFramesPerSecond") var preferredFramesPerSecond: Int = 60 // 0 = use device max (ProMotion) + @AppStorage("preferredFramesPerSecond") var preferredFramesPerSecond: Int = 0 // 0 = use device max (ProMotion) @AppStorage("decoderCompatibility") var decoderCompatibility: Bool = false // Enable for stutter issues on some devices + @AppStorage("lowLatencyMode") var lowLatencyMode: Bool = false @AppStorage("showKeyboardButton") var showKeyboardButton: Bool = true + @AppStorage("syncKeyboardLayout") var syncKeyboardLayout: Bool = true + @AppStorage("layoutSyncHotkey") var layoutSyncHotkey: LayoutSyncHotkey = .ctrlSpace + @AppStorage("ctrlShiftEmulatesCmdSpace") var ctrlShiftEmulatesCmdSpace: Bool = false + @AppStorage("backtickEmulatesCmdSpace") var backtickEmulatesCmdSpace: Bool = false @AppStorage("saveSessionSettings") var saveSessionSettings: Bool = true + @State private var crashCopied: Bool = false + @State private var diagCopied: Bool = false let resolutionChoices: [Choice] @@ -103,6 +116,11 @@ struct SettingsView:View Slider(value: $cursorScale, in:0.1...4, step:0.1) .frame(width: 200) Text(String(format: "%.1f", cursorScale)) + } + CatItem("Local Cursor Overlay") + { + Toggle("", isOn:$localCursorOverlay) + .frame(width:80) } CatItem("Mouse Sensitivity") { @@ -110,6 +128,64 @@ struct SettingsView:View .frame(width: 200) Text(String(format: "%.1f", mouseSensitivity)) } + CatItem("Mouse Acceleration") + { + Slider(value: $mouseAcceleration, in:0...1.5, step:0.05) + .frame(width: 200) + Text(String(format: "%.2f", mouseAcceleration)) + } + CatItem("Scroll Sensitivity") + { + Slider(value: $scrollSensitivity, in:0.1...4, step:0.1) + .frame(width: 200) + Text(String(format: "%.1f", scrollSensitivity)) + } + CatItem("Natural Scrolling") + { + Toggle("", isOn:$naturalScrolling) + .frame(width:80) + } + } + CatTitle("Keyboard") + CatList() + { + CatItem("Sync layout with host") + { + Toggle("", isOn:$syncKeyboardLayout) + .frame(width:80) + } + CatItem("Layout switch hotkey") + { + MultiPicker(selection:$layoutSyncHotkey, options: + [ + Choice("⌃ + Space (macOS default)", LayoutSyncHotkey.ctrlSpace), + Choice("⌃ + ⇧ (macOS alt)", LayoutSyncHotkey.ctrlShift), + Choice("⌘ + Space", LayoutSyncHotkey.cmdSpace), + Choice("⌥ + Space", LayoutSyncHotkey.altSpace), + Choice("Alt + Shift (Windows)", LayoutSyncHotkey.altShift), + Choice("Off", LayoutSyncHotkey.none) + ]) + } + CatItem("⌃⇧ → ⌘Space (Mac host)") + { + Toggle("", isOn:$ctrlShiftEmulatesCmdSpace) + .frame(width:80) + } + CatItem("` key → ⌘Space (language switch)") + { + Toggle("", isOn:$backtickEmulatesCmdSpace) + .frame(width:80) + } + CatItem("Capture System Shortcuts") + { + Toggle("", isOn:$captureSystemKeys) + .frame(width:80) + } + CatItem("Windows Host Remap") + { + Toggle("", isOn:$windowsHostKeyboardRemap) + .frame(width:80) + } } CatTitle("Graphics") CatList() @@ -150,6 +226,27 @@ struct SettingsView:View Toggle("", isOn:$decoderCompatibility) .frame(width:80) } + CatItem("Low Latency Mode") + { + Toggle("", isOn:$lowLatencyMode) + .frame(width:80) + .onChange(of: lowLatencyMode) { newValue in + if newValue { + // S08 — Poor-Network profile. Strip the latency-adding + // knobs AND cap bitrate: on a weak uplink an uncapped + // encoder outruns the link, and the resulting queue + // (bufferbloat) is what actually makes a stream feel + // laggy on bad WiFi. This trades sharpness for + // responsiveness — each knob stays individually + // overridable afterward. + preferredFramesPerSecond = 0 // device-max present rate + decoder = .h265 // more quality per capped bit + noOverlay = true + decoderCompatibility = false // compat decode path adds latency + bitrate = 5 // poor-network ceiling (Mbps) + } + } + } } CatTitle("Misc") CatList() @@ -174,6 +271,36 @@ struct SettingsView:View Toggle("", isOn:$saveSessionSettings) .frame(width:80) } + CatItem("Last Crash Log") + { + Button(action: { + if let crash = CrashReporter.peek() { + UIPasteboard.general.string = crash + crashCopied = true + } else { + UIPasteboard.general.string = "(no crash recorded)" + crashCopied = true + } + }) { + Text(crashCopied ? "Copied!" : (CrashReporter.peek() == nil ? "None" : "Copy")) + .foregroundColor(CrashReporter.peek() == nil ? .gray : Color("AccentColor")) + } + } + CatItem("Diagnostics Log") + { + Button(action: { + if let diag = Diagnostics.peek() { + UIPasteboard.general.string = diag + diagCopied = true + } else { + UIPasteboard.general.string = "(no diagnostics recorded)" + diagCopied = true + } + }) { + Text(diagCopied ? "Copied!" : (Diagnostics.peek() == nil ? "None" : "Copy")) + .foregroundColor(Diagnostics.peek() == nil ? .gray : Color("AccentColor")) + } + } } Text(getVersionInfo()) .multilineTextAlignment(.center) @@ -200,7 +327,10 @@ struct SettingsView:View } func getVersionInfo() -> String { - return "Version \(Bundle.main.infoDictionary!["CFBundleShortVersionString"] ?? "Unknown versino")-\(Bundle.main.infoDictionary!["GitCommitInfo"] ?? "Unknown commit")" + let info = Bundle.main.infoDictionary + let version = info?["CFBundleShortVersionString"] as? String ?? "Unknown version" + let commit = info?["GitCommitInfo"] as? String ?? "Unknown commit" + return "Version \(version)-\(commit)" } } diff --git a/OpenParsec/Shared.swift b/OpenParsec/Shared.swift index 24af858..79094de 100644 --- a/OpenParsec/Shared.swift +++ b/OpenParsec/Shared.swift @@ -76,7 +76,11 @@ class SharedModel: ObservableObject { @Published var constantFps = false @Published var output = "none" @Published var displayConfigs: [ParsecDisplayConfig] = [] - + // Host OS as reported in the case-11 video config. -1 = unknown / not yet + // received. The Int→OS encoding is empirical (see HostOS.from); this raw + // value is surfaced for discovery logging and feature gating. + @Published var hostOS: Int = -1 + } class DataManager { diff --git a/OpenParsec/ViewContainerPatch.swift b/OpenParsec/ViewContainerPatch.swift index e94bb19..38f0274 100644 --- a/OpenParsec/ViewContainerPatch.swift +++ b/OpenParsec/ViewContainerPatch.swift @@ -29,32 +29,39 @@ fileprivate extension NSObject { /// We need to set these when the VM starts running since there is no way to do it from SwiftUI right now extension UIViewController { - private static var _childForHomeIndicatorAutoHiddenStorage: [UIViewController: UIViewController] = [:] - + // Q4: these used `[UIViewController: UIViewController]`, which strongly + // retains BOTH the parent VC (key) and the ParsecViewController (value). + // If teardown's nil-clear didn't run (abnormal disconnect), the entry — and + // with it the GLKView + every gesture recognizer hanging off the Parsec VC — + // leaked for the lifetime of the process, accumulating one set per reconnect. + // NSMapTable.weakToWeakObjects() holds neither side, so a dropped VC + // deallocates and its entry auto-empties even without an explicit clear. + private static let _childForHomeIndicatorAutoHiddenStorage = NSMapTable.weakToWeakObjects() + @objc private dynamic var _childForHomeIndicatorAutoHidden: UIViewController? { - Self._childForHomeIndicatorAutoHiddenStorage[self] + Self._childForHomeIndicatorAutoHiddenStorage.object(forKey: self) } - + @objc dynamic func setChildForHomeIndicatorAutoHidden(_ value: UIViewController?) { if let value = value { - Self._childForHomeIndicatorAutoHiddenStorage[self] = value + Self._childForHomeIndicatorAutoHiddenStorage.setObject(value, forKey: self) } else { - Self._childForHomeIndicatorAutoHiddenStorage.removeValue(forKey: self) + Self._childForHomeIndicatorAutoHiddenStorage.removeObject(forKey: self) } setNeedsUpdateOfHomeIndicatorAutoHidden() } - - private static var _childViewControllerForPointerLockStorage: [UIViewController: UIViewController] = [:] - + + private static let _childViewControllerForPointerLockStorage = NSMapTable.weakToWeakObjects() + @objc private dynamic var _childViewControllerForPointerLock: UIViewController? { - Self._childViewControllerForPointerLockStorage[self] + Self._childViewControllerForPointerLockStorage.object(forKey: self) } - + @objc dynamic func setChildViewControllerForPointerLock(_ value: UIViewController?) { if let value = value { - Self._childViewControllerForPointerLockStorage[self] = value + Self._childViewControllerForPointerLockStorage.setObject(value, forKey: self) } else { - Self._childViewControllerForPointerLockStorage.removeValue(forKey: self) + Self._childViewControllerForPointerLockStorage.removeObject(forKey: self) } setNeedsUpdateOfPrefersPointerLocked() } diff --git a/OpenParsec/audio.c b/OpenParsec/audio.c index 7dae111..c7936bc 100644 --- a/OpenParsec/audio.c +++ b/OpenParsec/audio.c @@ -213,14 +213,22 @@ void audio_destroy(struct audio **ctx_out) if (ctx->audio_buf[x]) AudioQueueFreeBuffer(ctx->q, ctx->audio_buf[x]); } - + + // C5 fix: free the shared silence buffer while the queue is still alive + // and BEFORE freeing ctx. The old order was free(ctx) -> *ctx_out = NULL + // -> AudioQueueFreeBuffer(ctx->q, ...), which dereferenced the just-freed + // ctx (use-after-free) and freed the buffer on an already-disposed queue. + if (ctx->q && silence_buf) { + AudioQueueFreeBuffer(ctx->q, silence_buf); + silence_buf = NULL; + } + if (ctx->q) AudioQueueDispose(ctx->q, true); free(ctx); *ctx_out = NULL; isStart = false; - AudioQueueFreeBuffer(ctx->q, silence_buf); silence_inqueue = silence_outqueue = 0; } @@ -271,8 +279,17 @@ void audio_cb(const int16_t *pcm, uint32_t frames, void *opaque) return; } - memcpy((*find_idle)->mAudioData, pcm, frames * 4); - (*find_idle)->mAudioDataByteSize = frames * 4; + // The idle buffer is a fixed BUFFER_SIZE (4096-byte) AudioQueue + // allocation, but `frames` is host-supplied and unbounded. Normal ~10ms + // chunks are well under 1024 stereo frames, so this clamp is a no-op on + // the happy path — but a post-stall recovery on a flaky link can deliver + // an oversized chunk, and `frames * 4 > BUFFER_SIZE` would memcpy past the + // buffer (heap overflow → crash/corruption). Clamp to what fits; playing a + // truncated chunk is strictly better than overrunning the allocation. + uint32_t bytes = frames * 4; + if (bytes > BUFFER_SIZE) bytes = BUFFER_SIZE; + memcpy((*find_idle)->mAudioData, pcm, bytes); + (*find_idle)->mAudioDataByteSize = bytes; if(!isStart) {