refactor(desktop): realtime hub as default voice path + revert pill UI#8002
refactor(desktop): realtime hub as default voice path + revert pill UI#8002vendz wants to merge 16 commits into
Conversation
…r from Voice Model RealtimeHubSettings no longer stores an enable flag or its own provider; the hub is the default voice path and its provider follows the existing Advanced 'Voice Model' picker (RealtimeOmniSettings). Drops the now-unused subtitle and CaseIterable conformance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… BYOK predicate isActive/ensureWarm/reconnect no longer gate on the removed isEnabled toggle. isActive now consults RealtimeHubSettings.canConnect (single source). Removes the floating-bar pill flag writes (isVoiceThinking/Speaking/voiceLevel) and the orphaned AVSpeech delegate; native-audio playback is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… picker The hub is always-on and follows the existing Voice Model picker, so the duplicate enable+provider cards are removed. Default Voice Model set to OpenAI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Users can still switch to Gemini or Auto in Advanced settings; this default also drives the realtime hub provider. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ctivity observer Keeps the QueryTracer wiring, router-skip/screenshot heuristics, and recordVoiceTurn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mixer RMS tap + pending-buffer tracking only fed the reverted speaking waveform; playback (enqueue/stop/engine-restart/config-rebuild) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…dev-only) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pill entry Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR follows up on the prior “realtime-as-hub” work by making the Realtime Hub the default push-to-talk voice path (with fallback to the legacy STT cascade when the hub can’t connect), and reverting the floating bar voice UI back to the pre-redesign visual/behavior.
Changes:
- Default the “Voice Model” setting to GPT Realtime 2 and route hub provider selection through that single picker (removing hub enable/provider UI).
- Revert floating bar voice UI (delete
VoiceActivityIndicator, remove thinking/speaking state plumbing, collapse on PTT release). - Simplify audio playback plumbing by removing the output-level tap and “playing” edge callbacks from
StreamingPCMPlayer.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| desktop/macos/Desktop/Sources/RealtimeOmni/RealtimeOmniSettings.swift | Default provider now GPT Realtime 2 via UserDefaults registration. |
| desktop/macos/Desktop/Sources/MainWindow/Pages/SettingsPage.swift | Removes hub toggle/provider settings; defaults Voice Model picker to GPT Realtime 2. |
| desktop/macos/Desktop/Sources/FloatingControlBar/VoiceActivityIndicator.swift | Deletes the pill redesign indicator implementation. |
| desktop/macos/Desktop/Sources/FloatingControlBar/StreamingPCMPlayer.swift | Removes level tap + playback state callbacks; simplifies buffer scheduling/stop. |
| desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSettings.swift | Makes hub provider derived from RealtimeOmniSettings (no separate picker). |
| desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubSession.swift | Updates session documentation to reflect BYOK vs ephemeral token auth. |
| desktop/macos/Desktop/Sources/FloatingControlBar/RealtimeHubController.swift | Removes enable-gating; simplifies UI state handling for hub turns. |
| desktop/macos/Desktop/Sources/FloatingControlBar/PushToTalkManager.swift | Collapses bar on hub commit; reintroduces imperative resize control. |
| desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarWindow.swift | Removes voice-activity observer; relies on listening state + explicit resizes. |
| desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift | Reverts to thin idle bar + listening UI with pulsing mic. |
| desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarState.swift | Removes thinking/speaking/level state and derived isVoiceActive/voiceActivity. |
| desktop/macos/CHANGELOG.json | Updates unreleased note to reflect the new (default) voice hub behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| private init() {} | ||
|
|
||
| /// The hub provider follows the user's "Voice Model" choice in Advanced settings — | ||
| /// there is no separate hub picker. The two map 1:1 (same underlying models), and | ||
| /// `.auto` is already resolved to a concrete provider by `effectiveProvider`. | ||
| var provider: RealtimeHubProvider { | ||
| get { | ||
| let raw = UserDefaults.standard.string(forKey: providerKey) | ||
| return raw.flatMap(RealtimeHubProvider.init(rawValue:)) ?? .openai | ||
| } | ||
| set { | ||
| UserDefaults.standard.set(newValue.rawValue, forKey: providerKey) | ||
| NotificationCenter.default.post(name: .realtimeHubSettingsDidChange, object: nil) | ||
| switch RealtimeOmniSettings.shared.effectiveProvider { | ||
| case .gptRealtime2: return .openai | ||
| case .geminiFlashLive, .auto: return .gemini | ||
| } | ||
| } |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26c445b7c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| switch RealtimeOmniSettings.shared.effectiveProvider { | ||
| case .gptRealtime2: return .openai | ||
| case .geminiFlashLive, .auto: return .gemini |
There was a problem hiding this comment.
With the hub provider now derived from RealtimeOmniSettings.shared.effectiveProvider, changing Advanced → Voice Model no longer notifies the hub: the picker only mutates @AppStorage/refreshes Auto, while RealtimeHubController.setup still observes only .realtimeHubSettingsDidChange. For managed non-BYOK users this leaves the old warm session in place; after switching providers, isActive becomes false because sessionProvider no longer matches and canConnect is false, so every PTT turn falls back to the legacy cascade until something else restarts/rewarms the hub. Please post/observe the realtime omni settings change and reconnect the hub when this provider source changes.
Useful? React with 👍 / 👎.
| } else if !barState.isVoiceListening && wasListening { | ||
| FloatingControlBarManager.shared.resizeForPTT(expanded: false) |
There was a problem hiding this comment.
If a proactive notification is visible while the user is holding PTT, releasing PTT takes this collapse branch and resizes the panel to the tiny PTT sliver even though currentNotification is still set. The notification presenter allows this state, and the previous reactive collapse guarded currentNotification == nil, so this regression leaves the notification mounted but clipped until it times out/dismisses. Skip the PTT collapse, or resize back to the notification dimensions, when barState.currentNotification is non-nil.
Useful? React with 👍 / 👎.
Greptile SummaryThis PR makes the Realtime Voice Hub the unconditional default voice path (removing the opt-in toggle), reverts the floating-bar pill UI to the pre-redesign look, and removes the supporting state machinery (
Confidence Score: 3/5Two issues in the controller need fixing before the always-on hub ships cleanly to all users. The notification wiring between RealtimeOmniSettings and RealtimeHubController is broken: changing the Voice Model in settings has no runtime effect for BYOK users because the hub listens for a notification that is now never posted. Separately, if the hub session errors while the user is still holding PTT, the bar is left expanded because exitVoiceUI() clears the listening flag without triggering a resize, and the imperative resize path in updateBarState() then sees no transition to act on. Both are in the hot path of the feature this PR is promoting to default-on. RealtimeHubController.swift (notification registration and exitVoiceUI resize gap) and RealtimeHubSettings.swift (the notification name that is now dead). Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant User
participant PTTManager as PushToTalkManager
participant HubCtrl as RealtimeHubController
participant HubSettings as RealtimeHubSettings
participant OmniSettings as RealtimeOmniSettings
participant Session as RealtimeHubSession
participant Bar as FloatingBarWindow
User->>PTTManager: PTT press
PTTManager->>HubCtrl: isActive?
HubCtrl->>HubSettings: canConnect (BYOK key?)
HubSettings->>OmniSettings: effectiveProvider
OmniSettings-->>HubSettings: .gptRealtime2 or .geminiFlashLive
HubCtrl-->>PTTManager: "true (BYOK) or session!=nil (managed)"
PTTManager->>HubCtrl: beginTurn()
PTTManager->>Bar: updateBarState() resizeForPTT expanded true
User->>PTTManager: PTT release
PTTManager->>HubCtrl: commitTurn()
PTTManager->>Bar: updateBarState() resizeForPTT expanded false
Session-->>HubCtrl: hubDidReceiveAudio pcm24k
HubCtrl->>HubCtrl: pcmPlayer.enqueue
Session-->>HubCtrl: hubDidFinishTurn
HubCtrl->>Bar: exitVoiceUI isVoiceListening already false
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant User
participant PTTManager as PushToTalkManager
participant HubCtrl as RealtimeHubController
participant HubSettings as RealtimeHubSettings
participant OmniSettings as RealtimeOmniSettings
participant Session as RealtimeHubSession
participant Bar as FloatingBarWindow
User->>PTTManager: PTT press
PTTManager->>HubCtrl: isActive?
HubCtrl->>HubSettings: canConnect (BYOK key?)
HubSettings->>OmniSettings: effectiveProvider
OmniSettings-->>HubSettings: .gptRealtime2 or .geminiFlashLive
HubCtrl-->>PTTManager: "true (BYOK) or session!=nil (managed)"
PTTManager->>HubCtrl: beginTurn()
PTTManager->>Bar: updateBarState() resizeForPTT expanded true
User->>PTTManager: PTT release
PTTManager->>HubCtrl: commitTurn()
PTTManager->>Bar: updateBarState() resizeForPTT expanded false
Session-->>HubCtrl: hubDidReceiveAudio pcm24k
HubCtrl->>HubCtrl: pcmPlayer.enqueue
Session-->>HubCtrl: hubDidFinishTurn
HubCtrl->>Bar: exitVoiceUI isVoiceListening already false
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| } | ||
| } | ||
|
|
||
| /// Resize window for PTT state (expanded when listening, compact circle when idle). | ||
| /// Expand snaps (animated:false) so the indicator + text never flash cramped while the | ||
| /// window grows; collapse animates for a smooth shrink back to the resting sliver. | ||
| func resizeForPTTState(expanded: Bool, animated: Bool = true) { | ||
| /// Resize window for PTT state (expanded when listening, compact circle when idle) | ||
| func resizeForPTTState(expanded: Bool) { | ||
| let size = expanded | ||
| ? NSSize(width: FloatingControlBarWindow.expandedWidth, height: FloatingControlBarWindow.expandedBarSize.height) | ||
| : FloatingControlBarWindow.minBarSize | ||
| // Idempotent: skip when already at the target size (avoids a no-op resize). | ||
| if abs(frame.width - size.width) < 1, abs(frame.height - size.height) < 1 { return } | ||
| resizeAnchored(to: size, makeResizable: false, animated: animated) | ||
| resizeAnchored(to: size, makeResizable: false, animated: true) |
There was a problem hiding this comment.
The old signature was resizeForPTTState(expanded: Bool, animated: Bool = true) and applyVoiceExpansion called it with animated: false on expand, so the window snapped open before content rendered. The new signature always passes animated: true. Given the simpler reverted UI this is likely imperceptible, but the deliberate snap-on-expand choice is now gone without a substitute.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…turn hub exit - Observe .realtimeOmniSettingsDidChange (the live 'voice model changed' signal) instead of the dead .realtimeHubSettingsDidChange, so switching the Voice Model re-warms the hub on the newly selected provider for BYOK users. - exitVoiceUI() now collapses the bar when it clears isVoiceListening (mid-turn error / silent-tap cancel) — the transition-based updateBarState() would otherwise see no change and leave the bar expanded. Guarded against conversation/response/ notification/hover/onboarding. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nothing posts it after the hub toggle was removed; the controller now listens to .realtimeOmniSettingsDidChange instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The picker writes @AppStorage directly (bypassing the setter), so post the change ourselves — this re-warms the realtime hub on the newly selected provider. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1793120ba1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let provider = RealtimeHubSettings.shared.provider | ||
| if APIKeyService.byokKey(provider.byokProvider) != nil { return true } | ||
| return session != nil && sessionProvider == provider | ||
| if RealtimeHubSettings.shared.canConnect { return true } |
There was a problem hiding this comment.
Now that the hub no longer has an opt-in toggle, treating any stored BYOK value as active means users with a stale/revoked OpenAI or Gemini key never fall back to the legacy omni/STT path. After the hub socket fails authentication, isActive still returns true solely because the key exists, so every PTT press enters hub mode and attempts the failing realtime session instead of using the cascade that managed-token failures get.
Useful? React with 👍 / 👎.
… cascade fallback) isActive returned true whenever a BYOK key existed, so a stale/revoked OpenAI or Gemini key kept entering hub mode on every PTT and lost the turn to a failing realtime session (while managed-token failures already fell back). Now isActive requires a live, authenticated session for the selected provider (new hubConnected flag, set on the post-auth connect, cleared on teardown). A key/token that can't connect never routes a turn to the hub — PTT transparently uses the legacy cascade, so a broken hub never costs the user a turn. The hub re-warms in the background and resumes taking turns the moment it connects (also covers provider switches and idle-close reconnect windows). Reconnect churn is still capped via a named maxReconnectStrikes constant. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add get_memories, search_memories, search_conversations, get_conversations, create_action_item and update_action_item to the floating-bar voice agent so it answers "who am I / what do you know about me" and "most recent conversation" instead of refusing. Reads are synchronous and spoken; the system prompt no longer claims it can't see personal data and routes recency vs semantic vs spawn_agent. Tool dicts are static `let` (built once, not per reconnect). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the new HubTool cases in hubDidRequestTool to the existing APIClient.tool* endpoints (capped for voice: top 3/5, no transcripts). Reads/writes share one runToolAndSpeak helper (Task / do-catch / empty-fallback / log / sendToolResult) and a small arg() accessor; get_tasks output carries [id:…] so update_action_item can target a task. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Keep the harness tool-result switch exhaustive for the six new HubTool cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
What
Follow-up cleanup to the merged realtime-hub PR (#7957): make the realtime voice hub the default voice path and revert the floating-bar pill redesign.
Changes
realtimeHubEnabledopt-in. Every PTT turn routes through the hub whenever it can connect (BYOK key, or a server-minted ephemeral token for managed users); otherwise it falls back to the legacy STT cascade.VoiceActivityIndicator. Kept the agent voice-reply mic button (independent)./simplify): dropped the orphaned audio level-tap fromStreamingPCMPlayer, consolidated the hub's BYOK readiness intocanConnect, removed unusedRealtimeHubProvider.subtitle/CaseIterableand the empty AVSpeech delegate.Net: +116 / −665 across 12 files.
Verification
Deploy note
Managed (non-BYOK) Gemini users need
GEMINI_API_KEYset on the Rust desktop-backend (same asOPENAI_API_KEY); otherwise selecting Gemini falls back to STT.🤖 Generated with Claude Code