UltraClicky

An open-source, local AI agent that lives in your menu bar, sees your screen, and drives your Mac. Summon it next to your cursor, type a request, and it either answers + points at the screen (tutor mode) or does the task (agent mode) by controlling the Mac itself.

There are no cloud connectors (no Notion, Gmail, Calendar, etc.). The agent's only powers are: see the screen, and drive this Mac. The loop runs against your own OpenAI key through a speech-to-speech Realtime session when voice is active.

macOS 13+ · Swift · AppKit + SwiftUI · built with XcodeGen.

---

Setup — the only three things you do

1. Paste your OpenAI key into a local .env file in the project root:

   api-key='<your-openai-api-key>'
   

   (Copy .env.example to .env if you don't have one. Never commit your real key.)

2. Generate the project, then build and run:

   brew install xcodegen        # if you don't have it
   xcodegen generate            # creates Clicky.xcodeproj from project.yml
   open Clicky.xcodeproj         # then build & run (⌘R) in Xcode

   …or from the command line:

   xcodebuild -project Clicky.xcodeproj -scheme Clicky -configuration Debug build

3. On first launch, grant the three macOS permissions when prompted. The setup window lists each one with a button that deep-links to the right System Settings pane:

   • Screen Recording — so the agent can see what you see.
   • Accessibility — so it can read the UI and act in place (without moving your cursor).
   • Microphone — only for voice input from the mic button.

   These prompts require manual approval in System Settings and cannot be skipped or pre-granted by code. The setup window re-checks status when you return to it.

If api-key is missing or empty at launch, the setup window shows a clear message pointing you to .env. The key is only ever read from local runtime configuration — it is never hardcoded.

.env is ignored by Git and is not copied into the app bundle. If you launch the app from somewhere that cannot find the project-root .env, point at a specific file with the CLICKY_ENV_PATH environment variable.

---

How to use it

• Summon: press ⌃⌥ (Control + Option). A floating, semi-transparent input panel appears near your cursor. Press Esc to dismiss it.
• Pick a mode with the toggle:
  • Do it (agent) — Clicky executes actions to complete your request.
  • Show me (tutor) — read-only. Clicky explains and points at the relevant UI with an animated overlay, but never clicks, types, or changes anything. This is enforced in code: in tutor mode only screenshot, point_at, and done are allowed; click/type/press_key are rejected before execution.
• Watch the action log fill in as it works ("Clicked Export", "Typed filename", …).
• Hands-free voice: UltraClicky listens for speech, stops on silence, streams the segment into the Realtime session, and submits it through the selected mode. It pauses while acting or speaking, then resumes listening.
• Hold-to-talk: hold right ⌘ (Command), say your request, then release. UltraClicky submits that spoken segment through the selected mode and speaks the reply.
• Workflows: say something like "create workflow activate moon time to open Safari and go to x.com". Later, saying "activate moon time" runs that saved task.
• Kill switch: press ⌃⌥⎋ (Control + Option + Escape) at any time to instantly abort the loop, cancel the in-flight request, and clear pending actions.

How the loop works

On submit, Clicky captures the main display (ScreenCaptureKit), downscales it to ~1440px, JPEGs it, and sends it with your request and the tool definitions to the reasoning model. The model replies with exactly one tool call; Clicky executes it, captures a fresh screenshot, appends the result to the full message history, and sends the next turn — looping until the model calls done or it exhausts its step budget. The budget starts at 15 steps; if the model is still making progress it can call request_more_steps to extend it, up to a 40-step ceiling. Read-only steps (pointing, highlighting, remembering, reading the UI tree) reuse the previous screenshot instead of re-capturing, and app launches poll for readiness rather than sleeping a fixed interval.

The agent reasons through OpenAI's Realtime model (gpt-realtime-2), driven over a WebSocket using text, image input, and function calling (OpenAIRealtimeProvider). Completed microphone turns are first converted into text with gpt-transcribe; the resulting transcript becomes the same durable request the agent would receive from typing. The provider keeps a Realtime session open, appends new user/tool items, and reads back one forced tool call per agent step.

The control layer is cursor-preserving: it uses the Accessibility API to press elements and set text values in place, and only falls back to synthesized CGEvent mouse/keyboard input (which does move the cursor) when no Accessibility element is available.

---

Swapping the model

The reasoning model sits behind a ReasoningProvider protocol. The active provider is OpenAIRealtimeProvider (the Realtime API, gpt-realtime-2). A legacy OpenAIChatProvider is also included for experiments with /v1/chat/completions, but the app's active speech-to-speech path should stay on gpt-realtime-2.

You can override the model in .env:

model='gpt-realtime-2'

Memory compression is separate from the realtime reasoning model. UltraClicky keeps local memory under an estimated 16k-token budget and compresses it with gpt-5.4-mini by default:

memory-compression-model='gpt-5.4-mini'

The main agent prompt is structured for OpenAI prompt caching: stable role, tool-use rules, and guardrails stay at the start of the prompt, while variable runtime details like screen dimensions and memory are appended at the end.

---

Voice

UltraClicky starts a hands-free voice listener on launch when your API key and microphone permission are available. It records only speech-like segments, streams the raw 24kHz PCM16 audio into the Realtime session, waits for a committed-turn transcript, and sends that text with the current screenshot to gpt-realtime-2. The model speaks back while calling the same tools used by typed requests.

The microphone path cleans up audio automatically: a local adaptive gate checks the noise floor, speech-band energy, periodicity, and transient shape before a turn can start. A low-frequency filter and soft expander attenuate rumble, steady hiss, and short clicks before audio is buffered, so background noise does not become a request or get sent to the Realtime API.

A legacy VoiceProvider protocol and GPTRealtime2Provider skeleton are also included but gated behind a compile-time flag and not wired to any hotkey. The active voice path is already OpenAIRealtimeProvider in AppDelegate. To experiment with the older skeleton:

1. Set Features.voiceEnabled = true in Clicky/Sources/App/Features.swift.
2. Build out GPTRealtime2Provider.start(...) (the file documents the steps): open a realtime WebSocket session to gpt-realtime-2, stream mic audio, and emit the same tool calls into the agent loop. It reuses the same .env key and the same hardcoded system prompt (adapted into OpenAI's labeled realtime sections).

Note: the active realtime voice path streams microphone audio to OpenAI. The hidden kill switch still applies.

---

Safety, in short

• API key is only read from local runtime configuration — never compiled in, never committed, and never copied into the app bundle by the build.
• GitHub publishing: .env, .env.*, .envrc, built apps, archives, and debug symbols are ignored. Commit .env.example, not your real .env.
• Behavior guardrail: a hardcoded, non-editable system prompt (SystemPrompts.swift) is the primary control. It marks destructive/outbound actions (deleting data, emptying Trash, sending messages, purchases, entering credentials, changing System Settings, installing software, changing sharing/permissions) as RESTRICTED — the agent does them only if your typed request explicitly asks. It also treats anything it reads on screen as data, not as instructions.
• Backstop: a model-invisible kill switch (⌃⌥⎋), implemented purely in the app's input layer. It is never exposed as a tool, never mentioned in the prompt, and never sent to the model.
• No confirm-before-act: the agent acts immediately on each tool call. That is intentional — the guardrails and the kill switch are what keep it safe.

---

License

MIT. See LICENSE.

---

Project layout

project.yml                 XcodeGen spec (run `xcodegen generate`)
.env.example                template only; your real .env is ignored by Git
Clicky/
  Info.plist                usage-description keys, LSUIElement
  Clicky.entitlements       non-sandboxed; apple-events, mic, network
  Sources/
    main.swift              accessory app entry point
    App/                    AppDelegate, Features, ClickyError
    Config/                 EnvLoader (.env parser)
    Prompts/                SystemPrompts (hardcoded, not editable)
    Agent/                  AgentLoop, Tools, ReasoningProvider, OpenAIChatProvider
    Control/                MacController (AX + CGEvent), ScreenCapturer, KeyCodes
    Overlay/                OverlayController (point_at / flash)
    UI/                     SummonPanel/View/ViewModel, Setup window
    Input/                  HotkeyManager (summon + hidden kill switch)
    Permissions/            PermissionsManager
    Voice/                  Hands-free mic streaming + GPTRealtime2Provider (stub, off)