Skip to content

feat: enhance voice features with Smallest.ai integration and Jarvis wake functionality: #255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yashdev9274 merged 1 commit into from
Aug 6, 2026
Merged

feat: enhance voice features with Smallest.ai integration and Jarvis wake functionality: #255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ packages/
- Web app: `apps/web/.env` or `apps/.env.local`
- Required for auth: `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `NEXT_PUBLIC_BETTER_AUTH_URL`
- Required for DB: `DATABASE_URL`
- Voice/STT: `ELEVENLABS_API_KEY` (default provider), `STT_PROVIDER` (`elevenlabs`|`groq`), `ELEVENLABS_MODEL`, `STT_LANGUAGE`
- Voice/STT: `SMALLEST_API_KEY` (Smallest.ai Pulse STT — the only STT provider), `SMALLEST_MODEL`, `STT_LANGUAGE`. Voice TTS (spoken reply): `VOICE_REPLY` (`on`|`off`), `ELEVENLABS_VOICE_ID`, `ELEVENLABS_TTS_MODEL` (macOS `say` fallback when no key). Note `ELEVENLABS_API_KEY` doubles as both STT and TTS key historically; STT now always routes to Smallest.ai.
- Secrets: `INFISICAL_CLIENT_ID`, `INFISICAL_CLIENT_SECRET` (when Infisical is configured)

### Linting
Expand Down
43 changes: 43 additions & 0 deletions apps/supercode-cli/server/.env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,49 @@ GOOGLE_CSE_ID=""
# Firecrawl
FIRECRAWL_API_KEY=""

# ─── Voice / Speech-to-Text (STT) ──────────────
# Used by the CLI voice capture and POST /api/voice/transcribe.
# Voice capture uses Smallest.ai (Pulse STT) exclusively.
STT_PROVIDER="smallest"
STT_LANGUAGE="en"

# Smallest.ai (Pulse STT) — https://app.smallest.ai/dashboard/api-keys
SMALLEST_API_KEY=""
# "pulse-pro" (best English accuracy) or "pulse" (multilingual) by default
SMALLEST_MODEL="pulse-pro"
SMALLEST_LANGUAGE="en"

# ─── Voice / Text-to-Speech (TTS / spoken reply) ────
# The CLI speaks the assistant's answer aloud after a voice-triggered turn
# (Clicky-style). ElevenLabs TTS when ELEVENLABS_API_KEY is set, otherwise the
# built-in macOS `say` voice. Requires macOS for audio playback.
# Set VOICE_REPLY="off" to disable spoken replies entirely.
VOICE_REPLY="on"
# Voice used for ElevenLabs TTS
ELEVENLABS_VOICE_ID="21m00Tcm4TlvDq8ikWAM"
# "eleven_turbo_v2_5" (low latency) by default
ELEVENLABS_TTS_MODEL=""

# ─── Jarvis wake-work agent ─────────────────────
# ⚪ Optional. When voice input (or /jarvis) is triggered, the CLI launches
# these daily apps. WhatsApp opens in the native macOS desktop app; everything
# else opens in the browser named by JARVIS_BROWSER (default "Dia") rather than
# the OS default. Set the Slack/Linear URLs to your real workspace URLs;
# JARVIS_URLS is a comma-separated list of extra apps.
JARVIS_WAKE_PHRASE="jarvis wake up,jarvis wakeup,jarvis"
# Browser used to open the non-WhatsApp targets (macOS app name)
JARVIS_BROWSER="Dia"
# Native macOS app for WhatsApp (falls back to the URL when uninstalled)
JARVIS_WHATSAPP_APP="WhatsApp"
JARVIS_GITHUB_URL="https://github.com/yashdev9274/supercli"
JARVIS_WHATSAPP_URL="https://web.whatsapp.com"
JARVIS_SLACK_URL="https://join.slack.com/t/supercodeai/shared_invite/zt-43enen35h-glS1qR854YB~HUg2AIW0vg"
JARVIS_LINEAR_URL="https://linear.app/supercodeai/projects/all"
JARVIS_TWITTER_URL="https://x.com"
# New Warp terminal tab opened at this folder (uses Warp's new_tab deep link)
JARVIS_WARP_DIR="/Users/yashdewasthale/dev/saas/supercli"
JARVIS_URLS=""

# ─── Dodo Payments ─────────────────────────────
# 🟡 Required for paid-tier billing (checkout, webhooks, refunds)
DODO_PAYMENTS_API_KEY=""
Expand Down
24 changes: 17 additions & 7 deletions apps/supercode-cli/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,27 @@ The AI has access to file reading, searching, web fetching, and code execution t

### Voice Input

Voice capture requires `ffmpeg` and an STT provider API key.
Voice capture requires `ffmpeg` and a Smallest.ai API key.

| Env Var | Description | Default |
|---|---|---|
| `STT_PROVIDER` | STT provider (`elevenlabs` or `groq`) | `elevenlabs` |
| `ELEVENLABS_API_KEY` | ElevenLabs API key (required for ElevenLabs STT) | — |
| `ELEVENLABS_MODEL` | ElevenLabs model ID | `scribe_v1` |
| `GROQ_API_KEY` | Groq API key (required when `STT_PROVIDER=groq`) | — |
| `STT_LANGUAGE` | Transcription language | `en` |
| `SMALLEST_API_KEY` | Smallest.ai API key (https://app.smallest.ai/dashboard/api-keys) | — |
| `SMALLEST_MODEL` | Smallest.ai STT model | `pulse-pro` |
| `SMALLEST_LANGUAGE` | Transcription language | `en` |
Comment on lines +47 to +53

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the server-proxy transcription path.

Voice capture does not always require SMALLEST_API_KEY. canVoiceCapture() also permits an authenticated SUPERCODE_SERVER_URL proxy. Update this section to state that voice capture requires ffmpeg and either direct Smallest.ai credentials or a configured server proxy.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/supercode-cli/server/README.md` around lines 47 - 53, Update the voice
capture prerequisites section in the README to state that `ffmpeg` is required
along with either direct Smallest.ai credentials or an authenticated
`SUPERCODE_SERVER_URL` server proxy. Keep the existing environment-variable
table unchanged.


Press **Ctrl+Shift+V** during a chat session to start voice capture.
### Voice Reply (speaks the answer back)

After a voice-triggered turn, supercode reads the assistant's reply aloud using
ElevenLabs TTS when a key is available, otherwise macOS `say`. macOS only.

| Env | Description | Default |
|---|---|---|
| `VOICE_REPLY` | Enable spoken replies (`on`/`off`) | `on` |
| `ELEVENLABS_VOICE_ID` | ElevenLabs TTS voice | `21m00Tcm4TlvDq8ikWAM` |
| `ELEVENLABS_TTS_MODEL` | ElevenLabs TTS model | `eleven_turbo_v2_5` |

Press **Ctrl+V** (or **F2**) during a chat session to start voice capture. The
captured command is run as a normal agent turn, then the reply is spoken back.

## License

Expand Down
189 changes: 146 additions & 43 deletions apps/supercode-cli/server/src/cli/ai/chat/chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ import {
voiceCaptureFlow,
canVoiceCapture,
stopCapture,
speakText,
} from "src/voice/speech.ts"
import { isJarvisWake, runJarvisStart } from "src/voice/jarvis.ts"
import path from "node:path"
import { AtPicker, DragDropTracker } from "./at-picker.ts"
import {
Expand All @@ -112,6 +114,21 @@ async function getUserFromToken() {
return result.user
}

// Store the current user for feature gating
let currentUser: { id: string; name: string | null; email: string } | null = null

// Check if the current user is Yash Dewasthale (for feature gating)
function isYashDewasthale(): boolean {
if (!currentUser) return false
const name = currentUser.name?.toLowerCase() ?? ""
const email = currentUser.email?.toLowerCase() ?? ""
return (
name.includes("yash") && name.includes("dewasthale") ||
email === "yashdev.yvd@gmail.com" ||
email === "yash@supercode.ai"
)
}

export async function initConversation(userId: string, conversationId: string | null = null, mode = "chat") {
const thinking = createThinking("loading conversation")
const conversation = await getOrCreateConversation(conversationId, mode)
Expand Down Expand Up @@ -967,6 +984,10 @@ let stdinPrevWrapLines = 1
// transcribed text instead of wiping stdinInput to "".
let voiceJustCaptured = false

// Set when a voice capture auto-submits (Clicky-style) so the loop can speak
// the reply back once the assistant turn finishes. Consumed at most once.
let voiceAutoSubmitted = false

// Loaded skill context — injected as a system message so the AI uses it
// without pasting the full text into the user's input.
export let loadedSkillName: string | undefined
Expand Down Expand Up @@ -1309,48 +1330,7 @@ function stdinKeypress(_str: string, key: any) {
return
}

const resolve = stdinResolve
stdinResolve = null
// Clear slash list (content below input line)
for (let i = 0; i < slashListLines; i++) {
readline.moveCursor(process.stdout, 0, 1)
}
for (let i = 0; i < slashListLines; i++) {
readline.cursorTo(process.stdout, 0)
readline.clearLine(process.stdout, 0)
if (i < slashListLines - 1) {
readline.moveCursor(process.stdout, 0, -1)
}
}
slashListLines = 0
// Clear @ picker overlay
atPicker.close()
for (let i = 0; i < atListLines; i++) {
readline.moveCursor(process.stdout, 0, 1)
}
for (let i = 0; i < atListLines; i++) {
readline.cursorTo(process.stdout, 0)
readline.clearLine(process.stdout, 0)
if (i < atListLines - 1) {
readline.moveCursor(process.stdout, 0, -1)
}
}
atListLines = 0
// Clear drag-drop indicator
for (let i = 0; i < ddListLines; i++) {
readline.moveCursor(process.stdout, 0, 1)
}
for (let i = 0; i < ddListLines; i++) {
readline.cursorTo(process.stdout, 0)
readline.clearLine(process.stdout, 0)
if (i < ddListLines - 1) {
readline.moveCursor(process.stdout, 0, -1)
}
}
ddListLines = 0
ddTracker.clear()
process.stdout.write("\r\n")
resolve({ input: stdinInput, mode: stdinMode })
commitInput()
return
}

Expand Down Expand Up @@ -1566,6 +1546,11 @@ function stdinKeypress(_str: string, key: any) {
}

async function startVoiceCapture() {
if (!isYashDewasthale()) {
activeFooter?.setStatusMessage("⛭ Voice features are only available for Yash Dewasthale")
setTimeout(() => activeFooter?.setStatusMessage(""), 4000)
return
}
const check = canVoiceCapture()
if (!check.ok) {
const reason = check.reason ?? "unknown"
Expand All @@ -1575,24 +1560,117 @@ async function startVoiceCapture() {
}
const prevMode = voiceCaptureActive
voiceCaptureActive = true
activeFooter?.setStatusMessage("🎤 Recording... (voice key or Enter to stop)")
activeFooter?.setStatusMessage("🎤 Recording... (voice key or Enter to stop)")
try {
const text = await voiceCaptureFlow()
if (text) {
if (isJarvisWake(text)) {
// Wake-word caught — don't submit an agent turn, just start the
// workspace, speak a confirmation, and print the open summary.
const opened = await runJarvisAndSpeak()
process.stdout.write(
`\r\n ${chalk.hex(theme.green)("◆")} ${chalk.hex(theme.amber)("Jarvis")} woke — opening: ${opened.join(", ") || "none configured"}\r\n\n`,
)
if (stdinResolve) renderInput()
else voiceJustCaptured = true
return
}
stdinInput =
stdinInput.slice(0, stdinCursor) + text + " " + stdinInput.slice(stdinCursor)
stdinCursor += text.length + 1
slashSelected = -1
historyIndex = -1
if (stdinResolve) {
// Loop is idle awaiting input — Clicky-style: auto-submit the spoken
// command so the agent actually does the thing and speaks back.
voiceAutoSubmitted = true
commitInput()
Comment on lines +1583 to +1587

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Consume voice-submit state for non-agent slash commands.

Line 1586 sets voiceAutoSubmitted before parsing the captured input. If the input resolves to /jarvis, /help, /model, or another non-agent slash command, none of the response paths at Lines 2351, 2384, or 2402 clears the flag. The next typed agent response is then spoken unexpectedly.

Store the voice origin per submitted turn, or clear the flag in every slash-command branch that does not produce an assistant response.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/supercode-cli/server/src/cli/ai/chat/chat.ts` around lines 1583 - 1587,
Ensure the voice-submit state is consumed per submitted turn rather than leaking
into later responses: update the auto-submit flow around commitInput() and the
slash-command handling so /jarvis, /help, /model, and other non-agent commands
clear or otherwise isolate voiceAutoSubmitted, while preserving voice playback
for the current agent response.

} else {
// Agent is busy — just fill the input so the user can review/send later.
voiceJustCaptured = true
}
} else {
activeFooter?.setStatusMessage("🎤 No speech detected — press voice key to retry")
setTimeout(() => activeFooter?.setStatusMessage(""), 4000)
// Nothing was submitted — restore the input prompt in place.
if (stdinResolve) renderInput()
}
} catch (err) {
activeFooter?.setStatusMessage("⛭ Voice failed: " + (err instanceof Error ? err.message : err))
setTimeout(() => activeFooter?.setStatusMessage(""), 4000)
if (stdinResolve) renderInput()
} finally {
voiceCaptureActive = prevMode
}
}

// Wake Jarvis: launch the configured workspace targets in the default browser,
// speak a short confirmation, and report the status line for text. Returns the
// names that opened successfully.
async function runJarvisAndSpeak(): Promise<string[]> {
if (!isYashDewasthale()) {
return []
}
const { opened } = runJarvisStart()
const reply = opened.length
? `Jarvis online. Opening ${opened.slice(0, 3).join(", ")}${opened.length > 3 ? " and more" : ""}.`
: "Jarvis online, but no workspace apps are configured."
activeFooter?.setStatusMessage(`🤖 Jarvis online · ${opened.length} app${opened.length === 1 ? "" : "s"} opening`)
setTimeout(() => activeFooter?.setStatusMessage(""), 6000)
await speakText(reply)
return opened
}

// Commit whatever stdinInput currently holds as a submitted chat turn. Shared
// by the Enter key and the Clicky-style voice auto-execute path so both clear
// the input overlays (slash list, @ picker, drag-drop) and resolve the
// pending chatInput() promise identically.
function commitInput(): void {
const resolve = stdinResolve
if (!resolve) return
stdinResolve = null
// Clear slash list (content below input line)
for (let i = 0; i < slashListLines; i++) {
readline.moveCursor(process.stdout, 0, 1)
}
for (let i = 0; i < slashListLines; i++) {
readline.cursorTo(process.stdout, 0)
readline.clearLine(process.stdout, 0)
if (i < slashListLines - 1) {
readline.moveCursor(process.stdout, 0, -1)
}
}
slashListLines = 0
// Clear @ picker overlay
atPicker.close()
for (let i = 0; i < atListLines; i++) {
readline.moveCursor(process.stdout, 0, 1)
}
for (let i = 0; i < atListLines; i++) {
readline.cursorTo(process.stdout, 0)
readline.clearLine(process.stdout, 0)
if (i < atListLines - 1) {
readline.moveCursor(process.stdout, 0, -1)
}
}
atListLines = 0
// Clear drag-drop indicator
for (let i = 0; i < ddListLines; i++) {
readline.moveCursor(process.stdout, 0, 1)
}
for (let i = 0; i < ddListLines; i++) {
readline.cursorTo(process.stdout, 0)
readline.clearLine(process.stdout, 0)
if (i < ddListLines - 1) {
readline.moveCursor(process.stdout, 0, -1)
}
}
ddListLines = 0
ddTracker.clear()
process.stdout.write("\r\n")
resolve({ input: stdinInput, mode: stdinMode })
}

function ensureStdinHandler() {
const stdin = process.stdin
readline.emitKeypressEvents(stdin)
Expand Down Expand Up @@ -2048,6 +2126,17 @@ export async function chatLoop(
// (via setImmediate) so the text should already be visible.
voiceJustCaptured = true
process.stdout.write(`\r\n`)
} else if (result?.type === "jarvis") {
if (!isYashDewasthale()) {
process.stdout.write(
`\r\n ${chalk.hex(theme.red)("◆")} ${chalk.hex(theme.red)("Jarvis is only available for Yash Dewasthale")}\r\n\n`,
)
} else {
const opened = await runJarvisAndSpeak()
process.stdout.write(
`\r\n ${chalk.hex(theme.green)("◆")} ${chalk.hex(theme.amber)("Jarvis")} woke — opening: ${opened.join(", ") || "none configured"}\r\n\n`,
)
}
} else if (result?.type === "skills") {
if (result.skillName && result.message) {
loadedSkillName = result.skillName
Expand Down Expand Up @@ -2259,6 +2348,7 @@ export async function chatLoop(
if (result.content && result.content !== "(cancelled)") {
await addMessage(conversation.id, "assistant", result.content)
}
voiceAutoSubmitted = false
process.stdout.write(`\r\n ${chalk.hex(theme.amber)("◆")} cancelled\r\n`)
continue
}
Expand Down Expand Up @@ -2291,10 +2381,16 @@ export async function chatLoop(
footer,
)
if (agentResult.aborted) {
voiceAutoSubmitted = false
process.stdout.write(`\r\n ${chalk.hex(theme.amber)("◆")} cancelled\r\n`)
continue
}
await addMessage(conversation.id, "assistant", agentResult.content)
if (voiceAutoSubmitted) {
voiceAutoSubmitted = false
process.stdout.write("\r\n")
await speakText(agentResult.content)
}
lastUsage = agentResult.usage
lastElapsed = agentResult.elapsed
await maybeCompactConversation(conversation.id)
Expand All @@ -2303,6 +2399,11 @@ export async function chatLoop(
}

await addMessage(conversation.id, "assistant", result.content)
if (voiceAutoSubmitted) {
voiceAutoSubmitted = false
process.stdout.write("\r\n")
await speakText(result.content)
}

// Phase 8: in plan mode, persist the assistant's response to scratch
// so /plan execute can pick it up.
Expand All @@ -2326,6 +2427,7 @@ export async function chatLoop(
await maybeCompactConversation(conversation.id)
} catch (error: any) {
const errMsg = error?.message ?? "Unknown error"
voiceAutoSubmitted = false
process.stdout.write(`\r\n ${chalk.hex(theme.red)("◆")} ${chalk.hex(theme.red)(errMsg)}\r\n\n`)
} finally {
clearSkill()
Expand Down Expand Up @@ -2394,6 +2496,7 @@ export async function startChat(
console.log()

const user = await getUserFromToken()
currentUser = user
const conversation = await initConversation(user.id, conversationId, initialMode)

await chatLoop(aiProvider, conversation, workspaceInfo)
Expand Down
Loading
Loading