Realtime Captions STT

Real-time captions for the browser. Capture tab/screen audio, transcribe with Deepgram, translate with Google Cloud, and show a live overlay for multilingual accessibility.

What this repo includes

• Web app that captures tab/screen audio in the browser
• WebSocket backend that streams audio chunks for transcription and translation
• Live caption overlay in the frontend
• Translation cache + queue control to keep latency low

How it works

1. The user starts tab/screen audio capture in the frontend.
2. Audio is converted to 16-bit mono PCM and sent in short chunks via WebSocket.
3. The backend processes each chunk:
   • Transcription with Deepgram.
   • Translation with Google Cloud Translation (Service Account or API Key).
4. The frontend receives captions and displays them in real time.

Requirements

• Node.js 18+ (20+ recommended)
• Deepgram account with API Key
• Google Cloud Translation enabled (API Key or Service Account)

Quick start (local)

1. Install dependencies:

npm install
npm run install:all

2. Create the backend .env file (do not commit):

cd backend
copy .env.example .env

3. Fill in sensitive variables in backend/.env:

• DEEPGRAM_API_KEY
• TRANSLATE_PROJECT_ID (if using Service Account)
• TRANSLATE_API_KEY (if using API Key)
• GOOGLE_APPLICATION_CREDENTIALS (path to the Service Account JSON)

Use API Key or Service Account (ADC). At least one of these options is required for translation.

4. Run locally:

cd ..
npm run dev

• Backend (WebSocket): ws://localhost:8080
• Frontend: http://localhost:5173

Note: tab/screen capture requires a secure context; localhost is allowed.

Environment variables (summary)

• DEEPGRAM_API_KEY: Deepgram credential (STT)
• DEEPGRAM_MODEL: Deepgram model (default nova-3)
• DEEPGRAM_LANGUAGE: base/detection language (default multi)
• TRANSLATE_PROJECT_ID: GCP project (using Service Account)
• TRANSLATE_API_KEY: alternative to Service Account
• PORT: backend port (default 8080)
• TARGET_SAMPLE_RATE: 16000 recommended
• MAX_QUEUE: queue limit per client
• TRANSLATE_INTERIM: enables partial translation (default false)
• INTERIM_TRANSLATE_INTERVAL_MS: partial translation interval

Project structure

backend/
  server.js
  audio.js
  config.js
  translate.js
  .env.example
frontend/
  index.html
  src/
    App.jsx
    main.jsx
    styles.css

Notes

• For production, configure TLS and a secure origin in the backend.
• Tune TARGET_SAMPLE_RATE and chunk size to balance cost vs latency.

Contributing

Issues and PRs are welcome. If you plan a larger change, open an issue first to align on approach.