VoxClarity

A two-stage, personalized speech-to-text correction pipeline that runs entirely on a single consumer GPU. It transcribes speech with a quantized Whisper model, then uses a small quantized LLM to correct the transcription based on an individual speaker's own patterns — learned from a short calibration session, not from fine-tuning.

Originally motivated by ataxic dysarthria (a motor-speech condition that causes irregular, "scanning" speech), where generic ASR systems perform poorly because a speaker's errors are systematic to them, not random noise. See Scope & honesty notes below for what this project does and does not claim.

How it works

   mic audio
       │
       ▼
┌─────────────────┐      raw text      ┌──────────────────────┐
│  Whisper engine  │ ──────────────────▶│   Qwen correction LLM │ ──▶ corrected text
│ (faster-whisper) │                     │  (4-bit, few-shot)    │
└─────────────────┘                     └──────────────────────┘
   int8_float16                           NF4 4-bit + double quant
   VAD-filtered                           few-shot personalization
   beam_size=1                            from calibration data

Stage 1 — acoustic (Whisper). faster-whisper running large-v3-turbo, quantized to int8_float16: model weights compressed to 8-bit integers, live computation kept in float16 for precision. Voice-activity detection filters out silence before it ever reaches the model, and the correction stage is skipped entirely if no speech is detected — no wasted inference, no hallucinated corrections on silence.

Stage 2 — semantic correction (Qwen). Qwen2.5-1.5B-Instruct, quantized to 4-bit using the NF4 format (bitsandbytes) with double quantization. Rather than fine-tuning per user, the model is given a handful of the user's own (garbled → intended) example pairs directly in its prompt as real conversation turns — in-context learning, no training step. Session corrections accumulate during use, so the model has more examples to work from later in a session than it did at the start.

Stage 3 — optional retrieval (RAG). Uploaded documents (PDF/text) are chunked and embedded with a CPU-resident model (nomic-embed-text-v1.5), stored in a per-user persistent ChromaDB collection. Before each correction, the pipeline searches the user's indexed documents for relevant context and injects it into Qwen's prompt — so it can recognize domain-specific names/terms it would otherwise have no way to know. Runs on CPU specifically so it never competes with Whisper/Qwen for VRAM.

Both GPU-resident models load lazily — nothing touches the GPU until the first time it's actually needed — and together stay comfortably within a 6GB VRAM budget on hardware like an RTX 3060.

Features

• Push-to-talk audio capture, entirely in memory (no temp files written to disk)
• Per-user profiles with a 10-sentence calibration flow that builds a personalized correction baseline from your own voice
• A live translation session with a feedback loop — correct a bad output once, and it's saved as a new example for next time
• Optional document upload (PDF/text): indexed into a per-user vector store and automatically pulled in as context so corrections can use domain-specific vocabulary
• Per-stage latency and VRAM usage reported on every correction
• A VRAM status view and graceful model unload on exit

Project structure

speech_project/
├── config.py              # Typed, validated settings loaded from .env
├── main.py                # CLI entry point
├── audio/
│   ├── capture.py         # Push-to-talk microphone recording
│   └── preprocessor.py    # Normalization, silence trimming, resampling
├── models/
│   ├── whisper_engine.py  # Lazily-loaded, quantized Whisper wrapper
│   └── llm_engine.py      # Lazily-loaded, quantized Qwen correction wrapper
├── profiles/
│   ├── manager.py         # JSON-backed user profile storage (CRUD)
│   └── calibration.py     # 10-sentence calibration recording flow
├── rag/
│   ├── embedder.py        # CPU-resident embedding model wrapper
│   ├── indexer.py         # Document chunking + ChromaDB storage
│   └── retriever.py       # Similarity search over indexed documents
├── translation/
│   └── pipeline.py        # Orchestrates Whisper -> Qwen (+ optional RAG context)
└── utils/
    └── vram_monitor.py    # GPU VRAM usage via nvidia-smi

Setup

Requirements: Python 3.12+, an NVIDIA GPU with 6GB+ VRAM, and a recent driver.

git clone https://github.com/Tirth2116/Speech_AI.git
cd Speech_AI
python -m venv .venv
.venv\Scripts\activate          # Windows
pip install -r requirements.txt

torch needs a CUDA-enabled build specifically — a plain pip install torch can silently resolve to a CPU-only wheel depending on your platform. If python -c "import torch; print(torch.cuda.is_available())" prints False, reinstall with:

pip install --force-reinstall torch --index-url https://download.pytorch.org/whl/cu128

Create a .env file in the project root (this file is intentionally not committed — paths like the model cache directory are specific to your machine):

# Model cache paths — keep large model files off your OS drive if possible
HF_HOME=D:/ML_Cache/huggingface
TRANSFORMERS_CACHE=D:/ML_Cache/huggingface
TORCH_HOME=D:/ML_Cache/torch
XDG_CACHE_HOME=D:/ML_Cache

# Model identifiers
WHISPER_MODEL=deepdml/faster-whisper-large-v3-turbo-ct2
WHISPER_COMPUTE_TYPE=int8_float16
QWEN_MODEL=Qwen/Qwen2.5-1.5B-Instruct

# Audio settings
SAMPLE_RATE=16000
CHANNELS=1
PTT_KEY=ctrl

Usage

python main.py

This opens a menu: create a profile (which walks you through recording 10 calibration sentences), then start a translation session — hold the configured key, speak, release, and see the raw transcription alongside the corrected text. If a correction is wrong, you can type the right answer and it's saved for next time. Optionally, upload a PDF or text file from the menu to have its content available as context for corrections.

Current status

Phase 1 (the core MVP pipeline) and Phase 2 (retrieval-augmented correction) are both complete: configuration, audio capture, both quantized model engines, user profiles and calibration, document indexing/retrieval, the orchestration pipeline, and the CLI are all implemented and tested against a real GPU.

Not yet built: QDoRA adapter fine-tuning on real dysarthric speech datasets (planned: Google Colab training for both the acoustic and correction models), and a server/mobile frontend (currently CLI-only).

Scope & honesty notes

• This is a personal learning project, built and tested primarily with the developer's own (non-impaired) speech. It demonstrates the personalization mechanism — few-shot correction from a calibration set — but has not been validated against a genuinely dysarthric speaker population, and no claim of clinical validity is made.
• Calibration audio and profile data are never committed to this repository (see .gitignore) — they're personal data and stay local to whoever runs the app.