Skip to content

levidehaan/JetsonOrinTTS

BranchesTags

Repository files navigation

STT Service for NVIDIA Jetson Orin

Created by Levi DeHaan

High-performance Speech-to-Text service optimized for NVIDIA Jetson Orin Developer Edition using Faster-Whisper with CUDA acceleration.

Features

  • GPU-Accelerated Transcription: 4.1x faster than CPU using CUDA with float16 precision
  • YouTube Video Support: Direct transcription from YouTube URLs with audio extraction
  • Asynchronous Processing: Upload files and get results via job ID system
  • Queue Management: Processes multiple files in order with queue status
  • SQLite Storage: Persistent job and result storage
  • Multiple Audio Formats: Supports WAV, MP3, M4A, FLAC, OGG, OPUS, MP4, MKV, WEBM
  • RESTful API: Simple HTTP endpoints for all operations
  • Performance Monitoring: Tracks processing times and confidence scores
  • Real-time GPU Monitoring: Visual system monitoring with graphs and metrics
  • Automated Service Management: Smart startup scripts with health checks

System Requirements

  • NVIDIA Jetson Orin Developer Edition
  • CUDA 12.x
  • Ubuntu 20.04/22.04
  • Python 3.8+
  • 4GB+ available RAM

Quick Start

1. Setup

# Clone and setup
git clone <repository-url>
cd TTSsystem

# Full automated setup (recommended)
make setup

# OR manual setup
make install-deps
make install-python
make all

2. Start Service

Option A: Systemd Service (Recommended - Survives Reboots)

# Install as system service (one-time setup)
sudo cp stt-service.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable stt-service
sudo systemctl start stt-service

# Use the control script for easy management
./stt-service-control.sh status     # Check status and test endpoints
./stt-service-control.sh restart    # Restart service
./stt-service-control.sh logs       # View logs
./stt-service-control.sh youtube <url>  # Test YouTube transcription

# Direct systemctl commands
sudo systemctl start stt-service    # Start
sudo systemctl stop stt-service     # Stop
sudo systemctl restart stt-service  # Restart
sudo systemctl status stt-service   # Status

Option B: Manual Startup (Development)

# Smart startup with GPU monitoring and health checks
./start_service.sh

# Custom port
./start_service.sh 3000

# Restart service
./start_service.sh restart

# Stop service
./start_service.sh stop

3. Monitor GPU Usage

# Beautiful real-time GPU/CPU monitoring
python3 monitor_gpu.py

# Update every 0.5 seconds
python3 monitor_gpu.py 0.5

4. Test Transcription

# Test with audio file
python3 test_client.py path/to/audio.wav

# Test YouTube video transcription
curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://www.youtube.com/watch?v=VIDEO_ID"}' \
  http://localhost:8080/youtube

API Endpoints

YouTube Video Transcription (Primary Feature)

POST /youtube
Content-Type: application/json

# IMPORTANT: Use JSON format, NOT form data
curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://www.youtube.com/watch?v=VIDEO_ID"}' \
  http://localhost:8080/youtube

# ❌ WRONG - This will cause JSON parse errors:
# curl -F "url=https://www.youtube.com/watch?v=VIDEO_ID" http://localhost:8080/youtube

Response:

{
  "job_id": "job_1757939346_605ce77a",
  "status": "queued",
  "message": "YouTube audio downloaded and queued for transcription",
  "video_info": {
    "duration": 2823,
    "file_size": 541938218,
    "title": "Video Title Here",
    "uploader": "Channel Name"
  }
}

Upload Audio File

POST /upload
Content-Type: multipart/form-data

curl -F "audio=@/path/to/audio.wav" http://localhost:8080/upload

Response:

{
  "job_id": "job_1694648400_a1b2c3d4",
  "status": "queued",
  "message": "Audio file uploaded successfully and queued for processing"
}

Get Transcription Results

GET /results/{job_id}

curl http://localhost:8080/results/job_1694648400_a1b2c3d4

Response:

{
  "job_id": "job_1694648400_a1b2c3d4",
  "status": "completed",
  "text": "Complete transcription with timestamps...",
  "processing_time_seconds": 448.99,
  "confidence_score": 0.8671,
  "model_used": "faster-whisper-base",
  "language": "en",
  "segments": [
    {
      "start": 0.0,
      "end": 3.0,
      "text": "Hello world",
      "confidence": 0.95
    }
  ],
  "created_at": 1694648400,
  "completed_at": 1694648412
}

Service Status

GET /status

curl http://localhost:8080/status

Health Check

GET /health

curl http://localhost:8080/health

Architecture

┌─────────────────┐    ┌──────────────┐    ┌─────────────────┐
│   HTTP Client   │───▶│ HTTP Server  │───▶│   Job Queue     │
│  (YouTube URLs) │    │  (Crow C++)  │    │   (C++ MT)      │
└─────────────────┘    └──────────────┘    └─────────────────┘
                              │                      │
                              ▼                      ▼
                       ┌──────────────┐    ┌─────────────────┐
                       │   SQLite     │    │  GPU Transcriber│
                       │   Database   │    │ (Faster-Whisper)│
                       └──────────────┘    └─────────────────┘
                                                     │
                                                     ▼
                                            ┌─────────────────┐
                                            │  CUDA Runtime   │
                                            │ (Jetson Orin)   │
                                            └─────────────────┘

GPU Performance

The system uses dedicated GPU acceleration:

  • GPU Environment: .venv_cuda with CUDA-enabled PyTorch
  • Transcriber: transcriber_gpu.py with optimized GPU inference
  • Performance: 4.1x speedup vs CPU transcription
  • Model Device: cuda:0 (Jetson Orin GPU)
  • Precision: float16 for optimal memory usage
  • Processing Speed: ~1.19 MB/s for large audio files

Performance Optimization

The system is optimized for Jetson Orin with:

  • CUDA Acceleration: Uses GPU for inference (4.1x faster than CPU)
  • Float16 Precision: Reduces memory usage and increases speed
  • VAD Filtering: Voice Activity Detection for better accuracy
  • Beam Search: Optimized beam size for quality/speed balance
  • Memory Management: Efficient memory usage for long audio files
  • YouTube Integration: Direct audio extraction and processing
  • Background Processing: Asynchronous job queue with status tracking

Configuration

Edit transcriber_gpu.py to adjust GPU settings:

  • Model size: tiny, base, small, medium, large
  • Compute type: float16, int8_float16, int8
  • Device: cuda (automatic GPU detection)
  • Beam size, temperature, and other Whisper parameters

Monitoring Tools

GPU Performance Monitor

# Real-time visual monitoring with graphs
python3 monitor_gpu.py

# Features:
# - GPU utilization with bar graphs and sparklines
# - CPU usage per core with frequencies
# - Memory usage visualization
# - Temperature monitoring with color coding
# - Power consumption tracking
# - Historical trend graphs

System Stats (Alternative)

# Basic Jetson monitoring
tegrastats --interval 1000

# Watch mode
watch -n 1 tegrastats

Service Management

Systemd Service (Production)

The systemd service provides automatic startup, restart on failure, and proper logging:

# Service control script (recommended)
./stt-service-control.sh status     # Status and health check
./stt-service-control.sh restart    # Restart service
./stt-service-control.sh logs       # View recent logs
./stt-service-control.sh test       # Test all endpoints
./stt-service-control.sh youtube <url>  # Test YouTube workflow

# Direct systemctl commands
sudo systemctl status stt-service   # Detailed status
sudo systemctl restart stt-service  # Restart
sudo systemctl stop stt-service     # Stop
sudo systemctl start stt-service    # Start
sudo systemctl disable stt-service  # Disable auto-start
sudo systemctl enable stt-service   # Enable auto-start

# View logs
sudo journalctl -u stt-service -f   # Follow logs
sudo journalctl -u stt-service -n 50  # Last 50 lines

Manual Startup Script (Development)

The start_service.sh script provides:

  • Auto-detection: GPU, CUDA, Python environments
  • Health Checks: Service validation and endpoint testing
  • Port Management: Conflict detection and custom ports
  • Logging: Structured logs with timestamps
  • Process Management: Clean start/stop/restart operations
./start_service.sh         # Start on port 8080
./start_service.sh 3000    # Start on port 3000
./start_service.sh restart # Restart service
./start_service.sh stop    # Stop service

Troubleshooting

Critical Issues and Fixes

YouTube Videos Return Identical Transcriptions

Symptom: Different YouTube videos return the same transcription text Cause: YouTube downloader returns wrong audio file paths Fix: The file selection logic in youtube_downloader.py has been updated to properly match downloaded files to their titles

JSON Parse Errors with "--" Characters

Symptom: [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid number; expected digit after '-'; last read: '--' Cause: stderr output from youtube-dlp or Whisper corrupting stdout JSON Fix: Added 2>/dev/null redirection in transcriber commands and contextlib.redirect_stdout(sys.stderr) in transcriber_gpu.py

Text Truncation Issues

Symptom: Transcription results show "... [truncated - see full result in file]" Cause: Legacy truncation logic limiting output to 1000 characters Fix: Removed truncation logic to always return full JSON results

Service Won't Start - Port Conflicts

Quick Fix:

# Kill any services using port 8080
lsof -ti:8080 | xargs -r kill -9
sleep 1

# Rebuild and restart manually
make -C build && ./build/stt_service &

Wrong Virtual Environment

Symptom: Transcriber fails with environment errors Cause: Service using .venv instead of .venv_cuda Fix: Updated all scripts to use .venv_cuda for GPU acceleration

Manual Service Management

When automated scripts fail, use manual commands:

# Full manual restart sequence
pkill -f stt_service                    # Stop all instances
lsof -ti:8080 | xargs -r kill -9        # Free port 8080
make -C build                           # Rebuild with fixes
./build/stt_service &                   # Start service

# Test service is working
curl http://localhost:8080/health

# Test YouTube workflow
curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}' \
  http://localhost:8080/youtube

Debugging Component Issues

Test YouTube Downloader Directly

# Test individual components
source .venv_cuda/bin/activate

# Test YouTube downloader
python3 youtube_downloader.py "https://www.youtube.com/watch?v=VIDEO_ID" uploads 2>/dev/null

# Test GPU transcriber
python3 transcriber_gpu.py "uploads/audio_file.wav" base

# Check downloaded files
ls -la uploads/ | tail -5

Verify Correct File Selection

# Check that different videos create different files
bash -c 'source .venv_cuda/bin/activate && python3 youtube_downloader.py "URL1" uploads' | jq -r '.audio_file'
bash -c 'source .venv_cuda/bin/activate && python3 youtube_downloader.py "URL2" uploads' | jq -r '.audio_file'

# Files should be different and match video titles

GPU/CUDA Issues

# Check GPU status
nvidia-smi
python3 monitor_gpu.py  # Visual monitoring

# Check CUDA installation
nvcc --version

# Check Python CUDA support
source .venv_cuda/bin/activate
python3 -c "import torch; print(torch.cuda.is_available())"
python3 -c "import torch; print(torch.cuda.get_device_name())"

Service Issues

# Check service status
curl http://localhost:8080/health

# View logs
tail -f logs/stt_service_*.log
tail -f logs/gpu_transcription_*.log

# Restart service
./start_service.sh restart

Memory Issues

  • Reduce model size (base → small → tiny)
  • Set compute_type to int8_float16
  • Process shorter audio segments
  • Monitor with python3 monitor_gpu.py

Build Issues

# Check dependencies
make check

# Clean rebuild with CMake policy fix
rm -rf build && mkdir build
cd build && cmake .. -DCMAKE_POLICY_VERSION_MINIMUM=3.5 && make -j4

YouTube Download Issues

  • Check internet connection
  • Verify YouTube URL format
  • Check available disk space in uploads/ directory
  • View download logs in service output

File Structure

TTSsystem/
├── start_service.sh           # Smart service startup script
├── stt-service-control.sh     # Systemd service management script
├── stt-service.service        # Systemd service configuration
├── monitor_gpu.py             # GPU monitoring with visualizations
├── transcriber_gpu.py         # GPU-optimized transcriber
├── transcriber.py             # CPU fallback transcriber
├── youtube_downloader.py      # YouTube audio extraction
├── build/stt_service          # Main service binary
├── .venv_cuda/                # CUDA Python environment
├── .venv/                     # CPU Python environment
├── logs/                      # Service and transcription logs
├── uploads/                   # Downloaded/uploaded audio files
├── results/                   # Transcription result files
└── src/                       # C++ source code
    ├── server/                # HTTP server implementation
    ├── transcriber/           # Transcription wrapper
    ├── database/              # SQLite database management
    └── queue/                 # Job queue system

Performance Metrics

Real-world performance on NVIDIA Jetson Orin:

  • GPU Speedup: 4.1x faster than CPU
  • Processing Speed: ~1.19 MB/s for large files
  • Memory Usage: ~535MB RAM for 48-min video
  • Accuracy: 86.71% average confidence
  • Model Load Time: ~2.5 seconds
  • Languages: Auto-detection (English optimized)

Development

Build Options

make all        # Release build
make debug      # Debug build with symbols
make clean      # Clean all build files

# CMake build (if make fails)
rm -rf build && mkdir build
cd build && cmake .. -DCMAKE_POLICY_VERSION_MINIMUM=3.5 && make -j4

Testing

# System compatibility check
make check

# Test YouTube transcription
curl -X POST -H "Content-Type: application/json" \
  -d '{"url":"https://www.youtube.com/watch?v=dQw4w9WgXcQ"}' \
  http://localhost:8080/youtube

# Monitor GPU during processing
python3 monitor_gpu.py

Adding Features

  1. New endpoints: Modify src/server/http_server.cpp
  2. Database schema: Update src/database/db_manager.cpp
  3. GPU processing: Edit transcriber_gpu.py
  4. Service logic: Update src/transcriber/transcriber_wrapper.cpp

Known Issues

Resolved Issues (Fixed in Current Version)

  • ✅ YouTube file selection bug: Fixed - different videos now produce different transcriptions
  • ✅ Text truncation: Fixed - full transcription results are now returned
  • ✅ JSON parse errors: Fixed - stderr redirection prevents output corruption
  • ✅ Virtual environment: Fixed - using .venv_cuda for GPU acceleration

Current Issues

  • Port conflicts: Service may fail if port 8080 is in use (use manual restart: lsof -ti:8080 | xargs -r kill -9 && make -C build && ./build/stt_service &)
  • Large files: Videos >1GB may take significant time (monitor with python3 monitor_gpu.py)
  • CMake version: Some builds require CMake policy override for nlohmann/json dependency
  • Form data requests: YouTube endpoint requires JSON format, not form data (see API documentation)

License

MIT License

About

a project for the jetson orin developer kit that runs a TTS server for youtube videos

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages