Lip-Reading AI

An end-to-end PyTorch starter project for visual speech recognition (lip-reading) from silent videos and live webcam input.

What this project includes

• Video preprocessing with automatic mouth-region extraction
• Character-level lip-reading model with CNN + Transformer or BiLSTM + CTC
• Training pipeline for your own dataset
• Offline transcription from a video file
• Live webcam transcription with a rolling frame buffer
• Dataset preparation from local videos or a JSON download manifest

Important reality check

Lip-reading is much harder than audio speech recognition. This codebase is complete and runnable, but real-world accuracy depends heavily on:

• A large labeled dataset
• Consistent frontal faces
• Good lighting and frame rate
• Proper training time on a GPU

This project is a solid foundation, not a guaranteed production-grade recognizer out of the box.

Dataset format

Organize the dataset like this:

dataset/
  train/
    sample_0001.mp4
    sample_0001.txt
    sample_0002.mp4
    sample_0002.txt
  val/
    sample_1001.mp4
    sample_1001.txt

Each .txt file should contain the ground-truth transcript for the matching video.

Install

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Train

python train.py --data-dir dataset --epochs 40 --batch-size 4 --device cuda --encoder-type transformer

Model checkpoints are saved to checkpoints/.

Prepare a dataset from your own videos

If you already have video files and transcripts:

python prepare_dataset.py --source-dir raw_videos --output-dir dataset --copy

If you have a CSV with your own videos:

python prepare_local_videos.py --csv my_videos.csv --output-dir dataset

If your files are GRID-style names like bbaf2n.mpg:

python prepare_grid_corpus.py --source-dir "C:\Users\sriam\Videos\examples\s1.mpg_vcd\s1" --output-dir dataset

If you want the GRID videos converted to silent .mp4 files during preparation:

python prepare_grid_corpus.py --source-dir "C:\Users\sriam\Videos\examples\s1.mpg_vcd\s1" --output-dir dataset --silent-mp4

If you want to mix GRID data with your own labeled videos:

python prepare_mixed_dataset.py --grid-source "C:\Users\sriam\Videos\examples\s1.mpg_vcd\s1" --custom-csv my_videos.csv --output-dir dataset_mixed --silent-mp4

If you only want silent copies of a video folder:

python strip_audio.py --source-dir "C:\Users\sriam\Videos\examples\s1.mpg_vcd\s1" --output-dir silent_videos

If you want a label template generated from a video folder first:

python make_labels_template.py --videos-dir C:\Users\sriam\Videos\examples --output-csv my_videos.csv

If you have direct downloadable video URLs:

python prepare_dataset.py --manifest sample_manifest.json --output-dir dataset --copy

Transcribe a silent video

python infer_video.py --video path\to\video.mp4 --checkpoint checkpoints\best.pt --device cuda

Live webcam transcription

python live_infer.py --checkpoint checkpoints\best.pt --device cuda

Press q in the webcam window to quit.

Demo Notes

Check Saved Video

python infer_video.py --video "C:\project\lipsync\New project\grid_dataset\train\bbaf2n.mp4" --checkpoint "C:\project\lipsync\New project\grid_checkpoints_lipnet\best.pt" --device cpu --max-frames 75 --grid-decode

Check Live Webcam

python live_infer.py --checkpoint "C:\project\lipsync\New project\grid_checkpoints_lipnet\best.pt" --device cpu

Live Demo Reminders

• Speak the word or phrase first, then press Space
• Press C to clear the buffer before the next try
• Press Q to quit live mode
• Use saved-video inference as the main demo, and show live mode as a prototype

Demo Screenshots

Saved Video Inference

Live Webcam Inference

Tips for better results

• Keep the face centered and mostly frontal
• Use 25 to 30 FPS video when possible
• Avoid fast head turns and occlusion
• Normalize transcripts to lowercase text
• Start with short phrases before scaling up

Project structure

lipread_ai/
  config.py
  dataset.py
  decode.py
  model.py
  preprocess.py
  utils.py
  vocabulary.py
train.py
infer_video.py
live_infer.py
prepare_dataset.py
prepare_local_videos.py
make_labels_template.py
prepare_grid_corpus.py
prepare_mixed_dataset.py
strip_audio.py

Local training walkthrough

See LOCAL_TRAINING_GUIDE.md for a focused workflow for training on your own clips.