Sign Language Detection using LSTM & MediaPipe

Project Overview

This project is a Real-Time Sign Language Recognition System built using:

• MediaPipe Holistic for landmark extraction
• LSTM (Long Short-Term Memory) neural networks for sequence learning
• TensorFlow/Keras for deep learning
• OpenCV for real-time video processing

The system recognizes hand gestures in real-time and classifies them into predefined actions.

Currently supported gestures:

• ✋ hello
• 🙏 thanks
• 🤟 iloveyou

Objective

To build an end-to-end deep learning pipeline that:

1. Captures live video from webcam
2. Extracts pose, face, and hand landmarks
3. Collects structured gesture sequence data
4. Trains an LSTM model
5. Performs real-time gesture prediction

Project Pipeline

1️⃣ Data Collection (collect_data.py)

• Captures webcam frames

• Uses MediaPipe Holistic

• Extracts:

  • 33 Pose landmarks
  • 468 Face landmarks
  • 21 Left-hand landmarks
  • 21 Right-hand landmarks

• Total features per frame: 1662

• Saves 30 frames per sequence

• 30 sequences per action

• Total samples: 90 sequences

2️⃣ Model Training (train_model.py)

• Loads collected .npy files

• Normalizes feature data

• Splits dataset using stratified split (80/20)

• Trains LSTM-based deep learning model

• Uses:

  • Dropout regularization
  • EarlyStopping
  • ReduceLROnPlateau
  • Adam optimizer (learning_rate = 0.0005)

3️⃣ Model Architecture

Input Shape: (30 frames, 1662 features)

LSTM(128) → Dropout(0.3)
LSTM(128) → Dropout(0.3)
LSTM(64)  → Dropout(0.3)
Dense(128)
Dense(64)
Output Dense(3, softmax)

Model Performance

Model Accuracy: 88.89%

Confusion Matrix

[[6 0 0]
 [0 6 0]
 [2 0 4]]

Classification Report

Class	Precision	Recall	F1-score	Support
hello	0.75	1.00	0.86	6
thanks	1.00	1.00	1.00	6
iloveyou	1.00	0.67	0.80	6

Overall Accuracy: 0.89 Macro Average F1: 0.89 Weighted Average F1: 0.89

📈 Interpretation of Results

• Perfect classification for thanks
• Strong performance for hello
• Shows strong performance across all classes, with slight overlap between similar gesture patterns.
• Model generalizes well despite small dataset size

🧪 Dataset Details

Parameter	Value
Actions	3
Sequences per action	30
Frames per sequence	30
Total sequences	90
Feature size	1662
Train/Test Split	80/20

🛠️ Technologies Used

• Python 3.10
• OpenCV
• MediaPipe
• NumPy
• TensorFlow 2.15
• Keras
• Scikit-learn
• Matplotlib

How to Run

1️⃣ Create Virtual Environment

python -m venv new-env
new-env\Scripts\activate

2️⃣ Install Dependencies

pip install mediapipe==0.10.9 opencv-python numpy scikit-learn tensorflow==2.15.0 matplotlib

3️⃣ Collect Data

python collect_data.py

4️⃣ Train Model

python train_model.py

5️⃣ Run Real-Time Prediction

python real_time_prediction.py

Project Structure

sign-language-detection/
│
├── MP_Data/
│   ├── hello/
│   ├── thanks/
│   └── iloveyou/
│
├── collect_data.py
├── train_model.py
├── real_time_prediction.py
├── action.h5
└── README.md

🧠 Key Learning Outcomes

• Sequence modeling using LSTM
• Landmark extraction using MediaPipe
• Handling high-dimensional feature vectors (1662 features)
• Preventing overfitting with Dropout & EarlyStopping
• Using stratified data splitting
• Model evaluation using confusion matrix & classification report
• Real-time computer vision pipeline development

🔮 Future Improvements

• Increase dataset size to 60–100 sequences per action
• Add more gestures
• Apply gesture smoothing for stable real-time predictions
• Convert model to TensorFlow Lite for deployment
• Deploy as web app using Streamlit
• Add probability confidence bars
• Improve background robustness

Project Impact

This project demonstrates:

• Deep Learning for sequence modeling
• Real-time computer vision
• End-to-end ML pipeline development
• Practical understanding of LSTM architectures

Suitable for:

• AI/ML internships
• Computer Vision projects
• Deep Learning portfolio