Gesture-based Smart Home Interface using Sign Language and Hand Pose Recognition
🔧 Built with FastAPI, MediaPipe, React, MQTT and Home Assistant.
🔍 Empowering accessibility through open-source gesture control.
Gestalyze is an open-source project that enables real-time hand gesture recognition for controlling smart home devices. By combining computer vision and machine learning techniques, the system interprets static hand poses — including alphabet signs from sign language — and sends commands to home automation platforms like Home Assistant.
The system is modular, lightweight, and designed for low-power environments using technologies such as MQTT, MediaPipe Hands, and React. Users can extend, train custom models, and integrate new gesture mappings for full control over their smart environment.
- ✋ Real-time hand detection and finger tracking with MediaPipe.
- 🔠 ASL static alphabet recognition using trainable ML classifiers.
- 💡 Smart home integration via MQTT and Home Assistant plugins.
- ⚡ Low-latency WebSocket communication between frontend and backend.
- 📷 Frontend web app built in React + WebRTC.
- 🌍 Multilingual interface with i18n support.
- 🧠 Supports custom and pre-trained gesture models.
- 📊 Visualization tools for metrics, training and evaluation.
- 🧪 Designed for reproducible research and experimentation.
This repository is currently structured as a monorepo to simplify development, deployment, and cross-component coordination. As the project matures, it is expected that individual components — such as the frontend, backend, and Home Assistant integrations — may be separated into distinct repositories to improve modularity and scalability.
gestalyze/
├── backend/ # Gestalyze backend server
│ ├── inference/ # ML model inference and processing
│ ├── mqtt/ # MQTT client and message handling
│ └── ssl/ # SSL certificates and configuration
├── config/ # Project-wide configuration files (e.g., MQTT)
├── datasets/ # Public and custom gesture datasets
├── docs/ # Assets and documentation support files
├── frontend/ # React web app for video capture and UI
├── homeassistant/ # Plugins and custom components for Home Assistant
│ ├── custom_components/ # Custom Home Assistant integration
│ └── gestalyze_plugin/ # Home Assistant plugin for gestalyze
├── models/ # Serialized ML models (.joblib)
├── training/ # Training scripts and model experiments
├── website/ # Public website (gestalyze.com)
└── Makefile # Project task automation
Requires Python 3.10+, Node.js 20+, and Mosquitto MQTT
# Clone repository
git clone https://github.com/yourname/gestalyze.git
cd gestalyze
# Install Python & Node dependencies
make install
# Start development servers
make devOpen your browser to http://localhost:5173 to launch the interface.
To train your own gesture recognition model:
# Example with custom dataset
make extract-landmarks DATASET_NAME=your_dataset
make train MODEL_TYPE=custom DATASET_NAME=your_datasetModels are saved to models/ and can be switched via the interface.
Common tasks for development and training:
make install— Installs Python and Node.js dependenciesmake dev— Starts both frontend and backend in development modemake build— Builds the frontend for productionmake run— Runs the backend with built frontend and MQTTmake run-ssl— Runs backend with SSL enabledmake train MODEL_TYPE=custom DATASET_NAME=name— Trains a gesture modelmake extract-landmarks DATASET_NAME=name— Extracts MediaPipe landmarks from datasetmake clean— Cleans compiled files, build folders and cachesmake mqtt-start,make mqtt-stop— Start/stop the Mosquitto brokermake ssl-key-gen— Generates self-signed SSL certificatesmake deploy-ha— Deploys custom integration to Home Assistantmake website-dev— Starts the public website in development modemake website-build— Builds the public website for productionmake version— Displays version infomake help— Shows available commands
For advanced options and custom variables, inspect the Makefile.
Gestalyze communicates with Home Assistant via MQTT. You can:
- Use our plugin (external service)
- Or the custom integration under
homeassistant/custom_components/gestalyze/
This enables the use of sensors and event-based automations based on detected hand gestures.
All gesture processing happens locally on your device.
No images or biometric data are transmitted or stored externally.
See Privacy Policy for full details.
- Full system documentation: https://gestalyze.com
- Datasets, models and training configuration
- Code walkthroughs, diagrams and tutorials
This project is licensed under the Apache License 2.0 — see LICENSE for details.
Pull requests, feature ideas and issue reports are welcome. For contributions, see CONTRIBUTING.md.
- MediaPipe by Google for hand tracking
- Home Assistant for smart home integration
- Hugging Face and Kaggle for gesture datasets
- All open-source contributors who inspire and make projects like this possible
Gestalyze is still under active development. Some upcoming features and improvements include:
- Support for multiple simultaneous frontend clients.
- Real-time model hot-swapping via the interface.
- Integration with additional smart home platforms.
- Gesture recording and labeling tools for data collection.
- Extended dynamic gesture recognition (sequential actions).
- Accessibility testing with real users and community feedback.
- CI/CD automation for releases and plugin deployment.
- Docker-based deployment flow.
- Unit testing for every module.
- Improved model modularity for easier experimentation and integration.
- Continued refinement of gesture recognition model accuracy.
© 2025 — Initially developed by rbcorrales.