FingerSolve: AI-Powered Hand Gesture Math Solver

This project uses MediaPipe hand tracking and a neural network to recognize hand gestures and answer user-given questions through gesture-based interaction.

Description

FingerSolve is an AI-powered web application that enables users to solve basic arithmetic problems using hand gestures instead of typing answers.
Designed with accessibility in mind, it combines real-time hand tracking, gesture classification, and a friendly web interface to make math more interactive and inclusive.

The app leverages MediaPipe for extracting hand landmarks and a custom-trained TensorFlow model for recognizing American Sign Language (ASL) digits.
FingerSolve works best on mobile devices and provides instant feedback to make learning both fun and accessible.

Features

• Real-time hand tracking and gesture recognition using MediaPipe
• Neural network classifier trained on ASL digits (63-point hand landmarks)
• Smooth, responsive React.js + Flask web interface
• Optimized for mobile devices and browsers
• Open-source and extendable for custom gestures or datasets

Try FingerSolve live here: https://fingersolveai-m3py.onrender.com/

Model Architecture

• Hand Tracking: MediaPipe extracts 21 keypoints per hand (63 total coordinates)
• Gesture Classification: Custom TensorFlow neural network predicts ASL digits
• Input Features: 63 landmark values per frame → one-hot encoded labels
• Frontend: Built in React.js for dynamic and mobile-friendly UI
• Backend: Flask API handles inference and math quiz logic

Dataset

• Training Data: Custom image dataset of ASL digits (0–9), captured via webcam
• Feature Extraction: 63 hand landmark coordinates generated by MediaPipe

Requirements

• Python 3.11 or higher
• Node.js (for frontend)
• Webcam-enabled device
• Required Libraries
  • Backend: TensorFlow / Keras, MediaPipe, Flask, NumPy, OpenCV
  • Frontend: React.js, TypeScript (optional)

Setup Instructions

# Clone the repository
git clone https://github.com/junaid-pathan/fingersolve.git
cd fingersolve

# If you do not have Python 3.11, create a virtual environment:

# macOS/Linux
python3.11 -m venv venv
source venv/bin/activate

# Windows
python3.11 -m venv venv
venv\Scripts\activate

# Install required Python packages
pip install -r requirements.txt

# Launch the backend server for gesture recognition
python modeL_for_gesture.py
flask run

# For the frontend (from /frontend directory)
cd frontend
npm install
npm start

# Open the app at http://localhost:5000
# Or use the live demo:
# https://fingersolveai-m3py.onrender.com/