Practice MediaPipe & OpenCV Portal 🚀

Github Repo: https://github.com/bernardbdas/practice-mediapipe

This project is a premium on-device Artificial Intelligence application built using the modern MediaPipe Tasks API, OpenCV, and Streamlit WebRTC.

It demonstrates on-device ML tasks structured exactly as classified in the official MediaPipe solutions navbar, utilizing Streamlit Navigation (st.navigation) for section categorization.

---

🌟 Classified Modules & Tasks

👁️ Vision Tasks

1. Object Detection: Locates and classifies everyday objects with bounding boxes (EfficientDet-Lite0).
2. Image Classification: Classifies images into 1,000+ categories (EfficientNet-Lite0) with visual bar charts.
3. Image Segmentation: Separates the user (foreground) from their background (Selfie Segmenter).
4. Interactive Segmentation: Segments a specific object based on user-targeted coordinates (Magic Touch).
5. Gesture Recognition: Identifies hand gestures (e.g. Open Palm, Closed Fist, Thumbs Up, Victory) in real-time.
6. Hand Landmark Detection: Tracks 21 3D landmarks per hand and counts fingers.
7. Image Embedding: Computes cosine similarity scores between two uploaded images to verify visual similarity (MobileNetV3).
8. Face Detection: Fast face boundary box location (BlazeFace).
9. Face Landmark Detection: Maps a detailed 468-point 3D face mesh.
10. Pose Landmark Detection: Tracks 33 body posture skeleton points.
11. Holistic Landmark Detection: Combines Face, Hand, and Pose landmarkers in a single pipeline.

📝 Text Tasks

12. Text Classification: Analyzes text inputs to determine Positive or Negative sentiment (BERT/Projection).
13. Text Embedding: Calculates semantic cosine similarity scores between two text blocks (Universal Sentence Encoder).
14. Language Detection: Identifies the language of text inputs (supports 110+ languages).

🔊 Audio Tasks

15. Audio Classification: Categorizes environmental and everyday sound events in .wav files (YAMNet).

💭 Generative AI Tasks

16. LLM Inference: On-device local chat playground running Gemma 2B CPU int4 model.

🏗️ Project Structure

• src/: Core source directory containing simplified imports packages.
  • main.py: Sets up st.navigation and builds categorized sidebar headers.
  • pages/: 18 organized Streamlit pages (Portal Home + 16 Tasks + Classic CV Comparison).
  • utils/:
    • models.py: Central model configuration registry.
    • styling.py: Implements custom glassmorphism theme and Google fonts.
    • webrtc_utils.py: Shared WebRTC and TURN server configurations.
    • model_downloader.py: Automatic cached XML-based downloader with progress bars.
• .mediapipe/: Local storage folder (git-ignored).
  • mediapipe_models.xml: Local copy of bucket directory listing.
  • tasks/: Directory containing cached downloaded task model files.
• notebooks/: Experimental Jupyter Notebooks.
• main.py: Root wrapper entry point.

---

🛠️ Setup & Installation

Running Locally (uv)

uv sync
uv run python main.py

Running with Docker

docker compose up --build

Access the application at http://localhost:8080.

---

📖 MediaPipe Platform Setup Guides Reference

For developers deploying these solutions to production environments, here is a quick setup reference across target platforms:

🐍 Python

• Package: pip install mediapipe
• Usage: Load task models using BaseOptions and create_from_options(options). For vision tasks, convert OpenCV BGR frames to RGB before processing.

🤖 Android (Java/Kotlin)

• Dependency: Add implementation 'com.google.mediapipe:tasks-vision:latest.release' in build.gradle.
• Permissions: Add <uses-permission android:name="android.permission.CAMERA" /> in your manifest.
• Model Loader: Bundle model files in the assets/ folder and load using BaseOptions.builder().setModelAssetPath(...).

🌐 Web (JavaScript/TypeScript)

• Dependency: Install @mediapipe/tasks-vision via npm or load CDN scripts.
• WASM Loading: Vision tasks require loading WASM files at runtime. Configure FilesetResolver.forVisionTasks("https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm") before instantiating task objects.

🍎 iOS (Swift/Objective-C)

• Dependency: Add pod 'MediaPipeTasksVision' inside your Podfile.
• Permissions: Configure NSCameraUsageDescription in Info.plist to request camera access.
• Instantiation: Instantiate tasks using FaceDetector(options: options) or similar task options configurations.