Gesture-Based 3D Design and Physics Validation System

This project is an MVP for natural hand-based interaction in 3D environments. It replaces mouse-style manipulation with temporal hand-motion intent recognition.

Core Pipeline

Camera -> MediaPipe landmarks -> sequence model -> gesture action -> 3D scene update -> physics validation feedback

MVP Features

• Real-time single-hand tracking in browser using MediaPipe Hands.
• Temporal windowing of landmarks and action prediction request to backend.
• Three.js scene with base objects (cube, sphere, cylinder).
• Gesture-driven object operations:
  • create
  • grab/move
  • rotate
  • scale
  • join (placeholder)
  • delete
• Lightweight physics checks:
  • stability score
  • stress risk score (heuristic in MVP)
  • recommendations

Recommended Project Scope

Keep this first version intentionally simple:

• 5 to 6 core actions only.
• Primitive objects only (box, sphere, cylinder).
• Approximate physics, not full FEA.

Folder Structure

• backend/: FastAPI service for action inference and physics checks.
• frontend/: Browser app for webcam tracking + 3D interaction.
• ml/: model training scripts and notes.

Quick Start

1) Backend

cd backend
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

2) Frontend

Run the React app with Vite.

cd frontend
npm install
npm run dev

Then open: http://localhost:5500

The frontend expects backend at: http://localhost:8000

API Endpoints

• GET /health: backend health check.
• POST /predict-action: predicts action from recent hand landmark sequence.
• POST /physics/validate: returns stability and stress heuristics.

Week-by-Week Plan

• Week 1-2: Hand tracking and landmark streaming.
• Week 3-4: Train LSTM/Transformer for temporal gestures.
• Week 5: Integrate gesture actions with Three.js object control.
• Week 6: Add physics validation module.
• Week 7: Improve UX, demo scripts, and evaluation.

Suggested Evaluation Metrics

• Gesture classification: accuracy, F1 score, confusion matrix.
• Interaction latency: camera-to-action delay (ms).
• Physics proxy quality: correlation with expected stability outcomes for known test shapes.

Future Extensions

• Multi-hand support and bimanual operations.
• Full rigid body simulation integration (ammo.js or cannon-es).
• Design auto-fix agent inspired by PhysiOpt style optimization loops.