HELIOS is a browser-native, multimodal spatial operating system designed to bridge the gap between human intent and digital data. By fusing MediaPipe-powered neural hand tracking with Web Speech AI, it transforms the standard 2D browser into a 3D command center where data is manipulated as a physical entity.
A Real-Time Spatial Web Browser with Gesture + Voice Control
HELIOS is a fully functional web browser built inside a 3D spatial environment. It runs in a normal browser, uses your camera for hand tracking, and your microphone for voice control.
Simply open index.html in a modern web browser (Chrome/Edge recommended).
- Camera: Required for hand tracking
- Microphone: Required for voice control
Your workspace is now active. Use gestures or voice commands to control the system.
| Gesture | Action |
|---|---|
| Point | Move 3D cursor |
| Pinch | Click / Select |
| Pinch + Drag | Move window |
| Swipe Left/Right | Switch tabs |
| Swipe Up | Show tab overview |
| Open Palm | Show overview (planned) |
| Two-Hand Stretch | Resize window (planned) |
| Command | Action |
|---|---|
| "Open [website]" | Navigate to website |
| "Go to [URL]" | Open URL |
| "Search for [query]" | Google search |
| "Next tab" | Switch to next window |
| "Previous tab" | Switch to previous window |
| "Close window" | Close active window |
| "Minimize" / "Maximize" | Window controls |
| "Focus mode" | Dim inactive windows |
| "Reset workspace" | Close all windows |
| "Overview" | Show all tabs |
Website Shortcuts:
- "Open YouTube" → youtube.com
- "Open GitHub" → github.com
- "Open Google" → google.com
- etc.
| Shortcut | Action |
|---|---|
Ctrl/Cmd + T |
New window (prompts for URL) |
Ctrl/Cmd + W |
Close active window |
Tab |
Next window |
Shift + Tab |
Previous window |
Ctrl + Space |
Toggle tab overview |
Ctrl + F |
Focus mode |
Ctrl + R |
Reset workspace |
Alt + 1-4 |
Arrange windows (grid/cascade/horizontal/circular) |
Ctrl/Cmd + Shift + S |
Save session |
Ctrl/Cmd + Shift + L |
Load session |
Esc |
Close overview |
HELIOS consists of 5 integrated subsystems:
- Three.js-based 3D environment
- Post-processing effects (bloom, tone mapping)
- 60fps target performance
- Dynamic lighting system
- Central reactive sphere
- Responds to voice volume
- Provides visual feedback
- Particle field animation
- Energy rings
- MediaPipe hand tracking
- Real-time gesture recognition
- Pinch, swipe, spread detection
- 3D hand position mapping
- Cursor feedback
- Web Speech API integration
- Natural language processing
- Command parsing and routing
- Website shortcuts
- Command history
- CSS3D rendering for HTML content
- Real iframe-based browsing
- Spatial window positioning
- Drag-and-drop functionality
- Tab management
- Session persistence
- Multiple layout modes
- Procedural sound generation
- Spatial audio feedback
- Ambient background
- Event-based sounds
- Real-time hand tracking
- Voice command recognition
- 3D spatial environment
- AI core visualization
- Multiple floating windows
- Tab navigation
- Window dragging
- Window controls (close/minimize/maximize)
- Audio feedback
- Session saving/loading
- Tab overview mode
- Focus mode
- Multiple window layouts
- Keyboard shortcuts
- Real browser functionality (iframes)
- Two-hand resize gestures
- Depth-based window interaction
- Voice-based web search within pages
- Bookmark management
- History navigation
- Multiple workspace profiles
- VR/AR mode
- Collaborative browsing
- Advanced gesture customization
Edit css/core.css:
:root {
--primary-cyan: #00f2ff;
--secondary-magenta: #ff0055;
}Edit js/core/RenderEngine.js:
// Lower pixel ratio for better performance
this.renderer.setPixelRatio(1);
// Reduce bloom intensity
const bloomPass = new UnrealBloomPass(..., 1.5, ...);Edit js/core/WindowManager.js:
this.maxWindows = 8; // Change this value- Recommended: Chrome 90+, Edge 90+
- Required APIs:
- WebGL 2.0
- Web Speech API
- MediaPipe (via CDN)
- getUserMedia (camera access)
- Camera: Any webcam (720p+ recommended)
- Microphone: Built-in or external
- GPU: Moderate GPU for 3D rendering
- RAM: 4GB+ recommended
- Check browser permissions
- Ensure camera is not in use by another app
- Try refreshing the page
- Check microphone permissions
- Speak clearly and at moderate volume
- Ensure you're using a supported browser (Chrome/Edge)
- Close other browser tabs
- Lower window count
- Reduce browser window size
- Check GPU acceleration is enabled
- Check browser console for errors
- Ensure iframes are not blocked by CORS
- Some sites block iframe embedding
HELIOS/
├── index.html # Main entry point
├── css/
│ └── core.css # All styling
├── js/
│ ├── main.js # Application orchestrator
│ └── core/
│ ├── RenderEngine.js # 3D rendering
│ ├── AICore.js # AI visualization
│ ├── GestureEngine.js # Hand tracking
│ ├── VoiceEngine.js # Voice control
│ ├── WindowManager.js # Window system
│ └── AudioSystem.js # Sound feedback
└── README.md # This file
- Camera captures video feed
- MediaPipe detects hand landmarks (21 points per hand)
- Landmarks mapped to screen/3D coordinates
- Gestures recognized from landmark patterns
- Actions triggered based on gestures
- Microphone captures audio
- Web Speech API transcribes speech
- Natural language processing extracts intent
- Commands routed to appropriate handlers
- Visual/audio feedback provided
- Windows exist as CSS3D objects
- Each window contains a real iframe
- Positioned in 3D space via Three.js
- User interactions update 3D positions
- Smooth animations via lerping
HELIOS demonstrates what browsing could look like in a spatial computing future:
- No flat tabs → Floating spatial windows
- No mouse → Natural hand gestures
- No typing → Voice commands
- No 2D limits → Full 3D workspace
It's a statement: Browsing doesn't have to be flat.
This is a demonstration project showcasing spatial computing concepts.
Built with:
- Three.js - 3D rendering
- MediaPipe - Hand tracking
- Web Speech API - Voice recognition
Concept & Development: HELIOS - A spatial operating system for the web
HELIOS is a proof-of-concept that could evolve into:
- Browser Extension - Overlay on existing browsers
- Standalone App - Native desktop application
- VR/AR Platform - Full immersive experience
- Collaborative Tool - Multi-user spatial browsing
- Developer Platform - SDK for spatial web apps
Experience the future of browsing. Welcome to HELIOS.