Demo • Features • Architecture • Setup • Usage
Sora_Demo.mp4
Sora identifying objects in real-time and engaging in a natural voice conversation.
Sora is a state-of-the-art Agentic AI Assistant designed to bridge the gap between text-based LLMs and real-world interaction.
Built for the 2026 AI landscape, Sora isn't just a chatbot; it is a ReAct Agent (Reasoning + Acting). By orchestrating Google Gemini 2.0 Flash with LangGraph, the agent autonomously decides when to trigger its visual cortex to "see" the world. Leveraging Groq's LPU for sub-second inference, Sora delivers human-parity response times for speech and vision.
- 🧠 Agentic Decision Making: Uses LangGraph to decide autonomously when it needs visual context. It doesn't look until it needs to.
- 👁️ Active Vision: Powered by Llama 3.2 Vision on Groq, Sora captures and analyzes webcam frames to answer questions like "Do I look tired?" or "What am I holding?"
- 👂 Ultra-Low Latency Speech: Near-instant transcription using Whisper V3 on Groq's LPU hardware.
- 🗣️ Expressive TTS: Responds with lifelike, emotive audio using ElevenLabs (or lightweight gTTS for offline use).
- 💻 Split-Screen UI: A sleek Gradio interface featuring a live video feed alongside a real-time terminal-style chat.
Sora operates on a modular pipeline where specialized models handle specific sensory inputs, all coordinated by the central Gemini "Brain."
graph TD
User((User)) -->|Voice Input| STT["Whisper Large V3<br>(via Groq)"]
STT -->|Transcribed Text| Agent{"LangGraph ReAct Agent<br>(Gemini 2.0 Flash)"}
Agent -- "Requires Visuals?" --> Decision{Decision Node}
Decision -- Yes --> VisionTool[Vision Tool]
VisionTool -->|Capture Frame| OpenCV[OpenCV Webcam]
OpenCV -->|Image + Query| Llama["Llama 3.2 Vision<br>(via Groq)"]
Llama -->|Visual Analysis| Agent
Decision -- No --> Agent
Agent -->|Final Text Response| TTS["ElevenLabs / gTTS"]
TTS -->|Audio Output| User
| Component | Technology | Role |
|---|---|---|
| Orchestration | LangGraph / LangChain | Agent state management & tool routing |
| Brain (LLM) | Google Gemini 2.0 Flash | Core reasoning and instruction following |
| Visual Cortex | Llama 3.2 Vision (Groq) | Instant image-to-text analysis |
| Hearing (STT) | Whisper V3 (Groq) | Real-time speech transcription |
| Voice (TTS) | ElevenLabs | High-fidelity voice synthesis |
| Interface | Gradio & OpenCV | Web UI and webcam stream management |
git clone <repository-url>
cd Sora-Agentic-Assistant
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt
Create a .env file in the root directory:
GOOGLE_API_KEY="your_gemini_key"
GROQ_API_KEY="your_groq_key"
ELEVENLABS_API_KEY="your_elevenlabs_key"
- Start the Assistant:
python sora.py
- Enable Vision: Click the "Start Camera" button in the Gradio UI.
- Talk to Sora:
- Natural Query: "Hey Sora, what do you think of my setup?"
- Action: Sora will capture a frame, analyze the room with Llama 3.2, and speak its feedback.
- Chat History: Review the text logs in the sidebar while listening to the audio response.
- Long-term Memory: Integrate VectorDB for project-specific memory.
- Action Execution: Allow Sora to control local applications (Spotify, Browser).
- Multi-turn Vision: Continuous video stream analysis instead of snapshots.
Built with ❤️ for the 2026 Agentic AI Era.
Star this repo if you find it helpful! ⭐