Omni Studio

Video download + multimodal annotation tool.

Quick Start

Backend

cd backend
uv sync
cp .env.example .env
# Edit .env — fill in AIHUBMIX_API_KEY

uv run uvicorn app.main:app --reload --port 8000

Frontend

cd frontend
npm install
npm run dev

Open http://localhost:5173

---

Architecture

Omni-Studio/
├── backend/           FastAPI + SQLite
│   └── app/
│       ├── routers/   download / videos / ai / config
│       ├── services/  downloader (yt-dlp) · ai_service (litellm) · export
│       └── models.py  SQLAlchemy ORM
├── frontend/          React + Vite + Ant Design
│   └── src/
│       ├── pages/     Download · VideoList · Annotate · Settings
│       └── components/Annotate/  VideoPlayer · SegmentTimeline · RightPanel
└── storage/
    ├── videos/        hash.mp4
    └── audio/         hash.wav

Features

• Download YouTube / Bilibili / local videos via yt-dlp
• Real-time download progress (SSE)
• Audio extracted as 16kHz mono WAV
• Hash-named storage (SHA256, 16-char)
• Visual segment timeline (drag boundaries, add/delete, no gaps)
• Description mode: video_describe + audio_describe + speech_describe per segment
• Reason mode: per-segment descriptions + global Q&A (prompt / understanding / reasoning_process / answer)
• AI description via Gemini 2.0 Flash (native video+audio multimodal)
• AI reasoning via Claude Sonnet 4.6
• STT via Whisper
• Export as JSON / JSONL, optionally bundled with media as ZIP
• Configurable type / sub-type taxonomies and AI model lists

AI Models (via aihubmix + LiteLLM)

Task	Default Model
Video+Audio describe	gemini/gemini-2.0-flash
Reasoning Q&A	anthropic/claude-sonnet-4-6
Speech transcription	whisper-1