Every major vision system today is stateless. It sees a frame. It outputs boxes. It forgets.
┌──────────────────────────────────────────┐
Frame t=0 ──────►│ Detector │──► [person, bicycle, car] │
Frame t=1 ──────►│ Detector │──► [person, bicycle, car] │ ← no memory
Frame t=2 ──────►│ Detector │──► [person, bicycle, car] │ ← no context
└──────────────────────────────────────────┘
No depth. No identity. No relationships. No history. No prediction.
CortexVision is built to fix that.
Video Stream
│
▼
┌───────────────────────────────────────────────────────────┐
│ PERCEPTION YOLO-World · Depth Pro · SAM2 │
│ ByteTrack · RTMPose │
├───────────────────────────────────────────────────────────┤
│ FUSION Detection + Depth + Segmentation │
│ → Metric 3D object positions │
├───────────────────────────────────────────────────────────┤
│ SCENE GRAPH Spatial relations between all entities │
│ left_of · behind · near · overlapping │
├───────────────────────────────────────────────────────────┤
│ MEMORY Every object. Every interaction. │
│ Full trajectory history. │
├───────────────────────────────────────────────────────────┤
│ WORLD MODEL Global state. Event engine. Predictions. │
├───────────────────────────────────────────────────────────┤
│ NAVIGATION BEV · Occupancy Grid · Path Planning │
├───────────────────────────────────────────────────────────┤
│ REASONING VLM over world state. Future prediction. │
└───────────────────────────────────────────────────────────┘
│
▼
Structured world understanding — not just bounding boxes
Can a machine construct and maintain a temporally consistent world model from monocular video streams using multimodal perception, scene graphs, memory, and predictive reasoning?
Traditional vision systems answer: What objects exist?
CortexVision answers: What exists, where is it, how is it changing, what is it interacting with, and what will happen next?
Five models run in parallel on every frame:
| Model | Task | Output |
|---|---|---|
| YOLO-World | Open-vocabulary detection | Bounding boxes for any object class |
| Grounding DINO | Language-conditioned detection | Text-prompted localization |
| Depth Pro | Metric monocular depth | Per-pixel real-world distances (meters) |
| SAM2 | Video segmentation | Per-object pixel masks across frames |
| ByteTrack | Multi-object tracking | Persistent IDs across 500+ frames |
| RTMPose | Human pose estimation | Body keypoints for action understanding |
The first research challenge: fusing three imperfect signals into one coherent 3D representation.
Detection ──┐
├──► Depth-Aware Fusion ──► Object @ (x, y, z) meters
Depth ──┤
│ "Person_12 is 2.1m away, slightly left of center"
Segmentation ─┘ "Bicycle_03 is 4.5m away, directly ahead"
Not "person is closer than bicycle" — actual metric positions in 3D space.
Components:
Detection Fusion— associates depth map regions with detected bounding boxesDepth Fusion— lifts 2D boxes into metric 3D coordinatesObject Association— links detections across models into unified entity representationsScene Builder— assembles all entities into a coherent frame-level scene
A core research contribution. Every frame produces a metric scene graph — not just what exists, but the structured spatial relationships between everything.
Person_12 ──[1.8m · in_front_of]──► Bicycle_03
Person_12 ──[0.4m · near]──────────► TrafficCone_07
Bicycle_03 ──[left_of]──────────────► ParkedCar_11
ParkedCar_11 ──[contains]───────────► Passenger_05
Supported relations:
left_of right_of in_front_of behind
near overlapping contains approaching
The graph is serialized and stored — queryable at any point in time.
Unlike any frame-based system, CortexVision never forgets.
Every tracked entity gets a persistent memory record:
Person_12
──────────────────────────────────────────────────
frame 1 │ Appeared at (2.3m, left-center)
frame 12 │ Walking east at ~1.2 m/s
frame 31 │ Slowing — approaching Bicycle_03
frame 48 │ Reached Bicycle_03
frame 61 │ Mounted bicycle
frame 62+ │ Accelerating northeast
Four memory stores:
| Store | What it tracks |
|---|---|
Object Memory |
State history of every entity |
Trajectory Store |
Full position + velocity timelines |
Interaction Memory |
Every recorded entity-entity interaction |
History Buffer |
Raw frame-level scene snapshots |
All memory, all entities, all events — unified into a single coherent world state.
Entity Store ── all known objects + current states
World State ── global snapshot at time T
Event Engine ── detects and logs semantic events
("Person mounted bicycle", "Vehicle stopped")
Future Predictor ── forecasts likely next states
Transforms the world model into spatial representations for robotics and assistive systems.
- BEV Projection — Bird's-Eye View: top-down 2D map from monocular video
- Occupancy Grid — per-cell passability scores across the scene
- Corridor Detection — free navigable paths identified in real time
- Path Planning — safe route generation accounting for dynamic obstacles
The top of the stack. A VLM receives the full world state and reasons over it in natural language.
Input: World state @ T=247 + event history + future predictor output
Output: "Person_12 has been stationary for 4 seconds near an obstacle.
Based on trajectory, they are likely preparing to cross.
Collision risk with Vehicle_03 is elevated — 73% probability
of path intersection within 6 seconds."
Reasoning capabilities:
Temporal Understanding— what changed and whenVLM Reasoning— natural language queries over the world stateEvent Explanation— why did this happen?Future Prediction— what happens next?
| ID | Title | Description |
|---|---|---|
| C1 | Depth-Aware Object Fusion | Lifting 2D detections into metric 3D using monocular depth |
| C2 | Metric Scene Graph Generation | Spatially-grounded relational graphs with real-world distances |
| C3 | Persistent Object Memory | Long-horizon state tracking with full interaction history |
| C4 | Temporal Scene Graph Evolution | Tracking how spatial relationships change over time |
| C5 | Predictive World Modeling | Forecasting future interactions from observed patterns |
| C6 | Navigation-Oriented Scene Understanding | Perception representations designed for path planning |
|
🎯 Open-Vocabulary Detection Detect any object — not just fixed label sets. Models: YOLO-World, Grounding DINO |
📏 Metric Depth Estimation Real distances. Not relative rankings. Model: Depth Pro |
|
🔲 Precise Segmentation Pixel-level object boundaries. Model: SAM2 |
🔗 Persistent Tracking Same identity. Hundreds of frames. Model: ByteTrack |
|
⏳ Temporal Scene Graphs Relationships that evolve: |
🔮 Future Event Prediction What happens next: |
CortexVision/
│
├── 📱 app/ ← Demo entry points & inference scripts
├── 📊 benchmarks/ ← Evaluation harness & baseline comparisons
├── 🗃️ checkpoints/ ← Model weights (gitignored)
├── ⚙️ configs/ ← YAML configs for every component
├── 🧠 core/ ← Fusion, scene graph, memory, world model
├── 📦 datasets/ ← Loaders & preprocessing pipelines
├── 📚 docs/ ← Architecture docs, diagrams, writeups
├── 📈 evaluation/ ← Metrics: depth, tracking, scene graph quality
├── 🧪 experiments/ ← Run logs, ablations, result snapshots
├── 🤖 models/ ← Wrappers: YOLO, SAM2, Depth Pro, ByteTrack
├── 📓 notebooks/ ← Analysis & visualization notebooks
├── 🖼️ outputs/ ← Inference outputs (gitignored)
├── 📄 paper/ ← Draft writeup & figures
├── 🚀 scripts/ ← Training, inference, export utilities
├── ✅ tests/ ← Unit tests per module
├── 🛠️ tools/ ← Visualization & debugging helpers
└── 🔧 utils/ ← Shared utilities & helpers
┌──────────────┬──────────────────────────────────────────────────────┐
│ Perception │ YOLO-World · Grounding DINO · Depth Pro │
│ │ SAM2 · ByteTrack · RTMPose │
├──────────────┼──────────────────────────────────────────────────────┤
│ Deep Learning│ PyTorch 2.0+ · torchvision · CUDA │
├──────────────┼──────────────────────────────────────────────────────┤
│ Vision │ OpenCV · Pillow · scikit-image │
├──────────────┼──────────────────────────────────────────────────────┤
│ Data │ NumPy · Pandas · NetworkX (scene graphs) │
├──────────────┼──────────────────────────────────────────────────────┤
│ Reasoning │ Hugging Face Transformers (VLM integration) │
├──────────────┼──────────────────────────────────────────────────────┤
│ Visualization│ Streamlit · Matplotlib · Gradio │
├──────────────┼──────────────────────────────────────────────────────┤
│ Language │ Python 3.10+ │
└──────────────┴──────────────────────────────────────────────────────┘
CortexVision sits at the intersection of five active research areas:
| Area | Relevance |
|---|---|
| World Models | Maintaining internal representations of dynamic environments |
| Scene Graph Generation | Structured relational understanding of visual scenes |
| Monocular Depth Estimation | Inferring 3D structure from 2D video — no LiDAR required |
| Multi-Object Tracking | Persistent identity across long temporal horizons |
| Embodied AI | Perception systems designed to act, not just observe |
Active research — not a polished library. Components built, tested, and ablated iteratively.
Perception & Core
- Multi-model perception pipeline (YOLO-World + Depth Pro + SAM2 + ByteTrack)
- Metric depth fusion layer
- Per-object 3D position estimation
- Object memory store
- Trajectory tracking & storage
- Metric scene graph generation
World Modeling
- Temporal scene graph evolution (in progress)
- World model entity store (in progress)
- Event engine & semantic event detection (planned)
- Future predictor module (planned)
Reasoning & Navigation
- VLM reasoning integration (planned)
- BEV projection & occupancy grid (planned)
- Path planning layer (planned)
- Full evaluation suite (planned)
