A real-time multi-object tracking engine for autonomous-driving-style perception, written in modern C++17 — Kalman filter, Hungarian association, and lifecycle management in a multithreaded pipeline that holds real time on a laptop CPU.
Real YOLOv8 detections on real street footage, tracked by the C++ engine with stable per-object IDs:
A camera and a detector can say "there is a car, a person, a bicycle" in a single frame, but a single frame has no memory. This engine adds the memory: it follows each object across time, assigns a stable ID, and holds that ID even when the object is briefly occluded or the detector misses it for a frame. It is the tracking stage between an object detector and downstream planning in a perception pipeline, engineered to run in real time on a CPU, stay current under load, and not lose object identities.
- Real-time on a CPU. Sub-millisecond tracking latency; sustains 30–60 FPS on a laptop with frames to spare.
- Tracking-by-detection core. Per-object constant-velocity Kalman filter, IoU cost solved with the Hungarian algorithm, and explicit track lifecycle for births, confirmations, occlusions, and deaths.
- Graceful degradation under load. A drop-oldest queue sheds stale frames instead of building a backlog, so latency and memory stay bounded when the detector can't keep up.
- Multithreaded pipeline. Capture, inference, and tracking run as decoupled stages connected by bounded queues with deliberate drop-oldest and lossless policies.
- Built-in instrumentation. True p50/p95/p99 per stage and a load-sweep benchmark.
- Modular and swappable. Detector and source sit behind clean interfaces: ONNX YOLOv8 on live video, or offline replay for deterministic benchmarking.
- Zero-dependency core. The engine builds with only a C++17 compiler and CMake; OpenCV/ONNX are needed only for the live video path.
Measured by the code in this repository.
Real video (YOLOv8n detections on a 3000-frame street clip, 8-thread CPU):
| metric | value |
|---|---|
| frames processed | 3000 |
| detections tracked | 23,408 |
| effective tracking FPS | 30.0 |
| tracking latency p99 | 0.22 ms |
| end-to-end latency p99 | 0.39 ms |
| frames dropped | 0 |
Tracking accuracy (labeled sequence, bundled CLEAR-MOT evaluator):
| metric | value |
|---|---|
| MOTA | 88.3% |
| MOTP (IoU) | 0.908 |
| ID switches | 0 |
| false positives | 0 |
Accuracy (MOTA/MOTP) is reported on labeled data; on arbitrary footage the engine reports throughput and latency, since no ground truth exists to score against. Both paths run the same tracker.
Under detector overload (60 FPS capture, rising inference cost), the pipeline sheds frames so effective FPS settles at min(capture, detector_cap) and end-to-end latency stays bounded instead of growing, while tracking latency stays sub-millisecond and identities stay stable.
Capture ──drop-oldest──▶ Inference ──lossless──▶ Tracking
(thread) queue cap=2 (thread) queue cap=1024 (main thread)
frames ONNX YOLOv8 / Kalman predict
offline replay IoU + Hungarian
lifecycle mgmt
Three stages run concurrently and communicate only through bounded queues. The two queue policies are the core design decision: capture-to-inference drops the oldest frame under pressure (a moving platform wants the freshest frame, not a stale backlog), while inference-to-tracking is lossless (dropping there would corrupt sequential track state). Exactly one link can drop and one can block, which keeps the concurrency simple and bounds memory regardless of how far the detector falls behind.
- Motion model: 8-state constant-velocity Kalman filter on
[cx, cy, aspect, height, velocities], with noise scaled by box height so smaller, distant objects carry more uncertainty. - Association: IoU cost matrix between predicted track boxes and detections, solved optimally with the Hungarian algorithm, then IoU-gated.
- Lifecycle: born tentative, confirmed after N consecutive hits (filters clutter), coasts on the Kalman prediction through misses (occlusion handling), deleted after max-age misses.
Core engine — only a C++17 compiler and CMake required:
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -jOptional Python tooling (sequence generation, evaluation, real-video detection, demo GIF):
pip install numpy scipy ultralytics opencv-python pillowOffline replay on a labeled sequence, with accuracy:
python3 tools/generate_sequence.py
./build/tracker_app --det data/det.txt --out data/tracks.txt --fps 60
python3 tools/evaluate_mot.pyReal video — YOLOv8 detections, then C++ tracking, then an annotated clip:
python3 tools/detect_video.py clip.mp4 --out data/det.txt
./build/tracker_app --det data/det.txt --out data/tracks.txt --fps 30
python3 tools/visualize.py --video clip.mp4 --tracks data/tracks.txt --out data/tracked.mp4Standard benchmark — MOT16 ships detections and ground truth:
./build/tracker_app --det MOT16/train/MOT16-09/det/det.txt --out data/tracks.txt --fps 30
python3 tools/evaluate_mot.py --gt MOT16/train/MOT16-09/gt/gt.txt --tracks data/tracks.txtLive camera with on-screen visualization (requires OpenCV):
cmake -S . -B build -DWITH_OPENCV=ON && cmake --build build -j
./build/tracker_live 0 yolov8n.onnxLoad sweep:
bash tools/benchmark.sh 60Useful flags: --fps, --det-latency-ms (emulate detector cost), --detect-queue, --iou, --n-init, --max-age.
include/percep/ geometry, matrix, kalman, track, tracker, hungarian,
bounded_queue, profiler, pipeline, source, offline, onnx_detector
src/ kalman.cpp tracker.cpp hungarian.cpp pipeline.cpp
sources/offline.cpp detectors/onnx_detector.cpp viz/opencv_visualizer.cpp
main.cpp (offline + benchmark) main_live.cpp (live video)
tools/ generate_sequence.py evaluate_mot.py detect_video.py
visualize.py make_gif.py benchmark.sh
CMakeLists.txt
- Appearance embeddings (DeepSORT-style) to reduce ID switches through long occlusions.
- Ego-motion compensation and a constant-turn-rate motion model for a moving camera.
- TensorRT / ONNX Runtime detector backend with INT8 for higher live frame rates.
- Spatial gating of the cost matrix to scale association to very high object counts.
Issues and pull requests are welcome. For larger changes, open an issue first to discuss the approach. Please keep the core engine dependency-free and run the benchmark and evaluator before and after a change.
MIT — see LICENSE.