Skip to content

README.md

Latest commit

 

History

History
421 lines (351 loc) · 21.5 KB

README.md

File metadata and controls

421 lines (351 loc) · 21.5 KB
Raw

mot_tros — TROS-based Multi-Object Tracking on RDK X5

Standalone, self-contained MOT (Multi-Object Tracking) package for the D-Robotics RDK X5 board. Runs YOLO26 detection (6-class, int8 quantized) on the BPU and tracks with HybridSORT (default), ByteTrack, OCSort, OCLBoost, or BoT-SORT (StrongSORT is a stub). BoT-SORT, DeepOCSORT, and HybridSORT optionally fuse a committed MobileNetV4 ReID model. Input comes from a live MIPI camera (TROS mipi_cam → zero-copy HbmMsg1080P) or a local video file, with automatic file fallback when no camera is attached. Both Python and C++ implementations are provided and run out-of-the-box — the YOLO26s .bin is committed under models/ as the default, with YOLO26n and YOLO26l variants committed alongside (see Model variants).

Features

  • 🎯 BPU inference — YOLO26 6-class detection via TROS hbm_runtime (Python) or the raw hbDNN C API (C++), NV12 input, ~19 ms/frame (YOLO26s).
  • 🔄 Multiple trackers — ByteTrack, OCSort, HybridSORT (default), OCLBoost (BoostTrack), BoT-SORT. HybridSORT, DeepOCSORT, and BoT-SORT optionally fuse a real MobileNetV4 ReID model (see ReID); the rest are pure-motion. StrongSORT is a documented stub. See Trackers.
  • 📷 Live MIPI camera — TROS mipi_cam zero-copy HbmMsg1080P shared memory; automatic fallback to a video file when no sensor is connected.
  • 🎞️ Local video — pass any *.mp4/*.avi path to --source/--video.
  • 🐍 Python & C++ parity — same tracker library, two entry points.
  • 🔧 reg-aware post-processing — auto-switches between YOLO26 LTRB decode (reg_max=1) and YOLO11 DFL16 decode (reg_max=16) so both model families work from one code path.
  • 📊 Three model scales — YOLO26s (committed default), YOLO26n (smallest), YOLO26l (most accurate) share one deploy chain; swap by passing --model.
  • 🆔 Optional ReID — a committed MobileNetV4 256-d ReID model fuses appearance into BoT-SORT / DeepOCSORT / HybridSORT (bicycle/motorcycle stay pure-motion). See ReID.

Trackers

Six tracking algorithms are implemented as parallel pure-NumPy (Python) and pure-C++ (Eigen) classes in the same library. Five are usable end-to-end as pure-motion trackers (IoU/Kalman only, no ReID model); StrongSORT is a stub that raises NotImplementedError because it fundamentally requires a ReID embedding model, which is out of scope for this package.

Both the Python (mot_tros_demo.py) and C++ (mot_tros_demo) entry points expose the same --tracker flag with the same names, so the two are interchangeable. Default is hybridsort.

--tracker class motion model association strategy notes
bytetrack ByteTracker 8-D Kalman (xywh + v) two-stage IoU (high/low-conf split) fast, simple, strong default
ocsort OcSortTracker 7-D Kalman + velocity (OCF) observation-centric rescue + IoU handles occlusion / short gaps; update() also takes the frame
hybridsort (default) HybridSortTracker 9-D Kalman (adds score & velocity) three-stage: byte → OCR → TCM pure-motion HybridSORT; best on this detector's lower-conf outputs
occluboost OccluBoostTracker 7-D Kalman + velocity IoU + DLO boost (IoU/MHD/shape mix) BoostTrack-style boosting of low-conf matches
botsort BotSortTracker 8-D Kalman (xywh + v) two-stage IoU + proximity fuse BoT-SORT; optional ReID (min-fusion) via --reid-model
strongsort StrongSortTracker — — stub — raise NotImplementedError; needs a ReID model

The five usable trackers all have pure-motion paths (no appearance features, no camera-motion-compensation). BoT-SORT, DeepOCSORT, and HybridSORT additionally support optional ReID appearance fusion when a ReID model is attached (see ReID); without one they are byte-identical to their pure-motion form. The MOT17 HOTA column in the model table is each model's best tracker (e.g. YOLO26l + botsort = 60.84).

ReID

Three trackers — botsort, deepocsort, hybridsort — optionally fuse a real MobileNetV4 Conv-Small ReID model (mnv4_reid_256x256_nv12.bin, 256-d float embedding, 256×256 NV12 input, 1.77 MB). When --reid-model is set, update(dets, img) crops each high-confidence detection, runs it through the BPU model, and fuses the L2-normalized embedding with the motion cost. Each tracker uses its own fusion rule, mirroring boxmot 1:1:

  • botsort — min-fusion: cost = min(iou_cost, emb_cost), with an appearance gate (emb > appearance_thresh → 1) and an IoU-proximity gate (no overlap → ignore appearance). Track EMA α=0.9.
  • deepocsort — weighted injection: final_cost = -(iou + angle + emb_sim), with an IoU gate before the adaptive-weight (computeAwMaxMetric) step. Track EMA is confidence-adaptive (α = af + (1-af)(1-trust)).
  • hybridsort — weighted injection into the 4-point VDC cost: final = w0·(-(iou+angle)) + w1·emb_dist (emb is a cosine distance, opposite sign to DeepOCSORT). Requires MOT_HYBRIDSORT_EG > 0 (default 0 = ReID loaded but idle). Track EMA is confidence-weighted.

Class routing (v1): the model is a single joint person+vehicle model. person(0), car(2), truck(4), bus(5) get real ReID features; bicycle(1) and motorcycle(3) are routed to pure-motion (zero embedding → cosine distance 1.0 = appearance ignored → falls back to IoU). get_features(boxes, img, classes=None) returns a (N, 256) matrix row-aligned with the input boxes; classes=None (backward-compat) extracts all.

# Python — pass the ReID model; appearance gate tightened to 0.5 (real ReID)
# (the loose 0.9 default was for the old placeholder classifier)
python3 mot_tros_demo.py --source MOT17-02-DPM.mp4 --tracker deepocsort \
  --reid-model models/mnv4_reid_256x256_nv12.bin --reid-appearance-thresh 0.5

# HybridSORT additionally needs the appearance-term weight > 0
MOT_HYBRIDSORT_EG=0.5 python3 mot_tros_demo.py --source MOT17-02-DPM.mp4 \
  --tracker hybridsort --reid-model models/mnv4_reid_256x256_nv12.bin

# C++ (build with make -j2; 4+ cores stall the X5)
cd cpp && ./build.sh
./build/mot_tros_demo --model ../models/best_bpu_bayese_640x640_nv12.bin \
  --video ../MOT17-02-DPM.mp4 --tracker botsort --reg 1 --classes-num 6 \
  --reid-model ../models/mnv4_reid_256x256_nv12.bin --reid-appearance-thresh 0.5
# HybridSORT: MOT_HYBRIDSORT_EG=0.5 ./build/mot_tros_demo --tracker hybridsort ...

The BPU .bin consumes NV12 uint8 and applies mean/scale at compile time (data_mean_and_scale, ImageNet mean/std), so the ReID wrapper does BGR→NV12 (not boxmot's float-RGB /255+mean/std). The detector and ReID model coexist as independent hbDNN handles; inference is synchronous, so detect + reid run serially. The embedding output is float32 (backbone is int8 internally, output node stays float) — no dequant step needed; a guarded int8-dequant branch is present but stays a no-op. Input size (256×256) and embedding dim (256) are probed at runtime from the .bin, never hard-coded. See models/README.md for the model card and docs/REID_MNV4_BOARD_BENCH.md for the full board benchmark.

ReID performance on the board (MOT17-02-DPM, 600 frames, ~13 dets/frame)

Benchmark models: detector = YOLO26s 640x640 NV12 (best_bpu_bayese_640x640_nv12.bin, 6 classes, reg_max=1); ReID = MobileNetV4 Conv-Small 256x256 NV12 (mnv4_reid_256x256_nv12.bin, 256-d embedding). ReID runs serially per crop (~3.5 ms/crop end-to-end, 1.08 ms BPU-only), so it is the dominant cost when enabled. Measured with rendering disabled (--no-render) so the numbers reflect detect+track+ReID, not draw overhead.

path tracker ReID avg FPS ms/frame
Python DeepOCSORT OFF 14.25 54.4
Python DeepOCSORT ON (MNv4) 7.71 112.7
Python HybridSORT OFF 11.46 70.7
Python HybridSORT ON (MNv4) 6.72 131.2
C++ DeepOCSORT OFF 23.30 25.4
C++ DeepOCSORT ON (MNv4) 11.88 65.0
C++ HybridSORT OFF 23.09 25.8
C++ HybridSORT ON (MNv4) 11.92 65.0

C++ is ~1.5× faster than Python with ReID ON (11.9 vs 7.7 FPS) — modest, because per-crop ReID latency is BPU+NV12-bound and identical across languages (3.48 ms C++ vs 3.49 ms Python). The real MNv4 model also beats the prior placeholder (resnet18 logits) by 15–21% in Python FPS. Full decomposition (per-crop BPU-only vs full pipeline, frame breakdown, stress test) in docs/REID_MNV4_BOARD_BENCH.md.

Input read/decode included (RDK X5, C++, MOT17-02-DPM 1080p, 20 s)

The C++ summary now reports three per-frame medians: read/decode measures source->read() (file I/O plus video decode), detect+track+ReID excludes input and rendering, and read+detect+track+ReID is their per-frame sum. The following controlled runs use --no-render, so they intentionally exclude frame clone/drawing/display/video writing while including input read/decode.

tracker ReID read/decode (ms) detect+track+ReID (ms) read+detect+track+ReID (ms) avg FPS
HybridSORT OFF 17.04 25.61 42.70 23.15
HybridSORT ON (MNv4, MOT_HYBRIDSORT_EG=0.5) 17.24 58.52 76.16 13.03

So, with image reading included, the measured median end-to-end compute path is 42.70 ms/frame without ReID and 76.16 ms/frame with ReID for this 1080p video. Wall-clock FPS is lower than 1000 / median because it also includes occasional long-tail frame reads and loop/setup overhead. Reproduce with:

# Pure motion
./cpp/build/mot_tros_demo --model models/best_bpu_bayese_640x640_nv12.bin \
  --video MOT17-02-DPM.mp4 --tracker hybridsort --reg 1 --classes-num 6 \
  --duration 20 --no-render

# ReID must have a non-zero HybridSORT appearance weight to execute.
MOT_HYBRIDSORT_EG=0.5 ./cpp/build/mot_tros_demo \
  --model models/best_bpu_bayese_640x640_nv12.bin --video MOT17-02-DPM.mp4 \
  --tracker hybridsort --reid-model models/mnv4_reid_256x256_nv12.bin \
  --reg 1 --classes-num 6 --duration 20 --no-render

ReID tracking accuracy (MOT17-02 / MOT17-11, person-only HOTA)

HOTA (%) on the two held-out MOT17 sequences, same detector (YOLO26s) and the same tracker configs — pure-motion vs the MNv4 joint ReID vs an OSNet-x0.25 ReID baseline. HybridSORT + MNv4 is the best combination (55.81, +5.58 over its OSNet variant and +11.26 over pure-motion), which is why HybridSORT is the default tracker. BoT-SORT and DeepOCSORT barely move — their motion model is already strong on these pedestrian sequences.

tracker 纯运动 (pure-motion) MNv4-ReID OSNet-ReID
BoTSORT 53.53 53.60 53.60
DeepOCSORT 48.81 48.49 49.66
HybridSORT 44.55 55.81 50.23

Read-out: for HybridSORT the appearance term is the whole point — it lifts HOTA by ~11 points once a real ReID model is attached, and the lighter MNv4 (256-d, 1.77 MB) beats the heavier OSNet (512-d) by +5.58. For BoTSORT / DeepOCSORT the IoU/velocity motion model already captures most of the association signal on MOT17 pedestrians, so ReID adds little accuracy — pick them for speed, pick HybridSORT+ReID for accuracy.

⚠️ Benchmark caveat: always measure ReID overhead with rendering OFF. With frame.clone() + drawTracks() + putText() enabled, rendering dominates (~40 ms/frame at 1080p) and masks the ReID cost — all configs converge to ~22 FPS regardless of ReID, which is misleading. And remember MOT_HYBRIDSORT_EG (HybridSORT's appearance weight) defaults to 0, so HybridSORT's ReID is loaded but idle unless that env var is set.

Selecting a tracker

# Python — now supports --tracker (mirrors the C++ flag)
python3 mot_tros_demo.py --source MOT17-02-DPM.mp4 --tracker botsort
python3 mot_tros_demo.py --source MOT17-02-DPM.mp4 --tracker ocsort --save out.mp4

# C++
cd cpp && ./build.sh
./mot_tros_demo --model ../models/best_bpu_bayese_640x640_nv12.bin \
                --video ../MOT17-02-DPM.mp4 --tracker hybridsort \
                --reg 1 --classes-num 6 --save out_cpp.mp4

Common tuning flags (both entry points): --track-thresh (activation), --match-thresh (association IoU/threshold), --track-buffer (max lost frames). The Python demo also reads MOT_TRACK_THRESH / MOT_MATCH_THRESH / MOT_TRACK_BUFFER env vars; the C++ demo additionally takes --min-conf and --det-freq.

Model

The production model ships in models/:

field value
file models/best_bpu_bayese_640x640_nv12.bin
architecture YOLO26s, 6 classes (person bicycle car motorcycle truck bus)
reg_max 1 (direct LTRB box decode, not DFL16)
input 1×3×640×640, NV12 runtime / RGB train, data_scale 1/255
outputs 6 NHWC tensors (3 strides × cls/box)
quantization int8, march=bayes-e, max-percentile 0.99995 calibration
size / MD5 10.75 MB / b438704015efbf46cb392ef9a69afa70
board latency ~19.4 ms / frame
merged-val mAP50 / mAP50-95 0.626 / 0.397
MOT17 best HOTA (botsort) 53.60

Trained on BDD100K + CrowdHuman + MOT17 (rare classes oversampled). Two high-precision quantization variants (KL-divergence, per-layer mix) were A/B-tested on the board and both measured worse than max-int8 (see Quantization results and docs/REPORT_NOTES.md), so max-int8 is the committed model.

Model variants

Three YOLO26 scales share one deploy chain (same 6 classes, reg_max=1, int8, march=bayes-e, identical max-percentile calibration). All three bins are committed under models/; YOLO26s is the default, swap to n / l by passing --model.

model bin size merged-val mAP50 / mAP50-95 MOT17 best HOTA board latency committed
YOLO26s (default) 10.75 MB 0.626 / 0.397 botsort 53.60 19.4 ms ✅ ✅
YOLO26n 3.35 MB 0.555 / 0.346 hybridsort 50.49 10.0 ms ✅ ✅
YOLO26l 26.74 MB 0.688 / 0.443 botsort 60.84 49.1 ms ✅ ✅

Accuracy scales s→n→l as expected (l is +4.6 mAP50-95 over s, +7.2 MOT17 HOTA). Latency scales with bin size: n is ~2× faster than s, l is ~2.5× slower — all three measured on-board (100 held-out BDD imgs, min-of-7, std <0.5 ms, BPU inference only; full-pipeline e2e FPS on MOT17 video is n≈5.1 / s≈4.5 / l≈4.1). Correctness audited on-board across 3 dimensions (output-shape + box legality, class/score distribution, cross-scale IoU consistency) — all PASS, no degenerate boxes, no rare-class dropout. Swap any variant by passing --model models/<bin>.bin --reg 1. See docs/BOARD_BENCH_NL.md for the full bench log.

Per-class mAP50 (merged val, 14370 imgs): person/car are the strongest across all scales; bicycle/motorcycle lag (small targets, fewer samples).

class YOLO26s YOLO26n YOLO26l
person 0.816 0.767 0.862
car 0.796 0.751 0.829
truck 0.625 0.558 0.679
bus 0.617 0.546 0.672
bicycle 0.472 0.372 0.553
motorcycle 0.431 0.339 0.531

Layout

mot_tros/
├── mot_tros_demo.py        # Python main: detect + multi-tracker (--tracker)
├── mot_detector.py         # Detector wrapper (MOTDetector)
├── mot_tracker.py          # Pure-NumPy trackers (HybridSort, ByteTrack, ...)
├── reid_extractor.py       # BPU ReID wrapper (NV12 crops → 256-d L2-norm emb)
├── ultralytics_yolo_det.py # TROS hbm_runtime inference wrapper (reg-aware)
├── preprocess.py           # BGR→NV12, letterbox
├── postprocess.py          # DFL/LTRB decode, NMS, coord scaling
├── cam_capture.py          # TROS mipi_cam ROS subscription (video fallback)
├── cam_fifo_bridge.py      # Python→FIFO bridge that feeds the C++ binary
├── bench_*.py              # MOT17 FPS / ReID latency / frame-breakdown benches
├── models/                 # committed production .bin + model notes
├── tests/test_reid_fusion.py  # ReID class-routing / pure-motion fallback tests
├── docs/REPORT_NOTES.md    # report material: arch / deploy / quant experiments
├── docs/BOARD_BENCH_NL.md  # n/s/l model board benchmark log
├── docs/REID_MNV4_BOARD_BENCH.md  # MNv4 ReID board benchmark (speed + latency)
└── cpp/                    # C++ implementation
    ├── CMakeLists.txt
    ├── mot_tros_demo.cc    # main: FrameSource (file/V4L2/FIFO) + detect + track
    ├── yolo_detector.hpp   # hbDNN-based YOLO detector (reg-aware, header-only)
    ├── reid_extractor.hpp  # hbDNN-based ReID wrapper (header-only)
    ├── mot_tracker.{hpp,cpp}  # 6 trackers (ByteTrack/OCSort/HybridSort/OCLBoost/BotSort + StrongSORT stub)
    ├── kalman_filter.{hpp,cpp}
    ├── build.sh            # builds with -j2 (4+ cores stall the X5)
    └── run_cpp.sh          # run file or camera mode

Prerequisites (on the X5)

  • TROS humble: source /opt/tros/humble/setup.bash
  • Python: hbm_runtime, numpy, scipy, opencv-python (all preinstalled)
  • C++ build: cmake, g++, OpenCV, Eigen3, /usr/include/dnn + libdnn
  • MIPI sensor: an sc132gs is connected to this board (1088×1280 bgr8).

The model is already in models/, so no download step is needed. For a test video, drop one next to the scripts (e.g. symlink or copy MOT17-02-DPM.mp4), or pass any local video file via --source / --video.

Python

# Local video file (any mp4/avi on the board):
python3 mot_tros_demo.py --source MOT17-02-DPM.mp4
# Live MIPI camera (with automatic file fallback):
python3 mot_tros_demo.py
# Override output / duration:
python3 mot_tros_demo.py --source MOT17-02-DPM.mp4 --save out.mp4 --duration 30

Output: output_tros.mp4. Defaults target the bundled YOLO26s model (6 classes, reg=1, score_thres=0.35). To run a YOLO11 model instead:

python3 mot_tros_demo.py --model <yolo11.bin> --reg 16 --classes-num 80

Key flags: --reg (1=YOLO26 LTRB, 16=YOLO11 DFL), --classes-num, --score-thres, --tracker, --duration.

C++

cd cpp
./build.sh                       # cmake + make -j2
# Local video file, loop to ~60 s:
./mot_tros_demo --model ../models/best_bpu_bayese_640x640_nv12.bin \
                --video ../MOT17-02-DPM.mp4 --tracker hybridsort \
                --reg 1 --classes-num 6 --save output_tros_cpp.mp4
# Live MIPI camera (launches mipi_cam + bridge):
./run_cpp.sh camera

The camera mode starts the TROS mipi_cam node (zero-copy HbmMsg1080P on /hbmem_img), runs cam_fifo_bridge.py to decode frames and pipe them to the C++ binary through a FIFO, and the C++ binary runs detection+tracking. This keeps ROS out of the C++ build while still using the real MIPI sensor.

If no camera is connected, both Python and C++ fall back to MOT17-02-DPM.mp4 so the pipeline can always be validated. Defaults are YOLO26s (6 classes, reg=1); pass --reg 16 --classes-num 80 for a YOLO11 model.

The C++ detector reg=1 path mirrors the Python decode_ltrb_boxes math but could not be compiled on x86 (no hbDNN headers / aarch64 OpenCV). It is expected to build cleanly on the X5; if a tweak is needed it is localized to the cfg_.reg == 1 branch in cpp/yolo_detector.hpp.

Quantization results

Three int8 calibration strategies were A/B-tested on 100 held-out BDD images (excluding the calibration set, rare classes oversampled), ab_compare.py, min-of-3 latency. Lower is worse (fewer detections):

calibration detections @0.25 detections @0.15 latency
max-percentile (committed) 1372 1789 19.4 ms
mix (per-layer cosine select) 1234 (−10%) 1615 (−10%) 19.4 ms
kl (KL-divergence) 1100 (−20%) 1470 (−18%) 19.4 ms

Key findings (counter-intuitive): KL globally suppresses the classification logit and drops rare classes (bicycle/motorcycle) entirely; mix repairs the rare-class loss but still loses 10% of car/person. Mix's per-layer cosine selection picked layers that were more cosine-similar to the float model on all 6 outputs yet detected fewer objects — output cosine ≠ mAP. Full data and methodology in docs/REPORT_NOTES.md.

Verification on the board

  1. Python: python3 mot_tros_demo.py --source MOT17-02-DPM.mp4 --reg 1 --classes-num 6 --save out.mp4 → person boxes + HybridSORT IDs render correctly (regression check: before the reg branch, DFL16 decode collapsed every box to zero).
  2. C++: cd cpp && ./build.sh && ./mot_tros_demo --model ../models/best_bpu_bayese_640x640_nv12.bin --video ../MOT17-02-DPM.mp4 --reg 1 --classes-num 6 --save out_cpp.mp4.

Notes

  • The C++ tracker lib is identical to the rdk_model_zoo MOT sample; only the demo entry point was rewritten to support camera/FIFO sources and a target duration.
  • HybridSORT here ships with an optional ReID appearance branch (off by default; enabled with --reid-model + MOT_HYBRIDSORT_EG). With ReID off it is the pure-motion variant matching the other trackers in this repo.
  • reg_max=1 is why YOLO26 cannot reuse the YOLO11 DFL16 decode — see models/README.md.

License

Released under the Apache License 2.0. See LICENSE for details.