evaluate optional SONIC inference backends: upstream-style C++ and Python+CUDA

[MiaoDX]

Problem

#190 covers clarity and real-model validation for the current Python + CPU ONNX Runtime path.

A separate question remains: should roboharness support additional SONIC inference backends, especially:

1. upstream-style C++ inference path
2. Python + CUDA inference path (for example ONNX Runtime CUDA / TensorRT-capable deployment)

Right now the repo's SONIC implementation is Python orchestration plus onnxruntime on CPU. That is fine as a portable baseline, but it may diverge from upstream assumptions and it may not be the right path for performance-sensitive use.

Principle

If we support extra backends, they must be treated as real supported inference stacks, not experimental names backed only by mocks.

That means backend support must be justified by:

• a clear runtime contract
• real-model execution
• planner-mode validation
• tracking-mode validation

Goal

Evaluate and, if justified, add optional higher-performance SONIC inference backends without breaking the existing Python + CPU path.

Scope

1. Backend options to compare

Document the tradeoffs of:

• current Python + CPU ONNX Runtime path
• Python + CUDA ONNX Runtime path
• Python + TensorRT-backed ONNX Runtime path, if practical
• upstream-style C++ inference wrapper/integration

For each, capture:

• runtime dependencies
• portability
• CI feasibility
• expected latency / throughput benefits
• closeness to upstream implementation
• maintenance burden
• packaging complexity

2. Backend abstraction

If we support more than one path, define a small inference backend abstraction so controller logic does not hardcode a single runtime choice.

Examples of backend selection inputs:

• device=cpu|cuda
• backend=ort_cpu|ort_cuda|cpp
• environment variable or controller constructor argument

3. Real-model validation requirements

Any non-CPU backend must prove with real models:

• same input/output contract as current path
• planner mode works
• tracking mode works
• outputs are finite
• session loading works reliably
• behavior differences vs CPU path are documented

4. Rollout order

A reasonable rollout order might be:

1. keep current ort_cpu as default baseline
2. add optional ort_cuda support when CUDA provider is available
3. evaluate whether a separate C++ path is worth the complexity

But the issue should decide this based on evidence, not preference.

Non-goals

• Do not remove the current Python + CPU path
• Do not assume the upstream C++ path is automatically better without measurement
• Do not merge backend-specific complexity into the main controller logic without a clear interface
• Do not claim support for a backend if it only passes mocked tests

Acceptance criteria

• A design note compares CPU Python, Python+CUDA, and C++ backend options
• A recommended rollout order is documented
• Backend selection interface is defined if multiple paths are supported
• At least one GPU-capable Python backend is prototyped or rejected with evidence
• Upstream-style C++ backend feasibility is documented with concrete integration requirements
• Validation plan covers planner and tracking modes for every supported backend using real models

Related

• #190 — clarify current inference stacks and add real-model Python-path integration coverage
• #86 — planner-side SONIC controller
• #92 — encoder+decoder tracking pipeline

Notes

This issue is about optional backend support and evaluation.
It should not block validating the current Python path first, but any backend we add should meet the same real-model standard.