Detection stage looks non-deterministic — same input, ~85% different detections between runs

[Rome-1]

Hi — thanks again for open-sourcing the whole pipeline and the sample data. I've been trying to reproduce the vasculature render from the ultrasound-brain post, and while digging in I ran into something that looks like a reproducibility bug.

I installed ultratrace-ulm straight from the repo (no changes) and ran it on the public sanitized_neutral_ultratrace.h5. The beamforming is deterministic — I get identical beamformed compounds across runs. But the detection stage is a different story.

Running track with the adaptive recipe (--svd-method adaptive --frame-rate 222 --sigma-threshold 2.0 --knee-filter --temporal-sigma 0) on the same single acquisition twice, I get ~39,000 and ~38,000 detections — similar counts, but when I compare the actual detected positions at voxel resolution, only about 14% overlap. The median nearest-neighbor distance between the two runs' detections is ~0.27 mm, so it's not sub-pixel jitter — the peaks themselves move, and roughly 6 of every 7 detections are different between two runs of the identical code on identical data.

I think I know what's going on: spectral_centroid_cutoff scores the SVD modes with a phase-sensitive |rfft(u.real)|², and the eigenvectors from eigh on the float32 covariance (matrix @ matrix.conj().T) aren't stable across runs. So the chosen clutter cutoff — the number of modes removed — shifts from run to run, which changes the filtered magnitude everywhere, which moves all the z-score peaks. And since ~99.9% of the detections are a noise floor sitting right at 2σ (only ~48 of ~42,000 detections per acquisition actually link into tracks), that near-threshold floor reshuffles almost completely each run.

If that's right, it'd explain why I can't match the figure: the detection field I'm linking from is mostly a non-reproducible noise floor, so the tracks and the final render come out different — and much noisier — every time. A few things would let me get past it:

1. The clutter cutoff you actually used — the number of SVD components removed per acquisition, or an --svd-n-components value that reproduces the figure. Pinning that removes the non-determinism entirely, which is what I'd really like.
2. Is the public sanitized_neutral_ultratrace.h5 the exact input behind the figure, and what's the real acquisition frame rate? I couldn't find a PRF in the H5, so I've been passing --frame-rate 222, and that value feeds the adaptive cutoff through the 100 Hz tissue boundary, not just the color scale.
3. If it's easy, the actual tracks_smoothed.pkl / tracks.bin behind the figure, or the exact track + track-viewer command — I'm most curious about --smooth-method (gaussian vs 3dulm), --sigma-threshold, --min-track-length, and the viewer's --min-length. Even just the figure's track-length distribution (median frames per track) would tell me whether I should be getting long linked tracks or lots of short ones.
4. Which viewer produced the screenshot? The web/track_viewer in the repo doesn't have the s1/s2 switcher or the scale bar, so I'm assuming the figure came from a different front-end. If there's any masking or filtering between the smoothed tracks and the final image, that'd be good to know too.

Happy to share a minimal repro (two track runs on one acquisition plus the overlap computation) if it's useful. Thanks for taking a look.

[ennucore]

Hi! We used to have CPU and GPU versions of the tracking, with the GPU version being non-deterministic due to the use of a randomized SVD. I think the randomness wasn't coming from changes in the cutoff, but from the changes in the components themselves.
We pushed an update that makes the tracking algorithm better and removes the non-deterministic version.

The frontend in the blog is different. The ultratrace we released corresponds to s1, not s2.
The tracks (in the web viewer's format) are:
s1
s2
And here is the pickle file: pickle

On the PRF: the framerate is 222, the physical pulse repetition frequency is 4x that, because we send 4 plane waves per frame.

[Rome-1]

Hi, thanks for the detailed reply. This is really helpful and the tracks plus pickle are exactly what I needed.

On determinism: that matches what I found. The run-to-run variation comes from the randomized SVD (the temporal subspace itself changes between runs), not the spectral cutoff. On my side I'd already switched to a deterministic eigendecomposition of the complex128 Gram matrix, so I get bit-stable results now. Good to hear you removed the nondeterministic path too. I don't see that change in the public repo yet (the latest public commit predates your note). Is it in the private repo, and do you plan to push it public?

On the render: I rebuilt the s1 viewer from full_tracks_smoothed.pkl and it lines up closely with the blog figure. Thanks for clarifying s1 vs s2 and that the front end is separate.

One small clarification on the transmit sequence: each loop's raw IQ has 5 steered plane-wave slots (tx_delays is (5,134), a symmetric sweep with a 0-degree center), but you mentioned 4 plane waves per frame (which matches 888 Hz over 222.43 Hz). Do you compound all 5 in the released data, or only 4? I just want to be sure I'm compounding the same set you did.

One request: the pickle includes per-detection data for acquisition 0 only. Any chance you could share the detections for the other 215 acquisitions? I'm testing a global (min-cost-flow) re-tracking that already beats the greedy tracker per acquisition, and dataset-wide detections would let me evaluate it properly.

Thanks again, I really appreciate you engaging on this.