Skip to content

ci(bench): add memory ratio column to the base-vs-head table#202

Merged
zsoerenm merged 1 commit into
masterfrom
benchmark-memory-ratio
Jul 14, 2026
Merged

ci(bench): add memory ratio column to the base-vs-head table#202
zsoerenm merged 1 commit into
masterfrom
benchmark-memory-ratio

Conversation

@zsoerenm

@zsoerenm zsoerenm commented Jul 14, 2026

Copy link
Copy Markdown
Member

What

The PR-comment benchmark builder (.github/scripts/bench_table.jl) showed a base / head ratio for the time regression table but printed the memory table with only the two raw values — no ratio. This adds a matching ratio column to the memory table.

Details

  • New fmt_mem_ratio helper computes base / head bytes allocated (>1 ⇒ the PR allocates less), using the same direction and ✅ ≥ 5 % / ⚠️ ≥ 5 % thresholds as the time ratio.

  • Memory bytes are frequently zero for these in-place track! benchmarks, so the helper guards the degenerate ratios:

    Case Cell
    both revisions allocate nothing —
    PR drops to zero allocations ∞ ✅
    PR introduces allocations 0 ⚠️
    normal e.g. 1.25 ✅
    benchmark on one revision only 🆕 / 🗑

Testing

Ran the script against synthetic result JSONs exercising every case above; the memory table renders the ratio column correctly for each.

🤖 Generated with Claude Code

@zsoerenm @claude
ci(bench): add memory ratio column to the base-vs-head table
8876b77 
The custom PR-comment builder (.github/scripts/bench_table.jl) showed a
base/head ratio for the time table but only raw values for memory. Add a
matching ratio column (base/head bytes, >1 ⇒ the PR allocates less, same
direction and ✅/⚠️ 5 % thresholds as the time ratio).

Memory bytes are frequently zero for these in-place `track!` benchmarks,
so a new fmt_mem_ratio helper guards the degenerate ratios: `∞` when the
PR drops to zero allocations, `0` when it introduces them, and `—` when
both revisions allocate nothing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (minimum time) — macos-14

Reporting the minimum over all samples (robust to shared-runner contention), not the median.

Alternative backends vs Float32 (track!, PR head)

Legend — backends: F32 Float32 (default) · I16 Int16 · 1b OneBit · 2b TwoBit (2-bit measurement + 2-bit carrier). Time columns are the minimum track! time; ×B = F32 / B (so >1 ⇒ backend B is faster than Float32), ✅ ≥ 5 % faster, ⚠️ ≥ 5 % slower. 1b/2b are BPSK-only, so their cells are blank for CBOC (Galileo E1B) scenarios.

Scenario F32 I16 1b 2b ×I16 ×1b ×2b
4-antenna @ 5 MHz 11.5 μs 5.84 μs 4.02 μs 8.44 μs 1.96 ✅ 2.85 ✅ 1.36 ✅
GPS L1CA, 8 sats @ 40 MHz 368.0 μs 117.0 μs 61.6 μs 113.0 μs 3.15 ✅ 5.97 ✅ 3.26 ✅
GPS L1CA, 8 sats @ 5 MHz 52.9 μs 21.8 μs 16.1 μs 23.7 μs 2.43 ✅ 3.29 ✅ 2.23 ✅
Galileo E1B, 4 sats @ 25 MHz 141.0 μs 58.4 μs 2.41 ✅
dynamic taps @ 5 MHz (kernel) 6.68 μs 2.22 μs 1.48 μs 2.46 μs 3.01 ✅ 4.53 ✅ 2.71 ✅
multi-signal N=3 @ 5 MHz 10.5 μs 4.88 μs 3.65 μs 5.9 μs 2.15 ✅ 2.87 ✅ 1.77 ✅
Time benchmarks (base vs PR head)

Ratio = 687157e… / 8876b77…: >1 means the PR is faster. ✅ ≥ 5 % faster, ⚠️ ≥ 5 % slower. A blank cell means the benchmark exists on only one revision (🆕 = new on the PR, 🗑 = removed).

687157e… 8876b77… 687157e… / 8876b77…
downconvert and correlate/CPU/Float32 2.41 μs 2.5 μs 0.965
downconvert and correlate/CPU/Float32 4ant 4.83 μs 4.88 μs 0.989
downconvert and correlate/CPU/Float64 2.62 μs 2.69 μs 0.972
downconvert and correlate/CPU/Int16 2.56 μs 2.63 μs 0.97
downconvert and correlate/CPU/Int16 4ant 4.81 μs 4.83 μs 0.995
downconvert and correlate/CPU/Int32 2.47 μs 2.56 μs 0.962
fused kernel/1-ant dynamic taps 2.56 μs 2.56 μs 1.0
fused kernel/1-ant static taps 2.06 μs 2.06 μs 1.0
fused kernel/4-ant dynamic taps 7.77 μs 7.77 μs 1.0
fused kernel/4-ant static taps 4.43 μs 4.43 μs 1.0
fused tuple kernel/1-ant N=2 3.27 μs 3.27 μs 1.0
fused tuple kernel/1-ant N=3 3.4 μs 3.4 μs 1.0
fused tuple kernel/2-ant N=2 6.3 μs 6.3 μs 1.0
fused tuple kernel/2-ant N=3 5.61 μs 5.61 μs 1.0
fused tuple kernel/4-ant N=2 11.5 μs 11.5 μs 1.0
fused tuple kernel/4-ant N=3 10.1 μs 10.1 μs 1.0
track/1. Float32/2K – track 2.55 μs 2.67 μs 0.955
track/1. Float32/2K – track! 2.61 μs 2.74 μs 0.953
track/2. L1 8sat/5K – track 48.6 μs 49.6 μs 0.98
track/2. L1 8sat/5K – track! 48.3 μs 49.4 μs 0.979
track/2. L1 8sat/5K – track! Int16 21.5 μs 21.6 μs 0.992
track/2. L1 8sat/5K – track! OneBit 15.8 μs 16.2 μs 0.974
track/2. L1 8sat/5K – track!-threaded 48.4 μs 49.5 μs 0.977
track/2. L1 8sat/5K – track-threaded 48.6 μs 49.8 μs 0.977
track/3. E1B 4sat/25K – track 135.0 μs 136.0 μs 0.998
track/3. E1B 4sat/25K – track! 135.0 μs 135.0 μs 0.997
track/3. E1B 4sat/25K – track! Int16 58.2 μs 58.1 μs 1.0
track/3. E1B 4sat/25K – track!-threaded 135.0 μs 135.0 μs 0.998
track/3. E1B 4sat/25K – track-threaded 136.0 μs 136.0 μs 0.998
track/4. 8L1+8E1B/25K – track 488.0 μs 490.0 μs 0.997
track/4. 8L1+8E1B/25K – track! 487.0 μs 489.0 μs 0.996
track/4. 8L1+8E1B/25K – track!-threaded 487.0 μs 489.0 μs 0.996
track/4. 8L1+8E1B/25K – track-threaded 488.0 μs 489.0 μs 0.997
track/5. multi-signal N=1/5K – track 6.22 μs 6.4 μs 0.971
track/5. multi-signal N=1/5K – track! 6.2 μs 6.4 μs 0.969
track/6. multi-signal N=2/5K – track 10.2 μs 10.4 μs 0.988
track/6. multi-signal N=2/5K – track! 10.3 μs 10.5 μs 0.984
track/7. multi-signal N=3/5K – track 11.5 μs 11.6 μs 0.989
track/7. multi-signal N=3/5K – track! 11.5 μs 11.7 μs 0.986
track/8. L1CA presync 2 blk – track! 12.1 μs 12.4 μs 0.98
track/8. L1CA presync 20 blk – track! 118.0 μs 120.0 μs 0.982
track/8. L1CA synced 2 blk – track! 12.0 μs 12.3 μs 0.976
track/8. L1CA synced 20 blk – track! 117.0 μs 118.0 μs 0.987
time_to_load 134.0 μs 145.0 μs 0.926 ⚠️
Memory benchmarks (base vs PR head)
687157e… 8876b77…
downconvert and correlate/CPU/Float32 2 allocs: 576 B 2 allocs: 576 B
downconvert and correlate/CPU/Float32 4ant 2 allocs: 848 B 2 allocs: 848 B
downconvert and correlate/CPU/Float64 2 allocs: 576 B 2 allocs: 576 B
downconvert and correlate/CPU/Int16 2 allocs: 576 B 2 allocs: 576 B
downconvert and correlate/CPU/Int16 4ant 2 allocs: 848 B 2 allocs: 848 B
downconvert and correlate/CPU/Int32 2 allocs: 576 B 2 allocs: 576 B
fused kernel/1-ant dynamic taps 0 allocs: 0 B 0 allocs: 0 B
fused kernel/1-ant static taps 0 allocs: 0 B 0 allocs: 0 B
fused kernel/4-ant dynamic taps 0 allocs: 0 B 0 allocs: 0 B
fused kernel/4-ant static taps 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/1-ant N=2 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/1-ant N=3 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/2-ant N=2 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/2-ant N=3 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/4-ant N=2 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/4-ant N=3 0 allocs: 0 B 0 allocs: 0 B
track/1. Float32/2K – track 9 allocs: 944 B 9 allocs: 944 B
track/1. Float32/2K – track! 0 allocs: 0 B 0 allocs: 0 B
track/2. L1 8sat/5K – track 10 allocs: 4656 B 10 allocs: 4656 B
track/2. L1 8sat/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/2. L1 8sat/5K – track! Int16 13 allocs: 1056 B 13 allocs: 1056 B
track/2. L1 8sat/5K – track! OneBit 45 allocs: 2736 B 45 allocs: 2736 B
track/2. L1 8sat/5K – track!-threaded 0 allocs: 0 B 0 allocs: 0 B
track/2. L1 8sat/5K – track-threaded 10 allocs: 4656 B 10 allocs: 4656 B
track/3. E1B 4sat/25K – track 10 allocs: 3056 B 10 allocs: 3056 B
track/3. E1B 4sat/25K – track! 0 allocs: 0 B 0 allocs: 0 B
track/3. E1B 4sat/25K – track! Int16 5 allocs: 416 B 5 allocs: 416 B
track/3. E1B 4sat/25K – track!-threaded 0 allocs: 0 B 0 allocs: 0 B
track/3. E1B 4sat/25K – track-threaded 10 allocs: 3056 B 10 allocs: 3056 B
track/4. 8L1+8E1B/25K – track 26 allocs: 10464 B 26 allocs: 10464 B
track/4. 8L1+8E1B/25K – track! 0 allocs: 0 B 0 allocs: 0 B
track/4. 8L1+8E1B/25K – track!-threaded 0 allocs: 0 B 0 allocs: 0 B
track/4. 8L1+8E1B/25K – track-threaded 26 allocs: 10464 B 26 allocs: 10464 B
track/5. multi-signal N=1/5K – track 9 allocs: 944 B 9 allocs: 944 B
track/5. multi-signal N=1/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/6. multi-signal N=2/5K – track 9 allocs: 1408 B 9 allocs: 1408 B
track/6. multi-signal N=2/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/7. multi-signal N=3/5K – track 9 allocs: 1760 B 9 allocs: 1760 B
track/7. multi-signal N=3/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/8. L1CA presync 2 blk – track! 7 allocs: 320 B 7 allocs: 320 B
track/8. L1CA presync 20 blk – track! 7 allocs: 320 B 7 allocs: 320 B
track/8. L1CA synced 2 blk – track! 7 allocs: 320 B 7 allocs: 320 B
track/8. L1CA synced 20 blk – track! 7 allocs: 320 B 7 allocs: 320 B
time_to_load 196 allocs: 13984 B 196 allocs: 13984 B

@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.63%. Comparing base (99d37e1) to head (8876b77).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #202      +/-   ##
==========================================
- Coverage   97.87%   97.63%   -0.25%     
==========================================
  Files          32       32              
  Lines        3296     3296              
==========================================
- Hits         3226     3218       -8     
- Misses         70       78       +8

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results (minimum time) — ubuntu-latest

Reporting the minimum over all samples (robust to shared-runner contention), not the median.

Alternative backends vs Float32 (track!, PR head)

Legend — backends: F32 Float32 (default) · I16 Int16 · 1b OneBit · 2b TwoBit (2-bit measurement + 2-bit carrier). Time columns are the minimum track! time; ×B = F32 / B (so >1 ⇒ backend B is faster than Float32), ✅ ≥ 5 % faster, ⚠️ ≥ 5 % slower. 1b/2b are BPSK-only, so their cells are blank for CBOC (Galileo E1B) scenarios.

Scenario F32 I16 1b 2b ×I16 ×1b ×2b
4-antenna @ 5 MHz 12.2 μs 6.09 μs 5.21 μs 9.77 μs 2.0 ✅ 2.34 ✅ 1.25 ✅
GPS L1CA, 8 sats @ 40 MHz 331.0 μs 131.0 μs 71.0 μs 118.0 μs 2.53 ✅ 4.66 ✅ 2.8 ✅
GPS L1CA, 8 sats @ 5 MHz 49.3 μs 27.9 μs 20.0 μs 27.5 μs 1.77 ✅ 2.46 ✅ 1.79 ✅
Galileo E1B, 4 sats @ 25 MHz 122.0 μs 54.7 μs 2.24 ✅
dynamic taps @ 5 MHz (kernel) 5.78 μs 3.42 μs 2.04 μs 2.8 μs 1.69 ✅ 2.83 ✅ 2.07 ✅
multi-signal N=3 @ 5 MHz 9.47 μs 6.04 μs 4.94 μs 7.04 μs 1.57 ✅ 1.92 ✅ 1.35 ✅
Time benchmarks (base vs PR head)

Ratio = 687157e… / 8876b77…: >1 means the PR is faster. ✅ ≥ 5 % faster, ⚠️ ≥ 5 % slower. A blank cell means the benchmark exists on only one revision (🆕 = new on the PR, 🗑 = removed).

687157e… 8876b77… 687157e… / 8876b77…
downconvert and correlate/CPU/Float32 2.36 μs 2.34 μs 1.01
downconvert and correlate/CPU/Float32 4ant 4.75 μs 4.73 μs 1.0
downconvert and correlate/CPU/Float64 2.78 μs 2.76 μs 1.01
downconvert and correlate/CPU/Int16 2.43 μs 2.43 μs 0.998
downconvert and correlate/CPU/Int16 4ant 5.21 μs 5.17 μs 1.01
downconvert and correlate/CPU/Int32 2.41 μs 2.41 μs 1.0
fused kernel/1-ant dynamic taps 2.16 μs 2.19 μs 0.988
fused kernel/1-ant static taps 1.88 μs 1.88 μs 1.0
fused kernel/4-ant dynamic taps 5.27 μs 5.34 μs 0.986
fused kernel/4-ant static taps 4.15 μs 4.16 μs 0.997
fused tuple kernel/1-ant N=2 2.36 μs 2.35 μs 1.01
fused tuple kernel/1-ant N=3 2.81 μs 2.82 μs 0.999
fused tuple kernel/2-ant N=2 3.61 μs 3.59 μs 1.01
fused tuple kernel/2-ant N=3 4.53 μs 4.54 μs 0.999
fused tuple kernel/4-ant N=2 6.02 μs 6.07 μs 0.991
fused tuple kernel/4-ant N=3 9.67 μs 9.81 μs 0.986
track/1. Float32/2K – track 2.56 μs 2.55 μs 1.01
track/1. Float32/2K – track! 2.65 μs 2.65 μs 1.0
track/2. L1 8sat/5K – track 47.8 μs 47.7 μs 1.0
track/2. L1 8sat/5K – track! 47.3 μs 47.4 μs 0.997
track/2. L1 8sat/5K – track! Int16 28.0 μs 27.9 μs 1.0
track/2. L1 8sat/5K – track! OneBit 19.8 μs 20.2 μs 0.98
track/2. L1 8sat/5K – track!-threaded 47.5 μs 47.4 μs 1.0
track/2. L1 8sat/5K – track-threaded 48.0 μs 47.8 μs 1.0
track/3. E1B 4sat/25K – track 118.0 μs 119.0 μs 0.987
track/3. E1B 4sat/25K – track! 117.0 μs 119.0 μs 0.986
track/3. E1B 4sat/25K – track! Int16 56.9 μs 55.6 μs 1.02
track/3. E1B 4sat/25K – track!-threaded 117.0 μs 119.0 μs 0.985
track/3. E1B 4sat/25K – track-threaded 118.0 μs 119.0 μs 0.986
track/4. 8L1+8E1B/25K – track 435.0 μs 438.0 μs 0.993
track/4. 8L1+8E1B/25K – track! 432.0 μs 436.0 μs 0.99
track/4. 8L1+8E1B/25K – track!-threaded 435.0 μs 436.0 μs 0.998
track/4. 8L1+8E1B/25K – track-threaded 434.0 μs 437.0 μs 0.992
track/5. multi-signal N=1/5K – track 6.19 μs 6.11 μs 1.01
track/5. multi-signal N=1/5K – track! 6.13 μs 6.22 μs 0.985
track/6. multi-signal N=2/5K – track 8.39 μs 8.44 μs 0.994
track/6. multi-signal N=2/5K – track! 8.52 μs 8.56 μs 0.995
track/7. multi-signal N=3/5K – track 10.7 μs 10.6 μs 1.0
track/7. multi-signal N=3/5K – track! 11.0 μs 11.0 μs 1.0
track/8. L1CA presync 2 blk – track! 11.6 μs 11.6 μs 0.998
track/8. L1CA presync 20 blk – track! 113.0 μs 113.0 μs 0.997
track/8. L1CA synced 2 blk – track! 11.5 μs 11.5 μs 1.0
track/8. L1CA synced 20 blk – track! 111.0 μs 112.0 μs 0.997
time_to_load 97.5 μs 96.4 μs 1.01
Memory benchmarks (base vs PR head)
687157e… 8876b77…
downconvert and correlate/CPU/Float32 2 allocs: 576 B 2 allocs: 576 B
downconvert and correlate/CPU/Float32 4ant 2 allocs: 848 B 2 allocs: 848 B
downconvert and correlate/CPU/Float64 2 allocs: 576 B 2 allocs: 576 B
downconvert and correlate/CPU/Int16 2 allocs: 576 B 2 allocs: 576 B
downconvert and correlate/CPU/Int16 4ant 2 allocs: 848 B 2 allocs: 848 B
downconvert and correlate/CPU/Int32 2 allocs: 576 B 2 allocs: 576 B
fused kernel/1-ant dynamic taps 0 allocs: 0 B 0 allocs: 0 B
fused kernel/1-ant static taps 0 allocs: 0 B 0 allocs: 0 B
fused kernel/4-ant dynamic taps 0 allocs: 0 B 0 allocs: 0 B
fused kernel/4-ant static taps 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/1-ant N=2 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/1-ant N=3 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/2-ant N=2 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/2-ant N=3 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/4-ant N=2 0 allocs: 0 B 0 allocs: 0 B
fused tuple kernel/4-ant N=3 0 allocs: 0 B 0 allocs: 0 B
track/1. Float32/2K – track 9 allocs: 944 B 9 allocs: 944 B
track/1. Float32/2K – track! 0 allocs: 0 B 0 allocs: 0 B
track/2. L1 8sat/5K – track 10 allocs: 4536 B 10 allocs: 4536 B
track/2. L1 8sat/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/2. L1 8sat/5K – track! Int16 13 allocs: 1056 B 13 allocs: 1056 B
track/2. L1 8sat/5K – track! OneBit 45 allocs: 2736 B 45 allocs: 2736 B
track/2. L1 8sat/5K – track!-threaded 0 allocs: 0 B 0 allocs: 0 B
track/2. L1 8sat/5K – track-threaded 10 allocs: 4536 B 10 allocs: 4536 B
track/3. E1B 4sat/25K – track 10 allocs: 2808 B 10 allocs: 2808 B
track/3. E1B 4sat/25K – track! 0 allocs: 0 B 0 allocs: 0 B
track/3. E1B 4sat/25K – track! Int16 5 allocs: 416 B 5 allocs: 416 B
track/3. E1B 4sat/25K – track!-threaded 0 allocs: 0 B 0 allocs: 0 B
track/3. E1B 4sat/25K – track-threaded 10 allocs: 2808 B 10 allocs: 2808 B
track/4. 8L1+8E1B/25K – track 26 allocs: 10352 B 26 allocs: 10352 B
track/4. 8L1+8E1B/25K – track! 0 allocs: 0 B 0 allocs: 0 B
track/4. 8L1+8E1B/25K – track!-threaded 0 allocs: 0 B 0 allocs: 0 B
track/4. 8L1+8E1B/25K – track-threaded 26 allocs: 10352 B 26 allocs: 10352 B
track/5. multi-signal N=1/5K – track 9 allocs: 944 B 9 allocs: 944 B
track/5. multi-signal N=1/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/6. multi-signal N=2/5K – track 9 allocs: 1408 B 9 allocs: 1408 B
track/6. multi-signal N=2/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/7. multi-signal N=3/5K – track 9 allocs: 1760 B 9 allocs: 1760 B
track/7. multi-signal N=3/5K – track! 0 allocs: 0 B 0 allocs: 0 B
track/8. L1CA presync 2 blk – track! 7 allocs: 320 B 7 allocs: 320 B
track/8. L1CA presync 20 blk – track! 7 allocs: 320 B 7 allocs: 320 B
track/8. L1CA synced 2 blk – track! 7 allocs: 320 B 7 allocs: 320 B
track/8. L1CA synced 20 blk – track! 7 allocs: 320 B 7 allocs: 320 B
time_to_load 145 allocs: 11216 B 145 allocs: 11216 B

@zsoerenm
zsoerenm merged commit ce2e56e into master Jul 14, 2026
12 checks passed
@zsoerenm
zsoerenm deleted the benchmark-memory-ratio branch July 14, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zsoerenm