test(bench): sustained-duration measurement for MemWAL HNSW parity bench

[touch-of-grey]

Summary

Adds sustained-duration measurement to the MemWAL HNSW parity bench so it reports steady-state throughput under continuous load rather than a short burst. This follows up the AVX-512 distance work in #7009.

Changes (bench-only, no library changes):

• --insert-seconds / --query-seconds: run the write (graph build) and read (query) workloads in a loop for a fixed wall-clock duration; report aggregate throughput over all passes (insert_passes / query_passes).
• insert_core breakdown: times the insertion itself separately from per-build graph allocation + teardown.
• Both knobs added to the Lance bench and the hnswlib reference bench; run_parity_suite.sh gains INSERT_SECONDS / QUERY_SECONDS.

Motivation: a sub-second query window gave noisy/optimistic numbers and hid AVX-512 frequency throttling. Measuring 30 s of continuous load makes read/write parity (and where it doesn't hold) reproducible.

Latest perf results (merged main, c7i.12xlarge, 48 threads, dim=1024, m=12, ef=64, k=10)

Sustained 30 s read + 30 s write per size; AVX-512 throttles 3.78 GHz → ~2.5 GHz under all-core load (affects both impls).

Read (query_qps), Lance / hnswlib:

rows	ratio
100k	1.01
500k	0.995
1M	0.996

Write — insertion compute only (insert_core), Lance / hnswlib:

rows	ratio
100k	0.99
500k	0.98
1M	0.96

Write — end-to-end incl. per-build graph alloc + teardown:

rows	ratio
100k	0.96
500k	0.89
1M	0.87

Takeaways the improved bench makes visible:

• Read is at parity under sustained throttled load (confirms perf(mem_wal): match hnswlib throughput via runtime AVX-512 f32 distance #7009 holds; the burst window wasn't hiding a regression).
• Insertion compute is at parity — AVX-512 distance keeps pace even while downclocked.
• The end-to-end write gap at scale is entirely graph allocation/teardown (Lance's per-node Vec/Mutex/Arc vs hnswlib's flat arrays), not the algorithm — and it's allocator-sensitive: with mimalloc/jemalloc as the global allocator Lance is actually faster than hnswlib (≈1.08–1.25×). No in-tree change is warranted; using a modern allocator for the memtable workload closes it.

cc @jackye1995 — please review.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[touch-of-grey]

@jackye1995 ping for review. Bench-only follow-up to #7009: adds 30s sustained read/write measurement + insert_core breakdown. Latest results in the description — sustained read at parity (0.99-1.01x), insertion compute at parity (0.96-0.99x); the end-to-end write gap at scale is purely graph alloc/teardown (allocator-sensitive: mimalloc/jemalloc make Lance faster), not the algorithm.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[jackye1995]

Looks good to me