Autoresearch/perf optimization 20260320 - #6
Draft
jahfer wants to merge 46 commits into
Draft
Conversation
Replace heap-allocated AtomicPtr<Marker<K,V>> with inline atomic fields (AtomicU8 for state, AtomicIsize for move destination). The K,V data in InsertCell/DeleteCell markers was never read by other threads - only the operation state matters for synchronization. This eliminates all heap allocations from the cell operation hot path, particularly benefiting rebalance-heavy workloads. Benchmark improvements: - Random inserts: +63% to +281% throughput - Point lookups: +5% to +9% at larger dataset sizes - Mixed workloads: +8% to +14% - Sequential inserts: +2% to +32%
Relax SeqCst atomic orderings to Acquire on read paths for improved performance, especially on ARM/AArch64 where SeqCst requires full memory barriers. Changes: - CellGuard::from_raw(): version, marker_state, move_dest loads → Acquire - CellGuard::cache(): version load → Acquire - btree_map rebalance: source/dest cell reads → Acquire The seqlock validation re-read in from_raw() remains SeqCst to prevent reordering of data reads past the validation check on weakly-ordered architectures. Acquire ordering is sufficient because we only need pairwise synchronization between a writer and its readers for each cell, not total ordering across unrelated cells. Added 6 tests to verify Acquire semantics work correctly.
Replace expensive key.clone() in CellGuard::from_raw() with a simple is_some() check. The actual key/value cloning is now deferred to cache() when the data is actually needed. This optimization reduces overhead in the hot iteration path where code frequently checks is_empty() to find empty slots or skip cells without needing their actual contents. The seqlock validation still protects against torn reads since the is_some() check happens within the version-validated window. Before: clone key on every from_raw() call (~10% overhead) After: check is_some() in from_raw(), clone only in cache() Added tests verifying deferred cloning behavior.
Change version.store() and marker_state.store() calls after UnsafeCell writes from SeqCst to Release ordering in the insert paths of btree_map.rs. Release ordering is sufficient because it ensures all preceding writes (including UnsafeCell writes to key/value) are visible to any thread that subsequently performs an Acquire load on these atomics. This is the "publication" side of the Release/Acquire pattern - we don't need total ordering with writes to other cells, only pairwise synchronization with readers of this cell. Changes: - 6 stores relaxed: lines ~140, ~144, ~174, ~178, ~223, ~227 - Added 5 tests verifying Release/Acquire synchronization semantics All 116 tests pass. Step 2 of atomic ordering relaxation plan.
Change compare_exchange and compare_exchange_weak orderings from SeqCst/SeqCst to AcqRel/Acquire for marker_state CAS operations. Changes: - cell.rs: update_marker_state() CAS now uses AcqRel/Acquire - btree_map.rs: Move marker CAS uses AcqRel/Acquire - btree_map.rs: Best-effort cleanup CAS (version + marker) uses AcqRel/Acquire Rationale: - Success (AcqRel): Acquire synchronizes with prior writer's Release, Release publishes our claim to subsequent readers - Failure (Acquire): Observes the blocking value; no write on failure so Release adds nothing Added 5 tests verifying AcqRel/Acquire CAS semantics: - test_cas_acqrel_synchronizes_with_prior_writes - test_cas_failure_observes_blocking_value - test_rebalance_cas_publishes_move_marker - test_best_effort_cleanup_cas_correctness - test_cas_release_sequence_with_move_dest
Step 4 of atomic ordering relaxation plan: Change move_dest.store() from SeqCst to Release in both btree_map.rs and cell.rs. Release ordering is sufficient because move_dest is stored before the CAS that sets marker_state to Move. The CAS uses AcqRel ordering, creating a release sequence that readers synchronize with via Acquire on marker_state. Changes: - btree_map.rs: move_dest.store() in rebalance loop - cell.rs: move_dest.store() in update_marker_state() Tests added: - test_move_dest_release_visible_after_marker_acquire - test_move_dest_release_prevents_reorder_after_cas - test_update_marker_state_move_dest_release - test_concurrent_readers_observe_move_dest
Change destination cell version.store() and marker_state.store() from SeqCst to Release ordering in rebalance (lines ~502-509). Release ensures preceding UnsafeCell writes (key/value) are visible to readers who Acquire-load these atomics.
…ookup=53.0, rand_lookup=50.6, mixed=117.8)
Result: {"status":"keep","total_ms":438.13,"seq_insert_ms":112.38,"rand_insert_ms":105.93,"seq_lookup_ms":52.98,"rand_lookup_ms":50.65,"mixed_ops_ms":117.84}
…t only when Move state. Saves atomic load per cell iteration.
Result: {"status":"keep","total_ms":429.6,"seq_insert_ms":110.15,"rand_insert_ms":101.09,"seq_lookup_ms":51.45,"rand_lookup_ms":50.39,"mixed_ops_ms":115.39}
…Error/CellWriteError directly. -7.2% (429.60→398.74). Lookups improved ~19%.
Result: {"status":"keep","total_ms":398.74,"mixed_ops_ms":110.25,"rand_insert_ms":98.88,"rand_lookup_ms":41.2,"seq_insert_ms":105.56,"seq_lookup_ms":42.85}
… eliminates double-clone and OnceCell construction cost. -5.8% (398.74→375.72).
Result: {"status":"keep","total_ms":375.72,"mixed_ops_ms":103.48,"rand_insert_ms":92.86,"rand_lookup_ms":38.63,"seq_insert_ms":101.73,"seq_lookup_ms":38.32}
…loning key+value for non-matching cells. -1.8% (375.72→368.80).
Result: {"status":"keep","total_ms":368.8,"mixed_ops_ms":101.62,"rand_insert_ms":91.01,"rand_lookup_ms":38.13,"seq_insert_ms":98.57,"seq_lookup_ms":39.47}
…. Avoids cloning key+value for every non-empty cell during insert scan. -0.7% (368.80→366.30).
Result: {"status":"keep","total_ms":366.3,"mixed_ops_ms":102.08,"rand_insert_ms":90.21,"rand_lookup_ms":37.81,"seq_insert_ms":97.86,"seq_lookup_ms":38.34}
…oids GCD computation in rebalance hot loop. -4.4% (366.30→350.30). rand_insert -8.4%.
Result: {"status":"keep","total_ms":350.3,"mixed_ops_ms":97.3,"rand_insert_ms":82.62,"rand_lookup_ms":37.31,"seq_insert_ms":96.08,"seq_lookup_ms":37}
…t for u64 keys (350.30→351.92). Keep for correctness/style.
Result: {"status":"keep","total_ms":351.92,"mixed_ops_ms":97.55,"rand_insert_ms":83.46,"rand_lookup_ms":37.9,"seq_insert_ms":94.1,"seq_lookup_ms":38.91}
…d to compare_key/clone_value/cache. -22.2% (351.92→273.61). Lookups -40%!
Result: {"status":"keep","total_ms":273.61,"mixed_ops_ms":80.27,"rand_insert_ms":70.42,"rand_lookup_ms":22.46,"seq_insert_ms":77.89,"seq_lookup_ms":22.58}
…ad per empty cell iteration. -3.6% (273.61→263.88). Inserts -5%.
Result: {"status":"keep","total_ms":263.88,"mixed_ops_ms":75.04,"rand_insert_ms":67.32,"rand_lookup_ms":23.52,"seq_insert_ms":74.31,"seq_lookup_ms":23.7}
…er empty cell. -0.9% (263.88→261.48).
Result: {"status":"keep","total_ms":261.48,"mixed_ops_ms":76.04,"rand_insert_ms":67.29,"rand_lookup_ms":22.49,"seq_insert_ms":72.85,"seq_lookup_ms":22.25}
…matching cells. Only creates CellGuard for Move handling. -1.8% (261.48→256.75). Lookups -8.6%.
Result: {"status":"keep","total_ms":256.75,"mixed_ops_ms":74.84,"rand_insert_ms":66.1,"rand_lookup_ms":20.57,"seq_insert_ms":75.22,"seq_lookup_ms":20.02}
…s in insert. Avoids heap allocation per insert. -8.2% (256.75→235.65). rand_insert -39.5%!
Result: {"status":"keep","total_ms":235.65,"mixed_ops_ms":76.71,"rand_insert_ms":40,"rand_lookup_ms":23.03,"seq_insert_ms":73.26,"seq_lookup_ms":22.65}
…-lock avoids unnecessary write lock acquisitions. -3.4% (235.65→227.73).
Result: {"status":"keep","total_ms":227.73,"mixed_ops_ms":75.36,"rand_insert_ms":38.99,"rand_lookup_ms":20.71,"seq_insert_ms":72.62,"seq_lookup_ms":20.06}
…find/node_at/left_child/right_child, CellGuard is_empty/marker_state/move_dest, within_density_threshold. -38.4% (227.73→140.25)!
Result: {"status":"keep","total_ms":140.25,"mixed_ops_ms":47.18,"rand_insert_ms":26.66,"rand_lookup_ms":10.32,"seq_insert_ms":44.65,"seq_lookup_ms":11.43}
… improvement -1.2% (140.25→138.61).
Result: {"status":"keep","total_ms":138.61,"mixed_ops_ms":43.59,"rand_insert_ms":25.45,"rand_lookup_ms":10.28,"seq_insert_ms":41.87,"seq_lookup_ms":10.35}
…ovement -0.6% (138.61→137.72).
Result: {"status":"keep","total_ms":137.72,"mixed_ops_ms":43.02,"rand_insert_ms":25.3,"rand_lookup_ms":10.48,"seq_insert_ms":41.11,"seq_lookup_ms":10.58}
…lidation deferred to compare_key/clone_value. -3.1% (137.72→133.47).
Result: {"status":"keep","total_ms":133.47,"mixed_ops_ms":42.9,"rand_insert_ms":24.84,"rand_lookup_ms":9.92,"seq_insert_ms":42.59,"seq_lookup_ms":10.11}
…egligible perf impact (133.47→135.66).
Result: {"status":"keep","total_ms":135.66,"mixed_ops_ms":42.91,"rand_insert_ms":23.97,"rand_lookup_ms":10.72,"seq_insert_ms":42.82,"seq_lookup_ms":10.29}
… cells, only create CellGuard for actual insertion/rebalance. -11.9% (135.66→119.58). Inserts -22%, mixed -14%.
Result: {"status":"keep","total_ms":119.58,"mixed_ops_ms":37.94,"rand_insert_ms":18.2,"rand_lookup_ms":10.05,"seq_insert_ms":35.94,"seq_lookup_ms":10.87}
… check. Marginal improvement within noise (119.58→117.98 on best run).
Result: {"status":"keep","total_ms":117.98,"mixed_ops_ms":39.23,"rand_insert_ms":19.32,"rand_lookup_ms":11.63,"seq_insert_ms":36.76,"seq_lookup_ms":11.05}
…lls skip the Acquire atomic load entirely. -3.6% (117.98→113.73).
Result: {"status":"keep","total_ms":113.73,"mixed_ops_ms":38.01,"rand_insert_ms":18.45,"rand_lookup_ms":10.01,"seq_insert_ms":36.85,"seq_lookup_ms":10.1}
… perf change (113.73→116.85 noise).
Result: {"status":"keep","total_ms":116.85,"mixed_ops_ms":38.95,"rand_insert_ms":18.96,"rand_lookup_ms":10.69,"seq_insert_ms":37.63,"seq_lookup_ms":10.63}
… lock case. Marginal -1.0% (116.85→115.79 avg).
Result: {"status":"keep","total_ms":115.79,"mixed_ops_ms":37.88,"rand_insert_ms":17.75,"rand_lookup_ms":9.82,"seq_insert_ms":36.77,"seq_lookup_ms":10.25}
…ence. No measurable improvement (115.79→115.72). Keep for cleaner access.
Result: {"status":"keep","total_ms":115.72,"mixed_ops_ms":39.75,"rand_insert_ms":18.16,"rand_lookup_ms":9.18,"seq_insert_ms":38.15,"seq_lookup_ms":10.5}
…), iterate in reverse for spreading. Vec avoids ring buffer overhead. -3.5% (116.23→112.22).
Result: {"status":"keep","total_ms":112.22,"mixed_ops_ms":37.67,"rand_insert_ms":17.39,"rand_lookup_ms":9.52,"seq_insert_ms":36.87,"seq_lookup_ms":9.56}
…Avoids reallocation during density scan. -2.0% (112.22→109.96).
Result: {"status":"keep","total_ms":109.96,"mixed_ops_ms":35.65,"rand_insert_ms":17.24,"rand_lookup_ms":9.46,"seq_insert_ms":35.38,"seq_lookup_ms":9.47}
…hin_density_threshold. Eliminates O(8) linear scan per density check. -2.3% (109.96→107.43).
Result: {"status":"keep","total_ms":107.43,"mixed_ops_ms":35.39,"rand_insert_ms":16.75,"rand_lookup_ms":9.22,"seq_insert_ms":34.09,"seq_lookup_ms":9.59}
…ad code path, lets compiler generate tighter match. -1.4% (107.43→106.31 avg).
Result: {"status":"keep","total_ms":106.31,"mixed_ops_ms":34.59,"rand_insert_ms":17.88,"rand_lookup_ms":9.32,"seq_insert_ms":33.73,"seq_lookup_ms":9.92}
…thin noise (106.31→106.47).
Result: {"status":"keep","total_ms":106.47,"mixed_ops_ms":34.66,"rand_insert_ms":17.57,"rand_lookup_ms":10.11,"seq_insert_ms":34.22,"seq_lookup_ms":9.91}
…ersion validation. Lookups -16%. -2.1% total (106.47→104.26 best, ~105 avg).
Result: {"status":"keep","total_ms":104.26,"mixed_ops_ms":35.46,"rand_insert_ms":16.99,"rand_lookup_ms":8.29,"seq_insert_ms":33.79,"seq_lookup_ms":8.41}
…ition for find(). -4.5% (103.15→98.53). rand_insert -8.8%, mixed -5.6%.
Result: {"status":"keep","total_ms":98.53,"mixed_ops_ms":33.15,"rand_insert_ms":15.6,"rand_lookup_ms":8.2,"seq_insert_ms":33.46,"seq_lookup_ms":7.99}
… read paths. Within noise (98.53→98.76).
Result: {"status":"keep","total_ms":98.76,"mixed_ops_ms":33.63,"rand_insert_ms":15.61,"rand_lookup_ms":9.18,"seq_insert_ms":31.3,"seq_lookup_ms":9.04}
…ta dependency). Marginal -1.2% (98.76→96.10 best).
Result: {"status":"keep","total_ms":96.1,"mixed_ops_ms":32.15,"rand_insert_ms":15.38,"rand_lookup_ms":8.08,"seq_insert_ms":31.21,"seq_lookup_ms":8.09}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.