feat: add memtier_benchmark support and dragonfly benchmarks - #66
Merged
Conversation
add bench-memtier.sh for comprehensive benchmarking using memtier_benchmark, which provides better latency distributions and mixed workload testing (SET:GET ratios) compared to redis-benchmark. test matrix includes SET-only, GET-only, mixed 1:10 (read-heavy cache), mixed 1:1 (session store), large values, and no-pipeline latency tests across ember concurrent, ember sharded, redis, and dragonfly. update setup-vm.sh to install memtier_benchmark (build from source) and dragonfly (binary download) on GCP VMs alongside the existing redis and rust toolchain setup.
memtier runs can occasionally produce empty output when rapidly switching between servers under heavy load. add a single retry if the Totals line is missing, and a brief 0.5s settling pause between runs to reduce resource contention.
restructure the benchmark loop to complete all tests for one server before moving to the next, rather than interleaving tests across servers. this eliminates CPU contention on shared-machine benchmarks where multiple servers compete for cores.
memtier's output can contain bytes that truncate bash strings when captured in a variable. pipe stdout directly to awk for extraction instead of intermediate variable capture.
write memtier output to a temp file instead of piping or capturing in a variable. adds a warning to stderr when no Totals line is found, making failures visible for diagnosis.
pass the detected CPU core count via --shards instead of relying on the default auto-detection. makes the benchmark configuration explicit and visible in the output.
concurrent mode uses DashMap directly and doesn't benefit from the --shards flag. the flag still creates a full sharded engine with idle worker threads that may contend for CPU cores.
redis-benchmark --threads adds coordination overhead that reduces measured throughput on most systems. single-threaded mode gives more accurate and reproducible results. also bumps default requests from 100k to 500k for more stable measurements — 100k finishes too quickly to warm up properly.
the dragonfly release archive contains the binary as dragonfly-x86_64, not dragonfly. updated the find pattern to match the actual filename.
running all servers simultaneously caused CPU contention that produced sporadic 0 ops/sec results. restructured to start, benchmark, and stop each server individually. also added: - retry on 0 ops (memtier occasionally fails on first attempt) - health check between tests to detect server crashes - settling delay between tests for connection draining
tested on GCP c2-standard-8 with dragonfly v1.36.0. ember concurrent achieves 2.0-2.7x throughput vs dragonfly on pipelined GET/SET. dragonfly has notably higher p99 latency (~1ms vs ~0.4ms) on this single-machine setup.
kacy
added a commit
that referenced
this pull request
Feb 11, 2026
* feat: add memtier_benchmark support to benchmarking suite add bench-memtier.sh for comprehensive benchmarking using memtier_benchmark, which provides better latency distributions and mixed workload testing (SET:GET ratios) compared to redis-benchmark. test matrix includes SET-only, GET-only, mixed 1:10 (read-heavy cache), mixed 1:1 (session store), large values, and no-pipeline latency tests across ember concurrent, ember sharded, redis, and dragonfly. update setup-vm.sh to install memtier_benchmark (build from source) and dragonfly (binary download) on GCP VMs alongside the existing redis and rust toolchain setup. * fix: add retry logic and settling delay to memtier benchmark memtier runs can occasionally produce empty output when rapidly switching between servers under heavy load. add a single retry if the Totals line is missing, and a brief 0.5s settling pause between runs to reduce resource contention. * fix: run all tests per server to avoid cross-server contention restructure the benchmark loop to complete all tests for one server before moving to the next, rather than interleaving tests across servers. this eliminates CPU contention on shared-machine benchmarks where multiple servers compete for cores. * fix: pipe memtier output directly through awk memtier's output can contain bytes that truncate bash strings when captured in a variable. pipe stdout directly to awk for extraction instead of intermediate variable capture. * fix: use temp file for memtier output parsing write memtier output to a temp file instead of piping or capturing in a variable. adds a warning to stderr when no Totals line is found, making failures visible for diagnosis. * fix: explicitly pass --shards to ember-server in all bench scripts pass the detected CPU core count via --shards instead of relying on the default auto-detection. makes the benchmark configuration explicit and visible in the output. * fix: remove --shards from concurrent mode bench launches concurrent mode uses DashMap directly and doesn't benefit from the --shards flag. the flag still creates a full sharded engine with idle worker threads that may contend for CPU cores. * fix: remove multi-threaded redis-benchmark, increase default requests redis-benchmark --threads adds coordination overhead that reduces measured throughput on most systems. single-threaded mode gives more accurate and reproducible results. also bumps default requests from 100k to 500k for more stable measurements — 100k finishes too quickly to warm up properly. * revert: restore original benchmark defaults the apparent regression was caused by a degraded GCP VM, not benchmark methodology. a fresh c2-standard-8 instance reproduces the README numbers exactly with the original settings (100k requests, --threads=CPU cores). reverts: d2cd1d9, 65e11c3, 0f01eaa * fix: match dragonfly binary name in setup-vm.sh the dragonfly release archive contains the binary as dragonfly-x86_64, not dragonfly. updated the find pattern to match the actual filename. * fix: run memtier tests one server at a time with retry logic running all servers simultaneously caused CPU contention that produced sporadic 0 ops/sec results. restructured to start, benchmark, and stop each server individually. also added: - retry on 0 ops (memtier occasionally fails on first attempt) - health check between tests to detect server crashes - settling delay between tests for connection draining * docs: add dragonfly benchmark numbers to README tested on GCP c2-standard-8 with dragonfly v1.36.0. ember concurrent achieves 2.0-2.7x throughput vs dragonfly on pipelined GET/SET. dragonfly has notably higher p99 latency (~1ms vs ~0.4ms) on this single-machine setup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
bench/bench-memtier.shfor comprehensive benchmarking with memtier_benchmark alongside the existing redis-benchmark suitesetup-vm.sh(archive containsdragonfly-x86_64, notdragonfly)key design decisions
results (GCP c2-standard-8, 8 vCPU)
ember concurrent vs redis/dragonfly (64B values, pipeline=16):
what was tested