diff --git a/bench/README.md b/bench/README.md index 3e0cae37..074d1150 100644 --- a/bench/README.md +++ b/bench/README.md @@ -8,26 +8,39 @@ tested on GCP c2-standard-8 (8 vCPU Intel Xeon @ 3.10GHz), Ubuntu 22.04. ### throughput (requests/sec) +#### redis-benchmark + +| test | ember concurrent | ember sharded | redis | dragonfly | +|------|------------------|---------------|-------|-----------| +| SET (64B, P=16) | **1,897,056** | 911,363 | 1,011,232 | 897,714 | +| GET (64B, P=16) | **2,580,102** | 992,633 | 1,192,761 | 928,148 | +| SET (64B, P=1) | **199,600** | **199,600** | 99,800 | 199,600 | +| GET (64B, P=1) | **200,000** | **200,000** | 99,900 | 200,000 | + +#### memtier_benchmark + | test | ember concurrent | ember sharded | redis | dragonfly | |------|------------------|---------------|-------|-----------| -| SET (64B, P=16) | **1,859,464** | 863,823 | 992,380 | 551,514 | -| GET (64B, P=16) | **2,489,950** | 965,532 | 1,163,051 | 640,389 | -| SET (64B, P=1) | **222,123** | 199,840 | 117,647 | 222,222 | -| GET (64B, P=1) | **222,222** | 222,123 | 124,937 | 222,222 | +| SET (64B, P=16) | **1,781,625** | 1,139,899 | 1,121,736 | 1,002,678 | +| GET (64B, P=16) | **2,111,459** | 1,326,564 | 1,335,645 | 1,073,396 | +| mixed 1:10 (64B, P=16) | **2,153,732** | 1,298,726 | 1,305,540 | 1,056,071 | +| mixed 1:1 (64B, P=16) | **1,964,419** | 1,228,596 | 1,238,508 | 1,023,236 | +| SET (64B, P=1) | **255,763** | 159,418 | 161,217 | 304,207 | +| GET (64B, P=1) | **264,149** | 498,078 | 262,139 | 186,270 | ### vs redis | mode | SET | GET | notes | |------|-----|-----|-------| -| ember concurrent | **1.8x** | **2.1x** | best for simple GET/SET workloads | +| ember concurrent | **1.9x** | **2.2x** | best for simple GET/SET workloads | | ember sharded | 0.9x | 0.8x | channel overhead, but supports all data types | ### vs dragonfly | mode | SET | GET | notes | |------|-----|-----|-------| -| ember concurrent | **3.3x** | **3.8x** | dragonfly tested with default config | -| ember sharded | **1.6x** | **1.5x** | both use thread-per-core architecture | +| ember concurrent | **2.0x** | **2.7x** | redis-benchmark, pipelined | +| ember sharded | 1.0x | 1.1x | comparable throughput | **important caveat**: these benchmarks should be taken with a grain of salt. ember is a small indie project built for learning and experimentation. Redis and Dragonfly are production-grade systems developed by large teams over many years, battle-tested at massive scale. @@ -42,13 +55,14 @@ dragonfly in particular offers features ember simply doesn't have: ember's concurrent mode shows higher throughput on simple GET/SET because it's architecturally minimal — essentially a concurrent hashmap with RESP3 parsing. this simplicity is a tradeoff, not an advantage. for anything resembling production use, Redis and Dragonfly are the sensible choices. ember exists primarily as a learning project and for workloads where simplicity matters more than features. -### latency (50 clients, no pipelining) +### latency (48 clients, no pipelining, memtier_benchmark) -| server | p50 | p99 | p100 | -|--------|-----|-----|------| -| ember concurrent | 0.3ms | 0.4ms | 0.6ms | -| ember sharded | 0.3ms | 0.4ms | 0.6ms | -| redis | 0.3ms | 0.4ms | 0.5ms | +| server | p99 SET | p99 GET | +|--------|---------|---------| +| ember concurrent | 0.41ms | 0.38ms | +| ember sharded | 0.56ms | 0.51ms | +| redis | 0.55ms | 0.54ms | +| dragonfly | 0.98ms | 0.98ms | ### memory usage (~1M keys, 64B values) @@ -86,7 +100,7 @@ ember offers two modes with different tradeoffs: **concurrent mode** (`--concurrent`): - uses DashMap for lock-free access -- 1.8-2.1x faster than redis for GET/SET +- 1.9-2.2x faster than redis for GET/SET - only supports string operations - best for simple key-value workloads @@ -94,7 +108,7 @@ ember offers two modes with different tradeoffs: - each CPU core owns a keyspace partition - requests routed via tokio channels - supports all data types (lists, hashes, sets, sorted sets) -- ~0.9x redis throughput with pipelining, but 1.7x faster without pipelining +- ~0.9x redis throughput with pipelining, 2x faster without pipelining ## running benchmarks @@ -113,8 +127,11 @@ cargo build --release -p ember-server --features jemalloc # memory usage test ./bench/bench-memory.sh -# comprehensive comparison (redis + dragonfly) +# comprehensive comparison using redis-benchmark (redis + dragonfly) ./bench/compare-redis.sh + +# comprehensive comparison using memtier_benchmark (redis + dragonfly) +./bench/bench-memtier.sh ``` ### cloud VM benchmarking @@ -129,10 +146,15 @@ gcloud compute instances create ember-bench \ --image-family=ubuntu-2204-lts \ --image-project=ubuntu-os-cloud -# set up and run +# bootstrap (installs rust, redis, memtier_benchmark, dragonfly) +gcloud compute ssh ember-bench --zone=us-central1-a -- 'bash -s' < ./bench/setup-vm.sh + +# run benchmarks gcloud compute ssh ember-bench --zone=us-central1-a -bash -s < ./bench/setup-vm.sh -cd ember && ./bench/bench.sh +cd ember +./bench/compare-redis.sh # redis-benchmark suite +./bench/bench-memtier.sh # memtier_benchmark suite +./bench/bench-memory.sh # memory comparison # cleanup gcloud compute instances delete ember-bench --zone=us-central1-a @@ -145,21 +167,27 @@ gcloud compute instances delete ember-bench --zone=us-central1-a | `bench.sh` | full benchmark: ember (sharded + concurrent) vs redis | | `bench-quick.sh` | quick sanity check (~10 seconds) | | `bench-memory.sh` | memory usage with 1M keys | -| `compare-redis.sh` | comprehensive comparison with dragonfly support | -| `setup-vm.sh` | install dependencies on fresh ubuntu VM | +| `compare-redis.sh` | comprehensive comparison using redis-benchmark | +| `bench-memtier.sh` | comprehensive comparison using memtier_benchmark | +| `setup-vm.sh` | bootstrap dependencies on fresh ubuntu VM | ## configuration ```bash -# customize benchmark parameters +# customize redis-benchmark parameters BENCH_REQUESTS=1000000 BENCH_THREADS=16 ./bench/compare-redis.sh +# customize memtier_benchmark parameters +MEMTIER_THREADS=8 MEMTIER_CLIENTS=16 MEMTIER_REQUESTS=20000 ./bench/bench-memtier.sh + # customize memory test KEY_COUNT=5000000 VALUE_SIZE=128 ./bench/bench-memory.sh ``` ## environment variables +### compare-redis.sh (redis-benchmark) + | variable | default | description | |----------|---------|-------------| | `EMBER_CONCURRENT_PORT` | 6379 | ember concurrent mode port | @@ -171,6 +199,19 @@ KEY_COUNT=5000000 VALUE_SIZE=128 ./bench/bench-memory.sh | `BENCH_PIPELINE` | 16 | pipeline depth | | `BENCH_THREADS` | CPU cores | redis-benchmark threads | +### bench-memtier.sh (memtier_benchmark) + +| variable | default | description | +|----------|---------|-------------| +| `EMBER_CONCURRENT_PORT` | 6379 | ember concurrent mode port | +| `EMBER_SHARDED_PORT` | 6380 | ember sharded mode port | +| `REDIS_PORT` | 6399 | redis port | +| `DRAGONFLY_PORT` | 6389 | dragonfly port | +| `MEMTIER_THREADS` | 4 | memtier threads | +| `MEMTIER_CLIENTS` | 12 | clients per thread (48 total) | +| `MEMTIER_REQUESTS` | 10000 | requests per client (480k total) | +| `MEMTIER_PIPELINE` | 16 | pipeline depth | + ## micro-benchmarks for criterion micro-benchmarks: diff --git a/bench/bench-memtier.sh b/bench/bench-memtier.sh new file mode 100755 index 00000000..df40c34c --- /dev/null +++ b/bench/bench-memtier.sh @@ -0,0 +1,473 @@ +#!/usr/bin/env bash +# +# benchmark comparing ember, redis, and dragonfly using memtier_benchmark. +# +# memtier_benchmark provides better latency distributions and mixed workload +# testing compared to redis-benchmark. use alongside compare-redis.sh for a +# complete picture. +# +# usage: +# bash bench/bench-memtier.sh # full comparison +# bash bench/bench-memtier.sh --ember-only # only benchmark ember +# bash bench/bench-memtier.sh --quick # reduced test matrix +# +# environment variables: +# EMBER_CONCURRENT_PORT ember concurrent port (default: 6379) +# EMBER_SHARDED_PORT ember sharded port (default: 6380) +# REDIS_PORT redis server port (default: 6399) +# DRAGONFLY_PORT dragonfly server port (default: 6389) +# MEMTIER_THREADS memtier threads (default: 4) +# MEMTIER_CLIENTS clients per thread (default: 12) +# MEMTIER_REQUESTS requests per client (default: 10000) +# MEMTIER_PIPELINE pipeline depth for P>1 tests (default: 16) +# EMBER_BIN path to ember-server binary (default: ./target/release/ember-server) +# DRAGONFLY_BIN path to dragonfly binary (default: dragonfly) + +set -euo pipefail + +# --- configuration --- + +EMBER_CONCURRENT_PORT="${EMBER_CONCURRENT_PORT:-6379}" +EMBER_SHARDED_PORT="${EMBER_SHARDED_PORT:-6380}" +REDIS_PORT="${REDIS_PORT:-6399}" +DRAGONFLY_PORT="${DRAGONFLY_PORT:-6389}" + +MEMTIER_THREADS="${MEMTIER_THREADS:-4}" +MEMTIER_CLIENTS="${MEMTIER_CLIENTS:-12}" +MEMTIER_REQUESTS="${MEMTIER_REQUESTS:-10000}" +PIPELINE="${MEMTIER_PIPELINE:-16}" +EMBER_BIN="${EMBER_BIN:-./target/release/ember-server}" +DRAGONFLY_BIN="${DRAGONFLY_BIN:-dragonfly}" + +TOTAL_CONNECTIONS=$((MEMTIER_THREADS * MEMTIER_CLIENTS)) +TOTAL_REQUESTS=$((MEMTIER_THREADS * MEMTIER_CLIENTS * MEMTIER_REQUESTS)) +KEY_MAX=1000000 +RESULTS_DIR="bench/results" +TIMESTAMP=$(date +%Y%m%d-%H%M%S) + +EMBER_ONLY=false +QUICK_MODE=false + +for arg in "$@"; do + case "$arg" in + --ember-only) EMBER_ONLY=true ;; + --quick) QUICK_MODE=true ;; + *) echo "unknown flag: $arg"; exit 1 ;; + esac +done + +# --- helpers --- + +SERVER_PID="" + +cleanup() { + [[ -n "$SERVER_PID" ]] && kill "$SERVER_PID" 2>/dev/null && wait "$SERVER_PID" 2>/dev/null || true +} +trap cleanup EXIT + +start_server() { + local name=$1 + local port=$2 + shift 2 + + echo " starting $name on port $port..." + "$@" > /dev/null 2>&1 & + SERVER_PID=$! + wait_for_server "$port" "$name" +} + +stop_server() { + if [[ -n "$SERVER_PID" ]]; then + kill "$SERVER_PID" 2>/dev/null && wait "$SERVER_PID" 2>/dev/null || true + SERVER_PID="" + sleep 0.5 + fi +} + +wait_for_server() { + local port=$1 + local name=$2 + local retries=50 + while ! redis-cli -p "$port" ping > /dev/null 2>&1; do + retries=$((retries - 1)) + if [[ $retries -le 0 ]]; then + echo "error: $name did not start on port $port" >&2 + exit 1 + fi + sleep 0.1 + done +} + +# run memtier_benchmark and output "ops_sec p99_ms" on a single line. +# +# writes to a temp file to avoid both bash variable null-byte truncation +# and pipe buffering issues observed with certain memtier builds. +# retries once on failure since memtier occasionally produces no output. +run_memtier() { + local port=$1 + local pipeline=$2 + local ratio=$3 + local data_size=$4 + local attempt + + for attempt in 1 2; do + local tmpfile + tmpfile=$(mktemp) + + memtier_benchmark \ + -s 127.0.0.1 -p "$port" \ + --threads="$MEMTIER_THREADS" \ + --clients="$MEMTIER_CLIENTS" \ + --requests="$MEMTIER_REQUESTS" \ + --pipeline="$pipeline" \ + --data-size="$data_size" \ + --ratio="$ratio" \ + --key-minimum=1 \ + --key-maximum="$KEY_MAX" \ + --hide-histogram \ + > "$tmpfile" 2>/dev/null || true + + if grep -q "^Totals" "$tmpfile"; then + local parsed + parsed=$(awk '/^Totals/{printf "%.0f %.3f\n", $2, $7}' "$tmpfile") + local ops=${parsed%% *} + + # retry if memtier reported 0 ops (connection errors, etc.) + if [[ "$ops" -eq 0 ]] && [[ $attempt -eq 1 ]]; then + echo " retrying (0 ops on attempt 1)..." >&2 + rm -f "$tmpfile" + sleep 2 + continue + fi + + echo "$parsed" + rm -f "$tmpfile" + return + fi + + rm -f "$tmpfile" + if [[ $attempt -eq 1 ]]; then + echo " retrying (no output on attempt 1)..." >&2 + sleep 2 + fi + done + + echo "warning: no Totals in memtier output for port $port after 2 attempts" >&2 + echo "0 0.000" +} + +# pre-populate keys so GET tests hit data +populate_keys() { + local port=$1 + memtier_benchmark \ + -s 127.0.0.1 -p "$port" \ + --threads="$MEMTIER_THREADS" \ + --clients="$MEMTIER_CLIENTS" \ + --requests=5000 \ + --ratio=1:0 \ + --data-size=64 \ + --key-minimum=1 \ + --key-maximum="$KEY_MAX" \ + --hide-histogram \ + > /dev/null 2>&1 +} + +format_number() { + printf "%'d" "$1" +} + +calc_ratio() { + local a=$1 + local b=$2 + if [[ "$b" -gt 0 ]]; then + local ratio_x10=$(( (a * 10 + b / 2) / b )) + local ratio_int=$((ratio_x10 / 10)) + local ratio_frac=$((ratio_x10 % 10)) + echo "${ratio_int}.${ratio_frac}x" + else + echo "n/a" + fi +} + +# --- checks --- + +if ! command -v memtier_benchmark &> /dev/null; then + echo "error: memtier_benchmark not found." >&2 + echo " brew install memtier_benchmark # macOS" >&2 + echo " see bench/setup-vm.sh # linux" >&2 + exit 1 +fi + +if ! command -v redis-cli &> /dev/null; then + echo "error: redis-cli not found (needed for server health checks)." >&2 + exit 1 +fi + +if [[ ! -x "$EMBER_BIN" ]]; then + echo "building ember-server in release mode..." + cargo build --release -p ember-server +fi + +HAS_REDIS=false +HAS_DRAGONFLY=false + +if [[ "$EMBER_ONLY" == "false" ]]; then + if command -v redis-server &> /dev/null; then + HAS_REDIS=true + else + echo "note: redis-server not found, skipping redis benchmarks" >&2 + fi + + if command -v "$DRAGONFLY_BIN" &> /dev/null; then + HAS_DRAGONFLY=true + else + echo "note: dragonfly not found, skipping dragonfly benchmarks" >&2 + fi +fi + +mkdir -p "$RESULTS_DIR" + +CPU_CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1) + +echo "" +echo "=== memtier benchmark configuration ===" +echo "threads: $MEMTIER_THREADS" +echo "clients: $MEMTIER_CLIENTS per thread ($TOTAL_CONNECTIONS total)" +echo "requests: $MEMTIER_REQUESTS per client ($TOTAL_REQUESTS total)" +echo "pipeline: $PIPELINE" +echo "key range: 1 - $KEY_MAX" +echo "" + +# --- test matrix --- +# format: "label|data_size|pipeline|ratio" + +if [[ "$QUICK_MODE" == "true" ]]; then + TESTS=( + "SET (64B, P=$PIPELINE)|64|$PIPELINE|1:0" + "GET (64B, P=$PIPELINE)|64|$PIPELINE|0:1" + "mixed 1:10 (64B, P=$PIPELINE)|64|$PIPELINE|1:10" + "mixed 1:1 (64B, P=$PIPELINE)|64|$PIPELINE|1:1" + ) +else + TESTS=( + "SET (64B, P=$PIPELINE)|64|$PIPELINE|1:0" + "GET (64B, P=$PIPELINE)|64|$PIPELINE|0:1" + "mixed 1:10 (64B, P=$PIPELINE)|64|$PIPELINE|1:10" + "mixed 1:1 (64B, P=$PIPELINE)|64|$PIPELINE|1:1" + "SET (1KB, P=$PIPELINE)|1024|$PIPELINE|1:0" + "GET (1KB, P=$PIPELINE)|1024|$PIPELINE|0:1" + "SET (64B, P=1)|64|1|1:0" + "GET (64B, P=1)|64|1|0:1" + ) +fi + +# --- run benchmarks --- +# +# each server is started, benchmarked, and stopped individually. +# running all servers simultaneously causes CPU contention that +# produces unreliable results with memtier's higher thread count. + +echo "" +echo "running benchmarks (one server at a time)..." +echo "" + +# collect labels from the test matrix +declare -a LABELS=() +for test_spec in "${TESTS[@]}"; do + IFS='|' read -r label _ _ _ <<< "$test_spec" + LABELS+=("$label") +done + +# run all tests for a server, collecting ops and p99 into arrays +run_server_tests() { + local port=$1 + local name=$2 + local -n ops_arr=$3 + local -n p99_arr=$4 + + populate_keys "$port" + + echo " benchmarking $name..." + local first=true + for test_spec in "${TESTS[@]}"; do + IFS='|' read -r label data_size pipeline ratio <<< "$test_spec" + + # brief pause between tests to let connections fully drain + if [[ "$first" == "true" ]]; then + first=false + else + sleep 1 + # verify server is still alive between tests + if ! redis-cli -p "$port" ping > /dev/null 2>&1; then + echo " warning: $name not responding, waiting..." >&2 + sleep 3 + fi + fi + + local result ops p99 + result=$(run_memtier "$port" "$pipeline" "$ratio" "$data_size") + ops=${result%% *} + p99=${result##* } + ops_arr+=("${ops:-0}") + p99_arr+=("${p99:-0.000}") + done +} + +declare -a EC_OPS=() +declare -a ES_OPS=() +declare -a R_OPS=() +declare -a D_OPS=() +declare -a EC_P99=() +declare -a ES_P99=() +declare -a R_P99=() +declare -a D_P99=() + +# ember concurrent +start_server "ember concurrent" "$EMBER_CONCURRENT_PORT" \ + "$EMBER_BIN" --port "$EMBER_CONCURRENT_PORT" --concurrent +run_server_tests "$EMBER_CONCURRENT_PORT" "ember concurrent" EC_OPS EC_P99 +stop_server + +# ember sharded +start_server "ember sharded" "$EMBER_SHARDED_PORT" \ + "$EMBER_BIN" --port "$EMBER_SHARDED_PORT" +run_server_tests "$EMBER_SHARDED_PORT" "ember sharded" ES_OPS ES_P99 +stop_server + +# redis +if [[ "$HAS_REDIS" == "true" ]]; then + start_server "redis" "$REDIS_PORT" \ + redis-server --port "$REDIS_PORT" --save "" --appendonly no --loglevel warning + run_server_tests "$REDIS_PORT" "redis" R_OPS R_P99 + stop_server +fi + +# dragonfly +if [[ "$HAS_DRAGONFLY" == "true" ]]; then + start_server "dragonfly" "$DRAGONFLY_PORT" \ + "$DRAGONFLY_BIN" --port "$DRAGONFLY_PORT" --logtostderr --dbfilename "" + run_server_tests "$DRAGONFLY_PORT" "dragonfly" D_OPS D_P99 + stop_server +fi + +# --- output results --- + +DATE=$(date +%Y-%m-%d) + +echo "" +echo "========================================================================" +echo " memtier benchmark results — $DATE" +echo "========================================================================" +echo "" +echo "system: $CPU_CORES cores, $TOTAL_CONNECTIONS connections, $TOTAL_REQUESTS requests/test" +echo "" + +# build format string dynamically based on available servers +fmt="%-30s %16s %16s" +header_args=("test" "ember concurrent" "ember sharded") +divider_args=("----" "----------------" "-------------") + +if [[ "$HAS_REDIS" == "true" ]]; then + fmt="$fmt %12s" + header_args+=("redis") + divider_args+=("-----") +fi +if [[ "$HAS_DRAGONFLY" == "true" ]]; then + fmt="$fmt %12s" + header_args+=("dragonfly") + divider_args+=("---------") +fi + +# throughput table +echo "=== throughput (ops/sec) ===" +echo "" + +printf "$fmt\n" "${header_args[@]}" +printf "$fmt\n" "${divider_args[@]}" + +for i in "${!LABELS[@]}"; do + row_args=("${LABELS[$i]}" "$(format_number "${EC_OPS[$i]}")" "$(format_number "${ES_OPS[$i]}")") + if [[ "$HAS_REDIS" == "true" ]]; then + row_args+=("$(format_number "${R_OPS[$i]}")") + fi + if [[ "$HAS_DRAGONFLY" == "true" ]]; then + row_args+=("$(format_number "${D_OPS[$i]}")") + fi + printf "$fmt\n" "${row_args[@]}" +done + +echo "" +echo "" + +# p99 latency table +echo "=== p99 latency (ms) ===" +echo "" + +printf "$fmt\n" "${header_args[@]}" +printf "$fmt\n" "${divider_args[@]}" + +for i in "${!LABELS[@]}"; do + row_args=("${LABELS[$i]}" "${EC_P99[$i]}" "${ES_P99[$i]}") + if [[ "$HAS_REDIS" == "true" ]]; then + row_args+=("${R_P99[$i]}") + fi + if [[ "$HAS_DRAGONFLY" == "true" ]]; then + row_args+=("${D_P99[$i]}") + fi + printf "$fmt\n" "${row_args[@]}" +done + +echo "" +echo "" + +# ratio comparisons +if [[ "$HAS_REDIS" == "true" ]]; then + echo "=== ember vs redis ===" + echo "" + printf "%-30s %16s %16s\n" "test" "concurrent" "sharded" + printf "%-30s %16s %16s\n" "----" "----------" "-------" + for i in "${!LABELS[@]}"; do + ratio_c=$(calc_ratio "${EC_OPS[$i]}" "${R_OPS[$i]}") + ratio_s=$(calc_ratio "${ES_OPS[$i]}" "${R_OPS[$i]}") + printf "%-30s %16s %16s\n" "${LABELS[$i]}" "$ratio_c" "$ratio_s" + done + echo "" + echo "" +fi + +if [[ "$HAS_DRAGONFLY" == "true" ]]; then + echo "=== ember vs dragonfly ===" + echo "" + printf "%-30s %16s %16s\n" "test" "concurrent" "sharded" + printf "%-30s %16s %16s\n" "----" "----------" "-------" + for i in "${!LABELS[@]}"; do + ratio_c=$(calc_ratio "${EC_OPS[$i]}" "${D_OPS[$i]}") + ratio_s=$(calc_ratio "${ES_OPS[$i]}" "${D_OPS[$i]}") + printf "%-30s %16s %16s\n" "${LABELS[$i]}" "$ratio_c" "$ratio_s" + done + echo "" + echo "" +fi + +# ember modes comparison +echo "=== ember concurrent vs sharded ===" +echo "" +printf "%-30s %10s\n" "test" "ratio" +printf "%-30s %10s\n" "----" "-----" +for i in "${!LABELS[@]}"; do + ratio=$(calc_ratio "${EC_OPS[$i]}" "${ES_OPS[$i]}") + printf "%-30s %10s\n" "${LABELS[$i]}" "$ratio" +done +echo "" + +# --- save raw results --- + +RESULT_FILE="$RESULTS_DIR/${TIMESTAMP}-memtier.csv" +{ + echo "test,ember_concurrent_ops,ember_sharded_ops,redis_ops,dragonfly_ops,ec_p99,es_p99,redis_p99,dragonfly_p99" + for i in "${!LABELS[@]}"; do + echo "${LABELS[$i]},${EC_OPS[$i]},${ES_OPS[$i]},${R_OPS[$i]:-},${D_OPS[$i]:-},${EC_P99[$i]},${ES_P99[$i]},${R_P99[$i]:-},${D_P99[$i]:-}" + done +} > "$RESULT_FILE" + +echo "raw results saved to $RESULT_FILE" diff --git a/bench/setup-vm.sh b/bench/setup-vm.sh index 45eff963..44e5a392 100755 --- a/bench/setup-vm.sh +++ b/bench/setup-vm.sh @@ -2,16 +2,17 @@ # # set up a fresh linux VM for benchmarking # -# usage: ssh user@vm 'bash -s' < ./scripts/setup-vm.sh +# usage: ssh user@vm 'bash -s' < ./bench/setup-vm.sh # -# tested on ubuntu 22.04 +# tested on ubuntu 22.04 (GCP c2-standard-8) set -e -echo "=== installing dependencies ===" +echo "=== installing system dependencies ===" sudo apt-get update -sudo apt-get install -y build-essential git redis-server +sudo apt-get install -y build-essential git redis-server \ + autoconf automake libpcre3-dev libevent-dev pkg-config zlib1g-dev libssl-dev echo "" echo "=== installing rust ===" @@ -24,6 +25,61 @@ else source ~/.cargo/env fi +echo "" +echo "=== installing memtier_benchmark ===" + +if command -v memtier_benchmark &> /dev/null; then + echo "memtier_benchmark already installed" + memtier_benchmark --version +else + echo "building memtier_benchmark from source..." + git clone https://github.com/RedisLabs/memtier_benchmark.git /tmp/memtier_build + cd /tmp/memtier_build + autoreconf -ivf + ./configure + make -j"$(nproc)" + sudo make install + cd ~ + rm -rf /tmp/memtier_build + echo "memtier_benchmark installed" + memtier_benchmark --version +fi + +echo "" +echo "=== installing dragonfly ===" + +if command -v dragonfly &> /dev/null; then + echo "dragonfly already installed" +else + ARCH=$(uname -m) + if [[ "$ARCH" == "x86_64" ]]; then + echo "downloading dragonfly binary..." + DRAGONFLY_URL="https://github.com/dragonflydb/dragonfly/releases/latest/download/dragonfly-x86_64.tar.gz" + if curl -fsSL "$DRAGONFLY_URL" -o /tmp/dragonfly.tar.gz; then + mkdir -p /tmp/dragonfly_extract + tar -xzf /tmp/dragonfly.tar.gz -C /tmp/dragonfly_extract + # find the binary in the extracted archive (named dragonfly-) + DRAGONFLY_BIN=$(find /tmp/dragonfly_extract -name "dragonfly*" -type f ! -name "*.md" | head -1) + if [[ -n "$DRAGONFLY_BIN" ]]; then + sudo mv "$DRAGONFLY_BIN" /usr/local/bin/dragonfly + sudo chmod +x /usr/local/bin/dragonfly + echo "dragonfly installed" + else + echo "warning: could not find dragonfly binary in archive, skipping" + echo " benchmarks will work without dragonfly" + fi + rm -rf /tmp/dragonfly.tar.gz /tmp/dragonfly_extract + else + echo "warning: dragonfly download failed, skipping" + echo " install manually: https://github.com/dragonflydb/dragonfly/releases" + echo " benchmarks will work without dragonfly" + fi + else + echo "note: dragonfly binary not available for $ARCH, skipping" + echo " benchmarks will work without dragonfly" + fi +fi + echo "" echo "=== cloning ember ===" @@ -46,11 +102,15 @@ echo "=== verifying ===" ./target/release/ember-server --help | head -5 redis-server --version redis-benchmark --version +memtier_benchmark --version 2>/dev/null || echo "memtier_benchmark: not found" +dragonfly --version 2>/dev/null || echo "dragonfly: not found (optional)" echo "" echo "=== setup complete ===" echo "" echo "run benchmarks with:" -echo " ./scripts/bench.sh # full suite" -echo " ./scripts/bench-quick.sh # quick sanity check" -echo " ./scripts/bench-memory.sh # memory comparison" +echo " ./bench/bench-quick.sh # quick sanity check" +echo " ./bench/compare-redis.sh # redis-benchmark comparison" +echo " ./bench/bench-memtier.sh # memtier_benchmark comparison" +echo " ./bench/bench-memory.sh # memory usage test" +echo " ./bench/bench.sh # full suite"