Skip to content

feat: system benchmark script with redis comparison - #43

Merged
kacy merged 2 commits into
mainfrom
feat/system-benchmarks
Feb 7, 2026
Merged

feat: system benchmark script with redis comparison#43
kacy merged 2 commits into
mainfrom
feat/system-benchmarks

Conversation

@kacy

@kacy kacy commented Feb 7, 2026

Copy link
Copy Markdown
Owner

summary

  • adds bench/compare-redis.sh which uses redis-benchmark to measure end-to-end throughput (TCP + RESP3 + shard routing)
  • tests SET and GET at 3B, 64B, and 1KB value sizes, both pipelined (P=16) and non-pipelined (P=1)
  • supports --ember-only mode (no redis needed) and --json output for CI
  • saves raw CSV results to bench/results/ with timestamps
  • adds bench-compare and bench-quick Makefile targets

tested

ran bash bench/compare-redis.sh --ember-only on macOS, results:

test                      ember (rps)
SET (3B, P=16)              1,123,595
GET (3B, P=16)              1,010,101
SET (64B, P=16)             1,000,000
GET (64B, P=16)               952,381
SET (1KB, P=16)               602,409
GET (1KB, P=16)               869,565
SET (3B, P=1)                 136,798
GET (3B, P=1)                 137,362

design considerations

  • uses redis-benchmark because it's the standard tool and already speaks RESP — no custom load generator needed
  • starts and stops servers automatically with trap-based cleanup on any exit path
  • pre-populates keys before GET benchmarks so they hit existing data
  • environment variables allow tuning without modifying the script
  • CSV output enables historical tracking and regression detection

kacy added 2 commits February 7, 2026 12:11
adds bench/compare-redis.sh which uses redis-benchmark to measure
end-to-end throughput including TCP, RESP3 parsing, and shard routing.

test matrix:
- SET and GET at 3B, 64B, 1KB value sizes
- pipelined (P=16) for peak throughput
- non-pipelined (P=1) for latency measurement

features:
- --ember-only mode (no redis required)
- --json output for CI integration
- configurable via environment variables
- raw CSV results saved to bench/results/
- side-by-side comparison table with ratio column

includes bench/README.md, Makefile targets (bench-compare, bench-quick),
and .gitignore entries for result files.
@kacy
kacy merged commit b3dc2c8 into main Feb 7, 2026
4 of 5 checks passed
@kacy
kacy deleted the feat/system-benchmarks branch February 7, 2026 20:06
kacy added a commit that referenced this pull request Feb 11, 2026
feat: system benchmark script with redis comparison
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant