diff --git a/README.md b/README.md index 71d31fc4..516817ed 100644 --- a/README.md +++ b/README.md @@ -2,21 +2,24 @@ ember logo

-# โค๏ธโ€๐Ÿ”ฅ ember +# ember a low-latency, memory-efficient, distributed cache written in Rust. designed to outperform Redis on throughput, latency, and memory efficiency while keeping the codebase small and readable. ## features - **resp3 protocol** โ€” full compatibility with `redis-cli` and existing Redis clients -- **core commands** โ€” GET, SET, DEL, EXISTS, EXPIRE, TTL, TYPE, DBSIZE, INFO with proper semantics -- **list operations** โ€” LPUSH, RPUSH, LPOP, RPOP, LRANGE, LLEN with O(1) push/pop and auto-cleanup -- **sorted sets** โ€” ZADD (with NX/XX/GT/LT/CH flags), ZREM, ZSCORE, ZRANK, ZRANGE with WITHSCORES -- **sharded engine** โ€” shared-nothing, thread-per-core design with no cross-shard locking on the hot path +- **string commands** โ€” GET, SET (with NX/XX/EX/PX), MGET, MSET, INCR, DECR +- **list operations** โ€” LPUSH, RPUSH, LPOP, RPOP, LRANGE, LLEN +- **sorted sets** โ€” ZADD (with NX/XX/GT/LT/CH), ZREM, ZSCORE, ZRANK, ZRANGE, ZCARD +- **key commands** โ€” DEL, EXISTS, EXPIRE, TTL, PEXPIRE, PTTL, PERSIST, TYPE, SCAN +- **server commands** โ€” PING, ECHO, INFO, DBSIZE, FLUSHDB, BGSAVE, BGREWRITEAOF +- **sharded engine** โ€” shared-nothing, thread-per-core design with no cross-shard locking - **active expiration** โ€” background sampling cleans up expired keys without client access -- **memory tracking** โ€” per-shard byte-level accounting with configurable memory limits +- **memory limits** โ€” per-shard byte-level accounting with configurable limits - **lru eviction** โ€” approximate LRU via random sampling when memory pressure hits -- **pipelined connections** โ€” multiple commands per read for high throughput +- **persistence** โ€” append-only file (AOF) and point-in-time snapshots +- **pipelining** โ€” multiple commands per read for high throughput ## quickstart @@ -24,40 +27,73 @@ a low-latency, memory-efficient, distributed cache written in Rust. designed to # build cargo build --release -# run the server (defaults to 127.0.0.1:6379, no memory limit) +# run the server (defaults to 127.0.0.1:6379) ./target/release/ember-server -# or with a memory limit and eviction +# with memory limit and eviction ./target/release/ember-server --max-memory 256M --eviction-policy allkeys-lru +# with persistence +./target/release/ember-server --data-dir ./data --appendonly +``` + +```bash # connect with redis-cli -redis-cli SET hello world # => OK -redis-cli GET hello # => "world" +redis-cli SET hello world # => OK +redis-cli GET hello # => "world" +redis-cli MSET a 1 b 2 c 3 # => OK +redis-cli MGET a b c # => 1) "1" 2) "2" 3) "3" + +# expiration redis-cli SET temp data EX 60 -redis-cli TTL temp # => 59 -redis-cli DBSIZE # => (integer) 2 +redis-cli TTL temp # => 59 +redis-cli PTTL temp # => 59000 +redis-cli PERSIST temp # => (integer) 1 + +# counters +redis-cli SET counter 10 +redis-cli INCR counter # => (integer) 11 +redis-cli DECR counter # => (integer) 10 # lists -redis-cli LPUSH mylist a b c # => (integer) 3 -redis-cli LRANGE mylist 0 -1 # => 1) "c" 2) "b" 3) "a" -redis-cli RPOP mylist # => "a" +redis-cli LPUSH mylist a b c # => (integer) 3 +redis-cli LRANGE mylist 0 -1 # => 1) "c" 2) "b" 3) "a" # sorted sets -redis-cli ZADD board 100 alice 200 bob 150 charlie +redis-cli ZADD board 100 alice 200 bob redis-cli ZRANGE board 0 -1 WITHSCORES -redis-cli ZRANK board alice # => (integer) 0 -redis-cli ZSCORE board bob # => "200" +redis-cli ZCARD board # => (integer) 2 + +# iteration +redis-cli SCAN 0 MATCH "user:*" COUNT 100 +redis-cli DBSIZE # => (integer) 6 +redis-cli FLUSHDB # => OK ``` +## configuration + +| flag | default | description | +|------|---------|-------------| +| `--host` | 127.0.0.1 | address to bind to | +| `--port` | 6379 | port to listen on | +| `--max-memory` | unlimited | memory limit (e.g., 256M, 1G) | +| `--eviction-policy` | noeviction | `noeviction` or `allkeys-lru` | +| `--data-dir` | โ€” | directory for persistence files | +| `--appendonly` | false | enable append-only file logging | +| `--appendfsync` | everysec | fsync policy: `always`, `everysec`, `no` | + ## build & development ```bash -make check # fmt, clippy, tests -make build # debug build -make release # release build -make test # run all tests +make check # fmt, clippy, tests +make build # debug build +make release # release build +make test # run all tests +make docker-build # build docker image ``` +see [CONTRIBUTING.md](CONTRIBUTING.md) for development workflow and code standards. + ## project structure ``` @@ -66,20 +102,29 @@ crates/ ember-core/ core engine (keyspace, types, sharding) ember-protocol/ RESP3 wire protocol ember-persistence/ AOF and snapshot durability - ember-cluster/ raft, gossip, slot management + ember-cluster/ raft, gossip, slot management (wip) ember-cli/ interactive CLI tool ``` ## architecture -ember uses a shared-nothing, thread-per-core design inspired by [Dragonfly](https://github.com/dragonflydb/dragonfly). each core owns a partition of the keyspace with no cross-thread synchronization on the hot path. +ember uses a shared-nothing, thread-per-core design inspired by [Dragonfly](https://github.com/dragonflydb/dragonfly). each cpu core owns a partition of the keyspace with no cross-thread synchronization on the hot path. -| target | redis baseline | ember goal | -|--------|---------------|------------| +| metric | redis baseline | ember target | +|--------|---------------|--------------| | throughput | ~100k ops/sec/core | 500k+ ops/sec/core | | p99 latency | ~1ms | <200ยตs | | memory/key | ~90 bytes overhead | <40 bytes | +## security + +see [SECURITY.md](SECURITY.md) for: +- reporting vulnerabilities +- security considerations for deployment +- recommended configuration + +**note**: ember does not currently support authentication. always run behind a firewall or in a trusted network. + ## license MIT diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..7861789f --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,57 @@ +# security policy + +## supported versions + +| version | supported | +|---------|-----------| +| 0.1.x | yes | + +## reporting a vulnerability + +if you discover a security vulnerability in ember, please report it responsibly: + +1. **do not** open a public issue +2. email the maintainers directly at kacy@kacyfortner.com (or open a private security advisory on github) +3. include: + - a description of the vulnerability + - steps to reproduce + - potential impact + - any suggested fixes (optional) + +we aim to respond within 48 hours and will work with you to understand and address the issue. + +## security considerations + +### network exposure + +ember binds to `127.0.0.1` by default. if you expose it to a network: + +- use a firewall to restrict access to trusted clients +- consider running behind a reverse proxy with TLS termination +- ember does not currently support authentication (planned for future releases) + +### memory limits + +always configure `--max-memory` in production to prevent unbounded memory growth: + +```bash +ember-server --max-memory 1G --eviction-policy allkeys-lru +``` + +### untrusted input + +- command parsing is defensive and rejects malformed input +- buffer sizes are capped to prevent memory exhaustion attacks +- pattern matching (SCAN MATCH) uses an iterative algorithm to avoid regex DOS + +### persistence + +if using AOF or snapshots: + +- ensure the data directory has appropriate permissions +- aof files contain all write commands in binary format +- snapshots contain the full keyspace state + +## security updates + +security fixes are released as patch versions. we recommend staying up to date with the latest release.