From 5dc1617ee3340adcac5917aff54350e3bb29e335 Mon Sep 17 00:00:00 2001 From: Kacy Fortner Date: Wed, 18 Feb 2026 19:01:15 -0500 Subject: [PATCH] docs(readme): document clustering, add replication/HA roadmap - add cluster mode to features list - expand clustering section: one-line summary, manual quickstart, full command table (INFO, NODES, SLOTS, KEYSLOT, MYID, MEET, ADDSLOTS, DELSLOTS, FORGET, SETSLOT, COUNTKEYSINSLOT, GETKEYSINSLOT, MIGRATE, RESTORE, ASKING), note on upcoming REPLICATE/FAILOVER - add cluster config flags to configuration table (--cluster-enabled, --cluster-bootstrap, --cluster-port-offset, --cluster-node-timeout) - describe cluster mode in architecture section alongside sharded and concurrent modes - update status table: phase 5 stays in progress (clients not started), add phase 6 (replication/HA) as next, brief description of what remains - update stats: 1,049 tests, ~22k LOC (~41k with tests/comments) --- README.md | 69 +++++++++++++++++++++++++++++--- tests/integration/src/cluster.rs | 9 +---- 2 files changed, 65 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 88aec387..de25050e 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ a low-latency, memory-efficient, distributed cache written in Rust. designed to - **observability** — prometheus metrics (`--metrics-port`), enriched INFO with 6 sections, SLOWLOG command - **sharded engine** — shared-nothing, thread-per-core design with no cross-shard locking - **concurrent mode** — experimental DashMap-backed keyspace for lock-free GET/SET (2x faster than Redis) +- **cluster mode** — distributed keyspace across multiple nodes: 16384 hash slots, SWIM gossip, Raft-based topology, MOVED/ASK redirects, live slot migration - **active expiration** — background sampling cleans up expired keys without client access - **memory limits** — per-shard byte-level accounting with configurable limits - **lru eviction** — approximate LRU via random sampling when memory pressure hits @@ -122,7 +123,11 @@ ember-cli -p 6380 --tls --tls-insecure PING ## clustering -spin up a 3-node local cluster with one command: +ember distributes keys across 16384 hash slots using CRC16. each node owns a non-overlapping range of slots. gossip (SWIM) propagates membership and health; Raft commits topology changes; MOVED and ASK redirects keep clients pointing at the right node during normal operation and live slot migration. + +### quickstart + +the fastest way to get a 3-node cluster running locally: ```bash make cluster @@ -146,6 +151,27 @@ cluster_known_nodes:3 cluster_size:3 ``` +to start nodes manually: + +```bash +# node A — bootstrap a new cluster and own all slots initially +./target/release/ember-server --port 6379 --cluster-enabled --cluster-bootstrap \ + --data-dir ./data/node-a + +# node B and C — join with --cluster-enabled, then CLUSTER MEET into A +./target/release/ember-server --port 6380 --cluster-enabled --data-dir ./data/node-b +./target/release/ember-server --port 6381 --cluster-enabled --data-dir ./data/node-c + +# join B and C into the cluster +ember-cli -p 6380 cluster meet 127.0.0.1 6379 +ember-cli -p 6381 cluster meet 127.0.0.1 6379 + +# redistribute slots (5461 slots each after rebalancing) +ember-cli -p 6379 cluster addslots $(seq 0 5460 | tr '\n' ' ') +ember-cli -p 6380 cluster addslots $(seq 5461 10922 | tr '\n' ' ') +ember-cli -p 6381 cluster addslots $(seq 10923 16383 | tr '\n' ' ') +``` + connect to any node and start writing: ```bash @@ -156,7 +182,7 @@ ember-cli -p 6379 cluster info # cluster_state:ok, 16384 slots assigned ember-cli -p 6380 cluster nodes # 3 nodes, each with correct slot range ``` -clients that support cluster mode (including `redis-cli --cluster`) follow `MOVED` redirects automatically. the built-in `ember-cli` sends each command to the right node based on the key's hash slot. +clients that support cluster mode (including `redis-cli --cluster`) follow `MOVED` redirects automatically. `ember-cli` routes each command to the correct node based on the key's CRC16 hash slot. stop the cluster and clean up: @@ -165,6 +191,28 @@ make cluster-stop # SIGTERM the nodes, keep data make cluster-clean # stop + delete ./data/cluster/ ``` +### cluster commands + +| command | description | +|---------|-------------| +| `CLUSTER INFO` | cluster state, slot counts, node count | +| `CLUSTER NODES` | list all known nodes with their slot ranges | +| `CLUSTER SLOTS` | slot-to-node mapping in array form | +| `CLUSTER KEYSLOT key` | return the hash slot for a key | +| `CLUSTER MYID` | return this node's ID | +| `CLUSTER MEET host port` | join another node into the cluster | +| `CLUSTER ADDSLOTS slot [slot ...]` | assign slots to this node | +| `CLUSTER DELSLOTS slot [slot ...]` | remove slot assignments from this node | +| `CLUSTER FORGET node-id` | remove a node from the cluster view | +| `CLUSTER SETSLOT slot IMPORTING\|MIGRATING\|NODE\|STABLE ...` | manage slot migration state | +| `CLUSTER COUNTKEYSINSLOT slot` | count keys owned by a slot on this node | +| `CLUSTER GETKEYSINSLOT slot count` | list keys in a slot (used during migration) | +| `MIGRATE host port key db timeout [COPY] [REPLACE]` | move a key to another node | +| `RESTORE key ttl payload` | receive a migrated key | +| `ASKING` | tell the server to honor an ASK redirect for the next command | + +replication commands (REPLICATE, FAILOVER) are coming in the next phase — see the status table below. + see [ARCHITECTURE.md](ARCHITECTURE.md) for how clustering works under the hood (raft, gossip, hash slots, migration). ## protobuf storage @@ -256,6 +304,10 @@ distance metrics: **COSINE** (default), **L2** (squared euclidean), **IP** (inne | `--tls-ca-cert-file` | — | path to CA certificate for client verification | | `--tls-auth-clients` | no | require client certificates (`yes` or `no`) | | `--encryption-key-file` | — | path to 32-byte key file for AES-256-GCM encryption at rest (requires `--features encryption`) | +| `--cluster-enabled` | false | enable cluster mode with gossip-based discovery and slot routing | +| `--cluster-bootstrap` | false | bootstrap a new cluster as a single node owning all 16384 slots | +| `--cluster-port-offset` | 10000 | port offset for the cluster gossip bus (data_port + offset) | +| `--cluster-node-timeout` | 5000 | node timeout in milliseconds for failure detection | ## build & development @@ -333,11 +385,13 @@ tested on GCP c2-standard-8 (8 vCPU Intel Xeon @ 3.10GHz). see [bench/README.md] ## architecture -ember offers two execution modes: +ember offers two execution modes for the local keyspace: + +**sharded mode** (default): thread-per-core with channel-based routing. each shard owns a partition of the keyspace with no cross-shard locking on the hot path. supports all data types and enables atomic multi-key operations. -**sharded mode** (default): thread-per-core with channel-based routing. supports all data types (lists, hashes, sets, sorted sets). has channel overhead but enables atomic multi-key operations. +**concurrent mode** (`--concurrent`): lock-free DashMap access. 2-3x faster than Redis for simple GET/SET but limited to string operations. -**concurrent mode** (`--concurrent`): lock-free DashMap access. 2-3x faster than Redis but only supports string operations. +**cluster mode** (`--cluster-enabled`): multiple ember nodes, each running in sharded or concurrent mode, form a single distributed cache. the keyspace is partitioned into 16384 hash slots using CRC16 — each node owns a non-overlapping range. SWIM gossip propagates membership and health across nodes; topology changes (adding nodes, reassigning slots) go through a Raft consensus group so the cluster view is always consistent. clients are routed via MOVED redirects when a key lands on a different node, and ASK redirects during live slot migration. contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). @@ -350,8 +404,11 @@ contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). | 3 | data types (sorted sets, lists, hashes, sets) | ✅ complete | | 4 | clustering (raft, gossip, slots, migration) | ✅ complete | | 5 | developer experience (observability, CLI, clients) | 🚧 in progress | +| 6 | replication and high availability | 🔜 next | + +phase 6 will add leader/replica data streaming, `CLUSTER REPLICATE`, automatic failover via epoch-based elections, and `CLUSTER FAILOVER` for manual promotion. the topology structs already carry replica tracking fields; the commands are parsed — the replication stream is what remains. -**current**: 107 commands, 971+ tests, ~22k lines of code (excluding tests and comments) +**current**: 107 commands, 1,049 tests, ~22k lines of code (~41k including tests and comments) ## security diff --git a/tests/integration/src/cluster.rs b/tests/integration/src/cluster.rs index 59f50125..f6b2abdd 100644 --- a/tests/integration/src/cluster.rs +++ b/tests/integration/src/cluster.rs @@ -398,13 +398,8 @@ async fn cluster_meet_adds_peer() { let mut ca = a.connect().await; // introduce b to a - ca.ok(&[ - "CLUSTER", - "MEET", - "127.0.0.1", - &b.port.to_string(), - ]) - .await; + ca.ok(&["CLUSTER", "MEET", "127.0.0.1", &b.port.to_string()]) + .await; // node a should immediately see b in its local state let resp = ca.cmd(&["CLUSTER", "NODES"]).await;