Skip to content

Add Cluster Mode Enabled (CME) testing support - #12

Open
detemmienation wants to merge 5 commits into
valkey-io:unstablefrom
detemmienation:cme
Open

Add Cluster Mode Enabled (CME) testing support#12
detemmienation wants to merge 5 commits into
valkey-io:unstablefrom
detemmienation:cme

Conversation

@detemmienation

Copy link
Copy Markdown

This PR implements CME (Cluster Mode Enabled) testing support, addressing issue #3.

What's added

Three classes in src/valkey_test_case.py:

  • ClusterInfo — exposes typed accessors over a CLUSTER INFO reply (is_cluster_ok, cluster_known_nodes, cluster_slots_assigned).
  • ClusterNodeHandle — a single cluster-mode node; wraps the standard CLUSTER commands (MEET / REPLICATE / ADDSLOTSRANGE) and the waiters for the cluster to converge.
  • ClusterTestCase — orchestrates bootstrap. setup_cluster(num_shards, num_replicas_per_shard) starts the nodes, gossips them together, evenly distributes the 16384 hash slots across primaries, optionally attaches replicas, waits for cluster_state:ok, and returns a topology-awareValkeyCluster client.

Cluster formation is done purely through standard CLUSTER commands, and it reuses the framework's existing port reservation, server lifecycle, and nodes.conf cleanup.

Tests

tests/test_cluster_mode.py:

  • 3-shard cluster: healthy state, all slots assigned, read/write through the cluster client.
  • Slot distribution (2/3/5 shards): CLUSTER SLOTS covers 0..16383 exactly once, no overlap.
  • Replica topologies (3-shard + 1-replica, 2-shard + 2-replica): attachment, online counts, read/write.

Signed-off-by: Tracy <yuningt@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com>

@zackcam zackcam left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great start! One other thing could we update the readme to explain how to use this as well

Comment thread src/valkey_test_case.py Outdated
def start(self, connect_client=True):
super(ClusterNodeHandle, self).start(connect_client=connect_client)
if connect_client:
self._set_node_id()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we call this twice, we can most likely remove this call.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

Comment thread src/valkey_test_case.py Outdated
def knows():
nodesInfo = self.client.cluster("NODES")
for key in nodesInfo:
if re.match(nodeid, nodesInfo[key]["node_id"]):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the nodeid differ here, do we need to do a regex match or can we just do an equals? This works but equals might be cleaner

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Use == instead of re.match

Comment thread src/valkey_test_case.py Outdated
)
return

def start(self, connect_client=True):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the wait_for_ping=True parameter here? We use the start all nodes but might want compatibility incase some people are using the wait_for_ping

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forward wait_for_ping / connect_client args in start_all_node in commit 2e09c8a

Comment thread src/valkey_test_case.py Outdated
timeout=TEST_MAX_WAIT_TIME_SECONDS,
)

def wait_for_cluster_know_node(self, nodeid):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Typo should be known I think

Comment thread src/valkey_test_case.py Outdated

def get_cluster_client(self):
"""Return a cluster-aware client that follows MOVED/ASK redirections."""
from valkey.cluster import ValkeyCluster

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we import this at the top?

Comment thread src/valkey_test_case.py Outdated
self.nodes[i].client.readonly()
except Exception:
logging.warning(
"READONLY failed on replica port {}".format(self.nodes[i].port)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do we need the try catch here for?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isnt really needed, removed in 90c3ae2 !

Signed-off-by: Tracy <yuningt@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com>

@zackcam zackcam left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, can we get a quick update of the README

Signed-off-by: Tracy <yuningt@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants