Skip to content

[Fix]: cm handle DEAD-rejoin and STANDBY lifecycle to prevent false reshard triggers#30

Merged
ChillyAokiji merged 1 commit into
mainfrom
fix/deferred_reshard
Apr 14, 2026
Merged

[Fix]: cm handle DEAD-rejoin and STANDBY lifecycle to prevent false reshard triggers#30
ChillyAokiji merged 1 commit into
mainfrom
fix/deferred_reshard

Conversation

@ChillyAokiji

Copy link
Copy Markdown
Collaborator

Fix CM node lifecycle: DEAD rejoin and STANDBY state handling

When a DS restarts after being marked DEAD (i.e., the deferred reshard window had already expired and shards were redistributed), the CM previously set the node back
to RUNNING. This caused two problems:

  • False HB timeout loop: the rejoining DS had stale (expired) heartbeat records, so the HB scan immediately re-entered DEFERRED_RESHARD on the next scan cycle.
  • Inflated alive count: the empty DS was counted as a valid RUNNING node, skewing the reshard feasibility check (alive >= dataserver_min_num) and potentially
    receiving shards during rebalance despite holding no data.

Fix: introduce a STANDBY state for post-DEAD rejoins. A STANDBY node is tracked but excluded from alive counts and shard rebalance targets. If it goes silent again,
HB timeout transitions it directly to DEAD with no reshard (it holds no shards). A new unit test covers the full RUNNING → DEFERRED_RESHARD → DEAD → STANDBY → DEAD
lifecycle.

@Sebastian-Keith Sebastian-Keith left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ChillyAokiji
ChillyAokiji merged commit 8bb6d03 into main Apr 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants