feat(cluster): CLUSTER FAILOVER state machine

[kacy]

summary

implements CLUSTER FAILOVER [FORCE|TAKEOVER] — the manual failover command that promotes a replica to primary.

three modes:

• default: waits 500ms for the replication stream to deliver in-flight records, then proposes PromoteReplica + slot transfer via Raft for cluster-wide agreement. safe for planned maintenance failovers.
• force: same as default but skips the 500ms grace period. use when the primary is unreachable and you accept possible data loss of in-flight writes.
• takeover: bypasses Raft entirely. updates local state and propagates the promotion via gossip. use when Raft quorum is lost and you need an emergency promotion.

topology.rs — new ClusterState::promote_replica(replica_id):

• transfers all slots from the old primary to the promoted node in the slot map
• demotes the old primary to NodeRole::Replica (pointing back at the new primary)
• updates both nodes' replica lists and slots cache
• bumps the global config_epoch for conflict resolution

cluster.rs — ClusterCoordinator::cluster_failover(force, takeover):

• validates this node is a replica with a configured primary
• issues ClusterCommand::PromoteReplica + AssignSlots + RemoveSlots via Raft
• applies the state locally immediately so the node starts accepting writes without waiting for Raft reconciliation
• announces the role change via gossip
• adds writes_paused: AtomicBool with pause_writes / resume_writes for future primary-side coordination during default failover

connection.rs:

• replaces the stub with a call to cluster.cluster_failover(force, takeover)
• write rejection now also checks is_writes_paused() so a primary can be paused during coordinated failover

what was tested

• cargo test --workspace: 86 cluster unit tests, 68 server unit tests — all pass
• cargo clippy --workspace -- -D warnings: clean
• new tests: promote_replica_transfers_slots, promote_replica_rejects_non_replica, promote_replica_rejects_unknown_node (topology), failover_rejected_on_primary, failover_takeover_promotes_replica, writes_paused_blocks_and_resumes (cluster coordinator)

design considerations

the 500ms grace period in default mode is a heuristic — a proper implementation would track the exact replication offset per shard and wait for the replica to reach the primary's committed offset. that requires the replication client to expose its current offset back to the coordinator, which is a natural follow-up to pr 4 (automatic failover).

pause_writes and resume_writes on the primary side require a coordination channel (the primary needs to receive a pause request from the replica). the infrastructure is now in place; the protocol message is the remaining piece.