Milestones
List view
- No due date
## Implement zero controller infinite scalable metadata storage layer with SWIM and DS-RSM 1. SWIM Hardening (prerequisite for everything) - Gossip quality: We need either a bounded queue of recent events or weighted random selection. - Cluster bootstrap: Seed-node / join mechanism needed. - ShardLeader piggybacking: ShardLeader gossip event type (shard ID + leader node + term). This needs to be added to SwimPacket/Member or as a separate piggyback payload. 2. RPC Layer (TCP inter-node communication) SWIM uses UDP for lightweight membership. But Raft needs reliable, ordered messaging between nodes. We need: - A TCP-based RPC transport between cluster nodes (separate from the client TCP listener and the SWIM UDP socket) - Message types: AppendEntries, RequestVote, InstallSnapshot, and shard-management commands like OpenTopic/CloseTopic 3. Raft Consensus Engine (the core of DS-RSM) Each metadata shard runs its own independent Raft group. We need: - Raft state machine per shard: log, term, voted_for, commit_index, applied_index - Leader election: RequestVote RPC - Log replication: AppendEntries RPC - Snapshotting: for log compaction 4. Shard Manager The orchestration layer that ties SWIM membership to Raft groups: - Shard assignment: When the topology ring says key-range [X, Y) maps to nodes {A, B, C}, form a Raft group from those 3 nodes for that shard. - Rebalancing: When SWIM detects a node death → topology ring updates → affected shards need new Raft members. When a node joins → some shards need to migrate. - Shard-to-node mapping: hash(resource_key) → shard_id → Raft group → leader node 5. Metadata State Machine What each Raft shard actually stores and replicates: - Resource assignments (e.g., "Topic-Blue → Node-Y, Epoch=1") - The command log that the Raft group applies - Defined command types: AssignTopic, RevokeTopic, UpdatePartition, etc. 6. Client Routing & Cache Aimed at "pull + cache" model: - Client asks any node → node checks local cache → cache miss → hash key → find shard leader via ring → query shard leader → cache result - NotLeaderForTopic error → invalidate cache → re-query - Epoch-based staleness detection
No due date•2/2 issues closed