Raft

This repo implements the Raft consensus algorithm for managing a replicated log, based on this paper.

Consensus algorithms allow a collection of machines to work as a coherent group that can survive the failures of some of its members. Because of this they play a key role in building reliable large-scale software systems.

The goal of this project is learning, and going through it I learned the following:

• How consensus algorithms work, how they replicate data with consistency between nodes.
• How to write in Go.
• How to link internal services together using the RPC protocol.
• How goroutines work, and how to use them without getting deadlocks or race conditions, using locks like mutexes.
• How to write unit tests that covers different edge cases for the RPCs used in this project.

and many more small details.

Project Structure

Every node consists of a raftServer process running, which is implemented in raft-server/.

When a process is started it starts two servers:

• HTTP server
• gRPC server

both listening on the same port.

The HTTP server listens for requests coming from clients, it has these endpoints:

• POST /data to add a new log entry, requires key and value body parameters.
• GET /data/:key to check the current value of key in the hashmap.
• GET /state to get the current state of the node (used in the visualization web app).

The gRPC server listens for requests for the following RPCs:

• AppendEntries the leader node uses it to replicate data across the cluster, and also to send heartbeats and establish authority.
• RequestVote a candidate uses it to request a vote to become a leader from another node, which can reply with acceptance or rejection. A candidate becomes a leader if it gets a quorum (majority) of votes.

There are two tickers always running in the background:

• Election Ticker if a follower spends an amount of time (150 - 300ms) without getting any heartbeats from a leader, it becomes a candidate and requests votes.
• Heartbeat Ticker as a leader you send heartbeats every 50ms to say that you're alive and prevent elections.

What happens when a client sends a request?

• if the receiver is a follower it redirects it to the leader.
• when the leader receives it, it adds it to the log (uncommitted).
• on the next heartbeat the entry is sent as part of an AppendEntries request to all the followers; a follower appends the same log entry to its log and replies with success.
• if the leader gets a quorum of nodes that have this entry, it becomes committed, and then later it's applied to the database/hashmap/written to disk.
• then when a follower knows it's committed on the leader's node, it commits and applies the entry too.

All this preserves a consistent model of data across all nodes across failure cases.

A supervisor is added to run, modify, and monitor 3 different nodes.

Running

install dependencies

go mod download

run the supervisor

go run ./super

visit http://localhost:8080/ to see the control center, you can stop and start nodes, add entries and play around and see how the system can handle failure.

Final Words

This project is made with a lot of attention, it's based on many hours of studying, gathering ideas, and putting together far pieaces of knowledge, this project is why I love doing what I'm doing.

Thanks for caring, Thanks for giving it a look.