Consul

Notes on consul.io

Nodes use gossip protocol to communicate with each other in a distributed fashion.

Default Ports

  'dns'      => 8600,
  'http'     => 8500,
  'rpc'      => 8400,
  'serf_lan' => 8301,
  'serf_wan' => 8302,
  "server"   => 8300,

RAFT Consensus

Uses Server RPC

Since all servers participate as part of the peer set, they all know the current leader. When an RPC request arrives at a non-leader server, the request is forwarded to the leader. If the RPC is a query type, meaning it is read-only, then the leader generates the result based on the current state of the FSM. If the RPC is a transaction type, meaning it modifies state, then the leader generates a new log entry and applies it using Raft. Once the log entry is committed and applied to the FSM, the transaction is complete.

Because of the nature of Raft's replication, performance is sensitive to network latency. For this reason, each datacenter elects an independent leader, and maintains a disjoint peer set. Data is partitioned by datacenter, so each leader is responsible only for data in their datacenter. When a request is received for a remote datacenter, the request is forwarded to the correct leader. This design allows for lower latency transactions and higher availability without sacrificing consistency.

Gossip

• Serf LAN Gossip Pool - All Agents
• Serf WAN Gossip Pool - All Servers

LAN Gossip Pool

The LAN pool is used for a few purposes. Membership information allows clients to automatically discover servers, reducing the amount of configuration needed. The distributed failure detection allows the work of failure detection to be shared by the entire cluster instead of concentrated on a few servers. Lastly, the gossip pool allows for reliable and fast event broadcasts for events like leader election.

WAN

The WAN pool is globally unique, as all servers should participate in the WAN pool regardless of datacenter. Membership information provided by the WAN pool allows servers to perform cross datacenter requests. The integrated failure detection allows Consul to gracefully handle an entire datacenter losing connectivity, or just a single server in a remote datacenter.

Agents

• agent installed on every machine
• agent can function as server or client
• every agent runs (listening on 127.0.0.1 as the default):
• DNS Server
• Http Server
• Rpc Server
• every agent needs to be spoon fed at least one other agent IP to get started, except servers have extra requirement
• each agent can declare what services the machine it is hosted on offers, this information is queryable by local DNS or HTTP on each other server

Clients

Clients listen on DNS, HTTP, RPC by default on 127.0.0.1

Throw Away Example - Start a client (172.20.20.11), joined to an existing server (172.20.20.10)

consul agent -data-dir /tmp/consul -node=n2 -bind=172.20.20.11 -config-dir=/etc/consul.d -join=172.20.20.10

• data-dir : is where the files are stored
• node : the name of the node, defaults to hostname if not specified (can not have a dot in the name because of dns)
• config-dir : where to look for both the agent config and the service configs
• join : says the name of another node to spoon feed getting started
• bind : what nic ip to bind to (there are other options to bind to a device interface instead of ip)

Servers

• servers require quorum, like zookeeper
• prod servers have either 3 (survives 1 node fail) or 5 (survives 2 node failure)
• strongly consistent like zookeeper CP
• servers participate in leader elections and require some trickery to get up and going

$ consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -node=n1 -bind=172.20.20.10 -config-dir /etc/consul.d

• server : enables the agent to act as a server that participates in leader election etc
• bootstrap-expect : communicates how many servers are expected so that leader election waits for this many servers before electing a leader
• other settings are the same meaning as the client

DNS - dig

$ dig @127.0.0.1 -p 8600 web.service.consul SRV

• where web is a named service

Consul members

$ consul members

views the status, ips and names of all agents

Multiple Datacenters

• natively supports multiple datacenters
• uses optimized gossip protocol across regions

Health Checks

• you can write your own health checks that can be run by the agents to determine node availability

Web UI

• Consul comes with a client side modern js web site that can be hosted on any webserver, but by default is hosted internally.