DistFS

A work-in-progress personal distributed file system for learning distributed systems by building one I actually use.

Updated at: July 28, 2026

DistFS is a single-user distributed storage system focused on understanding real distributed systems concepts — consensus, consistency models, replication, locking, and fault tolerance — through implementation, not just theory.

This project exists for two reasons:

1. Re-learn distributed systems deeply enough to reason about them without notes.
2. Build a personal cloud storage system I actually trust with my own data.

The product is the forcing function for the learning.

---

Current Status

Work in progress — early architecture + implementation phase

Current focus:

• 3-node Raft cluster
• replicated metadata service
• CLI filesystem operations
• fault tolerance and leader election

Planned later:

• FUSE mount
• chunked file storage
• LAN deployment across real machines
• Tailscale access
• encryption support

---

Goals

DistFS aims to provide:

• Strongly consistent metadata
• Replicated file storage
• Single-user edit-in-place workflows
• Fault tolerance across 3 nodes
• Practical filesystem semantics
• Explicit CAP tradeoffs

This is intentionally:

• not multi-user
• not offline-first
• not a Dropbox clone
• not a research project inventing new algorithms

The goal is to use existing ideas (Raft, replication, linearizable metadata) correctly and understand them deeply.

---

Planned Architecture

                        Client (CLI / FUSE)
                                |
                    1. lookup / propose op
                                v
                  +-----------------------------+
                  |     Raft Metadata Cluster   |
                  |   (3 nodes, leader-elected) |
                  |                             |
                  |  - namespace / file tree    |
                  |  - chunk -> node mapping    |
                  |  - lease / lock coordination|
                  +-------------+---------------+
                                |
                    2. chunk locations returned
                                |
                                v
              +----------------+----------------+
              |                |                |
      +-------v------+ +-------v------+ +-------v------+
      | Chunkserver A| | Chunkserver B| | Chunkserver C|
      |  chunk store | |  chunk store | |  chunk store |
      +-------+------+ +-------+------+ +-------+------+
              |                |                |
              +----- 3. direct read/write -------+
                     (client <-> chunkserver,
                      replicated across N=3)

Architecture in Detail

DistFS splits into two planes, deliberately kept separate — an approach borrowed from GFS and HDFS:

1. Metadata plane — the Raft cluster

Three nodes run a Raft consensus group that owns everything about what exists, but not the file bytes themselves:

• The file/directory namespace (the tree structure a client would ls or stat)
• The mapping from each file to the chunk IDs that make it up, and which chunkservers hold each chunk
• Leases and locks that serialize concurrent operations on the same file

Every write to the namespace (creating a file, renaming it, updating chunk mappings) is proposed to the Raft leader, replicated to a majority of the 3 nodes, and only acknowledged once committed. This is what gives DistFS strong consistency for metadata — if the leader dies mid-write, a new leader is elected from nodes that already have the committed log entry, so no committed operation is lost. Because this plane only handles small, structured records (not raw file bytes), it stays fast even though every write pays a consensus round-trip.

2. Data plane — the chunkservers

Actual file content is split into chunks and stored on separate chunkserver processes (in the current single-machine setup, these can be co-located; the plan is to spread them across real machines on the LAN later). Each chunk is replicated across multiple chunkservers (N=3 to match the metadata cluster's fault tolerance target) so that losing one node doesn't lose data.

Chunkservers are intentionally "dumb" relative to the metadata cluster — they store and serve chunk contents, and report their own health/chunk inventory back to the metadata cluster, but they don't make consistency decisions themselves. Re-replication (if a replica falls behind or a node disappears) is driven by the metadata cluster noticing the gap and instructing a healthy chunkserver to catch up.

3. Request flow

1. Lookup/propose: the client asks the Raft metadata cluster to resolve a path, or proposes a change to the namespace (e.g. "create this file").
2. Resolve: the metadata leader answers with the chunk IDs for the file and which chunkservers currently hold each one — after committing any pending namespace change through Raft.
3. Direct I/O: the client talks straight to the relevant chunkservers to read or write chunk data, bypassing the metadata cluster for the actual bytes. This keeps the consensus layer free of bulk data traffic and lets storage/read throughput scale independently of metadata consistency.
4. Replication & healing: chunkservers replicate writes to their peers for that chunk; the metadata cluster tracks replica freshness and triggers re-replication if a node drops out or falls behind.

This split is the core design bet of the project: consistency-sensitive, low-volume metadata goes through Raft; throughput-sensitive, high-volume data goes through a simpler replicated chunk layer that trusts the metadata cluster for coordination. It mirrors how GFS/HDFS separate their master/NameNode from datanodes, adapted here for a single-user, 3-node, learning-focused deployment rather than a web-scale cluster.

Future additions build on top of this same shape without changing it: a FUSE mount and Tailscale access both just become new ways for a client to reach the same metadata-then-chunkserver path — not new architectural layers.

---

Build

(Build instructions to be added as the CLI and cluster bootstrap process solidify.)