sipfs

Demo Video: https://screen.studio/share/v2sr94IM

A bridge between Sia decentralized storage and IPFS.

This gives us access to the entire IPFS ecosystem (public gateways, IPNS, etc) all backed by Sia storage

Sia encrypts a file, splits it into erasure-coded slabs, and distributes each slab's sectors across hosts. Sector placement is mutable: hosts churn, contracts expire, sectors get migrated to keep the slab healthy. IPFS, by contrast, is content addressed — a CID is a hash of bytes, immutable for the lifetime of the content. The two systems have very different shapes.

sipfs reconciles them by publishing a small, immutable IPLD descriptor to IPFS that points at stable slab IDs rather than mutable sector locations. A separate routing layer — the bridge — resolves slab IDs to current host placement at the moment of download. Anyone with the root CID and a Sia Storage account can fetch the descriptor from any IPFS gateway, discover a bridge through the public DHT, fetch slab placement, and stream the decrypted bytes from Sia hosts.

Components

sipfs/
├── sipfs_bridge/   The bridge service: HTTP API + IPFS gateway + libp2p
│                   node. Pins objects to the Sia indexer under its own
│                   app key, publishes the IPLD descriptor to IPFS, and
│                   announces a routing CID per slab on the DHT.
│
├── sipfs/          Rust library + CLI (`sipfsc`). Talks to a local Kubo
│                   daemon for IPFS (block fetch + DHT FindProviders) and
│                   to a discovered bridge for slab placement, then
│                   streams object bytes from Sia hosts via sia_storage.
│
└── sipfs_go/       Earlier Go iteration of the consumer CLI, kept for
                    reference. Not built or tested anymore.

Architecture

Three things move around:

1. Sector data — encrypted shard bytes stored on Sia hosts. Each slab's sectors are erasure-coded so the slab survives some number of host failures. Sector placement is mutable.
2. The IPLD descriptor — a dag-cbor root + shard blocks listing slab IDs, byte offsets, lengths, plus the symmetric data key needed to decrypt. Lives on IPFS, content addressed by its root CID.
3. Slab placement records — JSON describing which sectors live on which hosts right now. Served by bridges over HTTP. Mutable.

The thing that holds the architecture together is the slab ID: Blake2b(min_shards | encryption_key | sector merkle roots). The digest deliberately excludes host keys, offsets, and lengths so the ID stays the same when the indexer migrates a sector to a new host.

Upload

                                  +-------------------+
                                  |   sipfs CLI       |
                                  |  (or any client   |
                                  |   with sia_storage)|
                                  +---------+---------+
                                            |
                          1. SDK upload     |
                          (sectors → hosts) |
                                            v
                                  +-------------------+
                                  |    Sia hosts      |
                                  +---------+---------+
                                            |
                          2. POST /pin      |
              { dataKey, slabs, metadata }  |
                                            v
                                  +-------------------+
                                  |   sipfs bridge    |
                                  |                   |
                                  | * pins slabs to   |
                                  |   indexer (own    |
                                  |   app key)        |
                                  | * publishes IPLD  |
                                  |   descriptor      |
                                  | * announces       |
                                  |   routing CIDs    |
                                  |   on the DHT      |
                                  +---------+---------+
                                            |
                                            v
                                       root CID

The client uploads sector bytes through the Sia SDK, which returns a list of Slab records — (encryption_key, min_shards, sectors, offset, length) — and a per-object data key. The client POSTs the data key, slabs, and any application metadata to the bridge's /pin endpoint. The bridge:

• PinSlabs and PinObject against the Sia indexer under its own app key. The indexer needs both: PinSlabs enrols the slabs in the repair loop that migrates failing sectors; PinObject keeps the object reference alive. The sealed envelope's EncryptedDataKey is random — the envelope exists only to satisfy the indexer's storage contract. The real data key flows through IPFS, embedded in the root block.
• Builds the IPLD descriptor (root + N shard blocks, each holding up to 1024 slab references) and stores it in its blockstore. Bitswap will serve these to anyone who asks.
• Computes one routing CID per slab — CIDv1(0x300519, sha2-256(slab_id)) — and announces itself as the provider on the public DHT via batched ProvideMany.

Download

        +-----------------+
        |   sipfs CLI     |
        +--------+--------+
                 |
   1. Kubo block/get → Bitswap (root + shards)
                 |
                 v
        +-----------------+
        |  IPLD descriptor|   { length, dataKey, slabRefs… }
        +--------+--------+
                 |
   2. Kubo routing/findprovs(routingCID(slab[0]))
                 |
                 v
        +-----------------+
        |  bridge URL     |   parsed from peer's /https multiaddr
        +--------+--------+
                 |
   3. GET /sia/slabs/{id} for every slab
                 |
                 v
        +-----------------+
        |  PinnedSlab     |   { encryptionKey, minShards, sectors, … }
        +--------+--------+
                 |
   4. Reconstruct sia_storage::Object,
      stream decrypted bytes via SDK
                 |
                 v
              file out

The consumer never needs to know which bridge holds which object. The DHT does the discovery; the bridge's libp2p AddrsFactory advertises an HTTPS multiaddr (e.g. /dns4/ipfs-bridge.sia.dev/tcp/443/https) that the CLI parses out of the provider records to get a fetchable URL.

Why this is interesting

• Decoupled addressing. The IPFS root CID is stable forever. Sector placement underneath can churn — host repair, migration, contract renewal — without invalidating any pinned CID. Sia gives up immutability by design (it has to, to do repair); sipfs recovers it through indirection.

• No central directory. Bridges find each other through the public IPFS DHT. A root CID and a working IPFS node are enough to download. There is no bridges.txt, no DNS list, no SDK config pointing at a specific operator. Spin up a new bridge with an AddrsFactory and it appears.

• Trustless retrieval. The data key lives in the descriptor, not on the bridge. Bridges can't read object contents — they only translate slab IDs to placements. Multiple bridges can serve the same object with no shared trust assumption.

• Standard IPFS surface. The descriptor is plain dag-cbor. Kubo, Helia, public gateways, ipfs dag get, ipfs ls — all of them resolve the descriptor without modification. A sipfs-aware client is only needed for the retrieval step (slab IDs → host placement → decrypted bytes).

Repository tour

Path	What it is
sipfs_bridge/	Bridge service: HTTP API, IPFS gateway, libp2p node.
sipfs/	Rust consumer library + CLI (sipfsc).
*/README.md	Component-level docs with usage examples.

Quick demo

Make sure a local Kubo daemon is running (ipfs daemon, RPC on 127.0.0.1:5001 by default).

In another shell, build and run the consumer CLI:

cd sipfs
cargo run --release login
cargo run --release upload ./hello.mp4
# bafy...
cargo run --release download bafy... -o out.mp4

download doesn't take a --bridge flag. The CLI asks Kubo to fetch the descriptor and to walk the DHT for the slab routing CID; the provider record's HTTPS multiaddr is the bridge URL.

Status

This is a working demo, not a production system. Known caveats:

• The bridge's blockstore and routing store are in-memory; restart drops state. Persistent backends are a small swap (badger3 / flatfs) but unwired.
• The Rust client extracts the object data key by parsing the SDK's share_object URL fragment, since sia_storage exposes no public accessor. A Object::data_key() -> [u8; 32] upstream removes that workaround.
• The Rust client also reproduces sia_storage's seal/open path (HKDF + XChaCha20-Poly1305) externally, in sipfs/src/unsafe_obj.rs, so it can build a valid Object from a known data key + slab list. Same fix: expose those primitives upstream.
• The download path assumes one bridge serves all of an object's slabs. Real deployments wanting per-slab redundancy need a multi-provider resolver.