Skip to content

fix(#242): ship rootfs as content-addressed sidecar so hub packs are portable#247

Merged
WaylandYang merged 1 commit into
mainfrom
fix/242-hub-rootfs-sidecar
Jun 14, 2026
Merged

fix(#242): ship rootfs as content-addressed sidecar so hub packs are portable#247
WaylandYang merged 1 commit into
mainfrom
fix/242-hub-rootfs-sidecar

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Closes #242.

The bug

Hub snapshot packs couldn't restore on any host but the one that baked them. Firecracker freezes the rootfs's absolute path into the vmstate and reopens it verbatim at restore, but from-image bakes keep the rootfs at /var/cache/forkd/<image>.ext4outside the snapshot dir — so it was never packed, and snapshot.json didn't even record which rootfs. First fork on another host died with FC's cryptic Block: Error manipulating the backing file: No such file or directory. This almost certainly explains the near-zero hub asset usage (everyone who tried forkd pull + fork on their own host bounced).

The fix — sidecar + dedup

The rootfs ships as a content-addressed <sha256>.rootfs.zst sidecar beside the pack, not inside it — so the memory-delta pack stays ~16 MiB (mostly-zero RAM, 33× compression) while the rootfs (real binaries, ~12–50× compression) travels separately and dedupes: packs sharing a base rootfs reference one sha-named asset, and the puller caches by sha.

  • forkd-vmm: Snapshot records rootfs (the path FC froze); written by snapshot/from-image, inherited through compact. #[serde(default)] — old snapshot.json stays valid.
  • pack: emits the sidecar at zstd -19 (worth it — compressed once, downloaded by all), records RootfsRef { target_path, sha256, size } in the manifest. Additive field, no pack-version bump.
  • pull / unpack: places the rootfs at the recorded absolute path, skips when already present with a matching sha. Local unpack reads the sibling file; pull fetches the pack-URL sibling. Atomic (temp + rename) + sha-verified.
  • Graceful degradation: an unsatisfiable sidecar warns at pull time with an actionable message instead of the cryptic block error at first fork.
  • push (single presigned PUT can't place a sibling) warns that the sidecar must be uploaded alongside; the hub-publish flow is forkd pack (emits both files) + a release upload of both.

Verified end-to-end

Isolated privileged container (host networking untouched), full portability cycle:

      "vmstate": "/root/.local/share/forkd/snapshots/quickstart/vmstate",
      "memory": "/root/.local/share/forkd/snapshots/quickstart/memory.bin",
      "volumes": [],
      "rootfs": "/var/cache/forkd/python-3-12-slim.ext4"
    }
    ===== 3. pack → should emit .tar.zst + <sha>.rootfs.zst sidecar =====
    ==> packing snapshot 'quickstart' → /tmp/pk/qs.tar.zst
    ==> compressing rootfs sidecar 0abd0245d267a844.rootfs.zst (zstd -19, 1.50 GiB → once per base)...
        1.50 GiB → 29.1 MiB (52.8× ) in 29.5s
    ✓ wrote 15.5 MiB (512.0 MiB uncompressed; 33.0× compression) in 31.3s
      next: scp/upload, then `forkd unpack /tmp/pk/qs.tar.zst` on the target host
    --- /tmp/pk contents: ---
    total 45688
    drwxr-xr-x 2 root root     4096 Jun 14 10:37 .
    drwxrwxrwt 1 root root     4096 Jun 14 10:36 ..
    -rw-r--r-- 1 root root 30491184 Jun 14 10:37 0abd0245d267a844.rootfs.zst
    -rw-r--r-- 1 root root 16281099 Jun 14 10:37 qs.tar.zst
    
    ===== 4. simulate OTHER HOST: wipe rootfs + snapshot =====
    rootfs gone: ls: cannot access '/var/cache/forkd/*.ext4': No such file or directory
    
    ===== 5. unpack → should place rootfs from sidecar =====
    ==> unpacking /tmp/pk/qs.tar.zst ...
    ✓ unpacked tag 'quickstart' at /root/.local/share/forkd/snapshots/quickstart
      next: forkd fork --tag quickstart -n <N>
    ==> placing rootfs → /var/cache/forkd/python-3-12-slim.ext4 (from /tmp/pk/0abd0245d267a844.rootfs.zst)
    ✓ rootfs ready at /var/cache/forkd/python-3-12-slim.ext4
    --- rootfs restored? ---
    -rw-r--r-- 1 root root 1610612736 Jun 14 10:37 /var/cache/forkd/python-3-12-slim.ext4
    
    ===== 6. fork from the unpacked snapshot (proves portability) =====
    ==> forking 2 children from snapshot 'quickstart' (per-child netns)...
    ✓ all sockets up in 50 ms
    ✓ 2 restores fired in parallel in 6 ms
    ✓ total wall-clock: 56 ms
    ==> letting children settle for 2s...
    ✓ 2 / 2 children alive
    ==> shutting down...
        cleaned work_dir /tmp/forkd-fork-quickstart
    EXIT=0

Bake → pack (1.5 GiB rootfs → 29 MiB sidecar, 52.8×; main pack stays 15.5 MiB) → wipe rootfs+snapshot (simulate another host) → unpack (rootfs auto-placed) → fork 2/2 alive in 56 ms.

Scope / follow-ups

  • Bases (from-image / snapshot — the packable things) carry the rootfs. Daemon-side branches set rootfs=None for now (not a hub path) — branch inheritance is a clean follow-up.
  • Existing hub assets (python-numpy etc.) still need a re-bake + re-pack + re-upload of both files to become portable — an ops task, tracked separately. This PR fixes every pack produced from here on.

Test plan

  • forkd-cli 23 + forkd-vmm 39 unit tests pass (incl. new path-rewrite + round-trip)
  • cargo clippy --workspace --all-targets -D warnings + cargo fmt --check clean
  • Full portability E2E (above)
  • CI green

🤖 Generated with Claude Code

…e portable

Hub snapshot packs couldn't restore on any host but the one that baked
them: Firecracker freezes the rootfs's absolute path into the vmstate
and reopens it verbatim at restore, but from-image bakes keep the rootfs
at /var/cache/forkd/<image>.ext4 — OUTSIDE the snapshot dir — so it was
never packed, and snapshot.json didn't even record which rootfs. First
fork on another host died with FC's cryptic "Block: Error manipulating
the backing file: No such file or directory". This almost certainly
explains the near-zero hub asset usage.

Approach (the "sidecar + dedup" option):

- forkd-vmm: Snapshot records `rootfs` (the absolute path FC froze).
  Written by `forkd snapshot`/`from-image`; inherited through compact.
  `#[serde(default)]`, so older snapshot.json files stay valid.
- pack: emits the rootfs as a content-addressed `<sha256>.rootfs.zst`
  sidecar (zstd -19) NEXT TO the .tar.zst, recorded in the manifest as
  `RootfsRef { target_path, sha256, size }`. The 1.5 GiB ext4 never
  bloats the memory-delta pack (which stays ~16 MiB of mostly-zero RAM);
  the sidecar is sha-named so packs sharing a base rootfs reference —
  and pullers cache — one asset. Additive manifest field, no pack-format
  version bump.
- pull/unpack: places the rootfs at the recorded absolute path. Skips
  when it's already present with a matching sha (dedup across a chain).
  Local unpack reads the sidecar beside the pack; pull fetches it from
  the pack URL's sibling. Atomic (temp + rename) with sha verification.
- Unsatisfiable sidecar WARNS (doesn't fail) at pull time with an
  actionable message, instead of the cryptic block error at first fork.
- push (single presigned PUT) can't place a sibling, so it warns that
  the sidecar must be uploaded alongside; the hub-publish flow is
  `forkd pack` (emits both files) + a release upload of both.

Scope: bases (from-image / snapshot — the packable things) carry the
rootfs; daemon-side branches set rootfs=None for now (they aren't a hub
path) — branch inheritance is a clean follow-up.

Verified end-to-end in an isolated privileged container (host networking
untouched): bake fresh -> pack (1.5 GiB rootfs -> 29 MiB sidecar, 52.8x,
main pack stays 15.5 MiB) -> wipe rootfs + snapshot to simulate another
host -> unpack (rootfs auto-placed from sidecar) -> fork 2 children, 2/2
alive, 56 ms wall-clock. Existing hub assets still need a re-bake +
re-pack + re-upload (both files) to become portable — an ops follow-up;
this fixes every pack produced from here on.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@WaylandYang WaylandYang merged commit 53a8315 into main Jun 14, 2026
2 checks passed
@WaylandYang WaylandYang deleted the fix/242-hub-rootfs-sidecar branch June 14, 2026 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hub packs are not portable across hosts — vmstate hardcodes the rootfs absolute path and packs don't ship the rootfs

1 participant