fix(#242): ship rootfs as content-addressed sidecar so hub packs are portable#247
Merged
Conversation
…e portable
Hub snapshot packs couldn't restore on any host but the one that baked
them: Firecracker freezes the rootfs's absolute path into the vmstate
and reopens it verbatim at restore, but from-image bakes keep the rootfs
at /var/cache/forkd/<image>.ext4 — OUTSIDE the snapshot dir — so it was
never packed, and snapshot.json didn't even record which rootfs. First
fork on another host died with FC's cryptic "Block: Error manipulating
the backing file: No such file or directory". This almost certainly
explains the near-zero hub asset usage.
Approach (the "sidecar + dedup" option):
- forkd-vmm: Snapshot records `rootfs` (the absolute path FC froze).
Written by `forkd snapshot`/`from-image`; inherited through compact.
`#[serde(default)]`, so older snapshot.json files stay valid.
- pack: emits the rootfs as a content-addressed `<sha256>.rootfs.zst`
sidecar (zstd -19) NEXT TO the .tar.zst, recorded in the manifest as
`RootfsRef { target_path, sha256, size }`. The 1.5 GiB ext4 never
bloats the memory-delta pack (which stays ~16 MiB of mostly-zero RAM);
the sidecar is sha-named so packs sharing a base rootfs reference —
and pullers cache — one asset. Additive manifest field, no pack-format
version bump.
- pull/unpack: places the rootfs at the recorded absolute path. Skips
when it's already present with a matching sha (dedup across a chain).
Local unpack reads the sidecar beside the pack; pull fetches it from
the pack URL's sibling. Atomic (temp + rename) with sha verification.
- Unsatisfiable sidecar WARNS (doesn't fail) at pull time with an
actionable message, instead of the cryptic block error at first fork.
- push (single presigned PUT) can't place a sibling, so it warns that
the sidecar must be uploaded alongside; the hub-publish flow is
`forkd pack` (emits both files) + a release upload of both.
Scope: bases (from-image / snapshot — the packable things) carry the
rootfs; daemon-side branches set rootfs=None for now (they aren't a hub
path) — branch inheritance is a clean follow-up.
Verified end-to-end in an isolated privileged container (host networking
untouched): bake fresh -> pack (1.5 GiB rootfs -> 29 MiB sidecar, 52.8x,
main pack stays 15.5 MiB) -> wipe rootfs + snapshot to simulate another
host -> unpack (rootfs auto-placed from sidecar) -> fork 2 children, 2/2
alive, 56 ms wall-clock. Existing hub assets still need a re-bake +
re-pack + re-upload (both files) to become portable — an ops follow-up;
this fixes every pack produced from here on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #242.
The bug
Hub snapshot packs couldn't restore on any host but the one that baked them. Firecracker freezes the rootfs's absolute path into the vmstate and reopens it verbatim at restore, but
from-imagebakes keep the rootfs at/var/cache/forkd/<image>.ext4— outside the snapshot dir — so it was never packed, andsnapshot.jsondidn't even record which rootfs. First fork on another host died with FC's crypticBlock: Error manipulating the backing file: No such file or directory. This almost certainly explains the near-zero hub asset usage (everyone who triedforkd pull+ fork on their own host bounced).The fix — sidecar + dedup
The rootfs ships as a content-addressed
<sha256>.rootfs.zstsidecar beside the pack, not inside it — so the memory-delta pack stays ~16 MiB (mostly-zero RAM, 33× compression) while the rootfs (real binaries, ~12–50× compression) travels separately and dedupes: packs sharing a base rootfs reference one sha-named asset, and the puller caches by sha.Snapshotrecordsrootfs(the path FC froze); written by snapshot/from-image, inherited through compact.#[serde(default)]— old snapshot.json stays valid.RootfsRef { target_path, sha256, size }in the manifest. Additive field, no pack-version bump.forkd pack(emits both files) + a release upload of both.Verified end-to-end
Isolated privileged container (host networking untouched), full portability cycle:
Bake → pack (1.5 GiB rootfs → 29 MiB sidecar, 52.8×; main pack stays 15.5 MiB) → wipe rootfs+snapshot (simulate another host) → unpack (rootfs auto-placed) → fork 2/2 alive in 56 ms.
Scope / follow-ups
rootfs=Nonefor now (not a hub path) — branch inheritance is a clean follow-up.Test plan
cargo clippy --workspace --all-targets -D warnings+cargo fmt --checkclean🤖 Generated with Claude Code