Writable state per sandbox: immutable rootfs + per-sandbox volume (design review + ordered plan) #320

Follow-up to #317, and the write-up behind withdrawing #319. #317 established that children of one snapshot share a writable rootfs; this is the platform-level view, because it turns out not to be a build-runner problem.

What the consumers of forkd here actually need

Two consequences of reading it as one table rather than as "CI":

• Two interactive sandboxes of one image already share a writable rootfs. A developer's npm install or apt can be corrupted by another user's concurrent session, with no CI involved. The failure is silent, and it reads as tooling weirdness rather than as a sandbox problem.
• Serialising writers is not an option for the interactive or per-PR consumers. Telling a user their image is busy because someone else has it open is not a behaviour we can ship. So a single-writer invariant is a floor for the bursty consumers only, not the answer.

What forkd already has, and what is missing

The persistent half exists: VolumeSpec with a caller-chosen guest_path, and suspend/resume already requires a workspace-backed sandbox. What is missing is that a restored child cannot be given its own drive paths — the rootfs path and every volume path are frozen into the binary vmstate, with no drive_overrides on /snapshot/load to override them.

So the model I think forkd wants is: the rootfs is an immutable shared toolchain, and every sandbox's writable state lives on a per-sandbox volume. That is correct for all five consumers above, matches the workspace design already present, and is disk-backed — which matters most for the interactive case, where suspending must not dump a writable layer into memory.bin.

Known gaps to close alongside it:

• Writable volumes share the frozen-path defect. VolumeSpec.read_only defaults to false, so an attached volume is writable by default, and children restored from a tag that carries one all open the same file. Same class as the rootfs, different device.
• A read-only root cannot create a mount point. Any guest-side writable layer depends on the tree existing in the image; that is how /workspace broke feat(vmm,init): per-child writable layer in guest RAM; bakes boot the rootfs read-only #319.

Ordered plan

1. Adopt the model: read-only rootfs + per-sandbox writable volume. Guest-side, the writable trees become overlays whose upper is the volume rather than tmpfs, keeping the image content as the lower layer. Where no volume is attached, tmpfs remains the fallback. (Being built on our side; the guest-init work from feat(vmm,init): per-child writable layer in guest RAM; bakes boot the rootfs read-only #319 is the mechanism, with the upper relocated.)
2. Add drive_overrides to /snapshot/load in Firecracker, in the vendored fork first, with an upstream proposal as done for MemBackendConfig::shared. This is the single mechanism that makes the rootfs and the volumes per-child, covering forked children and workspaces alike.
3. Until 2 lands, hold the invariant in the controller: refuse two live children on the same rootfs path (not the same tag — the path is the thing that must not be shared, so the check stays correct under 1 and 2 and retires itself when paths diverge).
4. Fallback if 2 stalls: run each child in a private mount namespace with the frozen path bind-mounted from a per-child CoW block device (dm-snapshot; LVM thin already provides this on our host). This is a workaround for the same missing capability, needs one bind per drive per child, and should be treated as temporary.

One thing to settle before doing 2

Does the branch/pack path read the drive location back from the child's configuration? A bind mount changes what a path resolves to, not the path string Firecracker holds — so if a snapshot taken of a running child records that string, a branch of a child whose writes live on a CoW device would name the pristine base as its rootfs and silently drop the child's filesystem state. If that is how it works, per-child drives need flattening at branch time, and that is the piece that makes item 2 a project rather than a patch. Worth answering before either of the last two items is attempted.

References

• Restored children share one writable rootfs — concurrent sandboxes corrupt each other #317 — the original report, the mechanism, and the measurement (two children, one inode, 30u).
• feat(vmm,init): per-child writable layer in guest RAM; bakes boot the rootfs read-only #319 — the withdrawn attempt, and the reason: a fixed allowlist, RAM-bounded writes, and a suspend cost that scales with the writable footprint.
• Re-snapshot corruption: ext4 rootfs left dirty by vm.kill() produces EBADMSG and broken binaries #296 — the dirty-ext4 class, which an immutable rootfs also retires.