feat(vmm): give every restored child its own rootfs backing

[jrimmer]

Fixes the defect in #317 — concurrent children of one tag sharing a writable rootfs — and carries the version policy that comes with it.

The mechanism, and why it is this one

Neither released Firecracker has drive_overrides: I checked the v1.16.2 and v1.17.0 binaries, not the source, and neither contains the string. So the load-time override from #5774 is not available in a released build.

What a released Firecracker does have is PATCH /drives/{drive_id} on a restored VM, and it moves the device's storage rather than just its config. Measured: after re-pointing a child's rootfs, a write made inside the guest landed in the new backing and not in the original file.

So each child gets its own reflink clone of the tag's rootfs and is re-pointed at it before it runs:

1. cp --reflink=always <tag>/rootfs.ext4 <child>/rootfs.ext4     # free where the fs supports it
2. PUT   /snapshot/load {…, resume_vm:false}                     -> 204
3. PATCH /drives/rootfs {drive_id, path_on_host:<child backing>} -> 204
4. PATCH /vm {"state":"Resumed"}                                 -> 204

The ordering is load-bearing, and step 2 is why. Loading paused leaves the guest un-executed, so nothing it does at boot — journal replay, /var/log, agent startup — can reach the shared base before the drive is re-pointed. Measured: the base stays byte-identical through boot while the backing changes.

Fail closed. A child whose rootfs cannot be re-pointed is not resumed. It would run against the shared base, which is the defect, so the failure surfaces and the child is dropped (killing its Firecracker) rather than run unsafely.

Two API details worth having: PATCH /drives requires drive_id in the body (without it a 400 reads as a body error, not a capability error), and PUT /drives to add a drive post-restore fails with PCI is not enabled since forkd boots pci=off — irrelevant here, relevant if anyone wants per-child extra devices later.

Evidence

Claim	How it was measured, on a KVM host running v1.17.0
children are isolated	two children wrote the same guest path; each read back its own bytes (child-1: AAA=1 BBB=0, child-2: AAA=0 BBB=1)
the shared base is never written	tag rootfs sha256 identical before and after both children wrote
backings are reclaimed on kill	2 backings → 0 after DELETE
a crash strands nothing	SIGKILL the controller, let systemd restart it: 2 backings → 0
daemon-baked tags are covered	POST /v1/snapshots now writes rootfs into snapshot.json; daemon branches inherit it
the predicate	21 assertions, each confirmed by running it

The crash case is interesting and shaped the sweep. On a restart, systemd kills the unit's cgroup, so the children die with the controller; the reap finds dead processes and correctly prunes rather than kills, so nothing reaches Vm::drop and nothing used to remove the backing — while the watchdog skips the directory because its socket file is still present. A sweep now runs before the reap, when those directories are unowned.

Its gate is process ownership, not file presence: a directory is skipped while any live Firecracker's --api-sock points inside it, and if any live Firecracker cannot be read, the sweep removes nothing. Leaking a backing costs disk; deleting a running VM's backing destroys its disk mid-job, so every ambiguity resolves toward keeping it.

The per-kill reclaim is kept alongside the sweep rather than replaced: it covers a process verified ours and confirmed dead at kill time, which the pre-reap sweep cannot see.

Consequences to plan for

• No Firecracker upgrade is required. An earlier revision required v1.15 on the claim that older builds accept PATCH /drives without moving storage. That was secondhand and wrong: measured on stock v1.5.1, v1.7.0, v1.10.1, v1.12.1, v1.14.1 and v1.17.0, a restored VM's writes move to the new file and /vm/config reports it. Existing snapshots keep loading; nothing needs re-baking. Every re-point is still verified by reading /vm/config back, and a child is not resumed unless it reports its own backing.
• The vendored fork is needed only for --live. It carries MemBackendConfig::shared, and forkd-vmm sends shared in exactly one place — the memfd path live-fork uses. Everything else runs on stock Firecracker.
• Reflink. The FICLONE constant fix landed separately as fix(vmm): use the real FICLONE ioctl number #322; this branch is rebased on it. On a filesystem without cloning a backing is a full copy per child (now made in the children's restore threads, in parallel), so pool size is a storage decision there. Warning, free-space gating and an ext4 policy are proposed as a follow-up.
• Branches and workspace suspends copy the rootfs inside the source's pause. The copy has to match the vmstate's instant. Near-free with reflink; a full copy without it — which matters most for live BRANCH, whose point is a short pause.
• Backings live under the work dir, which is std::env::temp_dir()-derived (now from one helper shared with the sweep). On a host where /tmp is tmpfs that charges a child's rootfs copy to RAM; TMPDIR relocates it.
• Writable volumes are still shared and are the same defect on /dev/vdb: VolumeSpec.read_only defaults to false and the path is frozen, so children of a tag carrying one all open the same file. Out of scope here; worth its own issue.

Not covered by the evidence

Both crash runs exercised the cgroup-kill flavour, because the children were spawned without a memory limit and so died with the controller. The other flavour — Firecracker outliving the controller, e.g. children in their own per-VM cgroup — exercises the kill path, which is verified through a plain DELETE but not yet through a crash.

Snapshots with no recorded rootfs keep the previous behaviour. Daemon-baked snapshots, branches and workspace suspends all record one now.

[jrimmer]

Defect found while checking what a consumer needs: branching a restored child is no longer durable

Branch used to be safe because a branch recorded the snapshot's rootfs, which persists. With per-child backings it records the child's own backing — inside a work dir that Drop and the sweep reclaim — so the clone dies with its source.

Measured on a v1.17 host with this branch deployed:

source child:            sb-6aa4f2a3-…
branch it -> tag:        br-clone created
kill the source child
spawn from br-clone:     400, "firecracker API PUT /snapshot/load returned 400:
                               Failed to restore from snapshot …"

Two things make it worse than a stale path. The clone's snapshot.json records no rootfs at all, so snapshot_restore_problem cannot see the missing file and forkd reports the FC load failure rather than "this tag's rootfs is gone" — an opaque 400. And the window is not just "source child exits": any restart whose sweep collects the backing breaks every tag derived from that child.

This matters beyond a corner case: Branch taken from a running sandbox is an ordinary way to clone or snapshot one, so clones would break intermittently depending on when their source was reaped.

The fix belongs here: on branch, materialise the child's rootfs into the new tag — a reflink clone of the backing, recorded as the new tag's rootfs — instead of leaving the vmstate pointing into an ephemeral work dir. That also restores the property that a tag's rootfs is its own, which pack/sidecar and snapshot_restore_problem both assume.

I would not merge this PR without it. I have not written it yet, and I am marking it here rather than leaving it for whoever reads the diff.

[jrimmer]

The clone path was dead: wrong FICLONE ioctl number

While checking what a consumer needed I measured the per-child cost on a ZFS pool and got a full copy — 12291 MiB of pool growth for one 12 GiB child, when both the source and the destination were on the same pool and that pool demonstrably clones (cp --reflink=always on the same files costs nothing).

chain::reflink_copy passes 0x40209409 for FICLONE. The kernel dispatches on the whole ioctl value including its size field, and that value encodes a 32-byte payload — FICLONERANGE's size — so it is answered with ENOTTY:

kernel:  #define FICLONE  _IOW(0x94, 9, int)      -> 0x40049409
on ZFS, same source file and directory:
  ioctl 0x40209409 -> ENOTTY        ioctl 0x40049409 -> clone

ENOTTY is in the list this function treats as "this filesystem does not support reflink", so it falls through to a full streamed copy — on every filesystem, silently, including the ones where cloning works. That is why the number being wrong was invisible: the failure path is the same as the legitimate one.

Two consequences beyond this PR: a restored child cost a full copy of the rootfs (measured above), and the bake path's "reflink preferred" baseline clone was a full copy too. Corrected in this PR; docs/VENDORED-FIRECRACKER.md-adjacent notes are not affected, but the cost model in the PR body is now accurate.

[WaylandYang]

Thanks. Pausing on a baked-in drive and re-pointing it with PATCH /drives before resume is the right mechanism, and the evidence table is exactly what this needed. Not merging yet. Beyond the branch-durability blocker you already flagged, here is what I found.

Already landed separately: the FICLONE fix you described (0x40209409 → 0x40049409) was never pushed to this branch; the diff doesn't touch chain.rs. It is in dev now as #322, credited to you and verified on an XFS reflink=1 loopback (old number → ENOTTY, new → shared extents). Please rebase onto dev.

Blocking

1. Branch durability (your comment): a branch taken from a restored child has to materialise the child's backing into the new tag and record it as that tag's rootfs.
2. No runtime Firecracker version gate. The CHANGELOG says PATCH /drives is accepted but doesn't move storage on FC < 1.15, so on an older binary children silently share the base again. That is the exact corruption this PR fixes, now with no signal. A documented minimum isn't enough; this needs to fail closed. Options: check firecracker --version when the controller/CLI starts, or GET /vm/config after the PATCH and refuse to resume unless the drive's path_on_host is the backing.
3. Read-only rootfs snapshots pay a full copy per child for nothing. The CLI read-only (squashfs) bake records snap.rootfs = cfg.rootfs, and restore clones whenever self.rootfs.is_some(). Every child of a read-only tag now copies the squashfs and PATCHes a drive that can never be written. Snapshot doesn't persist the drive's read-only flag (your ask #2 on #317), so the fix probably starts there: record it at bake and skip backings for read-only drives.
4. Cost on filesystems without reflink. On ext4 (the default on most hosts, including our test box) each child is a full streamed copy of the rootfs. The copies run sequentially in the loop before any load, so fork -n N costs N × rootfs in both disk and spawn latency, with no warning. At minimum: say once per restore that the fallback happened, gate on free space (the available_bytes helper from #316), and run the copies in the per-child threads. Whether ext4 hosts should get an explicit opt-in/refusal instead of silent full copies is a product call worth making in this PR.

Non-blocking

• sweep_stranded_backings scans std::env::temp_dir() for forkd-daemon-*. It matches where the controller creates work dirs only while both use the same TMPDIR, so please derive both from one helper.
• The upgrade note (re-bake every tag when moving to FC ≥ 1.15) is a release-level decision. We'll schedule it with the release rather than let it land silently in dev.

I can't exercise the KVM path on my side (the test host has no KVM), so your measured evidence plus a rerun after these changes will be what we rely on.

[jrimmer]

Thanks for the thorough review, and for landing the FICLONE fix as #322.

A correction first: my earlier comment said the FICLONE constant was "corrected in this PR". It wasn't — that commit never reached this branch. #322 is the fix; I'll rebase onto dev and drop any overlap.

Plan for this PR:

1. Branch durability — on branch from a restored child, reflink the child's backing into the new tag and record it as the tag's rootfs.
2. Fail closed on Firecracker < 1.15 — after the PATCH /drives, read back GET /vm/config and refuse to resume unless the rootfs path_on_host is the child's backing. That checks the behaviour itself rather than a version string, so it also covers patched or vendored builds.
3. Read-only rootfs — record the drive's read-only flag in the snapshot at bake, and skip the backing for read-only drives.

I'd like to split 4 (cost without reflink) into a follow-up PR, since it carries the product call you raised (silent full copy vs. opt-in vs. refusal on ext4). My proposal there: warn once per restore on fallback, gate on available_bytes, move the copies into the per-child threads, and refuse by default on a non-cloning filesystem unless an explicit opt-in is set. Happy to fold it in here instead if you'd rather review it all at once.

The non-blocking items (one helper for the work-dir root; the upgrade note belonging to the release) I'll take here.

I'll rerun the evidence table on KVM after the changes and post the results.

[jrimmer]

Rebased onto dev (with #322) and pushed. Blockers 1–3 are addressed; 4 is proposed as a follow-up as described above.

A correction that removes blocker 2's premise

The "Firecracker ≥ 1.15, else PATCH /drives is accepted but storage doesn't move" claim was secondhand and never measured. I measured it: restore a snapshot paused, PATCH /drives/<id> to a copy, resume, and watch which host file the guest's writes land in.

stock release (aarch64)	PATCH	guest writes land in	/vm/config reports
v1.5.1	204	new file	new path
v1.7.0	204	new file	new path
v1.10.1	204	new file	new path
v1.12.1	204	new file	new path
v1.14.1	204	new file	new path
v1.17.0	204	new file	new path

So there is no version floor and no re-bake: the upgrade note is gone from CHANGELOG and VENDORED-FIRECRACKER.md. The fail-closed check is still there, in the form you suggested: every re-point is followed by GET /vm/config, and the child is resumed only if Firecracker reports its own backing as the rootfs. Unit-tested against a fake API socket that answers 204 and keeps the old path: the child is never sent Resumed.

What changed

1. Branch durability. A snapshot of a restored child now owns its rootfs. While the source is paused, its drive is pointed at a clone in the new tag's dir, the snapshot is taken (so the vmstate records a durable path), and the drive is pointed home before resume. The clone is recorded as the tag's rootfs. If the drive can't be verified home, the source is killed rather than resumed onto the tag's disk; a drop guard covers every early return. This covers full, diff and live BRANCH, and workspace suspend, which had the same defect. Note that branches previously wrote rootfs: None, so children of a branch tag also shared the source's live backing, not only lost it when the source died.
2. Fail closed on the re-point — above.
3. Read-only rootfs. Rather than a new snapshot.json field, the restore thread reads is_read_only from /vm/config after the paused load — i.e. from the vmstate itself — and skips the backing. That also covers read-only tags baked before this change.
4. (partial) The per-child copies moved into the per-child restore threads, so they run in parallel. Warning, free-space gate and the ext4 policy are for the follow-up.

• Non-blocking: work-dir root and prefix now come from one helper used by both the spawn path and the sweep.

Evidence: forkd-vmm against stock Firecracker v1.17.0 on KVM

Guest /init writes a counter to /dev/vda offset 0 every 100 ms, so a file's first bytes show which host file the guest is writing.

check	result
two concurrent children of one writable tag each write their own backing	TICK…24 → …34 in each backing
the tag's rootfs through that	byte-identical
branch a child with the detour; source after resume	back on its own backing (…36 → …46)
the branch's rootfs after the source resumes	unchanged, holds the snapshot instant (…35)
kill the source	its backing is reclaimed
restore the branch after its source is gone	loads; the child continues from …35 on its own backing; branch rootfs unwritten
control: same branch without the detour, kill source, restore	PUT /snapshot/load returned 400 … Failed to restore devices … No such file — the defect reproduced
read-only tag, two children	both restored, no backing made, nothing to detour

Not covered yet

• This ran on aarch64 KVM through forkd-vmm directly, without netns, so multi-child batches ran as two single-child batches of the same tag. The controller's BRANCH/suspend handlers are covered by build, clippy and unit tests, not yet by a KVM run through the daemon. I'll rerun the full table through the controller on our x86 host (v1.17.0) and post it here before this leaves draft.
• The two forkd-vmm tests that fail on this machine (apply_diff_copies_only_data_regions, assemble_chain_memory_produces_correct_bytes) fail identically on dev: they assume 4 KiB pages and this host has 16 KiB.

[jrimmer]

CI note: bd01919 fixes a flaky test from #314, not part of this change. The .prev sweep tests used made-up "dead" pids 1000/2000, which can be live on a runner, and the sweep rightly leaves a live pid's backup alone. Setting the pid to 1 reproduces the exact failures; they now use pids above PID_MAX_LIMIT. Happy to split it into its own PR if you prefer.