Add OCI image support by henrybear327 · Pull Request #191 · sysprog21/elfuse

henrybear327 · 2026-07-09T21:34:53Z

Please see the prior work: #34

The idea is that we implement basic OCI operations, such as pull, unpack, run, prune, status, policy (also the the counterparts like rmi, to complete the lifecycle so we can handle the resource clean up, etc.)

Instead of relying on pure C, this PR relying on the existing go tooling (since container ecosystems are mainly built with go) - go-containerregistry, crane, and umoci. It's about 2.8k lines of go code (about 5k lines of test go code). I think it's a lot more maintainable compared to the 38k lines of C code (this is without the full OCI lifecycle).

The main idea is that this go code is exposed as a command-line tool, handling just the OCI related operations.

Tracking issue: #31

Commands to try out

build/elfuse-container list
build/elfuse-container pull alpine:3
build/elfuse-container inspect alpine:3
build/elfuse-container list
build/elfuse-container rmi --force e7a1a92a5bfe 
build/elfuse-container list

build/elfuse-container run --entrypoint /usr/local/bin/python3 python:3.12 -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'
build/elfuse-container run alpine:3 /bin/sh -c 'echo hello from elfuse'

Summary by cubic

Adds OCI image support with a new pure‑Go CLI elfuse-container to pull, unpack, inspect, list, run, rmi, and prune images over a spec‑compliant OCI image‑layout. Extracts a shared C launcher and adds case‑sensitive APFS sparsebundle caches with per‑run COW on macOS; CI enforces layout conformance, cross‑tool interop, and end‑to‑end HVF runs.

New Features
- CLI: pull, unpack, inspect, list, run, rmi, prune (supports --platform, --insecure for loopback registries only, digest pins/prefixes, and --json on inspect/list).
- Store uses an OCI image‑layout via go-containerregistry; interop with crane, skopeo, umoci (scripts/oci-interop.sh); reachability GC over index.json and blobs/sha256/.
- macOS runs default to a case‑sensitive APFS sparsebundle per digest with per‑run clonefile COW; --plain-rootfs fallback; run --keep retention protected by bundle markers; non‑Darwin builds support pull/inspect/unpack/GC for CI.
Hardening & Fixes
- Shared C launch entry (core/launch.c/.h) for positional ELF and elfuse-container run; adds --user, --workdir, --env, --clear-env; stages initial IDs so auxv matches --user; forwards INT/TERM/QUIT/HUP.
- Procfs/dev shims: /dev/full (ENOSPC on write), /dev/console (O_PATH safe), and /proc/sys/kernel/{ostype,osrelease,hostname} backed by cached uname; runtime /etc injection and named user/group resolution are rootfs‑bounded.
- Store updates serialized with <store>/.lock; run auto‑pulls only when a ref is truly absent; rmi refuses to drop a live or kept cache without --force; rmi/prune do reachability GC; prune --cache [--all] [--dry-run] safely detaches/reaps caches using bundle locks.
- Unpack finalizes exact modes regardless of umask and cleans partial trees on failure; CI builds elfuse-container, runs Linux OCI layout/interop, adds macOS sparsebundle tests + run‑less lifecycle, and includes an HVF run + Python workload; docs add docs/oci-design.md and expanded usage/testing.

^{Written for commit ebee672. Summary will update on new commits.}

jserv

Rebase latest main branch, resolve conflicts, and refine per review messages.

jserv · 2026-07-09T23:09:02Z

Commands to try out

build/elfuse oci list
build/elfuse oci pull alpine:3
build/elfuse oci inspect alpine:3
build/elfuse oci list
build/elfuse oci rmi --force e7a1a92a5bfe 
build/elfuse oci list
build/elfuse-container run --entrypoint /usr/local/bin/python3 \\n    python:3.12 -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'
build/elfuse oci run alpine:3 /bin/sh -c 'echo hello from elfuse'

Leveraging existing Go-based tools and packages is a great step toward full OCI image support. I agree with this change.

I suggest repositioning the build/elfuse binary as an efficient Linux syscall-to-macOS/Darwin runtime, while implementing build/elfuse-container in Go using OCI-related packages.

In other words, build/elfuse oci should not be considered a valid command. OCI-specific functionality should reside in elfuse-container, not in the elfuse executable.

Runtime file injection de-symlinked only the /etc directory itself; the per-file os.WriteFile still followed an image-shipped symlink at etc/{hostname,hosts,resolv.conf}, letting a malicious image redirect the write outside the rootfs. Named --user resolution had the same flaw on the read side: a symlinked etc/passwd or etc/group made lookupPasswd/lookupGroup read host account files. Route both through os.OpenRoot (the same containment the layer unpacker already uses): injection unlinks the existing entry and recreates it O_EXCL, and passwd/group opens are rootfs-bounded. Regression tests pin both behaviors. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Mode finalization ran only when an entry carried setuid/setgid/sticky bits, but the creation modes passed to os.Root.OpenFile and MkdirAll are masked by the process umask: under e.g. umask 0077 a layer's 0755/0644 entries unpacked as 0700/0600 and were never corrected. Chmod every created file and directory entry to its exact tar mode (applyMode), and split the ensure-parent path out (ensureParent) so finalizing an entry cannot reset the mode of an already-unpacked parent directory to the 0755 default. Regression test unpacks under umask 0077 and checks exact modes, including a 0700 parent that a later child entry must not widen. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Both run paths infer "already unpacked" from the rootfs path existing (csrun.go and the plain-directory path in commands.go), but unpackImage created the destination before applying layers and left it behind on failure. A run after a failed unpack therefore skipped the unpack and executed against the truncated tree. Delete the destination on unpack failure. The cleanup applies only when unpackImage created the directory itself, so an explicit pre-existing `unpack --rootfs DIR` target is never removed. Regression test drives a two-layer image whose second layer fails and checks both the cleanup and the pre-existing-directory guard. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pin/rmi updated refs.json with an unlocked load-modify-save cycle and a fixed temp filename, so two concurrent pulls (or a pull racing an rmi) could clobber each other's temp file and drop a just-recorded pin -- a later unpack/run then reports the image as not pulled even though its pull succeeded. index.json has the same read-modify-write shape inside the layout package. Add an exclusive flock on <store>/.lock held across pin's cycle, addImage's check-append-pin, and rmi's whole resolve-modify-GC sequence, and give savePins a unique temp name. A 16-writer concurrent-pin regression test asserts no entry is lost. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

cmdRun treated every s.image failure as "not pulled" and fell into the auto-pull path, so a corrupt refs.json or unreadable layout triggered a surprise network pull instead of reporting the store problem. Introduce an errNotPulled sentinel wrapped by digestFor and resolvePinnedTarget (user-facing message text is unchanged) and gate the auto-pull on errors.Is. A regression test pins the two error kinds apart. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two GC robustness fixes: DirEntry.Info() returning ENOENT (a blob reclaimed by a concurrent rmi/prune between ReadDir and Info) aborted the whole GC pass; skip the vanished entry instead, matching the IsNotExist tolerance already used elsewhere in gc.go. A last-pin rmi committed the pin removal to refs.json before removing the manifest descriptor from index.json. If descriptor removal then failed, the image was stranded: no ref resolves to it, the descriptor keeps every blob live, and prune never removes descriptors. Reorder the writes so the same failure window leaves a stale pin over a removed descriptor, which a retried rmi resolves and completes (RemoveDescriptors is a filter; re-removal is a no-op). Regression test forces the descriptor write to fail and checks the pin survives and the retry finishes. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pruneCSBundle ran the crash-recovery sweep -- including an unconditional hdiutil detach -force of any attached volume -- before checking whether the digest was still pinned, so a non---all `prune --cache` could rip the rootfs out from under an active run (the sweep cannot tell a crashed leftover mount from a live one by mount state alone). Two guards fix this. The live[key] pin check now runs before the sweep, so a plain prune never touches a pinned bundle; a crashed pinned bundle is recovered by the next run's provision or by --all. And sweepCSBundle now reports a volume busy instead of detaching when a run-<pid> clone of a live process remains inside it, protecting the --all and legacy/unpinned paths too. The gated darwin round-trip now covers the busy path, and folds in two review fixes of its own: the orphan clone uses a never-assignable pid (999999999 > kern.maxpid) instead of a reaped pid that the OS could reuse mid-test, and a t.Cleanup force-detach keeps failed runs from leaking an attached volume. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

clearDir followed a symlink at the mount-point path, so pre-attach cleanup of a corrupt or tampered store could empty a directory outside the OCI cache. Reject a symlinked mount dir with an error instead. Also drop csMount.imagePath: it was set but never read in production (every consumer derives <bundle>/rootfs.sparsebundle itself), so the field was state to maintain with no behavior behind it. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

runMainSubprocess read the stdout pipe to EOF before touching the stderr pipe -- the sequential-read pattern the os/exec docs warn can deadlock once the unread stream fills its ~64KB buffer. Hand both streams to exec.Cmd as bytes.Buffers instead, which the package drains concurrently; less code and no deadlock potential as outputs grow. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The jq expression comparing registry truth against the store pin returned every manifest matching os/arch, so a manifest list with two matching entries (several variants, or a future extra descriptor) produced a multi-line string and failed the equality check on a valid image. Wrap the selection in first(...) and exclude BuildKit attestation manifests, mirroring what crane.Pull(WithPlatform) resolves. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The prune summary presented one number as a uniform on-disk-allocation figure, but blob bytes come from logical file sizes while cache-dir bytes come from st_blocks allocation. Say so in the pruneReport doc and mark the user-facing total approximate rather than pretending to a single metric. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

v1.18.7 fixes an out-of-bounds read in s2.NewDict. The dependency is indirect (via go-containerregistry) and nothing here imports the s2 package, so there is no reachable exposure -- this is dependency hygiene while the report is fresh. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Every pre-launch failure path unlinks a FUSE-materialized temporary ELF (elf_host_temp) before returning; the --env/--clear-env OOM branch was the lone omission and leaked the temp file on disk. Mirror the sibling paths. Reported by cubic review on PR sysprog21#191. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

henrybear327 · 2026-07-11T12:22:40Z

Putting this PR as draft for now. Experimenting with Claude code and the setup went wrong, the PoC code polluted the codebase.

jserv

Refine per review messages from @cubic-dev-ai and squash commits for further review.

henrybear327 · 2026-07-11T16:12:53Z

Refine per review messages from @cubic-dev-ai and squash commits for further review.

I am in the process of resetting the changes.

The original 8 commits would be amended for changes as those were the logically reviewable units.

jserv · 2026-07-11T16:26:44Z

The original 8 commits would be amended for changes as those were the logically reviewable units.

Avoid using Semantic Commit Messages, Instead, stick to How to Write a Git Commit Message,

henrybear327 · 2026-07-13T13:30:45Z

Improved test coverage (go test cases using table tests format for better readability)
Improved CI coverage to test the lifecycle of the OCI code
Improved the documentation to document the design, implementation, and limitation
Improved the commit message

henrybear327 · 2026-07-13T14:23:43Z

I am doing a sweep of the existing AI review comments.

henrybear327 · 2026-07-13T21:09:34Z

PR description also updated. Ready for another pass @jserv @Max042004

Thanks.

jserv

Place copyright notice for each Go source file.

jserv

The dominant theme in the container-run path: run touches per-digest mount and clone state with no lock, while prune/rmi/other runs mutate the same state. A per-digest flock held for the lifetime of a run would close several of the inline findings at once.

Additional finding (in src/syscall/io.c, not part of this diff so it cannot be anchored inline): the new /dev/full node's ENOSPC-on-write is bypassed by the copy-path syscalls. copy_fd_range writes the output through the raw host fd and never routes through proc_intercept_write, so sendfile/copy_file_range/splice targeting a /dev/full guest fd write into the backing /dev/zero and report success where Linux returns -ENOSPC. Fix: thread the output guest fd into copy_fd_range and run write chunks tagged with a proc_path through proc_intercept_write, mirroring the read-side intercept.

Two items were considered and rejected: an apparent AT_EMPTY_PATH removal in fs.c (a diff-direction artifact -- the branch is behind main, which added that support; main...HEAD touches fs.c zero times), and a claimed uname-cache race (cached_uname is static const, resolved at compile time).

elfuse_launch owns guest bring-up: guest_bootstrap_prepare, the FUSE-temp unlink, the sysroot casefold probe, proc_set_ids, vCPU creation, GDB init/sync/wait, the run loop, gdb_stub_shutdown, the shim counter and syscall histogram dumps, and guest_destroy. main() retains the original CLI argv (proctitle rewriting), option parsing, sysroot provisioning, the shebang loop, the --gdb x86_64 guard, host cwd, and the heap resource cleanup. The new --user, --workdir, --env, and --clear-env flags map onto launch_args_t fields so a container front end can set the guest identity, working directory, and environment without patching the runtime; `elfuse-container run` drives exactly this interface. The --user identity is staged before bring-up (proc_set_initial_ids) so the auxv AT_UID/AT_GID snapshot taken by build_linux_stack matches what getuid()/getgid() later report; --workdir rejects non-absolute paths up front instead of silently resolving them against the host cwd; and a Rosetta guest's FUSE-materialized temp ELF, kept alive for the translator during the run, is unlinked at teardown and on the post-prepare error paths instead of leaking.

Container images probe paths that macOS does not provide or gates behind root. Synthesize the ones common entrypoints depend on: /dev/full (reads a NUL stream, writes fail with ENOSPC), /dev/console (reserved for the kernel on macOS, so non-root processes cannot open the real one), and /proc/sys/kernel/{ostype,osrelease,hostname}, which mirror a cached utsname so procfs and uname(2) agree on the platform string.

elfuse-container is a standalone Go binary that owns the OCI image pipeline: pull, unpack, inspect, and run over an OCI image-layout store, with run resolving Entrypoint/Cmd/Env/User/WorkingDir and exec'ing the existing elfuse launch path. elfuse itself stays a pure Linux syscall-to-Darwin runtime with no OCI commands; elfuse-container locates it as a sibling binary ($ELFUSE_BIN overrides for tests). The pipeline is hardened at the trust boundaries: refs.json and index.json updates are serialized by an exclusive store flock, run auto-pulls only when a ref is genuinely absent (store corruption surfaces instead of triggering a network pull), runtime /etc injection and symbolic --user resolution are rootfs-bounded via os.OpenRoot so image-controlled symlinks cannot escape, unpacked modes are finalized with an explicit chmod so a restrictive host umask cannot corrupt layer permissions, and a failed unpack removes the partial rootfs it created rather than leaving it to be executed later. Tests cover the pin table (flock serialization, corrupt refs.json), run-spec resolution, layer application (ordering, whiteouts, opaque directories), /etc injection fallbacks, exec argv shape, inspect output, and CLI dispatch. Production code gains small test seams -- function variables for crane.Pull, the exec entry, and hostname/resolv lookups -- so tests can drive failure paths in-process, plus a nil-object guard when parsing refs.json. Layer application enforces whiteout scoping: opaque markers spare the layer's own additions regardless of tar entry order, a malformed bare .wh. entry fails extraction instead of deleting its directory, a directory entry replaces a lower layer's non-directory rather than resolving through a planted symlink, and entry bodies are bounded to the header size. inspect propagates output-write failures, /etc must be a directory (or absent/symlink) before runtime files are injected, --platform rejects empty components, and an explicit run --platform is validated against the pinned image instead of silently launching another architecture.

A plain directory rootfs on the default case-insensitive APFS volume folds Linux filenames that differ only by case. Default runs now use a case-sensitive APFS sparsebundle per pinned manifest digest, with a per-run clonefile COW rootfs so guest writes never mutate the warm base tree and repeated runs skip the unpack. The plain-rootfs path remains available with --plain-rootfs, and the non-Darwin stub keeps elfuse-container buildable for pull/inspect/unpack tests on Linux. Darwin tests drive runCaseSensitive through seams for provisioning, clonefile, spawn, cleanup, and exit, and cover sparsebundle provisioning, mount/detach, and attach-failure teardown with a fake hdiutil; the mount-probe and force-detach hooks are function variables so tests need no real disk images. The spawn path intercepts SIGHUP alongside INT/TERM/QUIT and installs the handler before the child starts, so a hangup or an early signal still flows through the forward/reap/teardown path. elfuse is invoked with a "--" separator before the guest command, so an image Entrypoint beginning with "-" cannot steer the host launcher. hdiutil attach failures keep stderr in the error message (stdout stays clean for the plist parse), and the parsed mount path is XML-entity-decoded so a store path containing "&" or quotes still round-trips.

Add list/images, rmi, and prune on top of the OCI image-layout store. rmi and prune use reachability GC: shared manifests, configs, and layers stay on disk while any remaining ref reaches them. Cache cleanup handles plain rootfs caches and macOS sparsebundle caches through platform-specific seams, with two safety rules for active workloads: a prune without --all never touches a still-pinned digest's bundle, and a volume still hosting a live run's clone is never force-detached. Tests cover reachability GC (shared blobs, stale temp blobs, digest prefixes), prune cache safety rules, the darwin cache sweep through the mount-probe/reap/detach seams, and end-to-end command wrappers over pull/run/list/rmi/prune, with reapOrphanClonesFn and isMountPointFn as swappable probes so the sweep is testable without real disk images. prune's GC sweep and list's snapshot run under the store lock, closing the windows where a concurrent pull's fresh blobs could be reclaimed before their descriptor lands or a listing could observe a half-removed ref. The live-clone probe fails closed on an unreadable mount directory (the caller's next step is a force-detach), rmi --force refuses to drop a cache whose volume a live run still uses, and list reports the full os/arch/variant platform.

The on-disk store is the contract: pulls must produce a valid OCI image-layout that other tools can read and that agrees with registry truth on the manifest digest. Conformance tests pin the layout shape, and scripts/oci-interop.sh cross-checks the store with crane, skopeo, and umoci. CI splits by runner capability: a Linux job runs the pure-Go store paths (pull, unpack, inspect, lifecycle) without Hypervisor.framework, a hosted macOS job builds and tests the darwin sparsebundle code and drives a run-less pull/inspect/list/rmi/prune lifecycle smoke, and the self-hosted release leg boots guests end to end under HVF: the alpine:3 default-entrypoint smoke plus a full pull -> inspect -> list -> run -> rmi -> prune lifecycle that runs python:3.12-slim with an --entrypoint override and asserts the teardown half of the lifecycle.

Cover the two-binary model in README and docs: usage.md documents the elfuse-container commands and flags, testing.md the offline/CI validation split, and oci-design.md records the design rationale -- the C/Go boundary, the image-layout store with its refs.json pin table, layer application, run paths, and lifecycle GC -- plus an explicit scope-and-limitations accounting of which OCI features are and are not implemented. oci-design.md also records the concurrency model (store metadata is lock-serialized; per-run mount ownership for concurrent runs of one image is intentionally out of scope), and usage.md notes the resolv.conf fallback nameserver and its split-DNS implication.

henrybear327 requested review from Max042004 and jserv and removed request for Max042004 July 9, 2026 21:35

henrybear327 mentioned this pull request Jul 9, 2026

Add OCI image support: pull, unpack, run, prune, status, policy #34

Closed

This comment was marked as resolved.

Sign in to view

jserv requested a review from Max042004 July 9, 2026 22:18

henrybear327 force-pushed the oci/setup branch from df0bd89 to db0791c Compare July 9, 2026 22:29

jserv requested changes Jul 9, 2026

View reviewed changes

henrybear327 force-pushed the oci/setup branch 2 times, most recently from c18f864 to ed17166 Compare July 10, 2026 21:56

This comment was marked as resolved.

Sign in to view

henrybear327 marked this pull request as draft July 11, 2026 12:21

jserv requested changes Jul 11, 2026

View reviewed changes

henrybear327 force-pushed the oci/setup branch from 2afb675 to 2fd5111 Compare July 11, 2026 16:18

henrybear327 force-pushed the oci/setup branch 2 times, most recently from 1e6dee1 to 2bf9787 Compare July 12, 2026 19:56

henrybear327 force-pushed the oci/setup branch from 2bf9787 to e9206d9 Compare July 13, 2026 13:30

henrybear327 marked this pull request as ready for review July 13, 2026 14:12

henrybear327 requested a review from jserv July 13, 2026 14:13

henrybear327 force-pushed the oci/setup branch from e9206d9 to 76c8f3b Compare July 13, 2026 14:21

This comment was marked as resolved.

Sign in to view

henrybear327 force-pushed the oci/setup branch 3 times, most recently from 36df58c to 72eac43 Compare July 13, 2026 21:09

henrybear327 force-pushed the oci/setup branch 2 times, most recently from 22b57ac to dcf7e45 Compare July 13, 2026 21:15

jserv reviewed Jul 14, 2026

View reviewed changes

Comment thread .github/workflows/main.yml

jserv requested changes Jul 14, 2026

View reviewed changes

jserv reviewed Jul 14, 2026

View reviewed changes

henrybear327 force-pushed the oci/setup branch from dcf7e45 to 0674752 Compare July 14, 2026 20:10

henrybear327 added 4 commits July 14, 2026 22:11

henrybear327 force-pushed the oci/setup branch from 0674752 to 02e974b Compare July 14, 2026 20:11

henrybear327 added 3 commits July 14, 2026 22:51

henrybear327 force-pushed the oci/setup branch from 02e974b to ebee672 Compare July 14, 2026 20:57

Uh oh!

Conversation

henrybear327 commented Jul 9, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

This comment was marked as resolved.

Uh oh!

jserv left a comment

Choose a reason for hiding this comment

Uh oh!

jserv commented Jul 9, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

henrybear327 commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jserv left a comment

Choose a reason for hiding this comment

Uh oh!

henrybear327 commented Jul 11, 2026

Uh oh!

jserv commented Jul 11, 2026

Uh oh!

henrybear327 commented Jul 13, 2026

Uh oh!

henrybear327 commented Jul 13, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

henrybear327 commented Jul 13, 2026

Uh oh!

Uh oh!

jserv left a comment

Choose a reason for hiding this comment

Uh oh!

jserv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

henrybear327 commented Jul 9, 2026 •

edited by cubic-dev-ai Bot

Loading

henrybear327 commented Jul 11, 2026 •

edited

Loading

jserv left a comment •

edited

Loading