Support diskless directly with bootc

[cgwalters]

Chatting with someone here at KubeCon, we realized a nice way to diskless would be to use PXE boot, have bootc in the initramfs, then fetch the image to RAM and pivot into it.

This would replace squashfs with OCI which would be pretty cool.

[kfox1111]

This would be great. :)

[kfox1111]

It could potentially be used as a base implementation for kubevirt/kubevirt#13212.

[scottcwang]

Dracut dmsquash-live and livenet currently support root=live:LABEL=*, root=live:UUID=*, root=live:PARTLABEL=*, root=live:PARTUUID=*, root=live:CDLABEL=*, root=live:nfs*, root=live:http://*, root=live:https://*, root=live:ftp://*, root=live:torrent://*, root=live:tftp://*. Perhaps some of this existing logic could get reused to parse the root dracut argument, find the container image, and pull it into the same RAM disk that dracut already uses for live booting.

[chaserhkj]

This issue looks stale but I think it is still cool. I do have a coreos diskless workstation deployment myself in my home lab so this is very relevant to me, too. I'll see if I can make it work and make a contribution.

[chaserhkj]

Ok I got a proof-of-concept working here

I wanted to make fedora coreos image to work with this first, but apparently when generating initramfs for live boot osbuild did something extra on top of the dracut-produced initramfs file and coreos dracut module is depending on these extended stuff to pull the hooks and services correctly and setup for a live system.

I am thinking about making the stamp file myself and see what part of coreos-live machinery would just work with this (apparently there would be parts that won't work like the image mounting part). But maybe trying this with a simpler bootc image actually makes more sense.

[chaserhkj]

I switched the base image to test from fedora-coreos to fedora-bootc and it was much better, the project linked above should be usable now.

Just not very sure how this could be integrated into bootc itself and become a contribution.

Do we want to add extra dracut modules that ships with bootc and enables live boot? The trouble is that probably most of the bootc images need some distribution-specific modifications to work with live boot anyways.

Also since I am currently just use umoci for extracting the oci archive this is pulling some external dependency and I am not using any of the bootc tool code at all (after all it's really just reading a container image).

[cgwalters]

Also since I am currently just use umoci for extracting the oci archive this is pulling some external dependency and I am not using any of the bootc tool code at all (after all it's really just reading a container image).

That's cool for a PoC work for sure, but I think we want bootc status and such to operate, and / should still be readonly by default, and ultimately we do want to support soft reboots into new updates - this type of stuff needs to go beyond just using umoci/skopeo to copy from the registry.

[cgwalters]

Thanks for working on this!!

Just not very sure how this could be integrated into bootc itself and become a contribution.
Do we want to add extra dracut modules that ships with bootc and enables live boot? The trouble is that probably most of the bootc images need some distribution-specific modifications to work with live boot anyways.

We already do have a dracut module today for the initramfs setup for composefs. (Of course we shouldn't be tied to dracut, this can just be a systemd unit that runs in the initramfs, which dracut just installs, but so could another tool)

I think basically it'd accept e.g. bootc.image.spec=registry.example.com/someimage@sha256:abcd123 or so as a karg.

WDYT?

[chaserhkj]

The POC is sound, I have used it for some while on my diskless workstation, but I totally agree that a proper integration into bootc project would need to do much more than the rough skopeo/umoci pull.

What is good is that I have been improving the POC for a bit as well so I feel some of the ideas and experiences there would be transferable. So what the POC is doing is that it tries to setup an ext4 partition on a zram device, pulling the image to it and unpacking the rootfs, then bind mount to make / read only and /etc /var read-write. This emulates a real bootc setup and somehow worked out well. Zram really helps a lot since the unpacking process might eat a lot of tmpfs space that could lead to a problem. We might want this too in the proper implementation.

In the POC I am also using kexec to load the kernel image from the pulled image as well. This eliminates the implicit requirement that the bootloader loaded kernel and the image kernel needs to be kmod-compatible. This might be the ultimate route bootc wants to pursue on the soft reboot direction, too.

For the proper implementation, I would envision we do it with composefs backend and extend the current initramfs module to:

1. Match a particular karg (bootc.image.spec=) to activate a new setup-live-root subcommand in the initramfs binary
2. Setup a zram device and create a fsverify-supporting FS on it, as physical root (maybe take another karg to determine which fs to use)
3. call create_composefs_filesystem and commit_image to pull the image to composefs repo on physical root and create rootfs image out of it
4. call write_composefs_state to write the deployment state in composefs repo
5. Extract the image id created in step 3 and hand over the normal booting process to setup-root.

If this sounds like a good plan, I could start with a draft PR on this.