wip: zstd:chunked support via oci-client - #137
Conversation
This code pulls images via containers-image-proxy-rs (which is a wrapper around skopeo). It's a bit too large to have it in the top-level lib.rs, and we're about to add an alternate implementation, so split it out. We keep a pull() function in place for compatibility reasons to avoid needing to update all callers: we will start using this function to dispatch to the correct implementation depending on the reference format. Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
This is based on the "pull" example in the zstd-chunked-rs repository. Signed-off-by: Allison Karlitskaya <allison.karlitskaya@redhat.com>
|
There's some cool code here, and I think it's really useful actually to have this as an option even if it's hidden/experimental just to use as a comparison. The main issue is really c/image (the important part of skopeo here) vs oci-client as expanded on in https://github.com/containers/containers-image-proxy-rs/?tab=readme-ov-file#why and it's for that reason I can't imagine using this by default in bootc at least anytime in the nearly visible future. Things like ECR credential helpers are highly load bearing. OTOH, other folks could reasonably make other choices. |
|
Ya. My first instinct is to stick this behind a feature flag (and maybe also do the same with the skopeo and potential-future-oci-dir paths)... it could be useful for the kind of scenario where we don't need the skopeo features. In terms of those features, I feel like:
I feel like we could maybe close some of these gaps by adding some code to skopeo to do the work of resolving registry aliases, taking care of the initial authentication, etc. and call out to that. I'm also sort of of the (optimistic) opinion that maybe we can have a Rust-based stack for all of this stuff at some point, and honestly, the sooner the better... and although this piece of code here leaves a lot of the other questions unanswered, it's maybe one part of moving towards a world where we could image a feasible solution to that puzzle... |
cgwalters
left a comment
cgwalters
left a comment
There was a problem hiding this comment.
The first commit here though seems sane to me, let's split it into its own PR and get it in?
→ #138 |
|
So the biggest unanswered question here is what to do about the sha256-to-fsverity mapping. It's currently stored as raw My initial impulse was to store symlinks in the repository, but: this is just about the worst possible thing we can do. There's no actual benefit to being able to follow sha256 symlinks into the object files and this is only used for zstd:chunked. I'd actually prefer something more like an sqlite database storing sha256->fs-verity mappings. Also: it might make sense to store that in @alexlarsson is currently trying to solve a very similar issue for the ostree mapping... |
zstd:chunked layers are multi-frame zstd streams (one frame per chunk, plus skippable framing metadata), and gzip layers may legally contain multiple members. The decoders stop at the first frame or member boundary, so such layers import a truncated tar and fail with "unexpected EOF reading tar entry" (bootc-dev/bootc#2408). Enable multiple_members(true) on both decoders so the full stream decodes; libzstd skips skippable frames natively at any position. This is full-pull correctness only: nothing exploits the chunked TOC for partial pulls, which remains composefs#137. Raise the async-compression floor to 0.4.43, which fixed an infinite loop on a corrupt later frame with multiple_members enabled (Nullus157/async-compression#470); older versions could hang on a malformed blob once the flag is on. Generated-by: AI Signed-off-by: Andrew Dunn <andrew@dunn.dev>
2 participants
No description provided.