Record the SandboxConfigRef in snapshot storage instead of sandbox assets - #1423
Open
Zoe Zhao (zoez7) wants to merge 3 commits into
Open
Record the SandboxConfigRef in snapshot storage instead of sandbox assets#1423Zoe Zhao (zoez7) wants to merge 3 commits into
Zoe Zhao (zoez7) wants to merge 3 commits into
Conversation
…sets Snapshots used to be self-describing: the manifest written beside the checkpoint images carried the full sandbox asset set (binary URLs, hashes, pause image), and a restore rebuilt the sandbox from whatever the manifest said. That pins a snapshot to raw asset content with no way to tell which SandboxConfig it came from, or to notice that the config has been deleted or re-created since. Make the durable storage objects record a reference to the SandboxConfig Kubernetes object instead: - The snapshot manifest now carries only the SandboxConfig reference (name, UID, resourceVersion) alongside the identity, file list, and scope - never asset content. - Checkpoint reports the reference from the actor's on-node sandbox record (written at Run/Restore with the assets the sandbox actually booted with) in its response, and the pause and suspend finalizers stamp it onto the LocalSnapshotInfo / ActorSnapshot records. A paused-origin suspend takes the reference from LocalSnapshotInfo: no checkpoint runs on that path, and the pause that wrote it got the reference from its own checkpoint response. Every path reaching a finalizer holds a fresh response, since workflow retries re-send the Checkpoint RPC, so the reference never has to survive across invocations. - A resume resolves the restore's assets from the reference recorded on the snapshot whose sandbox runs the guest (the replacement template's golden config for a repointed actor, the golden snapshot's for a DATA_ON_GOLDEN restore, the actor's own snapshot's otherwise), erroring with FailedPrecondition when the object is gone or was re-created under the same name. Snapshots recorded before the reference existed report none and fall back to the pool's config. RestoreRequest therefore now requires sandbox_assets: the manifest is no longer a source of binaries. atelet's node-local flow is otherwise unchanged: Run/Restore write the per-actor on-node record (asset set plus, now, the reference) and Checkpoint/Terminate read it back, so those requests still carry no sandbox config. Checkpoint pins the record's reference into the manifest, and Restore cross-checks the manifest's reference against the request's assets as defense in depth. The update-sandbox-config e2e flow asserts provenance on the snapshots the suspends produce (where restores actually read it), and gains the WorkerPool SandboxConfig override hook it uses to point template B's pool at the copied config.
Collaborator
the manifest fetch is a bit of a latency limiter right now, I think we need to start planning to drop this otherwise agree directionally |
Zoe Zhao (zoez7)
marked this pull request as ready for review
September 3, 2026 00:53
A DATA restore cold-boots the guest from the template spec; no manifest's memory image resumes. After an actor is repointed at a template with a different SandboxConfig, the control plane rightly resolves the request's assets from the replacement template's golden snapshot, so they no longer match the reference recorded on the actor's own snapshot manifest and the defense-in-depth check rejected the resume with FailedPrecondition. Only enforce the cross-check when the guest actually resumes from a manifest's state (FULL and DATA_ON_GOLDEN restores).
The SandboxConfig reference stamped on snapshot records and manifests carried a resourceVersion that nothing consumed: an in-place update to a SandboxConfig (same UID, new revision) would silently resolve different binaries under a previously captured snapshot. Every resume path now holds the recorded reference to the exact object revision. resolveSandboxAssetsByRef fails with FailedPrecondition when the object's resourceVersion no longer matches the recorded one, just as it already did for a UID mismatch; atelet's non-DATA manifest cross-check likewise rejects a resourceVersion mismatch between the guest manifest's reference and the request's assets. A snapshot or golden recorded under a superseded revision fails resume loudly instead of booting binaries it was not captured with. resourceVersion is the only usable version signal: SandboxConfig has no status subresource, so metadata.generation never increments.
Zoe Zhao (zoez7)
requested review from
Benjamin Elder (BenTheElder) and
Dmitry Berkovich (dberkov)
September 3, 2026 00:53
Zoe Zhao (zoez7)
force-pushed
the
snapshot-sandbox-config-ref
branch
from
September 3, 2026 00:55
f9de248 to
ccb3ba6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change makes durable snapshot storage record a reference to the SandboxConfig object instead:
LocalSnapshotInfo/ActorSnapshot.FailedPreconditionwhen the SandboxConfig is gone or was re-created under the same name. Older snapshots without a reference fall back to the pool's config.RestoreRequestnow requiressandbox_assets, since the manifest is no longer a source of binaries.Also relaxes
UpdateActortemplate validation to compare volumes by name rather than list order, since volumes are referenced by name and order carries no meaning.Test plan
updatesandboxconfige2e suite asserting SandboxConfig provenance on the snapshots that suspends produce, including the repointed-actor restore path.