atelet: node-local file cache library (filecache, M1) - #1517
Open
Dmitry Berkovich (dberkov) wants to merge 6 commits into
Open
atelet: node-local file cache library (filecache, M1)#1517Dmitry Berkovich (dberkov) wants to merge 6 commits into
Dmitry Berkovich (dberkov) wants to merge 6 commits into
Conversation
This was referenced Sep 5, 2026
Introduce cmd/atelet/internal/filecache, the foundation of a node-local artifact cache: opaque entry keys (content-addressed sha256 and immutable-URI forms), the entries/tmp on-disk layout, a startup sweep for crash debris (unfinished fetches, interrupted evictions), and byte accounting for a GC budget. Golden snapshot restores download their files per actor with no reuse, and sandbox-asset fetches race concurrent downloads of the same asset; this package is the shared cache that will back both paths. Retrieval (singleflight fetch, atomic publication, hardlink-out) and eviction build on this skeleton in follow-up changes.
GetFileTo materializes a cached artifact at a destination path via hard link, fetching it on a miss. Concurrent callers for one key share a single fetch (singleflight), and the fetch runs detached from the callers' contexts bounded by the store's fetch timeout, so one canceled caller never aborts a download other callers are waiting on. There is no negative caching: a failed fetch reaches every waiting caller and the next call starts fresh. A fetch lands in tmp/, must produce a regular file, is made read-only (0444) so a consumer's in-place write fails loudly instead of corrupting the shared copy, and is published with one atomic rename. A hit links out and touches the entry's last-use clock under a shared lock that eviction will hold exclusively, closing the hit-vs-evict window. Destinations must not exist and must be absolute paths on the cache's mount; cross-filesystem destinations fail with a dedicated error rather than a silent copy.
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m1
branch
from
September 5, 2026 03:13
6f7da13 to
f7d72fb
Compare
copyFile and its hole-preserving machinery lived in package main, usable only by atelet's own checkpoint staging. The filecache package is about to need the same copy (its copy-out mode hands consumers a private, hole-preserved copy of a cached artifact), so move the code where both can import it. Mechanical move, with one seam added: Copy(src, dst *os.File) exposes the engine on caller-owned handles, for callers that must open the source before its name can vanish or create the destination with O_EXCL. CopyFile keeps its os.Create semantics for the existing caller.
GetFileTo serves hits as read-only hard links, which is only safe for consumers that never write the staged file in place. GetFileCopyTo serves the same read-through cache as a private copy instead: the caller owns the resulting inode outright (mode 0600) and may mutate it freely, holes are preserved, and the destination may live on any filesystem. The copy reads a handle opened under the hit lock, so an eviction racing the copy retires only the entry's name — the bytes survive until the copy completes. Fetch dedup is unchanged: concurrent calls for one key share a single flight.
EvictUnused frees cache space least-recently-used first until a byte target is met, with a two-phase retire: inside the key's singleflight and the hit lock, a victim is re-verified (a moved last-use clock or an in-flight fetch vetoes) and renamed to a .rm-* dir, making it invisible to lookups; the slow physical deletion runs after all retires, outside the locks the hot path contends, so hits and fetches never wait on it. Entries younger than the store's min age are never touched, covering the window between publication and a consumer's first link. Entries whose data a consumer still hard-links may be retired but count as pending rather than freed bytes - the kernel returns that space when the last consumer link goes - so eviction can only ever cost a re-download, never break a consumer. FreedBytes is credited per entry only after its physical removal succeeds; a failed removal leaves the bytes in a .rm-* dir for the startup sweep and out of the freed count.
State the package's consumer-protection contracts in the package doc (link-out immunity, min-age sizing, the read-only shared-bytes rule, and key immutability), and pin them with a race-detector stress test: getters and evictors hammer the same keys concurrently, and every get must succeed with intact content - eviction may force refetches but can never fail a caller, corrupt a served file, or leave half-states in the store.
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m1
branch
from
September 5, 2026 04:59
f7d72fb to
ee7dd93
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements milestone M1 of the node-local artifact cache proposed in #690 (design in the issue comment): a generic
cmd/atelet/internal/filecachepackage that will back golden-snapshot restores (today re-downloaded per actor on every start/resume) and later the sandbox-asset fetches. it reads well commit by commit:atelet: add filecache store skeleton— constructor-onlyKeys (SHA256Keycontent-addressed,URIKeyfor immutable sources; prefix-disjoint canonical forms, entry dir =sha256(key)), theentries/+tmp/+.rm-*layout,SweepDebris(startup crash-debris reaper),TotalBytes(GC budget measure), debug-onlymeta.json.atelet: add filecache singleflight retrieval (GetFileTo)— atomic get-and-link: per-key singleflight oncontext.WithoutCancel+ fetch timeout (a canceled caller never aborts the download others wait on; no negative caching); fetch intotmp/, validate, chmod0444(in-place writes fail loudly instead of poisoning shared bytes), publish by one atomic rename; hit = hard link + LRU touch underhitMu.RLock. Path-basedFileFetchersoategcs's sparse zstd download plugs in unchanged;%wwrapping end-to-end forateerrorsclassification.atelet: move the sparse file copy helpers into internal/sparsefile— mechanical move ofcopyFile/copySparse/kernelCopyRange(and their tests) out of package main so filecache can reuse them; addsCopy(src, dst *os.File)for caller-owned handles (source opened before its name can vanish, destination createdO_EXCL).filecache: add GetFileCopyTo for consumers that mutate staged files— the second serving mode: a private, hole-preserving copy (mode0600) instead of a read-only hard link, for consumers that rewrite staged files in place (ateom-microvm rewritesconfig.jsonat restore and merges deltas intomemory-rangesat suspend — a shared inode would be corrupted). The copy reads a handle opened under the hit lock, so an eviction racing the copy retires only the entry's name; a copy needs no same-mount constraint.atelet: add filecache eviction (EvictUnused)— pressure-driven only: min-age gate, unlinked-first then LRU ordering, stop at target. Two-phase retire inside the key's singleflight +hitMuexclusive (moved last-use clock or in-flight fetch vetoes; rename to.rm-*), slowRemoveAllafter all retires outside the hot-path locks. Stats distinguishRetired(namespace removal, irreversible at rename) fromFreedBytes(credited only after physical removal succeeds) andPendingBytes(retired but consumer-linked; kernel frees later). Copied-out entries carry no links, so eviction is free to take them — existing copies are private inodes and unaffected.atelet: document filecache contracts and stress the get/evict races— package-doc contracts (link-out immunity, copy-out privacy, min-age sizing rule, read-only shared bytes, key immutability) plus a race-detector stress test: concurrent getters and evictors on shared keys; every get must succeed with intact content.The core safety property throughout: eviction can only ever cost a refetch — never break a consumer. Hard-linked files are protected by the link itself (the consumer's inode survives eviction); copies are private inodes; the min age covers the publish-to-use window.
Follow-ups per the design: M2 wires a golden store into
Restore(downloadExternalCheckpoint/downloadCombinedCheckpoint) with a GC driver loop — gVisor restores get hard links, micro-VM restores get copies; M3 addsGetFile/GetDir+ the sandbox-record root set and migratesfetchAsset/fetchGVisorRelease.Tested:
go test -race -count=3 ./cmd/atelet/internal/filecache/; every commit builds and passes tests individually;golangci-lint, gofmt, and boilerplate checks clean.🤖 Generated with Claude Code