atelet: add filecache store skeleton - #1512
Closed
Dmitry Berkovich (dberkov) wants to merge 1 commit into
Closed
Conversation
Introduce cmd/atelet/internal/filecache, the foundation of a node-local artifact cache: opaque entry keys (content-addressed sha256 and immutable-URI forms), the entries/tmp on-disk layout, a startup sweep for crash debris (unfinished fetches, interrupted evictions), and byte accounting for a GC budget. Golden snapshot restores download their files per actor with no reuse, and sandbox-asset fetches race concurrent downloads of the same asset; this package is the shared cache that will back both paths. Retrieval (singleflight fetch, atomic publication, hardlink-out) and eviction build on this skeleton in follow-up changes.
This was referenced Sep 5, 2026
Collaborator
Author
|
Closing in favor of a single consolidated PR carrying the whole M1 library as reviewable commits — the fork-based stacking made per-PR diffs and CI awkward. Replacement PR link to follow. |
Collaborator
Author
|
Consolidated into #1517. |
Max Smythe (maxsmythe)
pushed a commit
that referenced
this pull request
Sep 5, 2026
`main` is red since #1480 merged: its regenerated `cmd/ateapi/internal/controlapi/zz_generated.validation.go` was produced by a stale generator build and references `deepEqualImpl_`, which doesn't exist — `cmd/ateapi` (and everything importing `controlapi`) fails to compile, and every open PR inherits the failure through CI's merge-with-main (see #1512's `run-tests` for an example, and `main`'s own post-merge `pr-workflow` runs for #1480/#1499). Fix is a pure regeneration: `hack/update/codegen.sh` on current `main` changes exactly one line, `deepEqualImpl_` → `ateDeepEqual` (the helper the current generator actually emits). After it, `go build ./cmd/ateapi/...` succeeds and `go test ./cmd/ateapi/internal/controlapi/` passes. No hand-written changes — the diff is generator output only. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of the node-local artifact cache proposed in #690 (see the design comment there): a generic
cmd/atelet/internal/filecachepackage that will back golden-snapshot restores (today re-downloaded per actor) and sandbox-asset fetches (today raced by concurrent misses, never evicted).This PR is the concurrency-free skeleton:
key.go): constructor-onlyKeywith two forms —SHA256Keyfor content-addressed artifacts andURIKeyfor immutable-URI sources. Canonical forms are prefix-disjoint, part joins are unambiguous (NUL separator), and the entry dir name issha256(canonical)so a future GC root set can match entries by hashing keys.filecache.go):New+WithMinAge/WithFetchTimeout(defaults mirror imagecache's 2m/10m), theentries/+tmp/+.rm-*layout, and the debug-onlymeta.jsonsidecar helpers.SweepDebris: startup reaper for crash debris (unfinished fetches undertmp/, interrupted evictions as.rm-*); never touches published entries; joins removal errors instead of stopping at the first.TotalBytes: usage measure for the future GC driver's byte budget; tolerates entries vanishing mid-walk.Follow-ups (stacked):
GetFileTo(singleflight fetch + atomic publication + hardlink-out), thenEvictUnused(LRU + min-age + two-phase retire).Tested with
go test -race ./cmd/atelet/internal/filecache/;make verifyclean except the pre-existingclang-format-missing failure inproto-fmt.sh(no protos touched).🤖 Generated with Claude Code