fix(release): stop compiling cryo and Go in the release path - #97
Merged
Conversation
Releases were exceeding goreleaser's 1h timeout and failing. The release Dockerfile compiled two things per architecture - cryo (Rust: polars + alloy) and the Go binary - and the arm64 leg did both under QEMU. Neither architecture finished in time. The buildx GHA cache that was meant to absorb this never worked, so every release started from a cold cache. It was broken two ways over: the goreleaser container is never passed ACTIONS_RUNTIME_TOKEN, and the default docker driver it falls back to inside that container cannot import or export type=gha at all. The flags are removed rather than left misleading. Both compiles now happen ahead of the release: - Dockerfile.cryo + .github/workflows/cryo-image.yml build the pinned cryo as ethpandaops/cryo:<sha>, each architecture on a native runner, joined into a manifest. It only runs when the pin actually moves, and reads the pin back out of the root Dockerfile so there is a single source of truth. - The release Dockerfile now compiles nothing. It takes cryo from that image and the server binary from goreleaser, which already cross-compiles both architectures natively with CGO_ENABLED=0. Also fixes ldflags that pointed at cmd.Release and cmd.GitCommit, which do not exist - the symbols live in internal/version. -X against a missing symbol fails silently, so released archives have been reporting the version as "dev". This mattered here because the images previously got their version via make build-binary and now inherit goreleaser's ldflags. Note the release Dockerfile now expects a prebuilt binary in its context, so it is no longer buildable standalone with `docker build .`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Releases are taking >1h and failing. Run 30244015572 is the clearest example:
The release Dockerfile compiled two things per architecture — cryo (Rust: polars + alloy) and the Go binary — and the arm64 leg did both under QEMU. Neither architecture finished in time.
The buildx GHA cache meant to absorb this never worked, so every release started cold. It was broken two ways over:
.github/workflows/goreleaser.yamlruns goreleaser in a container but only forwardsGITHUB_TOKEN,DOCKER_USERNAME,DOCKER_PASSWORD,RELEASE_SUFFIX—ACTIONS_RUNTIME_TOKENnever reaches it, sotype=ghahas no credentials.~/.docker/buildxon the host, so inside the containerdocker buildx buildfalls back to thedefaultdocker driver, which can't import/exporttype=ghaat all.ignore-error=trueon--cache-tosilently swallowed the failure.Those flags are removed rather than left misleading.
Approach
Both compiles now happen ahead of the release.
Dockerfile.cryo+.github/workflows/cryo-image.ymlbuild the pinned cryo asethpandaops/cryo:<sha>, each arch on a native runner (ubuntu-24.04/ubuntu-24.04-arm), joined into a manifest. It only runs when the pin moves, and readsARG CRYO_SHAback out of the rootDockerfileso there's a single source of truth.The release
Dockerfilenow compiles nothing — cryo comes from that image, and theserverbinary from goreleaser, which already cross-compiles both arches natively withCGO_ENABLED=0.GitHub-hosted runners on both legs: they're parallel, so wall clock is bound by the slower (arm) leg either way. cryo reaches its final crate in ~85s natively, so this is minutes.
Drive-by bug
ldflags pointed at
cmd.Release/cmd.GitCommit, which don't exist — the symbols are ininternal/version(what the Makefile uses).-Xagainst a missing symbol fails silently, so released archives have been reporting versiondev. This became load-bearing here: the images previously got their version viamake build-binaryand now inherit goreleaser's ldflags.Verified with
go version -mon both arch binaries:Testing
Dry-ran the pipeline locally with a stubbed cryo image:
cryois onPATHand executable asappuser(uid 1000)goreleaser checkpasses (remaining deprecations are pre-existing)docker buildx build --checkonDockerfile.cryo: no warningsMerge order
Merge this first, let
cryo imagepublish, then tag. The release needsethpandaops/cryo:559b654…to exist.ethpandaops/cryodoesn't exist on Docker Hub yet. It'll be auto-created on first push providedDOCKERHUB_TOKENcan create repos — if it's scoped to existing repos only, create it manually first.No "make it public" step needed:
goreleaser-crossruns/entrypoint.sh, which logs into Docker Hub fromDOCKER_USERNAME/DOCKER_PASSWORD, so theFROM ethpandaops/cryo:<sha>pull is authenticated. That also keeps you off anonymous pull rate limits.Note
The release Dockerfile now expects a prebuilt binary in its context, so
docker build .no longer works standalone. Nothing in the README or Makefile referenced it. Easy to add aDockerfile.devif that's wanted.🤖 Generated with Claude Code