Skip to content

fix(release): stop compiling cryo and Go in the release path - #97

Merged
Savid merged 1 commit into
masterfrom
fix/release-timeout-prebuilt-cryo
Jul 28, 2026
Merged

fix(release): stop compiling cryo and Go in the release path#97
Savid merged 1 commit into
masterfrom
fix/release-timeout-prebuilt-cryo

Conversation

@Savid

@Savid Savid commented Jul 28, 2026

Copy link
Copy Markdown
Member

Problem

Releases are taking >1h and failing. Run 30244015572 is the clearest example:

06:54:20  • building docker image  image=.../execution-processor:0.2.0-arm64v8
06:54:20  • building docker image  image=.../execution-processor:0.2.0-amd64
07:52:11  ⨯ release timed out after 1h0m0s   error=context deadline exceeded

The release Dockerfile compiled two things per architecture — cryo (Rust: polars + alloy) and the Go binary — and the arm64 leg did both under QEMU. Neither architecture finished in time.

The buildx GHA cache meant to absorb this never worked, so every release started cold. It was broken two ways over:

  1. .github/workflows/goreleaser.yaml runs goreleaser in a container but only forwards GITHUB_TOKEN, DOCKER_USERNAME, DOCKER_PASSWORD, RELEASE_SUFFIXACTIONS_RUNTIME_TOKEN never reaches it, so type=gha has no credentials.
  2. Buildx builder instances live in the CLI's ~/.docker/buildx on the host, so inside the container docker buildx build falls back to the default docker driver, which can't import/export type=gha at all. ignore-error=true on --cache-to silently swallowed the failure.

Those flags are removed rather than left misleading.

Approach

Both compiles now happen ahead of the release.

Dockerfile.cryo + .github/workflows/cryo-image.yml build the pinned cryo as ethpandaops/cryo:<sha>, each arch on a native runner (ubuntu-24.04 / ubuntu-24.04-arm), joined into a manifest. It only runs when the pin moves, and reads ARG CRYO_SHA back out of the root Dockerfile so there's a single source of truth.

The release Dockerfile now compiles nothing — cryo comes from that image, and the server binary from goreleaser, which already cross-compiles both arches natively with CGO_ENABLED=0.

GitHub-hosted runners on both legs: they're parallel, so wall clock is bound by the slower (arm) leg either way. cryo reaches its final crate in ~85s natively, so this is minutes.

Drive-by bug

ldflags pointed at cmd.Release/cmd.GitCommit, which don't exist — the symbols are in internal/version (what the Makefile uses). -X against a missing symbol fails silently, so released archives have been reporting version dev. This became load-bearing here: the images previously got their version via make build-binary and now inherit goreleaser's ldflags.

Verified with go version -m on both arch binaries:

-ldflags="-s -w -X github.com/ethpandaops/execution-processor/internal/version.Release=v0.2.0-c6b3bce ..."

Testing

Dry-ran the pipeline locally with a stubbed cryo image:

  • both arch binaries build in ~4s, statically linked, correct architectures — confirming the in-Docker Go build was pure waste
  • images assemble; cryo is on PATH and executable as appuser (uid 1000)
  • goreleaser check passes (remaining deprecations are pre-existing)
  • docker buildx build --check on Dockerfile.cryo: no warnings

Merge order

Merge this first, let cryo image publish, then tag. The release needs ethpandaops/cryo:559b654… to exist.

ethpandaops/cryo doesn't exist on Docker Hub yet. It'll be auto-created on first push provided DOCKERHUB_TOKEN can create repos — if it's scoped to existing repos only, create it manually first.

No "make it public" step needed: goreleaser-cross runs /entrypoint.sh, which logs into Docker Hub from DOCKER_USERNAME/DOCKER_PASSWORD, so the FROM ethpandaops/cryo:<sha> pull is authenticated. That also keeps you off anonymous pull rate limits.

Note

The release Dockerfile now expects a prebuilt binary in its context, so docker build . no longer works standalone. Nothing in the README or Makefile referenced it. Easy to add a Dockerfile.dev if that's wanted.

🤖 Generated with Claude Code

Releases were exceeding goreleaser's 1h timeout and failing. The release
Dockerfile compiled two things per architecture - cryo (Rust: polars +
alloy) and the Go binary - and the arm64 leg did both under QEMU. Neither
architecture finished in time.

The buildx GHA cache that was meant to absorb this never worked, so every
release started from a cold cache. It was broken two ways over: the
goreleaser container is never passed ACTIONS_RUNTIME_TOKEN, and the default
docker driver it falls back to inside that container cannot import or
export type=gha at all. The flags are removed rather than left misleading.

Both compiles now happen ahead of the release:

- Dockerfile.cryo + .github/workflows/cryo-image.yml build the pinned cryo
  as ethpandaops/cryo:<sha>, each architecture on a native runner, joined
  into a manifest. It only runs when the pin actually moves, and reads the
  pin back out of the root Dockerfile so there is a single source of truth.

- The release Dockerfile now compiles nothing. It takes cryo from that
  image and the server binary from goreleaser, which already
  cross-compiles both architectures natively with CGO_ENABLED=0.

Also fixes ldflags that pointed at cmd.Release and cmd.GitCommit, which do
not exist - the symbols live in internal/version. -X against a missing
symbol fails silently, so released archives have been reporting the
version as "dev". This mattered here because the images previously got
their version via make build-binary and now inherit goreleaser's ldflags.

Note the release Dockerfile now expects a prebuilt binary in its context,
so it is no longer buildable standalone with `docker build .`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Savid
Savid merged commit c44422d into master Jul 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant