Repository navigation
fix(runtime): don't cancel podman start/run mid netns setup (RIG-5334) - #2305
Merged
mattwilkinsonn merged 2 commits intoOct 15, 2026
Merged
Conversation
A signal to the podman client after pasta writes its pid file but before podman writes `rootless-netns/resolv.conf` leaves pasta alive with no `resolv.conf`. podman reuses that netns, so every later run fails with `mount resolv.conf to "/etc/resolv.conf": no such file or directory`. Caller ctx cancellation (runner shutdown) SIGKILLed in-flight `podman start`. `PodmanCLI.Start`, the pasta canary `podman run`, and the stack adapters' `podman run` now ignore caller cancellation. They stay bounded by their own command timeout and return the ctx error after exit. Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
😎 This pull request was merged. |
|
Compass engineering docs preview: https://compass-runner-rig-5334-podm.compass-eng-docs.pages.dev Deployed from |
…on (RIG-5334) Review fix: a run that succeeded after caller cancellation was reported as cancelled, so the stack dropped a live container from its teardown record. Return the command's own result; the caller's next ctx check still unwinds. Cancellation tests now prove the stub is alive while gated, so a regressed CommandContext kill fails deterministically. Co-authored-by: Matt Wilkinson <matt@rigel.build>
Contributor
Author
|
Review for head
|
rigel-mintaka
marked this pull request as ready for review
October 15, 2026 08:16
mattwilkinsonn
deleted the
compass-runner/rig-5334-podman-no-cancel-kill
branch
October 15, 2026 15:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes RIG-5334: e2e podman probe fails with
rootless netns: mount resolv.conf to "/etc/resolv.conf": no such file or directory, and every later podman run fails the same way.Root cause
setupPastastarts pasta, writesrootless-netns-conn.pid, then writesrootless-netns/resolv.conf.resolv.conf.getOrCreateNetnssees a live pasta, so it reuses the broken netns for every later run.resolv.conf, conn pid alive. The netns was created about 0.4 s before the stack test signalled the runner to shut down. The runner's ctx cancellation madeexec.CommandContextSIGKILL the in-flightpodman start.Local repro on podman 5.8.6, signalling
podman unshare --rootless-netnsonce the pid file appears:podman startof a bridge containerChange
PodmanCLI.Start, the pasta-canarypodman run, and the stack adapters'podmanExec.runnow run underWithoutCancel(ctx)plus their own command timeout.ctx.Err(), so existing cleanup paths are unchanged.Tests
TestPodmanStartSurvivesCallerCancellation/TestPodmanRunSurvivesCallerCancellation: the stub podman blocks after start. The test cancels the ctx, then releases the stub, and asserts the stub ran to completion and the call returnedcontext.Canceled. Both were red before the fix: the stub was killed.TestPodmanStartTimeoutKillsHungProcess/TestPodmanRunTimeoutKillsHungProcess: the per-command timeout still kills a hung podman.go test -race -count=1 ./internal/runtime/... ./internal/stack/...: ok.golangci-lint: 0 issues.