Repository navigation
fix(csharp-ci): disable MSBuild node reuse on shared hetzner runners - #40
monsieurleberre wants to merge 2 commits into
Conversation
The hetzner-3/hetzner-4 self-hosted runners share one host and OS user. When one job finishes, its orphan-process cleanup kills idle MSBuild node processes that a different, still-running job on the sibling runner had left attached to for node reuse, so that job's build dies mid-flight with MSB4166 (node terminated unexpectedly). Set MSBUILDDISABLENODEREUSE: '1' on the build-and-test job so MSBuild never registers or reattaches to a shared out-of-proc node in the first place. scala-ci.yaml runs no dotnet/MSBuild and needs no change. Fixes the reusable-workflow half of peacefulstudio/canton-dotnet-sdk-internal#1086; that repo's own inline jobs were already fixed in PR #1312, but env vars set by a workflow_call caller do not propagate into this workflow's own job env.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Closing without merging: superseded by the root-cause fix in peacefulstudio/infrastructure#412 (a separate .NET install per runner slot), live on the hetzner host since 2026-09-26 ~10:30Z. Job logs on hetzner-4 already show Disabling node reuse here would mask the soak criterion on peacefulstudio/canton-dotnet-sdk-internal#1086 (any MSB4166 after that time means #412 wasn't the whole story) and would slow every consumer's build, including on GitHub-hosted runners that never had the problem. Branch kept in case the soak fails and this workaround is needed after all. |
1 participant
Root cause
hetzner-3andhetzner-4are two self-hosted runners sharing one host andone OS user. When a job finishes, the runner's orphan-process cleanup kills
leftover MSBuild "node" processes belonging to that job — but it doesn't
distinguish those from idle out-of-proc MSBuild nodes that a different,
still-running job on the sibling runner registered for node reuse and is
still relying on. That sibling job's build then dies mid-flight with
MSB4166(node terminated unexpectedly).Evidence (MSBuild source)
XMake.csProcessNodeReuseSwitch—MSBUILDDISABLENODEREUSE=1forces node reuse off regardless of the/nrswitch or project settings.CommunicationsUtilities.cs:657— node-reuse eligibility check reads this env var before a node will register itself for reuse.NodeProviderOutOfProcBase.cs:265-307— the out-of-proc node provider's attach/reuse path is skipped entirely when node reuse is disabled, so no shared node is ever registered or reattached to.With the var set, MSBuild never registers/reattaches to a shared out-of-proc
node, so a sibling job's cleanup can no longer kill a node this job depends
on.
What changed
.github/workflows/csharp-ci.yaml: addedMSBUILDDISABLENODEREUSE: '1'tothe
build-and-testjob'senv:(the only job in this file that runsdotnet/MSBuild and can land on the self-hostedhetzner-3/hetzner-4pool —
normalize,coverage-output,matrix-output,matrix-commentand
packare all pinned toubuntu-latest).scala-ci.yamlwas checked and left untouched: it's pure Scala/sbt, itnever shells out to
dotnet/MSBuild anywhere, so it isn't exposed to thisbug.
csharp-publish-public.yamlwas checked too: itspublishjob ishardcoded to
runs-on: ubuntu-latestand can never land on the sharedself-hosted pool, so it's out of scope for this fix.
CHANGELOG.md: added the[Unreleased]entry.Background
This is the follow-up half of a fix already shipped in the caller repo.
peacefulstudio/daml-codegen-csharp-internalhit this as issue#1086,
and PR #1312
fixed it for that repo's own inline CI jobs. But that repo's CI also calls
this repo's
csharp-ci.yaml@v2andscala-ci.yaml@v2as reusableworkflows — and env vars set by a
workflow_callcaller do not propagateinto the called workflow's own job
env:, so the #1312 fix never reachedthese two reusable workflows. This PR applies the same fix inside
github-actionsitself.Reaching consumers
This only takes effect for consumers once the
v2tag moves to a releasethat includes this commit — merging to
devalone does not change what@v2resolves to. I'm not moving the tag or cutting a release myself; perthis repo's
CONTRIBUTING.md#release-process, that's the maintainer's stepafter merge (
git tag -a v2.X.Y ... && git tag -fa v2 ... && git push origin v2.X.Y v2).