Deflake CI: shared-timeout sizing, plus a runner-handshake diagnosis - #2019
Merged
Conversation
SubAgentActorTests.Approve_once_does_not_leak_between_subagent_tool_calls used a 5-second literal on its outer Ask. Every sibling approval test in the file uses the class constant ApprovalAskTimeout, which is 30 seconds. This test drives two sequential approval round-trips. It makes more hops than any sibling, so it is the least able to tolerate a 5-second budget. A starved Windows CI runner made it flake. The literal now uses the class constant. The Ask keeps an explicit timeout, because Ask falls back to akka.actor.ask-timeout, which is infinite by default. Deletion would trade a flaky wait for a hang.
ErrorCorrelationTests kept four 5-second literals on ExpectMsgAsync<ErrorOutput>. Each one waits for a full provider-failure turn: actor spawn, Akka.Persistence recovery, a failed stream, and the error classification. A starved CI runner can exceed 5 seconds on that path. The literal measures scheduler load, not correctness. This commit adds the same Config-property raise that PR #2014 applies to this class, and drops the four literals so they inherit it. LlmSessionTestBase seals ConfigureAkka, so the Config property is the only seam. The comment text matches #2014 word for word, so the two changes agree where they overlap. The Ask literals in this file stay explicit. Ask falls back to akka.actor.ask-timeout, which is infinite by default. Deletion would trade a flaky wait for a hang. Production code is untouched. A green test pays no added time. The raise only slows failure reporting.
Two shared test helpers held 5-second budgets. Each one bounds a multi-hop actor operation, not a correctness property. A short budget on a shared helper makes a whole group of tests flake at once. ShellApprovalHarness stops the approval actor twice: once to prove a persistent grant survives a restart, and once on dispose. Both stops wait for a persistence flush and an actor teardown. Every shell-approval test runs through this harness. ApprovalRehydrationTests resolves the session child at three sites, one of them the shared ColdRespawnAsync helper. The resolve waits for the actor spawn and the Akka.Persistence recovery. About twenty cold-respawn tests depend on it. Each budget moves to 15 seconds and carries a comment that states what it bounds. Production code is untouched. A green test pays no added time. The raise only slows failure reporting.
Aaronontheweb
enabled auto-merge (squash)
August 19, 2026 20:10
Aaronontheweb
disabled auto-merge
August 19, 2026 20:54
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates the remaining CI deflake items into one PR, in the shape of #2001.
Doctrine
Every raise here is environment sizing, not a correctness change. A short
budget on a multi-hop actor operation measures scheduler load on a starved
CI runner. It does not measure the property under test. Production code is
untouched, and a green test pays no added time — the raise only slows
failure reporting.
Ask literals stay explicit.
Askfalls back toakka.actor.ask-timeout,which is infinite by default. Deletion of an Ask timeout would trade a
flaky wait for a hang.
1. SubAgent Ask literal inconsistency
SubAgentActorTests.Approve_once_does_not_leak_between_subagent_tool_callsused a 5-second literal on its outer
Ask. Every sibling approval test inthe file uses the class constant
ApprovalAskTimeout, which is 30 seconds.This test drives two sequential approval round-trips. It makes more hops
than any sibling, so it tolerates a 5-second budget least. It flaked on
Windows CI. The literal now uses the class constant.
2. ErrorCorrelation leftovers
ErrorCorrelationTestsheld four 5-second literals onExpectMsgAsync<ErrorOutput>. Each waits for a full provider-failure turn:actor spawn, Akka.Persistence recovery, a failed stream, and the error
classification.
This commit adds the same
Config-property raise that #2014 applies to thisclass, then drops the four literals so they inherit it.
LlmSessionTestBasesealsConfigureAkka, so theConfigproperty is theonly seam. The comment text matches #2014 word for word.
Merge note for #2014: the
Configproperty is byte-identical on bothbranches, so it merges clean. Two hunks in
Each_error_turn_gets_a_distinct_CorrelationIdwill conflict, because thisPR edits the
ErrorOutputline and #2014 edits theTurnCompletedlinedirectly under it. No line separates them, so git cannot merge the pair.
The resolution is the union: keep both deletions. This conflict is
unavoidable in either order of merge.
3. Shared-infrastructure raises
Two shared helpers held 5-second budgets. A short budget on a shared helper
makes a whole group of tests flake at once.
ShellApprovalHarnessstops the approval actor twice — once to prove apersistent grant survives a restart, once on dispose. Both stops wait for
a persistence flush and an actor teardown. Every shell-approval test runs
through this harness. Both budgets move to 15 seconds.
ApprovalRehydrationTestsresolves the session child at three sites, oneof them the shared
ColdRespawnAsynchelper. The resolve waits for theactor spawn and the Akka.Persistence recovery. About twenty cold-respawn
tests depend on it. All three budgets move to 15 seconds.
Each site carries a comment that names what the budget bounds.
4. Demo.AppHost xUnit v3 handshake corruption — diagnosed, not fixed
Incident:
The mechanism is now proven from the runner code, and it is not ours:
-assemblyInfo(
TestProcessLauncherAdapter.GetAssemblyInfo, xunit.runner.visualstudio3.1.5). It then reads all of stdout and parses it as one JSON object.
ConsoleRunner.PrintAssemblyInfowrites that one JSON object and returns.ConsoleRunner.Run(xunit.v3.runner.inproc.console 3.2.2) queues awatchdog. One second after the entry point returns, if the process still
lives, it writes to stdout:
Console.WriteLine("Waiting 10 seconds for foreground threads to exit...").The plain-text branch runs because
-assemblyInfois not automated mode;the automated branch would emit a JSON diagnostic instead.
with
non-object.No Netclaw code writes to stdout on this path. The only
Console.WriteLinein the suite sits inside the opt-in test body, which is skipped without
NETCLAW_RUN_DEMO_SMOKE=1. The assembly has one module initializer, and itonly sets an environment variable. A local
-assemblyInfoprobe on Linuxreturns clean JSON and exits in 72 ms. The trigger is process-exit latency
above one second on the Windows runner.
No fix is applied, because the minimal fix is not in our code. Two options
for a follow-up, both maintainer calls:
samples/Netclaw.Demo.AppHost.IntegrationTestsfrom thepr_validationsweep. Every test in it is opt-in and skips without Dockerand
NETCLAW_RUN_DEMO_SMOKE=1, so the sweep buys no coverage and pays afragile process probe.
demo_smoke.ymlalready runs the suite on its own.-assemblyInfoprobeshould not write plain text to a stdout channel it defines as JSON.
Related but separate:
skunkworks/json-stdout-hygienecovers the NetclawCLI
--jsonenvelope, not the xUnit runner handshake. It does not applyhere.
Gates
dotnet build --nologo -v q— 0 warnings, 0 errorsSubAgentActorTests— 66 passedErrorCorrelationTests— 3 passedApprovalRehydrationTests— 20 passedShellPolicyEvidenceFixtureTests+ShellApprovalDispositionMatrixTests— 332 passeddotnet slopwatch analyze— 0 issues./scripts/Add-FileHeaders.ps1 -Verify— all files have headers