Skip to content

Make stalled companion-build retry tests deterministic under host load - #753

Merged
shiny-code-app[bot] merged 3 commits into
mainfrom
work/736-deterministic-watchdog
Oct 3, 2026
Merged

shiny-code-app[bot] merged 3 commits into
mainfrom
work/736-deterministic-watchdog

Conversation

@shiny-code-app

@shiny-code-app shiny-code-app Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Host load could consume the stalled-build fixture's one-second silence budget before its fake retry completed, failing otherwise passing CI. The fixture now injects a virtual Bash clock and sleeps into an isolated copy of the production script, with FIFO events synchronizing each fake build. The production watchdog, retry policy, workflows and installed runtime are unchanged.

Coverage exercises stall → success, compiler error without retry, a second stall, and a compiler error on retry. An unrelated blocked process must survive; leaked blocked build descendants are released and fail the fixture. Events precede terminal output so watchdog termination cannot prevent their delivery.

Validation:

  • All 115 release-workflow tests pass using uv run --no-project --with cryptography python -m unittest Tests.ScriptsTests.test_release_workflows -q.
  • 48 watchdog scenarios pass across six concurrent workers after the final fixes. A deliberate descendant-discovery mutation is rejected and its leaked fixture child released.
  • Full scripts/commit-gate.sh passed: Python lanes, Swift build and 1,203 Swift tests.
  • Changed-files JetBrains inspection: UNKNOWN, timed out waiting for another inspection's lifecycle lock; no native inspection ran. The CP-736-D1 brief permits recording an IDE UNKNOWN and continuing.
  • Docs and .github/github.json checked; no validation command, policy or product behavior changed.

Independent review: three read-only Anthropic claude-opus-5-5 passes. Fixed terminal-event ordering and descendant leak detection. Final pass found no blocking defect. Remaining Low suggestions are declined with bounded reasons: real termination sleeps violate the no-real-time test rule (the fixture detects actual leaked readers, but does not qualify production grace-period timing); exact exit 65 is not the production contract, whose failed-marker path returns 1 while its completed pipeline can return 65 (this host's Bash preserves 65 across repeated wait calls); source-spelling assertions are explicitly forbidden by AGENTS.md. No grace-period duration or hypothetical alternate clock implementation coverage is claimed.

Refs #736. Trial CP-736-D1; admitted under cbusillo/direction's spare-capacity rule recorded in the claim. No release dispatch, signing, entitlement, access or production change.

Integrated head a98d600 includes latest main 7d17469; the author test file is byte-unchanged by the bot base update. A fresh Anthropic claude-opus-5-5 review read that integrated head and reports no blocking defect. Its Low limits are retained: production TERM/KILL grace timing is not qualified by virtual sleeps; a rare signal-settling race at the FIFO cleanup probe is unconfirmed (not reproduced in 48 concurrent final scenarios or prior-head green CI, not claimed refuted); cleanup failure can mask a timeout diagnostic while still releasing the child and failing; future alternative clock syntax is outside this patch. Real-sleep/source-text remedies conflict with AGENTS.md. Current failure decisions and bounded retry are executable coverage; exact failure status is not the contract. Fresh integrated-head CI is pending; prior-head 4/4 green checks do not qualify it.

@shiny-code-app

shiny-code-app Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

CI attempt 1 failed in run 37143506404: Run routine Python test lane exceeded its existing five-minute step deadline while executing the unchanged Tests/ScriptsTests/test_companion_upload_script.py. The changed release-workflow suite was not reached in that lane; its final local run passed all 115 tests. This is an unrelated suite/runner timing failure, not evidence that the deterministic watchdog test failed. CodeQL/Analyze Swift passed. Preserve this failed attempt; once cleanup/checks are terminal and the watcher authorizes it, retry the failed CI job within the three-retry budget. No test, timeout, CI policy or production script will be changed to bypass this failure. Trial CP-736-D1.

@shiny-code-app

shiny-code-app Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

The unchanged companion-upload suite passes locally: 19 tests in 35.371 seconds, using fixture-only export tooling. The failed CI attempt timed out after five minutes at that file; the changed watchdog suite had not run. Treating this as unrelated runner/scheduling timing for one failed-job rerun, once the current head's cleanup/check evidence is terminal. No CI timeout, unrelated test or release workflow changed.

@shiny-code-app
shiny-code-app Bot merged commit aa24693 into main Oct 3, 2026
4 checks passed
@shiny-code-app
shiny-code-app Bot deleted the work/736-deterministic-watchdog branch October 3, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant