test(windows): harden the release-verification harness and pin its contracts - #3327
test(windows): harden the release-verification harness and pin its contracts#3327liugddx wants to merge 3 commits into
Conversation
…ntracts Split out of apache#3265 at its reviewer's request so these fixes merge on their own evidence and that PR stands on the installer transaction alone. Contents: - Read the CDP port from the DevTools stderr announcement (waitForDevToolsPort) instead of pre-reserving one; widen renderer discovery to 90s with per-probe AbortSignal bounds and errno cause chains (four observed CI failures in this family). - waitForUsableRenderer: poll the renderer-usable state with the deadline as the sole authority — one stalled Runtime.evaluate used to fail the whole gate (run 32352924376); the WebSocket handshake now has its own bound so a port that accepts but never speaks fails the probe, not the lane. - Tolerate taskkill exit 128 when the relaunched instance already exited; the authoritative assertion remains waitForInstalledProcessesToExit. - Match the versioned uninstall DisplayName ('Maka 0.1.11'): the -eq 'Maka' filter matched nothing, deterministically, and every reader of the scan was blind. - Bound every PowerShell probe that runs under a polling deadline (the anti-pattern apache#3241 names), and let waits tolerate one failed probe: a failed enumeration is never treated as 'no processes'. - waitForUninstallRegistrationToClear: a detached uninstaller deletes its registry keys after waitUntilMissing sees the files disappear; wait for the registration to clear before the next install, with the one-registry-call residual window stated precisely. - directoryTreeManifest/diffTreeManifests shared exports for the rollback gate, now recording empty directories so their loss is visible; capture upgrade-state evidence on a relaunch version mismatch. - Commit the table-driven contract tests as scripts/verify-windows-harness.test.mjs and wire them into the CI planner test step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Generated-by: Claude Fable 5
The upgraded-app smoke pipes stderr but only waitForDevToolsPort's temporary listener ever read it: once removed, the paused stream lets Chromium's --enable-logging=stderr output fill the pipe and block the child, and the evidence the pipe exists to preserve is lost. Attach the same persistent collector every sibling smoke uses and append its tail to renderer-readiness failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Generated-by: Claude Fable 5
e9b64be to
8a7f9e6
Compare
…aunch Run 32378497920: taskkill /T /F on the force-run instance exceeded its 30s bound on a wedged runner and failed the gate, even though the authoritative assertion - waitForInstalledProcessesToExit, which fails with the live process list if anything from the install tree still runs - was one line below. Treat a kill that overran its bound like exit 128: the kill is the mechanism, the exit wait is the assertion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Generated-by: Claude Fable 5
|
All lanes are green at head d6f443f: the unified CI job (harness contract tests included, running pre-install as pinned by #3317's step contract) and the full |
|
@Astro-Han @M4n5ter PTAL |
Run 32392391944 failed at the packaged renderer smoke with the 30-second CDP deadline and no app stderr - the exact flake family apache#3327 fixes; this branch is based on main, which predates those harness repairs.
hqhq1025
left a comment
There was a problem hiding this comment.
[P1] scripts/verify-windows-installer-lifecycle.mjs:47 — Retry WMI failures inside the relaunch deadline. This new 30-second bound correctly turns a wedged Get-CimInstance into a rejection, but scripts/verify-windows-autoupdate.mjs:410 directly awaits listInstalledProcesses() inside a 120-second relaunch loop. Run 32340493254 failed on exactly this path after one transient WMI timeout even though the upgraded process may already have relaunched. Catch failed rounds, continue until the outer deadline, and report the last probe error only if no later enumeration succeeds; use a shorter or remaining-budget probe timeout and add a failure-then-success regression.
[P2] scripts/verify-windows-autoupdate.mjs:492 — Reuse this termination policy in cleanup. The main path now treats exit 128 and timed-out taskkill as mechanism failures before relying on waitForInstalledProcessesToExit, but the finally block at lines 607-613 still calls taskkill directly. Run 32378497920 recorded the same stale-PID exit 128 from cleanup. After a later smoke or verification failure, that exception skips the authoritative exit wait and the uninstaller, potentially leaving a stale uninstall registration on the runner. Extract one stop-and-verify helper for both paths and complete the uninstall/registration barrier during cleanup.
Codex-assisted review performed under the maintainer-approved review workflow.
|
A data point for the bounded-probe half of this PR, from #3225's lane.
That last one is the shape you describe in the PR body: with no bound the step has no upper limit at all, so the cost is a wedged runner rather than a fast red. Good to see Reference: https://github.com/apache/maka/actions/runs/32387807797 (job 96534477439). Nothing needed from you — just adding evidence in case it's useful for the review. |
Summary
Windows release-verification harness hardening, split out of #3265 at its reviewer's request ("land the CI-stability fixes first, as their own PR") so these fixes merge on their own evidence and #3265 stands or falls on the installer transaction alone. All changes are under
scripts/plus one workflow test-step line; no packaged-product code is touched.What's fixed
CDP attach and renderer readiness (four observed failures in this family across #3161/#3240/#3265 lanes):
waitForDevToolsPortreads the port from the DevTools stderr announcement (--remote-debugging-port=0) instead of pre-reserving a port that Electron may race.AbortSignalbounds (a hung connect once turned a "90 seconds" failure into ~6 minutes), errno cause chains in the failure message.waitForUsableRenderer: polls the renderer-usable state with the deadline as the sole authority — a single stalledRuntime.evaluateused to fail the whole gate (run 32352924376,CDP renderer evaluation timed out.while the renderer was still booting). The WebSocket handshake now has its own bound, so a port that accepts TCP but never speaks fails one probe instead of hanging the lane (feat(win): Abort-path installer rollback with backup retention #3265 review finding 3821101776).Uninstall registration reads (the cause of the deterministic red in run 32363159920):
DisplayName -eq 'Maka', but electron-builder's defaultuninstallDisplayNameis"${productName} ${version}"(NsisTarget.js:473), so the registered name isMaka 0.1.11and the filter matched nothing — every reader was blind. Now matches the versioned name.waitForUninstallRegistrationToClear: an NSIS uninstaller launched without_?=detaches and deletes its registry keys tens of seconds after the files disappear; anything installing into that window gets its fresh registration deleted. The wait's comment states the residual window precisely (one registry call wide — the uninstall key at uninstaller.nsh:250, the install key at :254; finding 3820922176).Bounded probes under polling deadlines (findings 3820232602 / 3820922173 / 3821101780 — the anti-pattern #3241 names; kept as explicit per-site bounds rather than a
runCommanddefault, per the accepted argument that codesign/notarization have no honest upper bound):timeoutMs, including the evidence capture that runs on an already-wedged machine.Evidence and coverage (finding 3819778535):
directoryTreeManifest/diffTreeManifests(shared exports the feat(win): Abort-path installer rollback with backup retention #3265 gate consumes) now record empty directories so their loss is visible.scripts/verify-windows-harness.test.mjs(manifest shape/sorting/empty-dir/symlink-fail-closed, diff table, runCommand timeout kill, DevTools port announcement, usable-renderer deadline semantics, both waits' tolerate-and-deadline contracts) and run in the CI planner test step on every PR — 19 pass locally on Windows (1 skip: symlink creation needs privilege), and the step runs them on Linux too.Verification
node --checkon all touched files; biome clean;verify-windows-harness.test.mjs19 pass / 1 skip;ci-test-plan.test.mjs17 pass (step-name pin updated to the renamed step, contract unchanged: planner tests still run before dependency setup, unconditionally).testlane runs both test files.Relates to #3265 (review findings 3819778548, 3820232602, 3820922173, 3820922176, 3821101776, 3821101780, 3819778535) and #2142.