Skip to content

test(tools): a wire assertion must be its own wait - #1082

Merged
andrewgazelka merged 1 commit into
mainfrom
test/wire-assertions-are-their-own-wait
Jul 30, 2026
Merged

andrewgazelka merged 1 commit into
mainfrom
test/wire-assertions-are-their-own-wait

Conversation

@andrewgazelka

@andrewgazelka andrewgazelka commented Jul 30, 2026

Copy link
Copy Markdown
Member

smash-selector-e2e failed in CI with exactly one assertion:

FAILED S1 picked the Skeleton and the action bar never said so: []

The server had sent that line. Run 30500640846 recorded the failure at
00:21:41.7509285 and logged the line it was waiting for at
00:21:41.7511018:

00:21:41.7508758  15.94s [S1 ] <- chat: Kit set to Skeleton.
00:21:41.7509285  15.94s      FAILED S1 picked the Skeleton and the action bar never said so: []
00:21:41.7511018  15.94s [S1 ] <- action bar: You are the Skeleton.

The check waited on client.kit, which is set from the chat packet
lobby::choose sends one ahead of the action bar, and drain takes a single
recv per pump, so a burst split across two reads satisfies the wait with the
asserted-on packet still in flight. Production code was not at fault.

That fix landed on its own in #1083. This is the rule that stops it recurring.

What this adds

nix/verify-wire-assertions.py, wired in as a flake check so the subtractive
gate in nix/ci/flake-gate.nix picks it up without anybody adding it anywhere.
The rule: an assertion about a field the packet handler fills must be preceded,
in the same function, by a wait whose predicate reads that same field.
must_become and must_not_become make the predicate the assertion, so waiting
on one thing and asserting on another is not expressible through them.

The rule is deliberately narrow. It only looks at functions that wait at all,
because the bug is not "forgot to wait", it is "waited, and for the wrong
packet". The observed field set is derived from what the packet handlers fill
rather than listed, so a new handler brings its field under the rule the day it
lands. Two limits are documented in the script: it does not follow a field
through a helper call, and it says nothing about functions that never wait.

What it flags today, and what I did with it

Three sites, which is what the rule reports on this tree rather than a set I
picked. I deliberately did not hand-convert the other 64 assertions: that diff
is mechanical, and a conversion with a subtly wrong predicate turns a real
assertion into one that cannot fail while the gate stays green.

site verdict
a_click_picks_the_mob the shipped bug, now must_become
the mid-match refusal real: slept settle(1.0) where a second happened to be generous. Now must_not_become for the kit and must_become for the refusal
the lobby-count assertion false positive. Takes the escape hatch, with a reason

The escape hatch is # not-a-wire-assertion: <why> and the reason is part of
the syntax: a marker with nothing after the colon is itself an error, because a
bare marker is a mute button.

Break-it results

The lint was broken three ways and watched, through the real flake check. Three
cases and not one, because a rule that rejects everything passes the
reject-the-bad-thing test perfectly: the first version of this script flagged 67
of 67 assertion sites.

  • rejects the shape that shipped: smash-selector.py:1191: in a_click_picks_the_mob, this fails on action_bar, which the packet handler fills, and no wait before it reads it.
  • accepts a correct must_become: 14 client scripts checked, 0 problems
  • rejects a marker with no reason: smash-selector.py:1349: not-a-wire-assertion with no reason after the colon. The reason is the point: say which non-wire thing this asserts.

Because two live assertions changed, the gate they live in was re-run rather
than trusted: nix build .#checks.x86_64-linux.smash-selector-e2e, rc=0, with
both converted claims passing.

Enforcement, stated plainly

The check runs in CI automatically via the Flake job and nobody has to
remember it. It does not block a merge: ruleset 566717 on main carries no
required_status_checks rule, so no check in this repository is a required
context and a red gate is enforced by somebody noticing. ENG-10827.

@andrewgazelka
andrewgazelka enabled auto-merge July 30, 2026 02:42
@github-actions github-actions Bot added the test label Jul 30, 2026
@github-actions

Copy link
Copy Markdown

Benchmark Results for general

ray_intersection/aabb_size_0.1                     [  18.7 ns ...  18.8 ns ]      +0.06%
ray_intersection/aabb_size_1                       [  18.6 ns ...  18.6 ns ]      -0.03%
ray_intersection/aabb_size_10                      [  18.7 ns ...  18.6 ns ]      -0.19%
ray_intersection/ray_distance_1                    [   1.3 ns ...   1.3 ns ]      -0.09%
ray_intersection/ray_distance_5                    [   1.3 ns ...   1.3 ns ]      +0.12%
ray_intersection/ray_distance_20                   [   1.3 ns ...   1.3 ns ]      +0.19%
overlap/no_overlap                                 [  15.4 ns ...  15.6 ns ]      +0.75%
overlap/partial_overlap                            [  15.4 ns ...  15.5 ns ]      +0.33%
overlap/full_containment                           [  14.5 ns ...  14.5 ns ]      +0.12%
point_containment/inside                           [   5.4 ns ...   5.5 ns ]      +0.53%
point_containment/outside                          [   5.7 ns ...   5.7 ns ]      -0.12%
point_containment/boundary                         [   5.4 ns ...   5.4 ns ]      +0.17%

Comparing to dd8311f

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.65%. Comparing base (b3870e4) to head (afa2fee).
⚠️ Report is 1 commits behind head on main.

@@           Coverage Diff           @@
##             main    #1082   +/-   ##
=======================================
  Coverage   54.65%   54.65%           
=======================================
  Files         361      361           
  Lines       33257    33257           
  Branches     1259     1259           
=======================================
  Hits        18178    18178           
  Misses      14793    14793           
  Partials      286      286           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@andrewgazelka
andrewgazelka force-pushed the test/wire-assertions-are-their-own-wait branch from fddc5ee to 04030cf Compare July 30, 2026 02:49
@github-actions

Copy link
Copy Markdown

Benchmark Results for general

ray_intersection/aabb_size_0.1                     [  17.2 ns ...  17.2 ns ]      +0.05%
ray_intersection/aabb_size_1                       [  17.2 ns ...  17.2 ns ]      +0.12%
ray_intersection/aabb_size_10                      [  17.2 ns ...  17.2 ns ]      -0.03%
ray_intersection/ray_distance_1                    [   1.5 ns ...   1.5 ns ]      +0.17%
ray_intersection/ray_distance_5                    [   1.5 ns ...   1.5 ns ]      -0.24%
ray_intersection/ray_distance_20                   [   1.5 ns ...   1.5 ns ]      -0.01%
overlap/no_overlap                                 [  15.6 ns ...  15.6 ns ]      -0.20%
overlap/partial_overlap                            [  15.6 ns ...  15.6 ns ]      +0.19%
overlap/full_containment                           [  14.8 ns ...  14.9 ns ]      +0.37%
point_containment/inside                           [   6.2 ns ...   6.2 ns ]      +0.32%
point_containment/outside                          [   6.1 ns ...   6.1 ns ]      +0.29%
point_containment/boundary                         [   7.0 ns ...   6.9 ns ]      -0.33%

Comparing to 34178ea

andrewgazelka added a commit that referenced this pull request Jul 30, 2026
…1083)

`smash-selector-e2e` fails intermittently with

    FAILED S1 picked the Skeleton and the action bar never said so: []

and the line it says never arrived is in the same log **0.2 ms later**.

The server is not at fault. `lobby::choose` sends two packets, a chat
line
and then a separate `Channel::ActionBar` line. The check waited on
`client.kit`, which `on_chat` sets from the *first* of the two, then
read
the action bar with no further wait. The harness reads one packet per
pump,
so a burst split across two reads leaves the second packet still on the
wire
when the assertion runs. Draining harder would not help; the bytes had
genuinely not arrived.

Every other action bar assertion in the suite either settles first or
waits
on the action bar itself. This was the only one keying its wait on a
different packet that merely precedes the one it asserts on.

Proved in **both** directions on a Linux builder, because the unfixed
tree
also passes on a good run and one green run here proves nothing:

  fixed      PASS ... the server answered ['You are the Skeleton.']
negative with the `Channel::ActionBar` send deleted from
`lobby::choose`,
             rc=1, failing after the full 10 s wait rather than passing
             vacuously

That the negative control was necessary rather than merely thorough is
now
measured: `smash-hud-e2e` and `smash-selector-e2e` each failed one CI
run
and passed the next on effectively the same tree. Two of five failures
in
that suite are coin flips.

Split out of #1082 deliberately. That PR carries this fix plus a 322
line
static analyzer that stops the whole class being written again, and the
analyzer deserves an unhurried review. This half is costing a 35 to 90
minute CI run every time it fires and reads as a server bug to whoever
sees
it next, so it should not wait on the discretionary half.

Diagnosed, fixed and controlled by a subagent; the negative control was
its
own idea. Split and landed by me.

(sent by an AI agent via Claude Code)
`smash-selector-e2e` failed in CI with "S1 picked the Skeleton and the
action bar never said so: []" while the server had sent that exact line.
Run 30500640846 recorded the failure at 00:21:41.7509285 and logged the
line it was waiting for at 00:21:41.7511018, a fifth of a millisecond
later. The check waited on `client.kit`, which is set from the chat
packet `lobby::choose` sends one ahead of the action bar, and `drain`
takes a single `recv` per pump, so a burst split across two reads
satisfies the wait with the asserted-on packet still in flight.

Sweeping for other instances finds today's and not tomorrow's, so this
adds the rule instead: an assertion about a field the packet handler
fills must be preceded, in the same function, by a wait whose predicate
reads that same field. `must_become` and `must_not_become` make the
predicate the assertion, so waiting on one thing and asserting on
another is not expressible through them.

The rule is deliberately narrow. It only looks at functions that wait at
all, because the bug is not "forgot to wait", it is "waited, and for the
wrong packet", and flagging every synchronous assertion is how a check
becomes noise and its escape hatch becomes a mute button. The observed
field set is derived from what the packet handlers fill rather than
listed, so a new handler brings its field under the rule the day it
lands.

Three sites, which is what the rule flags on this tree rather than a
hand-picked set. The selector's kit announcement and the mid-match
refusal are real: the first is the shipped bug, the second slept
`settle(1.0)` where a second happened to be generous. The lobby-count
assertion takes the escape hatch, which carries a required reason
because a bare marker is a mute button.

A single green run proves nothing about this class, which is why the fix
was verified against a negative control that deletes the announcement
rather than against one passing build. The unfixed tree passes on a good
run: measured directly, and confirmed across the suite the same night,
where `smash-hud-e2e` and `smash-selector-e2e` each failed one CI run
and passed the next on effectively the same tree. Two of five failures
in that suite are coin flips.

This converts "you can wait on the wrong packet" into "you can write the
wrong predicate", which is a real trade: the second is an ordinary bug,
and the first reads as a server defect and sends someone into the wrong
code for an afternoon.
@andrewgazelka
andrewgazelka force-pushed the test/wire-assertions-are-their-own-wait branch from 04030cf to afa2fee Compare July 30, 2026 02:55
@github-actions

Copy link
Copy Markdown

Benchmark Results for general

ray_intersection/aabb_size_0.1                     [  18.6 ns ...  18.6 ns ]      -0.04%
ray_intersection/aabb_size_1                       [  18.7 ns ...  18.7 ns ]      +0.03%
ray_intersection/aabb_size_10                      [  18.7 ns ...  18.7 ns ]      +0.15%
ray_intersection/ray_distance_1                    [   1.3 ns ...   1.3 ns ]      +0.07%
ray_intersection/ray_distance_5                    [   1.3 ns ...   1.3 ns ]      -0.24%
ray_intersection/ray_distance_20                   [   1.3 ns ...   1.3 ns ]      +0.42%
overlap/no_overlap                                 [  16.1 ns ...  16.1 ns ]      -0.13%
overlap/partial_overlap                            [  16.0 ns ...  16.0 ns ]      +0.01%
overlap/full_containment                           [  14.5 ns ...  14.4 ns ]      -0.14%
point_containment/inside                           [   5.4 ns ...   5.4 ns ]      +0.28%
point_containment/outside                          [   5.8 ns ...   5.8 ns ]      -0.10%
point_containment/boundary                         [   5.4 ns ...   5.4 ns ]      +0.02%

Comparing to b3870e4

@andrewgazelka
andrewgazelka merged commit ec8c9fd into main Jul 30, 2026
11 of 12 checks passed
@andrewgazelka
andrewgazelka deleted the test/wire-assertions-are-their-own-wait branch July 30, 2026 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant