Skip to content

test: check every simulation suite runs in exactly one group - #550

Merged
MarcusKainth merged 1 commit into
mainfrom
test/group-coverage
Sep 18, 2026
Merged

MarcusKainth merged 1 commit into
mainfrom
test/group-coverage

Conversation

@MarcusKainth

Copy link
Copy Markdown
Owner

What this changes, and why

scripts/test-group.sh packs the simulation suites across the lettered
groups by name, and gives one group everything the others do not name.
Nothing checked either half of that. A suite that no group names, on a day
when no group takes the rest, would run nowhere and the matrix would stay
green without it. A suite two groups name runs twice and costs one of them
its budget. Neither shows up in a run's own output: a group prints the tests
it ran, not the ones nobody gave it.

The test reads the script's own sim_<letter>= lines and the
native/tests/sim_*_live.rs files on disk, and fails when a suite is named
by two groups, when a group names a suite that is not there, or when no case
takes the rest. It parses the binary() terms rather than evaluating a
filterset, so a filter that stops being a plain list of them fails here
rather than being read wrong.

It needs no server, so it runs in native-rest with the other suites
outside the simulation.

The hole this guards is currently closed, and I checked before writing.
The report was that #547 removed the catch-all. It did not: native-sim-e
took it over from native-sim-b. I put a throwaway sim_zzz_probe_live.rs
in the tree and listed it through each group's own filter:

native-sim-a selects the new suite: 0
native-sim-b selects the new suite: 0
native-sim-c selects the new suite: 0
native-sim-d selects the new suite: 0
native-sim-f selects the new suite: 0
native-sim-e (catch-all) selects the new suite: 1
native-rest selects the new suite: 0

So this is a guard on a property that holds today rather than a fix for a
break. It is worth having anyway: nothing enforced that the catch-all stays,
the packing in #547 was checked by hand once and never again, and the same
listing run by hand is what the test now does on every run.

Evidence

$ cargo test -p clickdoom-native --test group_coverage
test every_simulation_suite_runs_in_exactly_one_group ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
exit=0

Biting, one failure mode at a time, each by editing the script and putting
it back:

$ # sim_compact_live added to sim_b as well as sim_a
named by more than one group: ["sim_compact_live in ['a', 'b']"]

$ # native-sim-e's filter changed from the catch-all to ($sim_e)
no group takes the suites the others do not name, so a new suite would run nowhere

$ # sim_gone_live added to sim_a
sim_a names sim_gone_live, which is not on disk

Invariants

None. This adds a test that reads two files off disk.

Spec impact

  • None. No contract in SPEC.md is touched

Checks

  • make gates. Not run whole: this adds one test and a sentence to
    DEVELOPING.md. The test itself is above, green and shown failing three
    ways.
  • No AI attribution trailers in the commits

Anything else

One thing this does not cover, found while checking the above and worth its
own change rather than being bundled here. .config/nextest.toml cuts a
test off at 1,800 s and its comment calls that "about 1.7x the slowest test
measured on a CI runner (sim_pain_live, 1,073 s)". On the five runs #547 was
packed from, the slowest test is sim_missile_live's at 1,234 s median and
1,347 s at its worst, so the cutoff is 1.34x the worst rather than 1.7x, and
a test's own time moves 35% at the extreme. A suite that grows past the
cutoff is killed rather than reported slow, and the failure reads as a
timeout rather than as anything about the test.

Written mostly by Claude Opus 5.

@github-actions github-actions Bot added area: docs The prose: READMEs, ADRs, and the contributor documents area: native Native mode: the tic simulation and renderer as SQL, and the WAD loader labels Sep 18, 2026
`scripts/test-group.sh` packs the simulation suites across the lettered
groups by name and gives one group everything the others do not name.
Nothing checked that. A suite named by no group and reaching no catch-all
would run nowhere and the matrix would stay green without it; a suite named
by two would run twice and cost a group its budget. Neither shows up in a
run's own output.

The test reads the script's `sim_<letter>=` lines and the suites on disk. It
fails when a suite is named twice, when a group names a suite that is not
there, and when no case takes the rest. It parses the `binary()` terms
rather than evaluating a filterset, so a filter that stops being a plain
list of them fails here rather than being read wrong.

It needs no server, so it runs in native-rest with the other suites outside
the simulation.
@MarcusKainth
MarcusKainth merged commit 219a367 into main Sep 18, 2026
19 checks passed
@MarcusKainth
MarcusKainth deleted the test/group-coverage branch September 18, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: docs The prose: READMEs, ADRs, and the contributor documents area: native Native mode: the tic simulation and renderer as SQL, and the WAD loader

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant