test: check every simulation suite runs in exactly one group - #550
Merged
Merged
Conversation
MarcusKainth
force-pushed
the
test/group-coverage
branch
from
September 18, 2026 06:41
6b07729 to
9912f00
Compare
4 tasks
`scripts/test-group.sh` packs the simulation suites across the lettered groups by name and gives one group everything the others do not name. Nothing checked that. A suite named by no group and reaching no catch-all would run nowhere and the matrix would stay green without it; a suite named by two would run twice and cost a group its budget. Neither shows up in a run's own output. The test reads the script's `sim_<letter>=` lines and the suites on disk. It fails when a suite is named twice, when a group names a suite that is not there, and when no case takes the rest. It parses the `binary()` terms rather than evaluating a filterset, so a filter that stops being a plain list of them fails here rather than being read wrong. It needs no server, so it runs in native-rest with the other suites outside the simulation.
MarcusKainth
force-pushed
the
test/group-coverage
branch
from
September 18, 2026 08:10
9912f00 to
592136b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
scripts/test-group.shpacks the simulation suites across the letteredgroups by name, and gives one group everything the others do not name.
Nothing checked either half of that. A suite that no group names, on a day
when no group takes the rest, would run nowhere and the matrix would stay
green without it. A suite two groups name runs twice and costs one of them
its budget. Neither shows up in a run's own output: a group prints the tests
it ran, not the ones nobody gave it.
The test reads the script's own
sim_<letter>=lines and thenative/tests/sim_*_live.rsfiles on disk, and fails when a suite is namedby two groups, when a group names a suite that is not there, or when no case
takes the rest. It parses the
binary()terms rather than evaluating afilterset, so a filter that stops being a plain list of them fails here
rather than being read wrong.
It needs no server, so it runs in
native-restwith the other suitesoutside the simulation.
The hole this guards is currently closed, and I checked before writing.
The report was that #547 removed the catch-all. It did not:
native-sim-etook it over from
native-sim-b. I put a throwawaysim_zzz_probe_live.rsin the tree and listed it through each group's own filter:
So this is a guard on a property that holds today rather than a fix for a
break. It is worth having anyway: nothing enforced that the catch-all stays,
the packing in #547 was checked by hand once and never again, and the same
listing run by hand is what the test now does on every run.
Evidence
Biting, one failure mode at a time, each by editing the script and putting
it back:
Invariants
None. This adds a test that reads two files off disk.
Spec impact
Checks
make gates. Not run whole: this adds one test and a sentence toDEVELOPING.md. The test itself is above, green and shown failing threeways.
Anything else
One thing this does not cover, found while checking the above and worth its
own change rather than being bundled here.
.config/nextest.tomlcuts atest off at 1,800 s and its comment calls that "about 1.7x the slowest test
measured on a CI runner (sim_pain_live, 1,073 s)". On the five runs #547 was
packed from, the slowest test is
sim_missile_live's at 1,234 s median and1,347 s at its worst, so the cutoff is 1.34x the worst rather than 1.7x, and
a test's own time moves 35% at the extreme. A suite that grows past the
cutoff is killed rather than reported slow, and the failure reads as a
timeout rather than as anything about the test.
Written mostly by Claude Opus 5.