Skip to content

perf: occupancy and light classify each block state once, and tests run two executables at a time - #1244

Merged
stellarfeline merged 10 commits into
mainfrom
fix/ci-time
Nov 1, 2026
Merged

perf: occupancy and light classify each block state once, and tests run two executables at a time#1244
stellarfeline merged 10 commits into
mainfrom
fix/ci-time

Conversation

@stellarfeline

@stellarfeline stellarfeline commented Oct 31, 2026

Copy link
Copy Markdown
Owner

What

  • Occupancy (assembled.rs, blockstate.rs, sculk.rs): each block state is classified once, keyed by its interned key, instead of once per cell. Cell sets are gathered sorted and built in one pass.
  • Light (light.rs): DarkSurvey::measure reads the dense light field directly, and LightField::over packs each state once.
  • CI configuration, its own commit (tools/ci/cargo-test-concurrent.py --jobs 2): every test executable is read from cargo's JSON listing and run whole, two at a time, then the doc tests. The run prints R run of L listed and fails on any mismatch. Tested in tools/tests/test_cargo_test_concurrent.py.

Measured (remote, 4 cores, alternated base/head runs; numbers in the commit bodies)

  • One gallery detail commit: 44.6 s → 29.5 s wall, 949G → 727G instructions.
  • Sequential cargo test: 3563 s → 2879 s. With the concurrent runner: 2360 s.
  • CI estimate for the rust job: about 71 min, down from 1 h 42 min on main. This PR's CI run is the real number.

Unchanged

  • Output is byte-identical: base vs head for the primary build (1569 files) and the site-plan point (378 files); head built twice also matches.
  • Binding lines: build, site-plan build and analyze diffs are empty.
  • Gallery coverage report: byte-identical.
  • No test ignored, filtered or reduced, and no seed changed.

🤖 Generated with Claude Code

stellarfeline and others added 10 commits October 31, 2026 05:06
@stellarfeline @claude
perf(occupancy): classify each block state once, not each cell
d7f0e2f 
`occupancy_over`, `climb_holds` and `sculk::Listening::of` asked the
`blockshape` and registry rules of every cell's state text: string
parsing per cell over a world of millions of cells and a few hundred
states. Each reading is a function of the state alone, so each is now
made once per distinct interned state (keyed by `BlockState::key`, the
interned text's identity) and looked up per cell. Nothing iterates in
key order; the sets built are the same.

Profile that found it (perf, frame pointers, one `detail` commit of the
gallery site-plan point, 8 cores): the commit's main thread is serial
for ~19 s of a ~34 s wall; `occupancy_over` is ~7 s of that, called
four times per battery (light::final_world, light::lit_world,
blockout::check, blockout::crossings); `sculk::Listening::of` ~1 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
perf(light): the dark survey reads the dense field, and a state packs…
9b54b04 
… once

`DarkSurvey::measure` copied the whole flooded field into a
`BTreeMap` (`LightModel::flood`) to read the light of the cells it
examines; it now reads the dense field (`LightField::light_at`), which
answers what the map's `unwrap_or(0)` answered: a lit cell's value, 0
for an unlit cell or one outside the box. `LightField::over` resolved a
cell's byte through a map keyed by the block text (a string comparison
per cell); it now keys by the text's address and length, so an
interned state is one key.

In the commit profile `relight_with` was ~4 s of the main thread, 70%
of it the map copy and the string-keyed lookups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
perf(reach): the flight memo finds a node's child by one probe
6e9a7af 
A flight-memo walk (`flights::lookup`) chose the child of each run by
walking the parent's sibling list, one load from anywhere in `nodes`
per sibling. With a fixed-period profile (`cpu-clock`, 3 ms, frame
pointers) of one gallery site-plan commit, the single compare in that
walk held 13.5% of all CPU samples, and `World::expand`, which inlines
the walk, 70% inclusive. Each shard now indexes its edges by
`(parent, code)`; the sibling lists stay as the record of a node's
children, which a split moves to the run's old rest. The walk answers
the same child: a parent never holds two children of one code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
revert: perf(reach): the flight memo finds a node's child by one probe
530a414 
Measured A/B on one gallery site-plan commit (8 cores, perf stat,
two alternated runs): the index raised user instructions from 762G to
866G and task-clock from ~80 s to ~122-136 s. The sibling lists are
short; the probe costs more than the walk it replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
perf(occupancy): the cell sets are built once from sorted cells
e8c00db 
`occupancy_over` walks the block map in ascending cell order and
inserted each cell into its `BTreeSet`s one search at a time; after
the per-state classification, those inserts were most of what was
left of it (fixed-period profile of one gallery site-plan commit). The
cells are now gathered in order and each set is collected in one pass;
the sets hold the same cells (a waterlogged fluid cell, gathered twice
as a source, is held once, as before).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
ci(rust): the test executables run two at a time
c94579a 
CI configuration, its own commit. `cargo test --workspace` runs the
workspace's test executables one after another; the long ones are
sequences of `delvec` runs whose single-threaded stretches leave most
of a 4-core runner idle. `tools/ci/cargo-test-concurrent.py` builds
every test target, reads the executables from cargo's own JSON
listing, runs each once and whole, two at a time (each to its end),
then the doc tests, and states `R run of L listed, F failed`.

What it bought, measured on its own: remote build machine, tree
530a414 (the engine changes before this commit), 4 cores
(`taskset -c 8-11`), already built, `/usr/bin/time` over the whole
step:
  cargo test --locked --workspace --no-fail-fast
    wall 3606.8 s, user 9276.0 s, sys 500.4 s (2.71 cores busy)
  cargo-test-concurrent.py --jobs 2 --locked
    wall 2913.6 s, user 9960.7 s, sys 491.5 s (3.59 cores busy)
  -19.2% wall; 373 of 373 executables run, 0 failed; 376 `test result`
  lines in both logs. Other rounds shared the machine during both runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
style(light): rustfmt
3f1ae40 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
chore(perf): the round's measurements, before and after
aa9cf77 
No code. The numbers the commits above were judged by.

Instrument: the remote build machine (16 cores, WSL2, other rounds
sharing it throughout), dev profile without debug info, `taskset`
for a fixed core set; `cargo test`'s own `finished in` per binary as
the first method, `/usr/bin/time` over the whole step and `perf stat`
(instructions:u, task-clock) as the second.

Before, `main` @ 1f068cd:
- CI (run 40837884057, 4 vCPU): test step 1 h 39 min = 10 min build +
  88.5 min of binaries (sum of 376 `finished in`, 5310 s); top:
  starter_program 771 s, detail_check_agrees 588, unittests 308,
  detail_verb 289, remedy_reachability 248, place_reads 231.
- remote, 8 cores: sum 2764 s, step 2784 s.
- remote, 4 cores: sum 3547 s, step 3563 s; starter_program 549,
  detail_check_agrees 473, unittests 199, detail_verb 170.
- Where the time goes (perf, fixed 3 ms period, frame pointers, one
  `detail` commit of the gallery site-plan point): 85% of CPU in
  worker threads, 70% under `World::flown`/`World::expand` (the
  flight-memo walks: memory-latency bound); the main thread is
  serial for ~19 s of a ~33 s wall: `occupancy_over` ~7 s (string
  classification per cell, four calls per battery), light
  `relight_with` ~4 s, `blockout::check` ~3.5 s.

One commit of node/annex on that point, 4 cores (`taskset -c 0-3`),
three alternated runs, min wall: base 44.6 s / 949G instructions;
head 29.5 s / 727G instructions (-34% wall, -23% instructions). The
written campaign, library and stdout are byte-identical to base's.

After, this branch, remote, 4 cores (`taskset -c 8-11`):
- `cargo test --locked --workspace --no-fail-fast`: step 2879 s
  (user 8261 s), sum 2865 s (-19%); 376 test results, 0 failed;
  starter_program 352 (-36%), detail_check_agrees 303 (-36%),
  place_reads 107 (-28%), detail_atomic_write 70 (-26%),
  detail_verb 145 (-15%).
- `tools/ci/cargo-test-concurrent.py --jobs 2 --locked`: step 2360 s
  (user 8341 s): -34% against base's sequential 3563 s; 373 of 373
  executables run, 0 failed.

Town branch `feat/town-merge` @ c965e9f, read only, 4 cores:
starter_program 817 s -> 615 s and detail_check_agrees 1246 s ->
1135 s with the occupancy commits cherry-picked (its own light change
already matches this branch's; that file conflicts trivially).

Gallery, base `delvec` (1f068cd) against head: primary build 1569
files and site-plan point build 378 files byte-identical, and head
built twice byte-identical; binding lines diff empty (build 72,
site-plan build 73, analyze 23); `check-gallery-coverage.py` report
JSON and index byte-identical (80 ledgers, 0 zero bindings).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@stellarfeline @claude
fix(tests): a nested build names the compiler that built the test
6db1f87 
The concurrent runner launches test executables directly, so they no
longer inherit rustup's RUSTUP_TOOLCHAIN from `cargo test`. The nested
`cargo run` of gallery/prefab-generator (env!("CARGO"), the exact pinned
cargo) then runs `rustc` through the rustup proxy, which resolves a
toolchain per invocation from its working directory: registry crates
(cwd in ~/.cargo/registry) got the machine's default toolchain, the
workspace's path crates got the pin. One target directory held crates of
two compilers (E0514), and concurrent nested builds each started
installing the default toolchain (CI: stable + rust-src) at once, racing
on ~/.rustup/downloads.

Reproduced locally (default toolchain nightly): the reach_graph test
executable run directly, without RUSTUP_TOOLCHAIN, fresh nested target:
exit 101, 10 E0514 lines, serde compiled by 1.100.0-nightly. Same run
with only RUSTUP_TOOLCHAIN=1.97.1 set: 2 passed. With this change and
without RUSTUP_TOOLCHAIN: 2 passed, 0 E0514.

common::build_toolchain sets CARGO and RUSTC (the rustc beside the
cargo that built the test) on a command; common::nested_cargo is the
cargo command with it. Every nested toolchain process the tests spawn
takes it: the gallery generator (common, reach_graph) and
gallery_domain.py materialise (starter_program), which runs $CARGO.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd
@stellarfeline @claude
chore(perf): the CI test command, green at 6db1f87 on a CI-shaped too…
02ef09d 
…lchain

Measured on the remote build machine (16 CPU, sccache on, incremental
off), 6db1f87, `python3 tools/ci/cargo-test-concurrent.py --jobs 2
--locked` exactly as ci.yml runs it, DELVEWRIGHT_CLIENT_JAR set to the
pinned jar. Clean target and clean gallery/prefab-generator/target. A
private RUSTUP_HOME whose default is stable 1.99.1 with no rust-src (the
CI runner's shape; the machine's own default is the pin, which hides the
defect), RUSTUP_TOOLCHAIN unset.

  test executables: 373 run of 373 listed, 0 failed
  test result lines: 376; passed 5137, failed 0, ignored 12
  the four CI failures and the two other generator tests: ok
  wall: 2019 s
  stable's installed components after the run: cargo, rust-std, rustc
  (no nested process reached the default toolchain)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd
@stellarfeline
stellarfeline merged commit 9c50a20 into main Nov 1, 2026
23 checks passed
@stellarfeline
stellarfeline deleted the fix/ci-time branch November 1, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@stellarfeline