Repository navigation
chore(sync): main into the town branch (CI speedup), and the town tests run in the tree group - #1247
Merged
Merged
Conversation
…un two executables at a time (#1244) * perf(occupancy): classify each block state once, not each cell `occupancy_over`, `climb_holds` and `sculk::Listening::of` asked the `blockshape` and registry rules of every cell's state text: string parsing per cell over a world of millions of cells and a few hundred states. Each reading is a function of the state alone, so each is now made once per distinct interned state (keyed by `BlockState::key`, the interned text's identity) and looked up per cell. Nothing iterates in key order; the sets built are the same. Profile that found it (perf, frame pointers, one `detail` commit of the gallery site-plan point, 8 cores): the commit's main thread is serial for ~19 s of a ~34 s wall; `occupancy_over` is ~7 s of that, called four times per battery (light::final_world, light::lit_world, blockout::check, blockout::crossings); `sculk::Listening::of` ~1 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(light): the dark survey reads the dense field, and a state packs once `DarkSurvey::measure` copied the whole flooded field into a `BTreeMap` (`LightModel::flood`) to read the light of the cells it examines; it now reads the dense field (`LightField::light_at`), which answers what the map's `unwrap_or(0)` answered: a lit cell's value, 0 for an unlit cell or one outside the box. `LightField::over` resolved a cell's byte through a map keyed by the block text (a string comparison per cell); it now keys by the text's address and length, so an interned state is one key. In the commit profile `relight_with` was ~4 s of the main thread, 70% of it the map copy and the string-keyed lookups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(reach): the flight memo finds a node's child by one probe A flight-memo walk (`flights::lookup`) chose the child of each run by walking the parent's sibling list, one load from anywhere in `nodes` per sibling. With a fixed-period profile (`cpu-clock`, 3 ms, frame pointers) of one gallery site-plan commit, the single compare in that walk held 13.5% of all CPU samples, and `World::expand`, which inlines the walk, 70% inclusive. Each shard now indexes its edges by `(parent, code)`; the sibling lists stay as the record of a node's children, which a split moves to the run's old rest. The walk answers the same child: a parent never holds two children of one code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * revert: perf(reach): the flight memo finds a node's child by one probe Measured A/B on one gallery site-plan commit (8 cores, perf stat, two alternated runs): the index raised user instructions from 762G to 866G and task-clock from ~80 s to ~122-136 s. The sibling lists are short; the probe costs more than the walk it replaced. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(occupancy): the cell sets are built once from sorted cells `occupancy_over` walks the block map in ascending cell order and inserted each cell into its `BTreeSet`s one search at a time; after the per-state classification, those inserts were most of what was left of it (fixed-period profile of one gallery site-plan commit). The cells are now gathered in order and each set is collected in one pass; the sets hold the same cells (a waterlogged fluid cell, gathered twice as a source, is held once, as before). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(rust): the test executables run two at a time CI configuration, its own commit. `cargo test --workspace` runs the workspace's test executables one after another; the long ones are sequences of `delvec` runs whose single-threaded stretches leave most of a 4-core runner idle. `tools/ci/cargo-test-concurrent.py` builds every test target, reads the executables from cargo's own JSON listing, runs each once and whole, two at a time (each to its end), then the doc tests, and states `R run of L listed, F failed`. What it bought, measured on its own: remote build machine, tree 530a414 (the engine changes before this commit), 4 cores (`taskset -c 8-11`), already built, `/usr/bin/time` over the whole step: cargo test --locked --workspace --no-fail-fast wall 3606.8 s, user 9276.0 s, sys 500.4 s (2.71 cores busy) cargo-test-concurrent.py --jobs 2 --locked wall 2913.6 s, user 9960.7 s, sys 491.5 s (3.59 cores busy) -19.2% wall; 373 of 373 executables run, 0 failed; 376 `test result` lines in both logs. Other rounds shared the machine during both runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(light): rustfmt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(perf): the round's measurements, before and after No code. The numbers the commits above were judged by. Instrument: the remote build machine (16 cores, WSL2, other rounds sharing it throughout), dev profile without debug info, `taskset` for a fixed core set; `cargo test`'s own `finished in` per binary as the first method, `/usr/bin/time` over the whole step and `perf stat` (instructions:u, task-clock) as the second. Before, `main` @ 1f068cd: - CI (run 40837884057, 4 vCPU): test step 1 h 39 min = 10 min build + 88.5 min of binaries (sum of 376 `finished in`, 5310 s); top: starter_program 771 s, detail_check_agrees 588, unittests 308, detail_verb 289, remedy_reachability 248, place_reads 231. - remote, 8 cores: sum 2764 s, step 2784 s. - remote, 4 cores: sum 3547 s, step 3563 s; starter_program 549, detail_check_agrees 473, unittests 199, detail_verb 170. - Where the time goes (perf, fixed 3 ms period, frame pointers, one `detail` commit of the gallery site-plan point): 85% of CPU in worker threads, 70% under `World::flown`/`World::expand` (the flight-memo walks: memory-latency bound); the main thread is serial for ~19 s of a ~33 s wall: `occupancy_over` ~7 s (string classification per cell, four calls per battery), light `relight_with` ~4 s, `blockout::check` ~3.5 s. One commit of node/annex on that point, 4 cores (`taskset -c 0-3`), three alternated runs, min wall: base 44.6 s / 949G instructions; head 29.5 s / 727G instructions (-34% wall, -23% instructions). The written campaign, library and stdout are byte-identical to base's. After, this branch, remote, 4 cores (`taskset -c 8-11`): - `cargo test --locked --workspace --no-fail-fast`: step 2879 s (user 8261 s), sum 2865 s (-19%); 376 test results, 0 failed; starter_program 352 (-36%), detail_check_agrees 303 (-36%), place_reads 107 (-28%), detail_atomic_write 70 (-26%), detail_verb 145 (-15%). - `tools/ci/cargo-test-concurrent.py --jobs 2 --locked`: step 2360 s (user 8341 s): -34% against base's sequential 3563 s; 373 of 373 executables run, 0 failed. Town branch `feat/town-merge` @ c965e9f, read only, 4 cores: starter_program 817 s -> 615 s and detail_check_agrees 1246 s -> 1135 s with the occupancy commits cherry-picked (its own light change already matches this branch's; that file conflicts trivially). Gallery, base `delvec` (1f068cd) against head: primary build 1569 files and site-plan point build 378 files byte-identical, and head built twice byte-identical; binding lines diff empty (build 72, site-plan build 73, analyze 23); `check-gallery-coverage.py` report JSON and index byte-identical (80 ledgers, 0 zero bindings). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(tests): a nested build names the compiler that built the test The concurrent runner launches test executables directly, so they no longer inherit rustup's RUSTUP_TOOLCHAIN from `cargo test`. The nested `cargo run` of gallery/prefab-generator (env!("CARGO"), the exact pinned cargo) then runs `rustc` through the rustup proxy, which resolves a toolchain per invocation from its working directory: registry crates (cwd in ~/.cargo/registry) got the machine's default toolchain, the workspace's path crates got the pin. One target directory held crates of two compilers (E0514), and concurrent nested builds each started installing the default toolchain (CI: stable + rust-src) at once, racing on ~/.rustup/downloads. Reproduced locally (default toolchain nightly): the reach_graph test executable run directly, without RUSTUP_TOOLCHAIN, fresh nested target: exit 101, 10 E0514 lines, serde compiled by 1.100.0-nightly. Same run with only RUSTUP_TOOLCHAIN=1.97.1 set: 2 passed. With this change and without RUSTUP_TOOLCHAIN: 2 passed, 0 E0514. common::build_toolchain sets CARGO and RUSTC (the rustc beside the cargo that built the test) on a command; common::nested_cargo is the cargo command with it. Every nested toolchain process the tests spawn takes it: the gallery generator (common, reach_graph) and gallery_domain.py materialise (starter_program), which runs $CARGO. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd * chore(perf): the CI test command, green at 6db1f87 on a CI-shaped toolchain Measured on the remote build machine (16 CPU, sccache on, incremental off), 6db1f87, `python3 tools/ci/cargo-test-concurrent.py --jobs 2 --locked` exactly as ci.yml runs it, DELVEWRIGHT_CLIENT_JAR set to the pinned jar. Clean target and clean gallery/prefab-generator/target. A private RUSTUP_HOME whose default is stable 1.99.1 with no rust-src (the CI runner's shape; the machine's own default is the pin, which hides the defect), RUSTUP_TOOLCHAIN unset. test executables: 373 run of 373 listed, 0 failed test result lines: 376; passed 5137, failed 0, ignored 12 the four CI failures and the two other generator tests: ok wall: 2019 s stable's installed components after the run: cargo, rust-std, rustc (no nested process reached the default toolchain) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ce, tests two at a time) Merges origin/main @ 9c50a20 (#1244's perf change) into the town branch @ 51c0d67. One textual conflict, in crates/delvec/src/compiler/light.rs, DarkSurvey::survey: both sides made the same code change (read the dense LightField per cell instead of copying LightModel::flood's map) and wrote different comments for it. The merged comment carries both claims: the map is the field's lit cells copied out, it is huge for a site plan reaching the horizon, and a cell it does not hold reads 0. The town branch adds no nested cargo build that would need common::nested_cargo / build_toolchain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd
`tools/tests/test_town.py` imports the creator plugin's `scripts/starter.py` and `scripts/town.py` at module level. It was not listed in `conftest.WHOLE_TREE`, so it fell to the narrow `tool-tests` job by default, whose declared reach is the tools and the engine readers, and that job's reach guard failed the run during collection: reach guard: outside the reach during collection: ['open .claude/skills/.../scripts/starter.py', 'open .claude/skills/.../scripts/town.py'] The placement was wrong, not the reach: `tool-tests` is declared as the job that tests tools under `tools/`, and tests of the plugin run under `tool-tests-tree`, where `test_starter.py` (the sibling testing the same script) already sits. The file joins it there; the group's `why` names both readers. Measured on the remote, `DW_CI_REACH_GROUP=tool-tests python3 -m pytest -q tools/tests` (pytest 9.1.1): red before (1084 passed, 2 skipped, guard names the two reads), see the commit's round report for the green counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd
No file changes; the measurements of 52600d7 (merge of main @ 9c50a20 into feat/town-merge @ 51c0d67, plus the reach-guard fix), taken on the remote build machine (Linux, 16 CPU) with CI's commands. - reach guard, `DW_CI_REACH_GROUP=tool-tests python3 -m pytest -q tools/tests` (pytest 9.1.1): before the fix (d0cdee0) 1084 passed, 2 skipped, guard red naming `scripts/starter.py` and `scripts/town.py` during collection; after, exit 0, 1036 passed, 2 skipped, 29451 reads judged against 19 globs. `tool-tests-tree`: exit 0, 809 passed, 2 skipped. - `bash tools/ci/fmt-workspaces.sh --check`: 2 workspaces swept, 0 not clean. `cargo clippy --locked --workspace --all-targets -- -D warnings`: exit 0. - `python3 tools/ci/cargo-test-concurrent.py --jobs 2 --locked`: exit 0, 375 run of 375 listed, 0 failed; 5201 passed, 0 failed, 12 ignored. - `gallery-baseline.py --compare 51c0d67`: GREEN; 10 builds each side, 10502 emitted paths each, 0 added / 0 removed / 0 changed, 0 recorded values moved; warning rows 271 against the ledger's 271. Coverage 1222 bound + 17 refusal-proven of 1239, 0 unaccounted. - content demo/town-4 @ 7fb8a92, `delvec --prefabs prefabs build campaigns/foot-of-the-mountain` with release builds of the town tip and of this tree: both exit 0, 267 files each, tree digest 75bd382cb430... on both, `diff -rq` empty, every DW line identical; DW1076: 1 warning (blockout/node/castle, 439 outlined, 438 held, 1 floating). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019qiQFhKtTpTbYnn9NZBbSd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
main@ 9c50a20 intofeat/town-mergeand fixes the one red of the town pull request.light.rsDarkSurvey::survey, where both sides had made the same change (LightField::newandlight_at); one comment now carries both. Gallery output is byte-identical to the branch tip (10 builds, 10502 paths, 0 changed); warnings 271 = 271.tools/tests/test_town.pyimports the plugin'sstarter.pyandtown.pyat collection, but sat in the narrowtool-testsgroup. It now runs intool-tests-treebesidetest_starter.py, which tests the same script; no glob changed. The guard was red before the fix and clean after, with CI's command.cargo test375 of 375 executables, 5201 passed, 0 failed;tool-tests1036 passed andtool-tests-tree809 passed; every docs, binary, gallery and coverage job exits 0. The town builds byte-identical to the branch tip (267 files), with DW1076 = 1.🤖 Generated with Claude Code