perf: vein census 3 -- nine deterministic wins from real-program profiles

crates/rusty_alloc/CHANGELOG.md

@@ -7,6 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Performance
+
+Measured against 2.2.5 with callgrind, allocator instructions, outputs
+identical: Endless Sky start-up -6.85 %, lua -2.23 %, perl -1.64 %, sqlite
+-0.11 %; Rust `GlobalAlloc` workloads up to -4.24 % whole-program. Peak RSS
+within +-0.4 %. Details: `docs/opps.md`, "Vein census 3".
+
+- **Pages carve in growing batches.** A page's first extension is still one
+  4 KiB OS page; each later one carves as many blocks as the page already
+  holds. Each extension costs a slow-path trip, and those trips, not the
+  linking, were the cost. Thread start-up is unchanged.
+- **A cross-thread free into a full page goes straight to its owner** (the
+  `free`/`operator delete` exports test the owner first). Rust's `dealloc`
+  keeps the 2.2.5 order, which keeps `__rust_dealloc` inlinable into drop
+  glue. Trade-off: a same-thread free of a large or huge block costs 2 more
+  instructions (about 1.4 % of that path).
+- **The slow path looks up a small size's bin from a table.**
+- **`operator new`'s slow path has a smaller frame** (`malloc_or_with`, a
+  typed OOM handler).
+- **A moving `realloc` resolves the block's page once, not twice.**
+- **Exited threads' heap boxes are reused**, saving an `mmap`/`munmap` pair
+  and a page fault per thread.
+- **Rust: the natural-alignment arm of `GlobalAlloc::alloc` is out of line,
+  and `usable_size` is inlinable.**
+- **MSVC builds with `panic = "unwind"`** take back one of the three
+  instructions per free that 2.2.5's fix cost them.
+
+### Added
+
+- `alloc::malloc_or_with` and the `alloc::OnOom` trait: `malloc_or` with the
+  OOM handler as a type.
+- `alloc::free_inline_flags_first`: `free_inline`'s body in the 2.2.5 test
+  order, for Rust's `GlobalAlloc::dealloc`.
+
 ## [2.2.5](https://github.com/Remade-With-Rust/rusty_alloc/compare/rusty_alloc-v2.2.4...rusty_alloc-v2.2.5) - 2026-10-08
 
 ### Fixed

crates/rusty_alloc/UNSAFE.md

@@ -20,7 +20,7 @@ unsafe in DEPENDENCIES, and ours are `libc` plus bindings-only `windows-sys`.
 
 | Module | Count | What the `unsafe` is for | Last audit |
 |---|---:|---|---|
-| `alloc.rs` | 110 | The public entry points: raw-pointer reads on the malloc fast path (must never form `&mut` on the shared empty-heap sentinel), pointer-derived metadata on the free path (`segment_of`/`page_of`), block-content copies in the realloc family. **+15 on 2026-08-22**: the free path's two `asm!` sites — the memory-destination `used--` whose flags drive the retire branch (its `label` block is a separate item and carries its own `unsafe`), and the fused `cmp {tid}, fs:0` in both `free_inline` and `free_general` — plus `malloc_or`/`malloc_or_slow`, which give `operator new` a fast path whose miss is a tail call. Each asm reads or writes exactly one field it already had a valid pointer to; none widens what the surrounding code could already touch. **+2 on 2026-09-24 (instruction-count campaign, LEDGER):** `realloc` split into an inline null test and `unsafe fn realloc_live(p: NonNull<u8>, ..)` — one signature, and the `NonNull::new_unchecked` sits inside the block that already called it, on a pointer the line above proved non-null. The body is the old one unchanged; nothing new is dereferenced. **+10 on 2026-09-24 (round two, LEDGER):** (a) `realloc_move`, the moving arm shared by the growth and the shrink — one `unsafe fn`, its body block, and the two call-site blocks: the old inline move code, now reached with the copy length each arm already holds; (b) `zalloc` in `malloc`'s raw-read shape — its fast-path block reads through `hb`, which may be the sentinel, exactly as `malloc` does (no `&mut`, no write unless a block was popped, which proves the heap is real), and the cold `unsafe fn zalloc_slow` that performs the sentinel test the fast path dropped (signature + block); (c) `free_local_owned` / `free_local_no_xheap`, `owner_heap` split so its heap-creating fallback is a tail call — two signatures and their two blocks, the same loads and the same `free_local_at` as before; (d) `malloc_aligned_pow2`, an `unsafe fn` only because its caller must have proven the alignment a power of two (a violation is not memory-unsafe — the bound test still refuses a mask at or above half a segment). Nothing new is dereferenced anywhere; each carries its SAFETY line. The baseline had drifted to 106 without a row edit; recorded now. **+4 on 2026-10-07 (the Miri Stacked Borrows failure on `main`):** the deferred-free hook used to fire inside `Heap::malloc_generic`, where `&mut self` is protected, so a hook that allocates re-entered the heap through a raw pointer under that borrow. It now fires at the allocation entries before any `&mut Heap` exists: `fire_deferred_if_due` (an `unsafe fn` and its block: a raw read and write of the heap's own `deferred_due` flag, then the hook) and `malloc_slow_deferred` (an `unsafe fn` and its block: the same, then the `malloc_generic` call `malloc_slow` already made). The nine entry blocks that call it existed before; nothing new is dereferenced | 2026-10-07 |
+| `alloc.rs` | 123 | The public entry points: raw-pointer reads on the malloc fast path (must never form `&mut` on the shared empty-heap sentinel), pointer-derived metadata on the free path (`segment_of`/`page_of`), block-content copies in the realloc family. **+15 on 2026-08-22**: the free path's two `asm!` sites — the memory-destination `used--` whose flags drive the retire branch (its `label` block is a separate item and carries its own `unsafe`), and the fused `cmp {tid}, fs:0` in both `free_inline` and `free_general` — plus `malloc_or`/`malloc_or_slow`, which give `operator new` a fast path whose miss is a tail call. Each asm reads or writes exactly one field it already had a valid pointer to; none widens what the surrounding code could already touch. **+2 on 2026-09-24 (instruction-count campaign, LEDGER):** `realloc` split into an inline null test and `unsafe fn realloc_live(p: NonNull<u8>, ..)` — one signature, and the `NonNull::new_unchecked` sits inside the block that already called it, on a pointer the line above proved non-null. The body is the old one unchanged; nothing new is dereferenced. **+10 on 2026-09-24 (round two, LEDGER):** (a) `realloc_move`, the moving arm shared by the growth and the shrink — one `unsafe fn`, its body block, and the two call-site blocks: the old inline move code, now reached with the copy length each arm already holds; (b) `zalloc` in `malloc`'s raw-read shape — its fast-path block reads through `hb`, which may be the sentinel, exactly as `malloc` does (no `&mut`, no write unless a block was popped, which proves the heap is real), and the cold `unsafe fn zalloc_slow` that performs the sentinel test the fast path dropped (signature + block); (c) `free_local_owned` / `free_local_no_xheap`, `owner_heap` split so its heap-creating fallback is a tail call — two signatures and their two blocks, the same loads and the same `free_local_at` as before; (d) `malloc_aligned_pow2`, an `unsafe fn` only because its caller must have proven the alignment a power of two (a violation is not memory-unsafe — the bound test still refuses a mask at or above half a segment). Nothing new is dereferenced anywhere; each carries its SAFETY line. The baseline had drifted to 106 without a row edit; recorded now. **+4 on 2026-10-07 (the Miri Stacked Borrows failure on `main`):** the deferred-free hook used to fire inside `Heap::malloc_generic`, where `&mut self` is protected, so a hook that allocates re-entered the heap through a raw pointer under that borrow. It now fires at the allocation entries before any `&mut Heap` exists: `fire_deferred_if_due` (an `unsafe fn` and its block: a raw read and write of the heap's own `deferred_due` flag, then the hook) and `malloc_slow_deferred` (an `unsafe fn` and its block: the same, then the `malloc_generic` call `malloc_slow` already made). The nine entry blocks that call it existed before; nothing new is dereferenced **+13 on 2026-10-08 (vein census 3, `docs/opps.md`):** every one is a split or a re-shape of an existing path, and nothing new is dereferenced. `free_inline_impl` + `free_inline_flags_first` + `free_body` (the free fast path split by test order and by whether the page is already known: three `unsafe fn`s and the blocks that forward the contract); a second owner-test `asm!` arm whose label calls `remote_free` exactly as before; `remote_free_flagged` (an `unsafe fn`, its `unalign` block and its `remote_free` block: the same unalign `free_general` did for a remote aligned block); `realloc_live`'s page resolution (the `segment_of`/`page_of`/flags read `usable_size` did) and `realloc_move_at` replacing `realloc_move` (same allocate-copy-free, freeing through the page already resolved); `malloc_or_with`/`malloc_or_slow_with` (the `malloc_or` fast path and cold arm with a typed handler). The MSVC-unwind `sub`/`setle` arm sits inside the existing fast-path block | 2026-10-08 |
 | `heap.rs` | 80 | Owner-thread page/queue manipulation under raw pointers (no two `&mut Page` may coexist), the aligned-allocation peek, span carving. **+3 on 2026-08-19**: `try_unlink_huge_segment` split out of `remove_huge_segment`. **+15 on 2026-08-22**: `malloc_generic` split into a small entry plus `malloc_generic_walk`, with `grow_front`, `try_guarded` and `drain_delayed` as cold arms — each split adds an `unsafe fn` signature and its block while dereferencing nothing the single function did not — and the immortal `EMPTY_DELAYED` sentinel that lets the heartbeat read its list without a null test. **+2 on 2026-09-07 (P4e, §2.15 of `docs/plans/small-metal.md`)**: the reclamation fixes — one `unsafe` for the periodic `generic_collect` sweep and one for the reclaim-and-retry that runs before `malloc_generic` returns null. Both call `collect_inner`, which allocates nothing and touches only this heap’s own pages on the owner thread; each carries its SAFETY line. `malloc_generic` itself became a safe wrapper over the renamed `malloc_generic_once`, so the split added no signature. **+2 on 2026-09-08:** the medium collect-and-retry ahead of the heartbeat -- one `unsafe` around `page_collect` + `page_pop` on the bin queue front, one reading `free_is_zero` off the page just popped. Both are the SAME operations `malloc_generic_walk` performs on the same page a few lines later, on the owner thread under the heap lock: the block moves earlier, nothing new is dereferenced. Each carries its SAFETY line. **+2 on 2026-09-16 (Openheimer):** the two refuse arms in `malloc_aligned_at_slow` when `bins::aligned_at_from` reports that `block + offset + align` would wrap for a caller-chosen offset — each is one `free_local` of the block just allocated on this thread, which has not escaped. **+1 on 2026-09-24 (instruction-count campaign, LEDGER):** the medium collect-and-retry's `page_collect` + `page_pop` block became two blocks so the `saw_remote_free` latch is stored between them — same two calls on the same page, owner thread, in the same order; nothing new is dereferenced. **+1 on 2026-09-24 (round two):** a medium miss grows the front page directly — one block around the `capacity < reserved` test and the `grow_front` call, on the same page the block above just collected, which is `grow_front`'s whole contract **+2 on 2026-09-25 (CURIOSITY ROUND FOUR):** the medium arm pops the front page BEFORE collecting it — one `page_pop` block and the `free_is_zero` read beside it, the same two operations the collect-then-pop below already performs on the same owner-thread page. **+1 on 2026-10-07 (`docs/opps.md` vein census, vein 2):** `collect_inner` hands the per-heap cached large span (`Heap::large_cache`) back with `retire_span` — one block around `segment_of` on the cached page's own address and the same `retire_span` a large free performed before the cache existed, on the owner thread. **+2 the same day:** `free_large_span` — an `unsafe fn` and its block — is the large-span arm of `free_local_at` moved into its own function (double-free check, cache, free-run flag); the same operations on the same page, owner thread | 2026-10-07 |
 | `init.rs` | 42 | Thread/heap lifecycle: the initial-exec TLS slot (`global_asm!` + fs-relative asm reads), thread-pointer register reads (`fs:0`/`gs:0x30`/`tpidrro_el0`), heap-box creation/teardown, the abandonment path run inside platform TLS destructors. **+1 on 2026-09-09 (`firmware-what-is-left.md` §7.3): an ATTRIBUTE, not an operation** — `unsafe(link_section = ".rodata.…")` on `EMPTY_HEAP_BOX`, bare metal only, so the never-written sentinel lives in flash instead of costing 1,752 bytes of RAM. The token is counted because the census is a text search; the contract it rests on (raw reads only, no page ever stores its address) is the one the `Sync` impl below already carries, and a write would now fault against the flash cache instead of silently landing. `create_heap`'s template copy from it is a `ptr::read` inside the block that already existed. **+5 on 2026-09-16 (Openheimer, OH-rusty_alloc-13):** thread exit now abandons EVERY heap the dying thread still owns, not only the one in `done_slot` — a first-class heap that was never `heap_delete`d, or one installed with `set_default_heap`, used to stay DELAYED under a dead owner forever. `take_heap_owned_by` unlinks one such box under `HEAPS_LOCK` (one block: the registry walk); `thread_done` reads `owner_tid` off its own box and calls the split-out `unsafe fn thread_done_one` for the primary box and for each extra one (three sites plus the signature). Nothing new is dereferenced that the single function did not; each carries its SAFETY line | 2026-09-16 |
 | `segment.rs` | 45 | Segment/page metadata addressing: the mask trick (`segment_of`), `page_of`'s contract-based indexing — **the bound is now PROVED for every in-segment offset by `proofs.rs` (Kani), not merely asserted** — span tiling, purge/recommit. The baseline had drifted to 36 without a row edit; recorded now. **+2 on 2026-10-04 (`docs/plans/recommit-failure-ignored.md`):** `restore_for_reuse` — an `unsafe fn` and its block — is the guard-lift and re-commit `segment_free` and `huge_free` each did inline, moved into one function so both can act on its answer: a segment whose memory the OS will not make accessible and committed again is released whole or retired in place, never recycled. It touches the same range the inline code did; `segment_free`'s own block became the expression that calls it (no net change). `span_recommit` now returns whether the span is backed (same block, same call). **+5 the same day, all `#[cfg(test)]`:** `a_failed_recommit_is_a_failed_span_not_an_unbacked_one` writes and frees the blocks it allocated (four blocks) and frees them at the end (one); each carries its SAFETY line. **+2 on 2026-10-07 (`docs/opps.md` vein census, vein 1):** `scrub_recycled` — an `unsafe fn` and its block — replaces the whole-header `write_bytes` that `segment_alloc` and `huge_alloc` did on every recycled chunk: it zeroes the 2 KB owner table (all of it for a Normal segment, entry 0 for a Huge one, which overwrites the rest) and the first `slots` page slots, inside the same header the old call covered. The other slots of a Normal segment are zeroed where the bump cursor first carves them (inside `span_alloc`'s existing block); a Huge segment reads only slots 0 and 1, and `wait_no_remote_in_flight` now scans slot 1 alone there. Nothing new is dereferenced | 2026-10-07 |