gc: a pacing full at about the cost of its live mark; promoted-cohort fulls (#10182) [land after #10249]

changelog.d/10241-pacing-full-cost.md

@@ -0,0 +1,117 @@
+A synchronous full over a heap of promoted JSON trees costs about half as much,
+and a full can now be paced by the bytes promoted since the previous full
+(#10182). Stacked on #10220.
+
+**Cheaper full** (each change has a planted test and a sabotaged twin):
+
+- **Runs expanded only where the sweep reshapes.** `GcCycleState::new_full`
+  no longer expands every described promoted page run.
+  `PendingOldUnregister::defer` now expands the runs on a dead object's pages
+  before it invalidates that object.
+- **One-pass census.** An unbounded census hands each arena block to
+  `census_whole_block`, which parses headers and start bits in one loop.
+  - A block the preceding minor stamped while promoting it in place is adopted
+    from a record built during that stamping (`gc/trace/adopt_census.rs`),
+    without being read again.
+  - Adoption happens only in a promoted-cohort full that follows that minor
+    at the same safepoint.
+  - A census block is found through a direct-mapped 1 MiB window index.
+- **One-pass sweep and fewer hole scans.** An unbudgeted sweep walks each
+  block in one pass. The hole-list rebuild skips live blocks the census proved
+  hole-free, filtering the block list in place.
+- **Cheaper mark.** The mark reads its per-object facts (proxy tracing, weak
+  holder) once per object. It sets a pointer-free object's mark bit without
+  queueing it. It prefetches ahead in the worklist and before a range
+  descriptor's children. `GcMutableSlot` classifies `external` lazily.
+- **Stack scrub.** Between the census and the root scan, a full zeroes 16 KiB
+  of dead stack below its own frame. Census residue there had become one extra
+  conservative root. That root kept a dead 7 MB string alive and cost
+  `records_array_8m:roundtrip` +30 MiB peak RSS.
+
+On #10220's probe (one 20 MB tree in the old generation, `gc()` in a loop), a
+steady full drops from 38 ms to 25 ms: census 4.9 → 2.4 ms, mark 25 → 18 ms,
+sweep 7.2 → 3.3 ms.
+
+The probe's pacing full drops from 80 ms to 27 ms: rebuild 23 → 0 ms, sweep
+21 → 4.4 ms. On base that full fires at an allocation point, with a forced
+conservative scan. Here it is a promoted-cohort full at the safepoint, and it
+fires earlier. The `gc()` that follows therefore sweeps more, but still costs
+less than base's: 42 ms against 47 ms.
+
+**Promoted-cohort fulls** (`gc/promoted_cohort.rs`):
+
+- **Trigger.** A copying minor at a precise safepoint runs a full right after
+  it when the bytes promoted since the last full reach
+  `max(nursery cap, old live at the last full << shift)`. That full uses the
+  safepoint's roots, with no forced conservative scan.
+- **Backoff.** A full that reclaims less than half its cohort raises `shift`
+  (capped at 3). A productive full resets it to 0.
+- **Old-reclaim baseline untouched.** The cohort does not change that
+  baseline, so both baseline tests #10204 broke still pass.
+- **Diagnostics.** `PERRY_GC_DIAG=1` prints `[gc-promoted-cohort]` and adds
+  `promoted_since_full=` / `cohort_bound=` to `[gc-trigger]`.
+
+On a real cohort full in `records_array_20m:parse` the pause is 30–32 ms, down
+from the 62–68 ms #10220 measured.
+
+**Cohort survival feed** (`gc/promoted_cohort/survival.rs`):
+
+- **Defect.** Every full resets the untraced-promotion budget, so a cohort
+  full every ~live bytes kept `14_grow_then_churn` promoting its churn
+  untraced for good. No minor measured again: 13 cohort fulls, 0 copied
+  objects, wall 429 → 594 ms against #10220.
+- **Measurement.** A cohort full's sweep measures how much of what the minor
+  at its own safepoint promoted is still live. Blocks walked whole report their
+  live bytes; dead blocks reclaimed unwalked count as 0.
+- **When it is fed.** The figure replaces the young-survival predictor (below
+  950‰) only when it equals what a minor would measure. After the mark, the
+  unmarked objects on the minor's dirty old pages have their dirty slots read.
+  A slot pointing into the promoted blocks means a dead remembered parent
+  held them, and then nothing is fed.
+- **Why the gate.** The JSON rows measure 500‰ (8m scan: 333‰), because each
+  minor promotes the dead previous tree along with the live one. Their tree
+  arrays are exactly such dead parents. Fed unconditionally, the next minor
+  evacuates both trees: `records_array_20m:parse` +20.0 % CPU / +54.9 MiB,
+  `records_array_8m:scan` +14.1 % / +31.2 MiB (mini, best of 3).
+- **Diagnostics.** `[gc-promoted-cohort]` adds `promoted_by_minor=`,
+  `live_of_promoted=`, `survival_permille=`, `minor_view=` and `predictor=`.
+
+**Only in-place promotions fill the cohort.** A copying minor tenures an object
+only after it survived a minor, so a full scheduled for tenured bytes is
+futile. `12_large_live_set`'s one cohort full was reached by 21.2 MB of tenured
+bytes over a 16 MB bound and reclaimed 8.3 MB; every cohort full on the JSON
+rows was reached by in-place promotions alone. The old-reclaim baseline credit
+still takes every promoted byte.
+
+Laptop, interleaved, 7 rounds (median wall / peak RSS):
+
+| probe | #10220 | before these two changes | now |
+|---|---|---|---|
+| `14_grow_then_churn` | 0.44 s / 284.6 MiB | 0.63 s / 131.1 MiB | 0.29 s / 65.1 MiB |
+| `12_large_live_set` | 0.26 s / 109.1 MiB | 0.28 s / 115.6 MiB | 0.24 s / 109.0 MiB |
+
+On the mini, every JSON row's collection schedule (minors, fulls, cohort
+sizes, bounds and reclaimed bytes) is identical before and after both changes,
+and CPU stays within ±1.0 %, so the table below stands.
+
+**This does not meet #10182's acceptance bar.** Interleaved best-of-3 on the
+same tree, base `9a05821b9e`:
+
+| row | base | this branch | best node/bun |
+|---|---|---|---|
+| `records_array_20m:parse` | 143.8 ms / 240 MiB | 209.6 / 167 | 208.0 / 220 |
+| `records_array_20m:scan` | 153.7 / 240 | 218.4 / 167 | 212.0 / 225 |
+| `records_array_20m:sparse` | 145.5 / 240 | 214.7 / 167 | 208.4 / 220 |
+| `records_object_20m:parse` | 146.3 / 240 | 210.9 / 167 | 207.4 / 220 |
+| `records_array_20m:roundtrip` | 105.0 / 260 | 117.2 / 231 | 162.1 / 261 |
+| `records_array_8m:scan` | 182.0 / 189 | 227.1 / 127 | 188.7 / 110 |
+
+- **Target rows.** All four now beat node/bun on RSS but are +0.8 to +3.0 %
+  over the best CPU. Each iteration promotes its dead predecessor's tree, so
+  each row runs three fulls of about 22 ms net against a lead of 58–64 ms.
+- **Other rows.** `records_array_8m:scan` loses its CPU lead (4 cohort fulls),
+  and `records_array_20m:roundtrip` is +12 % CPU.
+- **Mechanism commits alone** (cohort trigger disabled locally): no row is
+  outside ±2 % CPU in both of two interleaved matrices. The 20 MB rows read
+  +0.0 to +1.8 %. `records_array_8m:parse`/`sparse` peak RSS falls from
+  109 MiB to 97/98 MiB.

crates/perry-runtime/src/arena/mod.rs

@@ -147,23 +147,23 @@ pub(crate) use stats::{old_gen_in_use_bytes_recomputed, old_gen_in_use_bytes_res
 // page_meta.rs (public + pub(crate) classification/page-meta API)
 pub(crate) use page_meta::{
     arena_header_is_object_start, classify_heap_generation, classify_heap_space,
-    classify_heap_space_in_range, generation_page_for_addr, materialize_all_promoted_page_runs,
-    old_arena_block_range_index, old_arena_block_ranges, old_arena_page_index_remove_object,
-    old_arena_source_blocks_for_pages, old_arena_walk_objects_on_pages, old_object_page_overlaps,
-    old_object_single_page, old_page_account_dirty_slot, old_page_account_dirty_slots,
-    old_page_account_promoted_object, old_page_account_swept_object, old_page_account_swept_tally,
-    old_page_clear_dirty, old_page_mark_dirty, old_page_meta_snapshot, old_page_summary,
-    old_pages_begin_gc_cycle, old_pages_reset_sweep_accounting, record_arena_object_start,
-    unregister_old_object_pages, unregister_old_objects_batch, HeapGeneration, HeapSpace,
-    OldArenaPageObjectCursor, OldArenaSourceBlockSelection, OldPageMeta, OldPageSummary,
-    OldPageSweepTally,
+    classify_heap_space_in_range, generation_page_for_addr,
+    materialize_promoted_page_runs_for_object, old_arena_block_range_index, old_arena_block_ranges,
+    old_arena_page_index_remove_object, old_arena_source_blocks_for_pages,
+    old_arena_walk_objects_on_pages, old_object_page_overlaps, old_object_single_page,
+    old_page_account_dirty_slot, old_page_account_dirty_slots, old_page_account_promoted_object,
+    old_page_account_swept_object, old_page_account_swept_tally, old_page_clear_dirty,
+    old_page_mark_dirty, old_page_meta_snapshot, old_page_summary, old_pages_begin_gc_cycle,
+    old_pages_reset_sweep_accounting, record_arena_object_start, unregister_old_object_pages,
+    unregister_old_objects_batch, HeapGeneration, HeapSpace, OldArenaPageObjectCursor,
+    OldArenaSourceBlockSelection, OldPageMeta, OldPageSummary, OldPageSweepTally,
 };
 
 #[cfg(test)]
 pub(crate) use page_meta::{
     deferred_old_page_registrations_len, generation_page_base,
     old_arena_page_index_clear_for_tests, old_page_meta_for_tests,
-    old_page_meta_snapshot_calls_for_tests, pending_promoted_page_runs, register_block_space,
-    register_promoted_page_run, reset_old_page_meta_snapshot_calls_for_tests,
+    old_page_meta_snapshot_calls_for_tests, pending_promoted_page_runs, promoted_page_run_pending,
+    register_block_space, register_promoted_page_run, reset_old_page_meta_snapshot_calls_for_tests,
     DEFERRED_OLD_PAGE_REGISTRATION_CAP, GENERATION_CLASS_SHIFT, GENERATION_PAGE_SIZE,
 };

crates/perry-runtime/src/arena/page_meta/mod.rs

@@ -759,16 +759,37 @@ pub(crate) fn materialize_promoted_page_runs(pages: impl IntoIterator<Item = usi
     }
 }
 
-/// Expand every pending run.
+/// Expand the pending runs of every page `[header, header + total_size)`
+/// overlaps. One `Cell` read when no run is pending anywhere.
 ///
-/// Called before anything that can reshape an old-gen block — the full and
-/// budgeted cycle constructors, whose sweep frees objects in place and whose
-/// holes are then refilled by `old_free` with objects of a different size. A
-/// run's `last_header` is an address remembered at promotion time; once
-/// boundaries inside it can move, that address stops being a header boundary.
-/// Expanding first keeps the run representation confined to the window in which
-/// promoted blocks are immutable: between the promotion and the next old-gen
-/// sweep.
+/// #10182: this is how a sweep keeps a run exact without expanding every run
+/// up front. A run's `last_header` is an address remembered at promotion time,
+/// and it stops describing the page once a boundary inside it can move — which
+/// happens only when an object on the page dies: the sweep invalidates its
+/// header (`PendingOldUnregister::defer`, `invalidate_dead_old_arena_header`)
+/// and `old_free` may later refill the hole with an object of another size.
+/// Both removers expand the object's pages first, so a page keeps its run
+/// exactly while every object on it survives, and a block the sweep reclaims
+/// whole discards its runs unexpanded (`unregister_old_block_pages`).
+#[inline]
+pub(crate) fn materialize_promoted_page_runs_for_object(header: usize, total_size: usize) {
+    if !OLD_GEN_PAGE_PROMOTED_RUNS_NONEMPTY.with(Cell::get) || header == 0 || total_size == 0 {
+        return;
+    }
+    // #7624: every expansion runs against a flushed deferral buffer (see
+    // `expand_promoted_run`). A synchronous full has no births to defer; a
+    // budgeted one can, across its mutator windows.
+    flush_deferred_old_page_registrations();
+    let first = generation_page_for_addr(header);
+    let last = generation_page_for_addr(header + total_size - 1);
+    materialize_promoted_page_runs(first..=last);
+}
+
+/// Expand every pending run. Test builds only since #10182: full and budgeted
+/// cycles used to call this from their constructor, expanding the runs of
+/// blocks the sweep then released whole; the sweep now expands a page only
+/// where it reshapes one ([`materialize_promoted_page_runs_for_object`]).
+#[cfg(test)]
 pub(crate) fn materialize_all_promoted_page_runs() {
     if !OLD_GEN_PAGE_PROMOTED_RUNS_NONEMPTY.with(Cell::get) {
         return;
@@ -815,6 +836,12 @@ pub(crate) fn pending_promoted_page_runs() -> usize {
     OLD_GEN_PAGE_PROMOTED_RUNS.with(|runs| runs.borrow().len())
 }
 
+/// Is `page`'s object list still DESCRIBED by a pending run? Tests only.
+#[cfg(test)]
+pub(crate) fn promoted_page_run_pending(page: usize) -> bool {
+    OLD_GEN_PAGE_PROMOTED_RUNS.with(|runs| runs.borrow().contains_key(&page))
+}
+
 pub(crate) fn unregister_block_generation(base: usize, size: usize) {
     if base == 0 || size == 0 {
         return;

crates/perry-runtime/src/arena/promote.rs

@@ -465,6 +465,14 @@ fn stamp_and_index_block(block: &ArenaBlock, liveness: PromotionLiveness) -> (us
     let mut run_count = 0usize;
     let mut run_bytes = 0usize;
     let mut run_headers: Vec<usize> = Vec::new();
+    // #10182: record the census facts of an untraced promotion's blocks for the
+    // promoted-cohort full that may follow at this safepoint (`adopt_census`).
+    let mut census = if describe {
+        crate::gc::AdoptableBlockBuilder::begin(block.data as usize, block.offset, block.size)
+    } else {
+        crate::gc::AdoptableBlockBuilder::begin(block.data as usize, 0, block.size)
+    };
+    let mut stopped_early = false;
 
     let mut offset = 0usize;
     while offset < block.offset {
@@ -478,6 +486,7 @@ fn stamp_and_index_block(block: &ArenaBlock, liveness: PromotionLiveness) -> (us
         if total < crate::gc::GC_HEADER_SIZE || total > block.size - aligned {
             // Same guard the arena walkers use: an implausible size means we
             // have run off the end of the initialised region.
+            stopped_early = true;
             break;
         }
         let obj_type = unsafe { (*header).obj_type };
@@ -489,6 +498,7 @@ fn stamp_and_index_block(block: &ArenaBlock, liveness: PromotionLiveness) -> (us
         // be looked at again.
         unsafe {
             crate::gc::stamp_header_promoted_in_place(header);
+            census.note(header, aligned);
         }
         objects += 1;
 
@@ -552,6 +562,7 @@ fn stamp_and_index_block(block: &ArenaBlock, liveness: PromotionLiveness) -> (us
             run_bytes,
         );
     }
+    census.finish(!stopped_early);
     debug_assert_eq!(
         offset, block.offset,
         "a promoted block did not parse to its own bump offset — its tail is \

crates/perry-runtime/src/arena/tests.rs

@@ -1546,8 +1546,8 @@ fn deferred_registration_flush_sites() {
             "expand_promoted_run",
             "expands a DESCRIBED promoted page into the object list. Every \
              caller has already flushed: the four readers/removers do so as \
-             their #7624 obligation, `materialize_all_promoted_page_runs` runs \
-             immediately after `old_pages_begin_gc_cycle`, and \
+             their #7624 obligation, `materialize_promoted_page_runs_for_object` \
+             flushes before it expands (#10182), and \
              `register_promoted_page_run` is inside the promotion walk covered \
              by the entry above",
         ),

crates/perry-runtime/src/arena/tests_promoted_runs.rs

@@ -141,10 +141,11 @@ fn a_full_cycle_expands_every_pending_run_before_it_can_sweep() {
         let pages = register_region_runs(&region);
         assert!(pending_promoted_page_runs() > 0);
 
-        // What `GcCycleState::new_full` calls. A run's bounds are addresses
-        // captured at promotion; once the sweep can free objects inside the
-        // block and `old_free` can refill the holes, those bounds stop being
-        // object boundaries.
+        // What full and budgeted cycle constructors called before #10182 (the
+        // sweep now expands a page only where it reshapes one). A run's bounds
+        // are addresses captured at promotion; once the sweep can free objects
+        // inside the block and `old_free` can refill the holes, those bounds
+        // stop being object boundaries.
         materialize_all_promoted_page_runs();
 
         assert_eq!(pending_promoted_page_runs(), 0);

crates/perry-runtime/src/arena/walk.rs

@@ -268,6 +268,27 @@ impl ArenaObjectCursor {
         self.finished
     }
 
+    /// Hand out the next whole block as `(global block index, data, offset,
+    /// size)`, snapshotted exactly as `next_budgeted` would walk it, for a
+    /// caller that parses the block itself (#10182). Honours `set_skip_blocks`.
+    /// Only valid at a block boundary ([`Self::at_block_boundary`]); `None`
+    /// once the cursor is exhausted.
+    pub(crate) fn next_whole_block(&mut self) -> Option<(usize, usize, usize, usize)> {
+        debug_assert!(self.at_block_boundary());
+        if !self.ensure_current_block() {
+            return None;
+        }
+        let block = self.current_block.take()?;
+        self.offset = 0;
+        Some((block.block_idx, block.data, block.offset, block.size))
+    }
+
+    /// No block has been entered by `next_budgeted` (the start of the walk, or
+    /// the point right after a block was exhausted).
+    pub(crate) fn at_block_boundary(&self) -> bool {
+        self.current_block.is_none() && self.offset == 0
+    }
+
     /// Never enter the blocks whose global index is set in `skip` (#10182).
     /// Must be installed before the first `next`; a block the cursor is
     /// already inside is not affected.

crates/perry-runtime/src/gc/barrier/mod.rs

@@ -220,7 +220,7 @@ unsafe fn process_dirty_slot_work(
     *changed |= *slot != before;
 }
 
-fn dirty_slot_ranges_for(
+pub(super) fn dirty_slot_ranges_for(
     range: HeapSlotRange,
     dirty_pages: &crate::fast_hash::PtrHashSet<usize>,
     stats: &mut RememberedSetTraceStats,