gc: a full mark-sweep promotes its Eden survivors in place; bounded promoted cohort (#10182) [does not meet acceptance]

changelog.d/10204-gc-full-promotes-eden-survivors.md

@@ -0,0 +1,10 @@
+**gc: a synchronous full mark-sweep promotes its young generation in place, and a nursery collection whose promotion would exceed a promoted-cohort bound runs as that full (#10182). Draft: does NOT meet its acceptance bar — see below.**
+
+Mechanism:
+
+- `GcCycleState::maybe_promote_young_after_full` (`gc/cycle.rs`): after the full's sweep and before its remembered-set clear, a synchronous, generational, precise-root full with complete old→young tracking retags every in-use Eden/survivor block to old-gen and finishes the promotion through the copying minor's `arena/promote.rs` machinery (`retag_young_for_in_place_promotion` / `finish_in_place_promotion`, `PromotionLiveness::AssumeAllLive`). It promotes only when the sweep's live from-space bytes are ≥ 95% (`PROMOTE_SURVIVAL_THRESHOLD_PERMILLE`) of the young bytes still in use after the sweep — the bytes the promotion would carry. The sweep invalidates the headers of dead young objects when a promotion is planned (`IncrementalSweepState::invalidating_dead_young_headers`), so the described page runs index exactly the survivors. A promoting full publishes a zero from-space live share (#7901) and leaves no young generation.
+- Promoted-cohort bound (`gc/policy.rs`): bytes promoted since the last full, bounded by `max(one base nursery cap, 2 × old live at the last full)`. It is NOT an arm of `old_reclaim_pressure_due` (that reinstates #7592/#7965's futile full and breaks the two tests pinning it); `promoting_full_preempts_nursery_minor` consults it one promotion ahead at the nursery safepoint.
+- The full's young census is not fed to the copying minor's predictor: measured to turn later minors into evacuations (20m roundtrip 131 → 170 ms, 8m scan 338 → 415 ms) with no RSS gain.
+- Counters: `full_promotion_cycles`, `full_promotion_declined_cycles`, `full_promoted_objects`, `full_promoted_bytes`, `nursery_minors_preempted_by_full`; diag lines `[gc-full-promote] plan|promoted|declined`, `[gc-trigger] kind=PromotingFull`, `promoted_since_full=` / `cohort_bound=`.
+
+Measured (JSON matrix, 22 rows, interleaved best of 3, same tree, loaded host): `records_array_20m:roundtrip` 189 ms / 284 MiB → 128 ms / 193 MiB (node/bun best 162 / 261); 8m parse/sparse RSS 109 → 97–98 MiB at equal CPU; but `records_array_20m` parse/scan/sparse and `records_object_20m:parse` regress 1.5–2.5× CPU (past the node/bun best) and `records_array_8m:scan` 1.8–2.1× CPU. The fulls that pay for the RSS mark a live 29–58 MB tree; that mark alone exceeds the rows' CPU lead. gc-ratchet: `14_grow_then_churn` changes seven gated counters (its 1 MB nursery env makes the cohort floor 1 MB, so pre-emption replaces 4 minors with declined fulls: peak RSS −59%, wall ×2.1); `11_collect_at_depth` copies 0.2% fewer objects (manual `gc()` fulls now promote); every other probe's gated counters unchanged.

crates/perry-runtime/src/gc/cycle.rs

@@ -598,6 +598,19 @@ pub(super) struct GcCycleState {
     sweep: Option<SweepTraceStats>,
     freed_bytes: u64,
     outcome: Option<GcCollectOutcome>,
+    /// #10182: may this cycle promote its young generation in place when its
+    /// own sweep measures it live? Decided in the constructor because the
+    /// SWEEP has to know (it invalidates the headers of the dead young objects
+    /// it reclaims so the promotion walk can tell survivors apart); the
+    /// decision itself is taken after the sweep, from the measurement.
+    promote_young_in_place: bool,
+    /// From-space occupancy at cycle start — the denominator of that
+    /// measurement. Captured in the constructor, before anything is reclaimed.
+    young_in_use_at_start: usize,
+    /// #10182: did the promotion actually run? The census `publish_reclaim_outcome`
+    /// hands the next copying minor depends on it — after a promotion the
+    /// from-space live share is zero, because there is no from-space left.
+    promoted_young_in_place: bool,
 }
 
 impl GcCycleState {
@@ -639,10 +652,29 @@ impl GcCycleState {
         // "already traced"). Every black birth is also pushed as a mark
         // seed — see `gc_note_black_birth`.
         super::barrier::GC_BIRTH_EXTRA_FLAGS.with(|cell| cell.set(GC_FLAG_MARKED));
+        let progress_kind = trigger_kind.progress_kind(GcCollectionKind::Full);
+        // #10182: a full mark-sweep is the one collection that can promote its
+        // Eden survivors for free — it has already proven them live, and its
+        // own sweep has already reclaimed everything around them. Whether it
+        // MAY is `full_promotion_planned` (one gate, shared with the safepoint
+        // pre-emption); whether it DOES is decided after the sweep, from the
+        // measurement.
+        let promote_young_in_place = super::full_promotion_planned(progress_kind);
+        let young_in_use_at_start = if promote_young_in_place {
+            crate::arena::copying_from_space_in_use_bytes()
+        } else {
+            0
+        };
+        if promote_young_in_place && crate::gc::gc_diag_enabled() {
+            eprintln!(
+                "[gc-full-promote] plan young_bytes={young_in_use_at_start} survivor_bytes={}",
+                crate::arena::copying_active_survivor_in_use_bytes()
+            );
+        }
         Self {
             collection_kind: GcCollectionKind::Full,
             trigger_kind,
-            progress_kind: trigger_kind.progress_kind(GcCollectionKind::Full),
+            progress_kind,
             phase: GcCyclePhase::BuildValidPointerSet,
             trace,
             active_elapsed: start.elapsed(),
@@ -661,6 +693,9 @@ impl GcCycleState {
             sweep: None,
             freed_bytes: 0,
             outcome: None,
+            promote_young_in_place,
+            young_in_use_at_start,
+            promoted_young_in_place: false,
         }
     }
 
@@ -722,6 +757,9 @@ impl GcCycleState {
             sweep: None,
             freed_bytes: 0,
             outcome: None,
+            promote_young_in_place: false,
+            young_in_use_at_start: 0,
+            promoted_young_in_place: false,
         }
     }
 
@@ -1497,7 +1535,9 @@ impl GcCycleState {
                 .with_dead_collection_finalize(
                     full_trace,
                     full_trace && !self.progress_kind.is_budgeted(),
-                ),
+                )
+                // #10182: see `GcCycleState::promote_young_in_place`.
+                .invalidating_dead_young_headers(self.promote_young_in_place),
             );
         }
         let done = self
@@ -1527,6 +1567,11 @@ impl GcCycleState {
             trace.old_pages = crate::arena::old_page_summary();
         }
         self.sweep = Some(sweep);
+        // #10182: the young generation is proven and its garbage is reclaimed;
+        // this is the one moment a non-moving full can hand its survivors to
+        // old-gen for nothing. Before the remembered-set clear below, which is
+        // exact precisely because no young generation remains afterwards.
+        self.maybe_promote_young_after_full(&sweep);
         // #7598: seed promote-on-first-copy from THIS completed collection.
         // Every cycle reaching here is a full or a non-copying minor — the two
         // blind spots of `retune_after_scavenge`, which only copying minors
@@ -1548,6 +1593,139 @@ impl GcCycleState {
         self.phase = GcCyclePhase::Reclaim;
     }
 
+    /// #10182: let a full mark-sweep promote its Eden survivors in place.
+    ///
+    /// # Why here
+    ///
+    /// A full is non-moving and promotes nothing (#7592's latch comment in
+    /// `gc/mod.rs`), so a parse/scan loop over document-sized inputs strands a
+    /// dead tree per iteration: the copying minor correctly measures the tree
+    /// live, promotes it untraced (#7888), and it dies one iteration later in
+    /// old-gen where only a full can reclaim it. The cohort bound below makes
+    /// that full happen; this is what stops the full from handing the NEXT
+    /// tree straight back to a copying minor that would evacuate all 58 MB of
+    /// it (survival drops as soon as two trees share Eden).
+    ///
+    /// # Ordering, and why it is this and not another
+    ///
+    /// Directly after the sweep completes, before `ReclaimSubphase::RememberedSet`:
+    ///
+    ///  * **After the sweep** because the sweep is what makes the blocks
+    ///    promotable. It has finalized and invalidated every dead young object
+    ///    on them (`invalidating_dead_young_headers`), reset and released the
+    ///    blocks that held nothing live, and left the survivors' page state
+    ///    correct. Promoting before it would hand old-gen the garbage too.
+    ///  * **Before the remembered-set clear** because that clear is
+    ///    unconditional on a full and is *exact* only once no young generation
+    ///    remains — which is true here by construction: `retag_young_for_in_place_promotion`
+    ///    takes every in-use Eden and survivor block, both semispaces.
+    ///  * **Before `publish_reclaim_outcome`**, so `finish_full_old_reclaim_baseline`
+    ///    measures an old generation that already contains the promoted bytes.
+    ///    That is also why nothing calls `credit_promoted_bytes_to_old_baseline`
+    ///    here: the credit exists so a MINOR's promotion does not read as
+    ///    old-gen growth, and a full overwrites the baseline outright moments
+    ///    later. Crediting first would only add the bytes to the promoted
+    ///    cohort that the same function then resets to zero — and resetting it
+    ///    is right, because a cohort a full just traced is verified, not
+    ///    assumed.
+    ///
+    /// # Liveness
+    ///
+    /// `PromotionLiveness::AssumeAllLive`, which after this sweep means
+    /// "everything still parseable", because the sweep zeroed the `obj_type` of
+    /// everything it reclaimed on these blocks. The cheap described page-runs
+    /// are therefore exact rather than approximate, and the traced path's
+    /// per-object header list — ~1M entries for a 58 MB tree — is not built.
+    fn maybe_promote_young_after_full(&mut self, sweep: &SweepTraceStats) {
+        if !self.promote_young_in_place {
+            return;
+        }
+        let young_bytes = self.young_in_use_at_start;
+        let live_bytes = sweep.arena_live_from_space_bytes as usize;
+        // The denominator is what the promotion would CARRY, not what the cycle
+        // started with: the sweep has already reset and released every young
+        // block that held nothing live, so those bytes cannot become old-gen
+        // garbage. What remains in use is exactly the set `retag_young_for_in_place_promotion`
+        // captures, and the threshold keeps the meaning it has on the copying
+        // minor — at most 5% of the promoted bytes are dead. Measured on
+        // `records_array_20m:parse`: 58.0 MB young at cycle start, 29.0 MB live,
+        // 29.7 MB still in use after the sweep (976‰ of what would be promoted).
+        //
+        // ★ The measurement is deliberately NOT fed to the copying minor's
+        // predictor (`note_young_survival`). The young census of a full is taken
+        // at whatever point the full runs, and on the #10182 rows that is a
+        // point where Eden still holds a parse result the JSON construction
+        // grace (`gc/json_defer.rs`) let die: 7‰ on `records_array_20m:roundtrip`,
+        // 333‰ on `records_array_8m:scan`. Fed to the predictor it turns the
+        // next minors into full evacuations of the following tree. Measured
+        // interleaved on the same binary (best of 3): 20m roundtrip 169.8 ms
+        // fed vs 131.0 ms not fed, 8m scan 415.2 ms / 168 MiB fed vs 338.0 ms /
+        // 155 MiB not fed, every other row of the 22-row matrix within noise.
+        let promotable_bytes = crate::arena::copying_from_space_in_use_bytes();
+        if !super::full_promotion_survival_holds_up(promotable_bytes, live_bytes) {
+            super::note_full_promotion_declined();
+            if crate::gc::gc_diag_enabled() {
+                eprintln!(
+                    "[gc-full-promote] declined young_bytes={young_bytes} live_bytes={live_bytes} \
+                     promotable_bytes={promotable_bytes} carried_live_permille={} eden_live_bytes={} \
+                     eden_dead_bytes={} arena_live_bytes={} freed_bytes={}",
+                    super::full_young_survival_permille(promotable_bytes, live_bytes),
+                    sweep.eden_live_bytes,
+                    sweep.eden_dead_bytes,
+                    sweep.arena_live_bytes,
+                    sweep.freed_bytes,
+                );
+            }
+            return;
+        }
+        let promotion = crate::arena::retag_young_for_in_place_promotion(false);
+        if promotion.is_empty() {
+            // Nothing in use to promote: the sweep reset every young block. The
+            // retag captured nothing, so there is nothing to undo.
+            super::note_full_promotion_declined();
+            return;
+        }
+        let reserved_bytes = promotion.reserved_bytes();
+        let blocks = promotion.block_count();
+        super::note_promoted_young_capacity(reserved_bytes);
+        let stats = crate::arena::finish_in_place_promotion(
+            promotion,
+            crate::arena::PromotionLiveness::AssumeAllLive,
+        );
+        self.promoted_young_in_place = true;
+        super::note_in_place_promotion(stats.bytes, stats.live_bytes, stats.objects);
+        super::note_full_promotion(stats.bytes, stats.objects);
+        // Deliberately NOT `instruments::note_copying_minor_moved`: that bumps
+        // `copying_minor_cycles`, and a full counted as a copying minor is
+        // #7025's shape — a liveness counter summing two collectors, so a cell
+        // can pass having run no copying minor at all.
+        if let Some(trace) = self.trace.as_mut() {
+            trace.old_pages = crate::arena::old_page_summary();
+        }
+        if crate::gc::gc_diag_enabled() {
+            eprintln!(
+                "[gc-full-promote] promoted blocks={blocks} objects={} bytes={} live_objects={} \
+                 promoted_live_bytes={} reserved_bytes={reserved_bytes} young_bytes={young_bytes} \
+                 live_bytes={live_bytes} carried_live_permille={} cycles={} declined={}",
+                stats.objects,
+                stats.bytes,
+                stats.live_objects,
+                stats.live_bytes,
+                super::full_young_survival_permille(promotable_bytes, live_bytes),
+                super::full_promotion_cycles(),
+                super::full_promotion_declined_cycles(),
+            );
+        }
+        debug_assert_eq!(
+            crate::arena::copying_from_space_in_use_bytes(),
+            0,
+            "a promoting full must leave the young generation EMPTY: the \
+             remembered-set clear below is exact only if nothing young remains, \
+             and the next copying minor would otherwise evacuate a nursery that \
+             still holds the tree this full just proved live"
+        );
+    }
+
     fn step_reclaim(&mut self, budget: GcWorkBudget) {
         self.reclaim_state
             .get_or_insert_with(ReclaimCycleState::new);
@@ -1750,7 +1928,15 @@ impl GcCycleState {
         let (arena_live_bytes, from_space_live) = match self.sweep {
             Some(sweep) => (
                 sweep.arena_live_bytes as usize,
-                Some(sweep.arena_live_from_space_bytes as usize),
+                // #10182: a promoting full leaves NO from-space. Publishing the
+                // sweep's from-space share here would hand the next copied
+                // minor bytes to subtract that are no longer in from-space at
+                // all — the #7901 double-charge, one collection removed.
+                Some(if self.promoted_young_in_place {
+                    0
+                } else {
+                    sweep.arena_live_from_space_bytes as usize
+                }),
             ),
             None => (crate::arena::arena_live_allocated_bytes(), None),
         };

crates/perry-runtime/src/gc/diag_sites.rs

@@ -54,11 +54,15 @@ pub(super) fn trigger_decision(site: &'static str, kind: &'static str) {
     let next_malloc = policy::GC_NEXT_MALLOC_TRIGGER.with(Cell::get);
     let old_in_use = crate::arena::old_gen_in_use_bytes();
     let old_free = old_free_bytes();
+    let promoted_since_full = policy::promoted_bytes_since_full();
+    let cohort_bound =
+        policy::promoted_cohort_bound_bytes(policy::GC_OLD_LIVE_AT_LAST_FULL.with(Cell::get));
     eprintln!(
         "[gc-trigger] site={site} kind={kind} arena_total={arena_total} next_base={next_base} armed={armed} \
          from_space={from_space} nursery_cap={nursery_cap} old_in_use={old_in_use} old_free={old_free} \
          old_reclaimable={old_reclaimable} external_side={external} old_baseline={old_baseline} \
          old_band={old_band} old_threshold={old_threshold} old_pending={old_pending} retaining={retaining} \
+         promoted_since_full={promoted_since_full} cohort_bound={cohort_bound} \
          malloc={malloc} next_malloc={next_malloc}"
     );
 }

crates/perry-runtime/src/gc/heap_budget.rs

@@ -164,6 +164,35 @@ budget_scaled_accessor!(
     12,
     2 * 1024 * 1024
 );
+/// #10182: floor of the promoted-but-unverified old-gen cohort — **one base
+/// nursery cap**, budget-scaled like every other threshold in this file.
+///
+/// Not a constant of its own, and not the 64 MB the first draft used. The
+/// cohort this bounds is produced one nursery at a time: every in-place
+/// promotion hands the whole young generation to old-gen, so the quantum of
+/// the thing being bounded IS the nursery cap. A floor of one quantum is the
+/// tightest bound that cannot fire twice for a single promotion, and it is
+/// what makes a parse/scan loop over document-sized inputs reach the
+/// fulls-dominated regime instead of stranding a dead tree per iteration
+/// (#10182). At 64 MB — four quanta on the 16 MB default cap — the arm fired
+/// only after the third or fourth promotion, i.e. after the peak it was meant
+/// to cap had already been set: measured 200/276/210 MiB against main's
+/// 188/256/187 on the three target rows.
+///
+/// The denominator tracks `gc_scavenge_nursery_cap_bytes`'s own default, so a
+/// budget-constrained device gets a floor in the same proportion to its heap
+/// that a desktop gets to the 16 MB cap.
+pub(crate) fn gc_promoted_cohort_floor_dyn_bytes() -> usize {
+    static CACHED: OnceLock<usize> = OnceLock::new();
+    *CACHED.get_or_init(|| {
+        budget_scaled(
+            super::policy::gc_scavenge_nursery_cap_bytes(),
+            1,
+            24,
+            1024 * 1024,
+        )
+    })
+}
 budget_scaled_accessor!(
     gc_copy_promotion_handoff_min_dyn_bytes,
     GC_COPY_PROMOTION_HANDOFF_MIN_BYTES,

crates/perry-runtime/src/gc/mod.rs

@@ -232,8 +232,10 @@ mod native_stack_scan;
 mod promote_in_place;
 use promote_in_place::*;
 pub use promote_in_place::{
-    first_cycle_promotion_attempts, first_cycle_promotion_rollbacks, in_place_promoted_objects,
-    in_place_promotion_cycles, untraced_promoted_objects, untraced_promotion_cycles,
+    first_cycle_promotion_attempts, first_cycle_promotion_rollbacks, full_promoted_bytes,
+    full_promoted_objects, full_promotion_cycles, full_promotion_declined_cycles,
+    in_place_promoted_objects, in_place_promotion_cycles, nursery_minors_preempted_by_full,
+    untraced_promoted_objects, untraced_promotion_cycles,
 };
 /// Instrument-liveness counters (#7604): copying minors completed, objects
 /// relocated, loop back-edge polls reached. Mode-independent — they count what