perf(gc): cut the copying minor's per-live-object trace cost (straight-line object scan, hoisted per-object facts, exact memos)

changelog.d/11676-gc-per-object-trace-cost.md

@@ -0,0 +1,10 @@
+perf(gc): the copying minor's per-live-object trace cost is cut by about a third (#11549). A copying minor over a fully-live binary tree (131,076 promoted objects, `btree` n=20) went from 1,472 to 953 instructions per live object, not counting the first-cycle barrier-arming walk that #11668 addresses separately (callgrind, `gc_collect_minor_with_trigger_inner` only). The biggest part was plumbing, not marking. On that tree, instructions fall 24.6% (n=20) and 22.5% (n=40). `gc_ratchet` 02 falls 7.9%, 12 falls 8.7%, retain 4.9% and qs stringify 2.3%. Peak RSS with THP off is within ±1% on every row except moment, whose GC schedule is nondeterministic.
+
+- **Ordinary objects take a straight-line scan** (`gc/copying_object_scan.rs`). The drain used to reach an object's slots through the generic rewrite walk, the `HeapChildSlotIterator` state machine and two indirect visitor calls per slot. It now builds the same slots as a plan: the residual-prototype edge, the shape record, field range and payload selection (`heap_payload_slot_selection_from` itself, so the mask logic has one copy), the carrier note, the keys edge, the meta record, the payload and the overflow fields, in the generic walk's order. It declines only up front (a full trace or layout-scan trace in progress, or a type table that no longer describes `GC_TYPE_OBJECT` this way). In test and debug-assertion builds every object it scans is also enumerated by the generic walk and the two slot lists must match. `gc::tests::copying_object_scan` shows the path is taken, that it still scans with the residual-prototype latch armed, and that a plan that drops a slot is refused.
+- **The parent's remembering question is asked once per object** (`ParentRemembering`), and the slot's generation is classified only when an answer depends on it. On a whole-block promoting cycle (`skip_remembering`), no slot is classified. Before this, every slot paid a page-map probe for an answer nothing read. It is checked against `barrier_parent_needs_remembering` over old, young and malloc parents, with a sabotaged twin.
+- **Raw words consult the mark memo before classifying.** The memo holds only addresses that classified this cycle, and a classification cannot change within a cycle. Every shaped receiver's `keys` word hits it. Test and debug builds re-derive the premise.
+- **`SHAPE_LAYOUTS` memoizes its last pointer-mask answer.** The table is reachable mutably only through `DerefMut`, which clears the memo, so the memo is exact by construction. `gc::tests::shape_layout_table` has a sabotaged twin that keeps the memo across a write.
+- **The old-carrier note skips its generation probe when it would change nothing** (both record flags already set). `gc::tests::copying_object_scan` shows that a promoted receiver still notes its shape, and that claiming every shape noted loses the note.
+- The drain's slot walk is instantiated for its visitor (`visit_gc_rewrite_slots_inline`), and its classification is inlined. Other callers keep the shared `dyn` walk.
+
+Not in this change: the full collection. dotenv's fulls are dominated by the valid-pointer-set build, the sweep and side-table clears, not by tracing. The per-cycle side-table passes that dominate `qs` minors are also not here: the closure box-capture prune, the per-object layout owner prune and its sort, and the remembered-set rebuild.

crates/perry-runtime/src/gc/copying.rs

@@ -1,4 +1,4 @@
-use super::copying_parent_facts::weak_holder_fact;
+use super::copying_parent_facts::{weak_holder_fact, ParentRemembering};
 use super::copying_phase::{
     finalize_dead_copied_minor_from_space_side_allocations, CopyingMinorPhase as Phase,
     CopyingMinorPhaseDiag as PhaseDiag,
@@ -416,10 +416,17 @@ impl CopyingNurseryCollector {
         if addr == self.memo_addr {
             return Some(self.memo_result);
         }
-        let ptr = self.ptrs.classify(addr)?;
+        let ptr = self.ptrs.classify_inline(addr)?;
         Some(self.mark_classified(addr, ptr))
     }
 
+    /// The memo's answer for `addr`, if `addr` is the last address a mark
+    /// classified successfully this cycle. See `memo_addr`.
+    #[inline(always)]
+    pub(super) fn memo_hit(&self, addr: usize) -> Option<usize> {
+        (addr == self.memo_addr).then_some(self.memo_result)
+    }
+
     /// [`mark_addr`](Self::mark_addr) for an address the caller has already
     /// classified: the memo, then the mark, without classifying again.
     #[inline]
@@ -710,17 +717,27 @@ impl CopyingNurseryCollector {
     }
 
     pub(super) unsafe fn scan_object_fields(&mut self, header: *mut GcHeader) {
+        // The common case, written out (#11549): see `gc/copying_object_scan.rs`.
+        if self.scan_plain_object(header) {
+            return;
+        }
         let mut changed = false;
         // LAZY, not eager. Reading the fact once per traced OBJECT regressed
         // all six fixtures (+0.88 % to +5.09 % instructions): a great many
         // traced objects — strings, pointer-free arrays — have no slot to
         // visit at all, and paid for an answer nobody then asked for.
         let mut weak_holder: Option<bool> = None;
-        visit_gc_rewrite_slots(header, |slot| unsafe {
+        // Same laziness as the weak fact, and the same per-object shape: see
+        // `ParentRemembering`.
+        let mut remembering: Option<ParentRemembering> = None;
+        let skip_remembering = self.skip_remembering;
+        visit_gc_rewrite_slots_inline(header, |slot| unsafe {
             slot.record_layout_read();
             let before = *slot.slot;
             let weak = *weak_holder.get_or_insert_with(|| weak_holder_fact(header));
-            self.visit_slot_with_weak_fact(slot.slot, header, weak, slot.external());
+            let remembering =
+                *remembering.get_or_insert_with(|| ParentRemembering::of(header, skip_remembering));
+            self.visit_slot_with_parent_facts(slot, header, weak, remembering);
             changed |= *slot.slot != before;
         });
         if changed {