gc: a pacing full at about the cost of its live mark; promoted-cohort fulls (#10182) [land after #10249] - #10241
Closed
proggeramlug wants to merge 27 commits into
Closed
proggeramlug wants to merge 27 commits into
proggeramlug wants to merge 27 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
proggeramlug
changed the title
added 19 commits
September 14, 2026 06:16
…es it (#10182) A synchronous full used to expand every pending promoted page run in its constructor, including the runs of blocks its block-granular sweep then released whole: 7-8 ms of a 62 ms pacing full on records_array_20m:parse and ~18 MiB of transient page lists. The sweep now expands a page right before it invalidates the first dead header on it (PendingOldUnregister::defer), the same order invalidate_dead_old_arena_header already keeps; a page on which every object survives keeps its run, and a block reclaimed whole discards its runs unexpanded.
The exact census reads every header in the arena. On a pacing full over two promoted 20 MB JSON trees it spent ~15 ms doing it, against ~4 ms for a plain header walk over the same 1.8 M objects: the cost was the per-object cursor call and the per-object recomputation of block constants, not the memory traffic. ValidPointerSetBuilder::census_whole_block parses a block itself (same alignment, stop conditions and walkability filter as next_budgeted), sets start bits through the block's chunk pointers, and applies the pointer range, start count and nursery classification once per block. Budgeted steps and classifier-mode sets keep the per-object walk.
…le-free (#10182) After the object walk, old_free_rebuild_from_live_old_blocks re-parsed every live old block looking for invalidated headers: on a pacing full that keeps one promoted 20 MB JSON tree, a second pass over the whole tree that finds no hole. The whole-block census now records whether a block holds any header that does not parse as an object, and the sweep records where it invalidated one; the rebuild skips a live old block that is hole-free by both, unchanged since the census. Counter: hole_rebuild_blocks_skipped; diag [gc-old-free] rebuild_skipped_blocks=.
A full's sweep called the arena cursor once per object and re-derived the object's block constants (old or general, from-space membership, age bumping) for each one. On a pacing full that keeps one promoted 20 MB JSON tree that was the sweep's largest cost after block skipping. The unbudgeted sweep now parses each block itself, keeps a marked, unpinned, unforwarded object inline with the block constants hoisted, and hands every other header to process_object unchanged. Budgeted sweeps keep the per-object cursor.
The full mark paid, for every traced slot: a page-generation lookup to build GcMutableSlot::external (read only by the copying minor), a thread-local read for the proxy-observation flag, a header re-read to rule out weak-holder classes, a dynamic callback per slot of a range descriptor, and — because the optimizer hoisted it above the arm check — a thread-local address fetch for the layout-scan counters. The slot's generation is now classified when asked, the proxy and weak-holder facts are read once per object, range descriptors are walked inline, and the layout counter's thread-local lives out of line behind the process-wide arm flag. classifier_verify_enabled tests its cached static before its thread-local.
Every membership query of the full mark first binary-searched the census block fences (about 150 on a 20 MB JSON pacing full), and after the per-slot hoists that search was the largest single cost left in the mark. The finished census now also builds a table with one entry per 1 MiB window: the greatest block base at or below the window start and the one base, if any, inside it. Arena blocks are at least 1 MiB and never overlap, so a lookup is a shift, a bounds check and one compare. Sets whose bases share a window, or that span more than 16 GiB, keep the binary search.
mark_field_into_worklist queued every newly marked object, and the drain then dispatched each one only to find a leaf descriptor with no slot to visit. Strings are half the objects of a parsed JSON tree. A pointer-free object that is not a forwarding stub is now marked and not queued; a forwarded one is still queued so its hop is followed.
The copying minor's drain prefetches the header a few worklist entries ahead; the full mark's drain did not, and on a promoted 20 MB JSON tree each header it dequeues is a cold DRAM read. A prefetch has no architectural effect and cannot fault.
…moting safepoint (#10182) Old-reclaim pacing cannot see a promoted tree that dies after its minor: every promotion credits the growth baseline (#7592/#7965). A document parse/scan loop lives in that blind spot, because each result's top-level array is born old and keeps its young contents reachable through remembered slots until a full proves it dead; on records_array_20m:parse every minor promotes two trees, one of them dead, and nothing ever collects them. A full is now due when the bytes promoted since the last full reach max(one base nursery, the old-gen live set that full verified << backoff). It is consulted only right after a nursery minor at a precise safepoint, so the full runs with precise roots and, after an in-place promotion, an empty young generation. It is not an arm of old_reclaim_pressure_due, whose credited baseline and the two tests #10204 broke are untouched. A cohort full that reclaims less than half its cohort doubles the bound (at most three times), so a heap whose promoted bytes stay live pays a logarithmic number of futile fulls; a productive one restores it. Diag: [gc-trigger] promoted_since_full= cohort_bound=, site safepoint_promoted_cohort, [gc-promoted-cohort] full cohort= bound= reclaimed= productive= backoff_shift=.
…just promoted (#10182) The cohort full runs at the same safepoint as the nursery minor whose promotion made it due, and that minor's untraced in-place promotion has just parsed every header of every block it promoted. The full's census parsed them again before the mutator ran a single instruction; on records_array_20m:parse those blocks are two of the three trees the census reads. When the minor's promotion can bring the cohort to its bound, the promotion walk now records for each block it parses whole the census's start bitmap and per-block facts, the flag facts computed by the census's own function after the promotion's own header writes. Only the cohort full started at the same safepoint may adopt a record, only for a block with the same address, bump offset and size, and every record is discarded when the safepoint returns. Test builds re-walk every adopted block with the census walk and assert the record agrees.
…e the sweep's page tally (#10182) A range descriptor (an array's elements, an all-pointer field range) is now walked twice: the first pass starts the header read of every pointer-tagged slot's target, the second marks. account_old_object is inlined into the whole-block sweep's live path.
…the liveness vector (#10182) The hole-free skip copied block_has_live and kept a second per-block vector for the sweep's invalidations. The rebuild now takes a block filter, and an invalidation clears the block's hole-free fact directly.
…he root scan (#10182) A conservative stack scan reads every word from its own stack pointer up, including slots of live frames that no call has written since a deeper frame returned. The one-pass census left a different heap address there than the per-object census did: on records_array_8m:roundtrip every alloc-point full found one more conservative root (8 against 7), kept a dead 7 MB stringify result, lost the 7 MB hole it would have left, and peak RSS rose from 129 to 159 MiB. A full now zeroes 16 KiB of dead stack after building its census; the root counts and the RSS are back to 129 MiB.
proggeramlug
force-pushed
the
gc/full-throughput
branch
from
September 14, 2026 04:19
9a05821 to
d6393e7
Compare
proggeramlug
force-pushed
the
gc/pacing-full-cost
branch
from
September 14, 2026 04:19
b321823 to
93881f6
Compare
Contributor
Author
|
Rebased onto the rebased #10220 (19 commits re-applied cleanly). Authoritative re-measurement on the quiet bench mini instead of the loaded laptop, all 50 JSON matrix rows, best of 3 interleaved rounds per engine against Node 26.5.1 and Bun 1.3.14, with the comparison engines measured in the same run (the
So on the authoritative host the target rows the description reports as missing at +0.8–3.0 % CPU are in fact ahead (0.93–0.97×). The one real miss this stack introduces is Still open before this can leave draft: the ungated gc-ratchet wall regressions the description lists ( |
proggeramlug
changed the title
…oted (#10182) A promoted-cohort full resets the untraced-promotion budget, so a workload that turns from building a live set to churning never re-measures: every minor promotes the churn untraced and every cohort full marks the whole live set to reclaim it (14_grow_then_churn: 13 cohort fulls, 0 copied). The full's sweep now measures, block by block, how much of what the minor at the same safepoint promoted is still reachable (the blocks the promotion walk recorded for census adoption). Below the in-place promotion threshold that figure replaces the young-survival predictor, so the next minor evacuates and measures instead of promoting on faith. A parse loop measures ~1000 there (the last minor promoted the tree being parsed) and keeps its untraced promotions.
added 7 commits
September 14, 2026 07:33
…10182) records_array_20m:parse measures 500 permille over the blocks its last minor promoted, not ~1000: that minor promotes the dead previous tree with the live current one, because the dead tree's born-old top array keeps its records reachable through remembered slots. Fed to the predictor, that made the next minor evacuate and copy both trees (~95 ms, +50 MB RSS). A minor measures what it reaches from roots and from every old object on its dirty pages, so the full's survival equals the minor's exactly when no object on those pages that the full's mark left unmarked holds a dirty slot into the promoted blocks. The minor now notes its remembered set (page keys only) while the promotion census records; the full checks those parents after its mark, before its sweep, and the predictor is fed only when the view is exact.
A copying minor tenures an object only after it survived a minor, so a cohort full scheduled for tenured bytes is futile by the only measurement a minor has. 12_large_live_set's one cohort full was reached by 21.2 MB of copy-tenured bytes over a 16 MB bound and reclaimed 8.3 MB; every cohort full on the JSON rows was reached by in-place promotions alone. The old-reclaim baseline credit still takes every promoted byte.
proggeramlug
changed the title
proggeramlug
marked this pull request as ready for review
September 14, 2026 08:20
This was referenced Sep 14, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for #10182, stacked on #10220 (base
gc/full-throughput,9a05821b9e).Two fixes added after the first review
A cohort full feeds the survival of the blocks its own minor promoted. Every full resets the untraced-promotion budget, so with cohort fulls running every ~26 MB the budget never reached the size that forces a measuring minor. A stale 1000-permille reading from a program's growth phase then kept every later minor promoting untraced, and the fulls did all the reclaiming: on
14_grow_then_churn, 26 of 28 minors promoted untraced and 13 cohort fulls each re-marked the whole live cache.The fix measures, with the full's own mark, how much of what the minor at the same safepoint promoted is still reachable, and feeds that to the promotion predictor. It is fed only when it provably equals what a minor would have measured: no parent left unmarked by the full, on the minor's own dirty old pages, refers into the promoted blocks. On the JSON rows that test fails (the dead predecessor tree's born-old array still refers into them), so nothing is fed and their schedules do not change. On the probe it holds, the predictor learns 4 permille, and the minors go back to copying.
Feeding the figure unconditionally below the promote threshold was measured and rejected: 20 MB parse +20.0 % CPU and +54.9 MiB, sparse +19.6 %, scan +18.4 %,
records_object_20m:parse+18.8 %, 8 MB scan +14.1 %.Only in-place promotions fill the cohort. A copying minor tenures an object only after it has already survived a minor, so a full scheduled for tenured bytes is futile by the one measurement a minor has.
12_large_live_set's only cohort full was reached by 21.2 MB of tenured bytes and reclaimed 8.3 MB; every cohort full on the JSON rows was reached by in-place promotions alone. The old-reclaim baseline credit still takes every promoted byte.What changed
The full gets cheaper (every soundness claim has a planted test and a sabotaged twin that shows the harm):
ea8a8ff59cGcCycleState::new_fullno longer expands every described promoted page run.PendingOldUnregister::deferexpands the runs on a dead object's pages before it invalidates the header.sweep_described_runs/sabotaged_expansion_order_keeps_counting_a_freed_object31b07d915fcensus_whole_block: one loop that parses headers, sets start bits and records block facts.census_whole_block(compares against the stepped census) /sabotaged_whole_block_census_is_caught_by_the_comparisonef57bdb203,5c6d287e57sweep_hole_rebuild/sabotaged_hole_census_loses_the_hole90dd47f908sweep_whole_block(compares against the stepped sweep) /sabotaged_whole_block_liveness_is_caught_by_the_comparisonf6fd156527proxy_trace_activeand the weak-holder fact once per object and walks Range descriptors inline.GcMutableSlot::externalis classified lazily. The layout-slot counter checks its armed flag inline.mark_slot_hoists/sabotaged_weak_holder_fact_keeps_the_weak_target_alived06177d792census_block_windows(compares against the binary search) /sabotaged_block_window_lookup_is_caught_by_the_comparison8b8119716aleaf_marks/sabotaged_forwarded_leaf_loses_its_target9106c65ccb,0b0fec712890f99d2ce5scrub_dead_stack(planted sentinel below SP)Promoted-cohort fulls (
58159c264d,392fce3104;gc/promoted_cohort.rs,gc/trace/adopt_census.rs):gc_safepoint_moving_minor, after a copying minor at a precise safepoint, a full runs when the bytes promoted since the last full reachmax(nursery cap, old live at last full << shift).safepoint_promoted_cohort; its trigger kind isOldGenBytes.shift, capped at 3. A productive full resets it.an_untraced_promotion_credits_the_old_reclaim_baselineandtest_old_reclaim_band_is_proportional_and_promotion_credits_baseline.stamp_and_index_blockrecords each promoted block's census (start bits, counts, type and obligation facts) while it stamps. The cohort full adopts those blocks instead of reading them again.cfg(test), every adopted block is re-walked and compared.PERRY_GC_DIAG=1:[gc-promoted-cohort] full cohort= bound= reclaimed= productive= adopted_census_blocks= backoff_shift=;[gc-trigger]gainspromoted_since_full=/cohort_bound=.promoted_cohort(7 tests, includingsabotaged_backoff_charges_a_retaining_schedule_a_full_per_live_set) andadopt_census(sabotaged_promotion_census_loses_rooted_objects).not_a_gc_pointer(counters).census.rs:PASS1_MARKEDis re-pinned with dated notes (26f90297e5,46ff6866d1).Phase tables
#10220's probe (
fullprobe.ts, one 20 MB tree in old gen, thengc()× 6), ms, min of 5Before and after each change. The steps used intermediate builds and temporary local switches;
materialize/sw_*come from temporary timers.Final head (
90f99d2ce5runtime) against base, production binaries,PERRY_GC_TRACEphase_us, min of 5.alloc_point_old_reclaimwith a forced conservative scan; here it issafepoint_promoted_cohort, which fires earlier.gc().A real pacing full in
records_array_20m:parse(n=4, warm 2)Base has no full on this row, so "before" is #10220's
k=1, floor 16 MBcohort bound as an allocation-point arm (1alloc). Steps are cumulative and use temporary timers, DIAG on, min of 7.atomicis the remembered-set rebuild;ctoris the full's setup, which was mostly run expansion.Final head, production binary,
PERRY_GC_TRACEphase_us, min of 5 (TRACE inflatesmark):Root scan. A forced conservative scan costs about 0.5 ms as a phase. What it really cost was #10220's rebuild skip: a conservative root can pin young objects, so the rebuild could not be skipped (
1alloc17.4 ms against2safepoint0.1 ms on the first full).22-row matrix
Interleaved best-of-3 on the same tree,
cells_gc.txt(n / warm as listed there).9a05821b9e; head (90f99d2ce5runtime); "cohort off" is head withpromotion_may_reach_boundreturningfalselocally.matrix_final1) agrees within ±3 % on every head row; the exception isrecords_array_8m:roundtrip, which swung up to 6 % in both arms.records_array_16k:scanrecords_array_1m:parserecords_array_1m:roundtriprecords_array_1m:scanrecords_array_1m:sparserecords_array_1m:stringifyrecords_array_20m:parserecords_array_20m:roundtriprecords_array_20m:scanrecords_array_20m:sparserecords_array_20m:stringifyrecords_array_8m:parserecords_array_8m:roundtriprecords_array_8m:scanrecords_array_8m:sparserecords_array_8m:stringifyrecords_object_20m:parserecords_object_20m:stringifyrecords_object_8m:parserecords_object_8m:stringifysmall_record:parsewide_1m:parserecords_array_16k:scanrecords_array_1m:parserecords_array_1m:roundtriprecords_array_1m:scanrecords_array_1m:sparserecords_array_1m:stringifyrecords_array_20m:parserecords_array_20m:roundtriprecords_array_20m:scanrecords_array_20m:sparserecords_array_20m:stringifyrecords_array_8m:parserecords_array_8m:roundtriprecords_array_8m:scanrecords_array_8m:sparserecords_array_8m:stringifyrecords_object_20m:parserecords_object_20m:stringifyrecords_object_8m:parserecords_object_8m:stringifysmall_record:parsewide_1m:parseMisses
records_array_8m:scan, 227.1 ms against best 188.7 (base 182.0). Four cohort fulls on an 8-iteration row with 6.7 ms of headroom. No bound that reclaims the 20 MB cohorts spares this row: it promotes the same shape at 8 MB, and one full costs more than its headroom (as gc: make a synchronous full cheaper per live object (#10182) [pacing retry does not meet acceptance] #10220 already noted).records_array_20m:roundtrip, +11.6 % CPU (two cohort fulls). It is still under the best (117 against 162) and saves 29 MiB.records_array_8m:parse/sparsepeak RSS falls from 109 to 97/98 MiB.old_in_use8.56 → 9.98 MB over six fulls); head stays flat at 9.04 MB.parse) and 6 to 7 (sparse) at unchanged CPU.records_array_8m:roundtripthen kept a dead 7 MB string and its hole: 129 → 159 MiB, deterministic, bisected to31b07d915f. With the scrub the root counts match base again, at 129 MiB.gc-ratchet, base against head
gc_ratchet.py measure --repeats 7,PERRY_NO_AUTO_OPTIMIZE=1, ownPERRY_RUNTIME_DIRper arm.shared_cirows); head has those same 30 plus 7 new ones, all in14_grow_then_churn.Why
14_grow_then_churnchanges.alloc_point_old_reclaimfull. After that the heap is small, so 15 of the remaining minors copy (330 objects) and free from-space; that is the evacuating minor'sfreed_bytes.copied_*and the minor's ownfreed_bytesgo to 0 andpromoted_*rises.Ungated metrics (medians of 7):
14_grow_then_churn14_grow_then_churn14_grow_then_churn12_large_live_set12_large_live_set12_large_live_setruns one futile cohort full (cohort 21.2 MB, reclaimed 8.3 MB) with no adopted blocks, so it pays a full census.09_try_catch_roots(0.88×),03_cross_gen_writes(0.94×) and04_dead_after_deep_stack(0.95×), all faster.Tests and gates
RUST_TEST_THREADS=1 cargo test --release -p perry-runtime --lib(at90f99d2ce5;46ff6866d1and the changelog commit change no Rust):That test asserts a
debug_assertionspanic and fails under--releaseon base too. Every new test above passes, including every sabotaged twin.SKIP_COMPILE_GATES=1 scripts/run_lint_gates.shat46ff6866d1:Compile tier, run step by step:
RUSTFLAGS="-D warnings" cargo check -p perry --binsRUSTFLAGS="-D warnings" cargo check --workspace --all-targets …could not compile perry-runtime (lib) due to 2 previous errors/(lib test) due to 3 previous errors, all atobject/global_this_webassembly.rs:192/201/662, as on main. Without-D warnings, perry-runtime all-targets reports only those three warnings.cargo clippy -p perry --binscargo clippy --workspace …./scripts/regen_api_docs.sh+ drift checkdocs/api/perry.d.ts/docs/src/api/reference.mddrift, as on main; both files restoredThat is main's three reds and nothing else.
python3 scripts/gc_runtime_root_holders.py:Seeded stress on worker cells.
PERRY_GC_SCHEDULE_SEED=1..4 PERRY_GC_SCHEDULE_RATE=0.2 PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=32, plus an unseeded seed-0 control, with the worker'sverifymode.records_array_8m:sparsemakes no copying minor at n=2, so the quarantine protected nothing on that cell.Seeded stress on
test_gap_gc*(same knobs, seeds 1–4, compared with the unseeded perry output and node):test_gap_gc_staging_args_rootingseed 4 dies withSIGBUS: a stale use of retired from-space at minor #0 (obj_type=2 size=40). The base binary built from9a05821b9efaults the same way, 3 of 3 runs.call_argument_rooting,http2_pending_event_callback_rooting,net_once_flags_rekey,next_request_importandrest_argument_rooting(undefined symbols against a bare runtime dir), andnamespace_and_computed_dispatch_rooting(relative fixture import).test_gap_gc_alloc_point_no_move(test_gap_gc_alloc_point_no_move.ts does not finish compiling on main (>23 min); #7682 coverage may be dark #8906).PERRY_GC_VERIFY_EVACUATION=1on all 22 worker cells, seeds 1 and 2 at rate 0.2,verifymode:[gc-verify] … evacuation_ok(verify lines = minors, 1–972 per run).records_array_8m:scan.Premises in the brief that did not hold
1allocagainst2safepoint), and the head regime has one more minor than base.records_array_8m:scanhas the same shape at 8 MB: four fulls with 6.7 ms of headroom.k=1, floor 16 MBarm at an allocation point. The bound here reaches 167 MiB at 30–32 ms per full.Process notes
/tmp/x.err.