gc: make a synchronous full cheaper per live object (#10182) [pacing retry does not meet acceptance] - #10220
gc: make a synchronous full cheaper per live object (#10182) [pacing retry does not meet acceptance]#10220proggeramlug wants to merge 11 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (19)
Included review availability: Your plan provides up to 8 included reviews per hour; 5 remain after this review. 📝 WalkthroughWalkthroughSynchronous full collections now use per-block object-start bitmaps, skip empty remembered-set rebuilds when census data proves they are unnecessary, and batch old-generation page accounting before unregister flushes. Tests, documentation, diagnostics, and root-holder audit data cover these changes. ChangesFull GC throughput optimizations
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~60 minutes Change: Refactor Sequence Diagram(s)sequenceDiagram
participant FullGC
participant Census
participant RememberedSet
participant OldGenSweep
participant PageMeta
FullGC->>Census: build block census
Census->>RememberedSet: determine whether rebuild is empty
RememberedSet-->>FullGC: install empty state or scan edges
FullGC->>OldGenSweep: sweep old objects
OldGenSweep->>PageMeta: apply batched page tally before unregister flush
Merge Risk: ⚪ Minimal · up to The documented lookup costs match the implemented paths, and no actionable merge-blocking issue remains. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 67.07% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 82 functions across 16 files. (3 skipped: 3 unsupported.)
✨ Finishing Touches 💡 3📝 Generate docstrings 💡
⚔️ Resolve merge conflicts 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
9a05821 to
d6393e7
Compare
|
Rebased onto Measured on the quiet bench mini (M1, best of 3 interleaved rounds per engine against Node 26.5.1 and Bun 1.3.14, |
|
Correction to my previous comment: the bench-mini run did not measure this PR on its own. The arm I called "mechanism" was this PR plus #10241's commits without its two pacing commits. That arm had the same 10 misses as |
Draft for #10182, stacked on #10217 (base
gc/block-granular-sweep). It makes a synchronous full collection cheaper per live object with three exact changes. The pacing retry that was the point of the cheaper full still does not meet acceptance, so no pacing change is included.What changed
A. Census membership from per-block object-start bitmaps (
gc/trace.rs)ValidPointerSet::containsused to do two binary searches: one over the first key of every census run, then one inside the run. On the probe below that was 1,141 of 2,196 mark samples.The census now opens one entry per arena block it walks (address order) and records each start as a bit:
contains(ptr)is now: range check, a binary search over the block fences, then one bit test. The hit still names its block for #10217's reachability record.enclosing_object(conservative words) takes the floor by scanning the bitmap backwards.Kept as they were: malloc-tracked objects (B-tree),
classifier_mode, and thePERRY_GC_VERIFY_CLASSIFIERdifferential check. The census asserts the alignment contract for every start it records.Chunking is load-bearing. A single contiguous bitmap vector grew into mimalloc's large-page class, and
records_array_1m:sparseread +4.8 MiB peak RSS (68 → 73, 7/7 runs) for a ~100 KB index. With chunks it reads 68 again.B. No remembered-set rebuild when nothing young is live (
gc/verify.rs,gc/cycle.rs,gc/trace/block_skip.rs)The rebuild runs in AtomicFinalize, before the sweep. It remembers a slot of a marked old parent when the child is nursery-classified or a registered malloc object.
So if both of these hold, it can only produce an empty set:
skip_rememberingpremise).In that case the full installs an empty sticky set without walking. The pre-cycle dirty-coverage repair in reclaim is unchanged. Diag:
[gc-remembered-rebuild] full skipped=young_generation_unmarked. Counter:FULL_REMEMBERED_REBUILDS_SKIPPED, classified in the holder inventory.C. Sweep page accounting once per page (
gc/oldgen/sweep_objects.rs,arena/page_meta/sweep_tally.rs)old_page_account_swept_objectallocated an overlapVecand did a page-meta hash lookup per swept old object.The sweep now sums consecutive single-page objects and applies the sum once per page. Multi-page objects keep the old call. The tally is always applied before a page-index flush (step end, or the 4,096-entry auto-flush), because that flush zeroes the accounting of a page it empties. That keeps the order of sums and resets identical on every page.
Cost of one full, before and after each part
Probe
scratchpad/ft/fullprobe.ts: one parsedrecords_array_20m.jsontree (585k objects), promoted to old gen by churn-driven minors, thengc()six times. Last three fulls, ms; two interleaved runs per arm agree within 1 ms.build_valid_pointer_set)trace_worklist)atomic_finalize)Same probe with the tree still young at each full (
parsethengc()): pause 63–68 → 46–51 ms, mark 42–44 → 25–26 ms. The rebuild (6.4 ms) and census (8 ms) are unchanged there, because the live tree is young.Tests
RUST_TEST_THREADS=1 cargo test --release -p perry-runtime --lib:That test asserts a
#[cfg(debug_assertions)]panic, so it cannot pass under--release. It fails on base as well.New tests. Each protective case has a sabotaged twin behind a
#[cfg(test)]switch that shows the harm:start_bitmap: every address (step 4) of every census block is compared with an independent arena walk. The population covers word boundaries, a 600 KB object (the floor search crosses zero words) and an oversized sorted block. Sabotage shifts every bit probe by one unit.full_rebuild_skip: each edge is planted with a raw store and no barrier, so only the rebuild can recover it. Sabotage forces the skip, and the known-live parent's edge is then unremembered. The guard for each child kind (young, malloc) is what prevents the skip in its own case.sweep_page_tally: every planted page's accounting is compared with an oracle computed from each object's known fate. The population has live, pinned and dead objects, multi-page objects, whole dead pages inside live blocks, and more than 4,096 dead objects. Sabotage applies the tally after the page-index flush.valid_pointer_membership_spans_every_census_run_including_the_partial_one_7646now checks block fences instead of run fences, and keeps both membership directions.test_minor_skips_whole_heap_old_to_young_rebuildpins one young object across its full, so that full still walks the old generation.Gates
scripts/run_lint_gates.sh: exactly main's three reds.run_lint_gates: 1 of 77 FAILED (compile tier SKIPPED); 2 CI-only skipped. The failure is[Public benchmark evidence freshness] python3 benchmarks/ci_public_baseline_check.py.-D warningsproduct: ok.-D warningshost-compatible, all targets: FAIL, onlyglobal_this_webassembly.rs:201/662(pre-existing). Because that failure stops the build,cargo check -p perry-runtime --all-targetswas run separately; its only warnings are the same file's lines 192/201/662.docs/api/perry.d.ts+10,reference.md+13). Both files restored.Other checks:
scripts/gc_runtime_root_holders.py:OK — 1425 holder declarations scanned.census.rs:PASS1_MARKEDis re-pinned forgc/cycle.rswith a dated re-audit: the provably-empty branch sits inside the window and adds no GC allocation, relocation, collection or callback.scripts/check_file_size.sh: OK.scripts/check_gc_doc_claims.py: OK.garbage-collector.mdgains a paragraph withgc-symbolmarkers to the new tests.Seeded stress (
PERRY_GC_SCHEDULE_SEED=1..4 RATE=0.2 PROTECT_FROMSPACE=1 DEPTH=32):retired_set=#per seeded run, except8m:sparse, which runs no copying minor and shows 0.retired_set0 and identical minor/full counts. So this probe exercises the new full paths but not the fuzzing.test_gap_gc*: 44 tests ran with the instrument live; 43 are identical to Node and across seeds 1–4.alloc_point_no_moveskipped (test_gap_gc_alloc_point_no_move.ts does not finish compiling on main (>23 min); #7682 coverage may be dark #8906).call_argument,http2_pending_event_callback,namespace_and_computed_dispatch,net_once_flags_rekey,next_request_import,rest_argument.staging_args_rootingSIGBUSes under seed 4 (retired_by_minor=#0) on base and head alike, 3/3 runs each.PERRY_GC_VERIFY_EVACUATION=1: exit 0, no stderr, VERIFY hash equal to the unverified run, on:k=1pacing switch;gc-ratchet
7 repeats, base and head back to back.
minor_cycles,step_cycles,copied_*,promoted_*,freed_bytes,heap_used_bytes).check --profile shared_ci: red on base and head with the same 30 gated rows and identical values.PERRY_GC_DIAG=1runs).JSON matrix: 22 rows, interleaved, best of 3, same tree, cpu ms / peak MiB
Every row is within ±2 % CPU and none is worse on RSS. The per-part runs (A alone, A+B) also stayed inside ±2 % after 7-rep rechecks of their outliers.
Target rows have the same regime on base and head (
PERRY_GC_DIAG=1). None runs a full, so the target rows cannot move on their own.Pacing retry: measured, not shipped
Same cohort bound as #10217, applied through a temporary local switch that is not in any pushed commit: old-reclaim is due when bytes promoted since the last full reach
max(floor, k × old live at last full). Interleaved best of 3, same tree; cpu ms / peak MiB:For comparison, #10217 measured
k=1on the same rows at 337.8–354.2 ms / 194–196 MiB. The cheaper full buys 68–86 ms and 15–17 MiB per row, but not the bar.Regime. Every full is
alloc_point_old_reclaim(OldGenBytes) behind the forced conservative scan. Per-full pause ms [census, mark, rebuild, sweep]:Why no row makes it on both axes:
k=1reaches RSS parity (179 MiB). The two fulls cost ~130 ms of pause against a ~64 ms lead. Fitting would need ≤ 32 ms per full over a 29 MB tree.k=0.5runs fulls. CPU stays under best (227 vs 234.9), but RSS reads 116–119 against 112.k=2andk=0.5each lose it on CPU.Where a k=1 full's time goes now.
sampleof 20m:parse over 60 iterations, symbolized; 1,007 samples in fulls:ValidPointerSetBuilder::step, O(all objects, including the dead promoted tree))GcCycleState::new_full→materialize_all_promoted_page_runs→expand_promoted_runThe last line re-materializes the described page runs of every in-place-promoted block, including blocks the block-skip sweep then releases whole. Deferring that materialization to pages a sweep actually reshapes, and a tighter census walk, are the next levers. They are not attempted here.
Corrections to the brief
find_floorover up to ~20k addresses, ~15 probes."VALID_POINTER_ARENA_RUN_CAPACITY), and they held every censused start, live and dead.contains14.8 ns (runs) vs 2.9 ns (bitmap) per query on the planted population.restore_surviving_dirty_coverageover its pre-cycle dirty snapshot in reclaim.skip_remembering).test_gap_gc*and the ratchet probes (plain and seeded), the rebuild never added a page the dirty-coverage repair missed.enclosing_object… measure both."PERRY_CONSERVATIVE_STACK_SCAN=full, 20m probe): root marking 0.42–0.49 ms head vs 0.47–0.56 ms base.records_array_16k:scanand over 330,000 ontest_gap_gc_symbol_local_rooting, so it is degenerate outside the target rows.scratchpad/ratchet_diff.pycrashes on the current ratchet JSON (correctnessis an object). A local fixed copy was used.Summary by CodeRabbit
Performance
Documentation