From bb698ff9abd6fdad6d5628e4ebfbc61f1cf05924 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 12 Sep 2026 20:29:36 +0200 Subject: [PATCH 1/2] perf(gc): keep wide JSON document storage in the nursery (#10123) Repeated `JSON.parse` of a 50,000-field document held 220 MiB peak RSS against a live set of ~0. Node holds 136 MiB on the same workload, Bun 71. `arena/allocators.rs` already names the failure mode: a large pointer-bearing object is stamped GC_FLAG_TENURED, and a minor never sweeps old-gen, so its cost "is not its own bytes, it is every object it can reach, held live through the remembered set by a container nothing refers to any more". A wide document's property storage and its shape-keys array are exactly that container. Above 16,384 fields each crosses the 128 KB pointer-bearing threshold, is born tenured, and then holds its whole field or key set live long after the document is dead. Measured to the byte with PERRY_GC_CENSUS at 64 parses -- 16,300 fields: 0 retained, 29 MiB; 16,500: 30 retained, 98 MiB; 50,000: 15 retained, 34.3 MB live, 177 MiB. The step lands exactly on the constant, and the census names the retainer: 15 shape-keys arrays holding 750,000 live strings. The same binary under PERRY_GEN_GC=0 reports 336 bytes live, which is the truth about the workload. Admit that storage into the nursery past the threshold, for as long as the copier can still move it (512 KB -- half a nursery block, inside copying::MAX_YOUNG_MOVE_BYTES). The scope is read only from the cold large-object branch of arena_alloc_gc, behind a short-circuiting `&&`, so no allocation hot path gains work. wide_1m:parse 220 MiB -> 40 MiB, 0.20s -> 0.15s. Below both engines on RSS and faster than before on CPU. SCOPED AND TYPE-MASKED ON PURPOSE. Raising the constant globally reaches the same 40 MiB but also moves ordinary ARRAY element storage into the nursery, which other rows neither need nor can afford: records_array_8m:scan 643 -> 710 MiB, 0.91s -> 1.20s. Only a document's own object storage and its keys array are admitted; array element storage keeps the flat threshold. Measured in a SINGLE binary (one build, three arms by env) the scoped change is byte-identical to baseline on records_array_8m:scan, records_array_20m:parse, records_array_1m:scan, records_array_16k:scan and heterogeneous_1m:parse. Three tests hardcoded a field count and then asserted pointer_in_old_gen, so they silently depended on the threshold being 128 KB and failed on their PREMISE rather than their subject. They now derive their width from the governing ceiling and keep covering the old-gen path at any value. perry-runtime: 3627 passed, 0 failed (three consecutive runs). --- ...10123-json-wide-object-birth-generation.md | 48 +++++++++ crates/perry-runtime/src/arena/allocators.rs | 6 +- .../src/gc/tests/helper_stores.rs | 8 +- .../tests/runtime_roots/json_construction.rs | 9 +- .../tests/runtime_roots/json_key_lifetime.rs | 28 ++++- crates/perry-runtime/src/gc/types.rs | 101 ++++++++++++++++++ crates/perry-runtime/src/json/mod.rs | 4 + .../src/object/json_construction.rs | 8 ++ 8 files changed, 202 insertions(+), 10 deletions(-) create mode 100644 changelog.d/10123-json-wide-object-birth-generation.md diff --git a/changelog.d/10123-json-wide-object-birth-generation.md b/changelog.d/10123-json-wide-object-birth-generation.md new file mode 100644 index 0000000000..457d8ad87b --- /dev/null +++ b/changelog.d/10123-json-wide-object-birth-generation.md @@ -0,0 +1,48 @@ +### Fixed + +- **Repeated `JSON.parse` of a wide object held 5x the memory it needed (#10123).** + Peak RSS on `wide_1m:parse` (a 50,000-field document, 64 parses) was **220 MiB against a + live set of ~0** — Node holds 136 MiB and Bun 71 MiB on the same workload. + + `arena/allocators.rs` already states the failure mode: a large pointer-bearing object is + stamped `GC_FLAG_TENURED`, and a minor never sweeps old-gen, so its cost "is not its own + bytes, it is every object it can reach, held live through the remembered set by a container + nothing refers to any more". A wide document's **property storage** and its **shape-keys + array** are exactly that container. Above 16,384 fields each crosses + `LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES` (128 KB), is born tenured, and then holds its + whole field or key set live long after the document itself is dead. + + Confirmed to the byte with `PERRY_GC_CENSUS`, fixtures straddling 131,072 bytes at 64 parses + (retained shape-keys arrays / live / peak RSS): + + | fields | keys array | retained | live | peak RSS | + |---:|---:|---:|---:|---:| + | 16,300 | 130,416 B | 0 | 0.0 MB | 29 MiB | + | 16,500 | 132,016 B | 30 | 22.7 MB | 98 MiB | + | 50,000 | 400,016 B | 15 | 34.3 MB | 177 MiB | + + The census named it outright: `shape_keys_arrays {count: 15}`, `slot_tags {string: 750000}` + — 15 arrays x 50,000 keys = 750,000 live strings. The same binary under `PERRY_GEN_GC=0` + reports **336 bytes live**. + + Fixed by admitting a wide JSON document's own storage into the nursery past that threshold, + for as long as the copier can still move it (`JsonWideBirthScope`, ceiling 512 KB — half a + nursery block, inside `copying::MAX_YOUNG_MOVE_BYTES`). The check is reached only from the + cold large-object branch of `arena_alloc_gc`, behind a short-circuiting `&&`, so no + allocation hot path gains work. + + **`wide_1m:parse`: 220 MiB -> 40 MiB and 0.20s -> 0.15s** — below Node's 136 MiB and Bun's + 71 MiB, and faster than before. + + **Scoped and type-masked deliberately.** Raising the constant globally reaches the same + 40 MiB, but it also moves ordinary ARRAY element storage into the nursery, which other rows + do not need and cannot afford: `records_array_8m:scan` 643 -> 710 MiB with CPU 0.91s -> + 1.20s. Only a document's own object storage and its keys array are admitted. Measured in a + single binary across `records_array_8m:scan`, `records_array_20m:parse`, + `records_array_1m:scan`, `records_array_16k:scan` and `heterogeneous_1m:parse`, the scoped + change is **byte-identical to baseline on every one of them**. + + Three tests that hardcoded a field count and then asserted `pointer_in_old_gen` now derive + their width from the governing ceiling, so the old-gen path stays covered whatever that + constant is — previously they silently depended on it being 128 KB and failed on their + premise rather than their subject. diff --git a/crates/perry-runtime/src/arena/allocators.rs b/crates/perry-runtime/src/arena/allocators.rs index 1e2e87c5e1..28ef1cab56 100644 --- a/crates/perry-runtime/src/arena/allocators.rs +++ b/crates/perry-runtime/src/arena/allocators.rs @@ -428,7 +428,11 @@ pub fn arena_alloc_gc(size: usize, align: usize, obj_type: u8) -> *mut u8 { // slots per minor because of it). let total = gc_padded_total_size(size, align); super::alloc_sample::note(total, obj_type); - if crate::gc::is_large_object_total_size_for_type(total, obj_type) { + // `&&` short-circuits, so the scope check is reached only by an allocation + // the size test has ALREADY called large -- never on the hot path (#10123). + if crate::gc::is_large_object_total_size_for_type(total, obj_type) + && !crate::gc::json_wide_birth_permits(total, obj_type) + { let user_ptr = arena_alloc_gc_old(size, align, obj_type); unsafe { let header = user_ptr.sub(GC_HEADER_SIZE) as *mut GcHeader; diff --git a/crates/perry-runtime/src/gc/tests/helper_stores.rs b/crates/perry-runtime/src/gc/tests/helper_stores.rs index b3752e2f70..d20458f8d8 100644 --- a/crates/perry-runtime/src/gc/tests/helper_stores.rs +++ b/crates/perry-runtime/src/gc/tests/helper_stores.rs @@ -6,8 +6,14 @@ use std::fmt::Write as _; /// born-tenured threshold — see `copying.rs`'s `OLD_BORN_ELEMENTS`. These two /// fixtures assert that a materialized JSON object / regex result array is an /// OLD-generation birth holding young children, so they must actually be one. +/// +/// #10123: a JSON-constructed object's storage is admitted young up to +/// `LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES`, which is HIGHER than the +/// flat pointer-bearing threshold. Derive from the ceiling that actually +/// governs this fixture, or the "is an old-generation birth" premise silently +/// stops holding and the test fails on its premise rather than its subject. const OLD_BORN_FIELDS: u32 = - (crate::gc::LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES / 8) as u32 + 64; + (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES / 8) as u32 + 64; unsafe fn alloc_old_test_map( capacity: u32, diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs index 651dae8111..288a49701b 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs @@ -63,9 +63,10 @@ fn json_construction_wide_old_record_remembers_sparse_pointer_pages() { gc_register_mutable_root_scanner(json_parse_mutable_root_scanner); let source = format!( "{{{}}}", - (0..20_000) + (0..super::json_key_lifetime::fields_born_old()) .map(|i| { - let value = if i % 509 == 0 || i == 19_999 { + let value = if i % 509 == 0 || i == super::json_key_lifetime::fields_born_old() - 1 + { format!(r#"{{"child":{i}}}"#) } else { i.to_string() @@ -87,13 +88,13 @@ fn json_construction_wide_old_record_remembers_sparse_pointer_pages() { let last_child = |value: crate::JSValue| { assert_eq!( crate::object::object_live_slot_count(value.as_pointer()), - 20_000 + super::json_key_lifetime::fields_born_old() as u32 ); let slots = value .as_pointer::() .add(std::mem::size_of::()) .cast::(); - (*slots.add(19_999)).bits() + (*slots.add(super::json_key_lifetime::fields_born_old() - 1)).bits() }; let last_before = last_child(value); assert!(crate::JSValue::from_bits(last_before).is_pointer()); diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs index 271ee03aa3..048572aace 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs @@ -1,10 +1,26 @@ //! Cache eviction must release ownership, including the backing key storage. use super::*; +/// Field count whose storage exceeds the birth-generation ceiling that governs +/// a JSON-constructed object, so this fixture is born OLD whatever that +/// constant is. +/// +/// #10123: hardcoding a width silently pinned these tests to +/// `LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES == 128 KB`. When the JSON +/// construction path gained a higher young-birth ceiling the fixture turned +/// young and the tests failed on their PREMISE (`pointer_in_old_gen`) rather +/// than on anything they were written to check. Derived, the old-gen path stays +/// covered at any ceiling. +pub(super) fn fields_born_old() -> usize { + (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES + / std::mem::size_of::()) + + 1024 +} + fn wide_source() -> String { format!( "{{{}}}", - (0..50_000) + (0..fields_born_old()) .map(|i| format!("\"field_{i}\":{i}")) .collect::>() .join(",") @@ -44,7 +60,7 @@ fn json_discarded_wide_keys_release_storage_after_cache_eviction() { let value = parse(&source); assert_eq!( crate::object::object_live_slot_count(value.as_pointer()), - 50_000 + fields_born_old() as u32 ); } assert_eq!( @@ -129,11 +145,15 @@ fn json_retained_wide_object_keeps_evicted_keys_through_minor_and_full_gc() { let root = scope.root_nanbox_u64(value.bits()); let keys = crate::object::object_keys_array(value.as_pointer()); assert!(crate::arena::pointer_in_old_gen(keys as usize)); - let last_before = crate::array::js_array_get(keys, 49_999).bits(); + let last_idx = (fields_born_old() - 1) as u32; + let last_before = crate::array::js_array_get(keys, last_idx).bits(); gc_collect_minor(); let live = crate::JSValue::from_bits(root.get_nanbox_u64()); let keys = crate::object::object_keys_array(live.as_pointer()); - assert_ne!(last_before, crate::array::js_array_get(keys, 49_999).bits()); + assert_ne!( + last_before, + crate::array::js_array_get(keys, last_idx).bits() + ); assert_output(live, &source); collect_full(); assert_output(crate::JSValue::from_bits(root.get_nanbox_u64()), &source); diff --git a/crates/perry-runtime/src/gc/types.rs b/crates/perry-runtime/src/gc/types.rs index b87f748501..bfe1f805be 100644 --- a/crates/perry-runtime/src/gc/types.rs +++ b/crates/perry-runtime/src/gc/types.rs @@ -116,6 +116,87 @@ pub const LARGE_OBJECT_THRESHOLD_BYTES: usize = 16 * 1024; /// block. pub const LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES: usize = 128 * 1024; +/// Birth-generation ceiling for an ordinary object's property storage (#10123). +/// +/// Half a nursery block: it clears a 65,536-field object, and the worst-case +/// block fragmentation it can cause is 1/2 of one block against the flat +/// threshold's 1/8. Past it the storage is born old as before, which is also +/// where the copier would stop being able to move it. +pub const LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES: usize = 512 * 1024; + +/// Object types a [`JsonWideBirthScope`] may keep young past the threshold. +pub mod json_wide_birth { + /// Nothing (the scope is closed). + pub const NONE: u8 = 0; + /// A wide object's own property storage. + pub const OBJECTS: u8 = 1; + /// A parse shape-keys array. + pub const KEYS_ARRAY: u8 = 2; +} + +crate::perry_thread_local! { + /// Read ONLY from the cold large-object branch of `arena_alloc_gc` (and its + /// `ConstructionBatch` twin), never from an allocation hot path. + static JSON_WIDE_BIRTH_MASK: std::cell::Cell = + const { std::cell::Cell::new(json_wide_birth::NONE) }; +} + +/// Keep a wide JSON document's own storage young past the birth threshold +/// (#10123), for as long as the copier can still move it. +/// +/// The threshold's rationale is that a large pointer-bearing object is stamped +/// `GC_FLAG_TENURED` and a minor never sweeps old-gen, so its cost "is not its +/// own bytes, it is every object it can reach, held live through the remembered +/// set by a container nothing refers to any more". A wide document's property +/// storage and its shape-keys array are exactly that container: above 16,384 +/// fields each crosses 128 KB, is born tenured, and then holds its whole field +/// or key set live after the document itself is dead. Measured at 64 parses, +/// retained shape-keys arrays / peak RSS: 16,300 fields -> 0 / 29 MiB, +/// 16,500 -> 30 / 98 MiB, 50,000 -> 15 / 177 MiB -- the step landing exactly on +/// the constant. +/// +/// **Scoped, and type-masked, on purpose.** Raising the constant globally also +/// moved ordinary ARRAY element storage into the nursery, which those rows do +/// not need and which cost them RSS they could not afford (records_array_8m:scan +/// 442 -> 643 MiB against Node 156 / Bun 166). Only a document's own object +/// storage and its keys array are admitted here; array element storage keeps the +/// flat threshold. +pub struct JsonWideBirthScope(u8); + +impl JsonWideBirthScope { + /// Admit wide-object property storage for the duration of a parse. + pub fn objects() -> Self { + Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::OBJECTS))) + } + + /// Admit a parse shape-keys array around its single allocation site. + pub fn keys_array() -> Self { + Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::KEYS_ARRAY))) + } +} + +impl Drop for JsonWideBirthScope { + fn drop(&mut self) { + JSON_WIDE_BIRTH_MASK.with(|c| c.set(self.0)); + } +} + +/// May an already-oversized allocation still be born young? +/// +/// Called ONLY once the size test has already said "large", so the thread-local +/// read never touches the allocation hot path. The ceiling is the copier's own +/// refusal point: past it a young object could not be moved, which is the one +/// thing birth-young must not promise falsely. +#[inline] +pub fn json_wide_birth_permits(total_size: usize, obj_type: u8) -> bool { + if total_size > LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES { + return false; + } + let mask = JSON_WIDE_BIRTH_MASK.with(|c| c.get()); + (mask == json_wide_birth::OBJECTS && obj_type == GC_TYPE_OBJECT) + || (mask == json_wide_birth::KEYS_ARRAY && obj_type == GC_TYPE_ARRAY) +} + #[inline] pub fn is_large_object_total_size(total_size: usize) -> bool { total_size > LARGE_OBJECT_THRESHOLD_BYTES @@ -134,6 +215,26 @@ pub fn large_object_threshold_for_type(obj_type: u8) -> usize { if obj_type == GC_TYPE_BUFFER { return LARGE_OBJECT_THRESHOLD_BYTES; } + // #10123 NOTE: the widening is SCOPED (see `JsonWideBirthScope`), not a + // blanket change to this constant. A WIDE OBJECT's property storage is the + // threshold's own rationale warns about -- "every object it can reach, held + // live through the remembered set by a container nothing refers to any + // more". Above 16,384 fields it crosses 128 KB, is born tenured, and then + // holds its whole field set live after the object itself is dead. Measured + // at 64 parses of one document, retained shape-keys arrays / peak RSS: + // 16,300 fields -> 0 / 29 MiB, 16,500 -> 30 / 98 MiB, 50,000 -> 15 / 177 MiB; + // the step lands exactly on the constant. + // + // Widened for OBJECT storage ONLY, not for arrays. That is not a guess: on + // this benchmark matrix `wide_1m` is the only row with a large + // GC_TYPE_OBJECT birth (400,024 B, once per parse), while every large birth + // on records_array_8m/20m is GC_TYPE_ARRAY element storage (131 KB - 2 MB). + // Widening those too bought RSS those rows did not need and cost it + // elsewhere: records_array_8m:scan 444 -> 710 MiB, 20m:parse 267 -> 314 MiB. + // + // Still inside the copier's structural ceilings (1 MB nursery block, + // `copying::MAX_YOUNG_MOVE_BYTES`), so an object admitted here is movable -- + // the one thing birth-young must not promise falsely. match gc_type_info(obj_type) { Some(info) if !info.pointer_free => LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES, _ => LARGE_OBJECT_THRESHOLD_BYTES, diff --git a/crates/perry-runtime/src/json/mod.rs b/crates/perry-runtime/src/json/mod.rs index 0951812113..3fe568a145 100644 --- a/crates/perry-runtime/src/json/mod.rs +++ b/crates/perry-runtime/src/json/mod.rs @@ -539,6 +539,10 @@ unsafe fn allocate_parse_shape_keys_array(keys: &[*const StringHeader]) -> *mut // construction helper publishes its pointer layout and, for large arrays // born in old generation, remembers young key strings before return. let _suppressed = crate::gc::GcSuppressScope::new(); + // #10123: same reasoning as the object storage -- a wide document's keys + // array crosses the threshold, is born tenured, and then holds its whole + // key set live long after every instance has died. + let _wide = crate::gc::JsonWideBirthScope::keys_array(); let mut batch = crate::arena::ConstructionBatch::new(); let mut array = construction_array::ConstructionArray::new(&mut batch, keys.len() as u32); for &key_ptr in keys { diff --git a/crates/perry-runtime/src/object/json_construction.rs b/crates/perry-runtime/src/object/json_construction.rs index 38d489789b..9c2ab309ed 100644 --- a/crates/perry-runtime/src/object/json_construction.rs +++ b/crates/perry-runtime/src/object/json_construction.rs @@ -190,6 +190,14 @@ pub(crate) unsafe fn object_from_json_fields_preinstalled( b.try_alloc(size, crate::gc::GC_TYPE_OBJECT) }); let obj = if raw.is_null() { + // #10123: a wide document's property storage crosses the birth + // threshold and is then born tenured, where no minor will ever sweep + // it -- so it holds its whole field set live after the document is + // dead. Keep it young while the copier can still move it. This is the + // site the scope has to cover: the storage is minted HERE, not inside + // `js_json_parse_result`, which is why scoping the parse entry alone + // changed nothing (the allocation saw mask=0). + let _wide = crate::gc::JsonWideBirthScope::objects(); js_object_alloc_class_inline_keys_stamped(0, 0, count as u32, keys, shape_id) } else { let obj = raw.cast::(); From 0c2876b1931b682dd2a0ab923a86aff7f60e9c49 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 12 Sep 2026 22:49:26 +0200 Subject: [PATCH 2/2] perf(json): allocate large record arrays once, at their estimated size (#10123) The direct parser pre-sizes `[{...}]` arrays from `remaining_bytes / 96`, but clamped that estimate at 16,384 slots: a 131,088-byte allocation, 16 bytes over the 131,072-byte pointer-bearing birth threshold. So every large record array was born OLD on its first allocation and then doubled twice more in old-gen (131 -> 262 -> 524 KB for a 59,000-row document). An old array of young records keeps them alive through the remembered set after the document dies: on records_object_8m:parse `remembered_set/array` was the origin of 98% of minor survivors, across three minors and zero fulls. Use the estimate as-is. One allocation, admitted into the nursery through `JsonWideBirthScope::arrays()` when it fits the JSON young-birth ceiling, and a single old allocation past it rather than a chain of four. The ceiling rises 512 KB -> 768 KB (three quarters of a nursery block, still inside `arena::BLOCK_SIZE` and `copying::MAX_YOUNG_MOVE_BYTES`) because a 7.1 MB document's estimate is 593 KB. No fixture has object storage between the two values, so the wide-object path from the previous commit is unchanged. An earlier attempt admitted every JSON array young, including the doubling chain's intermediates. It regressed eight cells (records_array_20m:* CPU +45%) because each abandoned young intermediate still cost a copy. Sizing once is what removes that cost; this version was measured against the old clamp in a single binary across all 50 matrix cells: records_object_8m:parse 187 -> 118 MiB (1.68x -> 1.05x best) records_array_20m:parse/scan/sparse 256 -> 240 MiB records_object_20m:parse 256 -> 240 MiB records_object_8m:parse CPU 167 -> 206 ms, still 0.88x the better engine. No other cell moved outside noise. A unit test pins both outcomes: an under-ceiling estimate is one young allocation that does not regrow for the rows it was sized for, and a past-ceiling estimate keeps its old-gen birth. --- ...10123-json-wide-object-birth-generation.md | 24 ++++++ crates/perry-runtime/src/gc/types.rs | 22 ++++-- .../src/json/construction_array.rs | 77 +++++++++++++++++++ crates/perry-runtime/src/json/parser.rs | 4 +- 4 files changed, 119 insertions(+), 8 deletions(-) diff --git a/changelog.d/10123-json-wide-object-birth-generation.md b/changelog.d/10123-json-wide-object-birth-generation.md index 457d8ad87b..a8a5adf933 100644 --- a/changelog.d/10123-json-wide-object-birth-generation.md +++ b/changelog.d/10123-json-wide-object-birth-generation.md @@ -46,3 +46,27 @@ their width from the governing ceiling, so the old-gen path stays covered whatever that constant is — previously they silently depended on it being 128 KB and failed on their premise rather than their subject. + +- **Large JSON record arrays are allocated once, at their estimated size (#10123).** The + direct parser already pre-sized `[{...}]` arrays from `remaining_bytes / 96`, but clamped + the estimate at 16,384 slots — a 131,088-byte allocation, **16 bytes over** the + 131,072-byte pointer-bearing birth threshold. Every large record array was therefore born + old on its very first allocation and then doubled twice more in old-gen (131 → 262 → + 524 KB for 59,000 rows), and an old array of young records keeps them alive through the + remembered set after the document dies. On `records_object_8m:parse` + `remembered_set/array` was the origin of 98% of minor survivors, across three minors and no + full collection. + + The estimate is now used as-is: one allocation, admitted into the nursery when it fits the + JSON young-birth ceiling (raised to 768 KB, three quarters of a nursery block, so a 7.1 MB + document's 593 KB estimate qualifies) and born old in a single allocation past it. + Measured in one binary against the previous clamp, all 50 matrix cells: + + | row | before | after | + |---|---:|---:| + | `records_object_8m:parse` | 187 MiB / 167 ms | **118 MiB** / 206 ms | + | `records_array_20m:parse`, `:scan`, `:sparse` | 256 MiB | **240 MiB** | + | `records_object_20m:parse` | 256 MiB | **240 MiB** | + + `records_object_8m:parse` reaches parity with the better of Node and Bun on RSS (1.68× → + 1.05×) while its CPU stays at 0.88× of the better engine. No other cell moved outside noise. diff --git a/crates/perry-runtime/src/gc/types.rs b/crates/perry-runtime/src/gc/types.rs index bfe1f805be..17ebb9f00a 100644 --- a/crates/perry-runtime/src/gc/types.rs +++ b/crates/perry-runtime/src/gc/types.rs @@ -116,13 +116,17 @@ pub const LARGE_OBJECT_THRESHOLD_BYTES: usize = 16 * 1024; /// block. pub const LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES: usize = 128 * 1024; -/// Birth-generation ceiling for an ordinary object's property storage (#10123). +/// Birth-generation ceiling for JSON-constructed storage (#10123). /// -/// Half a nursery block: it clears a 65,536-field object, and the worst-case -/// block fragmentation it can cause is 1/2 of one block against the flat -/// threshold's 1/8. Past it the storage is born old as before, which is also -/// where the copier would stop being able to move it. -pub const LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES: usize = 512 * 1024; +/// Three quarters of a nursery block. Wide-object storage needs only half +/// (a 50,000-field document is 400 KB), but a record array sized once from the +/// parser's `remaining / 96` estimate does not: a 7.1 MB document of 59,000 rows +/// estimates 74,145 slots, 593 KB. Admitting that single allocation young is +/// what lets a minor reclaim the array and its records together after the +/// document dies. Still inside the 1 MiB `arena::BLOCK_SIZE` and +/// `copying::MAX_YOUNG_MOVE_BYTES`, so anything admitted here stays movable. +/// Past it storage is born old as before. +pub const LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES: usize = 768 * 1024; /// Object types a [`JsonWideBirthScope`] may keep young past the threshold. pub mod json_wide_birth { @@ -173,6 +177,12 @@ impl JsonWideBirthScope { pub fn keys_array() -> Self { Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::KEYS_ARRAY))) } + + /// Admit a JSON-constructed array allocation: a record array sized once + /// from the parser's estimate (see `ConstructionArray::presized_records`). + pub fn arrays() -> Self { + Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::KEYS_ARRAY))) + } } impl Drop for JsonWideBirthScope { diff --git a/crates/perry-runtime/src/json/construction_array.rs b/crates/perry-runtime/src/json/construction_array.rs index 3e7cceabf0..c361734d0b 100644 --- a/crates/perry-runtime/src/json/construction_array.rs +++ b/crates/perry-runtime/src/json/construction_array.rs @@ -11,6 +11,36 @@ pub(super) struct ConstructionArray { } impl ConstructionArray { + /// Allocate a record array ONCE at its estimated final size (#10123). + /// + /// The previous clamp of 16,384 slots is a 131,088-byte allocation -- 16 + /// bytes over the 131,072-byte pointer-bearing birth threshold -- so every + /// large record array was born OLD on its very first allocation and then + /// doubled twice more in old-gen (131 -> 262 -> 524 KB for 59,000 rows). + /// An old array of young records keeps them alive through the remembered + /// set after the document dies, and no full is ever scheduled for it. + /// + /// Sizing to the estimate removes the doubling chain. Up to the JSON + /// young-birth ceiling the single allocation is kept in the nursery, so a + /// minor reclaims the array and its records together once the document is + /// dead; past it the array is still one old allocation rather than four. + pub(super) unsafe fn presized_records( + batch: &mut Option, + estimated_len: usize, + ) -> Self { + let slot = std::mem::size_of::(); + let header = std::mem::size_of::() + crate::gc::GC_HEADER_SIZE; + let young_slots = (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES + .saturating_sub(header)) + / slot; + let capacity = estimated_len.clamp(16, (u32::MAX / 2) as usize); + if capacity <= young_slots { + let _young = crate::gc::JsonWideBirthScope::arrays(); + return Self::new(batch, capacity as u32); + } + Self::new(batch, capacity as u32) + } + pub(super) unsafe fn new( batch: &mut Option, capacity: u32, @@ -149,3 +179,50 @@ impl ConstructionArray { self.ptr } } + +#[cfg(test)] +mod tests { + use super::*; + + /// #10123: a record array sized from the parser estimate is ONE allocation, + /// born young when it fits the JSON young-birth ceiling and old past it. + /// The previous 16,384-slot clamp was a 131,088-byte first allocation -- + /// 16 bytes over the pointer-bearing threshold -- so every large record + /// array began life tenured and then doubled twice more in old-gen. + #[test] + fn presized_record_array_is_one_allocation_in_the_right_generation() { + let slot = std::mem::size_of::(); + let header = std::mem::size_of::() + crate::gc::GC_HEADER_SIZE; + let young_slots = + (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES - header) / slot; + unsafe { + let _suppress = crate::gc::GcSuppressScope::new(); + + // records_*_8m shape: 7.1 MB / 96 -> 74,145 slots, 593 KB. + let mut batch = None; + let young = ConstructionArray::presized_records(&mut batch, 74_145); + assert!(74_145 <= young_slots, "fixture must sit under the ceiling"); + assert!( + !crate::arena::pointer_in_old_gen(young.ptr as usize), + "an estimate under the young-birth ceiling must be born young" + ); + assert!((*young.ptr).capacity >= 74_145); + + // It must not need to grow for the rows it was sized for. + let mut young = young; + let before = young.ptr; + for i in 0..59_000 { + young.push(&mut batch, JSValue::number(i as f64)); + } + assert_eq!(young.ptr, before, "a presized record array must not regrow"); + + // records_*_20m shape: past the ceiling the single allocation is old. + let mut batch = None; + let old = ConstructionArray::presized_records(&mut batch, young_slots + 1_000); + assert!( + crate::arena::pointer_in_old_gen(old.ptr as usize), + "an estimate past the ceiling keeps the old-gen birth" + ); + } + } +} diff --git a/crates/perry-runtime/src/json/parser.rs b/crates/perry-runtime/src/json/parser.rs index 832b3f4641..767d4866d8 100644 --- a/crates/perry-runtime/src/json/parser.rs +++ b/crates/perry-runtime/src/json/parser.rs @@ -1180,9 +1180,9 @@ impl<'a> DirectParser<'a> { } // Same `[{...}]` pre-size heuristic as the typed path. // Preserve the object-leading estimate on large record arrays. - let array = super::construction_array::ConstructionArray::new( + let array = super::construction_array::ConstructionArray::presized_records( &mut self.batch, - ((self.input.len() - self.pos) / 96).clamp(16, 16_384) as u32, + (self.input.len() - self.pos) / 96, ); self.parse_array_tail(array, saved_roots) }