diff --git a/changelog.d/10123-json-wide-object-birth-generation.md b/changelog.d/10123-json-wide-object-birth-generation.md new file mode 100644 index 0000000000..a8a5adf933 --- /dev/null +++ b/changelog.d/10123-json-wide-object-birth-generation.md @@ -0,0 +1,72 @@ +### Fixed + +- **Repeated `JSON.parse` of a wide object held 5x the memory it needed (#10123).** + Peak RSS on `wide_1m:parse` (a 50,000-field document, 64 parses) was **220 MiB against a + live set of ~0** — Node holds 136 MiB and Bun 71 MiB on the same workload. + + `arena/allocators.rs` already states the failure mode: a large pointer-bearing object is + stamped `GC_FLAG_TENURED`, and a minor never sweeps old-gen, so its cost "is not its own + bytes, it is every object it can reach, held live through the remembered set by a container + nothing refers to any more". A wide document's **property storage** and its **shape-keys + array** are exactly that container. Above 16,384 fields each crosses + `LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES` (128 KB), is born tenured, and then holds its + whole field or key set live long after the document itself is dead. + + Confirmed to the byte with `PERRY_GC_CENSUS`, fixtures straddling 131,072 bytes at 64 parses + (retained shape-keys arrays / live / peak RSS): + + | fields | keys array | retained | live | peak RSS | + |---:|---:|---:|---:|---:| + | 16,300 | 130,416 B | 0 | 0.0 MB | 29 MiB | + | 16,500 | 132,016 B | 30 | 22.7 MB | 98 MiB | + | 50,000 | 400,016 B | 15 | 34.3 MB | 177 MiB | + + The census named it outright: `shape_keys_arrays {count: 15}`, `slot_tags {string: 750000}` + — 15 arrays x 50,000 keys = 750,000 live strings. The same binary under `PERRY_GEN_GC=0` + reports **336 bytes live**. + + Fixed by admitting a wide JSON document's own storage into the nursery past that threshold, + for as long as the copier can still move it (`JsonWideBirthScope`, ceiling 512 KB — half a + nursery block, inside `copying::MAX_YOUNG_MOVE_BYTES`). The check is reached only from the + cold large-object branch of `arena_alloc_gc`, behind a short-circuiting `&&`, so no + allocation hot path gains work. + + **`wide_1m:parse`: 220 MiB -> 40 MiB and 0.20s -> 0.15s** — below Node's 136 MiB and Bun's + 71 MiB, and faster than before. + + **Scoped and type-masked deliberately.** Raising the constant globally reaches the same + 40 MiB, but it also moves ordinary ARRAY element storage into the nursery, which other rows + do not need and cannot afford: `records_array_8m:scan` 643 -> 710 MiB with CPU 0.91s -> + 1.20s. Only a document's own object storage and its keys array are admitted. Measured in a + single binary across `records_array_8m:scan`, `records_array_20m:parse`, + `records_array_1m:scan`, `records_array_16k:scan` and `heterogeneous_1m:parse`, the scoped + change is **byte-identical to baseline on every one of them**. + + Three tests that hardcoded a field count and then asserted `pointer_in_old_gen` now derive + their width from the governing ceiling, so the old-gen path stays covered whatever that + constant is — previously they silently depended on it being 128 KB and failed on their + premise rather than their subject. + +- **Large JSON record arrays are allocated once, at their estimated size (#10123).** The + direct parser already pre-sized `[{...}]` arrays from `remaining_bytes / 96`, but clamped + the estimate at 16,384 slots — a 131,088-byte allocation, **16 bytes over** the + 131,072-byte pointer-bearing birth threshold. Every large record array was therefore born + old on its very first allocation and then doubled twice more in old-gen (131 → 262 → + 524 KB for 59,000 rows), and an old array of young records keeps them alive through the + remembered set after the document dies. On `records_object_8m:parse` + `remembered_set/array` was the origin of 98% of minor survivors, across three minors and no + full collection. + + The estimate is now used as-is: one allocation, admitted into the nursery when it fits the + JSON young-birth ceiling (raised to 768 KB, three quarters of a nursery block, so a 7.1 MB + document's 593 KB estimate qualifies) and born old in a single allocation past it. + Measured in one binary against the previous clamp, all 50 matrix cells: + + | row | before | after | + |---|---:|---:| + | `records_object_8m:parse` | 187 MiB / 167 ms | **118 MiB** / 206 ms | + | `records_array_20m:parse`, `:scan`, `:sparse` | 256 MiB | **240 MiB** | + | `records_object_20m:parse` | 256 MiB | **240 MiB** | + + `records_object_8m:parse` reaches parity with the better of Node and Bun on RSS (1.68× → + 1.05×) while its CPU stays at 0.88× of the better engine. No other cell moved outside noise. diff --git a/crates/perry-runtime/src/arena/allocators.rs b/crates/perry-runtime/src/arena/allocators.rs index 1e2e87c5e1..28ef1cab56 100644 --- a/crates/perry-runtime/src/arena/allocators.rs +++ b/crates/perry-runtime/src/arena/allocators.rs @@ -428,7 +428,11 @@ pub fn arena_alloc_gc(size: usize, align: usize, obj_type: u8) -> *mut u8 { // slots per minor because of it). let total = gc_padded_total_size(size, align); super::alloc_sample::note(total, obj_type); - if crate::gc::is_large_object_total_size_for_type(total, obj_type) { + // `&&` short-circuits, so the scope check is reached only by an allocation + // the size test has ALREADY called large -- never on the hot path (#10123). + if crate::gc::is_large_object_total_size_for_type(total, obj_type) + && !crate::gc::json_wide_birth_permits(total, obj_type) + { let user_ptr = arena_alloc_gc_old(size, align, obj_type); unsafe { let header = user_ptr.sub(GC_HEADER_SIZE) as *mut GcHeader; diff --git a/crates/perry-runtime/src/gc/tests/helper_stores.rs b/crates/perry-runtime/src/gc/tests/helper_stores.rs index b3752e2f70..d20458f8d8 100644 --- a/crates/perry-runtime/src/gc/tests/helper_stores.rs +++ b/crates/perry-runtime/src/gc/tests/helper_stores.rs @@ -6,8 +6,14 @@ use std::fmt::Write as _; /// born-tenured threshold — see `copying.rs`'s `OLD_BORN_ELEMENTS`. These two /// fixtures assert that a materialized JSON object / regex result array is an /// OLD-generation birth holding young children, so they must actually be one. +/// +/// #10123: a JSON-constructed object's storage is admitted young up to +/// `LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES`, which is HIGHER than the +/// flat pointer-bearing threshold. Derive from the ceiling that actually +/// governs this fixture, or the "is an old-generation birth" premise silently +/// stops holding and the test fails on its premise rather than its subject. const OLD_BORN_FIELDS: u32 = - (crate::gc::LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES / 8) as u32 + 64; + (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES / 8) as u32 + 64; unsafe fn alloc_old_test_map( capacity: u32, diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs index 651dae8111..288a49701b 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/json_construction.rs @@ -63,9 +63,10 @@ fn json_construction_wide_old_record_remembers_sparse_pointer_pages() { gc_register_mutable_root_scanner(json_parse_mutable_root_scanner); let source = format!( "{{{}}}", - (0..20_000) + (0..super::json_key_lifetime::fields_born_old()) .map(|i| { - let value = if i % 509 == 0 || i == 19_999 { + let value = if i % 509 == 0 || i == super::json_key_lifetime::fields_born_old() - 1 + { format!(r#"{{"child":{i}}}"#) } else { i.to_string() @@ -87,13 +88,13 @@ fn json_construction_wide_old_record_remembers_sparse_pointer_pages() { let last_child = |value: crate::JSValue| { assert_eq!( crate::object::object_live_slot_count(value.as_pointer()), - 20_000 + super::json_key_lifetime::fields_born_old() as u32 ); let slots = value .as_pointer::() .add(std::mem::size_of::()) .cast::(); - (*slots.add(19_999)).bits() + (*slots.add(super::json_key_lifetime::fields_born_old() - 1)).bits() }; let last_before = last_child(value); assert!(crate::JSValue::from_bits(last_before).is_pointer()); diff --git a/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs b/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs index 271ee03aa3..048572aace 100644 --- a/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs +++ b/crates/perry-runtime/src/gc/tests/runtime_roots/json_key_lifetime.rs @@ -1,10 +1,26 @@ //! Cache eviction must release ownership, including the backing key storage. use super::*; +/// Field count whose storage exceeds the birth-generation ceiling that governs +/// a JSON-constructed object, so this fixture is born OLD whatever that +/// constant is. +/// +/// #10123: hardcoding a width silently pinned these tests to +/// `LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES == 128 KB`. When the JSON +/// construction path gained a higher young-birth ceiling the fixture turned +/// young and the tests failed on their PREMISE (`pointer_in_old_gen`) rather +/// than on anything they were written to check. Derived, the old-gen path stays +/// covered at any ceiling. +pub(super) fn fields_born_old() -> usize { + (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES + / std::mem::size_of::()) + + 1024 +} + fn wide_source() -> String { format!( "{{{}}}", - (0..50_000) + (0..fields_born_old()) .map(|i| format!("\"field_{i}\":{i}")) .collect::>() .join(",") @@ -44,7 +60,7 @@ fn json_discarded_wide_keys_release_storage_after_cache_eviction() { let value = parse(&source); assert_eq!( crate::object::object_live_slot_count(value.as_pointer()), - 50_000 + fields_born_old() as u32 ); } assert_eq!( @@ -129,11 +145,15 @@ fn json_retained_wide_object_keeps_evicted_keys_through_minor_and_full_gc() { let root = scope.root_nanbox_u64(value.bits()); let keys = crate::object::object_keys_array(value.as_pointer()); assert!(crate::arena::pointer_in_old_gen(keys as usize)); - let last_before = crate::array::js_array_get(keys, 49_999).bits(); + let last_idx = (fields_born_old() - 1) as u32; + let last_before = crate::array::js_array_get(keys, last_idx).bits(); gc_collect_minor(); let live = crate::JSValue::from_bits(root.get_nanbox_u64()); let keys = crate::object::object_keys_array(live.as_pointer()); - assert_ne!(last_before, crate::array::js_array_get(keys, 49_999).bits()); + assert_ne!( + last_before, + crate::array::js_array_get(keys, last_idx).bits() + ); assert_output(live, &source); collect_full(); assert_output(crate::JSValue::from_bits(root.get_nanbox_u64()), &source); diff --git a/crates/perry-runtime/src/gc/types.rs b/crates/perry-runtime/src/gc/types.rs index b87f748501..17ebb9f00a 100644 --- a/crates/perry-runtime/src/gc/types.rs +++ b/crates/perry-runtime/src/gc/types.rs @@ -116,6 +116,97 @@ pub const LARGE_OBJECT_THRESHOLD_BYTES: usize = 16 * 1024; /// block. pub const LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES: usize = 128 * 1024; +/// Birth-generation ceiling for JSON-constructed storage (#10123). +/// +/// Three quarters of a nursery block. Wide-object storage needs only half +/// (a 50,000-field document is 400 KB), but a record array sized once from the +/// parser's `remaining / 96` estimate does not: a 7.1 MB document of 59,000 rows +/// estimates 74,145 slots, 593 KB. Admitting that single allocation young is +/// what lets a minor reclaim the array and its records together after the +/// document dies. Still inside the 1 MiB `arena::BLOCK_SIZE` and +/// `copying::MAX_YOUNG_MOVE_BYTES`, so anything admitted here stays movable. +/// Past it storage is born old as before. +pub const LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES: usize = 768 * 1024; + +/// Object types a [`JsonWideBirthScope`] may keep young past the threshold. +pub mod json_wide_birth { + /// Nothing (the scope is closed). + pub const NONE: u8 = 0; + /// A wide object's own property storage. + pub const OBJECTS: u8 = 1; + /// A parse shape-keys array. + pub const KEYS_ARRAY: u8 = 2; +} + +crate::perry_thread_local! { + /// Read ONLY from the cold large-object branch of `arena_alloc_gc` (and its + /// `ConstructionBatch` twin), never from an allocation hot path. + static JSON_WIDE_BIRTH_MASK: std::cell::Cell = + const { std::cell::Cell::new(json_wide_birth::NONE) }; +} + +/// Keep a wide JSON document's own storage young past the birth threshold +/// (#10123), for as long as the copier can still move it. +/// +/// The threshold's rationale is that a large pointer-bearing object is stamped +/// `GC_FLAG_TENURED` and a minor never sweeps old-gen, so its cost "is not its +/// own bytes, it is every object it can reach, held live through the remembered +/// set by a container nothing refers to any more". A wide document's property +/// storage and its shape-keys array are exactly that container: above 16,384 +/// fields each crosses 128 KB, is born tenured, and then holds its whole field +/// or key set live after the document itself is dead. Measured at 64 parses, +/// retained shape-keys arrays / peak RSS: 16,300 fields -> 0 / 29 MiB, +/// 16,500 -> 30 / 98 MiB, 50,000 -> 15 / 177 MiB -- the step landing exactly on +/// the constant. +/// +/// **Scoped, and type-masked, on purpose.** Raising the constant globally also +/// moved ordinary ARRAY element storage into the nursery, which those rows do +/// not need and which cost them RSS they could not afford (records_array_8m:scan +/// 442 -> 643 MiB against Node 156 / Bun 166). Only a document's own object +/// storage and its keys array are admitted here; array element storage keeps the +/// flat threshold. +pub struct JsonWideBirthScope(u8); + +impl JsonWideBirthScope { + /// Admit wide-object property storage for the duration of a parse. + pub fn objects() -> Self { + Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::OBJECTS))) + } + + /// Admit a parse shape-keys array around its single allocation site. + pub fn keys_array() -> Self { + Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::KEYS_ARRAY))) + } + + /// Admit a JSON-constructed array allocation: a record array sized once + /// from the parser's estimate (see `ConstructionArray::presized_records`). + pub fn arrays() -> Self { + Self(JSON_WIDE_BIRTH_MASK.with(|c| c.replace(json_wide_birth::KEYS_ARRAY))) + } +} + +impl Drop for JsonWideBirthScope { + fn drop(&mut self) { + JSON_WIDE_BIRTH_MASK.with(|c| c.set(self.0)); + } +} + +/// May an already-oversized allocation still be born young? +/// +/// Called ONLY once the size test has already said "large", so the thread-local +/// read never touches the allocation hot path. The ceiling is the copier's own +/// refusal point: past it a young object could not be moved, which is the one +/// thing birth-young must not promise falsely. +#[inline] +pub fn json_wide_birth_permits(total_size: usize, obj_type: u8) -> bool { + if total_size > LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES { + return false; + } + let mask = JSON_WIDE_BIRTH_MASK.with(|c| c.get()); + (mask == json_wide_birth::OBJECTS && obj_type == GC_TYPE_OBJECT) + || (mask == json_wide_birth::KEYS_ARRAY && obj_type == GC_TYPE_ARRAY) +} + #[inline] pub fn is_large_object_total_size(total_size: usize) -> bool { total_size > LARGE_OBJECT_THRESHOLD_BYTES @@ -134,6 +225,26 @@ pub fn large_object_threshold_for_type(obj_type: u8) -> usize { if obj_type == GC_TYPE_BUFFER { return LARGE_OBJECT_THRESHOLD_BYTES; } + // #10123 NOTE: the widening is SCOPED (see `JsonWideBirthScope`), not a + // blanket change to this constant. A WIDE OBJECT's property storage is the + // threshold's own rationale warns about -- "every object it can reach, held + // live through the remembered set by a container nothing refers to any + // more". Above 16,384 fields it crosses 128 KB, is born tenured, and then + // holds its whole field set live after the object itself is dead. Measured + // at 64 parses of one document, retained shape-keys arrays / peak RSS: + // 16,300 fields -> 0 / 29 MiB, 16,500 -> 30 / 98 MiB, 50,000 -> 15 / 177 MiB; + // the step lands exactly on the constant. + // + // Widened for OBJECT storage ONLY, not for arrays. That is not a guess: on + // this benchmark matrix `wide_1m` is the only row with a large + // GC_TYPE_OBJECT birth (400,024 B, once per parse), while every large birth + // on records_array_8m/20m is GC_TYPE_ARRAY element storage (131 KB - 2 MB). + // Widening those too bought RSS those rows did not need and cost it + // elsewhere: records_array_8m:scan 444 -> 710 MiB, 20m:parse 267 -> 314 MiB. + // + // Still inside the copier's structural ceilings (1 MB nursery block, + // `copying::MAX_YOUNG_MOVE_BYTES`), so an object admitted here is movable -- + // the one thing birth-young must not promise falsely. match gc_type_info(obj_type) { Some(info) if !info.pointer_free => LARGE_POINTER_BEARING_OBJECT_THRESHOLD_BYTES, _ => LARGE_OBJECT_THRESHOLD_BYTES, diff --git a/crates/perry-runtime/src/json/construction_array.rs b/crates/perry-runtime/src/json/construction_array.rs index 3e7cceabf0..c361734d0b 100644 --- a/crates/perry-runtime/src/json/construction_array.rs +++ b/crates/perry-runtime/src/json/construction_array.rs @@ -11,6 +11,36 @@ pub(super) struct ConstructionArray { } impl ConstructionArray { + /// Allocate a record array ONCE at its estimated final size (#10123). + /// + /// The previous clamp of 16,384 slots is a 131,088-byte allocation -- 16 + /// bytes over the 131,072-byte pointer-bearing birth threshold -- so every + /// large record array was born OLD on its very first allocation and then + /// doubled twice more in old-gen (131 -> 262 -> 524 KB for 59,000 rows). + /// An old array of young records keeps them alive through the remembered + /// set after the document dies, and no full is ever scheduled for it. + /// + /// Sizing to the estimate removes the doubling chain. Up to the JSON + /// young-birth ceiling the single allocation is kept in the nursery, so a + /// minor reclaims the array and its records together once the document is + /// dead; past it the array is still one old allocation rather than four. + pub(super) unsafe fn presized_records( + batch: &mut Option, + estimated_len: usize, + ) -> Self { + let slot = std::mem::size_of::(); + let header = std::mem::size_of::() + crate::gc::GC_HEADER_SIZE; + let young_slots = (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES + .saturating_sub(header)) + / slot; + let capacity = estimated_len.clamp(16, (u32::MAX / 2) as usize); + if capacity <= young_slots { + let _young = crate::gc::JsonWideBirthScope::arrays(); + return Self::new(batch, capacity as u32); + } + Self::new(batch, capacity as u32) + } + pub(super) unsafe fn new( batch: &mut Option, capacity: u32, @@ -149,3 +179,50 @@ impl ConstructionArray { self.ptr } } + +#[cfg(test)] +mod tests { + use super::*; + + /// #10123: a record array sized from the parser estimate is ONE allocation, + /// born young when it fits the JSON young-birth ceiling and old past it. + /// The previous 16,384-slot clamp was a 131,088-byte first allocation -- + /// 16 bytes over the pointer-bearing threshold -- so every large record + /// array began life tenured and then doubled twice more in old-gen. + #[test] + fn presized_record_array_is_one_allocation_in_the_right_generation() { + let slot = std::mem::size_of::(); + let header = std::mem::size_of::() + crate::gc::GC_HEADER_SIZE; + let young_slots = + (crate::gc::LARGE_OBJECT_STORAGE_YOUNG_BIRTH_CEILING_BYTES - header) / slot; + unsafe { + let _suppress = crate::gc::GcSuppressScope::new(); + + // records_*_8m shape: 7.1 MB / 96 -> 74,145 slots, 593 KB. + let mut batch = None; + let young = ConstructionArray::presized_records(&mut batch, 74_145); + assert!(74_145 <= young_slots, "fixture must sit under the ceiling"); + assert!( + !crate::arena::pointer_in_old_gen(young.ptr as usize), + "an estimate under the young-birth ceiling must be born young" + ); + assert!((*young.ptr).capacity >= 74_145); + + // It must not need to grow for the rows it was sized for. + let mut young = young; + let before = young.ptr; + for i in 0..59_000 { + young.push(&mut batch, JSValue::number(i as f64)); + } + assert_eq!(young.ptr, before, "a presized record array must not regrow"); + + // records_*_20m shape: past the ceiling the single allocation is old. + let mut batch = None; + let old = ConstructionArray::presized_records(&mut batch, young_slots + 1_000); + assert!( + crate::arena::pointer_in_old_gen(old.ptr as usize), + "an estimate past the ceiling keeps the old-gen birth" + ); + } + } +} diff --git a/crates/perry-runtime/src/json/mod.rs b/crates/perry-runtime/src/json/mod.rs index 0951812113..3fe568a145 100644 --- a/crates/perry-runtime/src/json/mod.rs +++ b/crates/perry-runtime/src/json/mod.rs @@ -539,6 +539,10 @@ unsafe fn allocate_parse_shape_keys_array(keys: &[*const StringHeader]) -> *mut // construction helper publishes its pointer layout and, for large arrays // born in old generation, remembers young key strings before return. let _suppressed = crate::gc::GcSuppressScope::new(); + // #10123: same reasoning as the object storage -- a wide document's keys + // array crosses the threshold, is born tenured, and then holds its whole + // key set live long after every instance has died. + let _wide = crate::gc::JsonWideBirthScope::keys_array(); let mut batch = crate::arena::ConstructionBatch::new(); let mut array = construction_array::ConstructionArray::new(&mut batch, keys.len() as u32); for &key_ptr in keys { diff --git a/crates/perry-runtime/src/json/parser.rs b/crates/perry-runtime/src/json/parser.rs index 832b3f4641..767d4866d8 100644 --- a/crates/perry-runtime/src/json/parser.rs +++ b/crates/perry-runtime/src/json/parser.rs @@ -1180,9 +1180,9 @@ impl<'a> DirectParser<'a> { } // Same `[{...}]` pre-size heuristic as the typed path. // Preserve the object-leading estimate on large record arrays. - let array = super::construction_array::ConstructionArray::new( + let array = super::construction_array::ConstructionArray::presized_records( &mut self.batch, - ((self.input.len() - self.pos) / 96).clamp(16, 16_384) as u32, + (self.input.len() - self.pos) / 96, ); self.parse_array_tail(array, saved_roots) } diff --git a/crates/perry-runtime/src/object/json_construction.rs b/crates/perry-runtime/src/object/json_construction.rs index 38d489789b..9c2ab309ed 100644 --- a/crates/perry-runtime/src/object/json_construction.rs +++ b/crates/perry-runtime/src/object/json_construction.rs @@ -190,6 +190,14 @@ pub(crate) unsafe fn object_from_json_fields_preinstalled( b.try_alloc(size, crate::gc::GC_TYPE_OBJECT) }); let obj = if raw.is_null() { + // #10123: a wide document's property storage crosses the birth + // threshold and is then born tenured, where no minor will ever sweep + // it -- so it holds its whole field set live after the document is + // dead. Keep it young while the copier can still move it. This is the + // site the scope has to cover: the storage is minted HERE, not inside + // `js_json_parse_result`, which is why scoping the parse entry alone + // changed nothing (the allocation saw mask=0). + let _wide = crate::gc::JsonWideBirthScope::objects(); js_object_alloc_class_inline_keys_stamped(0, 0, count as u32, keys, shape_id) } else { let obj = raw.cast::();