Skip to content

merge train: land #10075 and #10077 - #10113

Merged
proggeramlug merged 9 commits into
mainfrom
train163
Sep 12, 2026
Merged

merge train: land #10075 and #10077#10113
proggeramlug merged 9 commits into
mainfrom
train163

Conversation

@proggeramlug

@proggeramlug proggeramlug commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Merge train landing #10075 and #10077 on top of main dc0d876fe
(train162). Both were CONFLICTING against current main.

The individual PRs merge as their own branch, so GitHub's close keywords do not
fire — each original is closed with a pointer comment after this lands, and the
issues it names are swept by hand.

PR issue commits
#10075 #10061 0c5348ce5, 8d66646ff, 6183863dd
#10077 #10060 a7c59a837, 1e79ee9e4, b27feab6f, 176e5ab86

The finding: #10077's front offset vs. code that postdates its branch point

#10077 gives a dense array a queue front offsetcapacity counts the
slots remaining AFTER the front, so logical element zero is no longer the end of
ArrayHeader, and every open-coded header + size_of::<ArrayHeader>() element
base becomes wrong for an array that has been shift()ed. The PR converts all
of them in its own tree, in both the runtime (array::storage::array_elements_ptr)
and codegen (LlBlock::array_elements_addr, 36 call sites across 20 files).

main has moved since that branch point and grew new ones the PR never saw. I
swept the merged tree for every element-base idiom and converted the two that
remained:

  • json_tape/cached_read.rs — the hot materialized-array read landed by
    perf(json): read traced materialized arrays without redundant resolution #10064 in train161. A materialized JSON array is an ordinary GC_TYPE_ARRAY
    that user code can shift(), so without this the read returns the wrong
    element. This is a genuine cross-PR interaction, not a style fix.
  • array/sort.rs — four more bases from the publish_sorted_values /
    apply_sorted_indices refactor.

The only size_of::<ArrayHeader>() uses left in the tree are allocation-size
validations (param_type_guard.rs) and GC payload arithmetic, not element
bases; codegen has no remaining literal +8 on an array pointer.

Conflict resolutions

Maintainer fix commits

  • fix(runtime): make the JSON cached read honour the array front offset
    the two conversions above, plus the six raw-handle debt sites fix(string): preserve UTF-16 slices and avoid quadratic suffix parsing #10075's new
    test modules introduced. --no-raise-vs <merge-base> refuses both a ceiling
    on a module absent at the base (string/slice_tests.rs) and a per-module
    raise (gc/tests/runtime_roots/string_slice.rs, 1 → 3), so they are
    converted: the post-collection re-reads now run the collection inside
    across_mut / across_const, and the argument-position reads become
    with_const_ptr. That also retires the module's one pre-existing site, so its
    ceiling line is deleted and the recorded total falls 945 → 944
    (--no-raise-vs origin/main: "none raised").
  • chore: bump workspace version to 0.5.1537 — neither PR bumped it, which
    is the maintainer's job at merge time; Cargo.lock regenerated with it.

Validation

run_lint_gates.sh                                              1 of 83 FAILED, 2 CI-only skipped
cargo check -p perry-stdlib                                    ok
cargo check -p perry-runtime --tests --features regex-engine   ok
RUSTFLAGS=-Dwarnings cargo check -p perry --bins                ok
RUST_TEST_THREADS=1 cargo test --release -p perry-runtime      3617 passed, 0 failed
RUST_TEST_THREADS=1 cargo test --release -p perry-codegen      1982 passed, 0 failed
RUST_TEST_THREADS=1 cargo test --release -p perry-hir           639 passed, 0 failed
RUST_TEST_THREADS=1 cargo test --release -p perry-stdlib        137 passed, 0 failed

6375 tests, zero failures. Exit codes are captured from each command itself, not
from a pipeline.

Only one lint gate fails, not the usual three. The two API-doc gates are
normally written off as a CARGO_TARGET_DIR artifact, because
scripts/regen_api_docs.sh builds with CARGO_TARGET_DIR but then reads a
hardcoded $ROOT/target/release/perry. Creating that symlink BEFORE the run
makes them genuinely pass with zero drift. Worth knowing: they are not merely
failing, they are destructive — the shell truncates the redirect target
before exec'ing the missing binary, so a run without the symlink leaves
docs/src/api/reference.md at 0 bytes, a 4229-line deletion sitting in the
worktree. The remaining failure, public benchmark evidence freshness, is
genuinely red on main since 2026-07-29 and is scoped out by #9969.

The four new gap tests this train adds were run directly against the pinned Node
26.5.1, with the static archives rebuilt first
(-p perry -p perry-runtime-static -p perry-stdlib-static) and their mtimes
confirmed newer than the last commit:

test_gap_array_shift_observable      PASS
test_gap_gc_array_shift_queue        PASS
test_gap_gc_string_suffix_cursor     PASS
test_gap_string_slice_utf16_suffix   PASS

The two GC fixtures are registered in test-parity/gc_repsel_corpus.txt, so
gc-stress and gc-moving-witnesses will execute them rather than leaving them
dark on disk.

Each suite was also checked for having actually run its subject: 10 tests from
the new array::shift_queue_tests, string::slice_tests,
gc::tests::copying::shift_queue and gc::tests::runtime_roots::string_slice
modules appear in the runtime log.

Validated at head 2aa97f529d45ab7041f5f3fc58f03e9ce45091f3.

Summary by CodeRabbit

  • Bug Fixes

    • Improved Array.prototype.shift() behavior for dense queues, aliases, garbage collection, growth, and observable edge cases.
    • Fixed slice(), substring(), and substr() handling at UTF-16 and surrogate boundaries, including lone surrogates and Unicode characters.
    • Improved correctness when arrays are serialized, mutated, or reused after repeated shifts.
  • Performance

    • Reduced copying and allocation overhead for repeated array shifts and eligible string-suffix parsing.
  • Tests

    • Added extensive regression coverage for Unicode strings, moving garbage collection, array queues, and benchmark validation.

Ralph Kuepper added 9 commits September 12, 2026 06:30
Conflicts resolved on the train (#10072/#10074 landed first):

- array/concat_reverse.rs, array/sort.rs: kept main's newer structure --
  #10072's three-arm linear append in js_array_concat, and the
  publish_sorted_values / with_sorted_indices refactor -- then applied this
  change's actual intent to it, replacing every open-coded
  `header + size_of::<ArrayHeader>()` element base with
  `crate::array::array_elements_ptr`. That includes four sites in sort.rs that
  postdate this branch's base and so were never converted here.
- array/push_pop.rs: union -- this change's offset-aware accessor plus main's
  TAG_HOLE -> undefined conversion, which spread needs and this branch predates.
- scripts/gc_runtime_root_holders.json: merged structurally, not textually. The
  PASS1_MARKED re-audit prose is base + main's #10055/#10054 paragraphs + this
  change's #10060 paragraph, and each `window.sources` pin is taken from the
  side that actually moved it (gc/census.rs from here, gc/mod.rs from main) so
  neither is reverted to the base hash.
- test-parity/gc_repsel_corpus.txt: both new entries kept.
#10077 gives a dense array a queue front offset: `capacity` counts the slots
remaining AFTER the front, so logical element zero is no longer the end of
`ArrayHeader`. That PR converted every open-coded
`header + size_of::<ArrayHeader>()` element base in its own tree — but two
places on current `main` postdate its branch point and were never converted:

- `json_tape/cached_read.rs` (from #10064, landed in train161) open-codes the
  base on the hot materialized-array read. A materialized JSON array is an
  ordinary `GC_TYPE_ARRAY` that user code can `shift()`, so after this train
  that read would return the wrong element. Routed through
  `crate::array::array_elements_ptr`.
- `array/sort.rs` gained four more open-coded bases with the
  `publish_sorted_values` / `apply_sorted_indices` refactor. Converted with the
  conflict resolution in the pick itself; only the rustfmt reflow lands here.

Also converts the six raw-handle debt sites #10075's two new test modules
introduced. `--no-raise-vs <merge-base>` refuses both a ceiling on a module that
was absent at the base (`string/slice_tests.rs`) and a per-module raise
(`gc/tests/runtime_roots/string_slice.rs`, 1 -> 3), so the sites are converted
rather than recorded:

- `gc/tests/runtime_roots/string_slice.rs`: the two post-collection re-reads now
  run the collection inside `across_mut` / `across_const`, and the kept-slice
  assertions inside `with_const_ptr`. That also retires the module's one
  pre-existing site, so its ceiling line is DELETED — the deletion is the
  receipt — and the recorded total falls 945 -> 944.
- `string/slice_tests.rs`: the three argument-position reads become
  `with_const_ptr`, which is the documented shape for a self-rooting entry
  point such as `js_string_slice`.
Train163 (#10075, #10077) lands on main at 0.5.1536; neither PR bumped the
version, which is the maintainer's job at merge time. Cargo.lock regenerated so
every workspace member's inherited version moves with it.
@proggeramlug
proggeramlug merged commit e8f9123 into main Sep 12, 2026
22 of 23 checks passed
@proggeramlug
proggeramlug deleted the train163 branch September 12, 2026 06:45
@coderabbitai

coderabbitai Bot commented Sep 12, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9dbebabc-a8d7-48c0-8457-4d65a7c1943e

📥 Commits

Reviewing files that changed from the base of the PR and between dc0d876 and 2aa97f5.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (179)
  • CLAUDE.md
  • Cargo.toml
  • benchmarks/array-shift-10060/.gitignore
  • benchmarks/array-shift-10060/README.md
  • benchmarks/array-shift-10060/after.json
  • benchmarks/array-shift-10060/array-push.ts
  • benchmarks/array-shift-10060/array-shift-queue.ts
  • benchmarks/array-shift-10060/before.json
  • benchmarks/array-shift-10060/checksum-reference.json
  • benchmarks/array-shift-10060/comparison.json
  • benchmarks/array-shift-10060/run.py
  • benchmarks/string_slice/.gitignore
  • benchmarks/string_slice/README.md
  • benchmarks/string_slice/baseline-artifacts.json
  • benchmarks/string_slice/baseline.json
  • benchmarks/string_slice/fixed-artifacts.json
  • benchmarks/string_slice/fixed.json
  • benchmarks/string_slice/measure.py
  • benchmarks/string_slice/public-baseline-check.json
  • benchmarks/string_slice/string-slice-astral.ts
  • benchmarks/string_slice/string-slice-parse-loop-ascii.ts
  • benchmarks/string_slice/string-slice-parse-loop-unicode.ts
  • benchmarks/string_slice/validation.json
  • changelog.d/10075-string-slice.md
  • changelog.d/10077-array-shift-queue.md
  • crates/perry-codegen/src/array_storage.rs
  • crates/perry-codegen/src/codegen/closure.rs
  • crates/perry-codegen/src/codegen/entry.rs
  • crates/perry-codegen/src/codegen/function.rs
  • crates/perry-codegen/src/codegen/method.rs
  • crates/perry-codegen/src/collectors/hir_facts.rs
  • crates/perry-codegen/src/collectors/mod.rs
  • crates/perry-codegen/src/collectors/suffix_strings.rs
  • crates/perry-codegen/src/collectors/suffix_strings/tests.rs
  • crates/perry-codegen/src/expr/array_pop.rs
  • crates/perry-codegen/src/expr/array_push.rs
  • crates/perry-codegen/src/expr/dispatch.rs
  • crates/perry-codegen/src/expr/element_shape_guard.rs
  • crates/perry-codegen/src/expr/i32_fast_path.rs
  • crates/perry-codegen/src/expr/index.rs
  • crates/perry-codegen/src/expr/index_get.rs
  • crates/perry-codegen/src/expr/index_get/guarded_array.rs
  • crates/perry-codegen/src/expr/index_get/inline_dyn_typed_array.rs
  • crates/perry-codegen/src/expr/index_set.rs
  • crates/perry-codegen/src/expr/index_set_guarded.rs
  • crates/perry-codegen/src/expr/index_set_packed_loop.rs
  • crates/perry-codegen/src/expr/logical_collections.rs
  • crates/perry-codegen/src/expr/masked_window.rs
  • crates/perry-codegen/src/expr/mod.rs
  • crates/perry-codegen/src/expr/ptr_numarray_access.rs
  • crates/perry-codegen/src/expr/string_window.rs
  • crates/perry-codegen/src/expr/suffix_cursor.rs
  • crates/perry-codegen/src/expr/this_super_call.rs
  • crates/perry-codegen/src/lib.rs
  • crates/perry-codegen/src/runtime_decls/strings.rs
  • crates/perry-codegen/src/stmt/cached_field_index_return.rs
  • crates/perry-codegen/src/stmt/loops.rs
  • crates/perry-codegen/src/stmt/mod.rs
  • crates/perry-codegen/src/stmt/stable_packed_loop.rs
  • crates/perry-ext-better-sqlite3/src/lib.rs
  • crates/perry-ext-http/src/client_overload.rs
  • crates/perry-ext-http/src/client_request_surface.rs
  • crates/perry-ext-http/src/server/types.rs
  • crates/perry-ffi/src/types.rs
  • crates/perry-runtime/src/array/alloc.rs
  • crates/perry-runtime/src/array/collection_tag_tests.rs
  • crates/perry-runtime/src/array/concat_reverse.rs
  • crates/perry-runtime/src/array/element_shape_tests.rs
  • crates/perry-runtime/src/array/fill_extend.rs
  • crates/perry-runtime/src/array/flat_clone.rs
  • crates/perry-runtime/src/array/forwarding_tests.rs
  • crates/perry-runtime/src/array/from_concat.rs
  • crates/perry-runtime/src/array/generic.rs
  • crates/perry-runtime/src/array/generic_object.rs
  • crates/perry-runtime/src/array/header.rs
  • crates/perry-runtime/src/array/header_gc_slots.rs
  • crates/perry-runtime/src/array/immutable.rs
  • crates/perry-runtime/src/array/indexing.rs
  • crates/perry-runtime/src/array/indexing_support.rs
  • crates/perry-runtime/src/array/iter_methods.rs
  • crates/perry-runtime/src/array/join.rs
  • crates/perry-runtime/src/array/jsvalue_api.rs
  • crates/perry-runtime/src/array/mod.rs
  • crates/perry-runtime/src/array/numeric_range.rs
  • crates/perry-runtime/src/array/push_pop.rs
  • crates/perry-runtime/src/array/reduce_right.rs
  • crates/perry-runtime/src/array/search.rs
  • crates/perry-runtime/src/array/shift_queue_tests.rs
  • crates/perry-runtime/src/array/sort.rs
  • crates/perry-runtime/src/array/splice_slice.rs
  • crates/perry-runtime/src/array/spread_dense_tests.rs
  • crates/perry-runtime/src/array/storage.rs
  • crates/perry-runtime/src/array/subclass.rs
  • crates/perry-runtime/src/array/subclass_elements.rs
  • crates/perry-runtime/src/array/tests.rs
  • crates/perry-runtime/src/array/tests_from_string_codepoints.rs
  • crates/perry-runtime/src/async_hooks.rs
  • crates/perry-runtime/src/buffer/encode.rs
  • crates/perry-runtime/src/buffer/from.rs
  • crates/perry-runtime/src/buffer/iter.rs
  • crates/perry-runtime/src/builtins/console.rs
  • crates/perry-runtime/src/builtins/formatting.rs
  • crates/perry-runtime/src/builtins/formatting/errors.rs
  • crates/perry-runtime/src/builtins/formatting/util_format.rs
  • crates/perry-runtime/src/builtins/globals.rs
  • crates/perry-runtime/src/builtins/table.rs
  • crates/perry-runtime/src/child_process/registry.rs
  • crates/perry-runtime/src/child_process/value_util.rs
  • crates/perry-runtime/src/gc/census.rs
  • crates/perry-runtime/src/gc/fromspace_scan.rs
  • crates/perry-runtime/src/gc/heap_snapshot.rs
  • crates/perry-runtime/src/gc/tests/array_pointer_slot_enumeration.rs
  • crates/perry-runtime/src/gc/tests/copying.rs
  • crates/perry-runtime/src/gc/tests/copying/all_pointer_elements_7469.rs
  • crates/perry-runtime/src/gc/tests/copying/shift_queue.rs
  • crates/perry-runtime/src/gc/tests/copying/survival_and_malloc.rs
  • crates/perry-runtime/src/gc/tests/helper_stores.rs
  • crates/perry-runtime/src/gc/tests/promote_in_place.rs
  • crates/perry-runtime/src/gc/tests/runtime_roots/callback_scanners.rs
  • crates/perry-runtime/src/gc/tests/runtime_roots/string_slice.rs
  • crates/perry-runtime/src/gc/tests/runtime_roots/transient_handles.rs
  • crates/perry-runtime/src/gc/tests/support.rs
  • crates/perry-runtime/src/gc/tests/young_log_tests.rs
  • crates/perry-runtime/src/gc/verify.rs
  • crates/perry-runtime/src/json/construction_array.rs
  • crates/perry-runtime/src/json/parser.rs
  • crates/perry-runtime/src/json/replacer.rs
  • crates/perry-runtime/src/json/stringify.rs
  • crates/perry-runtime/src/json/stringify_data_record.rs
  • crates/perry-runtime/src/json/stringify_flat.rs
  • crates/perry-runtime/src/json/stringify_nested_records.rs
  • crates/perry-runtime/src/json/stringify_primitive_array.rs
  • crates/perry-runtime/src/json/stringify_primitive_object.rs
  • crates/perry-runtime/src/json/stringify_record_output.rs
  • crates/perry-runtime/src/json/stringify_shape_template.rs
  • crates/perry-runtime/src/json/stringify_tojson_probe.rs
  • crates/perry-runtime/src/json_tape.rs
  • crates/perry-runtime/src/json_tape/cached_read.rs
  • crates/perry-runtime/src/node_stream_json.rs
  • crates/perry-runtime/src/object/alloc.rs
  • crates/perry-runtime/src/object/assert.rs
  • crates/perry-runtime/src/object/delete_rest.rs
  • crates/perry-runtime/src/object/field_get_set/entries_shape.rs
  • crates/perry-runtime/src/object/field_get_set/enumeration.rs
  • crates/perry-runtime/src/object/field_set_by_name/fast_paths.rs
  • crates/perry-runtime/src/object/gc_slots.rs
  • crates/perry-runtime/src/object/groupby.rs
  • crates/perry-runtime/src/object/has_own_helpers.rs
  • crates/perry-runtime/src/object/keys_lookup.rs
  • crates/perry-runtime/src/object/native_module/namespace_builders.rs
  • crates/perry-runtime/src/object/object_ops/accessors.rs
  • crates/perry-runtime/src/object/spill.rs
  • crates/perry-runtime/src/param_type_guard.rs
  • crates/perry-runtime/src/promise/combinators.rs
  • crates/perry-runtime/src/promise/then_probe.rs
  • crates/perry-runtime/src/proxy/put_value.rs
  • crates/perry-runtime/src/set.rs
  • crates/perry-runtime/src/string/char_ops.rs
  • crates/perry-runtime/src/string/mod.rs
  • crates/perry-runtime/src/string/slice_ops.rs
  • crates/perry-runtime/src/string/slice_range.rs
  • crates/perry-runtime/src/string/slice_tests.rs
  • crates/perry-runtime/src/string/split.rs
  • crates/perry-runtime/src/string/suffix_cursor.rs
  • crates/perry-runtime/src/thread.rs
  • crates/perry-runtime/src/typed_feedback.rs
  • crates/perry-runtime/src/typedarray/transform.rs
  • crates/perry-runtime/src/util_promisify.rs
  • crates/perry-stdlib/src/async_local_storage.rs
  • crates/perry-stdlib/src/sqlite/better.rs
  • scripts/addr_class_allowlist.txt
  • scripts/gc_runtime_root_holders.json
  • scripts/raw_handle_debt_baseline.txt
  • scripts/raw_handle_debt_files.txt
  • test-files/test_gap_array_shift_observable.ts
  • test-files/test_gap_gc_array_shift_queue.ts
  • test-files/test_gap_gc_string_suffix_cursor.ts
  • test-files/test_gap_string_slice_utf16_suffix.ts
  • test-parity/gc_repsel_corpus.txt

📝 Walkthrough

Walkthrough

This pull request adds dense Array.shift() queue storage, centralized logical array addressing, UTF-16 string boundary handling, compiler suffix cursors, runtime and GC tests, benchmark harnesses, and release metadata.

Changes

Array shift queue and storage

Layer / File(s) Summary
Dense queue storage and GC integration
crates/perry-runtime/src/array/*, crates/perry-runtime/src/gc/*
Dense shifts clear slots and advance the logical element start. Growth normalizes shifted storage. GC scans the logical live range.
Compiler and runtime address resolution
crates/perry-codegen/src/*, crates/perry-runtime/src/*, crates/perry-ext-*/*, crates/perry-stdlib/src/*
Array access paths use array_elements_ptr or array_elements_addr instead of fixed header offsets.
Array validation and regression coverage
test-files/test_gap_array_shift_*.ts, crates/perry-runtime/src/array/shift_queue_tests.rs, crates/perry-runtime/src/gc/tests/copying/shift_queue.rs
Tests cover observable shift behavior, aliases, growth, holes, moving GC, and young-reference retention.

String slicing and suffix cursors

Layer / File(s) Summary
UTF-16 range copying
crates/perry-runtime/src/string/slice_range.rs, crates/perry-runtime/src/string/slice_ops.rs
slice, substring, and substr share UTF-16 boundary handling that preserves split surrogate halves and lone-surrogate metadata.
Compiler suffix lowering
crates/perry-codegen/src/collectors/suffix_strings.rs, crates/perry-codegen/src/expr/suffix_cursor.rs, crates/perry-runtime/src/string/suffix_cursor.rs
Eligible non-escaping suffix locals use scalar cursor state and runtime operations for length, advancement, and character reads.
String validation and benchmarks
test-files/test_gap_string_*.ts, crates/perry-runtime/src/string/slice_tests.rs, crates/perry-runtime/src/gc/tests/runtime_roots/string_slice.rs, benchmarks/string_slice/*
Tests cover UTF-16 bounds, malformed data, aliases, closures, GC relocation, and benchmark parity.

Benchmark and release support

Layer / File(s) Summary
Array-shift benchmark
benchmarks/array-shift-10060/*
Adds deterministic queue-drain and push benchmarks, measurement scripts, before/after reports, checksums, and validation documentation.
String-slice benchmark
benchmarks/string_slice/*
Adds ASCII and Unicode benchmark workloads, measurement scripts, artifact manifests, result data, and validation metadata.
Release and audit metadata
Cargo.toml, CLAUDE.md, changelog.d/*, scripts/*, test-parity/gc_repsel_corpus.txt
Updates the workspace version, changelog entries, audit allowlists, debt baselines, and GC witness registration.

Estimated code review effort: 5 (Critical) | ~90 minutes

Change: Bug fix · Severity of issue fixed: Medium

Possibly related PRs

  • PerryTS/perry#8719: Adds stable packed-loop lowering that uses the array storage access contract changed here.
  • PerryTS/perry#8755: Carries stable packed-loop and array-index lowering paths updated for logical array element addresses.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch train163

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant