Skip to content

Share collection name and root storage with owned paths - #1041

Merged
helly25 merged 1 commit into
mainfrom
experiment/collection-owned-path-views
Nov 1, 2026
Merged

helly25 merged 1 commit into
mainfrom
experiment/collection-owned-path-views

Conversation

@helly25

@helly25 helly25 commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Reuse basename and root views within a collection-owned path whenever their bytes match its suffix/prefix. This cuts duplicated retained text without adding an interner or ownership wrapper. The large long-path/four-collection case is 26.6% faster on Linux and 8.1% faster on macOS, with about 37% less peak RSS on both.

AG;DR

Implementation and lifetime

Copy the full path once. Reuse exact suffix/prefix views for the basename and root; independently copy noncontained values. The arena owns all text and preserves addresses across growth and moves. No view borrows a walker buffer, and logical row/byte budget accounting remains unchanged. There is no new allocation policy, flag, dependency, lock, cache or executor layer.

Add an explicit noncontained-name/root lifetime test. Increase the existing growth/move fixture to 600-byte paths so it crosses a real 1 MiB arena block after #1040; the former 150-byte fixture no longer did. Existing empty, length-delimited, caller-destruction, ownership and budget cases remain applicable.

Retain the full O01-O22 optimization inventory, existing EP/JC/P references and acceptance criteria in docs/performance-analysis.md and TODO.md. This PR implements O01 only. Record closed #991/#1001 policies and superseded #1000 diagnostics without declaring the remaining unique executor work rejected. Publish the new decision report through the release-site map and update the changelog.

Evidence and limits

Baseline is merged main 16e70cb97de93c7ab4ae8c9e990e1842d50cc528, including #1040's fixed 1 MiB arena. Fresh release binaries use LLVM 23 and ThinLTO. Linux uses drogon's Ryzen 9950X KVM guest, tmpfs and CPU affinity; macOS uses this M5 Pro, warm APFS and worker allowances without affinity. Compare within-platform pairs, not absolute times between the two hosts.

Native measurements cover 26 cases, three sessions, discarded warmups and nine interleaved rounds retaining the fastest seven. There are 3,120 successful native invocations across both hosts, each checking count/size output independently. A separate 12-case collection kernel has three shuffled sessions and nine repetitions. Its checksum prevents dead-code removal but is not an independent output oracle. Untimed arena accounting and compiler-action inspection are separate from release timing.

Long-path 100k files, three workers, four collections Linux macOS
Elapsed change -26.57% -8.08%
Main peak RSS 357.05 MiB 364.36 MiB
Candidate peak RSS 225.41 MiB 230.77 MiB

At 100k long-root entries, actual retained text drops from 28,677,780 to 14,388,890 bytes, and fixed arena blocks from 28 to 14. One-thread collection kernels improve 17.22% on Linux and 25.68% on macOS. These are retained-collection results, not universal listing or process-memory claims.

Small costs remain: native paired medians increase up to 1.33% Linux / 1.64% macOS; kernel counterexamples reach 4.08% / 5.26%. All samples and counterexamples remain in the evidence. The four-thread kernel constructs separate collections and does not make production collection concurrent.

Measurement report and docs/benchmarks/collection-text-sharing-20261101.json.gz retain full raw samples, source patches, scripts, binary hashes, test logs, scoped lint output and separate accounting. Full-release C++ action inspection found -fno-exceptions throughout inspected XFF/extension/dependency target actions on both hosts; this does not describe prebuilt system libraries.

Validation and sequencing

  • Both baseline and candidate passed release //xff/engine:collect_test and //xff/engine:run_test on this Mac and Linux drogon.
  • Normal commit and push hooks passed, including scoped clang-tidy for the collection implementation/tests, formatting, policy checks and Markdown alignment.
  • Complete workspace plus extension release suites (//... and tools/extras.py --wildcards, --config=clang_release --config=xff_docs): macOS passed 302/302. Linux initially passed 301/302; full_extras_test expected FUSE to be unavailable because its required-capability marker was absent, whereas drogon successfully mounted the archive. Rechecking that target with the existing --config=xff_fuse_tests_required passed. The failure was an incorrect local test configuration, not a source change; no test expectation or implementation was changed.
  • Published-head GitHub CI is fully successful. The first MSan attempt timed out in the unrelated rg_full_test; the targeted retry passed on the unchanged head. All other required jobs passed without changing the PR head.

Standalone on current main, no unmerged dependency. The new goal retains all candidates and requires separate implementation, native verification and PR/CI qualification; no blanket acceptance of unmeasured candidates is implied by the inventory.

@helly25
helly25 merged commit cf23601 into main Nov 1, 2026
61 of 63 checks passed
@helly25
helly25 deleted the experiment/collection-owned-path-views branch November 1, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant