Skip to content

Pack owned matcher and deferred entry text - #1049

Merged
helly25 merged 2 commits into
mainfrom
perf/packed-visit-snapshots
Nov 2, 2026
Merged

helly25 merged 2 commits into
mainfrom
perf/packed-visit-snapshots

Conversation

@helly25

@helly25 helly25 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Pack pending matcher and deferred-entry text into one owned buffer, reducing allocation traffic while preserving entry lifetimes and ordered output. Long overlapping snapshots use about two-thirds fewer allocation calls; native elapsed gains are modest and vary by workload.

AG;DR

Implementation and ownership

  • Add OwnedVisit: one owned byte string plus offsets for path, basename and root. Exact suffix/prefix ranges share bytes; independent values append to the same allocation. Resolve views against the current string so short-string moves and vector relocation remain safe.
  • Preserve copied metadata, original root index, filesystem observers and archive filesystem owners. Moves disengage the old text slices/observer. Traversal-only metadata loaders remain outside retained snapshots, as before.
  • Use the record for pending matcher batches and deferred candidates; construct directly with emplace_back. Keep the existing vector-consumption/ordered-result protocol and worker admission policy.
  • Shard cohorts borrow names from the retained snapshot. The existing complete/superfluous/dedup tests caught a temporary-name lifetime error in the first prototype; the selected implementation passes those tests and sanitizer checks.
  • Collection arena policy, logical text lengths, safety controls and CLI behavior retain their existing contracts. No new batch arena, shared text owner or buffer-recycling API is introduced.

Measurements and decisions

Decision report and lossless evidence retain all source variants, raw repetitions, fixtures, executable/source hashes, scripts, allocation counters and qualification logs on this macOS host and Linux drogon.

  • For 1,000 long overlapping snapshots, allocation calls fall from 3,001 to 1,001. Requested bytes fall from 512,000 to 360,000 on macOS and 536,000 to 368,000 on Linux. Short-string text remains allocation-free. These are isolated ASan allocation counters, never sanitizer timings or live-memory claims.
  • The corrected counterbalanced native confirmation improves Linux 10,000-file rg filtering by 3.63% and 10,000-entry deferred shard grouping by 3.37%; corresponding macOS changes are -0.58% and -0.75%. Do not infer that every command is faster. The Linux large-comparison control is +3.33%, with changing signs across sessions, and all slower samples remain in the report.
  • Reject forced out-of-line construction: its two-host controls show no consistent advantage.
  • Final unstripped build outputs shrink by 560 bytes on each host: macOS 4,052,000 -> 4,051,440; Linux 12,295,424 -> 12,294,864.

The initial qualification-only driver rotated participants and reversed odd rounds, which cancels rotation for two variants. Preserve those populations explicitly; corrected rotation-only confirmation repeats 32 representative cases with three sessions, nine rounds/fastest seven and independent per-invocation output oracles on each host. The published benchmark runner already uses rotation without reversal and is unaffected. TODO.md records the retrospective audit of older two-variant qualification scripts.

Validation and portability

  • Complete development release suites: 306/306 on macOS and Linux, including composable extras and Linux's required-FUSE integration.
  • After direct-construction integration: five affected release/ASan suites on each host, Linux ownership/matcher/rg TSan suites, and the final test-fixture checks pass.
  • Scoped local clang-tidy covers all seven affected translation units; the enforcing commit hooks pass. Current pushed-head GitHub checks remain required before merge.
  • Tests cover destroyed caller text, shared/fallback bytes, counted slices and embedded nulls, empty/default/moved-from records, short/long copies and moves, vector growth, metadata/root identity, and archive-owner retention through the last snapshot.
  • Linux native timing uses tmpfs and 1/3/10-CPU affinity; macOS uses warm APFS without affinity enforcement. Separate in-memory controls use 1/4 worker allowances. Builds/lint/sanitizers do not overlap timing on the same host. Raw CPU time and child peak RSS are retained; first-output latency is not separately measured here.

The evidence archive has a narrowly scoped large-file-hook exception alongside the existing qualification reports; the normal archive-ignore rule requires explicitly staging this intended artifact. Documentation, the release-site mapping, CHANGELOG.md, and the O09/TODO tracking are updated together.

Based on merged #1048. The measured parent 5ca865a453acba2437909ec489c528b989bdf315 and merged main 7cf2a52441d626888b5840d09a4646d6dedca6b9 have identical trees; the unpublished branch was synchronized before committing. O10 bounded result-capacity reuse remains separate.

@helly25
helly25 merged commit 901c8fe into main Nov 2, 2026
32 checks passed
@helly25
helly25 deleted the perf/packed-visit-snapshots branch November 2, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant