Skip to content

Reuse caller-owned buffers for memory and archive member streams - #1068

Merged
helly25 merged 32 commits into
mainfrom
perf/caller-owned-streams
Nov 3, 2026
Merged

helly25 merged 32 commits into
mainfrom
perf/caller-owned-streams

Conversation

@helly25

@helly25 helly25 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Memory-backed and archive-member streams can now fill caller-owned buffers, avoiding per-chunk strings during archive reads. This substantially reduces allocation churn for nested containers while preserving read errors, sparse contents and shared budgets.

AG;DR

Dependency and merge order: #1067 and #1066 are merged. This PR is synchronized directly onto main; the synchronization changes only test execution allowance and ancestry, preserving the qualified runtime/test/benchmark implementation. Runtime qualification remains pinned to 492c1ef635837571d26c5c45b78f7c6ef1776798 with the #1065 dependency updates. Local qualification is not a substitute for this PR’s fresh main-based CI.

Implementation:

  • MemoryStream advertises ReadInto, copies only the initialized bytes available in the supplied span, and advances its independent cursor once. Owned reads remain supported.
  • Archive MemberStream exposes the same optional capability with the existing 64 KiB cap and decoder/error handling. Owned reads share the checked path. The existing archive session storage and shared reservation are reused; no new index, seeking, worker, lock, format parser, exception path or host-I/O route is introduced.
  • Explicitly incapable benchmark cursors preserve the legacy control. Separate benchmark families exercise actual memory sources and nested member sources; support code is common to all runtime variants. The benchmark gains a direct dependency on the existing mbo status macros target, not a new external dependency.
  • Committed tests cover empty/binary input, zero spans, independent offsets, untouched tails, mixed owned/borrowed positions, source lifetimes, 64 KiB member caps, late errors, four nested compression combinations, budget release, and sparse PAX/gzip holes with positive partial reads.

Qualification and limits:

  • Both hosts pass 313 selected root-and-extension release suites and five focused ASan suites. Linux also passes five focused MSan and five TSan suites, with FUSE required for the complete release selection. Actual scoped clang-tidy passes for all five changed translation units on both hosts. Source guards verify restoration after source-swapping builds and diagnostics.
  • Each host has 300 native rows and 576 memory runs across baseline, memory-only, member-only and combined variants. Independent confirmation adds 50 native rows and 81 memory runs, including a byte-identical baseline repeat. A separate two-case fixed-100,000-iteration check retains 18 runs per host. Complete native stdout/exit codes and memory fixture contents are checked. Timing never overlaps builds, lint or instrumentation on its host.
  • Large nested TAR native confirmation improves roughly 0.6–1.2% on Linux and 2–2.5% on macOS. Nested in-memory gains are larger on macOS. ZIP/gzip and ordinary/small controls remain decoder/startup dominated; slower controls are retained, including approximately 2–4% command movements and 3.6–3.7% Linux tiny-kernel movement in the fixed-iteration population. This is not a general archive-speedup claim.
  • A 10,000-member nested listing reduces cumulative requested allocation bytes from about 97 MB to 5 MB and requests from 61,503 to 60,097. These are separate ASan diagnostics, not release timing or peak/RSS savings. A tiny memory source instead requests about 54 KiB more with the same request count and existing budget reservation.
  • An isolated rewrite retaining a duplicate owned member-read implementation did not consistently improve controls and was rejected. Its 24 native rows and 72 memory runs per host remain separate. Some reference-repeat and byte-identical binary populations move substantially; the report explicitly limits attribution of small timing differences.
  • Full CLI sizes, all four runtime source/configuration identities, formatting gates, exact-output oracles, allocation diagnostics, failed setup and rejected pilot are retained in docs/benchmarks/caller-owned-streams-20261103.md and its lossless evidence archive. The initial nested warmup omitted depth two and was rejected before accepting any timing row; the qualified matrix uses --archive=all --archive-depth=2.

Normal enforcing commit and push hooks are required. Current-head CI determines merge readiness. O19b seeking/index/replay/admission and O19d direct final-buffer filling remain open; the broader performance goal is not complete.

The inherited CI correction raises the Linux benchmark-build limit to 60 minutes after #1067’s cold build plus expression measurements exceeded 30 minutes. Cases, repetitions and correctness checks are unchanged. Full CLI footprint: Linux +2,424 bytes; macOS +640 bytes (unstripped).

MSan integration-test deadline recovery:

  • Run 41568084072 reached the rg_full stdin cases after earlier cases passed, then the harness exceeded the target's 60-second limit. There was no MemorySanitizer finding in that log.
  • Set only //xff/cli:rg_full_test to timeout = "moderate" (300 seconds), retaining its small resource classification, every test case and all assertions. The exact published runtime plus this test attribute passes the complete suite locally under MSan in 5.4 seconds; private performance prototypes were explicitly excluded and restored afterward.
  • The fix is included with the required post-Borrow cached digests and streamline output records #1067 synchronization, avoiding a redundant rerun of the old base. Evidence: https://github.com/mboworks/xff/actions/runs/41568084072/job/125696569970 . The failed old context does not count toward readiness.

Base automatically changed from perf/borrow-output-values to main November 3, 2026 03:38
@helly25
helly25 merged commit 58ba30e into main Nov 3, 2026
32 checks passed
@helly25
helly25 deleted the perf/caller-owned-streams branch November 3, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant