Skip to content

Reuse archive input buffers without per-chunk allocations - #1066

Merged
helly25 merged 26 commits into
mainfrom
perf/reuse-archive-input-buffer
Nov 3, 2026
Merged

helly25 merged 26 commits into
mainfrom
perf/reuse-archive-input-buffer

Conversation

@helly25

@helly25 helly25 commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Archive reads now reuse caller-owned input storage instead of allocating a new string for every chunk. This reduces allocation churn and improves large TAR workloads while preserving decoder fallbacks, read budgets and error handling.

AG;DR

Dependency and merge order: #1065 and #1059 are merged. This PR is now based directly on main; #1067 remains its dependent follow-up. The synchronization retains both the archive-buffer implementation and all five merged dependency updates. Existing cross-platform timing below was collected on the fixed #1059 parent before #1065; it is historical qualification, not a claim that the updated dependency configuration has already been remeasured. Current-head CI verifies the synchronized configuration.

Implementation:

  • Each archive source session lazily allocates one uninitialized 64 KiB array when its cursor supports ReadInto; the libarchive callback borrows only the initialized byte range.
  • Legacy sequential decoders retain their owned-string reads. A capability change releases the inactive storage, retaining the existing single shared reservation. Reader destruction precedes callback-state destruction.
  • Short reads, EOF and source errors retain their behavior; impossible counts are rejected before exposing a view. No flags, dependencies, locks, worker changes, format parsers or host-I/O paths are added.
  • Nine isolated behavior cases cover stable storage, alternating capabilities, short reads, errors, impossible counts, EOF, legacy fallback, compression and shared-budget release. The reusable manual archive benchmark is indexed for scoped clang-tidy.

Cross-platform qualification on the fixed #1059 parent:

  • Both hosts: all 324 extension-enabled release suites pass; four focused ASan suites pass. Linux also passes four focused TSan and five focused MSan suites, with FUSE required for release testing.
  • Per host: 228 native rows and 144 in-memory runs, followed by 40 native confirmation rows and 63 memory confirmation runs. Independent fixtures and complete output are checked; timing never overlaps builds, lint or instrumented diagnostics on its host.
  • Large TAR native cases improve approximately 2–3% on Linux and 4–7% on macOS; confirmation retains gains of roughly 2% and 4–6% respectively. Compressed/ZIP and ordinary/small controls are retained, including slower cases. This is not a general archive-speedup claim.
  • A 10,000-member caller-owned listing drops cumulative requested allocations from 60,742 to 60,039 and requested bytes from about 50.6 MB to 4.57 MB. These ASan counts are separate diagnostics, not release timings or peak-memory savings. Observed maximum RSS is separately reported.
  • Unstripped full CLI changes: Linux +3,664 bytes; macOS unchanged. No dependency or licensing changes.

The report docs/benchmarks/archive-input-buffers-20261102.md links lossless source/configuration, timing, correctness, allocation and earlier rejected/prototype evidence. Every final variant was formatted with its actual filename before building; final source guards prevent validation against a builder-restored variant. Earlier populations remain separate rather than being attributed to the final code. O19b ZIP seeking/indexing and decoder admission remain open.

Local scoped clang-tidy passed for the three changed C++ translation units. Normal enforcing commit/push hooks and this PR’s current-head CI determine publication/readiness; local development validation does not substitute for those checks.

Base automatically changed from perf/borrow-normalized-shingles to main November 3, 2026 01:24
@helly25
helly25 merged commit 09e6bbf into main Nov 3, 2026
32 checks passed
@helly25
helly25 deleted the perf/reuse-archive-input-buffer branch November 3, 2026 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant