Skip to content

Skip discarded archive payloads through bounded owned cursors - #1071

Merged
helly25 merged 1 commit into
mainfrom
perf/forward-archive-skips
Nov 3, 2026
Merged

helly25 merged 1 commit into
mainfrom
perf/forward-archive-skips

Conversation

@helly25

@helly25 helly25 commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Advance eligible owned cursors over discarded archive payloads instead of reading every byte. This makes listing large uncompressed archives and reaching later members substantially faster while preserving sequential fallbacks, selected content, errors and shared read budgets.

AG;DR

Dependency #1070 is merged. This branch starts directly at current main 44e7e30102eb6c6ae122135189b6c62340c35bdb; its complete parent tree equals the qualification parent 38ac542f621abd9fb2610e0d01f8646f42428ab3. The six candidate source files retain their exact verified bytes after synchronization. Fresh checks in this PR's own context determine readiness; earlier PR checks are not substituted.

Implementation and resource contracts:

  • Add optional ReadStream::Skip: absence means unsupported without consuming input; a present count advances the existing cursor, may be short, and must not exceed the request. Zero alone is not EOF. Existing Read/ReadInto remain available and share the same position.
  • Memory cursors advance their existing offset. Host cursors lazily fstat their exclusively owned read-only regular-file descriptor, clamp by its observed size and lseek that same descriptor. They keep access/position/size state; ordinary reads gain no metadata syscall. No source-path reopening or cursor allocation per skip.
  • Register libarchive's skip callback only for requests of at least 64 KiB. Incapable sequential/member/decoder cursors fall back to read/discard. Leave XAR seeking unchanged; do not enable a ZIP index or decoder parallelism.
  • Keep callback buffers alive. Latch provider failures or oversized counts and make the fallback read report the error before touching the cursor again; the pinned libarchive skip proxy does not reliably terminate on a negative custom callback result.
  • Preserve memory/replay budgets, session reservation release and parent ownership. SourceSession gains no data fields. No writable host handle, host mutation, new dependency, worker, lock or CLI flag.
  • Make cursor ownership explicit by deleting object copy/move operations; callers transfer unique_ptr ownership. Sources explicitly retain their default copy/move operations. An inherited missing-special-members warning was found by the correct header filter and fixed.
  • Consolidate native metadata/seek status conversion into a value-returning read-only adapter, exercised by existing real syscall failures. No test-only production seam, executable-line exclusion or weaker coverage policy.

Verification:

  • Eleven new regressions cover unsupported/default cursors, zero/partial/max requests, shared positions, retained owners, native descriptors after path deletion, large discarded payloads, owned buffers, selected content, small/compressed inputs, sequential fallbacks, latched failures, invalid counts and truncation. Compile-time assertions prevent cursor object copies/moves.
  • All 313 selected root/extension release suites pass on macOS and Linux. Five focused ASan suites pass per host; five MSan and five TSan suites pass on Linux. Linux requires functioning FUSE during the complete suite.
  • Explicit clang-tidy checks cover all five actual modified translation units and both logical/physical read_source.h aliases on each host. The final status-helper implementation passes its affected runtime-unit recheck. The already tracked MBO-only default header-filter defect is not silently treated as header coverage.
  • Full Linux production coverage runs all 313 suites and passes every category. This candidate's own patch covers 61/61 lines and 28/28 branches. Absent patch-function metrics are handled by the same policy helper used in CI.
  • Normal enforcing commit and push hooks pass for f68cfa09c944da83946c66fd185ea579ab2876b5, including formatting, header-consumer clang-tidy and Trunk. Fresh current-head CI remains required. Qualification retains initial failures and final successful logs separately.

Performance and costs:

  • Preserve the first source-reopen experiment separately: both-host memory pilots and a Linux-only native pilot. It adds reopening/allocation and source/session state; the chosen cursor design avoids that work. Do not describe its incomplete native qualification as cross-platform.
  • The final four-participant matrix compares parent/API-only/eager/bounded variants: 37 memory and 40 native cases x three sessions per host. Native cases cover small/boundary/large TAR, stored/deflated ZIP, gzip TAR, nested TAR and ordinary files with one/three requested workers.
  • Separate final confirmation uses parent, a byte-identical parent copy, preceding selected and final selected binaries: 12 memory cases x five sessions, and 20 native cases x three sessions per host. Each native participant/case/session discards a warmup, retains fastest seven of nine interleaved measurements, and checks complete output/exits against independent Python fixtures. Linux uses tmpfs plus CPU affinity; macOS uses warm APFS. Requested workers do not imply parallel archive decoding. No same-host timing overlaps builds, lint, tests or instrumentation.
  • Final large native TAR/stored-ZIP cases improve roughly 79-87% Linux and 50-65% macOS. Nested TAR improves about 11%/7% for listing and 11%/6% for final-member content. Gzip-TAR confirmation controls stay within roughly 0.3%. Ordinary/boundary native controls cost at most 0.79% Linux and 0.56% macOS in confirmation.
  • Actual memory TAR listing of 100 x 1 MiB improves about 97% Linux / 92% macOS; reading its final member improves about 96% / 85%.
  • Retain slower and unstable virtual-only controls. One initial macOS 4 KiB kernel is +59.72%; final confirmation is -3.80% while an identical reference copy moves +4.75%. Other final macOS controls retain costs up to 4.42%. Native identical-reference movement is -10.61% to +1.63% Linux / -1.33% to +0.59% macOS. These are not universal speedup claims, and populations are not pooled.
  • The ownership-declaration refinement has identical CLI/kernel code sections on each host. The later status-adapter refinement changes CLI code and is included in final native confirmation; both memory kernel code sections remain identical. Complete debug-bearing binary hashes are retained independently.
  • Final full unstripped CLI grows 6,712 bytes Linux and 17,040 bytes macOS versus the parent, approximately 0.04% / 0.21%.

Documentation and retained work:

  • CHANGELOG, design, performance analysis, TODO and release-site mapping link docs/benchmarks/forward-cursor-skipping-20261103.md.
  • Its 5,017,324-byte lossless archive contains 803 labeled evidence files; SHA-256 b66c5d4a2785e9d4659314293b1ffd5545459d6c379a9e70bc54691d2f309461. It retains sources, scripts, compiler/configuration/link/binary identities, raw timings, independent oracles, validation/coverage logs and exact parent synchronization proof.
  • The exact size oracle independently reproduced an existing streamed-ZIP data-descriptor size limitation on the unchanged parent. Keep the source/proof and a separate correctness TODO; in-memory large-payload kernels use TAR, while native ZIP fixtures carry verified local-header sizes. Do not weaken the oracle or silently enable an unbudgeted seekable index.
  • This completes O19b1 forward skipping. O19b seekable indexing, repeated member/replay reuse and decoder admission remain open alongside all other retained performance candidates.

@helly25
helly25 merged commit 79f8b37 into main Nov 3, 2026
32 checks passed
@helly25
helly25 deleted the perf/forward-archive-skips branch November 3, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant