[helly25]

Keep directory read-ahead bounded and cancel work the traversal no longer needs, while reusing the command's existing workers. Cached siblings no longer hold execution slots needed for deeper traversal, and foreground work can run between speculative reads.

AG;DR

Scope and dependency

Draft follow-up stacked on PR #996, based on command-ownership head 2a0e3343d6be1e2e3a28327f1da9068774cf7a09. This isolates directory-frontier scheduling from useful-remaining-work admission and native allocation/timing experiments. No module, LLVM, linker, build default, benchmark driver, dataset selection, resource-accounting or platform-security settings change.

Implementation commit: 769d9cc72a11213a2c73c605532b56e526821561; tree: 0a3db67d5d703ad5d9782b0c2abea5ac95a6a826.

Implementation and invariants

• Retain the caller-inclusive --jobs=N command budget and at most N-1 persistent background threads. Directory/stat/content/comparison phases and nested mounted-filesystem walks reuse the same executor; workers remain leaf jobs and cannot submit/wait recursively in their own pool.
• Replace whole-sibling directory batch drains and all-child-aligned future/path arrays with an ordered frontier. Each admitted frontier holds at most twice its participating background-worker count in requests. All frontiers also share at most twice the started worker count in unfinished directory reads. Each speculative job performs one directory read plus its required metadata, then yields at a job boundary.
• Use separate foreground and speculative FIFO queues, alternating classes while both have work. This is nonpreemptive dispatch fairness, not fairness within a large stat/content/comparison drain or an interruptible VFS call. Per-directory dispatch adds queue/allocation work compared with whole-batch drains; task granularity must be measured, not assumed efficient.
• Track storage independently of unfinished-read slots. Completing a read releases its execution slot while retaining the result's byte charge. Consuming a parent request does not immediately refill that parent ahead of descent, allowing deeper frontiers to use available slots while older siblings remain cached.
• A move-only reservation owns a shared budget that can outlive its executor. Admission charges bookkeeping/path storage before dispatch. Retention replaces the total byte charge, cannot reduce it below the initial charge, and rejects an unrepresentable or over-budget result. Finish() is idempotent; destruction releases the remaining charges. Abseil mutexes guard shared accounting and dispatch state; result publication uses the existing Abseil completion protocol, not cancellation atomics.
• The provisional represented-storage allowance is 8 MiB, with a 512-byte bookkeeping estimate per request plus its path. Speculative listing charges include vector/string capacities and status messages/payloads. Inline commands do not allocate dispatch queues, a read-ahead budget, or frontier deques. These are conservative accounting charges, not measured allocator overhead or a hard RSS/total-traversal-memory cap.
• Oversized completed speculative results are discarded and read on demand if later required. A result already required by the caller while its VFS read is running transfers as ordinary traversal working memory without an additional lookahead payload charge, avoiding duplicate I/O. Required/ancestor listings, in-flight VFS buffers and allocator overhead are outside this speculative budget. Byte pressure can reduce useful overlap and discarded results can require repeat I/O; these are explicit performance controls.
• Known prune decisions are excluded before admission. Destroying an unused request cancels it; closing a frontier cancels the entire group. Workers check cancellation before and after VFS access. Queued canceled work skips filesystem access; an in-flight call must finish. Walk destruction still drains its leaf jobs before borrowed state disappears, leaving the executor reusable.
• Preserve traversal sorting, ordered output, post-order, root/depth/device/loop controls and archive ownership. Update the --jobs registry entry, generated XFF.md, parallel design and audit status in the same change. Add only narrow direct Cord/time dependencies required by storage accounting and controls.

Correctness controls

Executor controls exercise shared slot/byte admission, concurrent retention, saturating rejection, ownership release, reservations moved across threads/outliving the executor, idempotent completion, minimum bookkeeping charges and deterministic queue-class alternation.

In-memory traversal controls cover:

• pruning a queued request before it reaches the filesystem;
• stopping after the first visited child of a 1,000-sibling root with at most the four initially admitted reads under three participants, followed by executor reuse;
• nested frontiers at all six path-order modes, pre/post-order traversal, and byte allowances of 0, 128, 1,024, 8,192 and 1,048,576, with serial-equivalent visits and released accounting;
• deterministic nested progress with one background worker while a completed parent sibling remains cached: read-ahead FIFO markers prove both nested slots run before their first listing is consumed;
• an oversized speculative error payload being discarded, reread when required and reported once.

Existing command-budget, eager-stat, matching, comparison, ordering, archive and in-flight-stop tests remain. Markers/rendezvous are correctness synchronization, not accepted timing measurements.

Validation

• Final source passes bazel test --config=clang --config=xff_docs //... --test_output=errors: all 241 targets, including documentation/notice drift tests. This final run executes 87 targets and reuses valid cache entries for the rest; the initial all-extras run executed all 241.
• Each of the eight affected engine suites (command_workers_test, walk_test, walk_parallel_test, parallel_compare_test, parallel_match_test, rg_test, run_executor_test, run_test) passes under --config=clang --config=asan (ASan/UBSan) and --config=clang --config=msan.
• The same eight suites each pass ten repetitions under --config=clang --config=tsan --runs_per_test=10, including the expanded 60-mode/order/budget combinations, nested progress, cancellation and oversized error controls. Earlier narrower TSan repetitions also pass, but the final eight-suite run is the verification scope reported here.
• The canonical ./compile_commands-update.sh completes with all 297 expected first-party sources present. The normal commit hook passes enforcing clang-tidy, formatting, repository policies and Trunk checks. Initial lint passes identified optional engagement and a redundant lambda parameter-list issue; explicit guards and checked access resolve those findings without disabling checks or weakening presence assertions. No hook is bypassed. The worktree is clean and git diff --check passes.
• Normal push hooks pass, including enforcing clang-tidy and the repository policy/format checks; the remote branch matches the clean local implementation commit. No hosted check or native performance acceptance is inferred from local success.

Portability and remaining acceptance work

The implementation uses portable C++/Abseil and the existing thread-safe VFS interface. It introduces no Linux-only runtime calls or macOS configuration changes. Linux tests and hosted macOS correctness checks do not establish native M5 Pro performance safety.

Inherited directory admission (64 independent siblings), eager-stat admission (512 entries / 128-entry chunks), content bounds and accumulated-past-work cold matcher policy are unchanged for isolation. Count alone is not remaining cost, and completed work must not justify starting workers. Remaining-cost admission and measured task sizing remain necessary follow-ups.

The caller participates in foreground stat/content/comparison drains, but still waits when its required directory listing is owned by a queued/running speculative read. Thus an admitted directory-only frontier under --jobs=2 can use only its one background reader while the caller waits; thread reuse and a participant ceiling do not prove full useful participation. Caller assistance during those waits needs a separate correctness/performance control and remains open.

Separate draft PR #998 now implements caller-owned uncached reads and exactly-once claims of required queued reads. Its before-change control reproduces the one-reader gap, and its local all-extras suite plus ASan/UBSan, MSan and repeated TSan controls pass. That follow-up is not part of this PR's head; already-running required reads still require waiting, and useful-remaining-work admission, dispatch/grain costs and native performance acceptance remain open.

No native timing, allocation or RSS improvement is claimed. PR #995 retains unchanged-baseline provenance, allocation/counter controls and the isolated one-time Linux/x86 Abseil TSC calibration mechanism; this candidate does not disable that clock or attribute its 1 ms / 2 ms sleeps to each pthread creation. Required/ancestor listing lifetime is a separate memory finding, not solved by the speculative budget.

Keep this PR draft pending repeated/reordered complete prepared-grid A/B measurements with correct output hashes, metadata/stat-heavy and slow-storage controls, native allocation/RSS checks, and equivalent M5 Pro sampling, P/E-core accounting and A/B controls. Builds and instrumentation must not overlap accepted timing collection. This PR addresses bounded-frontier ownership/cancellation and job-boundary dispatch; it does not complete the original benchmark audit or establish that every worker now has enough useful remaining work.

Current qualification disposition (2026-11-07)

This proposal remains open as unqualified research; its branch and implementation are preserved. The successful checks above belong to its historical head/base context. They are not validation against current main, and this note changes no source, branch, base, CI policy or acceptance threshold.

PR #1125 has merged as 9bf1aad510d843d1d6338fbdf23107a30d339730. Its shared-stage-worker report rejects six separately implemented reuse designs, including a repeatable 15.55% Linux small-identical/1,000-pair/four-worker regression. That result does not establish that every unique mechanism in #992/#996/#997/#998 is rejected or superseded. Main retains specialized pools.

Preserve the unique command-wide unfinished-read/represented-byte reservation mechanism, cancellation before/after VFS calls, and alternating foreground/read-ahead dispatch. Main's typed directory/stat pool and bounded per-frontier requests do not implement that complete combination. Dropping an unused main request releases its completed result, but is not equivalent to this proposal's queued-work filesystem-access cancellation or command-wide retained-byte accounting.

The executor/copy audit demonstrates about 69% lower peak RSS versus #996 in its 65,536-file metadata fixture on both platforms, while the marginal Linux nested-listing time costs 17.5%. Neither figure is a general main-relative bound or current-head result. This PR still consumes the unqualified #996/#992 infrastructure and caller-inclusive policy; historical green child CI cannot promote it to main readiness.

The concrete next qualification is an isolated adaptation onto main's current typed slots/frontier, without the rejected admission/aggregate-budget policies. Preserve byte-reservation lifetime, minimum/saturating charges, exact errors/order, cancellation/draining and independent nested allowances. Measure enqueue/allocation cost, oversized-result rereads, completed-versus-in-flight storage, heterogeneous/slow VFS behavior and native RSS/time on both platforms. A represented-storage budget is not a total RSS or required-listing bound. This remains a useful open mechanism, not a fully rejected or fully superseded PR.