[helly25]

Let the calling thread read directories alongside the command's reusable workers. A required directory stays on the caller or is claimed from the queue exactly once, so a two-participant walk need not leave the caller idle while one worker performs all directory reads.

AG;DR

Scope and dependency

Draft follow-up stacked on PR #997, based on 769d9cc72a11213a2c73c605532b56e526821561. This separates caller participation from command ownership (#996), bounded frontier/cancellation (#997), and remaining-cost admission or native performance acceptance. There are no module, LLVM, linker, default, benchmark driver, dataset, resource-accounting or platform-security changes.

Implementation commit: 416e45a39e1432aeed3f2aed64f8e3795bac19f1; tree: 0fcaacf7f1d98757a8b54ea5dd24e432da7abd1c.

Finding and implementation

The before-change 65-sibling control preserved ordered output, exact read counts and the thread cap, but failed caller participation. Under --jobs=2, only one background reader accessed child directories, peak concurrent reads were one, and a two-directory rendezvous timed out. Under 3/10 participants, reader identities likewise excluded the caller. Reuse and a cap alone did not demonstrate useful work by the calling thread.

• Defer frontier filling until traversal processes a child rather than filling in the frontier constructor. Leave an uncached current directory on the caller and queue later siblings. This still allows speculation before the child's visitor can prune or stop; cancellation handles unused requests. Keep the inherited admission, frontier and command-wide read/storage limits.
• Represent a speculative read's move-only callback with shared, exactly-once claim state (RunWork). Either the caller or a dispatched worker takes callback ownership under an Abseil mutex, releases that mutex, then performs the leaf job. Losing claimants return immediately; result publication remains owned by the callback.
• Before waiting for a required queued result, mark it required and try to claim that exact read. Do not execute an unrelated slow sibling just to help a queue. If a worker already owns the required read, the caller still waits; this is not a guarantee of full utilization.
• Keep the caller-inclusive N participant budget with at most N-1 persistent background threads. Workers remain leaves: no recursive submission or waiting on their own executor, and no expression or output callbacks on directory workers.
• Preserve cancellation, error reporting, oversized-result handling and reservation ownership. Worker-owned reads are drained before borrowed walker/filesystem state dies. Caller-owned reads complete synchronously before traversal can return, even if a worker already completed their no-op queued wrapper.
• Update the --jobs registry self-documentation, generated reference, parallel design and audit status in the same change. Add the direct annotation dependency for the guarded in-memory trace fixture.

Controls and validation

New traversal controls force actual caller/worker overlap at participant budgets 2/3/10 and check serial-equivalent ordered visits, one read per child, caller identity and the total reader/concurrency cap. A second control occupies the sole background worker and requires the caller to visit the queued directory's file before releasing that worker; it then stops, verifies released accounting and reuses the executor.

Four executor controls cover a caller claiming queued move-only work, an already-running worker claim, simultaneous caller/worker claims, and a queued wrapper returning while caller-owned work is still executing. Move-only producer ownership and exactly-once execution are retained.

The command comparison rendezvous now requires all N participants including the caller, rather than only N-1 background readers. Both inventory phases must use the same reader identities, and the existing whole-command cap and comparison/content caller controls remain.

Existing fixtures are adjusted to hold or prune the first speculative sibling (d1) rather than the caller's uncached current directory (d0). The nested progress control still requires both nested reads before consumption; its total count adds the caller's parent read. No concurrency or cancellation requirement is relaxed.

• bazel test --config=clang --config=xff_docs //...: all 241 targets pass, including the documentation drift test. The initial run executes 238 targets with three valid cached results; the final-source recheck executes 87 with 154 valid cached results.
• Eight affected engine suites (command_workers_test, walk_test, walk_parallel_test, parallel_compare_test, parallel_match_test, rg_test, run_executor_test, run_test) pass ASan/UBSan and MSan. The initial ASan run executes all eight; its final recheck executes the newly expanded executor suite and reuses the other seven valid results. MSan executes all eight against the final source.
• Each of the same eight suites passes ten repetitions under --config=clang --config=tsan --runs_per_test=10, including the 60 ordering/budget combinations, cancellation, nested progress and caller/worker ownership controls.
• The canonical ./compile_commands-update.sh indexes all 297 expected first-party sources with zero missing. git diff --check and Markdown table checks pass. Normal commit and push hooks pass enforcing clang-tidy, formatting and repository policies; the commit also passes Trunk checks. The committed/pushed worktree is clean. No hooks are bypassed. All 30 hosted checks at this head are complete and successful or intentionally skipped, including Linux/macOS tests, sanitizer/lint jobs, adaptive benchmark shards, agreement controls and the combined benchmark comparison in workflow 38028058460. These are hosted correctness/build/gate results, not native M5 Pro performance acceptance.

Portability and remaining acceptance work

This uses portable C++/Abseil and the existing thread-safe VFS. There are no Linux-only runtime calls, OS-specific branches or macOS build changes. Linux correctness and hosted macOS checks do not establish native M5 Pro performance acceptance.

Directory admission remains the inherited 64-sibling count heuristic; eager-stat and content admission/grain are also unchanged. Count is not remaining cost, and completed serial work must not justify starting workers. Starting a cold pool and repeatedly dispatching small jobs must be paid for by useful remaining work; warm-pool reuse has a different cost. The extra claim state, allocation and mutex operations require native allocation/timing controls. Workers can perform many jobs over their command lifetime, but neither reuse nor this participation fix proves the jobs are large enough.

Keep this PR draft pending repeated/reordered full prepared-grid A/B controls, metadata/stat-heavy and slow-storage cases, native allocation/RSS checks, and equivalent M5 Pro sampling, P/E accounting and A/B measurements. Required/ancestor listing lifetime and nonpreemptive long foreground jobs remain separate findings. Builds, sanitizers and lint must not overlap accepted timing collection. No timing, memory improvement or completed audit is claimed.

Current qualification disposition (2026-11-07)

This proposal remains open as unqualified research; its branch and implementation are preserved. The successful checks above belong to its historical head/base context. They are not validation against current main, and this note changes no source, branch, base, CI policy or acceptance threshold.

PR #1125 has merged as 9bf1aad510d843d1d6338fbdf23107a30d339730. Its shared-stage-worker report rejects six separately implemented reuse designs, including a repeatable 15.55% Linux small-identical/1,000-pair/four-worker regression. That result does not establish that every unique mechanism in #992/#996/#997/#998 is rejected or superseded. Main retains specialized pools.

The unique contribution remains exactly-once ownership of a required queued read, plus caller-owned uncached reads. Main's current pool waits for its required task and does not expose this caller-claim mechanism. Preserve the claim/lifetime/rendezvous controls and #997 cancellation/byte ownership; a queued no-op wrapper completing must not imply that caller-owned work or borrowed filesystem state is finished.

The executor/copy audit retains useful caller-participation and RSS behavior, but the cumulative snapshot still costs +11.7% macOS broad listing and +36.3% long-name collection. Removing obsolete matcher admission does not erase all remaining counterexamples. This PR's historical green checks are only against #997, which still depends on the unqualified #996/#992 infrastructure.

A successor must isolate the claim mechanism on main's current frontier/slot ownership, preserve independent stage/root/mounted-walk allowances, and qualify exactly-once reads, early stop/prune, worker-owned versus caller-owned completion, ordered errors/output, claim-state allocation/mutex costs and paired native time/RSS on both platforms. Existing useful ownership tests and narrower latency improvements do not justify merging the regressing combined stack or closing this unique mechanism as superseded.