[helly25]

Keep traversal, content matching and file comparison on one reusable executor, with the calling thread included in the command's concurrency budget. This removes duplicated comparison pools and gives the caller useful batch work instead of leaving it idle beside additional workers.

AG;DR

Scope and dependency

This is a draft follow-up stacked on PR #992 (2c56ec76c9317dd38e0fe9389f3324721941dd67). It isolates command ownership and participant budgeting from the remaining-work admission experiment in that worktree. No LLVM/module/linker/build-default, benchmark driver, dataset selection or resource-accounting settings change here.

Implementation commits: 52153061b4b88f7e552dae4c73afbf68279dd3f2 (command ownership) and 2a0e3343d6be1e2e3a28327f1da9068774cf7a09 (deterministic metadata/filter overlap control).

The frozen shared-executor implementation constructed an executor inside each RunFindCore. Tree comparison launched a second coordinator using unconditional std::async, including under --jobs=1, so both sides could own separate worker pools. ParallelCompare then created another pool after collecting both inventories. Sharing traversal phases did not establish a command-wide budget.

Implementation

• RunFind now owns one lazily started executor until command completion. A resolved --jobs=N allowance includes the calling coordinator, leaving at most N-1 background threads. Standalone walks also reserve one participant for their caller; walk dispatch respects its own allowance when using an externally supplied executor.
• Comparison collects left and right inventories on the same caller, reusing that executor for both walks and later file-pair comparisons. The asynchronous side coordinator and separate comparison threads/mutex/generation protocol are removed. Both sides still run before error aggregation, and relative-path status, patch, summary and error handling retain their existing semantics.
• The coordinator claims independent eager-stat chunks, content-entry chunks and comparison pairs alongside background jobs. Atomic indices assign distinct result slots. Each phase waits for its submitted leaf jobs before replacing its input/results, and walker destruction still drains unused read-ahead before borrowed filesystem state disappears.
• Matcher participant subtraction handles zero/one remaining participant without unsigned underflow or starting idle workers. A deliberately expensive 16-entry prefix that completes the entire batch must leave the executor unstarted.
• Existing directory admission (64 siblings), stat admission (512 entries / 128-entry chunks), content batch bounds and cold matcher heuristic are inherited for experimental isolation. The accumulated-past-work matcher policy is not endorsed by this PR; remaining-work admission is still required.
• The --jobs registry details, generated XFF.md, API/design documentation and resource inspection reflect the caller-inclusive command budget. --explain reports command participants, background workers and the separate semicolon-form child-process allowance. External child processes remain independently limited to N; this is not one shared read/child-operation budget.
• Enforcing lint corrections make the walker explicitly noncopyable/nonmovable while jobs capture it, use const observers/shared handles, and correct test reserve/lambda/aggregate style. These do not change scheduling policy.

Correctness and portability controls

The new in-memory command fixture has 64 sibling directories and large regular files, without host filesystem dependencies. It forces a rendezvous to observe actual overlap rather than assuming fast calls use every worker. It checks calling-thread-only comparison under --jobs=1, a total participant cap at 2/3/10, reader reuse across both trees, caller participation in content/comparison work, and byte-identical ordered output against serial execution. Separate eager-stat controls cover 1/2/3/10 participants and require caller participation while preserving visit order. Existing controls cover pruning, stopping, archive ownership, partial chunks, metadata errors, prepared matcher state and repeated batches.

The full suite also exposed a pre-existing help fixture issue: local /usr/bin/printf is uutils coreutils 0.10.0 and treats its --help argument as its own option even following a format string. The retained unchanged baseline executable (f97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106) reproduces that output. The fixture now passes the literal argument through a POSIX child-shell printf; its parser-boundary assertions are preserved. ShellCheck's narrowly documented SC2016 suppression explains expansion in that child, not in the test shell. No production parser or execution implementation changes for this fixture.

Final-source validation:

• bazel test --config=clang --config=xff_docs //... --test_output=errors: all 241 targets pass at the corrected head, including generated documentation/notice drift checks (eight targets executed, the rest reused valid cache entries). An earlier complete all-extras run also passed.
• The eight affected engine suites (command_workers_test, walk_test, walk_parallel_test, parallel_compare_test, parallel_match_test, rg_test, run_executor_test, run_test) each pass under --config=clang --config=tsan, --config=clang --config=asan (ASan/UBSan), and --config=clang --config=msan at the initial implementation commit. The fixture-only follow-up reruns the complete rg_test suite successfully under all three sanitizer configurations. Other engine sources/tests are unchanged by that commit. These checks cover correctness, not native timing.
• The first enforcing lint pass identified the style findings above. After corrections, both normal commit and push hooks pass, including enforcing clang-tidy across 40 affected translation units, formatting, shell checks and repository policies. Neither hook was bypassed. The worktree is clean and git diff --check passes.

Hosted coverage fixture correction

Coverage at the initial implementation head failed only the required-reader-overlap assertion in MetadataConsumersKeepParallelFilteringAndOrderedResults; its output and exact-read-count assertions passed. With 300 entries, the fixture's first-read rendezvous can expire during the cold serial probe. Once dispatch starts, the caller is allowed to finish the small remaining batch before a worker reaches the filesystem, so the old fixture did not force the overlap it asserted. This does not establish a loss of metadata/filter eligibility.

The control now uses 600 metadata-demanding entries, warming the shared executor through eager stat work before content matching. Its first-read rendezvous forces actual content overlap; the >1 reader requirement, one content read per file, serial-output equivalence and all consumer/native/rg cases remain. Added scoped traces identify the consumer and mode. Existing cold-admission and completed-prefix controls stay separate. No runtime heuristic, coverage threshold or CI retry rule is changed.

Twenty instrumented repetitions pass with bazel coverage --config=clang-coverage --config=xff_docs //xff/engine:rg_test --test_filter=RgEngineTest.MetadataConsumersKeepParallelFilteringAndOrderedResults --runs_per_test=20 --combined_report=lcov '--instrumentation_filter=^//xff/engine:' --test_output=errors; the retained LCOV contains executed lines in both run.cc and parallel_match.cc. The initial broader target-pattern-like filter produced a no-coverage warning, so it is not counted as instrumented validation. The corrected head also passes the full all-extras suite, complete content suite and three sanitizer rechecks. Normal commit/push hooks pass for this follow-up. Hosted coverage at the corrected head now passes. All non-skipped hosted checks at 2a0e3343d6be1e2e3a28327f1da9068774cf7a09, including Linux/macOS tests, sanitizer jobs and benchmark builds, are complete and passing. These correctness/build checks do not establish native performance acceptance; the earlier failed job remains part of the review history.

Performance acceptance and remaining work

No native timing or memory improvement is claimed for this candidate. PR #995 retains the unchanged exact-source Linux baseline, allocation/counter controls and the isolated Abseil TSC calibration mechanism; this change does not disable that optional clock or attribute its one-time 1 ms / 2 ms sleeps to each pthread creation.

One reusable thread budget does not yet establish a fair or memory-bounded scheduler. FIFO directory drains can monopolize workers until an entire batch finishes; completed read-ahead listings lack a command-wide byte budget, unused reads are drained rather than canceled, and useful-work admission still needs remaining-cost controls. These are explicit follow-ups, not satisfied by green correctness tests.

The separate draft PR #997 now implements bounded/cancellable directory frontiers, separate unfinished-read and retained-storage accounting, and alternating foreground/read-ahead dispatch at job boundaries. Its local full-suite and sanitizer controls pass, but it claims neither a total RSS cap nor native performance acceptance. Remaining-cost admission, measured task sizing and useful caller participation during directory-result waits remain open; those concerns are not implicitly solved by either draft.

Separate draft PR #998, stacked on #997, addresses caller participation with uncached reads on the caller and exactly-once claims of required queued reads. Its local full-suite and sanitizer controls pass after a before-change control reproduces the caller-idle gap. Neither that correctness result nor worker reuse establishes sufficient remaining work or native performance acceptance; already-running required reads still require waiting. The implementation remains separate from this PR's head.

Sequential comparison inventory traversal can reduce overlap on narrow-frontier, high-latency storage. Repeated independent/reordered full-grid A/B measurements must include content, metadata and slow-storage controls. Native M5 Pro sampling, performance/efficiency-core accounting and equivalent A/B checks remain required. Portable C++ and Linux sanitizer success do not prove macOS performance safety. Keep this PR draft until those acceptance requirements are verified.

Current qualification disposition (2026-11-07)

This proposal remains open as unqualified research; its branch and implementation are preserved. The successful checks above belong to its historical head/base context. They are not validation against current main, and this note changes no source, branch, base, CI policy or acceptance threshold.

PR #1125 has merged as 9bf1aad510d843d1d6338fbdf23107a30d339730. Its shared-stage-worker report rejects six separately implemented reuse designs, including a repeatable 15.55% Linux small-identical/1,000-pair/four-worker regression. That result does not establish that every unique mechanism in #992/#996/#997/#998 is rejected or superseded. Main retains specialized pools.

The unique contribution remains command ownership and reusable comparison/traversal/content infrastructure, with explicit caller participation. It depends on #992's executor API and currently conflicts with its historical base. Its historical caller-inclusive aggregate ceiling changes work placement relative to main's independent stage allowances; do not treat that budget choice as a proven optimization or silently carry it into a successor.

The executor/copy audit retains Linux gains together with macOS broad 65,536-file listing +11.8% and long-name collection +39.6%. Those synthetic qualification snapshots exclude #991 and are not equivalent to this stale GitHub head. A successor must isolate ownership, concurrency policy and scheduling cost; preserve independent root/mount allowances; and qualify the collection, comparison, metadata/content and slow-frontier counterexamples on current main. The useful #997/#998 mechanisms do not make this combined runtime acceptable by inheritance.