Reuse one bounded executor for batched traversal and content work

XFF.md

@@ -564,7 +564,7 @@ See also: [Configuration](#topic-config), [Archives](#topic-archive), [Output](#
 <a id="flag-jobs"></a>
 
 - `-j N, -j=N, --jobs=N|all` - directory-read, eligible content-match, comparison and -exec workers _(global, find, xff, rg)_
-  `N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories on up to `N` worker threads; large directory listings can split required eager metadata reads across the same pool. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches also match in bounded parallel batches, with output kept in traversal order. Small initial batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
+  `N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories using one lazily started pool of up to `N` worker threads, reused for directory reads, required eager metadata and eligible content matching. Workers drain batches of independent work; small workloads stay on the coordinator. Lazy and metadata-free listings do not start stat workers. Independent content predicates and rg searches match in bounded parallel batches, with output kept in traversal order. Small initial batches, archive members, stateful expressions stay on the coordinator. Metadata-consuming output does not itself prevent eligible content filters from running in workers; reductions and rendering remain serial. Independent regular-file comparison pairs can also use bounded workers, preserving relative-path output order. Batches of only small or metadata-only comparisons stay inline; archive and owned filesystem sources stay serial. Xff may independently keep up to `N` semicolon-form `-exec` / `-execdir` children outstanding. Reads and children can overlap; `N` is not one shared operation budget. The children's truth value is therefore success on launch. The `... +` batch forms still run once after the walk and propagate a failing exit status. With no flag, xff uses one fewer than the detected cores, capped at 15 and floored at 1; find and rg modes use every detected core. `all` always means every detected core.
   See also: [Output](#topic-output), [Statistics](#topic-stats)
 
 <a id="flag-sort"></a>

docs/benchmark-correctness-audit.md

@@ -112,8 +112,9 @@ retrofit the binary identity of a historical measurement.
 
 Regular CI retains report artifacts for 30 days and executable build artifacts for three days;
 hosted backfill artifacts omit executables. Preserve available original binaries, build manifests
-and logs before cleanup or artifact expiry. Linux executable verification remains outstanding;
-the 72 checked Mac binaries do not verify the Linux measurements.
+and logs before cleanup or artifact expiry. The initial Mac audit left Linux executable
+verification outstanding; the native Linux follow-up below now covers the available local
+Linux artifacts. The 72 checked Mac binaries do not verify the Linux measurements.
 
 On Linux, verify each retained executable against its original report with the PR's
 `tools/benchmark_provenance.py --report=... --binary=... --retention-reference=...` command.
@@ -244,3 +245,76 @@ metadata and slow-storage controls on both platforms. Run affected correctness/c
 and appropriate sanitizers. Report absolute XFF time, reference time and same-run ratio together.
 The current evidence supports investigating specific costs; it does not establish that XFF
 will beat every reference tool on every workload.
+
+## Native Linux follow-up (2026-10-09)
+
+The native session uses merged audit driver commit
+`4ea895f63106f05df7c90b629d20433ae1e03016`. The original engine baseline remains
+`772994ab7905be77915978f06ea90ae5934aaeef`; both source revisions produce the same
+retained executable SHA-256,
+`f97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106`.
+Baseline builds use `clang_release`, with the historical ICU 70 build-library
+path supplied explicitly. Complete prepared-grid runs retain nine raw samples
+per cell, use the fastest seven estimator and store fixtures on verified tmpfs.
+
+### Retained Linux executables
+
+All 142 completed reports under the local Linux benchmark batch inventory match
+their original retained executables and their original build manifests. They
+cover 72 distinct commits and 38 distinct executable hashes; no mismatches were
+found. Verification uses `benchmark_provenance.verify_retention`, checks the
+standalone manifest against the report's embedded build record, and records a
+measurement-bound proof for each report. Historical observations are unchanged.
+
+The original reports and binaries remain under `/home/marcus/xff-benchmarks/batches`.
+The native proof inventory is retained at
+`/tmp/xff-linux-retention-verification/summary.json`, with per-report
+`binary-verification.json` files below that directory. These checks verify the
+available local Linux artifacts; they do not verify expired hosted artifacts.
+
+### Worker scheduling experiment
+
+Linux `kernel.perf_event_paranoid` is now `0`, and native `perf` hardware counters
+are available. Startup-boundary profiles identify worker creation and teardown,
+Abseil mutex slow paths and scheduling overhead. Keeping directory reads serial
+removes the small-work cliff but loses useful concurrency on large broad trees.
+[PR #991](https://github.com/mboworks/xff/pull/991) admits directory reads only
+when at least 64 independent sibling directories are available. Its hosted Linux
+and macOS tests, sanitizers, all three benchmark shards on each platform and
+combined benchmark comparison pass.
+
+The next experiment shares one run-owned executor across directory read-ahead,
+eager stat work and eligible content matching. Each admitted batch submits at
+most one drain job per useful worker. Directory jobs repeatedly claim independent
+listing reads; stat jobs claim 128-entry chunks; matcher jobs claim entry chunks.
+Only the coordinator recurses, evaluates traversal controls and consumes ordered
+results. Threads remain bounded and are reused until command completion.
+
+Correctness checks include nested read-ahead above the admission threshold,
+partial stat chunks with missing entries, repeated matcher transitions and
+executor reuse. An early-stop test deliberately holds a prefetched sibling read
+open while the coordinator quits: `Walk` must wait before destroying the walker,
+even though its executor survives for another phase. Explicit Abseil completion
+signals replaced the experiment's standard-library futures after expanded TSan
+coverage reported a shared-state completion/deallocation race.
+
+The interrupted `/tmp/xff-perf-shared-executor-v2` run is diagnostic only:
+correctness builds overlapped its measurements, and its measured source predates
+the final scheduling changes. It supplies no acceptance timing evidence.
+
+### Memory-accounting finding and remaining acceptance work
+
+Direct `/usr/bin/time -v` runs show approximately 5.3-5.4 MiB peak RSS for a small
+XFF invocation, while the Python comparison worker reports approximately
+22-23 MiB for the same command. A trivial native command also reproduces the
+Python worker's high floor. The forked launcher's inherited high-water mark must
+be corrected or explicitly excluded before using those reported peaks to choose
+arena sizes. This is separate from the worker scheduler experiment.
+
+The complete audit remains open. Final shared-executor timing must be collected
+after every build has finished, using the complete prepared grid. Repeated
+controls must cover content, eager metadata and slow storage. Native allocation
+profiles and the separate 20k/50k/100k single-worker slope investigation still
+need completion. Hosted macOS checks provide portability evidence for PR #991;
+they do not replace the specified native M5 Pro A/B and sampling controls for
+the shared-executor experiment.

docs/design-parallel.md

@@ -132,13 +132,34 @@ actions still occur in traversal order.
 
 ## Parallelism control
 
-A single knob, `-j N` (long form `--jobs`), configures independent limits: up to
-`N` directory-read workers, up to `N` eligible content-matcher workers, and at most
-`N` outstanding semicolon-form `-exec`/`-execdir` children. These are separate
-limits, not one shared `N`-operation budget. Content matching does not run alongside
-execution actions in the same expression. `-j 1` makes each part synchronous.
-`-j all` (`--jobs=all`) uses every detected core (`hardware_concurrency()`) for
-each limit, regardless of the active mode's default.
+A single knob, `-j N` (long form `--jobs`), caps one run-owned executor at `N`
+workers for directory reads, eager stat batches and eligible content matching.
+The coordinator retains traversal and output ownership. The executor starts
+lazily, grows only as useful work becomes available, and reuses its threads
+across phases and mounted-filesystem walks until the command finishes.
+Semicolon-form `-exec`/`-execdir` separately allows at most `N` outstanding child
+processes; content matching does not run alongside execution actions in the same
+expression. `-j 1` makes each part synchronous. `-j all` (`--jobs=all`) uses every
+detected core (`hardware_concurrency()`) for these limits, regardless of the
+active mode's default.
+
+The coordinator admits directory read-ahead when at least 64 independent sibling
+reads are available. It submits at most one drain job per worker; each job claims
+successive directories from that batch and publishes indexed listing results.
+Workers perform leaf reads and never recurse or wait on jobs in their own
+executor. Eager stat work similarly uses drain jobs over 128-entry chunks after
+at least 512 entries are available. This bounds executor queue operations by
+worker count while preserving coordinator visitation order.
+
+Content matching retains private evaluator and regex state for each logical
+worker. Decision-only matching collects up to 8,192 entries before flushing;
+rendered-content batches remain bounded at 256 entries because their output can
+be large. A cold executor keeps cheap small batches on the coordinator until
+8,192 entries have accumulated. A timed 16-entry prefix admits slower work
+earlier when that prefix takes at least 500 microseconds. Once traversal or
+matching has started the executor, batches of at least 16 entries may reuse it.
+Each submitted matcher job drains entry chunks, and results are consumed in
+original entry order.
 
 Under `-j > 1` the serial `-exec ... ;` / `-execdir ... ;` form launches its child
 on a bounded runner (at most `N` outstanding) instead of running it synchronously.
@@ -178,7 +199,8 @@ design-config.md):
   entry from the pruned subtree is visited or acted upon.
 - **`-quit`** - the coordinator stops visiting entries immediately. Workers only
   perform leaf directory reads and do not inspect the walk's stop flag; already
-  submitted reads finish while the pool is destroyed. Their results are ignored,
+  submitted drain jobs finish before their walker is destroyed, even when the
+  executor will be reused by a later phase. Their results are ignored,
   and no new expression actions are run. Exit status follows the normal model.
 - **`-depth` (post-order)** - children before parent. The ordering layer holds a
   directory's own visit until its subtree has been emitted; under `none` this

xff/cli/globals.cc

@@ -903,10 +903,10 @@ constexpr std::array kGlobals = std::to_array<GlobalFlag>({
         .details = "`N` is a positive integer; use `-j 4`, `-j=4`, or `--jobs=4`. The conventional attached "
                    "short form `-j4` is also accepted. Every form accepts `all` in place of `N`. `-j 1` makes "
                    "directory reads and `-exec ... ;` synchronous. At larger values, xff reads directories "
-                   "on up to `N` worker threads; large directory listings can split required eager metadata reads "
-                   "across the same pool. Lazy and metadata-free listings do not start stat workers. "
-                   "Independent content predicates and rg searches also match in bounded "
-                   "parallel batches, "
+                   "using one lazily started pool of up to `N` worker threads, reused for directory reads, "
+                   "required eager metadata and eligible content matching. Workers drain batches of independent "
+                   "work; small workloads stay on the coordinator. Lazy and metadata-free listings do not start "
+                   "stat workers. Independent content predicates and rg searches match in bounded parallel batches, "
                    "with output kept in traversal order. Small initial batches, archive members, stateful expressions "
                    "stay on the coordinator. Metadata-consuming output does not itself prevent eligible content "
                    "filters from running in workers; reductions and rendering remain serial. Independent "

xff/engine/BUILD.bazel

@@ -19,6 +19,30 @@ load("//xff:cc.bzl", "cc_binary", "cc_library", "cc_test")
 
 package(default_visibility = ["//visibility:private"])
 
+cc_library(
+    name = "run_executor_cc",
+    srcs = ["run_executor.cc"],
+    hdrs = ["run_executor.h"],
+    visibility = ["//xff:__subpackages__"],
+    deps = [
+        "@abseil-cpp//absl/base:core_headers",
+        "@abseil-cpp//absl/functional:any_invocable",
+        "@abseil-cpp//absl/synchronization",
+    ],
+)
+
+cc_test(
+    name = "run_executor_test",
+    size = "small",
+    srcs = ["run_executor_test.cc"],
+    deps = [
+        ":run_executor_cc",
+        "@abseil-cpp//absl/synchronization",
+        "@googletest//:gtest",
+        "@googletest//:gtest_main",
+    ],
+)
+
 # Directory traversal over a vfs::FileSystem: a worker pool reads (readdir+lstat)
 # in parallel while a single coordinator thread visits in --sort order.
 cc_library(
@@ -27,6 +51,7 @@ cc_library(
     hdrs = ["walk.h"],
     visibility = ["//xff:__subpackages__"],
     deps = [
+        ":run_executor_cc",
         "//xff/vfs:vfs_cc",
         "@abseil-cpp//absl/algorithm:container",
         "@abseil-cpp//absl/base:core_headers",
@@ -140,12 +165,15 @@ cc_test(
     size = "small",
     srcs = ["walk_test.cc"],
     deps = [
+        ":run_executor_cc",
         ":walk_cc",
         "//xff/vfs:local_fs_cc",
         "//xff/vfs:vfs_cc",
         "@abseil-cpp//absl/functional:function_ref",
         "@abseil-cpp//absl/status",
         "@abseil-cpp//absl/status:statusor",
+        "@abseil-cpp//absl/strings",
+        "@abseil-cpp//absl/synchronization",
         "@googletest//:gtest",
         "@googletest//:gtest_main",
         "@mboworks_mbo//mbo/testing:status_cc",
@@ -248,6 +276,7 @@ cc_library(
         ":collect_cc",
         ":evaluate_cc",
         ":expression_execution_cc",
+        ":run_executor_cc",
         "//xff/parser:parser_cc",
         "//xff/registry:registry_cc",
         "@abseil-cpp//absl/base:core_headers",