[helly25]

Fix native benchmark-launcher staging and stop compile-database indexing from claiming completeness after a failed source query. These checks protect the measurements and tooling used to evaluate scheduler changes.

Also record the stage-specific concurrency, admission-tuning and low-overhead analysis follow-up, without changing scheduling behavior or introducing flags. This consolidates the already-green #999/#1002 preparation and the newly reproduced main-only workflow fix into one validation cycle.

AG;DR

Consolidation and scope

• Rebase original Fix benchmark launcher staging and consolidate audit preparation #999 f9f8c2fa5e3e5c8930a94dfc625764c9522d3021 onto merged main 88d4facd15ebd00acf749222b3028de2e8916970. Its rebased patch ID matches the original.
• Fold original Record stage-specific concurrency and execution scheduling follow-up #1002 cc7b0dec173a68251c216e06b131fbbf18634061 into this branch. Its original TODO/design blobs were verified identical before adding JC07/JC08 below. Record stage-specific concurrency and execution scheduling follow-up #1002 was closed as superseded on 2026-10-10 at 12:44:10 UTC after this PR actually merged and its complete tree/doc preservation was verified; the original branch is retained.
• Both original PR heads had 30 terminal check runs (26 success/four adaptive skips) and successful Snyk status. Those checks are historical evidence, not current-head validation after the rebase/update.
• Carry no engine implementation from Evaluate directory worker admission after the clock fix #991/Reuse one bounded executor for batched traversal and content work #992/Reuse one bounded command-wide executor #996-Let the caller participate in directory reads #998/Plan content workers from remaining batch cost #1001. Bound and cancel directory read-ahead on the shared executor #997 remains unchanged. The separate coordinator-completion repair is local/unpublished and is not part of this PR.
• Combined implementation head: 2e3e789693f38a5f6e7fee4ffe209b6b2f7710ad; tree: 38d5ecc9e1789f0f0a187960c24daee1f7884da0. The clean three-commit batch descends directly from the actual current main, not an obsolete stacked base.
• No C++ source, expression behavior, jobs default, module/toolchain/linker, dataset grid/sampling, security setting, native resource-accounting algorithm or historical artifact changes.

Main-only native-launcher finding and fix

The post-#994 main Benchmarks run 38049489783 failed on both Ubuntu and macOS at Build paired optimized revisions. The five requested targets built successfully; the subsequent native-launcher query performed no build actions and cp could not find the queried top-level output. This was a deterministic staging failure, not a network issue or failed historical build. The main-only paired workflow was not exercised by the otherwise-green PR Test workflow.

The Python driver's dependency can build the native helper under a transitioned configuration instead of the top-level configuration being copied. Locally, building only //tools:benchmark_resources produced bazel-out/k8-opt-ST-f38e6014d46d/bin/tools/process_resources_native, while bazel-out/k8-opt/bin/tools/process_resources_native remained missing. An unqualified query also returned both cached configured outputs. Restricting the universe alone did not eliminate that ambiguity in the same graph; the explicit configuration selector did.

• Add //tools:process_resources_native to the main paired build's existing target list, with the same release/cache options. PR and hosted-backfill builds already request it explicitly.
• In PR, main and hosted-backfill workflows, query config(//tools:process_resources_native, target) so the copied path identifies the top-level configuration explicitly. This is not a guessed output path or selection of the first line from an ambiguous query.
• Strengthen the existing workflow guard to require an explicit release build of the native target before its query/copy. Add a guard covering configuration selection in all three workflows; preserve prebuilt-launcher handoff and keep shard measurement free of compilation.
• The behavior of target patterns matching multiple configured instances and lingering cached graphs is described in Bazel's cquery documentation. The project's pinned Bazel was also exercised directly; the selector returned exactly one path, and the corresponding release executable was built, copied and run successfully.

Fail-closed compile-source collection

The original audit found that compile_commands-update.sh suppressed a failing core query's diagnostics and swallowed the pipeline failure with || true. It continued with only 110 extra-module sources and reported zero missing/OK. The direct invalid core query returned exit 7 with a missing-target error. That partial result is not valid completeness evidence.

• Move expected-source collection into tools/collect_compile_sources.sh, with the updater's existing core/extra/API patterns and first-party label filter.
• Check each query status, preserve original diagnostics and identify the failing pattern. Keep results in owned temporary files until all queries succeed; a later failure emits no partial source list.
• Use POSIX awk so a legitimate empty/header-only query succeeds without swallowing genuine query/read/write errors. Reject failure to load the extra-module pattern list.
• Fail before the post-processor can announce completeness. Keep cleanup limited to the two files created by mktemp; document the contributor workflow.
• Four bashtests cover filtering, a later failed query after earlier success, successful empty queries and missing-argument usage errors. A prior real-graph control also verified that a nonexistent later target returns failure, preserves its diagnostics/pattern and emits zero stdout bytes.

Scheduling insight and actionable TODOs

The folded design/TODO change records work, not an implemented or settled API:

• JC01-JC06 explore a --jobs-* family for directory I/O, matching/comparison and external execution. Leave aggregate ceiling versus fallback, caller-inclusive budgets, defaults, all, repeated overrides and INI semantics open.
• Distinguish serial queued execution from synchronous expression continuation. A one-child queue alone does not preserve exit-status-dependent truth, short-circuiting or chained effects.
• Preserve prune/quit/post-order, captures/reductions, mutations and ordered-publication barriers. Speculative reads must not become unauthorized visits/actions.
• Compare role-aware command-owned execution with coordinated persistent stage pools; reuse threads, avoid nested pool waits and I/O starvation, and bound queues, retained bytes and outstanding children separately.
• Treat the user's roughly three-reader local and 50-core network observations as motivating hypotheses, not measured defaults; explicitly chosen I/O concurrency may exceed CPU count.
• JC07 separates directory-admission thresholds from worker budgets. Evaluate directory worker admission after the clock fix #991's experimental 64-sibling gate is not a universal optimum. Explore explicit serial/minimum-independent-work or cost controls, distinguish cold/warm costs, and qualify small slow frontiers as well as broad cached ones.
• JC08 requires opt-in super-low-overhead scheduling analysis selected once at setup. Disabled hot loops must have no diagnostic flag checks, timers, counters, extra atomics or diagnostic allocations. Enabled analysis uses bounded existing-observation aggregates and justified coarse/sampled timing, not additional filesystem probes, forced workers, reordering or unbounded per-path traces. Separate measured/predicted values and diagnostic output; inspect disabled compiled paths and retain off-baseline/enabled-overhead controls. Super-low overhead is sufficient, but an unverified zero-effect claim is not.
• Require native Linux/macOS and local/slow/network qualification with actual thread/child counts, useful work, correctness/effects/cancellation, first output, throughput and memory. Unsettled spellings require the existing unstable gate and same-change registry/help/tests when implemented.

Validation

• The strengthened explicit-build guard failed against the original main workflow. The final configuration-selector guards also failed against the original workflow queries. These negative results were retained before implementation; the fixed guards pass.
• Real Linux release control: building the Python driver alone reproduced the missing top-level launcher/different transitioned output. Explicitly building both targets, selecting the target configuration, checking the single executable path, copying it and executing /usr/bin/true through the copy succeeded. Native command exit/output/diagnostics were checked, and measured child RSS was nonzero.
• Five Bazel targets pass: benchmark_history_test, process_resources_test, benchmark_backfill_test, benchmark_compare_test, and collect_compile_sources_test. The history suite contains 28 tests including the new cross-workflow guard.
• Live bazel run //tools:repository_tooling_test passes all five repository-wide checks.
• Focused normal pre-commit passes formatting, actionlint, ShellCheck, Markdown and repository policies for all twelve changed paths. Clang-tidy is inapplicable because this batch changes no C++/headers/bzl sources.
• The complete canonical ./compile_commands-update.sh succeeds on this combined branch: 294 expected first-party sources, zero missing. This executes the final fail-closed collection stage, not merely the collector's isolated tests.
• Normal commit hooks, including Trunk, pass without bypasses; the final tracked worktree is clean. The one exact-lease push completed through normal hooks, including a successful broader-scope clang-tidy check of rebased main changes. Exact-head Test run 38050820172 completed successfully: 30 terminal check runs (26 success/four adaptive skips), successful Snyk, and no nonterminal checks. The old green PR runs were not reused as the strict-base merge gate.

Portability and limitations

The workflow fix is shared by Linux/macOS matrices, uses the pinned Bazel configuration selector and existing Bash/copy/build options, and changes no platform-specific launcher code. The source collector uses existing Bash, mktemp, POSIX awk and narrow cleanup, not GNU-only file-processing options. Both controls passed the exact-head ordinary cross-platform CI, including the macOS test and all six benchmark shards. The actual post-merge main-only paired benchmark jobs now pass on both platforms at squash bbc0bf9376c7fc41fe75094998c3c7e74bd02e94: Ubuntu job 114215537420 completed successfully at 13:01:20 UTC, and macOS job 114215537237 at 13:03:43 UTC on 2026-10-10. Their explicit native-helper build/query/copy steps and subsequent paired measurements succeeded. Terminal job/step metadata and literal timestamped command excerpts are retained under /tmp/xff-main-bbc0bf9-launcher-evidence/; cquery's resolved path is captured in a shell variable and is not printed, so no observed path value is claimed. Both post-merge main workflows are now terminal-success on that exact squash: Test 38052921500 completed at 13:20:48 UTC (24 jobs: 15 success, nine configured benchmark skips), and Benchmarks 38052921577 completed at 13:38:15 UTC (15 jobs: 11 success, four optional shard skips). Full paginated inventories confirm no nonterminal jobs; both paired builds, all six measurement shards, both agreement checks and aggregation succeeded. Terminal run summaries and complete job inventories are retained alongside the launcher evidence. This establishes actual Linux/macOS workflow execution, not native M5 Pro performance; a subsequent publication deployment has not been verified.

Workflow-text guards are regression guards, not substitutes for actual build/copy execution. The compile-source universe remains Bazel's declared graph rather than independent compiler-semantic proof. Extraction may already have written a database before the last query fails, but the command returns failure instead of announcing completeness. Legitimately empty queries remain valid.

This PR does not accept a scheduler optimization, implement analysis/flags, repair the 47 persistent full-stack Linux advisories, or claim M5 Pro/slow-storage performance. Frozen audit drivers, retained historical binaries and completed native collections remain unchanged. The batch used one consolidated exact-lease push. No unnecessary child rebases, old-head CI reruns or extra workflow dispatches were triggered.

Actual merge identity

The user-approved CI-gated squash auto-merge completed on 2026-10-10 at 12:42:02 UTC as bbc0bf9376c7fc41fe75094998c3c7e74bd02e94. Its parent is 88d4facd15ebd00acf749222b3028de2e8916970, and its tree is exactly the validated combined tree 38d5ecc9e1789f0f0a187960c24daee1f7884da0. Root fetched only the main tracking ref and verified a zero-difference comparison against head 2e3e789693f38a5f6e7fee4ffe209b6b2f7710ad, including final documentation bytes. No frozen/dirty worktree was checked out or modified during that sync. The earlier preview merge_commit_sha value was not treated as an actual merge.