[helly25]

Make executor startup and warmed dispatch costs observable separately, with fresh-process cold controls and independently checkable work. This diagnostic helps investigate worker admission and task sizing without changing runtime thresholds or claiming performance acceptance.

AG;DR

Scope and dependency

Draft diagnostic follow-up stacked on PR #998 (416e45a39e1432aeed3f2aed64f8e3795bac19f1). Implementation: c8929d16c77e318b15d87c5bfcbf07af0fa7058e; tree: 58bc0b9b3a4626c63ae859557551c93308e9e0a4.

Evidence and documentation follow-up: 1dbad41bcf1bfba5c3a579691cca9783dc94c717. The measured source remains the implementation commit above; adding archives and a release-page mapping does not relabel those binaries as the later head.

No shipping runtime, module, LLVM, linker, default, benchmark driver, dataset selection, accounting or platform-security setting changes. The private measurement library is consumed only by its correctness test and explicit manual diagnostic binary. Register that binary in the existing clang-tidy manual-target index so it cannot silently evade lint/header preparation.

Measurement contract

• Explicitly start a caller-inclusive N participant executor (N-1 background workers). Time start calls, a first trivial dispatch, warmed synthetic sweeps and teardown separately; reuse the same executor for every warmed sweep. This deliberately bypasses production admission policy to measure its costs.
• leaf queues a bounded batch beside a reserved caller chunk. claimable adds the caller/worker exactly-once callback and separate result/wrapper state. drain submits one chunk-draining leaf per background worker per sweep, with the caller also claiming atomic chunk indices. Workers never submit or wait recursively in the pool.
• Check output via item counts, queued-job counts, participant observations and a modulo-64-bit checksum. Runtime unsigned xor-shifts supply adjustable synthetic work; zero work units is the cheap-body control. Partial chunks and no-work drains retain correct totals.
• Observed identities count actual contributors, not forced rendezvous participants. A drain's caller can process zero chunks if workers claim them first; record that rather than inventing full participation.
• Require a fresh process for each cold sample. In-process repetition may already have paid Abseil's one-time clock calibration. Phase times include probe bookkeeping and omit some vector preparation/final destruction; the phase sum is not native process wall time.
• Keep all host output within annotated stream adapters and argv within its annotated ABI adapter. Add no VFS/host mutation path or shipping CLI flag.

Validation and evidence status

• The diagnostic test covers all three modes, partial grains 1/16/128, participant budgets 1/2/3/10, independently known sums, invalid modes/dimensions and representability rejection.
• Final-source executor_measurement_test and run_executor_test each pass ten repetitions under ASan/UBSan, MSan and TSan. Earlier normal and sanitizer runs also pass. These are correctness checks, not timing evidence.
• Canonical indexing includes all 300 expected first-party sources and the manual binary. The first indexing attempt had a bad diagnostic dependency/manual registration and exposed suppressed source-query failures; that incomplete 110-source result is not counted. The independent fail-closed indexing fix is PR #999, based on main, not mixed into this change.
• Normal commit and push hooks pass enforcing clang-tidy, formatting and repository policies; the commit also passes Trunk. Initial findings are resolved with unsigned shift operands, explicit arithmetic precedence, trailing commas and named literal ranges. No check is disabled, the test matrix is not reduced and the pushed worktree is clean.
• Optimized default and optional clock-control (ABSL_USE_UNSCALED_CYCLECLOCK=0) binaries are retained separately before collection, with system/home RCs disabled, announced workspace flags and clean source identity. LLD requires the already-retained ICU 70 build-library path; that environment and the rejected missing-library attempt are retained. This is a build dependency, not a toolchain/default change.
• Native repeated/reordered collection completed with terminal exit 0 and independent verification: 4,860 fresh invocations, 486 cells, 486 warmups and 4,374 raw samples. All modes × participants 1/3/10 × grains 1/16/128 × mix iterations 0/32/512 run in declared, reversed and seeded-shuffled orders. Every cell/version has one fresh warmup and nine fresh samples. Default/control order alternates by cell/repetition. CPUs 0..9 are ten distinct allowed physical (package,core) pairs; each invocation is restricted to its first N cores. All builds, sanitizers and lint finished before collection; frozen source/binary/helper/module identities are checked before and after.
• Retain nine lossless compressed archives and an indexed SHA-256/byte-count inventory in docs/benchmarks/native-executor-dispatch-controls.json: all raw stdout/resources/commands, three original analysis arrays, independently verified analysis, same-session ratio summary and complete provenance/reproduction scripts. Original executable/helper bytes remain at indexed local paths and were rehashed on export; public review archives do not include them. The independent offline verifier passes without /tmp binaries, GitHub or remeasurement, recalculating every oracle/order/resource relationship and all seven-of-nine estimates. Scripts and source/config/build logs are retained, not screenshots alone.
• Default cold start plus first dispatch has median 3,242.70 us (N=3) and 3,268.54 us (N=10) across retained per-cell medians; separate clock-control medians are 86.28 us and 298.70 us. This corroborates the separately profiled one-time Linux/x86 Abseil calibration; it is not 3 ms per pthread creation. Clock control remains diagnostic-only.
• Reusing threads does not amortize fine-grained dispatch: at mix 32/grain 16/N=3, separately dispatched leaf batches take 34.3747 ms versus their 1.5625 ms serial control; drains take 0.6466 ms versus their 1.5595 ms serial control. Same-session ratios over the three orders are 21.79..22.08 and 0.410..0.422 respectively. Zero-work drains still lose to serial, while heavier/grain-128 work benefits from useful parallelism. These are warmed synthetic phase times, not native wall/full-engine measurements; cold startup can outweigh the warmed saving.
• Fix the hosted release-site failure: the audit's new local Markdown link had no page mapping. Add the probe document to release-site.json; the release-site unit target and configured rendered-link validation pass locally. Use the same placeholder generated CLI-reference page as CI; this does not validate the real generated CLI reference. Initial failed job: https://github.com/mboworks/xff/actions/runs/38031016573/job/114151792817. No site policy is weakened.

Portability and limitations

The probe is portable C++/Abseil; there are no Linux-only runtime APIs or macOS configuration changes. The Linux collection harness uses verified physical-core affinity, which is not a claim of equivalent M5 Pro P/E scheduling. Hosted validation of the updated evidence/site-map head is pending. The harness retained topology, allowed CPUs and start/end load averages, but not CPU model/kernel version or per-sample thermal/load observations; do not claim a host-independent dispatch constant.

This is not a VFS/metadata/content/frontier benchmark. It excludes path and retained-storage accounting, expression/output work, actual storage latency and allocator/RSS attribution. Its batch join shape differs from a sliding nested frontier. Synthetic work units are not calibrated file costs. It cannot alone choose production thresholds, prove enough remaining work, establish full utilization or prove improvements against reference tools.

Keep this PR draft until its diagnostic evidence is reviewed. The full engine audit remains open: actual remaining-cost admission/grain tuning, metadata/stat-heavy and slow-storage controls, native allocation/phase profiling, repeated/reordered complete-grid A/B and native M5 Pro sampling/RSS/P-E controls remain necessary. Builds, sanitizer checks and lint must finish before native timing collection; no engine acceptance or completed audit is claimed.

[helly25]

Closing as superseded by merged #1035, not as a failed performance optimization. This PR is diagnostic infrastructure; its purpose was to distinguish fresh executor startup, first dispatch, warmed dispatch, teardown, and synthetic work-grain costs.

#1035 extracted the executor/probe implementation, its correctness tests and manual diagnostic registration into private testonly targets on main, independently of the rejected runtime stack. The diagnostic evidence and provenance remain in docs/benchmarks/executor-dispatch-probe.md, native-executor-dispatch-controls.json, and the retained compressed native-executor-dispatch archives. The later audit retains source/driver and measurement interpretation.

In particular, the old approximately 3.2 ms fresh-start observations and clock-control observations are historical controls for Abseil's one-time calibration, not a current per-thread creation cost or a portable admission constant. Warm synthetic leaf/drain results remain useful for understanding dispatch grain; they do not validate VFS traversal, output, storage, RSS or the production scheduling integrations.

There is no need to merge this ancestry-dependent diagnostic PR after its independently isolated implementation was merged. Shipping CLI isolation was checked for #1035 (byte-identical Linux CLI; matching non-debug Mach-O section payloads on macOS), and the diagnostics remain disconnected from the shipping CLI. Continue using that merged harness for further investigations; runtime acceptance for #992/#996/#997/#998 remains separate.

Evidence and interpretation: complete cross-platform audit, raw samples, exact source patches and identities. Integration measurements use explicitly recorded synthetic ports onto clock-corrected main 974d137026818bba1c1deaf6cfc4fffe676b314e; they are not relabeled as tests of this unchanged GitHub head or the later #1039 binary. Linux used drogon's Ryzen 9950X KVM guest, tmpfs and requested CPU affinity; macOS used the M5 Pro, warm APFS and worker allowances without affinity. Native comparisons used discarded warmups, nine interleaved samples retaining seven, and three differently ordered sessions with independent output oracles. In-memory populations have separate contracts. Compare variants within a host, not absolute Linux/macOS times.

This closure does not delete the branch, rewrite any commit, merge experimental runtime code, or claim that all worker/queue research is finished.