[helly25]

Retain the native Linux benchmark audit's startup and large-grid allocation findings with raw evidence. The small-work timing cliff is largely associated with one-time Abseil TSC calibration; deep traversal also retains substantially more listing/path memory. This documents the isolated controls and their limitations without changing runtime or build defaults.

AG;DR

Consolidation status (2026-10-10)

The user requested batching independently passing measurement changes to avoid serial CI cycles. This PR's exact-head checks are all green, and its original artifacts remain byte-identical. Its complete evidence has been folded into #994 at bd23f81e3b48ae6518408f6a4ba0a3d82495a671, rebased onto merged #993. #994 also corrects guest CPU/storage limitations and retains the original host observations. Only the combined #994 head is pushed and tested. Keep this original PR open until that shared gate succeeds and #994 actually merges, then close it as superseded; do not separately rebase or merge it. Original branches and artifact bytes are retained.

Fresh exact-source full-grid baseline

• Freshly build the exact initially requested source 772994ab7905be77915978f06ea90ae5934aaeef using its historical clang_release configuration. Clean source/HEAD/tree and tracked build-configuration hashes pass; the saved executable is f97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106, independently matching the original retained baseline bytes.
• Complete all thirteen prepared anchors, both v2 layouts, workers 1/3/10, required ten-physical-core allowance and tmpfs, nine samples/fastest seven plus one discarded warmup. Batch f586e579b2d57d0f9afa24e6d06849d6bc1df4d96654e5bcb349e77d8c6a9d1d is terminal-success, with 858 cells, 14,040 retained samples and 1,560 warmups: 15,600 correct fresh participant invocations. Compilation finishes before measurement; no other local builds/profiles overlap collection.
• Freeze accounting driver 0e296e40ca6bf523215d80d1dc02deb3503b796c, including Correct native benchmark accounting and retain audit evidence #994 fixes beyond original Verify benchmark build provenance and show binary evidence in the viewer #990, and independently retain native helper 60bcb395.... The contract records all driver hashes and method boundaries. Native wall times include the helper boundary and must not be pooled with old Python-launch measurements.
• Independently verify the actual report/binary/build manifests, historical source configuration, full task grid, fixture/anchor/oracle identities, affinity, all samples/output lengths, native resource metrics and fastest-seven estimates. The exact-source baseline reproduces the cliff: broad enumeration/jobs3 1.842 ms at 10 files to 5.241 ms at 20 (rg 3.103/3.230); deep 2.386 ms at 1k to 6.151 ms at 2k (rg 3.223/3.923). Retain absolute values and same-run ratios together, including single-worker and ten-worker controls.
• Retain a machine-readable index, complete derived analysis and provenance/reproduction scripts. The original 11 MB report's exact bytes are split into eleven lossless gzip byte chunks; thirteen archives in total stay under the existing file-size policy. Individual/reassembled hashes verify against original report and analysis bytes. No raw sample is discarded and no historical report is rewritten. Executable bytes remain locally, with their hashes and source/configuration/build logs retained in the archives.
• Preserve limitations: collection checks actual stdout against its oracle, but retained samples only record output lengths; post-run model/oracle regeneration checks internal consistency rather than an independent workload implementation. Generated MODULE.bazel.lock cannot be verified against a tracked source blob. One ordered Linux baseline is not repeated/reordered engine A/B or native Mac acceptance.
• Include a same-driver/exact-source M5 Pro baseline handoff script in the provenance archive, syntax-checked only on Linux. It checks the intended hardware, retains the helper before historical builds and records P/E-core keys/storage without Linux-affinity or RAM-disk claims. Native Mac accounting, baseline execution, Instruments/allocation and repeated/reordered controls remain outstanding.
• Commit f7534e5a91aef9b07077d29fdf719cfe4bf95ed6 is pushed and passes normal commit/push checks for all fifteen follow-up files. All four evidence families reverify after formatting; the exact report and analysis bytes remain unchanged. A disposable site-builder link check passes at /tmp/xff-exact-baseline-review-site.OeQY5v and renders all fourteen new artifact links to this immutable source commit. The first generated-input preflight failed; verified explicit inputs passed on retry. This local check uses a standalone reference placeholder and validates audit-document/artifact links, not CLI reference generation or Pages publication. Fresh hosted run 38017443685 passes release-site and Trunk; other checks remain pending.
• After the collector and verifier reached terminal success, bazel test //tools:release_site_test --config=clang passes all 32 cases for the current worktree. No local test/build overlaps the retained timing campaign. The separate visited-listing lifetime experiment is being prepared from merged Verify benchmark build provenance and show binary evidence in the viewer #990; no runtime change is included or accepted by this PR.

Large-grid CPU follow-up

• Retain 24 diagnostic cells on fresh, verified full-grid v2 broad/deep fixtures: 20k/50k/100k files, one physical CPU and one worker, covering enumeration, tree sorting, collection and content-needle matching. The unchanged retained baseline binary is independently verified. Each cell has an oracle-checked warmup, nine direct counter samples and fifty fresh invocations under perf record; all 1,440 invocations pass, including exact hierarchical tree-sort order. No lost samples or build/acceptance-timing overlap is reported.
• Enumeration's user instructions/file remain approximately 2.5k-2.6k across both layouts, despite deep traversal's much higher live heap. At 100k, collection increases user instructions from 251.3 to 376.1 million on broad and from 260.8 to 390.0 million on deep. These are actual whole-CLI workloads, not isolated causal phase costs, accepted wall timings or scheduler A/B results.
• Correct the profile denominator: the recording includes Python's repetition loop, so even the XFF-filtered initial reports print globally normalized percentages. Recompute shares from all XFF self periods, preserving unknown locations. Coarse exclusive labels use function names rather than argument-list type names.
• Independently parse all raw CPU-mode records. Their exact period sums match the filtered histograms in all 24 cases. Kernel-context sample IPs exist despite the event's exclude_kernel=1; retain these as unattributed, not kernel CPU time. Ordinary sample-location skid is a possible explanation, not an isolated host diagnosis. The cycles:up precise capability probe fails with No supported events found; no security/kernel setting is weakened. See perf_event_open for counter-exclusion/sample-mode semantics.
• Retain the index plus nine lossless gzip JSON archives of raw counters, self reports, callgraphs, sample-context records, event attributes, manifests and reproduction scripts. Deep 100k is partitioned by workload under the existing 500 KB file-size policy. Original perf.data and complete oracle arrays remain locally with hashes and reconstruction commands; these large originals are explicitly not included in the review archives. The index identifies the superseded local analysis and its corrected replacement without altering original observations.
• An independent export verifier confirms all original profile-byte hashes, raw texts/scripts, counter medians, sample period/context sums and all 24 unique workload keys against local artifacts.
• Document the command-wide scheduler gap: Reuse one bounded executor for batched traversal and content work #992 owns an executor per traversal; comparison runs two coordinators/two separately capped pools, then creates another comparison pool. Directory drains can monopolize workers; pending listings have no global retained-byte budget and queued unused reads drain after quit. The current single-coordinator executor cannot simply be shared concurrently. Required global ownership, coordinator accounting, bounded frontier, fair batches and native cross-platform acceptance remain unresolved.
• This follow-up changes documentation and evidence only. No engine code, defaults, LLVM/linker selection, macOS configuration or host security setting changes. Full precise phase attribution, metadata/slow-storage controls, repeated engine A/B and native M5 Pro controls remain open. The exact initial-source Linux baseline is now complete as documented above; it does not supply the Mac half or repeated acceptance.
• Commit 6c0dad53cd is pushed after normal commit/push checks pass for all eleven follow-up files. CPU, allocation and clock evidence all independently reverify; index formatting does not change raw values. The disposable site-builder link check passes at /tmp/xff-large-cpu-review-site.FQv8dN and renders the new evidence links to that immutable source commit. This local check uses a standalone reference placeholder, not generated CLI help. Its hosted release-site and Trunk checks passed; current-head validation is recorded above.
• The separate fresh exact-initial-source baseline at /tmp/xff-perf-baseline-exact772-native-session1/collection is terminal-success and independently verified, as documented above. Its handoff also records the rejected nonempty-output preflight before any compilation/measurement; the actual collection uses its own fresh directory. Original observations are unchanged. The export preserves the original embedded retention proof rather than inventing a separate proof-file requirement.

Scope and finding

• Documentation and retained measurement artifacts only, based on merged PR Verify benchmark build provenance and show binary evidence in the viewer #990 (4ea895f63106f05df7c90b629d20433ae1e03016). No engine, toolchain, linker, build flag, macOS configuration or host security setting changes.
• Record the historical Linux executable verification and ongoing shared-executor experiments, distinguishing diagnostic observations, failed controls and unmeasured follow-ups from acceptance.
• Retain 36 native startup profile cells: nine fresh perf stat observations, 200 fresh invocations under perf record, a warmup and syscall trace per cell. All 7,596 invocations pass the oracle. Eight additional symbolized sleep controls identify ReadPool::Run -> Abseil mutex wait-queue Enqueue -> CycleClock::Frequency -> MeasureTscFrequencyWithSleep.
• This Linux/x86 host lacks the sysfs TSC-frequency value. The pinned Abseil fallback requests 1 ms and 2 ms sleeps while calibrating its frequency. Treat this as one-time mutex-clock initialization, not a measured per-thread startup cost.

Isolated timing and mechanism controls

• Build the unchanged baseline source with only ABSL_USE_UNSCALED_CYCLECLOCK=0 for target and host compilation. Preserve release build options, source/configuration manifests and executable hashes. Four affected correctness suites pass. This is an experimental build, not a new default or historical release.
• Three independent timing sessions use ascending, descending and seeded-shuffle cell order. Each builds fresh full-grid broad/v2 and deep/v2 trees: 10/20/50/100/200/500/1k/2k/5k/10k/20k/50k/100k files, workers 1/3/10, verified ten-physical-core allocation and tmpfs. Enumeration covers every anchor; content needle/absent covers 5k/20k/50k/100k. Baseline, clock-only and rg participant order rotates between rounds.
• Retain nine samples and one discarded oracle-checked warmup per participant/cell, fastest seven. All 11,340 invocations pass; 378 cells retain 10,206 samples. Builds and instrumented profiling do not overlap these timings.
• Median same-session timing change: broad 20 files/jobs3 -60.3%, deep 2k/jobs3 -51.0%. Tiny broad 10/jobs1 regresses +3.2%, with +1.94% to +4.14% in all three sessions. Retain this counterevidence; do not claim no regressions.
• Twelve additional native mechanism controls cover baseline/clock-only at both boundaries and workers 1/3/10. All 132 invocations pass. Both parallel variants create exactly two workers, but only baseline requests the 1 ms and 2 ms calibration sleeps. Instrumented durations are not acceptance timings.
• Per-session means and ratios are summarized separately; samples are not pooled across sessions. Common native accounting includes helper startup in wall time. Absolute times are not comparable to the older Python-launch boundary.

Retention and validation

• Link a machine-readable index plus four gzip JSON archives from the audit page. Preserve all raw timing reports; startup counter/syscall records; calibration stacks; analyses; build/measurement manifests; exact reproduction scripts; fixture, oracle, binary and accounting-helper identities.
• Compressed and decompressed SHA-256 values are recorded and verified. Artifacts fit the normal added-file-size policy without changing hooks. Original executables and large perf.data files remain at recorded local retention paths; archives retain their hashes rather than their bytes. The old accounting helper is retained independently of the subsequently rebuilt helper.
• Offline archive verification checks original reports/scripts/native diagnostic records and retained helper bytes, reruns the raw timing analyzer, and reproduces the entire stored analysis except its verification timestamp.
• All 32 release-site tests pass; focused normal pre-commit checks and git diff --check pass. Commit 5f82c254b04cdfbd833c79543be7591bde88ae2e is pushed and passes normal commit/push hooks and Trunk checks for all six changed files. The index remains semantically identical after formatting; offline verification passes again. The disposable site-builder link check also completes successfully at /tmp/xff-clock-review-site.FHnU8Y, using a standalone reference placeholder and without publishing Pages. Current-head hosted validation is recorded above.

Portability and acceptance limits

Apple already disables this optional Abseil clock in the pinned source; that fact is not native macOS performance validation. No Linux-only result accepts #991's directory threshold, #992's shared-executor scheduling, or a new default clock configuration. The exact initial-revision Linux baseline and the single-worker workload allocation controls are complete as recorded here; native M5 Pro sampling/A-B, multiworker eager metadata, slow storage, candidate-scheduler allocation and precise phase-level CPU/slope attribution remain open. Separate draft PRs #996/#997/#998 address command ownership, bounded/fair directory lookahead and caller participation, with correctness/hosted controls, not accepted native performance. Remaining-cost admission and task sizing still require real workload and cross-platform controls. Keep engine experiments draft until the audit's actual acceptance gates pass.

Large single-worker allocation follow-up

• Extract Heaptrack 1.5.0 plus four dependencies into a private /tmp bundle; verify all package hashes without changing system packages or kernel permissions. A known C allocation control confirms all three explicit malloc/realloc calls, their symbolized stacks and the exact 6 MiB controlled live peak. Allocation-event accounting recovers peaks missed by short-process periodic snapshots.
• Collect one original Heaptrack profile and one separate uninstrumented native-launcher RSS control for each of six enumeration cells: both full-grid v2 layouts, 20k/50k/100k files, one worker on a verified physical core. All twelve commands pass the original output oracle. Full maximum-tree hashes and the unchanged retained baseline/helper bytes match before and after. This is diagnostic allocation evidence, not timing acceptance.
• At 100k, broad enumeration makes 223,942 allocation calls, peaks at 0.311 MiB tracked heap and records 5.531 MiB native RSS. Deep enumeration makes 211,212 allocation calls, peaks at 34.006 MiB tracked heap and records 40.191 MiB native RSS. RSS and requested live heap are distinct metrics; Heaptrack cannot cover non-interposed allocation mechanisms.
• At the deep 100k global peak, stack weights containing ReadDir::JoinPath total about 24.3 MiB, while those containing StatEntries and reserve total about 9.2 MiB. Source inspection shows recursive traversal retains parent listings and full path strings. This directs investigation toward listing/path lifetime and path-length scaling, not an arena-reserve guess. It does not establish the CPU-time contribution or validate a lifetime change.
• Retain a separate allocation index, six compressed per-cell JSON archives containing original zstd Heaptrack bytes, complete allocation/global-peak stacks, analyzer output, commands, manifests, native RSS records and exact reproduction scripts. Preserve deep 100k's 37 MB raw stdout as three lossless gzip byte chunks with offsets and reassembly checksums. No raw output is discarded or file-size policy weakened.
• Recompute every allocation/free event total. All collapsed allocation-stack weights sum exactly to call totals; all global-peak stack weights sum exactly to tracked heap peaks. Verify compressed/decompressed hashes, original native artifacts, embedded profile bytes, scripts, RSS records and lossless output reassembly. All six cells verify successfully after export and index formatting.
• Commit 8638a7eb0ce914ae4140034e5bef8f7101913cb4 is pushed and passes normal commit/push hooks and Trunk checks for all eleven follow-up files. Both evidence families verify again after formatting. The disposable site-builder link check passes at /tmp/xff-large-allocation-review-site.mX88Vj, using a standalone reference placeholder; actual new artifact links resolve to that immutable source commit. Current-head hosted validation is recorded above. No engine or default build change is included.

Exact-baseline workload allocation follow-up

• Commit 2b1d5a95f37775e3e3902611bca2391bc93c254c adds thirty allocation profiles and thirty separately uninstrumented native-resource observations, all using the exact initially requested source 772994ab7905be77915978f06ea90ae5934aaeef and retained executable f97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106. Both full-grid v2 layouts × 20k/50k/100k × tree sorting/collection/lazy stat/eager stat/content needle, pinned to one verified physical CPU with --jobs=1. Collection is terminal exit 0; all builds precede profiling. No runtime/build/security setting changes.
• Lazy stat uses -type f -size +0 -print0; eager stat uses -size +0 -type f -print0, making the metadata consumer unconditional under ExpressionMetadata. Independent fixture-byte-length/content oracles cover every output, including hierarchical tree order. Full materialized-tree hashes, binary/helper and driver hashes match before/after. The independent verifier regenerates all thirty oracles, checks all sixty raw outputs/native-resource relationships and recomputes every allocation/free event. These profiles and single native observations are diagnostic, not repeated timing or engine/reference A/B.
• At 100k, broad collection peaks at 26,862,227 tracked bytes and 32,243,712 native RSS bytes; deep collection peaks at 72,085,451 and 78,327,808 bytes respectively. Tree, lazy/eager stat and content retain approximately 0.33 MB tracked heap on broad and 35.65 MB on deep. Heap/RSS are different metrics and non-interposed allocation mechanisms remain excluded.
• Every collapsed allocation and global-peak stack sum matches its independently recomputed event total. Exclusive group precedence is EvalCollect, ReadDir::JoinPath, StatEntries plus reserve, stat path-copy helpers, then all other weight. At deep 100k collection's global peak, collection-associated stacks retain 36,456,976 bytes alongside 25,475,264 directory-path and 9,618,816 listing-reserve bytes. Broad collection-associated stacks retain 26,701,744 bytes. These are named stack groups, not precise CPU/phase attribution or proof that all collection storage is an arena.
• Metadata/content add calls without comparable peak growth on these fixtures; symbolized metadata stacks show transient StatFields string copies. Collection also changes eager metadata demand. Neither these observations nor warm tmpfs establish storage latency, worker admission, task sizing or native M5 Pro safety. Original artifacts remain at /tmp/xff-native-workload-allocation-session1.
• Publish docs/benchmarks/native-workload-allocation-controls.json and thirty-five gzip archives preserving all 360 original per-cell files: original zstd profiles, analyzer outputs, full allocation/global-peak stacks, native resources and both stdout streams. Identical bytes use exact reconstruction references; oversized JSON payloads split into ordered lossless raw-byte chunks under the unchanged 500 KB policy. The export verifier reconstructs all 360 files byte-exact against originals after normal formatting. Frozen driver/collection/verifier/analyzer/export source and full logs are archived. Executable/helper/profiler executable bytes remain local, with their identities and package provenance retained explicitly.
• Normal commit/push hooks and Trunk pass for all 37 follow-up files. The release-site unit target passes (cached, unchanged test/code), and the actual configured site renders and validates at /tmp/xff-workload-allocation-review-site.ggjbWt, binding the new index link to committed source 2b1d5a95f37775e3e3902611bca2391bc93c254c. As in CI, that disposable build supplies a standalone CLI-reference placeholder; it does not validate generated help. The public PR head is verified and the worktree is clean. Current-head hosted validation is pending; historical-head green checks above do not prove this new head. The complete audit remains open for remaining-cost admission, useful task sizing, real multiworker metadata/slow-storage controls, precise phase attribution, scheduler candidate allocation profiles, repeated/reordered full-grid A/B and native M5 Pro sampling/RSS/P-E controls.

[helly25]

Superseded by #994, merged to main as 88d4fac. Artifact preservation was verified against the merged tree: prior #995 benchmark artifacts remain byte-identical; the only additions under docs/benchmarks are native-linux-host-context.json, native-resource-accounting-controls.json, and native-resource-accounting-portability-controls.json.