Repository navigation
Correct native benchmark accounting and retain audit evidence - #994
Merged
Merged
Conversation
helly25
marked this pull request as ready for review
October 10, 2026 10:34
Rebase the measurement infrastructure onto merged PR #993 and fold the already-validated evidence from PR #995 into one merge-gating batch. Preserve accounting implementation and original artifact bytes. Correct completed hosted-validation notes and guest CPU/storage limitations; retain the original host-context observations. No scheduler, toolchain, platform or host-security changes.
helly25
force-pushed
the
fix/native-benchmark-resource-accounting
branch
from
October 10, 2026 10:47
0e296e4 to
bd23f81
Compare
helly25
enabled auto-merge (squash)
October 10, 2026 10:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Correct benchmark memory accounting and retain the native audit's measurements, raw evidence and limitations. This lands independent measurement infrastructure and evidence without accepting scheduler changes or changing XFF runtime defaults.
AG;DR
Combined merge batch
dac3747d63cad713def0b949a934dfe36f82da63, then fold Retain native startup, allocation and large-grid CPU audit evidence #995's already-validated documentation and measurement artifacts into the same PR. Current combined head isbd23f81e3b48ae6518408f6a4ba0a3d82495a671.0e296e40ca6bf523215d80d1dc02deb3503b796cand Retain native startup, allocation and large-grid CPU audit evidence #9952b1d5a95f37775e3e3902611bca2391bc93c254ceach completed 30 checks (26 successful, four expected adaptive-shard skips), plus successful external Snyk status. The accounting component includes actual macOS helper execution and all three successful macOS measurement shards./tmpand/dev/shmare tmpfs. The later host observation does not prove historical hypervisor state. Record these limitations without rewriting frozen contract fields.git diff --checkpass, normal commit and push hooks pass, and pinned Trunk formatting reports no issues on the two revised files. Trunk's normal merge-commit hook reports its built-in skip; no hook is bypassed. The actual disposable release-site preflight succeeds and binds the new host-context artifact link to this exact commit; like CI it uses a standalone reference placeholder, not generated CLI help or Pages publication.38046140395is queued for its single merge-gating CI cycle. Existing component checks are not labelled as checks of the combined head. Keep Retain native startup, allocation and large-grid CPU audit evidence #995 open until Correct native benchmark accounting and retain audit evidence #994 actually merges, then close it as superseded; do not independently update Retain native startup, allocation and large-grid CPU audit evidence #995 or the engine stack.38045263673,38045263735) to reserve capacity. Preserve Reuse one bounded executor for batched traversal and content work #992's repair verification, other non-superseded runs, existing artifacts and the frozen accounting-driver checkout used by retained measurements.Native accounting (original #994)
Finding and independent controls
The audit's roughly 25 MB floor was a measurement defect on Linux, not evidence that every small command allocated that much. Five independent 20-file controls per tool compare the original fresh Python worker, a native launcher and
/usr/bin/timeusing the same commands and output oracle. Small XFF runs report about 5.2 MiB with both native methods versus about 24 MiB with Python;truereports about 1.5 MiB with both native methods versus about 24 MiB with Python. Find and rg also lose the inherited Python floor.Retain all raw peaks, commands, executable identities and the correctness oracle in
docs/benchmarks/native-resource-accounting-controls.json. Its JSON values match/tmp/xff-resource-accounting-final-controls.json; normal repository hooks format whitespace. That initial optimized helper SHA-256 ise6766cea6431e91b5a5c1d16746079d59ed0e70bb13bce220e9796d23a09fb89, after its initial lint fixes. The original controls remain unchanged. After the Darwin macro follow-up, the optimized helper is60bcb395bb3d0eccccd0a3d0cb2407538b1f96c45d5e1e156a0bd4522f93328e; five additional native/Python/time controls per tool reproduce the finding and pass the same oracle. Retain those separately indocs/benchmarks/native-resource-accounting-portability-controls.json, matching/tmp/xff-resource-accounting-post-portability-controls.json. The earlier helper executable is independently retained with the clock experiment. These small controls establish an accounting defect; they are not engine-performance acceptance or the complete v2 workload.Implementation and boundaries
wait4; report individual peaks, exit codes and aggregate CPU time as JSON on a separate descriptor from command output.bazel-binexecutable during measurement. Direct-script subprocesses use the canonical runfile lookup without caller-relative repository mapping.Provenance and historical compatibility
--resource-launcherplumbing and document native/direct-script invocation, method partitioning, summed pipeline peaks and unchanged historical records. No engine scheduler, LLVM dependency, linker, host package or security setting is changed.Validation
bazel test //tools:python_tests --config=clangpass on native Linux. The ten directly affected suites also pass after the runfiles and renderer-warning regressions.4ea895f63106f05df7c90b629d20433ae1e03016. Its executable SHA-256 isf97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106, independently verified against retention. Both layouts measure successfully; an identical rerun reuses the verified build and completed report. Its frozen environment and report agree on the final helper identity and both accounting methods. Retain/tmp/xff-native-accounting-backfill-final-pilot, batchdbd9ca40a57b277e4d55a21d894927d66caf4bf53be138082086eaf1b02dcf4a. The earlier prototype pilot is retained separately.cannot resume incompatible batchbefore build or measurement. Historical pilot output remains unchanged.tools/clang_tidy.sh tools/process_resources_native.ccpass succeeds after fixing all reported findings. Narrow POSIX ABI exceptions have reasons; the lint policy is unchanged. All 57 tooling suites pass again after the helper changes; the repository-wide tooling check andgit diff --checkpass.57292fe4f0passes normal commit hooks, push hooks and Trunk checks for all 24 changed files. Enforcing clang-tidy also passes during both commit and push. The raw control JSON values remain identical after Trunk's formatting.Portability and remaining investigation
Ready for the independently scoped measurement-infrastructure merge requested by the user. Ordinary Linux/macOS CI passes, including actual hosted benchmark shard collection. Darwin units are implemented; independent native macOS accounting calibration and M5 Pro performance controls remain separate audit requirements, not conclusions supplied by this merge. Historical RSS cannot be corrected retroactively from new runs.
Hosted run
38008492687passed the release-site, pre-commit, Trunk, coverage, fuzz, MSan, TSan, ASan, GCC, minimal, generated-reference and clang-tidy checks. The macOS test job failed compiling::sigemptyset(...): the Darwin SDK defines it as a function-like macro. A local portability follow-up uses an unqualified invocation, with a comment explaining the SDK constraint.All six initial Linux/macOS benchmark shards also failed because their direct-Python invocation had no prebuilt launcher: the original shared archive retained only XFF. A local integration follow-up now builds/stages the helper in PR and main build jobs, includes it beside XFF in
benchmark-build.tar, checks that the extracted helper is executable and passes its explicit path to every shard. The measurement action does not build or run Bazel. Hosted backfill prepares the current helper before freezing its batch and compiling historical revisions. Two regression tests cover both workflow families, the shared action and the backfill ordering; the root test-data filegroup declares the inspected backfill workflow. Documentation explains the artifact and build/measurement boundary.After the unrelated timing collection and native diagnostic profiles completed, the follow-ups passed all 57 tooling suites, the repository-wide tooling check, pinned formatting, enforcing helper clang-tidy and
git diff --check. The optimized helper was then rebuilt and its independent RSS controls repeated successfully. Commit0e296e40ca6bf523215d80d1dc02deb3503b796cis pushed and passes normal commit/push hooks and Trunk checks, including actionlint, Bazel policy and enforcing clang-tidy. Formatting leaves the raw control JSON values unchanged. Fresh hosted run38011945259passes full Linux/macOS tests, both benchmark builds, all three Linux and all three macOS measurement shards, both agreement checks, combined comparison and finaldone. Release-site, pre-commit, Trunk, fuzz/coverage, GCC/minimal/reference, clang-tidy and MSan/TSan/ASan checks also pass. Additional adaptive shards are correctly skipped after initial agreement. The previous run's failures remain recorded above; the repaired run supplies actual hosted collection/portability evidence, not independent native Mac accounting acceptance.The 20k/50k/100k allocation/slope investigation, repeated/reordered engine A/B controls, slow-storage and metadata controls, and native macOS scheduler profiles remain separate audit requirements. The corrected RSS method is a prerequisite for interpreting those allocation measurements, not their completion.
Retained audit evidence (original #995)
Fresh exact-source full-grid baseline
772994ab7905be77915978f06ea90ae5934aaeefusing its historicalclang_releaseconfiguration. Clean source/HEAD/tree and tracked build-configuration hashes pass; the saved executable isf97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106, independently matching the original retained baseline bytes.f586e579b2d57d0f9afa24e6d06849d6bc1df4d96654e5bcb349e77d8c6a9d1dis terminal-success, with 858 cells, 14,040 retained samples and 1,560 warmups: 15,600 correct fresh participant invocations. Compilation finishes before measurement; no other local builds/profiles overlap collection.0e296e40ca6bf523215d80d1dc02deb3503b796c, including Correct native benchmark accounting and retain audit evidence #994 fixes beyond original Verify benchmark build provenance and show binary evidence in the viewer #990, and independently retain native helper60bcb395.... The contract records all driver hashes and method boundaries. Native wall times include the helper boundary and must not be pooled with old Python-launch measurements.MODULE.bazel.lockcannot be verified against a tracked source blob. One ordered Linux baseline is not repeated/reordered engine A/B or native Mac acceptance.f7534e5a91aef9b07077d29fdf719cfe4bf95ed6is pushed and passes normal commit/push checks for all fifteen follow-up files. All four evidence families reverify after formatting; the exact report and analysis bytes remain unchanged. A disposable site-builder link check passes at/tmp/xff-exact-baseline-review-site.OeQY5vand renders all fourteen new artifact links to this immutable source commit. The first generated-input preflight failed; verified explicit inputs passed on retry. This local check uses a standalone reference placeholder and validates audit-document/artifact links, not CLI reference generation or Pages publication. Fresh hosted run38017443685passes release-site and Trunk; other checks remain pending.bazel test //tools:release_site_test --config=clangpasses all 32 cases for the current worktree. No local test/build overlaps the retained timing campaign. The separate visited-listing lifetime experiment is being prepared from merged Verify benchmark build provenance and show binary evidence in the viewer #990; no runtime change is included or accepted by this PR.Large-grid CPU follow-up
perf record; all 1,440 invocations pass, including exact hierarchical tree-sort order. No lost samples or build/acceptance-timing overlap is reported.exclude_kernel=1; retain these as unattributed, not kernel CPU time. Ordinary sample-location skid is a possible explanation, not an isolated host diagnosis. Thecycles:upprecise capability probe fails withNo supported events found; no security/kernel setting is weakened. Seeperf_event_openfor counter-exclusion/sample-mode semantics.perf.dataand complete oracle arrays remain locally with hashes and reconstruction commands; these large originals are explicitly not included in the review archives. The index identifies the superseded local analysis and its corrected replacement without altering original observations.6c0dad53cdis pushed after normal commit/push checks pass for all eleven follow-up files. CPU, allocation and clock evidence all independently reverify; index formatting does not change raw values. The disposable site-builder link check passes at/tmp/xff-large-cpu-review-site.FQv8dNand renders the new evidence links to that immutable source commit. This local check uses a standalone reference placeholder, not generated CLI help. Its hosted release-site and Trunk checks passed; current-head validation is recorded above./tmp/xff-perf-baseline-exact772-native-session1/collectionis terminal-success and independently verified, as documented above. Its handoff also records the rejected nonempty-output preflight before any compilation/measurement; the actual collection uses its own fresh directory. Original observations are unchanged. The export preserves the original embedded retention proof rather than inventing a separate proof-file requirement.Scope and finding
4ea895f63106f05df7c90b629d20433ae1e03016). No engine, toolchain, linker, build flag, macOS configuration or host security setting changes.perf statobservations, 200 fresh invocations underperf record, a warmup and syscall trace per cell. All 7,596 invocations pass the oracle. Eight additional symbolized sleep controls identifyReadPool::Run-> Abseil mutex wait-queueEnqueue->CycleClock::Frequency->MeasureTscFrequencyWithSleep.Isolated timing and mechanism controls
ABSL_USE_UNSCALED_CYCLECLOCK=0for target and host compilation. Preserve release build options, source/configuration manifests and executable hashes. Four affected correctness suites pass. This is an experimental build, not a new default or historical release.Retention and validation
perf.datafiles remain at recorded local retention paths; archives retain their hashes rather than their bytes. The old accounting helper is retained independently of the subsequently rebuilt helper.git diff --checkpass. Commit5f82c254b04cdfbd833c79543be7591bde88ae2eis pushed and passes normal commit/push hooks and Trunk checks for all six changed files. The index remains semantically identical after formatting; offline verification passes again. The disposable site-builder link check also completes successfully at/tmp/xff-clock-review-site.FHnU8Y, using a standalone reference placeholder and without publishing Pages. Current-head hosted validation is recorded above.Portability and acceptance limits
Apple already disables this optional Abseil clock in the pinned source; that fact is not native macOS performance validation. No Linux-only result accepts #991's directory threshold, #992's shared-executor scheduling, or a new default clock configuration. The exact initial-revision Linux baseline and the single-worker workload allocation controls are complete as recorded here; native M5 Pro sampling/A-B, multiworker eager metadata, slow storage, candidate-scheduler allocation and precise phase-level CPU/slope attribution remain open. Separate draft PRs #996/#997/#998 address command ownership, bounded/fair directory lookahead and caller participation, with correctness/hosted controls, not accepted native performance. Remaining-cost admission and task sizing still require real workload and cross-platform controls. Keep engine experiments draft until the audit's actual acceptance gates pass.
Large single-worker allocation follow-up
/tmpbundle; verify all package hashes without changing system packages or kernel permissions. A known C allocation control confirms all three explicit malloc/realloc calls, their symbolized stacks and the exact 6 MiB controlled live peak. Allocation-event accounting recovers peaks missed by short-process periodic snapshots.ReadDir::JoinPathtotal about 24.3 MiB, while those containingStatEntriesandreservetotal about 9.2 MiB. Source inspection shows recursive traversal retains parent listings and full path strings. This directs investigation toward listing/path lifetime and path-length scaling, not an arena-reserve guess. It does not establish the CPU-time contribution or validate a lifetime change.8638a7eb0ce914ae4140034e5bef8f7101913cb4is pushed and passes normal commit/push hooks and Trunk checks for all eleven follow-up files. Both evidence families verify again after formatting. The disposable site-builder link check passes at/tmp/xff-large-allocation-review-site.mX88Vj, using a standalone reference placeholder; actual new artifact links resolve to that immutable source commit. Current-head hosted validation is recorded above. No engine or default build change is included.Exact-baseline workload allocation follow-up
2b1d5a95f37775e3e3902611bca2391bc93c254cadds thirty allocation profiles and thirty separately uninstrumented native-resource observations, all using the exact initially requested source772994ab7905be77915978f06ea90ae5934aaeefand retained executablef97833a8eb45f49b596665b47911c2df9a2558619778512fd31d1b38f2e56106. Both full-grid v2 layouts × 20k/50k/100k × tree sorting/collection/lazy stat/eager stat/content needle, pinned to one allocated guest CPU with--jobs=1. Collection is terminal exit 0; all builds precede profiling. No runtime/build/security setting changes.-type f -size +0 -print0; eager stat uses-size +0 -type f -print0, making the metadata consumer unconditional underExpressionMetadata. Independent fixture-byte-length/content oracles cover every output, including hierarchical tree order. Full materialized-tree hashes, binary/helper and driver hashes match before/after. The independent verifier regenerates all thirty oracles, checks all sixty raw outputs/native-resource relationships and recomputes every allocation/free event. These profiles and single native observations are diagnostic, not repeated timing or engine/reference A/B.EvalCollect,ReadDir::JoinPath,StatEntriesplusreserve, stat path-copy helpers, then all other weight. At deep 100k collection's global peak, collection-associated stacks retain 36,456,976 bytes alongside 25,475,264 directory-path and 9,618,816 listing-reserve bytes. Broad collection-associated stacks retain 26,701,744 bytes. These are named stack groups, not precise CPU/phase attribution or proof that all collection storage is an arena.StatFieldsstring copies. Collection also changes eager metadata demand. Neither these observations nor warm tmpfs establish storage latency, worker admission, task sizing or native M5 Pro safety. Original artifacts remain at/tmp/xff-native-workload-allocation-session1.docs/benchmarks/native-workload-allocation-controls.jsonand thirty-five gzip archives preserving all 360 original per-cell files: original zstd profiles, analyzer outputs, full allocation/global-peak stacks, native resources and both stdout streams. Identical bytes use exact reconstruction references; oversized JSON payloads split into ordered lossless raw-byte chunks under the unchanged 500 KB policy. The export verifier reconstructs all 360 files byte-exact against originals after normal formatting. Frozen driver/collection/verifier/analyzer/export source and full logs are archived. Executable/helper/profiler executable bytes remain local, with their identities and package provenance retained explicitly./tmp/xff-workload-allocation-review-site.ggjbWt, binding the new index link to committed source2b1d5a95f37775e3e3902611bca2391bc93c254c. As in CI, that disposable build supplies a standalone CLI-reference placeholder; it does not validate generated help. The public PR head is verified and the worktree is clean. Source head2b1d5a95f37775e3e3902611bca2391bc93c254csubsequently completed all 30 checks successfully or with the four expected adaptive skips. These component checks are historical evidence; the combined Correct native benchmark accounting and retain audit evidence #994 head still needs its own merge gate. The complete audit remains open for remaining-cost admission, useful task sizing, real multiworker metadata/slow-storage controls, precise phase attribution, scheduler candidate allocation profiles, repeated/reordered full-grid A/B and native M5 Pro sampling/RSS/P-E controls.