Skip to content

Reuse validated rg output transactions in matcher results - #1127

Merged
helly25 merged 3 commits into
mainfrom
perf/o22-transactional-record-publication
Nov 7, 2026
Merged

helly25 merged 3 commits into
mainfrom
perf/o22-transactional-record-publication

Conversation

@helly25

@helly25 helly25 commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Reuse each file's validated rg output as the matcher result instead of replaying and copying it. This reduces allocation traffic and improves large-output workloads while preserving native callback behavior, late-error handling and binary suppression. Ordinary whole-command search controls remain effectively unchanged.

AG;DR

Implementation and boundaries

  • RenderRgOutput returns the owned per-file output transaction and selection state. ParallelMatch transfers its existing reusable result allocation into rendering and takes the validated transaction back, avoiding the outer replay/copy and unused rg record-end storage.
  • Specialize only the ordinary rg publication path. Native grep retains its original local buffer, callback boundaries, frozen sink and bool/status return. Reduced and multiline rg keep validated capture. Restore heading and sink state on both success and failure; discard all output after late read failure or a binary marker.
  • Preserve the existing retained-capacity limit, worker counts, batching, CLI behavior, source ownership and serial archive barriers. This is not output streaming, pipeline overlap, a global byte bound or an oversized-output solution; O18 remains open.
  • Add committed regression cases for native callback boundaries/context separators, sink replacement, late read and binary failures, heading/state restoration, nonmatching scratch and reused-result capacity. The ASan fixture measures allocation counts only, never performance.

Evidence and limits

  • The report and lossless evidence are docs/benchmarks/transactional-rg-records-20261107.md and .json.gz (5,079,513 bytes; SHA-256 f32ec19cd3af284576a8f6af7e84fbedbf10dee48b044bfdc05909e0ffc7c466). The archive preserves all three variants, counterexamples, copied-image controls and complete source/raw/log records; no experimental runtime variant other than the selected native-in-place form enters this PR.
  • The selected 64-entry allocation diagnostic removes three allocations per entry with unchanged retained result capacity. Memory large-output cohort times improve 12.36% on Mac and 4.32% on Linux. Ordinary native rg-line cohort changes are -0.23%/+0.00%, with same-binary repeats -0.31%/-0.34%. Individual large native-output cases improve up to 11.35%/50.56%; these are not general CLI claims.
  • Retain costs: Linux native-line memory cohort +3.18%, Linux PCRE2 count/100/4 +8.53%, Mac PCRE2 native-lines/10,000/4 +6.63%, and all slower native controls. Do not attribute every difference to allocation traffic, capacity or unrelated-system noise.
  • The experiment measures lean release images based on c95140674254c371844f1fab27a64951a80dac55, not sanitizer timing or current release distributions. Lean images grow +2,816 bytes on Mac and +11,240 on Linux. Mac timings are unpinned; Linux uses pinned physical-core-separated logical CPUs. No native RSS improvement is claimed.
  • All retained variants comprise 101,736 raw measured observations, excluding warmups: three variants, both hosts, complete 128-case memory and 124-case native populations with three balanced sessions and nine repetitions retaining seven. This is not a selected-only population or total CLI-invocation count.

Documentation and publication

  • Update CHANGELOG, performance analysis and O22c2h tracking; keep O18's byte admission, ordered completion, head credit, cancellation/draining and oversized transaction requirements explicit.
  • Register the Markdown report and supported release-site files mapping for the exact gzip download. Add only its precise large-file exception, preserving all existing report/history entries. This adds about 5.08 MB per rendered release snapshot; no historical performance data is removed.

Validation and dependencies

The isolated development candidate passed all 323 documented core/registered-extension release targets on both hosts, including required real Linux FUSE, plus four focused canonical ASan targets per host. Source/evidence identities and all record hashes are preserved independently of public PR readiness. No local clang-tidy is run, as explicitly requested by the user.

Current integration is based on merged #1126/main d8cf61cccd4b0efed5058bf6fa45228e47162b48; five C++ source/test files and the frozen report/evidence were preserved byte for byte while adding publication metadata. Current-base integration passed the three optimized semantic targets (rg_test, parallel_match_test, evaluate_test); these cached results were accepted only after exact input verification. The exact twelve-file changed-file pre-commit gate passed with SKIP=clang-tidy; all 32 release-site unit tests, supported files/page configuration, precise allowlist positive/negative controls and diff checks passed without altering the staged bytes. O22c2h tracking is marked integrated; its own PR CI remains required. Normal signed commit c4d21aea2f6640386d49c4a2f66503472bd64f96 and normal push succeeded; its GPG signature verifies as good. At the initial publication this private clone had no automatic commit/push hooks configured, so the exact manual changed-file gate above is the initial local hook validation claimed. The later scoped Trunk formatter initialized its configured Git hooks; the subsequent signed Markdown correction and push ran those hooks successfully. No hook was disabled or bypassed. This PR must pass its own exact pushed-head GCC, coverage, sanitizers, CI clang-tidy, final benchmark aggregate and required checks; archived experiment results do not replace them. The four unqualified historical worker PRs #992/#996/#997/#998 remain untouched and are not dependencies.

CI Trunk identified one extra blank line between performance-analysis sections on the initial head. Follow-up signed commit 92469874d258200a1e07fcbabcc9c8b150602294 removes that single blank line; scoped trunk fmt recheck and changed-file pre-commit passed. All C++ source, tests, report and compressed evidence bytes remain unchanged. Healthy reachable collectors were not cancelled; the updated head requires its own CI.

CI clang-tidy then identified three narrow source issues. The follow-up keeps the selected native buffer and record-boundary vector local, makes the unused non-publishing specialization's boundary vector const, accumulates already-computed boolean matches with |=, and removes redundant typed-lambda parentheses. Fresh Mac development checks passed rg_test, parallel_match_test, and expression_allocations_test (three executed tests); scoped formatting, changed-file hooks with explicit local clang-tidy skip, and normal signed commit/push hooks passed. The immutable experiment archive still describes the original measured source; no byte-identical performance claim is made for this style correction. Fresh pushed-head CI, including actual CI clang-tidy, remains authoritative. No healthy collector was cancelled.

@helly25
helly25 merged commit 1786340 into main Nov 7, 2026
32 checks passed
@helly25
helly25 deleted the perf/o22-transactional-record-publication branch November 7, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant