Skip to content

Reuse requested content values across worker stages - #1053

Merged
helly25 merged 8 commits into
mainfrom
perf/carry-derived-content
Nov 2, 2026
Merged

helly25 merged 8 commits into
mainfrom
perf/carry-derived-content

Conversation

@helly25

@helly25 helly25 commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Reuse requested hashes and line counts from content already read by matching workers. Large parallel field workloads become substantially faster while raw file bytes stay confined to each evaluator.

AG;DR

  • Compile a deduplicated request from bound field renderers and histogram enums once per run. Preserve hash defaults/qualifiers and transformed fields; no per-file flag-name scanning or new CLI option.
  • Derive only for matched/selected entries whose worker snapshot already has bytes. Move only requested digest map nodes and line values into an optional result-indexed cache vector. Keep ParallelResult unchanged and allocate no side vector for ordinary filters.
  • Keep source bytes worker-local and preserve ordered coordinator consumption. Streaming sources without loaded snapshots retain lazy reads. Result limits avoid computing hashes for unconsumed matches. Unmatched/empty/regrowing batches, cached read errors, binary line counts, effect invalidation and archive serial admission keep their contracts.
  • Measure clean ThinLTO release executables on local macOS ARM64 and Linux x86-64 drogon: 50 native cases x three sessions and 26 whole-engine in-memory cells x three sessions per host. Use nine rounds/fastest-seven means, rotating participant order, independent native output oracles and Linux CPU affinity. No compilation/lint/sanitizer overlaps timing on a host.
  • Focused confirmation adds a byte-identical baseline participant: four native cases x five sessions and five kernels x three sessions per host. Larger four-worker content-field kernels confirm about 67-70% lower elapsed time on Linux and 72-73% on macOS. Heavy ten-worker native hash cases improve about 73-74%; one-worker cases stay near baseline. Gains combine avoided reads/copies and moving requested computation onto existing workers, not new threads.
  • Retain the initial macOS large-output control (+6.63%) and its confirmation (-0.52%), slower samples, source/binary identities and raw evidence. Initial native control medians are within about +1.55% Linux / +0.78% macOS; do not imply every workload is faster. This adds bounded batch cache values, not a total-process memory cap.
  • Update the existing O13 checklist, performance analysis, parallel design, changelog and release-site report mapping. Keep O14/O15 catalog/invariant preparation, O17/O18 scheduling and O22 output construction separate.

Validation: complete core/extension release suites pass 306/306 on both hosts, with required Linux FUSE mounts. Focused ASan passes five suites per host; Linux TSan passes three. All 42 affected translation units pass local scoped clang-tidy after fixing test const qualifiers. Snapshot and end-to-end tests verify one VFS read per file, requested-only/no-byte transfers, failures, defaults/transforms, limits, pool transitions, rg selections, and serial/parallel column/summary/histogram equality. Additional consumer tests receive focused release/ASan validation on both hosts and Linux TSan. Normal enforcing commit/push hooks and current pushed-head CI remain required before merge.

Based on merged #1052 (documentation/evidence only). Synchronization preserves exactly the previously validated and measured source tree. Current main-context pushed-head CI remains authoritative.

Base automatically changed from perf/direct-match-evaluator to main November 2, 2026 12:17
@helly25
helly25 merged commit dc4e702 into main Nov 2, 2026
32 checks passed
@helly25
helly25 deleted the perf/carry-derived-content branch November 2, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant