Skip to content

Aggregate ordinary comparison side totals directly - #1057

Merged
helly25 merged 16 commits into
mainfrom
perf/aggregate-comparison-totals
Nov 2, 2026
Merged

helly25 merged 16 commits into
mainfrom
perf/aggregate-comparison-totals

Conversation

@helly25

@helly25 helly25 commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Ordinary comparison summaries now build left and right totals directly, reducing duplicate per-path summary storage. Category summaries and expressions with actions retain their existing behavior.

AG;DR

Based on main after #1056 merged. Synchronization head ae7ec80094471b0d1d94f9aa654d1ffc13741d94 contains main fc2cd316e81d7319aa44912e786e7aad4492cd0a; its complete tree is identical to measured head cc00b95f0842dc8a07f7d936f12badde5f6af85f. Normal synchronization commit/push hooks pass; required CI is running on the current pushed head.

Implementation:

  • For overall, type and ext summaries with no category scope, reuse per-root aggregates to construct side-total columns instead of allocating summary vectors/maps under every path and rebuilding the same totals through a category map.
  • Reuse the existing independent-filter audit conservatively. Mixed/category scopes, complex summaries and expressions with actions, traversal controls, collections or verification retain the existing per-entry implementation.
  • Add no flag-name scans, new effect vocabulary, worker, lock, clock, exception mechanism, dependency or mutation route. Traversal and content observations remain in place.
  • This is O16a summary retention only: both complete tree maps remain, and first-output latency is unchanged. Document the still-open O16b completed-frontier interface and O17/O18 concurrency/backpressure boundaries, including empty/ignored completion, ordering, errors, effects, archives and root-relative ignore context.

Evidence and tradeoffs:

  • docs/benchmarks/aggregate-comparison-totals-20261102.md and compressed raw data retain source/compiler/build/link/binary identities, all samples, scripts, setup failures and validation logs.
  • Per host: 72 native cases x three sessions, broad/deep trees with 10/1,000/10,000 paths, one/three requested workers, discarded warmups, nine rotating rounds/fastest seven. Independent physical count/byte/category oracles and complete baseline/candidate output equality are required.
  • Separate in-memory whole-engine population: 30 cases x three sessions x two variants. Confirmation uses an identical baseline participant in eight native cases x five sessions and five in-memory cases x three sessions x three variants.
  • Large native totals improve about 16-21% on Linux and 4.6-5.2% on macOS in confirmation. In-memory savings are larger because native startup/traversal/I/O remain. Category/mixed/status controls and small commands retain their measured slower samples; no universal speedup is claimed.
  • Peak RSS and unstripped CLI size are reported explicitly. Full tree retention remains linear; this is not a bounded streaming algorithm.

Validation:

  • New isolated, read-only VFS tests compare ordinary totals with the existing action fallback across scopes, filters, repeated sinks, directory roots, file/directory changes and empty sides; expected physical counts/bytes are asserted independently.
  • 60 complete baseline/candidate formatting comparisons pass per host across plain, Markdown, JSONL, CSV and TSV.
  • All three modified translation units pass scoped local clang-tidy. All 323 extension-enabled release tests pass per host, with required Linux FUSE. Five focused ASan suites pass on each host and five Linux TSan suites pass. Current pushed-head CI and normal enforcing hooks remain required before readiness.

Update the canonical performance analysis, TODO, summary design, changelog and release-site mapping. Preserve broader expression, concurrency, archive, metadata, similarity and output work as open rather than claiming it complete from this narrower optimization.

Base automatically changed from perf/prepare-hash-operands to main November 2, 2026 20:24
@helly25
helly25 merged commit 6bebe94 into main Nov 2, 2026
32 checks passed
@helly25
helly25 deleted the perf/aggregate-comparison-totals branch November 2, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant