Skip to content

x86: claim affine2p output lines with PREFETCHW; drop input prefetches - #52

Merged
cjfields merged 1 commit into
mainfrom
perf/x86-prefetchw
Oct 17, 2026
Merged

cjfields merged 1 commit into
mainfrom
perf/x86-prefetchw

Conversation

@cjfields

Copy link
Copy Markdown
Member

Part of #31. Closes the Xeon (Skylake-SP) affine2p CIGAR gap.

Problem

After #49, the Xeon Gold 6150 was the only machine where gap-affine 2-piece high-memory CIGAR wasn't faster than C:

Rust / C M2 EPYC 7713 Xeon 6150
affine2p CIGAR 0.96 0.77 1.01
affine2p banded CIGAR – 0.75 1.28

Cause

Measured with perf on the Xeon, posted in #31.

  • The store buffer was full for 4.67 G cycles, against 0.38 G for C. About 60% of those stalls fell on the five output stores of the AVX2 affine2p kernel.
  • In high memory those outputs are fresh lines. Rust's stores missed L2 when claiming ownership of a line (RFO) about 158 M times, against about 88 M in C's kernel.
  • Ruled out along the way:
    • per-component victim outputs (no change);
    • split stores (12% vs C's 10%);
    • slab cache coloring (helps only Xeon banded, costs up to 4% on the EPYC);
    • the input prefetches themselves (not the cause, but no help either).

Change

  • The affine2p AVX2 kernel issues PREFETCHW on its five output arrays, 32 diagonals (two cache lines) ahead. Prefetch addresses use wrapping_offset.
  • All x86 input prefetches are removed: the per-step ones in aligner.rs and the in-kernel ones in affine.rs/affine2p.rs. The hardware prefetcher already covers these sequential streams.
  • ARM is unchanged, since all of this is x86-only.

Results

A/B at 45f419e, best of 3. Time relative to main; 32, 64 and 128 are the prefetch distance in diagonals:

Case Xeon: 32 Xeon: 64 Xeon: 128 EPYC: 32
affine2p CIGAR (m1k_5) 0.88 0.98 1.08 0.96
affine2p banded CIGAR (m1k_20) 0.65 0.91 1.05 1.03
affine2p CIGAR (l10k_5) 0.89 0.92 0.96 1.00
affine2p score 0.93 0.95 0.97 0.94
affine2p BiWFA 0.95 0.98 1.03 0.96
gap-affine CIGAR / score 0.98 / 0.98 1.01 / 0.97
edit (control) 0.99 0.99
  • Xeon, store-buffer stalls on affine2p CIGAR: 4.63 G → 0.43 G cycles, about C's level.
  • Xeon, against C: affine2p CIGAR 1.01× → 0.89×, banded 1.28× → 0.83×, 10 kb 0.97× → 0.87×.
  • EPYC: 2–6% faster on most affine2p cases; the rest is within noise. The edit control moved about 1%.

Testing

  • Output: identical to unchanged main in all 16 A/B cases across both machines (six builds each). Under Rosetta, the AVX2 build matches C on affine2p high, BiWFA and med, and gap-affine high.
  • cargo test --release: 271 passed.
  • cargo clippy --all-targets -D warnings: clean on ARM, x86-64 and x86-64-v3.
  • CI runs the differential harness on Linux x86, covering both runtime-selected and compile-time AVX2.

docs/performance.md stays pinned to fc51088, so its Xeon numbers predate this change. The next refresh of the README table would pick it up.

🤖 Generated with Claude Code

On a Xeon Gold 6150 (Skylake-SP), gap-affine 2-piece high-memory CIGAR was
store-buffer bound: the AVX2 kernel's five output stores go to fresh lines,
whose ownership requests (RFOs) missed L2 ~1.8x as often as in C's kernel,
leaving 4.67 G cycles of store-buffer-full stalls against C's 0.38 G.

The kernel now issues PREFETCHW on its five outputs 32 diagonals (two lines)
ahead, and all x86 input prefetches are removed (the hardware prefetcher
covers those sequential streams; they did not help in A/B). 32 beat 64 and
128 diagonals.

Xeon, time vs before: affine2p CIGAR 0.88x, banded 0.65x, 10 kb 0.89x,
score 0.93x, BiWFA 0.95x; store-buffer stalls 4.63 G -> 0.43 G cycles. EPYC
7713: 2-6% faster on most affine2p cases, the rest within noise. ARM is
unchanged. Output identical in every A/B case on both machines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cjfields
cjfields merged commit d4b6714 into main Oct 17, 2026
17 checks passed
@cjfields
cjfields deleted the perf/x86-prefetchw branch October 17, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant