Skip to content

Vectorize and fuse the med/low piggyback kernels - #45

Merged
cjfields merged 1 commit into
mainfrom
perf/piggyback-fused
Oct 15, 2026
Merged

cjfields merged 1 commit into
mainfrom
perf/piggyback-fused

Conversation

@cjfields

Copy link
Copy Markdown
Member

Part of #31. This speeds up the med/low memory modes (piggyback backtrace) with no change in output.

Cause

None of the piggyback kernels vectorized in the linked binary, including the per-component passes that had been split out to make them vectorize. LLVM rewrote each if c { a[i] } else { b[i] }, a choice between two loaded values, into a load from a selected address. That is a gather, and it blocks vectorization (the old linear kernel compiles to csel x25, x11, x25, eq; ldr w25, [x25, x8]).

Inspecting the crate's own --emit asm hides this when lto = true: that output comes from the pre-link pipeline, which skips loop vectorization. The linked binary (objdump -d) shows what actually runs.

Changes (src/compute/piggyback.rs)

  • Mask blends. Choices between loaded values use a mask blend, (a & m) | (b & !m), so both values are always loaded.
  • Fused loops. affine and affine2p are each one loop, as in C (wavefront_compute_affine*_idm_piggyback). M is computed from this cell's I/D, which stay in registers. Before, they took 6 and 12 passes.
  • No aliasing. Every loop, edit/indel/linear included, sits in an #[inline(never)] function that takes each array as a separate slice parameter, so LLVM knows nothing aliases. linear reuses the edit kernel, since the cell update is the same.
  • Ties unchanged: I1, I2, D1, D2, then mismatch, with later wins, comparing against the overall max as C does.

All five kernels now vectorize: NEON on this machine, and AVX2 with target-cpu=native on x86.

Results

Apple M2 Pro, best of 3, CIGAR, ms. m1k_5 is 20k × 1 kb at 5%; m1k_20 is 5k × 1 kb at 20%. C is 3282f83.

Case C main fused fused/main fused/C instr. fused/main
m1k_5 edit med 286 259 185 0.71 0.65 0.76
m1k_5 edit low 282 211 175 0.83 0.62 0.76
m1k_5 indel med 200 229 167 0.73 0.83 0.77
m1k_5 indel low 202 226 168 0.74 0.83 0.77
m1k_5 linear med 392 351 237 0.67 0.60 0.72
m1k_5 linear low 435 407 287 0.70 0.66 0.74
m1k_5 affine med 4160 3325 2241 0.67 0.54 0.57
m1k_5 affine low 4610 4032 2958 0.73 0.64 0.63
m1k_5 affine2p med 27180 20665 11721 0.57 0.43 0.49
m1k_5 affine2p low 29470 24332 15345 0.63 0.52 0.57
m1k_20 edit med 761 466 243 0.52 0.32 0.58
m1k_20 edit low 749 464 240 0.52 0.32 0.58
m1k_20 indel med 421 502 311 0.62 0.74 0.61
m1k_20 indel low 386 498 319 0.64 0.83 0.61
m1k_20 linear med 1090 891 478 0.54 0.44 0.56
m1k_20 linear low 1200 1079 647 0.60 0.54 0.61
m1k_20 affine med 9800 7584 4836 0.64 0.49 0.51
m1k_20 affine low 10880 9252 6474 0.70 0.60 0.59
m1k_20 affine2p med 73800 53301 30158 0.57 0.41 0.46
m1k_20 affine2p low 80400 64311 40923 0.64 0.51 0.54

Still to check on x86

The quick sweep on the EPYC 7713, run before this change, had med/low at 1.3–1.9× slower than C built with -march=native, where GCC vectorizes C's fused loops with AVX2. Re-running bench/sweep.py --filter ' (med|low)$' on hpcbioamd/hpcbio with this branch will show whether that gap is closed.

🤖 Generated with Claude Code

None of the piggyback kernels vectorized in the linked binary, including
the per-component passes split out for that purpose: LLVM turned each
choice between two loaded values into a load from a selected address (a
gather), which stops vectorization. (With lto = true the crate's own
--emit asm is the pre-link pipeline and doesn't show this.)

- Choose between loaded values with a mask blend, so both are loaded.
- Fuse affine and affine2p into one loop each, as C does (M from this
  cell's I/D, in registers), instead of 6 and 12 passes.
- Put each loop in an #[inline(never)] function taking every array as a
  separate slice (noalias), including edit/indel/linear.

All five kernels now vectorize. Tie order is unchanged (I1, I2, D1, D2,
then misms; later wins), and output is identical to C and main: the
differential harness is clean (only the 20 known #22 cases differ).
med/low is 0.52-0.83x its previous time on Apple M2 Pro and now faster
than C (indel med/low was 1.12-1.34x C).

Part of #31.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cjfields
cjfields merged commit 4d01834 into main Oct 15, 2026
17 checks passed
@cjfields
cjfields deleted the perf/piggyback-fused branch October 17, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant