Remove the AVX2 gather extend kernels (x86 stalls); race-safe bench runs

[cjfields]

Part of #31. This removes the AVX2 gather-based extend kernels, which made Rust up to 2× slower than C on x86 for long or divergent alignments.

Evidence

bench/sweep.py on hpcbio (Xeon Gold 6150, Skylake-SP; main aa091cf) had Rust up to 2.16× slower than generic (scalar) C on long or divergent edit/indel/linear alignments (l10k_5, m1k_20), while executing only about 0.6× the instructions. The EPYC 7713 quick run showed the same pattern (indel 1.22–1.25×). The Apple M2 never takes this path.

perf on l10k_5 indel score:

	C (scalar)	Rust
cycles	0.70 G	1.50 G
instructions	2.74 G	1.56 G
stall cycles (total)	0.013 G	0.487 G
stalls on L1 miss	0.5 M	4.6 M
4K-alias events	2.5 M	5.3 M
store-buffer stalls	0.12 M	0.20 M

• Hot spot: 88% of Rust's time is in extend_end2end_avx2, and 65% of that is on its two vpgatherdd. That kernel gathers 4 bytes of pattern and text for 8 diagonals at a time.
• Gather latency, not memory: cache misses, store-buffer stalls and 4K aliasing are all small, so the stall cycles come from the gathers themselves.
• Why it went unnoticed: Rust only used the kernel for wavefronts of 256+ diagonals, which the 1 kb/5% benchmark rarely produces. C's AVX2 build uses the same gathers and was also slower than its scalar build on the EPYC. On Skylake-SP, C-native uses its AVX-512 kernel instead.

Change

• Removed extend_end2end_avx2, extend_end2end_max_avx2 (the BiWFA variant) and their lzcnt helper. x86 now uses the blockwise 64-bit scalar extend, the same one ARM uses. The AVX2 compute kernels are unaffected, since they don't gather.
• Output is unchanged: both kernels compute the same offsets. Tests and clippy pass on ARM. CI covers x86, including the differential harness and the x86-64-v3 build.
• Docs: PERFORMANCE_LOG and PROGRESS entries, and a note in .claude/AVX2_MIGRATION.md, whose original analysis predicted that a gather extend was "unlikely to help".

Also: race-safe benchmark runs from a shared checkout

So several cluster nodes can run bench/sweep.py from one checkout at the same time:

• Datasets: each process writes a private temporary file and renames it into place. The content is deterministic, so a lost race is harmless.
• C builds (build_c.sh): each run builds in a private directory and moves it into place only if no finished build exists. If it loses the race, it keeps the other run's build; BSD mv fails and GNU mv nests, and both cases are handled. A directory left without a binary by an interrupted older run is rebuilt.
• Tested with 3×3 concurrent C builds (all succeed, one clean build, no leftovers) and with two concurrent sweeps generating the same datasets (files intact and identical to a single generation).

To confirm

Re-run bench/sweep.py on hpcbio and hpcbioamd once this is merged, or from this branch. Expected:

• Rust at or below generic C on l10k_5 and m1k_20 edit/indel/linear;
• the BiWFA (ultralow) affine2p gap may narrow too, since BiWFA used the max variant.

🤖 Generated with Claude Code