test: discover and pin third-party assembly corpus

[cpunion]

Summary

PR #35 and the standalone main repair in PR #39 have merged. This PR is rebased onto current xgo-dev/main and adds a strict, persistent third-party assembly coverage layer.

• scan the official index.golang.org feed and proxy.golang.org/cached-only module ZIPs without executing module code
• persist every successfully inspected exact module@version, including versions with no assembly; repeatable -seen-report inputs skip only those exact versions, while failures remain retryable
• ignore zero-byte, whitespace-only, and comment-only .s placeholders, while retaining unreadable or unusually large assembly for corpus validation rather than silently dropping it
• verify every pinned module is still @latest, filter packages by exact owning module path, and fail on package-load errors, inventory drift, translation failures, or LLVM object failures
• cover all 11 supported GOOS/GOARCH targets; targets with no selected assembly are validated as explicit zero inventories rather than skipped
• derive independent, non-fail-fast Qiniu CI jobs from every manifest entry
• use LLVM 22 for every object check, with no fallback to LLVM 23 or another major

Ecosystem scan and corpus

The continuous local scan has processed 210,000 official index records from 2019-04-10T00:00:00Z through 2019-09-14T10:28:12.476893Z. Its consolidated schema-2 state contains:

• 54,352 unique successfully inspected module@version pairs, whether or not they contain assembly
• 33,761 distinct module paths among completed and failed inspections
• 427 unique assembly-bearing matches
• 145 explicit failure records, none marked completed, so overlapping future scans retry them

The report state is incremental and resumable. A newly published version is not hidden by an older successful scan of the same module.

The pinned manifest now contains 24 modules and 239 target-specific assembly translations across Darwin, Linux, Windows, js/wasm, and wasip1/wasm:

• three llgo issue regressions contribute 54 translations: coder/websocket, klauspost/compress, and tmthrgd/go-hex
• 21 ecosystem modules contribute 185 translations
• this update adds anacrolix/mmsg, cespare/xxhash v1, klauspost/cpuid v1, klauspost/reedsolomon, minio/highwayhash, modern-go/gls, tmthrgd/go-bitwise, tmthrgd/go-popcount, and zeebo/this

Exact module ownership prevents a v1 suite such as klauspost/cpuid from silently including its nested /v2 module. Content inspection also removed false coverage: the Darwin x/net .s placeholders are comment-only, and modern-go/reflect2 contains empty placeholders, so neither is counted as executable assembly.

TDD instruction-family fixes

Native Go cross-assembly and failing real-library suites were used as the oracle before changing plan9asm. In addition to the earlier Snappy, LZ4, and go-bits regressions, the expanded corpus first reproduced failures in HighwayHash, Reed-Solomon, and go-popcount and then drove family-level fixes:

• amd64/386: SUBB; BSFQ/BSRQ/BSFL/BSRL and POPCNTL/POPCNTQ register-or-memory sources; PMULULQ; PUNPCKLLQ/PUNPCKHLQ; VPBROADCASTQ XMM/GPR/memory sources with XMM/YMM/ZMM destinations; and Go ABI aggregate call arguments
• ARM: MOVH/MOVHU plus BPL/BVS/BVC
• ARM64: both MOVKW halves, two- and three-operand MULW, MADDW/MSUBW, CCMP/CCMPW/CCMN/CCMNW, and BVS/BVC

Broadcast positive and negative forms were checked directly with the Go 1.27 amd64 assembler, preventing the shared lowering path from accidentally accepting invalid XMM destinations for VBROADCASTF32X2/VBROADCASTSD or immediate sources. Related spellings and memory/register variants are tested together.

The official Go assembler coverage baselines for Go 1.20 through Go 1.27 were updated only for the 12 newly supported amd64 forms across PMULULQ, POPCNTL/Q, PUNPCKLLQ/HLQ, and VPBROADCASTQ. On Go 1.27, amd64 reports 767 supported and 5,969 explicitly unsupported forms with zero parse errors. Every supported architecture is scanned; unsupported forms remain visible and are not reclassified as a non-target architecture or skipped.

Candidates are admitted only when their complete selected inventory passes. minio/sha256-simd remains outside the green manifest after its unpack instruction was fixed because it additionally requires an unsupported [16][]byte signature and inline LONG raw machine code.

Validation

• root and nested-module go test ./...
• root and discovery/corpus race tests
• root and nested-module go vet ./...
• actionlint and shellcheck
• scripts/check-go-asm-coverage.sh
• scripts/check-arm64-plan9-corpus.sh
• standard-library scan, translation, and LLVM 22 object compilation across every supported architecture
• all 24 third-party suites and all 239 selected translations across the complete 11-target matrix with LLVM 22.1.8

[codecov]

Codecov Report

❌ Patch coverage is 96.26437% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
amd64_lower_arith.go	89.06%	7 Missing ⚠️
amd64_lower_vec.go	96.73%	3 Missing ⚠️
amd64_lower_branch.go	95.83%	2 Missing ⚠️
arm64_lower_arith.go	98.36%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[fennoai]

Review summary

Large, well-executed PR extending amd64/arm64 opcode lowering, adding a reported-library corpus tool, and expanding conformance/coverage testing. The lowering additions are careful about LLVM shift UB, sign-vs-zero extension, and NZCV flag computation, and are backed by strong executable conformance tests validated against native Go oracles. I verified the specifically risky areas (SHLW/SHRW/SARW masking, BSFW/BSRW, arm64 ROR/REV/bitfield, CSEL family, setFlagsSub32, F/V register aliasing) and found them correct.

Findings below. Nothing here is blocking merge outright, but the CI injection and the PSRAL zero-fill are worth addressing.

Additional non-inline notes (lower confidence / design-level):

• Serial compile pipeline (performance). cmd/plan9asmll/main.go runs targets serially, tasks serially, and one blocking llc subprocess per file. This PR raises defaultMatrixTargets() from 8 to 11 targets and the new corpus CI runs the full matrix per library, so wall-clock is sum(targets) x files x (translate + llc) with zero parallelism, leaving CI cores idle. A bounded worker pool over tasks would cut runtime roughly linearly with core count.
• No subprocess timeouts (robustness). cmd/plan9asmcorpus/main.go (go list -m @latest, go mod download) and cmd/plan9asmll (llc) use exec.Command(...).Run()/CombinedOutput() with no context.WithTimeout. Since the corpus tool intentionally fetches network modules, a hung proxy or stuck llc blocks until the job-level timeout. Consider exec.CommandContext with per-command deadlines.
• @latest on manifest-controlled module paths (supply chain). runLibrary resolves and downloads whatever module the manifest names, validated only for non-emptiness/uniqueness. Not shell-injectable (passed as argv), but worth constraining to an allowlist of expected module prefixes or pinned versions with -check-latest=false in CI.
• Vector XMM/YMM branch duplication (maintainability). VPAND, VPCMPEQB/VPCMPGTB, VPUNPCKLBW/VPUNPCKHBW each carry two near-identical blocks differing only by width/parse/load/store; extracting a width-parameterized helper would halve the code and reduce drift risk.

Additional findings

• README.md:29: [P2] README target list is stale and now contradicts the code: This PR changed defaultMatrixTargets() in cmd/plan9asmll/main.go to return 11 targets, adding linux/arm, js/wasm, and wasip1/wasm. The README still lists only the original 8 and states linux/arm "is not part of plan9asmll -all-targets" (line 29) — now false. The new test TestDefaultMatrixTargetsCoversEveryPlan9Architecture asserts all 11, so the README directly contradicts shipped behavior and the test. Update the target list and remove/reword the linux/arm/wasm exclusion sentence.

[cpunion]

Addressed the review findings and LLVM toolchain alignment in the latest head:

• validate manifest-derived matrix ids and keep them out of shell source;
• add executable PSRAL >=32 semantic coverage and correct sign-fill behavior;
• update the README to the full 11-target matrix;
• align CI and all Go LLVM bindings with LLVM 22 (xgo-dev/llvm v0.9.9);
• require LLVM major 22 for automatic and explicit llc selection, with no fallback to LLVM 23 or older releases.

The post-merge PSRAL coverage-baseline repair was also split into the minimal PR #39 so main can be repaired independently.

Local validation used LLVM 22.1.8: root/nested Go tests, actionlint, shellcheck, official assembler coverage, the complete stdlib architecture-level/cross-OS matrix, and all reported-library suites across 11 targets passed.