Repository navigation
fix(native-cpu): multi-row products through the native Q8_0/Q4_0/Q5_0/Q5_1 kernels no longer zero the earlier rows - #1436
Merged
michalharakal merged 1 commit intoOct 21, 2026
Conversation
…/Q5_1 kernels no longer zero the earlier rows The FFM bindings of these four kernels staged a zero-filled output segment of outputOffset + outputDim floats and copied the whole reach back from index 0, so a product served row by row into one array (DefaultCpuOpsJvm) kept only its last row. Single-row decode steps were right; the quantize-at-load lane on an encoder (Laya on Q8_0 with the native provider) returned uniform probabilities. The bindings now copy back exactly their slice, as the K-quant bindings do. NativeQ8CanonicalWeightTransposedReproTest covers every packed format for 1, 7 and 50 rows and the SPI offset contract.
aharakal
approved these changes
Oct 21, 2026
michalharakal
added a commit
that referenced
this pull request
Oct 21, 2026
CHANGELOG [0.62.1] - 2026-10-21 over everything develop holds since 0.62.0: the FFM Q8_0/Q4_0/Q5_0/Q5_1 binding fix (#1436) as the headline, the YOLO move-out (#1437), the StableHLO softmax axis and logSoftmax fixes, the compile-flexibility explanation (#1432) and the SKEEP-008 draft (#1434). VERSION_NAME, CITATION, Antora version, README bom coordinate, What's New and contributors, Android tutorial coordinates; regenerated operator and kernel-support matrices.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-row activations through the native Q8_0, Q4_0, Q5_0 and Q5_1 kernels came back as zeros for every row but the last.
DefaultCpuOpsJvm.chooseQuantizedMatmul2Dserves a[rows, in] × packed [out, in]ᵀproduct by calling the format's SPI kernel once per row withoutputOffset = row * outinto one shared output array; the four FFM bindings staged a zero-filled output segment ofoutputOffset + outputDimfloats and copied the whole reach back from index 0, wiping the rows computed before each call. The K-quant bindings copy only their own slice and were never affected.Why nobody saw it: a single-row product (a decode step) is correct, and the GGUF serving paths are either single-row or
MemorySegment-backed (other bindings). It surfaces with the quantize-at-load lane on an encoder, where every token is a row: Laya on Q8_0 withskainet-backend-native-cpuon the classpath answered every question with uniform probabilities (found while building the LayaMoE example; the transformersLayaQuantizedParityTestreproduces it with the native provider on the test classpath — CI's classpath has none).The four bindings now copy back exactly the slice they produced. New
NativeQ8CanonicalWeightTransposedReproTest: every packed format (Q8_0, Q4_0, Q5_0, Q5_1, Q4_K, Q5_K, Q6_K) throughops.matmulWeightTransposedon a canonical weight for 1, 7 and 50 rows against the block-sequential dequant ground truth, with the native provider alone and with the default registry (second pass through the relayout cache), plus the SPI contract that a call at an output offset writes only its own row (sentinel rows before and after untouched). Before the fix the multi-row cases failed with all-zero rows; after it the worst deviation is 1e-4 of the output scale for the 32-element formats.Fix for a 0.62.1; no API change.
Verification
:skainet-backends:skainet-backend-native-cpu:jvmTest(29 suites) green; the new test's multi-row cases fail ondevelopand pass here.LayaQuantizedParityTestwith the native provider on the classpath against this build: [laya-q8] 45 values compared, max shift 0.0349, choices 5/5 agree; fp32 40293 ms, q8_0 76705 ms PASSscripts/pr-gate.sh --quick.