Repository navigation
perf(tokenizer): build a GGUF SentencePiece tokenizer in one pass over the vocab - #1447
Merged
Merged
Conversation
StreamingGGUFReader read every metadata value with its own RandomAccessSource.readAt (a fresh byte array each, one FileChannel read each on the file sources) and allocated a Pair and a boxed offset per array element. A tokenizer GGUF carries arrays of ~262k strings, floats and ints, so opening it cost over a million source calls. GgufMetadataCursor reads the source in 64 KB blocks and decodes little-endian values from the block; string and 32-bit arrays are decoded block by block straight into pre-sized lists. `fields` keeps its keys, order, value classes and element classes, and stays plain lists (a lazily boxing view was slower for consumers that walk the lists several times on an interpreted runtime). Truncated input and array counts larger than the file fail with IllegalArgumentException before allocating. StreamingGGUFReaderMetadataParityTest compares against the previous per-value parse (kept in the test) on every value type, a long string over the block size, a 262k-entry tokenizer-shaped file, a source that returns 7 bytes per read, the file source, truncated input and a GGUFWriter round trip; SKAINET_GGUF_FILE adds a real file. GgufTokenizerLoadBenchmarkTest times parse and tokenizer build. Gemma-3 tokenizer GGUF, JVM: file source 1,002 -> 21-45 ms, in-memory 55 -> 21-34 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…vocab TextFrontEnds.fromGguf derived the token list, the token types and the special-token map three to four times per build (fromGgufFields, specialTokenMap twice), each a full copy of a ~262k-entry list, and copied the scores twice (List<Float>, then FloatArray). Now the tokens, token types and special tokens are taken once and handed to an internal fromGgufFields overload; the token list is used as is when it holds only strings; the scores go straight into a FloatArray through an internal constructor (the public constructors and fromGgufFields(Map) are unchanged; apiCheck green). ArrayVocabulary fills its map from the last id down, one map operation per piece instead of containsKey + put, with the same first-occurrence-wins result. GgufTokenizerIdParityTest (env-gated: a GGUF, a corpus of JSON-string lines and a recorded ids file) compares encode() over a corpus, the vocabulary round trip and the special-token policy with a baseline recorded before the change: 26,803 texts identical on a Gemma-3 tokenizer GGUF. Build of that tokenizer: JVM -Xint 1,211 -> 276 ms; debuggable Android app on a 32-bit ARM box 4.5-5.4 -> 3.0-3.3 s first build in a process, 2.6-2.8 -> 1.4-1.7 s warm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michalharakal
changed the base branch from
perf/gguf-buffered-metadata
to
develop
October 25, 2026 13:17
…-metadata # Conflicts: # CHANGELOG.md
Merged
michalharakal
added a commit
to michalharakal/SKaiNET-core-upstream
that referenced
this pull request
Oct 25, 2026
… build, AVX2/FMA SGEMM CHANGELOG [0.62.3] - 2026-10-25: the buffered metadata cursor in StreamingGGUFReader (SKaiNET-developers#1446), the one-pass GGUF SentencePiece tokenizer build (SKaiNET-developers#1447), the AVX2/FMA lane of the dense FP32 SGEMM and the sub-block output fix of NativeFp32MatmulKernel (SKaiNET-developers#1445). VERSION_NAME, CITATION, Antora version, README bom coordinate, What's New and contributors, the Android tutorial coordinates; operator and kernel-support matrices (version stamp). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1446 (base is its branch; GitHub retargets to
developonce #1446 is merged).Summary
TextFrontEnds.fromGguffor a SentencePiece GGUF derived the token list, the token types and the special-token map three to four times per build (fromGgufFields,specialTokenMapfrom both), each a full copy of a ~262k-entry list, and copied the scores twice (List<Float>, thenFloatArray).SentencePieceTokenizer.fromGgufFields(fields, tokens, types, specials)overload;fromGgufFields(Map)stays and delegates.GgufTokenizerFields.tokensreturns the reader's list itself when every element is aString(no copy); token types and scores are read in one loop each into anIntArray/FloatArray.SentencePieceTokenizertakes the scores as aFloatArraythrough an internal primary constructor; the public constructor (now secondary) has the same signature.ArrayVocabularyfills its map from the last id down, one map operation per piece instead ofcontainsKey+put; first occurrence still wins.QwenByteLevelBpeTokenizer.fromGgufFieldsreuses its token list for the special-token map.apiCheckgreen for frontend-core, frontend-text, io-core: no public API change.Tried and dropped: an open-addressing
String → idtable inArrayVocabulary. On-Xintit was 14× slower than theHashMap(91 → 1,276 ms); on Android the platformHashMapis precompiled while our loop is not, so it can only lose there.Numbers (Gemma-3 tokenizer GGUF, 262,172 tokens)
TextFrontEnds.fromGguf-XintTests
GgufTokenizerIdParityTest(io-core jvmTest, env-gated:SKAINET_GGUF_FILE,SKAINET_TOKENIZER_CORPUSwith one JSON string per line,SKAINET_TOKENIZER_IDS): recordsencode()ids over a corpus plus the vocabulary round trip and the special-token policy, then compares later builds line by line. Baseline recorded before the change; 26,803 texts (incl. chat-template prompts with special tokens, mixed scripts, emoji, byte fallback) identical after it.:skainet-frontend:skainet-frontend-core:jvmTest(3),:skainet-frontend:skainet-frontend-text:jvmTest(68),:skainet-io:skainet-io-core:jvmTest(145),:skainet-io:skainet-io-gguf:jvmTest(179) green; frontend-core/-text compile for linuxX64, androidNativeArm32, js, wasmJs, android.🤖 Generated with Claude Code