[michalharakal]

Summary

StreamingGGUFReader read the header, KV and tensor-info sections one value at a time: one RandomAccessSource.readAt per primitive (a fresh byte array each; on JvmRandomAccessSource / AndroidRandomAccessSource one FileChannel read each), plus a Pair and a boxed offset per array element. A tokenizer GGUF carries arrays of ~262k strings, floats and ints, so opening one was well over a million source calls.

• New internal GgufMetadataCursor: pulls the source in 64 KB blocks through the buffer form of readAt, decodes little-endian values from the block (commonMain, no java.nio), grows the block for a single long value.
• String and 32-bit arrays are decoded block by block straight into pre-sized ArrayLists.
• fields is unchanged: same keys and order, same value and element classes (UInt for UINT32, Float, Int, String, …), still plain lists, so every consumer (as? List<*>, GGUFWriter, GgufMemoryPlan, the tokenizer builders) works as before.
• A truncated file or an array count larger than the rest of the file now fails with an IllegalArgumentException before allocating.
• No public API change; RandomAccessSource untouched.

An earlier variant exposed the primitive arrays as lazily boxing AbstractList views. It parsed faster but made the tokenizer build ~2× slower on an interpreted runtime (the build walks those lists several times), so values are boxed once into plain lists.

Numbers (Gemma-3 tokenizer GGUF, 6.5 MB, 262,172 tokens)

	before	after
JVM, file source	1,002 ms	21–45 ms
JVM, in memory	55 ms	21–34 ms
JVM -Xint, file / in memory	13.9 s / 1.4 s	0.5 s / 0.5 s
32-bit ARM Android device, debuggable app, file source	23.3 s	1.2 s
same, in memory, warm / first parse in process	2.0 s / 4.0–4.7 s	1.15 s / 3.0 s

Tests

• StreamingGGUFReaderMetadataParityTest: the previous parser kept in the test as LegacyFieldParser, compared with the new one per key, value class and element class: every value type incl. nested and empty arrays, a 120k-char string, a 262k-entry tokenizer-shaped file, a source that returns 7 bytes per read, the file source, truncated input, an oversized array count, a GGUFWriter round trip, and a real GGUF via SKAINET_GGUF_FILE (env-gated).
• GgufTokenizerLoadBenchmarkTest (io-core, env-gated timing harness, no assertions).
• :skainet-io:skainet-io-gguf:jvmTest, :skainet-io:skainet-io-core:jvmTest, :skainet-frontend:skainet-frontend-text:jvmTest green; io-gguf compiles for linuxX64, androidNativeArm32, js, wasmJs, android.

Follow-up: #1447 (tokenizer build, stacked on this branch).

🤖 Generated with Claude Code