Repository navigation

perf(tokenizer): build a GGUF SentencePiece tokenizer in one pass over the vocab - #1447

Merged

michalharakal merged 4 commits into

perf/tokenizer-one-pass-build

Oct 25, 2026

Merged

perf(tokenizer): build a GGUF SentencePiece tokenizer in one pass over the vocab#1447
michalharakal merged 4 commits into
developfrom
perf/tokenizer-one-pass-build

Commits on Oct 24, 2026
- gguf: parse metadata through a buffered cursor
  
  michalharakal
  and
  claude
  committed
  0ed276d
Commits on Oct 25, 2026