Baseline: bytes per token from offsets, not from vocabulary inspection

[AmitMY]

Stacked on #80.

The causal-LM baseline needs the UTF-8 bytes of each label to report bits per byte comparable to WeLT's. Each token's bytes were inferred from its vocabulary entry (token_byte_lengths), special-casing byte-level BPE, SentencePiece's ▁ and <0xNN> byte fallback, and off by one for SentencePiece's dummy prefix space.

Now each token counts the text since the previous token's end, from the tokenizer's offsets (which token_chunks already requested). This is exact for any tokenizer with offsets, and counts every byte once.

A character split across tokens counts in its first token; its other tokens have 0 bytes but are still scored as text. That's why a text_mask (scored labels that are not EOS) now marks the bits-per-byte numerator, instead of label_bytes > 0.

Tests

• A new test checks that each byte of mixed scripts and emoji (Pythia tokenizer) is counted exactly once, and EOS as 0 bytes. It replaces the per-tokenizer vocabulary test, which downloaded TinyLlama and Qwen tokenizers.
• test_baseline_scores_the_bytes_welt_scores (the baseline scores exactly the bytes WeLT scores) passes.
• The loss test covers a split character.
• A 30-step baseline training run reports 3.547 bits per byte, against 3.542 before.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f

---

Note

Medium Risk
Changes how baseline training metrics (bits per byte) are computed and could shift reported numbers slightly, but logic is better aligned with WeLT and covered by updated tests.

Overview
Replaces vocabulary-based per-token byte counting with offset-derived spans: each label’s label_bytes is the UTF-8 length of text since the previous token’s end, so every byte is counted once for any tokenizer that exposes offsets.

Bits-per-byte now uses a new text_mask (scored labels that are not EOS) for the loss numerator instead of label_bytes > 0, so continuation tokens after a split character (0 bytes) still contribute to cross-entropy while byte totals stay correct. token_byte_lengths is removed; token_chunks / token_examples / loss_func and dataset wiring are updated accordingly.

Tests swap the multi-model vocab check for a Pythia offset round-trip test and extend the loss test for split-character labels.

^{Reviewed by Cursor Bugbot for commit 4ab190b. Bugbot is set up for automated code reviews on this repo. Configure here.}