BitEmbedding from utf8-tokenizer instead of WeLT's own copy

[AmitMY]

Stacked on #78.

BitEmbedding from utf8-tokenizer

WeLT's ByteEmbedding (a byte table plus a zero-initialized projection of each byte's bits, folded into a plain table for export) moved to utf8-tokenizer as BitEmbedding (sign/utf8-tokenizer#14, released in 0.8.4). WeLT now imports it.

The parameter names are the same (weight, bit_proj), so existing checkpoints load unchanged.

The dependency is utf8-tokenizer[fast]>=0.8.5 with words-segmentation>=0.0.7: 0.8.5 renamed the control-tokens pattern words-segmentation uses (sign/utf8-tokenizer#16, sign/words-segmentation#9).

Tests

With 0.8.4 installed, test_model.py, test_model_overfitting.py, test_processor.py and test_train.py pass (train → export → vLLM generation): 40 passed, 1 skipped. The Docker image needs a rebuild to pick up 0.8.4.

🤖 Generated with Claude Code

https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f

---

Note

Low Risk
Localized embedding swap with compatible parameter names; main risk is version skew if environments don’t pick up the bumped tokenizer/segmentation deps.

Overview
Removes WeLT’s in-repo ByteEmbedding and wires byte embeddings through BitEmbedding from utf8-tokenizer for the bytes encoder and decoder.

Dependency floors move to words-segmentation>=0.0.7 and utf8-tokenizer[fast]>=0.8.5 (aligned control-token naming). weight and bit_proj are unchanged, so existing checkpoints should load without migration.

^{Reviewed by Cursor Bugbot for commit bef353f. Bugbot is set up for automated code reviews on this repo. Configure here.}