Repository navigation
Restore CONTROl_TOKENS_PATTERN, used by words-segmentation - #15
Merged
Merged
Conversation
#4 removed it as unused here, but words-segmentation imports it: with 0.8.3, importing words_segmentation fails. A test now keeps it as public API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restore
CONTROl_TOKENS_PATTERN, used by words-segmentation#4 removed
CONTROl_TOKENS_PATTERNfromutf8_tokenizer.controlas unused in this repo. But words-segmentation imports it (pretokenizer.py,languages.py), so with v0.8.3import words_segmentationfails with anImportError.The constant is restored byte-identical to v0.8.2. A test now pins it as public API: it matches exactly the non-whitespace control characters.
It was the only top-level name removed between v0.8.2 and v0.8.3.
🤖 Generated with Claude Code
https://claude.ai/code/session_015Xm62qoeaTunFSU5GYCQ1f
Note
Low Risk
Backward-compatibility restore of a documented public constant plus a unit test; no runtime behavior change beyond fixing imports for dependents.
Overview
Re-adds
CONTROl_TOKENS_PATTERNtoutf8_tokenizer.controlwith the same value as v0.8.2 (\x01-\x08\x0e-\x1f\x7f), fixingImportErrorfor downstream words-segmentation after it was dropped in v0.8.3.A new test treats the constant as public API and asserts its regex range matches all non-whitespace ASCII control characters (excluding tab, LF, VT, FF, CR, and space).
Reviewed by Cursor Bugbot for commit b8314e1. Bugbot is set up for automated code reviews on this repo. Configure here.