Repository navigation

fix: unicode token boundaries for non-ASCII “Other” scripts - #1

Merged

darkskygit merged 2 commits into

darkskygit:masterfrom

Konf:unicode_fix

Apr 5, 2026

Merged

fix: unicode token boundaries for non-ASCII “Other” scripts#1
darkskygit merged 2 commits into
darkskygit:masterfrom
Konf:unicode_fix

Commits on Mar 10, 2026
- Add proper word-split for unicode (SegmentScript::Other)
  Konf
  committed
  4dd7a00
- Add tests to cover unicode split issues
  Konf
  committed
  a38ef6f