Repository navigation
Word transform for text: enwik9 −5.76%, enwik8 −4.16% (1.1.0, format v3) - #6
Merged
Merged
Conversation
Segments that are mostly words are dictionary-coded before modelling: each frequent word becomes a one-to-three byte code (0x80-0xFF) from a dictionary built out of the segment itself and carried at the head of the stream, with case as a flag byte. Inside the 1- and 2-byte classes words are ordered by recursive principal-component bisection of their neighbour statistics, so a code's lead byte names a cluster of words used alike. The model treats code bytes as word characters, gets a mixer weight set for mid-code positions, and on transformed segments at -5/-7/-9 adds four contexts (skip bigram, word-class trigram from code lead bytes, word + punctuation, previous word + order 3). Gates decline the transform under 256 KB, under 40% letters, over 2% high bytes, with a detected stride, or when the stream would not shrink 10%; the nine Silesia files it declines come out byte-identical. Measured at -9, round-trip verified: enwik9 157,073,381 -> 148,026,632, enwik8 18,810,680 -> 18,027,359, Silesia per-file 35,583,396 -> 35,468,198 (dickens, webster, xml). -f1..-3 are 15-26% faster, -9 13%. Format: new segment kind 2, archive format v3. 1.1 reads v2 and v3; 1.0 refuses v3 with a version message. -w0 turns the transform off. sfuzz.py gains 26 kind-2 cases and a positive control. Release string 1.1.0.
ValisSowilo
force-pushed
the
feat/word-transform
branch
from
September 26, 2026 12:10
d696c9b to
0c565dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Text segments are dictionary-coded before modelling: each frequent word becomes a 1–3 byte code (
0x80–0xFF) from a dictionary built from the segment itself, carried at the head of the stream. Case is a flag byte. Inside the 1- and 2-byte classes, words are ordered by recursive principal-component bisection of their neighbour statistics, so a code's lead byte names a cluster of words used alike. The model is transform-aware:-5/-7/-9, transformed segments get four extra contexts: skip bigram, word-class trigram from code lead bytes, word + punctuation, and previous word + order 3.Design, every measured alternative, and the gates: ARCHITECTURE.md §29.
Results (−9, round-trip verified, same machine)
-s1000-s1000Against the published tables, that would place about 29th of 227 on LTCB (listed 35th) and 48th of 322 on Silesia (listed 49th). It hasn't been submitted.
Per preset on 16 MB of enwik8 (A-B-B-A ×4, medians):
-f1−10.2% / −15% time,-1−6.0% / −25%,-3−2.1% / −26%,-5−3.5% / ±0%,-7−2.5% / −6%,-9−2.7% / −13%.Costs:
-9peaks about 260 MB higher (four 2^22 tables). Silesia's max peak goes from 1,055 MB to 1,188 MB (webster).Format
-w0turns the transform off.Verification
sfuzz.py: +26 kind-2 attack cases and a third positive control. 68 cases × 2 modes pass.fuzz.py(648 round trips),tfuzz.py(60),gfuzz.py(300 trials): pass.-1/-5/-9, 0 failures.Not done here
scripts/bench_session.pytakes about 2.5 h.