Skip to content

Word transform for text: enwik9 −5.76%, enwik8 −4.16% (1.1.0, format v3) - #6

Merged
ValisSowilo merged 1 commit into
mainfrom
feat/word-transform
Sep 26, 2026
Merged

ValisSowilo merged 1 commit into
mainfrom
feat/word-transform

Conversation

@ValisSowilo

@ValisSowilo ValisSowilo commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

What

Text segments are dictionary-coded before modelling: each frequent word becomes a 1–3 byte code (0x80–0xFF) from a dictionary built from the segment itself, carried at the head of the stream. Case is a flag byte. Inside the 1- and 2-byte classes, words are ordered by recursive principal-component bisection of their neighbour statistics, so a code's lead byte names a cluster of words used alike. The model is transform-aware:

  • code bytes hash as word characters
  • there's a mixer weight set for mid-code positions
  • at -5/-7/-9, transformed segments get four extra contexts: skip bigram, word-class trigram from code lead bytes, word + punctuation, and previous word + order 3.

Design, every measured alternative, and the gates: ARCHITECTURE.md §29.

Results (−9, round-trip verified, same machine)

input 1.0.2 1.1
enwik9 -s1000 157,073,381 148,026,632 −5.76%
enwik8 -s1000 18,810,680 18,027,359 −4.16%
Silesia per-file total 35,583,396 35,468,198 −0.32%
— xml / dickens / webster −2.88% / −1.97% / −1.20%
— other nine files byte-identical 0

Against the published tables, that would place about 29th of 227 on LTCB (listed 35th) and 48th of 322 on Silesia (listed 49th). It hasn't been submitted.

Per preset on 16 MB of enwik8 (A-B-B-A ×4, medians): -f1 −10.2% / −15% time, -1 −6.0% / −25%, -3 −2.1% / −26%, -5 −3.5% / ±0%, -7 −2.5% / −6%, -9 −2.7% / −13%.

Costs:

  • Transformed text at -9 peaks about 260 MB higher (four 2^22 tables). Silesia's max peak goes from 1,055 MB to 1,188 MB (webster).
  • enwik9 peaks lower (2,847 / 3,235 MB vs 3,033 / 3,836 MB).
  • Files under 1 MB gain less, and files under 256 KB are left alone.

Format

  • New segment kind 2 and archive format v3. 1.1 reads v2 and v3; 1.0 refuses v3 with a clear version message (checked both ways).
  • -w0 turns the transform off.
  • Release string is now 1.1.0 (it said 1.0.1 at 1.0.2).

Verification

  • sfuzz.py: +26 kind-2 attack cases and a third positive control. 68 cases × 2 modes pass.
  • fuzz.py (648 round trips), tfuzz.py (60), gfuzz.py (300 trials): pass.
  • Transform-specific round-trip fuzz covering flag bytes, high bytes, case forms and the 32-letter limit: 190 cases at -1/-5/-9, 0 failures.
  • Gates checked by forcing the transform on files it declines: samba +10.5%, reymont +2.2%.

Not done here

  • The top-of-README preset table (8 presets × Silesia, one session) is still 1.0.2's and is marked as such. Re-running scripts/bench_session.py takes about 2.5 h.
  • The 1.1 enwik9 times aren't claimed; the machine wasn't quiet for that run.
  • No release has been tagged and nothing has been resubmitted to Mahoney.

Segments that are mostly words are dictionary-coded before modelling: each
frequent word becomes a one-to-three byte code (0x80-0xFF) from a dictionary
built out of the segment itself and carried at the head of the stream, with
case as a flag byte. Inside the 1- and 2-byte classes words are ordered by
recursive principal-component bisection of their neighbour statistics, so a
code's lead byte names a cluster of words used alike. The model treats code
bytes as word characters, gets a mixer weight set for mid-code positions, and
on transformed segments at -5/-7/-9 adds four contexts (skip bigram, word-class
trigram from code lead bytes, word + punctuation, previous word + order 3).

Gates decline the transform under 256 KB, under 40% letters, over 2% high
bytes, with a detected stride, or when the stream would not shrink 10%; the
nine Silesia files it declines come out byte-identical.

Measured at -9, round-trip verified: enwik9 157,073,381 -> 148,026,632,
enwik8 18,810,680 -> 18,027,359, Silesia per-file 35,583,396 -> 35,468,198
(dickens, webster, xml). -f1..-3 are 15-26% faster, -9 13%.

Format: new segment kind 2, archive format v3. 1.1 reads v2 and v3; 1.0
refuses v3 with a version message. -w0 turns the transform off.
sfuzz.py gains 26 kind-2 cases and a positive control. Release string 1.1.0.
@ValisSowilo
ValisSowilo merged commit 7677685 into main Sep 26, 2026
2 checks passed
@ValisSowilo
ValisSowilo deleted the feat/word-transform branch September 26, 2026 12:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant