Gleipnir 1.2: model 4 at -9, x86 contexts at every preset, huge pages

[ValisSowilo]

Summary

Gleipnir 1.2.0. -9 becomes a maximum-ratio preset built on a new model. -1 to -7 keep 1.1's model and add x86 instruction contexts only. On Linux, huge pages make it faster. Archive format v4.

Results (full corpora, every result round-trip verified)

	1.1	1.2	change
Silesia -9, per file	35,468,198	34,426,438	−2.94%
enwik8 -9 -s1000	18,027,359	17,599,927	−2.37%
enwik9 -9 -s1000	148,026,632	143,807,662	−2.85%
enwik9 -9 -s1000 -m1	—	141,380,893	−4.49%, below zpaq 6.42 -max

Against the published tables that would be about 33rd on Silesia (listed: 49th) and about 27th on LTCB (26th with -m1; listed: 35th). Nothing has been submitted.

Full Silesia, 1.1 and 1.2 interleaved file by file, -t1:

preset	size	time
-1	−1.33%	+8.9%
-3	−0.76%	+4.1%
-5	−0.54%	+4.2%
-7	−0.50%	+6.0%
-9	−2.94%	+90%

At -1 to -7 every file without x86 code is byte-identical to 1.1. The change is all in mozilla and ooffice (ooffice −12% at -1).

What changed

• Model 4 (-9): twelve additions, each still positive with the others on. The biggest:

  • a second mixer selected by the previous byte;
  • an x86 instruction model: a table-driven length decoder whose contexts see opcode, ModRM and operand position;
  • a run model;
  • a long-match model;
  • line and column contexts: nci −14%.
  • The rest: an order-3 indirect context, an order-0 context, linear-domain inputs, match-byte contexts, 4-byte follow-history, and the word-transform contexts on untransformed text.

  ARCHITECTURE.md §30 has the search: what each adds, what it costs, and what was rejected.

• Presets: model-4 features at -1 to -7 were measured too, and none beat stepping up a preset on full Silesia at equal time. The first cut had them and lost 1.5% at -5; the small screening set was two-fifths x86 code. So -1 to -7 take only the x86 contexts, which run only on x86 blocks.

• Compatibility: archive v4. v2 and v3 decode with the 1.1 model bit for bit, and r keeps a repaired archive's version. The bit path is compiled twice, so batches that run no model-4 code take the exact 1.1 path at 1.1 speed.

• Huge pages: madvise(MADV_HUGEPAGE) on every table of 4 MB and up. On Linux with THP in madvise mode, -9 runs 26% faster and -3 7% faster, output identical. It's a no-op elsewhere.

• Memory: a -9 worker now budgets about 1.5 GB (1.1: 1.1 GB). Silesia peaks at 1,579 MB. enwik9 -9 -s1000 peaks at 3.6 GB.

• Docs: README "What 1.2 changed", USAGE, SECURITY, LEADERBOARDS projections and dist texts are updated. Version 1.2.0 is set in packaging and CI. The ranking and per-file graphs are regenerated.

Not re-measured

The single-session 1.1 preset table and its ladder/speed/memory graphs are kept and labelled as 1.1's. 1.2's differences are in the interleaved table above rather than mixed into another session's timings.

Test plan

• fuzz.py, tfuzz.py, gfuzz.py, sfuzz.py, wfuzz.py (at -3, -5, -7, -9)
• Mixed x86/text/raster directory at -5/-7/-9 with -t6 -s1, extracted -t3, checked with t -D
• 1.1 archives extract with 1.2; a repaired 1.1 archive stays v3
• With features off, output body byte-identical to 1.1
• All twelve Silesia files at -1/-3/-5/-7/-9, enwik8 -9, and Calgary/Canterbury -9 round-trip; both enwik9 runs pass t -D
• ASan + UBSan round trips at -9/-7/-1 (Linux)
• CI