Add tokenization benchmarks comparing Links Notation with JSON, YAML, XML - #210
Merged
Conversation
Adding CLAUDE.md with task information for AI processing. This file will be removed when the task is complete. Issue: #209
… XML This implements UTF-8 character count benchmarks in all six supported languages: - Rust (primary, used in CI/CD for auto-generating BENCHMARK_RESULTS.md) - JavaScript - Python - C# - Go - Java Features: - Five benchmark test cases: employees, simple_doublets, triplets, nested_structure, config - Detailed markdown report with summary and per-case results - GitHub Actions workflow that automatically updates benchmark results on push to main - Consistent benchmark implementation across all languages producing identical results Results show Lino achieves on average: - 47.9% fewer characters vs JSON - 21.5% fewer characters vs YAML - 61.5% fewer characters vs XML Closes #209 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
konard
changed the title
Apply rustfmt formatting and fix clippy warning about redundant closure. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
konard
marked this pull request as ready for review
January 21, 2026 10:38
This reverts commit 222c3f1.
Member
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost estimation:
Now working session is ended, feel free to review and add any feedback on the solution draft. |
Member
Author
|
Get latest branch changes here. And make sure we use new fresh example for JSON serialization we recently supported (see pull requests / issues). We may also need to update SVG generation for our primary example, also double check SVG is done by code, so we can always regenerate examples, not manually. Update all website/docs and so on to be consistent with latest changes at #282 and others. Also update exactly all our dependencies in all supported languages. All I ask must touch all codebase and all supported languages we have. We also must recheck we do the best possible practices in token efficiency benchmarking. |
konard
marked this pull request as draft
September 5, 2026 10:12
Member
Author
|
🤖 AI Work Session Started Starting automated work session at 2026-09-05T10:12:06.050Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback. |
The previous benchmark could not support its own headline. It counted characters rather than tokens, and each format carried a hand-written file, so the JSON and the Links Notation being compared did not hold the same data. The ~45% saving it reported was an artefact of that. - benchmarks/datasets/ is now the only source of data, generated by a seeded PRNG in benchmarks/tools/generate-datasets.mjs so a rerun reproduces it and --check catches drift. Ten datasets cover uniform, semi-uniform and nested records, deeply nested configuration, keyed maps and tuples. - Every other representation is derived from that data by the Rust benchmark: three Links Notation forms, indented and compact JSON, YAML, XML, and CSV where a dataset is genuinely tabular. - Tokens are counted with tiktoken-rs in o200k_base (GPT-5, 4.1, 4o) and cl100k_base (GPT-4, 3.5), alongside characters and bytes. - No number is reported for a document that does not parse with the links-notation crate and decode back to the value it was built from. The Links Notation writer is ported from lino-objects-codec, including the single-line form, and gains a minimal-quoting mode: a string stays bare unless writing it that way would read back as something else. That probe found a real grammar constraint - a bare reference ends at a colon, so timestamps must be quoted - which is recorded in experiments/issue-209. Honest result over the ten datasets, o200k tokens against indented JSON: minimal quoting 29.5%, single line 24.4%, always quoted 23.7%, YAML 24.0%, XML -26.4%. Compact JSON is still cheaper at 38.3%.
The Rust benchmark validating its own output only shows the benchmark agrees with itself. benchmarks/tools/verify-representations.mjs reads the committed documents back with the libraries everyone else uses - lino-objects-codec for all three Links Notation forms, yaml, fast-xml-parser and csv-parse - and fails the run when one does not come back as its dataset. Corrupting a single scalar in a .yaml or .lino file is caught. Two emitter faults surfaced while writing it: - XML dropped the key of an empty array entirely, because an array writes its element once per item. It now writes one empty element, so the XML carries the same keys as the other representations. - YAML left yes, no, on and off as plain scalars. That is correct under the 1.2 core schema and wrong everywhere else: PyYAML would hand back a boolean. They are quoted now, which is what PyYAML, SnakeYAML and js-yaml all write. The new sparse_records dataset is what found them. It carries nulls, empty objects, empty arrays, an empty string, non-ASCII text, and strings such as 1e3, 0012 and yes whose content reads as another type - the cases where a writer that quotes too little silently changes the data rather than shrinking it.
Each language now parses every generated .lino with its own links-notation implementation, counts tokens with its own tokenizer, writes its results file and compares it against the Rust results, failing on any disagreement. The generated manifest gained the schema field the results files carry, so a reader that has no default for it agrees with the ones that do.
The remaining three supported languages now run the same contract as JavaScript, Python and Go: parse every generated .lino with their own links-notation implementation, count tokens with their own tokenizer, and fail when a measurement disagrees with the Rust results. All seven results files are now identical apart from the generator field.
The workflow still committed a benchmarks/benchmark_results.json that no longer exists, validated six languages against whatever was committed rather than against the run it had just made, and had no PHP at all. The Rust job now publishes the documents, results/rust.json and the report as an artifact, and every other job reads that artifact, so the six other languages re-count the exact bytes this run produced. A pull request fails on drift between the generator and the committed output; on main the same drift is what triggers the publish job, which commits generated/, results/ and BENCHMARK_RESULTS.md with GITHUB_TOKEN once every language has agreed. Toolchains follow the per-language workflows (Node 22, Python 3.13, .NET 10, Go 1.26, Java 21, PHP 8.4), and the workflow follows the conventions the rest of them share: checkout@v7 without persisted credentials, per-job concurrency, a verbose dispatch input and least-privilege permissions.
The four documents behind the README picture were maintained by hand, and had drifted: comparison.yaml carried three `id` fields no other document had. comparison.json is now the only file anyone edits. - generate.mjs writes the .lino, .yaml and .xml from it. The notation is whatever lino-objects-codec serialises the object to, which since #282 is the readable indented form, so the README shows what the codec actually writes rather than a form someone typed once. It decodes its own notation and parses its own YAML back before writing. - generate_comparison_svgs.py reads those documents instead of repeating them, sizes the grid from their content, highlights the quoted-string style the codec emits, and gained a --check mode. - generate_png.sh converts with --zoom rather than a fixed width and height, so a longer example widens the picture instead of stretching it. - The docs workflow runs both checks on every pull request that touches docs/comparison, and prints the diff when they drift.
The published site was broken. Its Vite base was still "/Protocols.Lino/" from before the repository was renamed, while Pages serves it from /links-notation/, so every asset 404'd: the live page loaded no CSS, no JavaScript and no comparison image. The base is now relative, which is correct wherever the site is served from and cannot rot on a rename. The content had rotted with it: - The playground ran a 300-line hand-rolled imitation of the parser that lived only in script.js, still announcing version 0.6.0. It now imports this repository's parser and shows the parse, the format back to notation, and the JSON the object codec reads out. The codec only reports a reading when re-encoding reproduces the document, so notation that is not an object graph is named as such rather than given a plausible wrong answer. - The version badge is injected from js/package.json at build time instead of being typed into index.html, where it had sat at v0.6.0 for ten minor releases. - Documentation cards covered three languages under their old package names (@linksplatform/protocols-lino, Platform.Protocols.Lino, platform-lino) and linked to js/README.md paths that 404 on Pages. All seven languages are listed with their current packages and links, and Quick Start shows all seven. - New example cards for the nested contexts parentheses open (#282) and for writing an object as notation. - .code-example, .doc-card and .example-card get min-width: 0. A grid item is min-width: auto, so a long line widened its column past 1fr and pushed the last Quick Start column out of the container once the row grew from three languages to seven. docs/website/dist was committed and stale; the pages workflow builds it fresh on every push, so it is now ignored rather than tracked. The npm audit matrix covers every directory with a lockfile instead of two.
TEST_CASE_COMPARISON.md covered Python, JavaScript, Rust and C#. Go, Java and PHP have been supported for several releases and were simply absent from it, so a test added in one of them was invisible to the only document that shows whether the implementations agree. The generator now reads all seven. The extractors are one table instead of four near-identical functions, and every link points at the line the test name is on rather than the attribute above it. Go is the one language that does not keep a test file per category, so its tests are filed by name into the categories the other languages define, and by file name when no other language has the test. The README quoted the counts by hand: six languages, and six numbers that were between 16 and 173 tests short. Both READMEs now carry a generated block, and `--check` fails when any of the three documents drifts. The `docs` workflow runs it on every pull request touching a test, and the pre-commit hook - which watched only .py, .js, .rs and .cs - now watches all seven languages, Go included, whose tests live next to the code rather than under tests/. The "Known Implementation Differences" list was also wrong on three of its four points: Python has multiline quoted strings (four tests), JavaScript has FormatConfig (js/src/FormatConfig.js), and LinksGroup is missing from four implementations, not one. Replaced with what the source actually shows, linked.
links_notation.__version__ was the literal "0.7.0" while the package released 0.16.1: the only version in the tree that scripts/version-consistency.mjs does not read, and the only one that had drifted, by nine minor releases. A literal in the source is a second place to remember, so there is no literal now. The version comes from the installed distribution's metadata, falling back to pyproject.toml - the one place it is written - when the package is imported from a checkout that was never installed, as the tests do. The read is a regex rather than tomllib because this package supports Python 3.9 and tomllib arrived in 3.11. The module docstring also still called the package Platform.Protocols.Lino.
Issue #282 settled what a parenthesis does to the lines inside it: `(` opens a nested context that starts fresh at indentation level zero and follows the root's rules, so the line breaks and indentation inside are structure. That answer has to hold in every implementation or a document means one thing to one library and another to the next. experiments/issue-282 held a JavaScript-only reproduction. This adds a check that parses the same document with all seven libraries in this tree - reaching into the working copy rather than a published package - and compares each formatted result against the single expected reading. All seven print `(value ((id 1) (label one)))`. Languages whose toolchain is missing are reported as skipped rather than failed. PHP needs an interpreter at least as new as php/composer.json asks for, which the runner finds by version rather than by binary name, because several distributions install an older one as `php`.
Issue #282 changed what a parenthesis means: `(` opens a nested context whose body starts fresh at indentation level zero and follows the root's rules, so a line break inside parentheses is structure. docs/grammar said so; not one of the nine READMEs did, so the documents a reader starts from still described the old reading by omission. Each README now shows the multi-line group next to the indented syntax it belongs with, in that language's own idiom, with the output it actually produces - every snippet here was run against the library in this tree. The two root READMEs also link the grammar, which nothing in the repository linked before. The Russian README was missing the indented syntax section the English one has; it has it now. CHANGELOG: the #282 entry named six implementations and PHP was not among them, though PHP was ported from Python with the fix already in place and has the tests to match.
Every manifest in the repository was checked against what the registry currently publishes, and each bump was followed by the check that would notice if it changed a result, rather than by a version number alone. npm: lino-objects-codec 0.7 -> 0.8 (docs/comparison, docs/website, benchmarks/tools), csv-parse 6 -> 7 and gpt-tokenizer 3 -> 4 in the benchmarks. The two major bumps are the reason the checks matter: the generated comparison documents, the 12 dataset files, the 81 generated benchmark documents and the JavaScript token counts are all unchanged, so the parsers and the tokenizer still read the same bytes the same way. Go: regexp2 1.10 -> 1.12 and uuid 1.3 -> 1.6 in the benchmark module, whose requirement on the library also stops claiming v0.0.0 and names the version that is actually tagged. The `replace` still points at ../../go, so the build keeps measuring the working tree. Java: maven-compiler-plugin 3.15 -> 3.16 and spotless 3.10.1 -> 3.10.2 in both poms, surefire 3.5.6 -> 3.6.0 in the library. C#: Microsoft.ML.Tokenizers 1.0.3 -> 2.0.0. That release asks for Microsoft.Bcl.Memory 9.0.4, which carries GHSA-73j8-2gch-69rq, so the benchmark pins the patched 10.x directly; the restore is back to zero vulnerability warnings and the token counts still agree with Rust. Python, Rust and PHP needed no change: every version they declare is already the newest published one.
Two things a reader of BENCHMARK_RESULTS.md was entitled to and did not get.
The footer said "against `links-notation` 0.1.0". 0.1.0 is the version of
the benchmark crate, which is what `env!("CARGO_PKG_VERSION")` expands to
there; the parser it measured was 0.16.1. `links_notation::VERSION` now
carries the library's own version - the same thing Python already exposes
as `__version__` - and the report reads it from there, so it cannot drift
from the parser again.
The report also claimed nothing about what it does not show, which for a
token-count benchmark is the easiest thing in the world to over-read. A
"What this does not measure" section now says it outright: fewer tokens is
a cost and not a capability, so nothing here is evidence that a model reads
one format better than another; both encodings are OpenAI BPE, chosen
because they are the two every implementation has a tokenizer for, and
other vendors segment text differently; only the document is counted, not
the prompt around it; and size is not speed. The same list is in
benchmarks/README.md, next to the method it qualifies.
`experiments/issue-209/lino-shapes` did not build at all: its path dependency pointed at `../../rust/links-notation`, which resolves to `experiments/rust/links-notation`, so cargo stopped at "No such file or directory" before it could compile. One `..` short. It runs again and prints the eleven shapes it was written to show. `experiments/issue-282/parity` failed rather than skipped on a machine whose `dotnet` is older than the framework the library targets: the C# check went straight to `dotnet run` and the SDK answered NETSDK1045, so the run reported a disagreement that was really a missing toolchain. It now reads the required framework out of the project and asks `dotnet --list-sdks` first, skipping the way the PHP check already skips when no new enough interpreter is installed. With a .NET 10 SDK on PATH all seven implementations still agree; without one, six run and C# is reported as skipped. The `target/` directory of the Rust experiment is ignored, which the parity experiment already did for its own.
A minor bump rather than a patch: `links_notation::VERSION` is new public API in the Rust crate, and the unreleased entries already carry the PHP implementation and the nested-context change to all seven parsers. All seven declare 0.17.0, so `scripts/version-consistency.mjs` passes. The manifests that name the library by version rather than by path follow: `rust/links-notation-benchmark/Cargo.toml`, `benchmarks/java/pom.xml` and the private website package. `benchmarks/go/go.mod` keeps naming 0.16.1, the last version actually tagged, because its `replace` builds the working tree regardless and pointing it at an untagged version would only be true after the release. The benchmark report and the seven results files were regenerated; all seven languages still agree, and the report footer now reads 0.17.0 because it reads the parser's version rather than its own. CHANGELOG records the token efficiency benchmark work for #209: what is measured, how the numbers are kept honest, the CI workflow that regenerates them, the dependency sweep and what the benchmark explicitly does not claim.
The header carried a hand-typed `v0.16.1`, while script.js already writes the same span from `__LIBRARY_VERSION__`, which vite.config.js reads out of js/package.json at build time. Two sources for one number is how the page came to advertise v0.6.0 for ten minor releases, so the literal is gone and the span starts empty. Screenshot regenerated from the built site to show the header now reading v0.17.0.
dev/log/issues/209/pulls/210/README.md, following the pattern of the 290/291 and 292/294 logs: what the benchmark rests on, the headline numbers, the four bugs found while verifying it and how each was traced, and what was run before pushing.
Six of the seven languages had a README.ru.md and Go did not, so README.ru.md sent a Russian reader to an English page with a note apologising for it. Go now has one, translated from README.md and carrying the same nested-contexts section issue #282 added to the English READMEs. That section was also missing from the six existing README.ru.md files: commit 7867587 added it to the nine English READMEs and to the root Russian one, but not to the per-language Russian ones, so a Russian reader still got the old reading by omission. Each now has it, with the same snippet its English counterpart shows. Three things the Go README claimed turned out not to be true, found by running its snippets rather than reading them: - `lino.StrPtr` does not exist. Nothing in the package exports it; the snippet did not compile. It now takes the address of a local, which is what the tests do. - The feature list named four sibling implementations when there are six. - The title did not match the other six READMEs. go/example_readme_test.go keeps the rest honest: the documented snippets are now Go examples with `// Output:` comments, so `go test` fails when the README and the package disagree, and a snippet naming a function that does not exist stops compiling. Also dropped the hand-typed `Version: 0.1.0` from the JS, Java and PHP package information sections - six releases stale, and the same class of bug as the website literal and the benchmark footer, so the registry badge is left as the single source. The JS README claimed MIT while js/package.json declares Unlicense.
python/pyproject.toml sets black and isort to line length 120; the benchmark lives outside python/, where neither the python workflow's lint job nor the benchmarks workflow looks, so it had been left at black's default. One wrapped call, unwrapped. `--check` still agrees with Rust.
`prettier --check .` runs over js/, and it writes emphasis as `_text_`. The nested-contexts section added in 7867587 used `*text*`, so the js workflow's format check would have failed on this branch; the Russian section added here inherited the same style.
The Java READMEs handed out 0.1.0 in their Maven and Gradle snippets and the PHP ones "^0.1" in Composer, sixteen minors behind the 0.17.0 every implementation declares, so a reader who copied them installed a release from 2025. Correcting the four literals alone would only reset the clock, so scripts/version-consistency.mjs — already a hard failure in CI, already reading the declared versions — now also reads every version typed into an installation snippet and fails when one disagrees with its implementation. Verified by reverting one snippet to 0.1.0 and watching the check exit 1.
"the nine English READMEs" was not a count of anything: the root plus the seven implementations is eight, and counting the two experiment READMEs makes ten. Stated as what it is now that Go has a Russian README — eight and eight.
konard
marked this pull request as ready for review
September 5, 2026 12:59
Member
Author
Working session summaryDone. PR #210 is out of draft and green: #210 Closing out this session
CI: all 16 workflows pass on the head commit One thing I deliberately left alone, since it needs a CI-matrix decision rather than a doc edit: the This summary was automatically extracted from the AI working session output. |
Member
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $55.616002📊 Context and tokens usage:Claude Opus 5: (11 sub-sessions)
Total: (25.3K new + 1.1M cache writes + 62.9M cache reads) input tokens, 539.7K output tokens, $55.616002 cost 🤖 Models used:
📎 Log file uploaded as Gist (20563KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Member
Author
✅ Ready to mergeThis pull request is now ready to be merged:
Monitored by hive-mind with --auto-restart-until-mergeable flag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #209 — how much of a model's context each format spends on the same information.
Links Notation is measured against JSON (indented and compact), YAML, XML and CSV over 11 seeded datasets covering the shapes real payloads have: uniform records, semi-uniform records, nested records, deeply nested configuration, keyed maps, sequences and tuples. The report is generated, not written:
.github/workflows/benchmarks.ymlregeneratesbenchmarks/BENCHMARK_RESULTS.mdon every push tomainwith the defaultGITHUB_TOKENand commits it only when it changes and the run succeeded.Per the issue, TOON is used as a reference for method only and is not one of the compared formats.
Headline result
Totals over all 11 datasets,
o200k_base(GPT-5 / GPT-4.1 / GPT-4o):Compact JSON (8004) is smaller than all of them and CSV smaller still. Both are carried in the report as reference points rather than quietly omitted, because a benchmark that only shows the comparisons it wins is not evidence.
Links Notation appears three times because a writer has a real choice: quote every string (what
lino-objects-codecwrites today), quote only where a bare word would read back as something else (the rule YAML plain scalars follow, so the like-for-like row), or stay on one line. All three decode back to the same value.What makes the numbers checkable
Each of these fails the build rather than warning:
benchmarks/tools/generate-datasets.mjswritesdatasets/*.jsonfrom seeded generators; the Rust benchmark derives every other representation from those. Nothing ingenerated/is hand-written, so no format can silently carry less information than another..linodocument and the compact JSON are decoded back and compared with the source value before a number is reported.benchmarks/tools/verify-representations.mjsre-reads every generated document withyaml,fast-xml-parser,csv-parseandlino-objects-codec— libraries with no stake in the outcome..linowith its own implementation and counts it with its own tokenizer. The seven files inresults/may differ only in which language wrote them.Point 4 is also why both encodings are OpenAI BPE:
o200k_baseandcl100k_baseare the only two with a tokenizer in all seven languages, and cross-language agreement is what keeps the numbers honest. The report states this alongside its other limits — no accuracy measurement, no surrounding prompt, no speed — under "What this does not measure".Beyond the benchmark
The review comment on this PR asked that the change reach the whole codebase and every supported language.
TEST_CASE_COMPARISON.mdnow cover Rust, JavaScript, Python, C#, Go, Java and PHP. Every command accepts--check, which is what CI runs.CHANGELOG.md.docs/comparison/comparison.linois the input;docs/comparison/generate.mjsproduces the SVGs and PNGs, and--checkfails if a committed asset differs from what the code produces. No more hand-edited SVG.lino-objects-codec, and issue Indentation is structural at the root but ignored inside ( ), so parenthesised groups cannot express nested records #282's nested-context semantics documented in all eight English READMEs and all eight Russian ones — the root plus every implementation, Go having had no Russian README at all — and on the website.go/example_readme_test.goruns them undergo test; that is how a documentedlino.StrPtrthat the package never exported was caught.Bugs found while verifying, and fixed here
links-notation0.1.0report::markdownreceivedenv!("CARGO_PKG_VERSION")frommain.rs, which expands to the benchmark crate's versionrust/links-notation/src/lib.rsexposespub const VERSION; the benchmark reports thatdocs/website/index.htmlcarried a hand-typed literal whilescript.jsalso wrote the same span from__LIBRARY_VERSION__^0.1scripts/version-consistency.mjsnow checks every documented install version against the version its implementation declaresgo/README.mddocumentedlino.StrPtrgo/example_readme_test.gomakes this class of drift a test failureexperiments/issue-209/lino-shapeswould not build..short;cargo metadataexited 101../../../rust/links-notationexperiments/issue-282/parity/run.shreported a false disagreementNETSDK1045and counted as a disagreement instead of a skipdotnet --list-sdkswhat it can target and skips, as the PHP check already didWebsite
Regenerated and rebuilt from the working tree; the header now reads
v0.17.0because it comes fromjs/package.jsonat build time.How to reproduce
Each of the other six languages has the same
--checkentry point;.github/workflows/benchmarks.ymlruns all seven and fails on any disagreement.Test plan
--checkmodescripts/version-consistency.mjs: all seven declare 0.17.0, and every documented install version matchesdotnet format, actionlint, zizmor, shellcheckThe evidence behind the above is committed at
dev/log/issues/209/pulls/210/README.md.