fix: write text as text in both readable forms instead of base64 - #46
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #45
A single control character makes encode and encode_line replace the whole string with base64, so a log message holding a newline stops being readable and greppable, and a key holding one silently turns its object into an array. See #45
encode() and encode_line() replaced a whole string with a base64 payload as soon as it held one control character, so a multi-line log message stopped being greppable. They now write the text itself: a value using the quote delimiter is wrapped in a run of delimiters the notation reads literally, a newline stays raw in the indented form, and only characters the chosen form cannot carry are percent-escaped under a new (escaped "...") marker. base64 stays where it was asked for, in encode_compact()/encode_obfuscated(), and (base64 "...") is still read. The shared fixtures gain the cases the issue asks for -- a multi-line string, strings holding the quote delimiter, a repeated value, a real traceback -- and a legacy section pinning that documents written up to 0.6.0 keep decoding.
Port the Rust change: the readable forms write a string as text, escape only the characters the form itself cannot carry, and quote with a run of delimiters instead of doubling them. `(escaped "...")` replaces the `(base64 "...")` marker, which stays readable for older documents.
encode() and encode_line() replaced a whole string with a base64 payload as soon as it held one control character, so a multi-line log message stopped being greppable. They now write the text itself: a value using the quote delimiter is wrapped in a run of delimiters the notation reads literally, a newline stays raw in the indented form, and only characters the chosen form cannot carry are percent-escaped under a new (escaped "...") marker. base64 stays where it was asked for, in encode_compact()/encode_obfuscated(), and (base64 "...") is still read. A key holding a control character now stays a key rather than turning the dict it belongs to into a list, and the conformance suite gains the legacy section that pins documents written up to 0.6.0 as still decodable.
The issue reports that parse_lino desynchronises on the doubled-quote form, so 241 of 6,000 fuzzed values came back as something else. The new test encodes twelve values -- quotes of both kinds, leading and trailing quotes, runs of quotes -- and asserts links-notation reads each one back as exactly the text that was written.
A single control character used to turn a whole string into base64, so a log message holding one newline hid its own text: the message, the stack trace and every word a reader would grep for. Both readable forms now write the text as it is and escape only what the form itself cannot carry -- the newline on a single line, the carriage return everywhere, and the remaining control characters -- with a new (escaped "...") marker whose payload is percent-escaped, so the escaped part stays readable too. A value holding the quote delimiter is written with a run of delimiters rather than doubled quotes, which the notation's own parser reads back unchanged. (base64 "...") is still decoded, so documents written up to 0.6.0 keep reading, and a key holding a control character now stays a key instead of turning its object into an array.
Every README said a value holding a control character is base64-encoded. It is not: both readable forms write the text as it is, use a run of delimiters for a value holding the quote, percent-escape only what the form cannot carry, and write a repeated value out every time. base64 is now described where it still happens -- the compact form, which asks for it by name. Also re-exports ESCAPED_MARKER from the JavaScript entry point, which Python, Rust and C# already expose, and adds the release notes for all four packages.
konard
changed the title
Working session summaryAll non-success checks are PR: #46 — ready for review, mergeable, all 9 workflow runs green on What was wrong
An unrelated bug surfaced while fixing it: a key holding a control character was written as What changed, in all four languagesBoth readable forms write text as text. Only what the form itself cannot carry is percent-escaped, in a Verification
One finding worth flagging: the four packages pin different This summary was automatically extracted from the AI working session output. |
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $22.087043📊 Context and tokens usage:Claude Opus 5: (6 sub-sessions)
Total: (17.0K new + 567.7K cache writes + 20.8M cache reads) input tokens, 236.2K output tokens, $22.087043 cost 🤖 Models used:
📎 Log file uploaded as Gist (7180KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
🎉 Auto-mergedThis pull request has been automatically merged by hive-mind.
Auto-merged by hive-mind with --auto-merge flag |
1 participant
Closes #45.
The problem
encode_line(andencode) treated every control character as unwritable, soone newline turned the whole string into base64:
The message, the stack trace and every word a reader would
grepfor went withit. In a real log that is most of the text.
The doubled-quote escape used for a value holding
"had the same effect for adifferent reason: it desynchronises the notation's own parser, so the value came
back as something else.
Both are reproduced by the new suites before anything else changed.
The change
Applied identically to Rust, JavaScript, Python and C#:
encodeandencode_linenever reach for base64. It stays reachableonly through
encode_compact()/encode_obfuscated(), which say so by name.holds the delimiter:
"""say "hi"""". The Rust suite proves the result readsback unchanged through
links_notation::parse_lino_to_links.encode.encode_lineonly, the newline — and nothing else — is escaped:(escaped "line one%0Aline two"). The payload is percent-escaped, so eventhe escaped part stays readable and greppable.
reference, so a log line never depends on another line.
quote delimiter, a repeated value, a tab, a carriage return, a control
character, a percent sign, a key holding a newline — 52 cases plus a
legacysection, byte-identical across the four languages.
Which characters each form escapes:
encodeencode_line%0A%0D%0D%XX%XXFixed along the way
A key holding a control character used to be written as
(base64 "…")in keyposition and read back as an array element, so
{"a\nb": "a\nb"}silentlydecoded to
["a\nb", "a\nb"]. Keys are now decoded as keys in all fourlanguages.
Backward compatibility
(base64 "…")is still decoded, and the two older quoting conventions (a run ofone delimiter meaning doubled-quote escaping, a run of two meaning the empty
value) still read. The
legacyfixture section pins this in all four suites, soevery document written by an earlier version keeps decoding.
Reproducing and verifying
rust/tests/plain_text_values.rs,js/tests/test_plain_text_values.test.js,python/tests/test_plain_text_values.py,csharp/tests/.../PlainTextValuesTests.cs— the same 9–10 tests in eachlanguage, written against the behaviour the issue asks for, failing before the
fix.
experiments/issue-45/quote-probe— the measurement behindquote(): whichdelimiter run each
links-notationrelease actually reads back unchanged.experiments/issue-45/README.mdrecords the result, including that the fourpackages pin different notation releases with different quoting rules (the
readable format is read by this repository's own tokenizer in every language,
so a value round-trips regardless).
All four suites pass locally: Rust 8 targets + clippy clean, JavaScript 497
tests +
npm run check, Python 488 tests + ruff + mypy, C# 507 tests +dotnet format --verify-no-changes+/warnaserror.Release notes are added for all four packages (
minorin each).