Read a # line as a comment in every parser, and write a leading # quoted - #304
Merged
Conversation
Adding .gitkeep for PR creation (default mode). This file will be removed when the task is complete. Issue: #301
Member
Author
🚨 Solution Draft FailedThe automated solution draft encountered an error: 🤖 Models used:
📎 Failure log uploaded as Gist (8377KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Member
Author
|
🤖 AI Work Session Started Starting automated work session at 2026-09-05T21:57:41.534Z The PR has been converted to draft mode while work is in progress. This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback. |
Comments are blanked before parsing, so every later character keeps its offset and a parse error still points at the original document. A parser can be told to treat # as an ordinary character with ParserConfig::without_comments(). Part of #301.
The parser blanks comments before handing the document to the generated
parser, so a parse error still points at the position in the document the
caller wrote. A line holding nothing but spaces now separates links the
way an empty line does, which is what a blanked comment line leaves
behind. new Parser({ comments: false }) reads # as an ordinary character.
Part of #301.
The quote scanner moves into a module of its own so that the comment stripper can ask it how far a delimited reference reaches. A line holding nothing no longer closes an indented block, which is what a blanked comment line leaves behind. Parser(comments=False) reads # as an ordinary character. Part of #301.
A formatter that writes `(a #tag)` writes a document that no longer reads back as itself: the `#` opens a comment, so the second reference is gone by the time the document is parsed again. A `#` anywhere else in a reference cannot open a comment (`issue#1047`), so only the first character decides. All seven implementations now quote such a reference, each with a test that fails without the fix.
The rule belongs next to the one it mirrors, so it is now in the Comments section of all sixteen READMEs, in the grammar overview and in the changelog.
A minor bump rather than a patch: every parser gains a `comments` option and reads `#` comments by default, which is new public API in all seven languages. All seven declare 0.19.0, so scripts/version-consistency.mjs passes, including the installation snippets in the Java and PHP READMEs. The manifests that name the library by version rather than by path follow: rust/links-notation-benchmark/Cargo.toml, benchmarks/java/pom.xml and the private website package, and the benchmark report footer names the version it measured. Refs #301
The generated comparison groups tests by the name of the file they live in, so the Rust suite, named `comment_tests.rs`, stood in a category of its own while the other six shared `Comments`. Rename it, and give the tests that differed by name alone the name their counterparts carry. The Python extractor matched `def test_...` only at the start of a line, so tests written as methods of a `Test...` class were invisible to it, and whole files, the comment suite among them, never reached the comparison. Allow the indentation a method carries.
konard
changed the title
# prose lines parse by accident and break on a bare colon# line as a comment in every parser, and write a leading # quoted
konard
marked this pull request as ready for review
September 5, 2026 23:18
Clippy 1.98, which the workflow runs, reads an array of byte characters as a byte string written the long way (`clippy::byte_char_slices`) and the lint is denied there.
Member
Author
Working session summaryPR #304 is finished and ready for review: #304 What landed since the last checkpoint:
CI on One note on my earlier local verification: This summary was automatically extracted from the AI working session output. |
Member
Author
🤖 Solution Draft LogThis log file contains the complete execution trace of the AI solution draft process. 💰 Cost: $32.572985📊 Context and tokens usage:Claude Opus 5: (7 sub-sessions)
Total: (17.8K new + 665.5K cache writes + 37.3M cache reads) input tokens, 286.4K output tokens, $32.572986 cost 🤖 Models used:
📎 Log file uploaded as Gist (11722KB)Now working session is ended, feel free to review and add any feedback on the solution draft. |
Member
Author
🎉 Auto-mergedThis pull request has been automatically merged by hive-mind.
Auto-merged by hive-mind with --auto-merge flag |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #301 —
#opens a line comment in all seven implementations, on by default, and can be turned off.The document from the issue is prose that a parser used to read as a link:
Now both are comments and hold no links, and a document a formatter writes reads back as itself.
The issue asked whether comments should exist at all. The decision on it — "We should have an option to disable comments in parsers, and by default we should support comments with
#. As it is a single character. That also will be an advantage over JSON." — is what this PR implements: comments are read by default everywhere, and a parser can be asked to treat#as an ordinary character again.Root cause
The notation had no comment syntax, so nothing ever skipped a line of prose.
#was simply an ordinary character in a reference, which is why# a bparsed by accident — as the three references#,aandb— and why one more character broke it. The four hand-written parsers were even quieter about it: they read# a: bas('# a': b), silently turning the prose into a link identifier.So the bug was not in how the colon was parsed. It was that prose was being parsed at all.
The rule
A
#opens a comment when it stands at the start of the document or after a space, a tab or a line break. The comment runs to the end of its line. Everywhere else a#is content:# a: ba: b # why(a: b)issue#1047issue#1047— a#inside a tokena: b#c(a: b#c)— a#that opens a token"# not a comment" a# not a commentanda— a#inside a delimited referenceparent\n # what the child is for\n childparentwith the single childchildReading a comment inside a delimited reference correctly means knowing how far that reference reaches, which is not something a regular expression can tell: any run of N delimiters opens and closes a reference and 2N of them stand for N delimiters written as content. So the scanner that reads a delimited reference was lifted out of each parser (
Quotesin C#,quotes.pyin Python, and their counterparts elsewhere) and is now shared with the comment stripper — the stripper reaches into a reference exactly as far as the parser does, by construction rather than by resemblance.Comments are blanked, not removed
Each implementation replaces the characters of a comment with spaces instead of cutting them out, so every character that follows keeps the offset, line and column it was written at. That is what keeps the positions from #302 honest:
still reports
Syntax error at line 2, column 12, with the offending line quoted under a caret, and not a position shifted by the length of the comment.blanking a comment keeps the length of the documentis a test in all seven suites.A blanked comment leaves a line of spaces behind, so a line of spaces now separates links the way an empty line does — also a test in all seven suites.
Turning comments off
A document written before comments existed can still be read:
parse_lino_with_config(source, &ParserConfig::without_comments())new Parser({ comments: false })Parser(comments=False)p := NewParser(); p.Comments = falsenew Parser(false)new Parser(comments: false)new Parser(10 * 1024 * 1024, 1000, false)rust/links-notation/examples/comments.rsshows both settings on one document.Writing a document back
A parser that skips comments makes the formatter's job stricter:
(a #tag)written unquoted no longer reads back as itself —#tagopens a comment, and reading the document again givesaalone, a syntax error or a different link, depending on the language. Every escaper now quotes a reference that begins with a#:This is the half of the change that is easy to miss, so it has its own tests in all seven suites (
a reference that begins with a hash is written quoted,a hash that cannot open a comment is left unquoted). Note that Rust'sDisplay for LiNodoes not escape at all — a pre-existing asymmetry, not touched here — so the Rust test goes throughformat_links_with_config, which is the path that escapes.Tests
A conformance suite shared by all seven implementations, written before the fix and watched failing:
The three parsers that report positions also assert that a comment does not move a later error, and Rust and JavaScript assert that a parser without comments still rejects the document from the issue.
Local runs, all green:
cargo test+cargo fmt --check+cargo clippy -D warnings,dotnet test --configuration Release(229/229) +dotnet format --verify-no-changes,bun test(237) +bun run lint+ Prettier,pytest(214 passed, 1 skipped) + Black + isort + flake8,go test ./...+gofmt -l,mvn test+spotless:check. PHPUnit runs in CI only — composer requires PHP >= 8.4 and this machine has 8.3.scripts/create-test-case-comparison.mjsmatcheddef test_...only at the start of a line, so Python tests written as methods of aTest...class were invisible to it and whole files never reached the comparison. With that fixed the Python count in the README goes from 146 to 215 — the 21 new comment tests plus 48 class-based tests that had been hidden all along — and the comment suites of all seven languages now sit in one## Commentscategory.Documentation
Every README (8 languages × 2 translations),
docs/grammar/GRAMMAR.md,docs/grammar/grammar.lino,docs/grammar/links-notation.ebnf,docs/grammar/syntax-diagrams.mdandCHANGELOG.mddescribe comments, the option that turns them off, and the quoting rule the formatter follows.Known divergence, left alone
With comments turned off, Python, Go, Java and PHP read
# a: bas('# a': b)while Rust, JavaScript and C# reject it. That is about whether an identifier may hold a space before a colon — it predates this PR and is the same looseness catalogued in #302 and #138. It is orthogonal to comments, so it is not fixed here; the shared suite asserts the rejection only in the two implementations that already reject.How to reproduce and verify
Asks all seven implementations about the same six documents and prints the answers next to each other; toolchains that are not installed are reported as skipped.
experiments/issue-301/README.mdrecords what the run says before and after.Release
Every implementation is bumped to 0.19.0 (
node scripts/version-consistency.mjs— "All 7 implementations declare 0.19.0"), so merging releases the change.