Repository navigation
Edit: reach readings record the bytes they read, as spans and looks - #520
Conversation
The engine names no element kinds: the lookahead that tells the line leaves the block open inside the chain block asks for a child of the kind that block already holds, not for a paragraph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The reopen kept the content marks whose source fell before the rest point, but marks read inside a nested block hold sources relative to that block, so they never compared past it: a nested list read again on every edit grew line_marks without bound (22 MB after 2000 edits of the 16k list fixture, 11x a fresh parse). Every map the flush made is complete, so the reopen now truncates to the identity run. The fuzz runner reads each input twice: once as appends, as before, and once switching to edits at the first head byte with the high bit set, so the edit oracle sees random inputs too. Edits whose ends fall after a continuation byte at a line start are skipped, since such positions are ambiguous in UTF-16. An api test makes a thousand edits to a small nested list and checks the session holds no more than after the first hundred; it fails without the fix (4x growth). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
… part 2) The compaction of plan 12.4 (E-D4). The pool keeps the slots each reading lets go for the next, so it holds what the most text the session held needed. The session keeps that peak, and an edit that leaves less than half of it reads the whole text: it drops the pool's released slots and slab first, so the new nodes take new slabs, and again after, so the old slabs go with the old nodes. The input's lines, the content marks, the line facts and the edit's scratch then shrink to the text (markdown_core_parser_compact, markdown_core_array_shrink). Ids stay, as for any whole reading. On the 16k list fixture an edit that deletes 90% of the text now leaves 1.35x the storage of a fresh parse, down from 7.6x. An api test checks 2x after such an edit, and fails without the compaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Three cases where a snapshot settled what a longer input reads otherwise, found by the edit-aware fuzz runner (4.6, then 4.1 at the flush): - A `$` pair whose body a code span begins makes no formula, but the pairing still took the delimiters between its two away, so a reading from a settle point between them paired a `*` the whole reading drops. The rejected pair now holds the run undecided from its opener (markdown_core_inline_state_undecide). - An attribute block whose quote the input's end leaves open, or closes at the very end, reads that quote as an unquoted value's byte; the input going on undoes it. The recogniser now says it ran out (`ran_out`), and the run reports the end read as it does for a block that does not close. - A topic's name reads the character after it whole, up to four bytes, but reported an end read only within three of the end. It now reports within four, and five after a `/`. Each case is a source of session_flush_matches_a_parse, which fails without its fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…tes it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…t openers A formula scanner asks how many `$` openers are unmatched to read a `$` as a closer. An opener that an emphasis pair around it takes away still counted at those scans, so a reading from past the emphasis, without the opener, could read a later `$` as an opener instead. The core now notes the rules whose scanners ask that count, and the start of a removed opener of such a rule is an undecided start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The wire's markdown_core_wire_session_edit answers an edit's document as a step's message. Swift's MarkdownSession.edit(from:to:with:), and Kotlin's and ECMAScript's edit(start, end, text), replace the text between two cursors in the session's unit and return the parse of the new text. An edit before the flush throws the new code sessionStreaming (SESSION_STREAMING); one out of bounds or backwards throws outOfBounds and leaves the context. Each binding sweeps an insertion and its undo over every cursor of a text and checks each document against a fresh parse, and checks that a node the edit did not reach is the old node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…s for Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Document.content holds existential markup, which == cannot compare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…step E3) didOpen parses the text into the document's session and flushes it, each didChange content change is one session edit, and completion locates the session at its cursor. The server holds no copy of the text: the new markdown_core_session_source lends the session's. A change whose scope is backwards is refused before any edit applies; an edit that cannot allocate closes the document. The sync oracle now compares the session's document with a fresh parse of its text after every change. Reading the session's exact-sized text showed a context query reading one byte past the text at a destination the text's end opens, such as `[a](`; it now reads only the text, with a facade test on an exact-sized buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…p 4b) MarkdownCoreLSP.Server wraps markdown-core-lsp: initialize, shutdown, phase, open, update, close and complete, with the protocol's Position, Scope and Change, Capabilities, Completions and ServerError. The C engine builds from source as the MarkdownCoreLSPC target. The conformance tests replay specs/lsp/transcripts through the server, bundled by the resource plugin, and the release manifest, source archive and product build carry the second product. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Kotlin's com.nouprax.markdown.core.lsp.Server and the ECMAScript entry point @nouprax/es-markdown-core/lsp join MarkdownCoreLSP.Server: one method per protocol method of markdown-core-lsp, with Position, Scope, Edit, Capabilities, Completions and an error carrying the protocol's code. Kotlin's JNI library and cinterop definition link the engine, whose natives register on the parser's JNI holder; the WebAssembly module compiles and exports it. A content change is now an Edit in Swift too, by the naming rule. Each binding's tests replay specs/lsp/transcripts in process: Kotlin from a generated common-test source, ECMAScript from the bundled conformance fixtures. The source-list and public-surface audits, the JVM surface record, the ABI dumps and the npm packaging test know the new sources and entry point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A new benchmark job, "Measure - completion latency", runs the stream benchmark's edit scripts through the language engine as a client typing them would: completion_runner opens a server on the document and sends each edit as didChange, and a measured request is one edit followed by a completion at the end of its text, both inside bench_request. The driver, scripts/benchmark/completion.mjs, reads each request's inclusive Ir from the callgrind call graph, split into the edit and the completion, and groups the requests by document size (grammar corpus, shapes at 16, 64 and 256 KiB) next to the plan's proposed budget. Report only (D5). A script is measured at up to 16 evenly spaced edits; a shape larger than 16 KiB is cut to its first 16 edits and measured at 4. Scripts whose edit splits a \r\n are skipped, since no protocol position names that place. The runner checks that the session ends in the script's text. The PR comment shows the size table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
An edit's splice cut, shifted and unlabeled one table at a time, so a reference definition whose label fell to a later heading looked the heading up in old-text terms and lost the label. Take and shift every table first, then move the labels on. A sweep of whole-line edits over texts whose labels cross tables checks the trees and tables against a fresh parse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Headings prepare runs its loop only when a heading waits, a tree of one leaf erases in that leaf, and an erase of nothing returns inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
An insertion that joins the front at the end appends in place, a cut of the last leaf's tail leaves the leaf and lowers only the counts above it, a one-leaf tree inserts in its leaf, a label table looks a label up once, and headings in order are read from their stretch of storage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The first leaf of a tree starts at eight entries and doubles to a full leaf while it is the only one, taken anew and copied so small blocks come from the allocator's caches. Definition tables search the leaf of the front's last entry directly, the uses flush walks runs, and withdraw skips tables whose front ends before the root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A first leaf that grows from eight entries costs every small document its growth in the parse stage; the leaf is taken at its full size, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Line places, the dropped facts and a leap's places step through the line records leaf by leaf, forward with markdown_core_tree_run and back with the new markdown_core_tree_back, instead of finding each line by rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The snapshot and the locate's look begin, read and end through the same S_look_begin, S_look_read and S_look_end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
record_nest records the nodes of a reopened definition's old bodies, and each record owns its node from then on; the body itself was never released. Two edits of a definition list that drop a body (a blank line removed before a term, then put back) left its node, and its slab, alive past the session's close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A session that still streams, and a document, locate through a session of their whole text, flushed, whose look reads at the cursor (THE LOOK OF A TEXT AS A WHOLE): the parse of the text before the cursor, with labels seeded from the whole document, is gone, and context_at always has a locus. A flushed session with an empty root reads its look with no window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
An edit on the line right after a paragraph's last line -- typing the first character of a new line at the paragraph's end -- rested at the paragraph's start and read it again whole, so every new line of a long paragraph cost as much as the paragraph. The edit now rests in the paragraph from its last line (THE LEAP), as an edit of one of its own lines does, and the reading decides whether the edited line goes on with it. Typing at the end of a 200-line paragraph: 180k -> 76k Ir per edit, the same as at 100 lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
An edit at a paragraph's end now parses its inline content from as late as a snapshot does, instead of from a seam two lines back. - The seam margin is 0: a seam may lie at the rest line's start. When the content no longer reaches past the picked seam (a delimiter row took its head, an underline, an interrupting line), S_seam_shrink falls back to the last seam the content still reaches past instead of reading the paragraph whole. - Settle starts inside the content's last line are INNER seams. They stand only for an edit from where the old content ended on, and only while the new content keeps all of the old (S_seam_holds), the same condition the snapshot's settle point relies on. - A paragraph carrying an element flag (a table's TABLE_VISITED) no longer leaps: its reopened node kept the flag the old reading set on a later line, so a line edited into a valid delimiter row made no table. This was wrong on main too. Test: a paragraph typed at its end, and lines written at its end, dump as a full parse after every edit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Every reach record now ends past the last byte its reading read, and one that met the text's end ends past it: - a block's close ends past the line ending of the line that closed it (markdown_core_parser_read_past); the splice notes its closes on the old line it resumes at (S_close_on, S_line_past); - every lookahead is noted, the one that read only the next line too; - link definitions, block identifiers, the properties envelope and the definition list term's peek note their true ends. Records are of two kinds. A span is a block's reading held open across a point: closes, and a paragraph open from before its definitions. A look read ahead and decided. One rest rule for edits and locates: a reading from before the rest point that crosses it stands only when it is a look that ended no later than the edit's start or the cursor, or a span of a block of the chain. The splice resumes on a line no reading read past. The paragraph +1 and first-child +1 rules, and the cut-front special case, go: what they covered is in the reach now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d43c51e622
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (lookahead->cursor && lookahead->active && !lookahead->waited) { | ||
| end = (size_t)(lookahead->cursor - source); | ||
| } else { | ||
| end++; |
There was a problem hiding this comment.
Distinguish container rejection from reaching EOF
When a lookahead stops because the next line does not carry its container prefix, S_lookahead_next sets active = false even though it has not reached EOF. This condition consequently records a reach through the entire document, forcing unrelated later edits to reparse from the rejected lookahead's origin. For example, opening > term\n: def\n\n followed by 1,000 para\n\n paragraphs and last, then inserting into last, rereads bytes 0–6019; the parent commit rereads only 6008–6019. This makes otherwise local edits scale with the entire remaining document. Preserve the cursor-based end for container rejection and extend the reach past EOF only when the lookahead actually encountered EOF.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 05d5745: the lookahead now records ended only when it met the input's end. A line a container rejected reaches past that line's terminator, or past the text's end only when it is the unterminated last line. Added hard-quote-lookahead-reach (quote term/def + 20 paragraphs; typing in last rereads [134, 140)).
Generated by Claude Code
…the text's end Readings track whether they met the input's end; only those reach past it. Audit fix: the reach ends ask markdown_core_parser_input_text for the size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05d5745660
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| *undecided = !line && !blank_lines && (lookahead.exhausted || lookahead.waited); | ||
| *reached = lookahead.cursor; | ||
| /* Past the line it read, or the text's end it met. */ | ||
| *reached = line ? lookahead.cursor : NULL; |
There was a problem hiding this comment.
Preserve identifier lookahead reach on container rejection
When the next line lacks its container prefix, markdown_core_parser_lookahead_next returns false without reaching EOF. Setting reached to NULL then makes S_reach record a dependency through the entire document, so unrelated later edits reparse from the identifier's owner. Reproduced with > - item\n>\n> #id#\npara\n\n, followed by 1,000 para\n\n paragraphs and last: inserting into last rereads bytes 0–6029, whereas the parent rereads only 6018–6029. The corrected ended handling in S_reach_lookahead does not cover this separate identifier-specific path. Preserve the cursor-based reach for container rejection and reserve the past-EOF reach for an actual encounter with the input's end.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in ebde4b1: the identifier's reading reaches past the text's end only when it met or waits at the input's end (or read an unterminated last line); a line a container rejected keeps the cursor. Added hard-identifier-lookahead-reach (your shape with 20 paragraphs; typing in last rereads [144, 150)).
Generated by Claude Code
The reread expectations of hard-type-in-middle-paragraph and the new quote-lookahead case are written by scripts/benchmark/workloads.mjs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…line Only a reading that met or waits at the input's end reaches past it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
BenchmarkCommit: Benchmark is required when CI inputs require execution. This ordinary PR comment does not create a review thread to resolve. Parse stagesBaseline:
Totals sum this finite workload; they are not elapsed time or a general speedup claim. Parser creation and release are not parsing and are in no figure here. Stage budget (+2% per document and stage): 1,584/1,584 passed, 0 exceeded. Required when CI inputs require execution. Largest stage ratios (up to 10 document stages)
Corpus: Grammar-equivalent reference comparisonsA = Core on dialect input; B = Core on common input; R = cmark/cmark-gfm on common input. Ir covers the two parse stages. Whole means the declared language; local excludes the unmatched host. Each input is measured once. No AST equivalence or equal native output cost is asserted.
Rejection of constructs only Core implementsNo reference implements these constructs, so none is measured. B = Core on the common input; C = Core on its control, the same input with the rejected construct's trigger bytes replaced by letters of the same width.
Attribute grammar
Core / lexbor instructions on the same recovered attributes: ASCII 0.7064×, UTF-8 0.7571×. StreamsSubjects: Stream regressions (+2% per workload and subject in p95, max and total, against
Bindings' C side (B): the MCB3 encode of each document an S window publishes and the release of its message, without building the platform values; no gate.
Edits: Ir per window of edits, R counting the two parse stages of the
Completion latencyCallgrind Ir per completion request: the didChange of one edit and the completion at the end of its text. At most 8 requests a script, at evenly spaced edits. The budget is the plan's proposed p95 wall time on a device, not compared with these counts; no gate.
|
A reading from before the line that stopped inside it was the opener of the line's first block (a list's variant from the list before); one that read the line whole, a lookahead or a close, decided its own line's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
|
Stream gate 6.4: one figure is over the 1.02x limit. It is The cause is the reach records. They now hold true bytes, so the two-line lookaheads they note for blank lines and feeds are larger. The owner's rule is to pay reach costs the architecture requires and report them, so I am leaving this unfixed. Merging needs the owner to accept the figure. Generated by Claude Code |
Part of item 3 of the no-flush plan (
no-flush.md§9.5 and §10). Every reach record is now exact, and edit and locate share one rest rule.True ends
Every reach record ends past the last byte its reading read. A reading that met the text's end ends past the end.
markdown_core_parser_read_past). When the splice closes blocks on the old line it resumes at, it notes those closes as that line's old reading did (S_close_on,S_line_past).Spans and looks
Each reach record is one of two kinds; a span carries a high bit on its end (
MARKDOWN_CORE_REACH_SPAN).Edits and locates use one rule (
markdown_core_parser_crossedgainsfrom). A reading from before the rest point that crosses it stands only in two cases: it is a look that ended no later than the edit's start or the cursor, or it is the span of a block on the chain.The paragraph "+1" rule, the first-child "+1" rule and the cut-front special case are removed; the reach now covers what they covered.
Tests
The targeted differential suites pass locally on a Debug build: incremental_hard, incremental_typing, incremental_lines, incremental_random_1 and api_engine.
Two reread expectations narrowed, both still matching a fresh parse. Typing in the middle paragraph of
one\n\ntwo\n\nthreenow rereads from 5 instead of 0, because the definition-list peek fromoneread onlytwo's first byte. The changes are inhard-type-in-middle-paragraph.scriptandtests/api/main.c.Benchmark (end-of-text edit Ir, 8 documents)
The other five documents are within ±1.3%. These numbers were measured before the span/look split and its locate change. I have not re-measured since.
Follow-up, pending owner review: §10 of the plan moves this state onto the tree itself, as node properties plus a children index.
Generated by Claude Code