Repository navigation
One look for the snapshot and every locate (unify step 2) - #517
Conversation
The reach of the readings past where an edit puts the parse back stands in the edit's back, measured from the text's end, as its lines do: a splice drops what the reading read again and notes nothing anew. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
What an edit puts aside of the registry and of the headings is measured from the text's end, as its lines and reach are, so the splice takes it back in the new text's terms and no delta crosses the element hooks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Nothing reads a published node's place again: the old children an edit keeps or takes back are measured from where the child before the old tail ends, by their extents, for the reading's candidates, its splice, its records and its publish, so the splice no longer walks the tail to move the places. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The FINAL prefix's edit check made GCC 15 stop inlining publish_relation_begin into publish_matched, a per-step cost on the stream session path. It is always inlined now, as its neighbours are. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The widths and the reread range are made at a session's first edit, so a session that is only streamed keeps the size it had: the 32 bytes they added moved the session into another malloc size class, and the heap's layout after it, which cost some stream windows 3%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…, stage 1)
## What
Before this change, an edit inside a list item, callout or footnote
reread the whole root child that held it: one edit near the end of a
long list read the entire list again. Now the reading starts at a rest
point inside that container, at any depth. It reopens the chain of
containers from the root's child down to the rest point. Then it reads
on from there until the top-level resync (E1a) takes the old blocks
back.
Plan: `e1b-nested.md`, stage E1b-1. Resync and splice at depth is the
next stage.
## How
- **Nested rest point** (`session_rest_in`, session.c). The search
descends while the held child is a container whose element sets
`reopens` and the edited line is past its first line. At each level it
picks the last child that:
- begins after a blank (or prefix-only) line;
- has a gap before its previous sibling, so no lazy line or split
reference paragraph runs into it;
- has no reading that started before the chain spanning its start.
Child windows come from `markdown_core_node_window`.
- **Reopen the chain** (`S_reopen_chain`, blocks.c). Each chain node
goes OPEN again, and its kept children are folded again. The new
`reopen_block` hook lets an element undo what its close decided. Here
that is the list's looseness. The engine still names no kinds: the
hook and the `reopens` flag come from the elements. Elements opt in
with `reopens`: list, callout, footnote and specimen. Tables are left
out.
- **The blank state before the rest point** (`markdown_core_parser_rest_blank`).
It tells each chain level whether the line before the rest point was
blank to it, so the deepest node's last-line-blank flag matches a
fresh parse.
- **The reach log.** It keeps each entry's raw end and builds a separate
prefix maximum (`furthest`) for the span queries. This lets the root
chain node's own close entry be found and dropped at reopen (it closes
again where the reading closes it). `markdown_core_parser_spanned_inside`
finds readings that start inside the chain.
- **Publish** (ast.c). Each chain node is recorded as open under its
owner, with its kept children FINAL. Definitions in a chain node keep
their place in the table (`publish_chain_rank`), so nested footnotes
list in source order.
Hot paths are unchanged. Everything new runs at edit time or behind
`edit->depth`.
## Results
- The differential fuzz compares against a fresh `Document.parse`:
20,000 container-heavy documents × 20 edits, plus 10,000 mixed ones.
All match. The rest points it hit were at depths 1 to 4.
- Bytes reread per edit over `specs/incremental`, before → after:
- list-ascii-16k: 11,694 → 5,833.
- list-utf8-16k: 12,125 → 6,196.
- Every other family is the same or lower. The whole set averages
10,719 → 10,466.
- The incremental oracle suites and the api_incremental tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
- A container with one child (a one-item list, a quote holding one list) is now descended too: the search needs a child to go into, not two. - A reopened root child closed by its own end condition (a directive's closer) notes its close at its last line, as one the input's end closes does, so that entry is the chain's own close too. - Definition lists and directives reopen (`reopens`), as the plan's E1b-1 lists them. A definition's body is not published and has no window, so no rest point is looked for below one. Bytes reread per edit over specs/incremental: quote-16k 15.9k -> 10.8k, opener-directive-16k 16.3k -> 10.3k, nameless-directives 213 -> 164; the whole set 10,466 -> 9,891 (main: 10,719). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…pth (step E1b, stage 2) A reading reopened inside a container now resyncs at the child the chain let go, at any level, and splices the rest of the old children back: each block of the chain closes as it closed before, with the flags its children fold refolded. Lists whose items abut (`abuts`) rest and resync between adjacent items and at their first item. `trailed` becomes per-kind (`trailed_kinds`), so list items no longer block a resync. The `hard` incremental family now runs on the session subject. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The engine names no element kinds: the lookahead that tells the line leaves the block open inside the chain block asks for a child of the kind that block already holds, not for a paragraph. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The reopen kept the content marks whose source fell before the rest point, but marks read inside a nested block hold sources relative to that block, so they never compared past it: a nested list read again on every edit grew line_marks without bound (22 MB after 2000 edits of the 16k list fixture, 11x a fresh parse). Every map the flush made is complete, so the reopen now truncates to the identity run. The fuzz runner reads each input twice: once as appends, as before, and once switching to edits at the first head byte with the high bit set, so the edit oracle sees random inputs too. Edits whose ends fall after a continuation byte at a line start are skipped, since such positions are ambiguous in UTF-16. An api test makes a thousand edits to a small nested list and checks the session holds no more than after the first hundred; it fails without the fix (4x growth). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
… part 2) The compaction of plan 12.4 (E-D4). The pool keeps the slots each reading lets go for the next, so it holds what the most text the session held needed. The session keeps that peak, and an edit that leaves less than half of it reads the whole text: it drops the pool's released slots and slab first, so the new nodes take new slabs, and again after, so the old slabs go with the old nodes. The input's lines, the content marks, the line facts and the edit's scratch then shrink to the text (markdown_core_parser_compact, markdown_core_array_shrink). Ids stay, as for any whole reading. On the 16k list fixture an edit that deletes 90% of the text now leaves 1.35x the storage of a fresh parse, down from 7.6x. An api test checks 2x after such an edit, and fails without the compaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Three cases where a snapshot settled what a longer input reads otherwise, found by the edit-aware fuzz runner (4.6, then 4.1 at the flush): - A `$` pair whose body a code span begins makes no formula, but the pairing still took the delimiters between its two away, so a reading from a settle point between them paired a `*` the whole reading drops. The rejected pair now holds the run undecided from its opener (markdown_core_inline_state_undecide). - An attribute block whose quote the input's end leaves open, or closes at the very end, reads that quote as an unquoted value's byte; the input going on undoes it. The recogniser now says it ran out (`ran_out`), and the run reports the end read as it does for a block that does not close. - A topic's name reads the character after it whole, up to four bytes, but reported an end read only within three of the end. It now reports within four, and five after a `/`. Each case is a source of session_flush_matches_a_parse, which fails without its fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…tes it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…t openers A formula scanner asks how many `$` openers are unmatched to read a `$` as a closer. An opener that an emphasis pair around it takes away still counted at those scans, so a reading from past the emphasis, without the opener, could read a later `$` as an opener instead. The core now notes the rules whose scanners ask that count, and the start of a removed opener of such a rule is an undecided start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The wire's markdown_core_wire_session_edit answers an edit's document as a step's message. Swift's MarkdownSession.edit(from:to:with:), and Kotlin's and ECMAScript's edit(start, end, text), replace the text between two cursors in the session's unit and return the parse of the new text. An edit before the flush throws the new code sessionStreaming (SESSION_STREAMING); one out of bounds or backwards throws outOfBounds and leaves the context. Each binding sweeps an insertion and its undo over every cursor of a text and checks each document against a fresh parse, and checks that a node the edit did not reach is the old node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…s for Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Document.content holds existential markup, which == cannot compare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…step E3) didOpen parses the text into the document's session and flushes it, each didChange content change is one session edit, and completion locates the session at its cursor. The server holds no copy of the text: the new markdown_core_session_source lends the session's. A change whose scope is backwards is refused before any edit applies; an edit that cannot allocate closes the document. The sync oracle now compares the session's document with a fresh parse of its text after every change. Reading the session's exact-sized text showed a context query reading one byte past the text at a destination the text's end opens, such as `[a](`; it now reads only the text, with a facade test on an exact-sized buffer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…p 4b) MarkdownCoreLSP.Server wraps markdown-core-lsp: initialize, shutdown, phase, open, update, close and complete, with the protocol's Position, Scope and Change, Capabilities, Completions and ServerError. The C engine builds from source as the MarkdownCoreLSPC target. The conformance tests replay specs/lsp/transcripts through the server, bundled by the resource plugin, and the release manifest, source archive and product build carry the second product. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A long heading's anchor, and a literal held in two parts, are made at their first read, which can be refused storage after the parse succeeded. The Swift builder read them as calls that cannot fail. It now makes them as it queues each node and throws the refusal from the parse or session step; the records built after read what the engine kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
When an edit shortens a heading that parsed from its seams, undoing the seams keeps its old children while the heading reads as a paragraph; the records named its old id but compared the paragraph's kind, so the heading's content lost its ids. The publish now matches the kind it named the node as, as it does the id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A heading named from its projection clears the name once, the key's choice of path inlines into the naming loops, and the two-lane Horner takes four bytes each turn of its loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…given hash Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A heading's node keeps its base (the projection of its text, split after an edit that leaps inside it) and a numeric suffix; its anchor is the base and "-k". One naming model serves every length: the committed pass and an edit's renames read names as pieces and compare them byte for byte, keyed by their measure, so there is no 1024-byte cut and no digest key. - Stems keep no bytes, only a measure and a witness (first member's base, else a child's, else the heir or relic), and are removed by id. - An edit's new bases wait for the edit's end and are written in place (the old base's kept front and back around the new middle). - The public anchor borrows the base when it is whole and unsuffixed, and is otherwise joined on first read. - A heading whose text projects to nothing borrows a static "section". - Roster ANCHOR keys are hashed by measure. Also fixes a locate crash present before this change: a heading's run that reached the text's end, prepared before a later heading's run began, lost the sink root and read NULL when it finished. Only a run that reaches the end now replaces the root. The OOM dump sweep parses a fresh document per allocation, since a read keeps the anchors it joins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…ader refuse Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…names as bytes The measure is now a polynomial in a generator G of the residues modulo 2^61 - 1, read through byte tables, so no repeating pattern of byte changes cancels; the Codex family of colliding headings no longer shares a measure. The committed naming pass, which reads every base whole, files names under a byte hash read in parts (markdown_core_key_stream), so a base's hash is read once for all its suffixes; edits keep the measure. Holders whose bases are of the name's base size are compared by suffix and one comparison of bases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…enerator Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
…er for every heading Swapping the projection's storage into the node left the pass a buffer to grow again from nothing for the next heading, several reallocations per heading; a copy is one allocation, as the joined anchor was before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
core/tree.{h,c} is one B+ tree of fixed-size entries (leaves of 64,
fanout 16) with a focus that splits it into a front in new text terms
and a back in old text terms, pending shifts, optional end measures
(tree_past), key ordering for label trees, and handles kept current by
a `moved` callback. It replaces the zipper and every hand-rolled index:
- input lines (facts dropped as lines move to the back), reach, uses
- committed tables (listed entries by id -> handle) and their labels
- roster places (record id -> handle)
- headings and heading-name stems
- session child widths
- the label key tree (map.c)
core/zipper.h is gone. Document reads use the pure tree_peek and
tree_seek, so concurrent read-only access stays safe.
The stream driver keeps its per-line cost within the stage gate: leaves
are not zeroed, the next line's start is read from the terminator, an
edit's line insert is a cold path, properties walk lines in runs
(tree_run), and a new child takes its offset from the line at hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
An edit's splice cut, shifted and unlabeled one table at a time, so a reference definition whose label fell to a later heading looked the heading up in old-text terms and lost the label. Take and shift every table first, then move the labels on. A sweep of whole-line edits over texts whose labels cross tables checks the trees and tables against a fresh parse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Headings prepare runs its loop only when a heading waits, a tree of one leaf erases in that leaf, and an erase of nothing returns inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
An insertion that joins the front at the end appends in place, a cut of the last leaf's tail leaves the leaf and lowers only the counts above it, a one-leaf tree inserts in its leaf, a label table looks a label up once, and headings in order are read from their stretch of storage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The first leaf of a tree starts at eight entries and doubles to a full leaf while it is the only one, taken anew and copied so small blocks come from the allocator's caches. Definition tables search the leaf of the front's last entry directly, the uses flush walks runs, and withdraw skips tables whose front ends before the root. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A first leaf that grows from eight entries costs every small document its growth in the parse stage; the leaf is taken at its full size, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Line places, the dropped facts and a leap's places step through the line records leaf by leaf, forward with markdown_core_tree_run and back with the new markdown_core_tree_back, instead of finding each line by rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
The snapshot and the locate's look begin, read and end through the same S_look_begin, S_look_read and S_look_end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
record_nest records the nodes of a reopened definition's old bodies, and each record owns its node from then on; the body itself was never released. Two edits of a definition list that drop a body (a blank line removed before a term, then put back) left its node, and its slab, alive past the session's close. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
A session that still streams, and a document, locate through a session of their whole text, flushed, whose look reads at the cursor (THE LOOK OF A TEXT AS A WHOLE): the parse of the text before the cursor, with labels seeded from the whole document, is gone, and context_at always has a locus. A flushed session with an empty root reads its look with no window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvzArumvqU8RFP8mWYPXpj
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
BenchmarkCommit: Benchmark is required when CI inputs require execution. This ordinary PR comment does not create a review thread to resolve. Parse stagesBaseline:
Totals sum this finite workload; they are not elapsed time or a general speedup claim. Parser creation and release are not parsing and are in no figure here. Stage budget (+2% per document and stage): 1,584/1,584 passed, 0 exceeded. Required when CI inputs require execution. Largest stage ratios (up to 10 document stages)
Corpus: Grammar-equivalent reference comparisonsA = Core on dialect input; B = Core on common input; R = cmark/cmark-gfm on common input. Ir covers the two parse stages. Whole means the declared language; local excludes the unmatched host. Each input is measured once. No AST equivalence or equal native output cost is asserted.
Rejection of constructs only Core implementsNo reference implements these constructs, so none is measured. B = Core on the common input; C = Core on its control, the same input with the rejected construct's trigger bytes replaced by letters of the same width.
Attribute grammar
Core / lexbor instructions on the same recovered attributes: ASCII 0.7064×, UTF-8 0.7571×. StreamsSubjects: Stream regressions (+2% per workload and subject in p95, max and total, against
Bindings' C side (B): the MCB3 encode of each document an S window publishes and the release of its message, without building the platform values; no gate.
Edits: Ir per window of edits, R counting the two parse stages of the
Completion latencyCallgrind Ir per completion request: the didChange of one edit and the completion at the end of its text. At most 8 requests a script, at evenly spaced edits. The budget is the plan's proposed p95 wall time on a device, not compared with these counts; no gate.
|
Step 2 of the unified read plan (one code path for stream, edit and LSP).
Changes
markdown_core_parser_snapshot(stream) andmarkdown_core_parser_look(flushed locate) now begin, read and end through the sameS_look_begin/S_look_read/S_look_end.markdown_core_document_contextnow locate through a temporary session of the whole text. That session is flushed, then its look reads at the cursor.markdown_core_parse_before_cursorandseed_labelsare deleted (they re-parsed the text before the cursor and seeded labels from the whole document).context_atnow always has a locus.markdown_core_context_locateis replaced bymarkdown_core_context_read/markdown_core_context_take, so a failed locate leaves the old context unchanged.record_nestrecords the nodes of a reopened definition's old bodies, but it never released the body node itself.incremental_undo_2andincremental_random_1..3caught it. The body is now released once its nodes are recorded.Testing
Generated by Claude Code