perf: lex block tokens only and reuse parsed blocks while streaming

[withkarann]

Description

parseMarkdownIntoBlocks runs on every streamed token. Two things made it slower than it needs to be:

1. It called Lexer.lex, which also runs marked's inline tokenizer. Only the raw block text is used, so it now calls blockTokens directly.
2. It lexed the whole document every time. While a response streams, text is only appended, so blocks before the last safe boundary are reused and only the rest is lexed again.

The boundary needs care. A block that ends with a blank line can still absorb the next block when that block's first line changes (2 is a paragraph, 2. is a list item), and a lone # ends the paragraph above it while #x continues it. So blocks are reused only up to the last one that ends with a blank line and is followed by at least two more blocks.

Type of Change

• Performance improvement

Changes Made

• lib/parse-blocks.tsx: call blockTokens instead of Lexer.lex; keep the last parse in one cache entry and re-lex only the tail when the input extends the previous input. Anything else falls back to a full parse.
• __tests__/parse-blocks-incremental.test.tsx: streams 40 tricky documents plus 400 seeded random ones, one, five, and varying characters at a time. Every prefix must equal a fresh full parse.
• __benchmarks__/parse-blocks.bench.ts: adds a case that streams text after a long document.
• Changeset (patch).

Testing

• All existing tests pass
• Added new tests for the changes
• Manually tested the changes

pnpm vitest bench --run __benchmarks__/parse-blocks.bench.ts, main vs this branch, Node 24 on Apple Silicon:

case	main (ops/s)	branch (ops/s)	change
single block	857,680	2,241,046	2.6x
many blocks (100)	8,175	18,267	2.2x
large table (100 rows)	4,979	16,209	3.3x
streaming text (50 steps)	3,217	8,273	2.6x
streaming after a long document (30 steps, new)	230	2,188	9.5x
realistic AI response	41,019	91,799	2.2x
multiple code blocks	1,361,710	1,392,361	1.0x

Code-only and HTML-only inputs have no inline content, so they do not change. The React re-render bench moves from 135 to 147 ops/s because it is dominated by React work.

Checklist

• My code follows the project's code style
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have created a changeset (pnpm changeset)

Additional Notes

The cache is a single module-level entry. Two documents streaming at once fall back to full parses, which is the current behavior.

One known difference: marked drops a duplicate link definition (a second [x]: url for the same label) from a full parse, so the streamed tail keeps text the full parse loses. Neither renders anything for it.

[vercel]

@withkarann is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.