Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion benches/js/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1619,7 +1619,10 @@ CWD-relative).
`loc` from `start`/`end` (UTF-16 offsets) + source via the ECMAScript (TS) / LF-only
(Svelte) line rules, and assert equality. TS is 100% exact; the two Svelte non-derivable
cases (the `<script>` `Program` tag-position override, the destructure `+1`-column quirk)
are classified, not failed. Exits 1 on any unexplained mismatch. The reference
are classified, not failed. Svelte's `name_loc` is gated the same way — the name span is
derived from each node's own `start`/`end` + type (tag run after `<`, attribute name at
the node, directive head token) and both it and its line/column must match the oracle.
Exits 1 on any unexplained mismatch. The reference
reconstruction a `no-locations` consumer would use. Run:
`deno run --allow-ffi --allow-read --allow-env --allow-net --allow-sys benches/js/diagnostics/no_locations_parity.ts`
- `diagnostics/reconstruct_vs_materialize.ts` — the **perf** sibling of the parity
Expand Down
142 changes: 94 additions & 48 deletions benches/js/diagnostics/no_locations_parity.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@
* offsets + source, and asserts they match — so a consumer holding only the
* no-locations wire can recover exact acorn/svelte `loc` on demand.
*
* Svelte's `name_loc` gets the same treatment. Its name span isn't a node
* `start`/`end`, but it is a function of them plus the node type — the tag-name run
* after `<`, an attribute name at the node start (a shorthand `{x}` naming its
* braces interior), a directive's whole head token — so this file derives the span
* and gates both it and its line/column against the oracle wire.
*
* Line rules (mirrored from `tsv_lang::LocationTracker`):
* - TypeScript / `.ts`: ECMAScript LineTerminators — \n, \r, \r\n (ONE), U+2028,
* U+2029 (`new_ecmascript_with_map`).
Expand Down Expand Up @@ -71,40 +77,73 @@ interface Tally {
mismatch: number;
name_loc_exact: number;
name_loc_mismatch: number;
prefix_ok: number;
prefix_off: number;
name_span_exact: number;
name_span_mismatch: number;
}

// Svelte name offset = node.start + a fixed per-node-type prefix (the STRETCH
// claim: the name span is derivable even without name_loc). Prefix = the bytes
// before the name: `<` for elements, the directive keyword + `:`, `{` for shorthand.
const NAME_PREFIX: Record<string, number> = {
RegularElement: 1,
Component: 1,
SvelteElement: 1,
SvelteComponent: 1,
SvelteSelf: 1,
SvelteWindow: 1,
SvelteDocument: 1,
SvelteBody: 1,
SvelteHead: 1,
SvelteFragment: 1,
SvelteBoundary: 1,
TitleElement: 1,
SlotElement: 1,
Attribute: 0,
ShorthandAttribute: 1,
OnDirective: 3, // on:
BindDirective: 5, // bind:
ClassDirective: 6, // class:
StyleDirective: 6, // style:
UseDirective: 4, // use:
TransitionDirective: 11, // transition:
InDirective: 3, // in:
OutDirective: 4, // out:
AnimateDirective: 8, // animate:
LetDirective: 4, // let:
};
// The Svelte name span is derivable from the node's own start/end + type, so the
// no-locations wire keeps `name_loc` recoverable too. Re-derived here rather than
// imported from the shipped helper (crates/tsv_wasm/npm/locations.js) — this file
// is the independent oracle that gates it.

/** Node types whose name is the tag-name run right after `<`. */
const ELEMENT_NAME_TYPES = new Set([
'RegularElement',
'Component',
'SvelteHead',
'SvelteWindow',
'SvelteBody',
'SvelteDocument',
'SvelteElement',
'SvelteComponent',
'SvelteSelf',
'SlotElement',
'SvelteFragment',
'SvelteBoundary',
'TitleElement',
]);

/** Node types whose name span is the whole directive head (`on:click|preventDefault`). */
const DIRECTIVE_NAME_TYPES = new Set([
'OnDirective',
'BindDirective',
'ClassDirective',
'StyleDirective',
'UseDirective',
'TransitionDirective', // `in:`/`out:` too — Svelte has no In/OutDirective type
'AnimateDirective',
'LetDirective',
]);

/**
* The chars that end an attribute/directive name run — tsv's parse of Svelte's
* `read_tag` (`/[\s=/>"']/`), ASCII whitespace only.
*/
const NAME_TERMINATORS = ' \t\n\r\v\f=/>"\'';

/** The `[start, end]` offsets a node's `name_loc` covers, or null if it carries none. */
function name_span_of(node: Record<string, unknown>, source: string): [number, number] | null {
const { type, name, start, end } = node as {
type: string;
name?: string;
start?: number;
end?: number;
};
if (typeof name !== 'string' || typeof start !== 'number' || typeof end !== 'number') return null;
if (ELEMENT_NAME_TYPES.has(type)) return [start + 1, start + 1 + name.length];
if (DIRECTIVE_NAME_TYPES.has(type)) {
let head_end = start;
while (head_end < end && !NAME_TERMINATORS.includes(source[head_end])) head_end++;
return [start, head_end];
}
if (type === 'Attribute') {
// a shorthand `{x}` names its braces interior, padding included (`{ x }`) —
// unlike a `<script>` attribute, whose literal name can itself be braced
if (source[start] === '{' && !source.startsWith(name, start)) return [start + 1, end - 1];
return [start, start + name.length];
}
return null;
}

function check_node(
node: Record<string, unknown>,
Expand Down Expand Up @@ -140,24 +179,31 @@ function check_node(
}
// Svelte name_loc: its `character` sub-field is the name offset; line/column
// must reconstruct from it exactly (the spine carries no pattern quirk).
const nl = node.name_loc as { start?: { line: number; column: number; character: number } } | undefined;
if (nl?.start) {
const nl = node.name_loc as
| {
start?: { line: number; column: number; character: number };
end?: { line: number; column: number; character: number };
}
| undefined;
if (nl?.start && nl.end) {
const got = loc_at(nl.start.character, starts);
if (got.line === nl.start.line && got.column === nl.start.column) t.name_loc_exact++;
else t.name_loc_mismatch++;
// Report-only heuristic: the name offset ≈ node.start + a per-type prefix.
// Approximate (some node spans lead with a space, so this is off by one) —
// a consumer needing the exact name span in the no-loc wire recovers it by
// searching the name string within [start,end], not by a fixed prefix. Not
// a gate; just surfaces how close the simple rule gets.
const prefix = NAME_PREFIX[node.type as string];
if (prefix !== undefined && typeof node.start === 'number' && node.start + prefix === nl.start.character) {
t.prefix_ok++;
// The name span itself — the part a no-loc consumer has to derive, since the
// wire drops `name_loc` whole. Per-type rule (tag name after `<`, attribute
// name at the node, directive head run), gated like `loc`.
const span = name_span_of(node, source);
if (span && span[0] === nl.start.character && span[1] === nl.end.character) {
t.name_span_exact++;
} else {
t.prefix_off++;
t.name_span_mismatch++;
if (t.name_span_mismatch <= 5) {
console.error(
` name span mismatch ${node.type as string} @${node.start}: got ${span ? `[${span[0]},${span[1]}]` : 'null'} want [${nl.start.character},${nl.end.character}]`,
);
}
}
}
void source;
}

function walk(value: unknown, starts: number[], is_svelte: boolean, source: string, t: Tally): void {
Expand Down Expand Up @@ -187,8 +233,8 @@ for (const language of ['typescript', 'svelte'] as Language[]) {
mismatch: 0,
name_loc_exact: 0,
name_loc_mismatch: 0,
prefix_ok: 0,
prefix_off: 0,
name_span_exact: 0,
name_span_mismatch: 0,
};
let checked = 0;
for (const f of by_lang[language] ?? []) {
Expand All @@ -215,10 +261,10 @@ for (const language of ['typescript', 'svelte'] as Language[]) {
);
if (is_svelte) {
console.error(
` name_loc line/col: exact ${t.name_loc_exact}, MISMATCH ${t.name_loc_mismatch}; name-offset≈start+prefix (report-only): ok ${t.prefix_ok}, off ${t.prefix_off}`,
` name_loc: line/col exact ${t.name_loc_exact}, MISMATCH ${t.name_loc_mismatch}; name span exact ${t.name_span_exact}, MISMATCH ${t.name_span_mismatch}`,
);
}
if (t.mismatch > 0 || t.name_loc_mismatch > 0) any_mismatch = true;
if (t.mismatch > 0 || t.name_loc_mismatch > 0 || t.name_span_mismatch > 0) any_mismatch = true;
}

if (any_mismatch) {
Expand Down
5 changes: 3 additions & 2 deletions benches/js/diagnostics/reconstruct_vs_materialize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@
*
* TypeScript reconstruction is EXACT (pure offset → line/col); Svelte is
* APPROXIMATE (the `<script>` `Program` tag-position override + destructure
* `+1`-column quirk + dropped `name_loc` — see `no_locations_parity.ts`), so its
* `+1`-column quirk + the in-tag comment `character` field — `name_loc` itself
* reconstructs exactly; see `no_locations_parity.ts`), so its
* B path is still a valid *perf* measurement but its output isn't Svelte's exact
* `loc`. Both are reported; the headline "reconstruct beats materialize" number is
* the exact TS one.
Expand Down Expand Up @@ -148,7 +149,7 @@ for (const language of ['typescript', 'svelte'] as Language[]) {
if (!t || t.files === 0) continue;
const exactness = language === 'typescript'
? 'EXACT reconstruction'
: 'APPROXIMATE — omits name_loc + 2 quirks; see no_locations_parity.ts';
: 'APPROXIMATE — 2 quirks + in-tag comment shape; see no_locations_parity.ts';
console.log(`\n${language} (${exactness}), ${t.files} files:`);
console.log(` A full wire, loc materialized in Rust : ${t.a.toFixed(2)}`);
console.log(
Expand Down
2 changes: 1 addition & 1 deletion crates/tsv_svelte/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ See [../../CLAUDE.md §Project Structure](../../CLAUDE.md#project-structure) for
- `parse(source, arena: &'arena Bump) -> Result<Root<'arena>>` — internal AST, bump-arena-allocated (caller owns the `Bump`). The Svelte parser creates the one arena per document and shares it with every embedded sub-AST — `tsv_ts` (`<script>` / `{expr}`) and `tsv_css` (`<style>`) — so the whole component is one bump-allocated graph. See [docs/architecture.md §Nested AST](../../docs/architecture.md#nested-ast-bump-arena-not-flatindexed).
- `format(&Root, source) -> String` — round-trips through the doc builder; `format_in(&Root, source, &DocArena)` is the same into a caller-provided doc arena (multi-file drivers reuse one via `DocArena::reset()`; embedded `<style>` shares this host arena via `tsv_css::format_embedded_in`, not a second per-block one — the CSS still renders to its own string, only doc-node storage is shared)
- `convert_ast_json_bytes(...) -> Vec<u8>`, `convert_ast_json_string(...) -> String`, and `convert_ast_json(...) -> serde_json::Value` — the public JSON AST (gated on `feature = "convert"`). The bytes variant is the **sole emission path** (FFI/CLI non-pretty; the string variant adds one output UTF-8 validation for `&str` boundaries — WASM `JSON.parse`, N-API): the writer (`ast/convert/write.rs`) walks the *internal* Svelte AST once and emits the wire JSON directly, never materializing a typed public tree, fusing byte→UTF-16 offset translation into the walk (per-language pipeline shapes: [docs/architecture.md §Closed Scope, Open Convention](../../docs/architecture.md#closed-scope-open-convention)). The Svelte spine (elements, blocks, tags, directives, attributes, `name_loc`) emits fused; embedded `<script>`/`{expr}` route through `tsv_ts`'s embedded writers and `<style>` children through `tsv_css`'s `write_css_node`. Template-expression comments (and `<script>` comments) fuse via an island-scoped attach: each comment-bearing island's wire node tree is recorded during a byte-space skeleton emit (`tsv_ts`'s `SkeletonRecorder`, driven by `ast/convert/special.rs`'s `build_*_writer_comments` — no re-parse of the emitted bytes), the acorn attach machinery in `ast/convert/comment_attachment.rs` walks the recorded tree, and the assignments fold into a span-keyed `WriterComments` the fused writer consults at each node's close (so `leadingComments`/`trailingComments` serialize in place). `convert_ast_json` is a thin wrapper (`serde_json::from_slice(&convert_ast_json_bytes(...))`) for the `Value` consumers (the CLI's `--pretty`, the fixture gate); not an independent conversion. Gated against the canonical Svelte parser's `expected.json`, including the multibyte and template-comment fixtures that exercise the fused offset translation and island-scoped attach.
- `convert_ast_json_bytes_no_locations(...)` / `convert_ast_json_string_no_locations(...)` — the opt-in **span-only** variant: drops *every* line/column object from the Svelte wire — the acorn `loc` on `<script>`/`{expr}` nodes (via `emit_loc` threaded into the embedded `tsv_ts` writers), the root-comment `loc`, **and the element/attribute/directive `name_loc`**. Only `start`/`end` offsets remain; because that removes all line/column emission, the LF line table is never queried. A name's exact span reconstructs as `node.start + a fixed per-node-type prefix` (`<`→+1, plain attribute→+0, directive→its keyword+`:` length, shorthand→+1) plus source. Mirrors acorn's `locations: false`; a distinct narrower product, not a change to the drop-in wire (which is byte-identical). Invariant: `tests/no_locations.rs`.
- `convert_ast_json_bytes_no_locations(...)` / `convert_ast_json_string_no_locations(...)` — the opt-in **span-only** variant: drops *every* line/column object from the Svelte wire — the acorn `loc` on `<script>`/`{expr}` nodes (via `emit_loc` threaded into the embedded `tsv_ts` writers), the root-comment `loc`, **and the element/attribute/directive `name_loc`**. Only `start`/`end` offsets remain; because that removes all line/column emission, the LF line table is never queried. `name_loc` stays derivable from a node's own `start`/`end` + type: an element names the tag run at `start + 1`, an attribute names `[start, start + name.len]` (a shorthand `{x}` its braces interior, padding included), and a directive its whole head token — the run from `start` to the first name terminator (`on:click|preventDefault`, not just `click`). The shipped `reconstruct_locations` helper (`crates/tsv_wasm/npm/locations.js`) restores it on that basis; `benches/js/diagnostics/no_locations_parity.ts` gates the derivation against the loc-bearing wire. Mirrors acorn's `locations: false`; a distinct narrower product, not a change to the drop-in wire (which is byte-identical). Invariant: `tests/no_locations.rs`.
- `script_content_spans(&Root) -> Vec<(u32, u32)>` — byte spans of the instance/module `<script>` contents (gated on `feature = "convert"`). The writer uses it to partition comments into template-expression comments (outside the spans) vs `<script>` comments

## Distinctives
Expand Down
9 changes: 6 additions & 3 deletions crates/tsv_wasm/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,12 @@ rules: `reconstruct_locations(ast, source, opts?)` (one-shot, adds `loc` to ever
node, **mutates in place**), `create_locator(source, opts?)` (amortized — holds
the prebuilt line table, exposes `loc_of(node)` / `reconstruct(ast)`), and a bare
`loc_of(node, source, opts?)` convenience. **Exact for TypeScript**; **approximate
for Svelte** (doesn't recover `name_loc`, doesn't replicate the `<script>`
tag-position or destructure `+1`-column parser quirks, and adds `loc` to template
nodes Svelte's own wire omits); **a no-op for CSS**. It rides the **parse-capable**
for Svelte** (doesn't replicate the `<script>` tag-position or destructure
`+1`-column parser quirks, leaves a root in-tag comment without its `character`
field, and adds `loc` to template nodes Svelte's own wire omits — but `name_loc`
is restored exactly, its span derived from each node's own `start`/`end` + type,
as is the name-shaped `loc` on shorthand-attribute identifiers, snippet names, and
simple-identifier block patterns); **a no-op for CSS**. It rides the **parse-capable**
packages only (`@fuzdev/tsv_parse_wasm`, `@fuzdev/tsv_wasm`) — it operates on the
parse wire, so the format-only package has no use for it. `patch_npm_package.ts`
copies it + the hand-written `npm/locations.d.ts` into the package root and
Expand Down
2 changes: 1 addition & 1 deletion crates/tsv_wasm/README_all.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ const root: Root = parse_svelte('<script>const x = 1;</script>');

Three formatters (`format_svelte`, `format_typescript`, `format_css`) take a source `string` and return the formatted `string`. Three parsers (`parse_svelte`, `parse_typescript`, `parse_css`) return a Svelte-compatible JSON AST; the `parse_*_json` variants return the AST as a compact JSON string instead (faster when writing to disk or the wire). For TypeScript and Svelte, `parse_{typescript,svelte}_no_locations` (and `_json_no_locations`) emit a **span-only** wire — `start`/`end` offsets, no per-node `loc` (Svelte also no `name_loc`) — ~46% smaller and faster to materialize, with line/column derivable from offsets + source. All throw on a parse error.

To turn a span-only wire back into a loc-bearing one, `reconstruct_locations(ast, source)` adds `loc` to every node (mutating in place; `structuredClone` first to keep the input) — **exact for TypeScript** (each node's `loc` value equals acorn's; the key is appended last, so an object consumer matches but a re-serialized tree won't byte-match the wire's key order), **approximate for Svelte** (no `name_loc`, and it skips Svelte's `<script>`/destructure position quirks), a no-op for CSS. For sparse lookups, `create_locator(source, opts?)` reuses one line table across `loc_of(node)` / `reconstruct(ast)` calls (pass `{language: 'svelte'}` for `.svelte`); a bare `loc_of(node, source)` is also exported.
To turn a span-only wire back into a loc-bearing one, `reconstruct_locations(ast, source)` adds `loc` to every node — and the Svelte `name_loc` — (mutating in place; `structuredClone` first to keep the input) — **exact for TypeScript** (each node's `loc` value equals acorn's; the key is appended last, so an object consumer matches but a re-serialized tree won't byte-match the wire's key order), **approximate for Svelte** (it skips Svelte's `<script>`/destructure position quirks and a root in-tag comment's `character` field; `name_loc` itself is exact), a no-op for CSS. For sparse lookups, `create_locator(source, opts?)` reuses one line table across `loc_of(node)` / `reconstruct(ast)` calls (pass `{language: 'svelte'}` for `.svelte`); a bare `loc_of(node, source)` is also exported.

AST types are bundled in `tsv_ast.d.ts` and re-exported from the package — `import type` any node directly.

Expand Down
2 changes: 1 addition & 1 deletion crates/tsv_wasm/README_parse.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ const ast = reconstruct_locations(parse_typescript_no_locations(src), src);
// every node now carries loc: {start: {line, column}, end: {line, column}}
```

`reconstruct_locations(ast, source)` walks the tree and adds `loc` to every node, **mutating in place** and returning it (`structuredClone(ast)` first if you need the input untouched). It's **exact for TypeScript** — each node's `loc` value equals acorn's exactly (the key is appended last, so an object consumer matches but a re-serialized tree won't byte-match the wire's key order) — and **approximate for Svelte**: it doesn't recover `name_loc` and doesn't replicate Svelte's `<script>` tag-position or destructure `+1`-column parser quirks; everything else reconstructs exactly. CSS is a no-op (there's no `loc` to rebuild).
`reconstruct_locations(ast, source)` walks the tree and adds `loc` to every node, **mutating in place** and returning it (`structuredClone(ast)` first if you need the input untouched). It's **exact for TypeScript** — each node's `loc` value equals acorn's exactly (the key is appended last, so an object consumer matches but a re-serialized tree won't byte-match the wire's key order) — and **approximate for Svelte**: it doesn't replicate Svelte's `<script>` tag-position or destructure `+1`-column parser quirks, and a root in-tag comment (`<div /* c */ class="x">`) reconstructs without the `character` field Svelte's wire gives it. Everything else reconstructs exactly, including the `name_loc` on elements, attributes, and directives — and the name-shaped `loc` Svelte reports on a shorthand attribute's identifier, a snippet name, and a simple-identifier block pattern. CSS is a no-op (there's no `loc` to rebuild).

For sparse or repeated lookups, `create_locator(source, opts?)` holds the prebuilt line-start table and exposes `loc_of(node)` and `reconstruct(ast)`; pass `{language: 'svelte'}` for a `.svelte` document (LF-only line rule). A one-shot `loc_of(node, source)` is also exported for the occasional single lookup (it rebuilds the table each call).

Expand Down
Loading