diff --git a/CLAUDE.md b/CLAUDE.md index ce21910..6565c84 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,82 +4,89 @@ Agent guidance for `codellm-devkit/codeanalyzer-typescript` (`cants`). ## What this project is -`cants` is a TypeScript/JavaScript static analyzer built on the TypeScript compiler -(via [ts-morph](https://ts-morph.com/)). It is the CLDK TypeScript backend: it emits -the **canonical schema v2** — one additive Code Property Graph — in **two projections**, -`analysis.json` and a **Neo4j** property graph. It mirrors its +`cants` = TypeScript/JavaScript static analyzer built on TypeScript compiler +(via [ts-morph](https://ts-morph.com/)). CLDK TypeScript backend: emits +**canonical schema v2** — one additive Code Property Graph — in **two projections**, +`analysis.json` and **Neo4j** property graph. Mirrors [Python](https://github.com/codellm-devkit/codeanalyzer-python) and [Java](https://github.com/codellm-devkit/codeanalyzer-java) sibling analyzers, so -output-shape parity with them is a first-class concern. +output-shape parity with them first-class concern. ## Schema v2 — the additive CPG (read this before touching output) -The output is **one scale-free structure**: a containment tree of nodes (id / kind / -`span` / children) with **typed edge overlays** (a CPG). Every classic artifact — symbol -table, call graph, CFG, PDG, SDG — is a *projection* of that one structure, and analysis -**levels** are how deeply it is populated (each level only ever *adds*, never rewrites): +Output = **one scale-free structure**: containment tree of nodes (id / kind / +`span` / children) with **typed edge overlays** (CPG). Every classic artifact — symbol +table, call graph, CFG, PDG, SDG — is *projection* of that one structure. Analysis +**levels** = how deep it populated (each level only *adds*, never rewrites): -- **L1** (`-a 1`): the tree to callable depth — `application → symbol_table{module} → +- **L1** (`-a 1`): tree to callable depth — `application → symbol_table{module} → types{}/functions{}/fields{} → callables{}` — plus `call` nodes in each callable's - `body{}` (with `callee` unresolved). `source` is stored once per module; every node's + `body{}` (`callee` unresolved). `source` stored once per module; every node's text slices off it via `span.bytes`. -- **L2** (`-a 2`): the `call_graph` edge list (callable→callable) at the application scope, - and the `callee` slot on each call node refined `null → id` (the one sanctioned mutation). -- **L3** (`-a 3`): the rest of `body{}` (statements + `@entry`/`@exit`) and the intra-callable +- **L2** (`-a 2`): `call_graph` edge list (callable→callable) at application scope, + and `callee` slot on each call node refined `null → id` (only sanctioned mutation). +- **L3** (`-a 3`): rest of `body{}` (statements + `@entry`/`@exit`) and intra-callable edge lists `cfg`/`cdg`/`ddg` (reaching-definitions, `prov:["reaching-defs"]`) hung on each callable. -- **L4** (`-a 4`): the synthetic `@formal_in:N`/`@formal_out`/`/actual_in:N`/`/actual_out` - vertices, the intra-caller `summary` edges, and the application-scope `param_in`/`param_out` - lists (the interprocedural SDG). +- **L4** (`-a 4`): synthetic `@formal_in:N`/`@formal_out`/`/actual_in:N`/`/actual_out` + vertices, intra-caller `summary` edges, and application-scope `param_in`/`param_out` + lists (interprocedural SDG). -**Identity is two-tier**: durable `can://////` ids at callable +**Identity two-tier**: durable `can://////` ids at callable depth and above; ordinal `@:` (or `@`) below. Intra-callable edge lists use **bare local ids**; cross-callable lists use **fully-qualified `can://…@local`** -ids. `L1 ⊆ L2 ⊆ L3 ⊆ L4` is a CI-checkable monotonicity gate (`test/schema-v2.test.ts`). The -model + every decision live in `.claude/SCHEMA_DECISIONS.md` (§ "Schema v2 migration") and the +ids. `L1 ⊆ L2 ⊆ L3 ⊆ L4` = CI-checkable monotonicity gate (`test/schema-v2.test.ts`). +Model + every decision live in `.claude/SCHEMA_DECISIONS.md` (§ "Schema v2 migration") and skillset's `canonical-schema.md`. -**Provider/client boundary:** the analyzer is a *pure graph provider* — it emits the graph -substrate (CFG/PDG/SDG + `summary` edges) and stops. Slicing and taint are reachability -*queries* over it and belong to the frontend SDK; never add a `taint_flows` section here. - -The v1→v2 emitter is a pure transform in **`src/schema/v2/`** (`emit.ts` reshapes the v1 -in-memory model, `dataflow.ts` maps `program_graphs` into the tree) — the parse/resolve/ -dataflow *compute* is untouched; only serialization is v2. - -The call graph defaults to the **union** of two backends: the TS compiler's resolver -and the embedded [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer (which -recovers higher-order/callback edges the resolver misses). Merged edges keep a +**Provider/client boundary:** analyzer = *pure graph provider* — emits graph +substrate (CFG/PDG/SDG + `summary` edges) and stops. Slicing and taint = reachability +*queries* over it, belong to frontend SDK; never add `taint_flows` section here. + +Schema v2 = **native model** (#96): stages build v2 tree directly (`src/schema/schema.ts`, +one model family — no v1 model, no emit-time reshape). Per-run passes stamp derived +layers (python parity): `assignIds` (can:// ids — per-run because ids embed `--app-name` +while cache round-trips tree), `l1Body` (`call_sites` → `body{}`), `heritage`, +`homing` + `l2Callees` (L2), `dataflow/attach` (L3/L4). `finalizeAnalysis` +(`src/schema/emit.ts`) runs them + assembles envelope + strips INTERNAL fields +(`call_sites`, `abs_path`, cache trio). + +Call graph defaults to **union** of two backends: TS compiler resolver +and embedded [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer (recovers +higher-order/callback edges resolver misses). Merged edges keep `provenance` tag (`tsc` / `jelly`); `--tsc-only` or `--call-graph-provider jelly` -selects one alone. +picks one alone. ## Architecture — follow the pipeline -The whole analyzer is one orchestration function: `analyze()` in `src/core.ts`. Read -it first; everything else is a stage it calls, in order: +Whole analyzer = one orchestration function: `analyze()` in `src/core.ts`. Read +it first; everything else is stage it calls, in order: -1. **materialize** (`src/build`) — resolve/prepare the target project's deps. +1. **materialize** (`src/build`) — resolve/prepare target project deps. 2. **buildSymbolTable** (`src/syntactic_analysis`) — modules, classes, interfaces, - enums, type aliases, namespaces, functions, methods, variables, decorators, and + enums, type aliases, namespaces, functions, methods, variables, decorators, JSDoc, with precise source spans. 3. **call graph** (`src/semantic_analysis`) — `selectProvider()` picks tsc / jelly / union; each provider returns edges + external (phantom) symbols. 4. **program graphs** (`src/dataflow`) — levels 3–4 (`-a 3`/`-a 4`): CFG → post-dominance/CDG → - access-path def-use → PDG → SCC-condensed bottom-up summaries → SDG. This is the *compute*; - it produces the internal `program_graphs` model, which `src/schema/v2/dataflow.ts` then maps - **into the v2 tree** (`body{}` + `cfg`/`cdg`/`ddg`/`summary` per callable + `param_in`/ - `param_out`). Decisions: `.claude/SCHEMA_DECISIONS.md`; contract + staged follow-ups: issue #2. -5. **cache** (`src/utils/cache.ts`) — content-hash cache under `.codeanalyzer/`, so - re-analysis only touches what changed (levels 3–4 also record summaries + - dependency edges in `graphs_summaries.json`). -6. **output** (`src/schema/v2`, `src/build/neo4j`) — `src/schema/v2/emit.ts` reshapes the v1 - compute model into the schema-v2 `analysis.json`; `src/build/neo4j` projects the *same* v2 - tree into a `graph.cypher` snapshot or an incremental Bolt push. `--emit neo4j` is always - **full-depth** (levels gate the JSON path only; combining `-a`/`--graphs` with it is an error). - -The **output** shape is schema v2 (`src/schema/v2/model.ts`, `V2Application` the top type); the -types in `src/schema` (`TSApplication`) are the *internal compute model* the emitter transforms. -The Neo4j schema (`src/build/neo4j/schema.ts`, v2.0.0) is versioned and enforced by a conformance -test — treat both as contracts and keep them in lockstep with the JSON. + access-path def-use → PDG → SCC-condensed bottom-up summaries → SDG. This is *compute* + (IR in `src/schema/graphs.ts`); `src/dataflow/attach.ts` writes it **onto tree** + (`body{}` + `cfg`/`cdg`/`ddg`/`summary` per callable + `param_in`/`param_out`). + Decisions: `.claude/SCHEMA_DECISIONS.md`; contract + staged follow-ups: issue #2. +5. **cache** (`src/utils/cache.ts`) — content-hash cache under `.codeanalyzer/`; stores + **id-free** builder tree only (ids/body/heritage = per-run layers; levels 3–4 also + record summaries + dependency edges in `graphs_summaries.json`). +6. **finalize + output** — `finalizeAnalysis` (`src/schema/emit.ts`, called by `analyze()`) + runs pass spine, returns `AnalysisResult` {`application` (wire `TSAnalysis` envelope), + `internal`, `program_graphs`, gates}; `src/utils/serialize.ts` writes envelope verbatim; + `src/build/neo4j` projects *same* envelope into `graph.cypher` snapshot or incremental + Bolt push. `--emit neo4j` always **full-depth** (levels gate JSON path only; combining + `-a`/`--graphs` with it = error). + +**Output** shape = schema v2 (`src/schema/schema.ts`: `TSAnalysis` envelope → +`TSApplication` root → `TSModule`/`TSType`/`TSCallable`/`TSField`/`TSBodyNode`). +Same types = the model stages build; INTERNAL fields never reach wire. +Neo4j schema (`src/build/neo4j/schema.ts`) versioned and enforced by conformance +test — treat both as contracts, keep in lockstep with JSON. ## Directory map @@ -90,36 +97,35 @@ test — treat both as contracts and keep them in lockstep with the JSON. | `src/options` | Parsed CLI options / `AnalysisOptions` | | `src/syntactic_analysis` | Symbol table (ts-morph traversal) | | `src/semantic_analysis` | Call-graph providers (tsc, jelly, union), phantoms | -| `src/dataflow` | L3/L4 program-graph **compute**: CFG, dominance/CDG, def-use, summaries, SDG | -| `src/schema` | `TSApplication` — the internal compute model + `signatureOf` + `program_graphs` | -| `src/schema/v2` | **the schema-v2 emitter**: `model.ts` (target shape) + `emit.ts` (tree/L1/L2) + `dataflow.ts` (L3/L4) | +| `src/dataflow` | L3/L4 program-graph **compute** (CFG, dominance/CDG, def-use, summaries, SDG) + `attach.ts` (IR → tree) | +| `src/schema` | **the native v2 model** (`schema.ts`) + per-run passes (`assignIds`/`l1Body`/`heritage`/`homing`/`l2Callees`) + `emit.ts` (`finalizeAnalysis`) + `signatureOf` + graphs IR | | `src/build` | Dep materialization; `build/neo4j` = the v2 graph projection (project/rows/cypher/bolt/schema) | -| `src/utils` | fs, caching, logging, serialization (`serialize.ts` → `toV2`), version | +| `src/utils` | fs, caching, logging, serialization (`serialize.ts` writes the envelope), version | | `test` | Bun tests + `fixtures/sample-app` + `fixtures/dataflow-app`; `schema-v2.test.ts` = the L1–L4 gates | ## Commands -- `bun run start -- --input /path/to/project` — run the analyzer from source. -- `bun run build` — compile the standalone `dist/cants` binary. +- `bun run start -- --input /path/to/project` — run analyzer from source. +- `bun run build` — compile standalone `dist/cants` binary. - `bun test` — run tests. Container tests: `bun run test:container` (needs Docker). - `bun run typecheck` — `tsc --noEmit`. - `bun run gen:schema` — regenerate `schema.neo4j.json`. -- `bun run gen:readme` — regenerate the README's `cants --help` block. +- `bun run gen:readme` — regenerate README's `cants --help` block. ## I implement features myself — you assist For feature work, **I write the implementation** to stay fluent in my own analyzer. -Act as a helper, not the author: +Act as helper, not author: -- **Don't write the feature code** or apply edits to implement it unless I explicitly +- **Don't write feature code** or apply edits to implement it unless I explicitly ask ("write this", "implement X", "apply it"). Default to guiding, not doing. -- **Do** move me fast: explain the relevant stage, point at prior art (e.g. an existing - call-graph provider in `src/semantic_analysis` as the template for a new one), sketch - signatures/types, outline an approach, and answer questions about the codebase. -- **Review on request:** when I share a diff or push, critique it — correctness, - **parity with the Python/Java backends**, schema conformance, missing tests, edge +- **Do** move me fast: explain relevant stage, point at prior art (e.g. existing + call-graph provider in `src/semantic_analysis` as template for new one), sketch + signatures/types, outline approach, answer questions about codebase. +- **Review on request:** when I share diff or push, critique it — correctness, + **parity with Python/Java backends**, schema conformance, missing tests, edge cases — and suggest concrete improvements. -- Scaffolding like tests or boilerplate is fine **when I ask**; otherwise leave the +- Scaffolding like tests or boilerplate fine **when I ask**; otherwise leave keyboard to me. - If you think I'm about to go wrong, say so briefly and let me decide — don't pre-empt by implementing the fix. @@ -127,63 +133,63 @@ Act as a helper, not the author: ## Rules 1. **Think before coding.** State assumptions explicitly; ask rather than guess. Push - back when a simpler approach exists. Stop when confused. -2. **Simplicity first.** Guide me toward the minimum idiomatic code that solves the + back when simpler approach exists. Stop when confused. +2. **Simplicity first.** Guide me toward minimum idiomatic code that solves the problem. Nothing speculative; no abstractions for single-use code. -3. **Issue → branch → work → PR.** Every change starts as an issue, on a branch named - `feat/issue-XXX`, `fix/issue-XXX`, `chore/issue-XXX`, and lands via a PR. +3. **Issue → branch → work → PR.** Every change starts as issue, on branch named + `feat/issue-XXX`, `fix/issue-XXX`, `chore/issue-XXX`, lands via PR. 4. **Guard the contract.** Changes to `src/schema` or Neo4j output must keep parity - with the sibling analyzers and pass the schema conformance test. + with sibling analyzers and pass schema conformance test. ## Goal-driven execution, as a teaching loop -Success is measured by the sole fact that **I understand it**. The success criterion: -I can point to the exact line of code where any feature lives, however remote or +Success measured by sole fact that **I understand it**. Success criterion: +I can point to exact line of code where any feature lives, however remote or obscure, and explain why it's there and how it behaves. -To that end, be my teacher and a Socratic one — not an answer key: +To that end, be my teacher and Socratic one — not answer key: -- Lead with questions that make me derive the answer; don't hand me the solution. -- Verify understanding, not just behavior — have me locate and explain the relevant - LOC, walk edge cases, and predict what a change would do before running it. -- Teach, help improve, and strengthen the weak spots you surface; circle back to them. -- The loop closes when I can **teach it back** and place every feature on a line, not - merely when the tests pass. -- Over the session, frequently — but not so much that I am stymied — ask spaced - repetition questions so concepts are internalized. +- Lead with questions that make me derive answer; don't hand me solution. +- Verify understanding, not just behavior — have me locate and explain relevant + LOC, walk edge cases, predict what change would do before running it. +- Teach, help improve, strengthen weak spots you surface; circle back to them. +- Loop closes when I can **teach it back** and place every feature on a line, not + merely when tests pass. +- Over session, frequently — but not so much that I am stymied — ask spaced + repetition questions so concepts internalized. -Learning progress is tracked globally, not per-repo: see the SRS deck and the +Learning progress tracked globally, not per-repo: see SRS deck and "continual learning" defaults in `~/.claude/CLAUDE.md`. # Auxiliary support tasks ## Tidy up the release announcement -Every `vX.Y.Z` tag makes the release workflow (`.github/workflows/release.yml`) -auto-post an announcement to the **Announcements** discussion (this repo, and the -org mirror in `codellm-devkit/.github`). That body is machine-generated from the -release notes / PR titles — it mis-categorizes changes, includes `chore(release)` -noise, and buries breaking changes. +Every `vX.Y.Z` tag makes release workflow (`.github/workflows/release.yml`) +auto-post announcement to **Announcements** discussion (this repo, and +org mirror in `codellm-devkit/.github`). That body machine-generated from +release notes / PR titles — mis-categorizes changes, includes `chore(release)` +noise, buries breaking changes. -When you're working in this repo, check whether the latest release's announcement +When working in this repo, check whether latest release's announcement still needs cleanup, and if so, fix it: 1. **Find it.** `gh release view --json tagName,publishedAt`; then list recent discussions via `gh api graphql` (repository → discussions), match category - `Announcements` and title `vX.Y.Z`. Keep the discussion node `id` and read its + `Announcements` and title `vX.Y.Z`. Keep discussion node `id`, read its `body`. -2. **Skip if already done.** If the body starts with `` (or already - reads as a clear, human-written announcement), do nothing. -3. **Otherwise rewrite it** into a clear, user-facing announcement, grounded in - `CHANGELOG.md` and the referenced PRs/diff (not the auto-grouping — verify each +2. **Skip if already done.** If body starts with `` (or already + reads as clear, human-written announcement), do nothing. +3. **Otherwise rewrite it** into clear, user-facing announcement, grounded in + `CHANGELOG.md` and referenced PRs/diff (not auto-grouping — verify each change; never invent anything): - - **breaking changes first**, each with a one-line migration step; - - plain-language highlights (what it does, not the PR title); + - **breaking changes first**, each with one-line migration step; + - plain-language highlights (what it does, not PR title); - upgrade lines — `pip install -U "codeanalyzer-typescript==X.Y.Z"`, or - `brew upgrade codellm-devkit/homebrew-tap/codeanalyzer-typescript`, or the + `brew upgrade codellm-devkit/homebrew-tap/codeanalyzer-typescript`, or shell installer one-liner; - - links to the GitHub release and `CHANGELOG.md`. -4. **Update in place.** Edit the discussion body with the GraphQL `updateDiscussion` - mutation (don't open a new one), prepend ``, and mirror the same - body to the org discussion. This task only reads code and edits Discussions — it - makes no commits. + - links to GitHub release and `CHANGELOG.md`. +4. **Update in place.** Edit discussion body with GraphQL `updateDiscussion` + mutation (don't open new one), prepend ``, mirror same + body to org discussion. This task only reads code and edits Discussions — makes + no commits. diff --git a/docs/design/specs/native-v2-model.md b/docs/design/specs/native-v2-model.md new file mode 100644 index 0000000..a67db81 --- /dev/null +++ b/docs/design/specs/native-v2-model.md @@ -0,0 +1,179 @@ +# Native v2 model — retire the v1 compute model and the emit-time transform + +- **Status:** implemented (branch `refactor/issue-096-native-v2-model`) +- **Scope:** `codeanalyzer-typescript` only — no wire change, no SDK change, no sibling change +- **Schema version:** 2.1.0, unchanged. This is an internal rewrite; `analysis.json` and the + Neo4j projection stay semantically identical at every level. +- **Analyzer version:** 1.0.0 → 1.1.0 (one MINOR at completion) +- **Parity precedent:** codeanalyzer-python, whose staged v2 chain made its schema models the + native compute model (`codeanalyzer/schema/py_schema.py`) with small per-run passes + (`assign_ids.py`, `l1_body.py`, `l2_callees.py`, `call_graph_ids.py`) and strip-at-emit for + internal fields (python d0084cb). + +## Problem + +The analyzer carries **two model families for one output**. Stages build the v1 +`TSApplication` (`src/schema/schema.ts`, 445 lines, plus `graphs.ts`), and every emit reshapes +the whole tree into `V2Application` through a ~720-line transform (`src/schema/v2/emit.ts` 436 + +`src/schema/v2/dataflow.ts` 282). The transform re-buckets containers (`classes`/`interfaces`/ +`enums`/`type_aliases`/`namespaces` → `types{}`; `methods`/`inner_callables` → `callables{}`; +`attributes`/`properties`/`members`/`variables` → `fields{}`), assigns `can://` ids +(`idFromSig`, emit.ts:97), converts `call_sites[]` to `body{}` call nodes (`toBody`, +emit.ts:139), filters attributes through a 30-key `DROP` set plus recursive null-pruning +(`carry`/`pruneNulls`, emit.ts:38–95), resolves heritage, homes external/synthesized endpoints, +and rewrites edge keys. + +The cost is structural: **every schema feature lands twice** — once in the v1 model and builders, +once in the transform — and the `DROP`/`carry` indirection means the wire shape of any node is +defined nowhere; it is the residue of a filter. The migration playbook that produced this +architecture calls the wrap-don't-rewrite emitter "the compat shim the migration leans on *while +both schemas coexist*". Coexistence ended when the v1 wire was removed (`toV2` is unconditional +in `src/utils/serialize.ts`); python has since retired its shim. This spec retires ours. + +## Decisions + +| Decision | Choice | +| --- | --- | +| Architecture | **One model family + small per-run passes** (python parity), not inline-everything | +| Wire gate | **Deep-equal goldens** captured from pre-rewrite `main`, both fixtures, `-a 1..4` + `graph.cypher`; key order free, arrays order-sensitive | +| Tracking | **One issue, one PR**; stages are commits, each green on the goldens | +| Release | **1.1.0 at completion**; stages merge nothing individually; no SDK lockstep | +| Terminal naming | `V2*` → `TS*` at teardown; `src/schema/v2/model.ts` folds into `src/schema/schema.ts`. As implemented: envelope `TSAnalysis`, root node `TSApplication` (python's `PyApplication` analog), internal working set `AnalysisInternal`, wire edges `TSCallGraphEdge`/`TSParamEdge` | +| Goldens lifetime | Transition-scoped — harness and goldens deleted at teardown | +| L3/L4 tree-attach | Moves to `src/dataflow/attach.ts` (python parity: its dataflow emits onto the tree, python 5600542) | + +**Why ids are a pass, not build-time.** Durable ids embed the app name +(`can://typescript//…`), and `--app-name` is per-invocation (`src/options/options.ts:15`), +while the symbol table round-trips through the analysis cache across runs +(`src/utils/cache.ts`). A tree with baked ids would go stale the moment the app name changes; +a tree without ids is cacheable forever. Python hit the same constraint — `assign_ids.py` stamps +ids fresh every run and returns the `signature → id` map the later passes join on. + +**The join key stays the signature.** `signatureOf` (`src/schema/signatures.ts`) remains the one +canonicalizer; builders, call-graph providers, the cache, and the dataflow compute all keep +speaking signature strings. `can://` ids exist only downstream of the assign-ids pass, exactly +as in python. + +## Target architecture + +### The model (`src/schema/schema.ts`, terminal state) + +The v2 shapes currently in `src/schema/v2/model.ts` become the model the stages build: + +- **Containers** (`TSModule`, `TSType`, `TSCallable`, `TSField` — today `V2Module` etc.) are + v2-bucketed (`types{}`/`functions{}`/`fields{}`/`callables{}`/`body{}`), carry `id` (stamped + per-run), `kind`, `span` only — the flat `start_line`/`end_line`/`start_column`/`end_column` + quartet disappears from container types (it is `DROP`ped from the wire today). +- **Leaf models** (`TSImport`, `TSExport`, `TSComment`, `TSCallableParameter`, `TSDecorator`, + `TSTypeParameter`, enum members, …) keep their current wire shapes, flat ints included — but + nullable fields become **optional**: the wire's "present or absent, never null" convention + (today enforced by `pruneNulls`) becomes the model's own convention, and builders omit instead + of writing `null`. The one sanctioned `null` stays: a `call` body node's `callee` at L1. +- **Internal fields** ride the model but never the wire, python-style (strip at emit, not a + field-by-field copy): `call_sites[]` and `callee_signature` (cache round-trip + the l1/l2 + passes; see python's l2_callees.py docstring for why the linker's resolutions are never + persisted), `module_name` (signature prefix), `path`/`file_path`, and the cache trio + `content_hash`/`last_modified`/`file_size`. + +### The spine (`src/core.ts`, terminal order) + +``` +materialize → buildSymbolTable (v2 buckets, no ids) + → assignIds(app, appName) # stamps can:// ids; returns sigToId + collisions + → populateL1Body # call_sites[] → body{} call nodes, callee: null + → resolveHeritage # extends_ids / implements_ids (TS-specific pass) + → [L2] provider.build per program # unchanged, signature-keyed + → [L2] homeEndpoints # externals + synthesized (2.1.0 compat index) → sigToId + → [L2] backfillCallees + reidentifyCallGraph # callee null→id; edge sig→id, source/target→src/dst + → [L3/4] buildProgramGraphs → attach # attach writes body/cfg/cdg/ddg/summary/param_* onto the tree + → envelope # schema_version / language / max_level / k_limit / analyzer + → saveCache (tree without ids, with internal fields) +``` + +`analyze()` returns the enveloped application. `src/utils/serialize.ts` shrinks to: JSON path = +strip internal fields + write; Neo4j path = `project(application)` directly — the projection +(`src/build/neo4j`) already consumes the v2 tree and does not change. + +### Module layout (terminal) + +| Path | Responsibility | +| --- | --- | +| `src/schema/schema.ts` | **The** model (v2 shapes, `TS*` names) + envelope types | +| `src/schema/ids.ts` | `can://` construction + `idFromSig`/`memberKey` (moved from emit.ts) | +| `src/schema/assignIds.ts` | id stamping pass → `sigToId`, collision gate | +| `src/schema/l1Body.ts` | `call_sites[]` → `body{}` call nodes | +| `src/schema/l2Callees.ts` | callee `null→id` + call-graph re-identification, dangling gate | +| `src/schema/homing.ts` | externals + synthesized-callable compat index (from emit.ts:298–360) | +| `src/schema/heritage.ts` | `extends_ids`/`implements_ids` resolution | +| `src/schema/signatures.ts` | unchanged | +| `src/schema/graphs.ts` | unchanged — the dataflow **compute IR**, no longer serialized on the app | +| `src/dataflow/attach.ts` | `applyDataflow` re-homed: program-graph IR → tree | +| *(deleted)* | `src/schema/v2/emit.ts`, `src/schema/v2/dataflow.ts`, `src/schema/v2/model.ts`, v1 container types, `DROP`/`carry`/`pruneNulls` | + +(Exact pass-file granularity may collapse siblings into one file at implementation time; the +pass *boundaries* above are the contract.) + +## Wire-stability gate + +Before any model change, capture goldens from `main`: `analysis.json` at `-a 1|2|3|4` and +`graph.cypher`, for `test/fixtures/sample-app` and `test/fixtures/dataflow-app`, committed under +`test/goldens/`. A transition test then asserts on every commit of the branch: + +- `analysis.json`: **deep-equal** after parsing — object key order free (JSON objects are + unordered; the SDK parses, never diffs bytes), arrays order-sensitive. If an array's order + proves nondeterministic, the comparator sorts that one list and says so in a comment. +- `graph.cypher`: compared as **sorted line sets** (row emission order follows object iteration + order, which the re-bucketing legitimately changes; the graph is the set of rows). +- `analyzer.version` is excluded from comparison. + +The existing standing gates — schema conformance, `L1 ⊆ L2 ⊆ L3 ⊆ L4` monotonicity +(`test/schema-v2.test.ts`), Neo4j conformance — run unchanged throughout and remain after the +goldens are deleted at teardown. + +## Stages (commits within the one PR; each green on goldens + full suite) + +*As implemented:* Stages 2 and 3 landed as one commit — the pass chain is order-coupled +(the L2 passes join on `assignIds` output, the attach on both), so relocating it into the +core spine piecemeal would have produced two incoherent halves. The goldens also cover a +third fixture (`anon-app`) for the 2.1.0 synthesized-callable homing paths. + +1. **Stage 0 — harness.** Goldens captured from the branch base + the deep-equal test. +2. **Stage 1 — L1 native.** Containers in `schema.ts` become v2-bucketed and span-only; + `builders.ts`/`symbolTable.ts` fill them; leaf nullables go optional; `assignIds`, `l1Body`, + `heritage` passes; every symbol-table reader (semantic_analysis, dataflow/extract, cache) + moves to the new buckets; `emit.ts` loses its tree walk and shrinks to L2 homing + dataflow + + envelope. The largest stage — it retires `carry`/`DROP` for containers. +3. **Stage 2 — L2 native.** Homing/backfill/re-identification become core-spine passes; + `emit.ts` loses its L2 block. Collision and dangling gates surface from the passes (today + `ToV2Result.collisions`/`dangling`; tests re-point). +4. **Stage 3 — L3/L4 native.** `src/schema/v2/dataflow.ts` → `src/dataflow/attach.ts`; + `program_graphs` leaves the application model (stays the internal return of + `buildProgramGraphs`); envelope moves to core; `emit.ts` is deleted. +5. **Stage 4 — teardown.** Delete v1 types and `src/schema/v2/`; rename `V2*` → `TS*`; delete + goldens + harness; update `CLAUDE.md`/`README`/`.claude/SCHEMA_DECISIONS.md`; CHANGELOG under + *Changed* (internal, no wire impact); bump 1.1.0. + +## What does not change + +- The wire: `analysis.json` all levels, `graph.cypher`, `schema.neo4j.json`, schema_version 2.1.0. +- `signatureOf` and signature strings as the internal join currency. +- Call-graph providers' interface (signature-keyed edges + externals + synthesized), the union + merge, and Jelly. +- The dataflow compute (`src/dataflow/*` stages 1–7, `graphs.ts` IR, worker pool). +- The Neo4j projection (`src/build/neo4j`) — it already consumes the v2 tree. +- The SDK: no model change, no version lockstep, no release ordering constraint. + +## Caveats and risks + +- **Cache compatibility:** old caches store v1 shapes; the existing `analyzer_version` + invalidation (`cache.ts`, loadCache) drops them wholesale on first post-rewrite run. No + migration code — by design. +- **Stage 1 blast radius:** semantic_analysis (950 lines) and dataflow (2,527 lines) read + symbol-table containers; how much of each actually touches the re-bucketed fields is + discovered in Stage 1, not before. The goldens bound the risk: any misread shows up as a + fixture diff, not a silent drift. +- **L3 worker boundary:** `src/dataflow/worker.ts` builds its own Project per file; if workers + consume serialized v1 module shapes across the process boundary, that surface migrates in + Stage 1 with the other readers. +- **Deep-equal is weaker than byte-equal** on object key order by construction; that is the + accepted trade (no consumer is order-sensitive), recorded here so nobody "fixes" it later. diff --git a/src/build/neo4j/project.ts b/src/build/neo4j/project.ts index 0f2ab20..6c217f1 100644 --- a/src/build/neo4j/project.ts +++ b/src/build/neo4j/project.ts @@ -1,16 +1,16 @@ /** - * project() — the pure projection from the schema-v2 additive-CPG tree (`V2Application`) to graph + * project() — the pure projection from the schema-v2 additive-CPG tree (`TSAnalysis`) to graph * rows. It walks the uniform tree (module → types/functions/fields → callables → body) emitting one * graph node per tree/body node keyed on its `can://` id, containment as HAS_x / DECLARES edges, and * every typed overlay (call_graph, cfg/cdg/ddg/summary, param_in/param_out) as a typed relationship. * No I/O: the writers (cypher snapshot / bolt incremental) consume the returned `GraphRows`. * - * The graph is a second projection of the SAME v2 shape the JSON path emits (serialize.ts → toV2), + * The graph is a second projection of the SAME v2 envelope the JSON path emits (finalizeAnalysis), * so JSON and graph never diverge. Every project-owned node carries `_module` (its owning file key, * for the incremental writer's per-module isolation); shared nodes (External) carry none. */ -import type { V2Application, V2BodyNode, V2Callable, V2External, V2Field, V2Module, V2Node, V2Root, V2Type } from "../../schema/v2"; +import type { TSAnalysis, TSApplication, TSBodyNode, TSCallable, TSField, TSModule, TSType } from "../../schema"; import { SCHEMA_VERSION } from "./schema"; import { type GraphRows, type NodeRef, type Props, RowBuilder, prune } from "./rows"; @@ -41,9 +41,9 @@ const KIND_LABEL: Record = { function_expression: "TSCallable", }; -export function project(app: V2Application, _appName?: string): GraphRows { +export function project(app: TSAnalysis, _appName?: string): GraphRows { const b = new RowBuilder(); - const root: V2Root = app.application; + const root: TSApplication = app.application; const appRef = b.node(["Application", "TSApplication"], "id", root.id, prune({ id: root.id, @@ -67,19 +67,19 @@ export function project(app: V2Application, _appName?: string): GraphRows { } // External library targets (shared nodes — no _module). - for (const ext of Object.values((root.external_symbols ?? {}) as Record)) { + for (const ext of Object.values(root.external_symbols ?? {})) { b.node([CAN, "TSExternal"], "id", ext.id, prune({ id: ext.id, kind: "external", name: ext.name, module: ext.module })); } // 2.1.0: `synthesized_callables` is mostly a compatibility index (old id → tree id) whose targets // are already projected as tree nodes. Only the residual fallback entries — a signature no // provider could name, recognisable because the map key IS the entry's own id — still need a // standalone node, so call-graph edges pointing at them do not dangle. - for (const [key, sc] of Object.entries((root.synthesized_callables ?? {}) as Record)) { + for (const [key, sc] of Object.entries(root.synthesized_callables ?? {})) { if (key !== sc.id) continue; b.node([CAN, "TSAnonymousCallable"], "id", sc.id, prune({ - id: sc.id, kind: "callable", name: str(sc.name), path: str(sc.path), - start_line: spanLine(sc, "start"), start_column: spanCol(sc, "start"), - _module: str(sc.path), + id: sc.id, kind: "callable", name: sc.name ?? null, path: sc.path ?? null, + start_line: sc.span?.start?.[0] ?? null, start_column: sc.span?.start?.[1] ?? null, + _module: sc.path ?? null, })); } @@ -96,13 +96,13 @@ export function project(app: V2Application, _appName?: string): GraphRows { // ---------------------------------------------------------------------------------------------- /** Walk a scope's child maps (module OR namespace: types + functions + fields). */ -function projectScope(b: RowBuilder, scope: V2Module | V2Type, parent: NodeRef, fileKey: string): void { +function projectScope(b: RowBuilder, scope: TSModule | TSType, parent: NodeRef, fileKey: string): void { for (const t of Object.values(scope.types ?? {})) projectType(b, t, parent, fileKey); for (const c of Object.values(scope.functions ?? {})) projectCallable(b, c, parent, "TS_DECLARES", fileKey); for (const f of Object.values(scope.fields ?? {})) projectField(b, f, parent, fileKey); } -function projectType(b: RowBuilder, t: V2Type, parent: NodeRef, fileKey: string): void { +function projectType(b: RowBuilder, t: TSType, parent: NodeRef, fileKey: string): void { const label = KIND_LABEL[t.kind] ?? "TSClass"; const node = b.node([CAN, label], "id", t.id, typeProps(t, fileKey)); b.edge("TS_DECLARES", parent, node); @@ -121,7 +121,7 @@ function projectType(b: RowBuilder, t: V2Type, parent: NodeRef, fileKey: string) /** An unnamed callable's signature ends with the positional segment `contributorName` gives it. */ const ANON_SIG = /\.$/; -function projectCallable(b: RowBuilder, c: V2Callable, owner: NodeRef, ownerRel: string, fileKey: string): void { +function projectCallable(b: RowBuilder, c: TSCallable, owner: NodeRef, ownerRel: string, fileKey: string): void { // An unnamed callable carries :TSAnonymousCallable alongside :TSCallable — one node, two labels, // reached by ordinary containment. That is what keeps pre-2.1.0 MATCH (:TSAnonymousCallable) // queries working and puts these nodes on the snapshot wipe's containment walk (issue #75). @@ -152,9 +152,9 @@ function projectCallable(b: RowBuilder, c: V2Callable, owner: NodeRef, ownerRel: for (const t of Object.values(c.types ?? {})) projectType(b, t, node, fileKey); } -function projectField(b: RowBuilder, f: V2Field, owner: NodeRef, fileKey: string): void { +function projectField(b: RowBuilder, f: TSField, owner: NodeRef, fileKey: string): void { const node = b.node([CAN, "TSField"], "id", f.id, prune({ - id: f.id, kind: "field", name: str(f.name), type: str((f as V2Node).type), ...span(f), _module: fileKey, + id: f.id, kind: "field", name: f.name, type: f.type ?? null, ...span(f), _module: fileKey, })); b.edge("TS_HAS_FIELD", owner, node); } @@ -163,40 +163,42 @@ function projectField(b: RowBuilder, f: V2Field, owner: NodeRef, fileKey: string // property flattening (v2 node attrs → Neo4j-legal scalars/arrays) // ---------------------------------------------------------------------------------------------- -function moduleProps(mod: V2Module, fileKey: string): Props { +function moduleProps(mod: TSModule, fileKey: string): Props { + // The wire strips the internal fields (module_name, content_hash) — the graph name is the + // file key, exactly as the historical projection of the stripped tree produced. return prune({ - id: mod.id, kind: "module", name: str((mod as V2Node).module_name) ?? fileKey, - is_tsx: bool((mod as V2Node).is_tsx), is_declaration_file: bool((mod as V2Node).is_declaration_file), - content_hash: str((mod as V2Node).content_hash), ...span(mod), _module: fileKey, + id: mod.id, kind: "module", name: fileKey, + is_tsx: mod.is_tsx, is_declaration_file: mod.is_declaration_file, + ...span(mod), _module: fileKey, }); } -function typeProps(t: V2Type, fileKey: string): Props { +function typeProps(t: TSType, fileKey: string): Props { return prune({ - id: t.id, kind: t.kind, signature: str(t.signature), name: str((t as V2Node).name), - base_classes: strArr((t as V2Node).base_classes), implements_types: strArr((t as V2Node).implements_types), - aliased_type: str((t as V2Node).aliased_type), - is_abstract: bool((t as V2Node).is_abstract), is_const: bool((t as V2Node).is_const), - is_exported: bool((t as V2Node).is_exported), is_ambient: bool((t as V2Node).is_ambient), + id: t.id, kind: t.kind, signature: t.signature, name: t.name, + base_classes: strArr(t.base_classes), implements_types: strArr(t.implements_types), + aliased_type: t.aliased_type ?? null, + is_abstract: t.is_abstract ?? null, is_const: t.is_const ?? null, + is_exported: t.is_exported, is_ambient: t.is_ambient, ...span(t), }); } -function callableProps(c: V2Callable, fileKey: string): Props { +function callableProps(c: TSCallable, fileKey: string): Props { return prune({ - id: c.id, kind: c.kind, signature: str(c.signature), name: str((c as V2Node).name), - return_type: str((c as V2Node).return_type), cyclomatic_complexity: num((c as V2Node).cyclomatic_complexity), - accessibility: str((c as V2Node).accessibility), accessor_kind: str((c as V2Node).accessor_kind), - is_static: bool((c as V2Node).is_static), is_abstract: bool((c as V2Node).is_abstract), - is_async: bool((c as V2Node).is_async), is_generator: bool((c as V2Node).is_generator), - is_exported: bool((c as V2Node).is_exported), is_ambient: bool((c as V2Node).is_ambient), - is_implicit: bool((c as V2Node).is_implicit), ...span(c), _module: fileKey, + id: c.id, kind: c.kind, signature: c.signature, name: c.name, + return_type: c.return_type ?? null, cyclomatic_complexity: c.cyclomatic_complexity, + accessibility: c.accessibility ?? null, accessor_kind: c.accessor_kind ?? null, + is_static: c.is_static, is_abstract: c.is_abstract, + is_async: c.is_async, is_generator: c.is_generator, + is_exported: c.is_exported, is_ambient: c.is_ambient, + is_implicit: c.is_implicit, ...span(c), _module: fileKey, }); } -function bodyProps(bn: V2BodyNode, id: string, fileKey: string): Props { +function bodyProps(bn: TSBodyNode, id: string, fileKey: string): Props { return prune({ - id, kind: bn.kind, of: str(bn.of), parent: str(bn.parent), + id, kind: bn.kind, of: bn.of ?? null, parent: bn.parent ?? null, callee: typeof bn.callee === "string" ? bn.callee : null, ...span(bn), _module: fileKey, }); } @@ -217,7 +219,7 @@ const edges = (x: unknown): Edge[] => (Array.isArray(x) ? (x as Edge[]) : []); /** A cross-edge endpoint is already the graph node's can:// id — pass it through unchanged. */ const idOf = (endpoint: string): string => endpoint; -function moduleKeyOf(mod: V2Module): string { +function moduleKeyOf(mod: TSModule): string { // id = can:////; the fileKey is everything after the 3rd '/' past the scheme. const m = /^can:\/\/[^/]+\/[^/]+\/(.+)$/.exec(mod.id); return m ? (m[1] as string) : mod.id; @@ -227,17 +229,5 @@ function span(n: { span?: { start: [number, number]; end: [number, number] } }): if (!n.span) return {}; return { start_line: n.span.start?.[0], end_line: n.span.end?.[0] }; } -function spanLine(n: V2Node, which: "start" | "end"): number | undefined { - const s = n.span as { start?: [number, number]; end?: [number, number] } | undefined; - return s?.[which]?.[0]; -} -function spanCol(n: V2Node, which: "start" | "end"): number | undefined { - const s = n.span as { start?: [number, number]; end?: [number, number] } | undefined; - return s?.[which]?.[1]; -} - -// value coercion (v2 attrs are `unknown`; Neo4j props must be scalars / homogeneous arrays) -const str = (v: unknown): string | null => (typeof v === "string" ? v : null); -const num = (v: unknown): number | null => (typeof v === "number" ? v : null); -const bool = (v: unknown): boolean | null => (typeof v === "boolean" ? v : null); -const strArr = (v: unknown): string[] | null => (Array.isArray(v) && v.every((x) => typeof x === "string") && v.length ? (v as string[]) : null); +// A present-but-empty string list is a non-fact in the graph (matches the historical projection). +const strArr = (v: string[] | undefined): string[] | null => (v && v.length ? v : null); diff --git a/src/build/neo4j/schema.ts b/src/build/neo4j/schema.ts index d3be823..acb7f1d 100644 --- a/src/build/neo4j/schema.ts +++ b/src/build/neo4j/schema.ts @@ -170,7 +170,7 @@ export const REL_TYPES: RelType[] = [ { type: "TS_SUMMARY", from: ["TSBodyNode"], to: ["TSBodyNode"], properties: { var: "string" } }, { type: "TS_PARAM_IN", from: ["TSBodyNode"], to: ["TSBodyNode"], properties: { var: "string" } }, { type: "TS_PARAM_OUT", from: ["TSBodyNode"], to: ["TSBodyNode"], properties: { var: "string" } }, - // Inheritance, projected from the `extends_ids`/`implements_ids` node props (schema/v2/emit.ts) — + // Inheritance, projected from the `extends_ids`/`implements_ids` node props (the heritage pass, schema/heritage.ts) — // resolved-only: an unresolved (external/library) supertype never reaches here. A `to` of `TSClass` // covers TS's `implements SomeClass` (structural, not just interfaces); an interface may itself // `extends` a class's instance type, hence `TS_EXTENDS` also allows a `TSInterface` source. diff --git a/src/core.ts b/src/core.ts index 6aeda80..5355e86 100644 --- a/src/core.ts +++ b/src/core.ts @@ -4,15 +4,20 @@ import { mergeCallGraphs, selectProvider } from "./semantic_analysis"; import { loadCache, saveCache } from "./utils"; import { materialize } from "./build"; import type { AnalysisOptions } from "./options"; -import type { TSApplication } from "./schema"; +import type { AnalysisInternal } from "./schema"; +import { type AnalysisResult, finalizeAnalysis } from "./schema/emit"; import { buildSymbolTable } from "./syntactic_analysis"; import { Logger } from "./utils"; +export type { AnalysisResult } from "./schema/emit"; + /** - * The orchestrator. Order mirrors the reference analyzers: materialize deps → build the symbol - * table → build the resolver call graph → cache the base → return the Application. + * The orchestrator. Order mirrors the reference analyzers (python core.py): materialize deps → + * build the symbol table → call-graph providers → program graphs → cache the id-free base → + * run the per-run pass spine (ids / body / heritage / homing / callees / attach) and assemble + * the wire envelope. Returns BOTH views: the wire `application` and the live `internal` tree. */ -export async function analyze(opts: AnalysisOptions): Promise { +export async function analyze(opts: AnalysisOptions): Promise { const log = new Logger(opts.verbosity); log.info(`analyzing ${opts.input} (level ${opts.analysisLevel})`); const cacheDir = opts.cacheDir ?? path.join(opts.input, ".codeanalyzer"); @@ -38,8 +43,8 @@ export async function analyze(opts: AnalysisOptions): Promise { const extraction = opts.analysisLevel >= 3 ? startExtraction(project, symbol_table, mat.tsConfigFilePath, opts, log) : null; // Call graph via the selected provider (union of tsc+jelly by default; --tsc-only / jelly opt-in). - // Only worth running at level >= 2: the v2 emitter discards call_graph/external_symbols/ - // synthesized_callables at -a 1 (homeExternals/homeSynthesized in src/schema/v2/emit.ts are + // Only worth running at level >= 2: finalizeAnalysis discards call_graph/external_symbols/ + // synthesized_callables at -a 1 (homeExternals/homeSynthesized in src/schema/emit.ts are // gated to `level >= 2`), so running the solve — including the heavier Jelly leg — at -a 1 // would compute a result that's thrown away. Levels 3/4 need the provider for callee // resolution and are always >= 2, so this gate is safe. @@ -66,7 +71,7 @@ export async function analyze(opts: AnalysisOptions): Promise { } const call_graph = cg.edges; - const app: TSApplication = { + const app: AnalysisInternal = { symbol_table, call_graph, external_symbols: cg.external_symbols, @@ -75,10 +80,10 @@ export async function analyze(opts: AnalysisOptions): Promise { // Level 3 join: stages 5–7 (summary wavefront + SDG) consume the extraction AND the // provider-backfilled callee signatures. Strictly flag-gated so -a 1/-a 2 cost nothing. - if (extraction) { - app.program_graphs = await buildProgramGraphs(extraction, symbol_table, opts, log); - } + const pg = extraction ? await buildProgramGraphs(extraction, symbol_table, opts, log) : null; - saveCache(cacheDir, { symbol_table, call_graph }); - return app; + // Cache the id-free base (ids/body/heritage are per-run layers stamped by finalizeAnalysis; + // the cached tree must stay --app-name-free). + saveCache(cacheDir, { symbol_table }); + return finalizeAnalysis(app, pg, opts); } diff --git a/src/schema/v2/dataflow.ts b/src/dataflow/attach.ts similarity index 93% rename from src/schema/v2/dataflow.ts rename to src/dataflow/attach.ts index 72121cd..ceea6a9 100644 --- a/src/schema/v2/dataflow.ts +++ b/src/dataflow/attach.ts @@ -1,6 +1,8 @@ /** - * L3/L4 dataflow → the v2 tree. Transforms the v1 `program_graphs` (CFG / PDG[=CDG+DDG] / SDG, - * keyed by (signature, integer node_id)) into the additive-CPG placement: + * L3/L4 dataflow → the v2 tree (the ATTACH step): the stage that computes the program graphs also + * writes them onto the tree (python parity). Transforms the internal `program_graphs` compute IR + * (CFG / PDG[=CDG+DDG] / SDG, keyed by (signature, integer node_id)) into the additive-CPG + * placement: * * L3 (-a 3): grow each callable's `body{}` with statement nodes (+ `@entry`/`@exit`), and hang * the intra-callable edge lists `cfg`/`cdg`/`ddg` (bare local ids) on the callable. @@ -10,18 +12,17 @@ * * A pure relabel of what the analyzer already computed — no re-analysis. See * .claude/SCHEMA_DECISIONS.md § "Schema v2 migration" and dataflow-graphs.md. - * Node-kind mapping (grounded in the v1 model): + * Node-kind mapping (grounded in the compute IR, schema/graphs.ts): * entry→'@entry', exit→'@exit' (also '@formal_out' as a PARAM_OUT/formal-out anchor), * param→contracted out of the L3 CFG; '@formal_in:N' at L4, statement→'line:col'. */ -import type { CfgEdge, GraphNode, PdgEdge, ProgramGraphs } from "../graphs"; -import type { TSApplication } from "../schema"; -import type { V2Callable, V2ParamEdge, V2Root } from "./model"; +import type { CfgEdge, GraphNode, PdgEdge, ProgramGraphs } from "../schema/graphs"; +import type { TSApplication, TSCallable, TSParamEdge } from "../schema"; interface LocalIds { canId: string; - callable: V2Callable; + callable: TSCallable; stmtLocal: Map; // entry/exit/statement node id → body local id paramN: Map; // param node id → declaration index N paramName: Map; // param node id → the `of` name @@ -29,7 +30,7 @@ interface LocalIds { } /** Build the per-callable node_id→local-id maps (single source-of-truth for every edge rewrite). */ -function buildLocalIds(canId: string, callable: V2Callable, nodes: GraphNode[]): LocalIds { +function buildLocalIds(canId: string, callable: TSCallable, nodes: GraphNode[]): LocalIds { const stmtLocal = new Map(); const paramN = new Map(); const paramName = new Map(); @@ -130,7 +131,7 @@ function emitL3(li: LocalIds, nodes: GraphNode[], cfgEdges: CfgEdge[] | undefine // L4 — synthetic vertices + summary (callable) + param_in/param_out (application) // ---------------------------------------------------------------------------------------------- -function emitL4(root: V2Root, pg: ProgramGraphs, info: Map): void { +function emitL4(root: TSApplication, pg: ProgramGraphs, info: Map): void { // Formal vertices + the deferred formal-out-routing ddg edges, per callable. for (const [sig, fg] of Object.entries(pg.functions)) { const li = info.get(sig); @@ -226,14 +227,13 @@ function emitL4(root: V2Root, pg: ProgramGraphs, info: Map): v * `callableBySig` locates each callable's v2 node (populated during the L1 walk). */ export function applyDataflow( - root: V2Root, - app: TSApplication, + root: TSApplication, + pg: ProgramGraphs, idBySig: Map, - callableBySig: Map, + callableBySig: Map, level: number, ): void { - const pg = app.program_graphs; - if (!pg || level < 3) return; + if (level < 3) return; const info = new Map(); for (const [sig, fg] of Object.entries(pg.functions)) { @@ -277,6 +277,6 @@ function cmp3(a: { src: string; dst: string; kind: string }, b: { src: string; d function cmpDdg(a: { src: string; dst: string; var?: string }, b: { src: string; dst: string; var?: string }): number { return cmp2(a, b) || (a.var ?? "").localeCompare(b.var ?? ""); } -function cmpEdgeVar(a: V2ParamEdge, b: V2ParamEdge): number { +function cmpEdgeVar(a: TSParamEdge, b: TSParamEdge): number { return a.src.localeCompare(b.src) || a.dst.localeCompare(b.dst) || (a.var ?? "").localeCompare(b.var ?? ""); } diff --git a/src/dataflow/index.ts b/src/dataflow/index.ts index 8993537..51ded96 100644 --- a/src/dataflow/index.ts +++ b/src/dataflow/index.ts @@ -37,9 +37,8 @@ import { type ProgramGraphs, type TSCallable, type TSCallsite, - type TSClass, type TSModule, - type TSNamespace, + forEachCallable, } from "../schema"; import type { Logger } from "../utils"; import { extractCallableData, indexCallableDecls } from "./extract"; @@ -76,11 +75,11 @@ export function startExtraction( const callables = collectCallables(symbol_table); // Partition callables by owning file (round-robin over the sorted file list) so each worker - // deeply visits only its share of the program. TSCallable.path is the declaration's ABSOLUTE + // deeply visits only its share of the program. TSCallable.abs_path is the declaration's ABSOLUTE // file path; the graph data carries the project-relative file key. const byFile = new Map>(); for (const [sig, c] of [...callables.entries()].sort(([a], [b]) => a.localeCompare(b))) { - const absPath = c.path; + const absPath = c.abs_path; const arr = byFile.get(absPath) ?? []; arr.push({ signature: sig, path: fileKeyOf(absPath, opts.input).fileKey, absPath }); byFile.set(absPath, arr); @@ -163,7 +162,7 @@ function extractSequential( for (const [sig, c] of [...callables.entries()].sort(([a], [b]) => a.localeCompare(b))) { const fn = astIndex.get(sig); if (!fn) continue; // bodiless (interface/abstract/ambient/implicit) or unmatchable - const data = extractCallableData(sig, fn, fileKeyOf(c.path, opts.input).fileKey, opts.input, opts.graphFieldDepth); + const data = extractCallableData(sig, fn, fileKeyOf(c.abs_path, opts.input).fileKey, opts.input, opts.graphFieldDepth); if (data) out.set(sig, data); } return out; @@ -195,7 +194,7 @@ export async function buildProgramGraphs( for (const site of (callables.get(sig) as TSCallable).call_sites) { const nodeId = containingNode(data, site); if (nodeId === null) continue; - refs.push({ nodeId, callee: site.callee_signature, argCount: site.argument_types.length }); + refs.push({ nodeId, callee: site.callee_signature ?? null, argCount: site.argument_types.length }); } refs.sort((a, b) => a.nodeId - b.nodeId || (a.callee ?? "").localeCompare(b.callee ?? "")); callSites.set(sig, refs); @@ -389,8 +388,8 @@ function persistSummaries( const entries: Record = {}; for (const sig of [...summaries.keys()].sort()) { const c = callables.get(sig); - // TSCallable.path is absolute; the symbol table is keyed by the project-relative file key. - const fileKey = c ? fileKeyOf(c.path, opts.input).fileKey : null; + // TSCallable.abs_path is absolute; the symbol table is keyed by the project-relative file key. + const fileKey = c ? fileKeyOf(c.abs_path, opts.input).fileKey : null; entries[sig] = { ...summaries.get(sig), content_hash: (fileKey && symbol_table[fileKey]?.content_hash) ?? null, @@ -404,34 +403,11 @@ function persistSummaries( } // ------------------------------------------------------------------------------------------------ -// Symbol-table collection (signature → callable), recursing through every container kind +// Symbol-table collection (signature → callable) — the shared containment walk (schema.ts) // ------------------------------------------------------------------------------------------------ function collectCallables(symbol_table: Record): Map { const out = new Map(); - for (const mod of Object.values(symbol_table)) collectModule(mod, out); + for (const mod of Object.values(symbol_table)) forEachCallable(mod, (c) => out.set(c.signature, c)); return out; } - -function collectModule(mod: TSModule, out: Map): void { - for (const f of Object.values(mod.functions)) collectCallable(f, out); - for (const c of Object.values(mod.classes)) collectClass(c, out); - for (const ns of Object.values(mod.namespaces)) collectNamespace(ns, out); -} - -function collectNamespace(ns: TSNamespace, out: Map): void { - for (const f of Object.values(ns.functions)) collectCallable(f, out); - for (const c of Object.values(ns.classes)) collectClass(c, out); - for (const n of Object.values(ns.namespaces)) collectNamespace(n, out); -} - -function collectClass(c: TSClass, out: Map): void { - for (const m of Object.values(c.methods)) collectCallable(m, out); - for (const ic of Object.values(c.inner_classes)) collectClass(ic, out); -} - -function collectCallable(c: TSCallable, out: Map): void { - out.set(c.signature, c); - for (const ic of Object.values(c.inner_callables)) collectCallable(ic, out); - for (const cl of Object.values(c.inner_classes)) collectClass(cl, out); -} diff --git a/src/index.ts b/src/index.ts index 0d4a836..3b6de23 100644 --- a/src/index.ts +++ b/src/index.ts @@ -11,8 +11,8 @@ async function main(): Promise { emitSchema(opts); return; } - const app = await analyze(opts); - await emit(app, opts); + const result = await analyze(opts); + await emit(result.application, opts); } catch (e) { const err = e as Error; process.stderr.write(`[codeanalyzer-ts] FATAL ${err.stack ?? err.message}\n`); diff --git a/src/schema/assignIds.ts b/src/schema/assignIds.ts new file mode 100644 index 0000000..e493c11 --- /dev/null +++ b/src/schema/assignIds.ts @@ -0,0 +1,63 @@ +/** + * Walk the symbol-table tree and stamp every node with its `can://` id — python's + * `assign_ids.py`. Runs once per analysis run (ids embed the per-invocation app name, so the + * cached tree is stored id-free and re-stamped on every run; stamping is overwrite-idempotent). + * + * Returns the `signature → id` map the later passes join on (callee backfill, call-graph + * re-identification, dataflow attach), the callable locator for the L3/L4 attach, and the + * id-uniqueness gate's collision list. + */ + +import { applicationIdOf, idFromSig, memberKey, moduleIdOf, modulePrefixOf } from "./ids"; +import type { AnalysisInternal, TSCallable, TSField, TSType } from "./schema"; + +export interface AssignedIds { + appId: string; + idBySig: Map; // signature → can:// id (types + callables) + callableBySig: Map; // locates each callable's node for the L3/L4 attach + collisions: string[]; // signatures that mapped to two distinct ids (L1 id-uniqueness gate) +} + +export function assignIds(app: AnalysisInternal, appName: string): AssignedIds { + const appId = applicationIdOf(appName); + const idBySig = new Map(); + const callableBySig = new Map(); + const collisions: string[] = []; + + const register = (sig: string, id: string): void => { + if (idBySig.has(sig) && idBySig.get(sig) !== id) collisions.push(sig); + idBySig.set(sig, id); + }; + + const doFields = (parentId: string, fields: Record | undefined): void => { + for (const [name, f] of Object.entries(fields ?? {})) f.id = `${parentId}/${name}`; + }; + + const doCallable = (moduleId: string, modulePrefix: string, c: TSCallable): void => { + c.id = idFromSig(moduleId, modulePrefix, c.signature); + register(c.signature, c.id); + callableBySig.set(c.signature, c); + for (const nested of Object.values(c.callables ?? {})) doCallable(moduleId, modulePrefix, nested); + for (const t of Object.values(c.types ?? {})) doType(moduleId, modulePrefix, t); + }; + + const doType = (moduleId: string, modulePrefix: string, t: TSType): void => { + t.id = idFromSig(moduleId, modulePrefix, t.signature); + register(t.signature, t.id); + doFields(t.id, t.fields); + for (const m of Object.values(t.callables ?? {})) doCallable(moduleId, modulePrefix, m); + for (const f of Object.values(t.functions ?? {})) doCallable(moduleId, modulePrefix, f); // namespace + for (const nt of Object.values(t.types ?? {})) doType(moduleId, modulePrefix, nt); // namespace + }; + + for (const [fileKey, mod] of Object.entries(app.symbol_table)) { + const moduleId = moduleIdOf(appId, fileKey); + const modulePrefix = modulePrefixOf(fileKey); + mod.id = moduleId; + doFields(moduleId, mod.fields); + for (const fn of Object.values(mod.functions ?? {})) doCallable(moduleId, modulePrefix, fn); + for (const t of Object.values(mod.types ?? {})) doType(moduleId, modulePrefix, t); + } + + return { appId, idBySig, callableBySig, collisions }; +} diff --git a/src/schema/emit.ts b/src/schema/emit.ts new file mode 100644 index 0000000..4ba933e --- /dev/null +++ b/src/schema/emit.ts @@ -0,0 +1,92 @@ +/** + * finalizeAnalysis — the per-run pass spine (invoked by `analyze()`, python-parity: core runs the + * passes; serialization is dumb): stamp the per-run layers onto the NATIVELY-built tree, assemble + * the envelope, and strip the INTERNAL fields. No reshaping happens anywhere — the builders + * construct the wire shapes directly (src/syntactic_analysis/builders.ts): + * + * assignIds — can:// ids (per-run: ids embed --app-name; the cache stays id-free) + * populateL1Body — call_sites → body{} `call` nodes, callee: null + * resolveHeritage — extends_ids / implements_ids (resolved-only) + * [L2] homeExternals / homeSynthesized / backfillCallees / reidentifyCallGraph + * [L3/4] applyDataflow — program_graphs → body{} + cfg/cdg/ddg/summary + param_in/param_out + * + * The returned application is a DEEP, INTERNAL-FIELD-STRIPPED copy: the live tree keeps + * `call_sites`, `abs_path`, and the cache metadata for the resolver/dataflow/cache, while every + * consumer of the emission (JSON writer, Neo4j projection, tests) sees exactly the wire. + */ + +import * as path from "node:path"; +import type { AnalysisOptions } from "../options"; +import { ANALYZER_VERSION } from "../utils/version"; +import type { AnalysisInternal, TSAnalysis, TSApplication } from "./schema"; +import type { ProgramGraphs } from "./graphs"; +import { assignIds } from "./assignIds"; +import { populateL1Body } from "./l1Body"; +import { resolveHeritageIds } from "./heritage"; +import { homeExternals, homeSynthesized } from "./homing"; +import { backfillCallees, reidentifyCallGraph } from "./l2Callees"; +import { applyDataflow } from "../dataflow/attach"; + +const LANGUAGE = "typescript"; +const SCHEMA_VERSION = "2.1.0"; +const ANALYZER_NAME = "codeanalyzer-typescript"; +/** Highest analysis level this emitter populates today (L1 tree, L2 call graph, L3/L4 dataflow). */ +const MAX_IMPLEMENTED = 4; + +/** INTERNAL model fields — never on the wire (see schema.ts header). */ +const INTERNAL_KEYS = new Set(["call_sites", "abs_path", "content_hash", "last_modified", "file_size"]); + +// ---------------------------------------------------------------------------------------------- +// entry point +// ---------------------------------------------------------------------------------------------- + +export interface AnalysisResult { + application: TSAnalysis; // the wire: deep, internal-field-stripped envelope + internal: AnalysisInternal; // the live internal working set (tree + sig-keyed provider outputs) + program_graphs?: ProgramGraphs; // the L3/L4 compute IR (already attached onto the wire tree) + idBySig: Map; // signature → can:// id (real callables + externals + synthesized) + collisions: string[]; // signatures that mapped to two distinct ids (L1 id-uniqueness gate) + dangling: string[]; // call-graph endpoints with no id home (L2 no-dangling gate; should be empty) +} + +export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null, opts: AnalysisOptions): AnalysisResult { + const level = opts.analysisLevel; + const appName = (opts.appName ?? (opts.input ? path.basename(opts.input) : "") ?? "").trim() || "app"; + + // L1 — stamp ids, derive body{}, project heritage (all overwrite-idempotent per-run passes). + const { appId, idBySig, callableBySig, collisions } = assignIds(app, appName); + populateL1Body(app); + resolveHeritageIds(app, idBySig); + + const root: TSApplication = { id: appId, kind: "application", symbol_table: app.symbol_table, call_graph: [], param_in: [], param_out: [] }; + + // L2 — home the off-tree edge endpoints, backfill `callee`, re-identify the call graph. + const dangling: string[] = []; + if (level >= 2) { + root.external_symbols = homeExternals(app, appId, idBySig); + root.synthesized_callables = homeSynthesized(app, appId, idBySig); + backfillCallees(app, idBySig); + root.call_graph = reidentifyCallGraph(app.call_graph ?? [], idBySig, dangling); + } + + // L3/L4 — grow body{} + cfg/cdg/ddg/summary on callables and param_in/param_out on the app. + let k_limit: number | undefined; + if (level >= 3 && pg) { + applyDataflow(root, pg, idBySig, callableBySig, level); + k_limit = pg.k_limit; + } + + const envelope: TSAnalysis = { + schema_version: SCHEMA_VERSION, + language: LANGUAGE, + max_level: Math.min(level, MAX_IMPLEMENTED), + ...(k_limit !== undefined ? { k_limit } : {}), + analyzer: { name: ANALYZER_NAME, version: ANALYZER_VERSION }, + application: root, + }; + // The wire copy: deep, detached from the live tree, internal fields stripped by key. + const application = JSON.parse( + JSON.stringify(envelope, (key, value) => (INTERNAL_KEYS.has(key) ? undefined : value)), + ) as TSAnalysis; + return { application, internal: app, ...(pg ? { program_graphs: pg } : {}), idBySig, collisions, dangling }; +} diff --git a/src/schema/heritage.ts b/src/schema/heritage.ts new file mode 100644 index 0000000..0b36ca9 --- /dev/null +++ b/src/schema/heritage.ts @@ -0,0 +1,37 @@ +/** + * Heritage projection pass (TS-specific): resolve each type's heritage SIGNATURES + * (`base_classes`/`implements_types`, which stay on the wire as the human-readable spine) into + * `can://` ids for the Neo4j EXTENDS/IMPLEMENTS overlay. Resolved-only: an external/library + * supertype that never maps to a first-party id is dropped, never nulled. Runs at every level + * (types are homed by the unconditional L1 walk) and is overwrite-idempotent (stale ids from a + * previous run under another app name are recomputed; unresolvable sets are deleted). + */ + +import type { AnalysisInternal, TSModule, TSType } from "./schema"; +import { forEachType } from "./schema"; + +function resolve(sigs: string[], idBySig: Map): string[] { + return sigs.map((s) => idBySig.get(s)).filter((x): x is string => x !== undefined); +} + +function doType(t: TSType, idBySig: Map): void { + delete t.extends_ids; + delete t.implements_ids; + const base = t.base_classes ?? []; + if (!base.length) return; + const impl = t.implements_types ?? []; + // A class's `base_classes` is the union of extends + implements; subtract `implements_types` + // to recover just the extended base class (0 or 1 — TS classes extend at most one class). + // Interfaces carry no `implements_types`, so their whole heritage is extends. + const extendsSigs = t.kind === "class" ? base.filter((s) => !impl.includes(s)) : base; + const extendsIds = resolve(extendsSigs, idBySig); + const implementsIds = resolve(impl, idBySig); + if (extendsIds.length) t.extends_ids = extendsIds; + if (implementsIds.length) t.implements_ids = implementsIds; +} + +export function resolveHeritageIds(app: AnalysisInternal, idBySig: Map): void { + for (const mod of Object.values(app.symbol_table) as TSModule[]) { + forEachType(mod, (t) => doType(t, idBySig)); + } +} diff --git a/src/schema/homing.ts b/src/schema/homing.ts new file mode 100644 index 0000000..7eea938 --- /dev/null +++ b/src/schema/homing.ts @@ -0,0 +1,87 @@ +/** + * L2 endpoint homing (TS-specific): give every off-tree call-graph endpoint an id home on the + * application root, so the no-dangling rule holds. + * + * - `homeExternals`: external library call targets → `can://…/@external//` nodes. + * - `homeSynthesized`: the 2.1.0 anonymous-callable compatibility index (pre-2.1.0 id → the + * tree id that replaced it), plus residual fallback nodes for signatures no provider could + * name (recognizable because the map key equals the entry's own id). + * + * Both register their ids into `idBySig`, which is why they run BEFORE the callee backfill and + * the call-graph re-identification (l2Callees.ts). + */ + +import type { AnalysisInternal, TSSpan } from "./schema"; + +/** A call target outside the project (an imported library member / builtin) — an edge endpoint, not a tree node. */ +export interface TSExternalNode { + id: string; + kind: "external"; + module: string; // the import/require specifier, e.g. "node:fs", "express" + name: string; // the called member, e.g. "readFileSync" +} + +/** A synthesized-callable index entry: a pointer node (id + kind) or a residual fallback node. */ +export interface TSSynthesizedNode { + id: string; + kind: string; + name?: string; + path?: string; + span?: TSSpan; +} + +/** External library call targets → `can://…/@external//` ids on the application root. */ +export function homeExternals(app: AnalysisInternal, appId: string, idBySig: Map): Record { + const out: Record = {}; + for (const [sig, ext] of Object.entries(app.external_symbols ?? {})) { + const id = `${appId}/@external/${ext.module}/${ext.name}`; + idBySig.set(sig, id); + out[id] = { id, kind: "external", module: ext.module, name: ext.name }; + } + return out; +} + +/** + * The compatibility index for anonymous callables (schema 2.1.0). + * + * Anonymous callables are real nodes in the containment tree, signed positionally + * (`.`) and reachable by containment. This map is not a node + * registry: it maps the **pre-2.1.0 id** of each anonymous callable — `@:`, + * derived from the old `:` signature — onto the tree id that replaced it, + * so a consumer holding an old id can still resolve it. + * + * The old host was the nearest enclosing callable the old rules could name, which is recovered by + * stripping the trailing `` chain. An anonymous callable directly under a module had no + * resolvable old id (the old emitter fell back to an opaque `@synthetic/` key that encoded the + * ambiguous `:` signature, which was not unique across files) — those are + * skipped rather than reproduced. + * + * Any signature the call-graph provider still could not name is homed here too, unchanged, so the + * no-dangling rule holds even if a provider reports a function-like node the tree missed. + */ +export function homeSynthesized(app: AnalysisInternal, appId: string, idBySig: Map): Record { + const out: Record = {}; + for (const [sig, id] of [...idBySig.entries()]) { + const m = /^(.*?)((?:\.)+)$/.exec(sig); + if (!m) continue; + const host = idBySig.get(m[1] as string); + if (!host) continue; // module-level anonymous callable — no resolvable pre-2.1.0 id + const last = /$/.exec(sig) as RegExpExecArray; + out[`${host}@${last[1]}:${last[2]}`] = { id, kind: "callable" }; + } + for (const [sig, sc] of Object.entries(app.synthesized_callables ?? {})) { + if (idBySig.has(sig)) continue; // the tree names it now + const m = /^(.*):?$/.exec(sig); + const enclosing = m ? idBySig.get(m[1] as string) : undefined; + const id = m && enclosing ? `${enclosing}@${m[2]}:${m[3]}` : `${appId}/@synthetic/${encodeURIComponent(sig)}`; + idBySig.set(sig, id); + out[id] = { + id, + kind: "callable", + name: sc.name, + path: sc.path, + span: { start: [sc.start_line, sc.start_column], end: [sc.start_line, sc.start_column], bytes: [0, 0] }, + }; + } + return out; +} diff --git a/src/schema/ids.ts b/src/schema/ids.ts new file mode 100644 index 0000000..860f0b5 --- /dev/null +++ b/src/schema/ids.ts @@ -0,0 +1,38 @@ +/** + * Canonical `can://` id construction for schema v2 (durable ids, ≥ callable depth) and the + * member-key rule for the tree's named maps. Pure functions, no runtime imports — mirrors + * python's `codeanalyzer/schema/ids.py`. + * + * Ids embed the app name (`--app-name`, per-invocation), while the symbol table round-trips the + * analysis cache across runs — so ids are NEVER baked at build time; `assignIds` stamps every + * node fresh each run (see assignIds.ts). + */ + +const LANGUAGE = "typescript"; + +export function applicationIdOf(appName: string): string { + return `can://${LANGUAGE}/${appName}`; +} + +export function moduleIdOf(appId: string, fileKey: string): string { + return `${appId}/${fileKey}`; +} + +/** The module/signature prefix: the file key without its TS/JS extension. */ +export function modulePrefixOf(fileKey: string): string { + return fileKey.replace(/\.d\.ts$/, "").replace(/\.(tsx|ts|jsx|js|mts|cts|mjs|cjs)$/, ""); +} + +/** The containment-path id of a descendant, derived from its dotted signature. */ +export function idFromSig(moduleId: string, modulePrefix: string, sig: string): string { + const tail = sig.startsWith(`${modulePrefix}.`) ? sig.slice(modulePrefix.length + 1) : sig; + return `${moduleId}/${tail.split(".").join("/")}`; +} + +/** The map key for a callable/type within its parent: the last signature segment (+ accessor tag). */ +export function memberKey(sig: string, accessorKind?: string | null): string { + const seg = sig.split(".").pop() ?? sig; + if (accessorKind === "getter") return `${seg}#get`; + if (accessorKind === "setter") return `${seg}#set`; + return seg; +} diff --git a/src/schema/index.ts b/src/schema/index.ts index fde1c87..828c38b 100644 --- a/src/schema/index.ts +++ b/src/schema/index.ts @@ -3,3 +3,4 @@ export * from "./schema"; export * from "./signatures"; export * from "./graphs"; +export * from "./emit"; diff --git a/src/schema/l1Body.ts b/src/schema/l1Body.ts new file mode 100644 index 0000000..903c562 --- /dev/null +++ b/src/schema/l1Body.ts @@ -0,0 +1,65 @@ +/** + * L1 body population — python's `l1_body.py`: materialize each callable's `body{}` `call` nodes + * from the INTERNAL `call_sites`, `callee: null` (the sanctioned null→id refinement happens at + * L2, l2Callees.ts). + * + * Rebuilds `body{}` WHOLESALE every run and deletes the derived edge lists — body, cfg/cdg/ddg/ + * summary, and callee resolution are per-run projections (they embed per-run ids and per-level + * depth), while `call_sites` are the cached source of truth. That wholesale rebuild is what makes + * the whole pass chain idempotent across repeated emissions at different levels. + */ + +import type { AnalysisInternal, TSBodyNode, TSCallable, TSCallsite, TSModule } from "./schema"; +import { forEachCallable } from "./schema"; + +/** + * The body key of each call site, in recording order: `line:col`, disambiguated `/2`, `/3`, … + * when chained calls share a start position. The SINGLE definition of the key sequence — l1Body + * builds with it and l2Callees re-derives the same pairing from it. + */ +export function* callBodyKeys(sites: TSCallsite[]): Generator<[string, TSCallsite]> { + const used = new Set(); + for (const cs of sites) { + const base = `${cs.start_line}:${cs.start_column}`; + let key = base; + for (let k = 2; used.has(key); k++) key = `${base}/${k}`; + used.add(key); + yield [key, cs]; + } +} + +function callNodeOf(cs: TSCallsite): TSBodyNode { + return { + kind: "call", + span: { + start: [cs.start_line, cs.start_column], + end: [cs.end_line, cs.end_column], + bytes: cs.bytes ?? [0, 0], + }, + callee: null, + method_name: cs.method_name, + ...(cs.receiver_expr != null ? { receiver_expr: cs.receiver_expr } : {}), + ...(cs.receiver_type != null ? { receiver_type: cs.receiver_type } : {}), + argument_types: cs.argument_types, + type_arguments: cs.type_arguments, + ...(cs.return_type != null ? { return_type: cs.return_type } : {}), + is_constructor_call: cs.is_constructor_call, + is_optional_chain: cs.is_optional_chain, + }; +} + +function resetCallable(c: TSCallable): void { + const body: Record = {}; + for (const [key, cs] of callBodyKeys(c.call_sites)) body[key] = callNodeOf(cs); + c.body = body; + delete c.cfg; + delete c.cdg; + delete c.ddg; + delete c.summary; +} + +export function populateL1Body(app: AnalysisInternal): void { + for (const mod of Object.values(app.symbol_table) as TSModule[]) { + forEachCallable(mod, resetCallable); + } +} diff --git a/src/schema/l2Callees.ts b/src/schema/l2Callees.ts new file mode 100644 index 0000000..7ab3fa0 --- /dev/null +++ b/src/schema/l2Callees.ts @@ -0,0 +1,44 @@ +/** + * L2 refinement passes — python's `l2_callees.py` + `call_graph_ids.py`: + * + * - `backfillCallees`: fill each L1 `call` body node's `callee` (null → id) from the call + * site's resolver-backfilled `callee_signature`. A declared target becomes its can:// id via + * `idBySig` (which, run after the homing pass, also names external and synthesized targets); + * an unresolved call site keeps the sanctioned `callee: null`. + * - `reidentifyCallGraph`: rewrite the provider edge list onto can:// endpoints in the wire + * shape ({src, dst, prov, weight}); endpoints with no id home are collected as `dangling` + * (the L2 no-dangling gate; should be empty) and their edges dropped. + */ + +import type { AnalysisInternal, TSCallEdge, TSCallGraphEdge, TSModule } from "./schema"; +import { forEachCallable } from "./schema"; +import { callBodyKeys } from "./l1Body"; + +export function backfillCallees(app: AnalysisInternal, idBySig: Map): void { + for (const mod of Object.values(app.symbol_table) as TSModule[]) { + forEachCallable(mod, (c) => { + for (const [key, cs] of callBodyKeys(c.call_sites)) { + if (!cs.callee_signature) continue; + const node = c.body[key]; + if (!node || node.kind !== "call") continue; + node.callee = idBySig.get(cs.callee_signature) ?? null; + } + }); + } +} + +export function reidentifyCallGraph( + edges: TSCallEdge[], + idBySig: Map, + dangling: string[], +): TSCallGraphEdge[] { + const out: TSCallGraphEdge[] = []; + for (const e of edges) { + const src = idBySig.get(e.source); + const dst = idBySig.get(e.target); + if (!src) dangling.push(e.source); + if (!dst) dangling.push(e.target); + if (src && dst) out.push({ src, dst, prov: e.provenance, weight: e.weight }); + } + return out; +} diff --git a/src/schema/schema.ts b/src/schema/schema.ts index df95f0f..7587bb0 100644 --- a/src/schema/schema.ts +++ b/src/schema/schema.ts @@ -1,23 +1,33 @@ /** - * The canonical CLDK analysis schema for TypeScript. + * The canonical CLDK analysis schema for TypeScript — the NATIVE model. The stages build this + * shape directly (schema v2, canonical-schema.md): one additive containment tree of nodes + * (`id` / `kind` / `span` / named child maps) that `analysis.json` and the Neo4j projection both + * emit. There is no second model and no emit-time reshape: what the builders construct is what + * the wire carries, minus the INTERNAL fields listed below (stripped at serialization). * - * Mirrors the identity-only Python schema (codeanalyzer-python/.../py_schema.py) field for - * field on the invariant spine — `TSApplication { symbol_table, call_graph, external_symbols }`, - * `Module → Class/Callable` nesting, identity-only `TSCallEdge` whose `source`/`target` are - * bare signature strings — and extends it at the leaves with TypeScript-native node kinds - * (interface / type-alias / enum / namespace) and typed fields (generics, modifiers, ...). - * See SCHEMA_DECISIONS.md. + * Mirrors python's `codeanalyzer/schema/py_schema.py` field-for-field on the invariant spine — + * `symbol_table{module → types{}/functions{}/fields{}}`, `type → callables{}/fields{}`, + * `callable → body{}` — and extends it at the leaves with TypeScript-native node kinds + * (interface / type_alias / enum / namespace) and typed fields (generics, modifiers, ...). + * + * Conventions: + * - A fact is PRESENT or ABSENT; never `null`. Optional fields are omitted, not nulled. The one + * sanctioned null is a `call` body node's `callee` at L1 (refined null→id at L2). + * - `id` fields are stamped per-run by `assignIds` (ids embed `--app-name`; the cached tree must + * stay app-name-free). Builders initialize them to "". + * - INTERNAL fields (never on the wire; serialize.ts strips them by key): `call_sites`, + * `abs_path`, `content_hash`, `last_modified`, `file_size`. They exist for the call-graph + * resolver, the dataflow join, and the analysis cache. * * All field names are snake_case so `JSON.stringify` emits keys the SDK Pydantic models parse. - * The matching Pydantic models live in python-sdk/cldk/models/typescript/models.py and MUST be - * co-evolved with this file. */ +import { modulePrefixOf } from "./ids"; + // ---------------------------------------------------------------------------------------------- -// Span (schema-v2) — the one universal attribute. `bytes` are char offsets into the owning -// module's `source` blob, so `source.slice(bytes[0], bytes[1])` reproduces the node's text -// (what the per-node `code` field used to hold). `start`/`end` are [line, column], 1-based, for -// display and addressing. Captured on every container node during the AST walk (builders.ts). +// Span — the one universal attribute. `bytes` are char offsets into the owning module's `source` +// blob, so `source.slice(bytes[0], bytes[1])` reproduces the node's text. `start`/`end` are +// [line, column], 1-based. // ---------------------------------------------------------------------------------------------- export interface TSSpan { @@ -27,13 +37,13 @@ export interface TSSpan { } // ---------------------------------------------------------------------------------------------- -// Leaf models +// Leaf models (wire shapes — flat line/col ints are part of the wire here) // ---------------------------------------------------------------------------------------------- export interface TSImport { module: string; // the module specifier, e.g. "./user" or "@nestjs/common" name: string; // the imported binding (or "" for side-effect imports / "*" for namespace) - alias: string | null; + alias?: string; is_type_only: boolean; // `import type { X } ...` import_kind: "named" | "default" | "namespace" | "side_effect"; start_line: number; @@ -43,9 +53,9 @@ export interface TSImport { } export interface TSExport { - module: string | null; // re-export source, e.g. "./user"; null for `export { x }` + module?: string; // re-export source, e.g. "./user"; absent for `export { x }` name: string; // exported name ("*" for `export * from`) - alias: string | null; + alias?: string; is_type_only: boolean; export_kind: "named" | "default" | "namespace" | "re_export"; start_line: number; @@ -63,25 +73,9 @@ export interface TSComment { end_column: number; } -export interface TSVariableDeclaration { - span?: TSSpan; // schema-v2 precise span - name: string; - type: string | null; - initializer: string | null; - value: unknown | null; - scope: "module" | "namespace" | "class" | "function" | "block"; - declaration_kind: "const" | "let" | "var" | "using" | "unknown"; - is_readonly: boolean; - is_exported: boolean; - start_line: number; - end_line: number; - start_column: number; - end_column: number; -} - export interface TSDecorator { name: string; // locally written name, e.g. "Get" - qualified_name: string | null; // checker-resolved FQN when available + qualified_name?: string; // checker-resolved FQN when available positional_arguments: string[]; // raw source fragments keyword_arguments: Record; // object-literal args flattened to key→source start_line: number; @@ -92,18 +86,18 @@ export interface TSDecorator { export interface TSTypeParameter { name: string; - constraint: string | null; // the `extends ...` clause text - default: string | null; // the `= ...` clause text + constraint?: string; // the `extends ...` clause text + default?: string; // the `= ...` clause text } export interface TSCallableParameter { name: string; - type: string | null; - default_value: string | null; + type?: string; + default_value?: string; is_optional: boolean; is_rest: boolean; is_readonly: boolean; // parameter property `constructor(readonly x: T)` - accessibility: string | null; // parameter property visibility (NestJS DI / TS shorthand) + accessibility?: string; // parameter property visibility (NestJS DI / TS shorthand) decorators: TSDecorator[]; // param decorators (e.g. @Param('id')) start_line: number; end_line: number; @@ -111,25 +105,114 @@ export interface TSCallableParameter { end_column: number; } +export interface TSOverloadSignature { + parameters: TSCallableParameter[]; + return_type?: string; + type_parameters: TSTypeParameter[]; + start_line: number; + end_line: number; +} + +/** + * INTERNAL — a recorded call site. Never on the wire (the wire's view is the `call` node in the + * owning callable's `body{}`, built per-run by the l1Body pass). Kept on the callable because the + * call-graph resolver joins on it (span-matched to the AST) and the cache round-trips it. + * `callee_signature` is backfilled in place by the tsc resolver. + */ export interface TSCallsite { method_name: string; - receiver_expr: string | null; - receiver_type: string | null; + receiver_expr?: string; + receiver_type?: string; argument_types: string[]; type_arguments: string[]; // explicit call type args, foo() - return_type: string | null; - callee_signature: string | null; // null when recorded; backfilled by the resolver call graph + return_type?: string; + callee_signature?: string; // absent when recorded; backfilled by the resolver call graph is_constructor_call: boolean; // `new X()` is_optional_chain: boolean; // `a?.b()` start_line: number; start_column: number; end_line: number; end_column: number; - bytes?: [number, number]; // schema-v2: char offsets [start, end] into module.source + bytes: [number, number]; // char offsets [start, end] into module.source +} + +// ---------------------------------------------------------------------------------------------- +// Body nodes — a callable's `body{}` map, keyed by local id (`line:col`, or `@tag` synthetic). +// L1: `call` nodes; L3 adds statements + @entry/@exit; L4 adds formal/actual param vertices. +// ---------------------------------------------------------------------------------------------- + +export interface TSBodyNode { + kind: string; // "call" | "statement" | "entry" | "exit" | "formal_in" | "actual_in" | … + span?: TSSpan; + callee?: string | null; // `call` nodes: null at L1, refined to an id at L2 (the one sanctioned null) + of?: string; // synthetic param vertices: the flowed name ("arg0", "$ret", a global path) + parent?: string; // actual_in/actual_out: the anchoring call-site statement's local id + // call-node attributes (copied from the recorded call site by the l1Body pass) + method_name?: string; + receiver_expr?: string; + receiver_type?: string; + argument_types?: string[]; + type_arguments?: string[]; + return_type?: string; + is_constructor_call?: boolean; + is_optional_chain?: boolean; +} + +// ---------------------------------------------------------------------------------------------- +// Intra-callable edge lists (L3/L4), bare local-id endpoints +// ---------------------------------------------------------------------------------------------- + +export interface TSCfgEdge { + src: string; + dst: string; + kind: string; +} +export interface TSCdgEdge { + src: string; + dst: string; +} +export interface TSDdgEdge { + src: string; + dst: string; + var?: string; + prov: string[]; // "reaching-defs" (L3 syntactic; sanctioned additive token) / "points-to" (L4) +} +export interface TSSummaryEdge { + src: string; + dst: string; + var?: string; +} + +// ---------------------------------------------------------------------------------------------- +// Field — module-level binding, class attribute / interface property, or enum member. +// One open-ish shape: each origin sets its own subset (matching what the origin declares). +// ---------------------------------------------------------------------------------------------- + +export interface TSField { + id: string; // `${parentId}/${name}` — stamped per-run by assignIds + kind: "field"; + span?: TSSpan; // absent for constructor parameter properties + name: string; + type?: string; + // module/namespace variable + initializer?: string; + scope?: "module" | "namespace"; + declaration_kind?: "const" | "let" | "var" | "using" | "unknown"; + is_exported?: boolean; + // class attribute / interface property + comments?: TSComment[]; + decorators?: TSDecorator[]; + accessibility?: string; + is_static?: boolean; + is_readonly?: boolean; + is_optional?: boolean; + is_abstract?: boolean; + // enum member + value?: string; // initializer text or computed const value } // ---------------------------------------------------------------------------------------------- -// Callable (function / method / constructor / accessor / arrow) +// Callable (function / method / constructor / accessor / arrow / function expression) // ---------------------------------------------------------------------------------------------- export type TSCallableKind = @@ -141,36 +224,19 @@ export type TSCallableKind = | "arrow" | "function_expression"; -export interface TSOverloadSignature { - parameters: TSCallableParameter[]; - return_type: string | null; - type_parameters: TSTypeParameter[]; - start_line: number; - end_line: number; -} - export interface TSCallable { - span?: TSSpan; // schema-v2 precise span (line/col + char offsets into module.source) + id: string; // can:// containment id — stamped per-run by assignIds + kind: TSCallableKind; + span: TSSpan; name: string; - path: string; // file path of the declaration - signature: string; // e.g. src/user.UserService.getUser — the edge id + signature: string; // e.g. src/user.UserService.getUser — the internal join key comments: TSComment[]; decorators: TSDecorator[]; parameters: TSCallableParameter[]; type_parameters: TSTypeParameter[]; - return_type: string | null; - code: string | null; - start_line: number; - end_line: number; - code_start_line: number; - call_sites: TSCallsite[]; - inner_callables: Record; - inner_classes: Record; - local_variables: TSVariableDeclaration[]; + return_type?: string; cyclomatic_complexity: number; - // --- TypeScript-native typed fields --- - kind: TSCallableKind; - accessibility: string | null; // public | private | protected | null + accessibility?: string; // public | private | protected is_static: boolean; is_abstract: boolean; is_async: boolean; @@ -180,172 +246,87 @@ export interface TSCallable { is_exported: boolean; is_ambient: boolean; // `declare` is_implicit: boolean; // synthesized default constructor - accessor_kind: string | null; // getter | setter | null + accessor_kind?: string; // getter | setter overload_signatures: TSOverloadSignature[]; + body: Record; // L1: `call` nodes (l1Body pass); L3+: full statements + callables?: Record; // nested callables (closures) — present only when non-empty + types?: Record; // nested (local) classes — present only when non-empty + cfg?: TSCfgEdge[]; // L3 + cdg?: TSCdgEdge[]; // L3 + ddg?: TSDdgEdge[]; // L3→L4 + summary?: TSSummaryEdge[]; // L4 + // INTERNAL (stripped from the wire) + abs_path: string; // ABSOLUTE file path of the declaration — the resolver's AST-index key + call_sites: TSCallsite[]; } // ---------------------------------------------------------------------------------------------- -// Class attribute -// ---------------------------------------------------------------------------------------------- - -export interface TSClassAttribute { - span?: TSSpan; // schema-v2 precise span - name: string; - type: string | null; - comments: TSComment[]; - decorators: TSDecorator[]; - initializer: string | null; - accessibility: string | null; - is_static: boolean; - is_readonly: boolean; - is_optional: boolean; - is_abstract: boolean; - start_line: number; - end_line: number; -} - -// ---------------------------------------------------------------------------------------------- -// Class -// ---------------------------------------------------------------------------------------------- - -export interface TSClass { - span?: TSSpan; // schema-v2 precise span - name: string; - signature: string; // e.g. src/user.UserService - comments: TSComment[]; - code: string | null; - decorators: TSDecorator[]; - base_classes: string[]; // spine: union of extends + implements (signature strings) - implements_types: string[]; // typed split: just the implemented interfaces - type_parameters: TSTypeParameter[]; - methods: Record; - attributes: Record; - inner_classes: Record; - is_abstract: boolean; - is_exported: boolean; - is_ambient: boolean; - start_line: number; - end_line: number; -} - -// ---------------------------------------------------------------------------------------------- -// Interface (TS node kind) -// ---------------------------------------------------------------------------------------------- - -export interface TSInterface { - span?: TSSpan; // schema-v2 precise span - name: string; - signature: string; - comments: TSComment[]; - code: string | null; - base_classes: string[]; // extended interfaces (signature strings) - type_parameters: TSTypeParameter[]; - methods: Record; // bodiless - properties: Record; - call_signatures: string[]; // raw text of call/construct signatures - index_signatures: string[]; // raw text of `[key: string]: T` - is_exported: boolean; - is_ambient: boolean; - start_line: number; - end_line: number; -} - -// ---------------------------------------------------------------------------------------------- -// Enum (TS node kind) +// Type — one node with a single `kind`; the buckets it populates depend on that kind: +// class / interface → callables{} + fields{}; enum → fields{}; type_alias → (leaf); +// namespace → types{} + functions{} + fields{} (a sub-file scope, same buckets as a module). // ---------------------------------------------------------------------------------------------- -export interface TSEnumMember { - span?: TSSpan; // schema-v2 precise span - name: string; - value: string | null; // initializer text or computed const value - start_line: number; - end_line: number; -} +export type TSTypeKind = "class" | "interface" | "enum" | "type_alias" | "namespace"; -export interface TSEnum { - span?: TSSpan; // schema-v2 precise span +export interface TSType { + id: string; // stamped per-run by assignIds + kind: TSTypeKind; + span: TSSpan; name: string; signature: string; comments: TSComment[]; - code: string | null; - members: TSEnumMember[]; - is_const: boolean; is_exported: boolean; is_ambient: boolean; - start_line: number; - end_line: number; + // class / interface / enum / namespace members + callables?: Record; + fields?: Record; + types?: Record; // namespace only + functions?: Record; // namespace only + // class + decorators?: TSDecorator[]; + base_classes?: string[]; // spine: union of extends + implements (signature strings) + implements_types?: string[]; // typed split: just the implemented interfaces + is_abstract?: boolean; + // class / interface / type_alias generics + type_parameters?: TSTypeParameter[]; + // interface + call_signatures?: string[]; // raw text of call/construct signatures + index_signatures?: string[]; // raw text of `[key: string]: T` + // enum + is_const?: boolean; + // type_alias + aliased_type?: string; // the RHS type text + // Heritage projection (resolved-only), stamped per-run by the heritage pass: + extends_ids?: string[]; // resolved can:// id(s) of the extended class/interface(s) + implements_ids?: string[]; // resolved can:// id(s) of implemented interfaces (classes only) } // ---------------------------------------------------------------------------------------------- -// Type alias (TS node kind) -// ---------------------------------------------------------------------------------------------- - -export interface TSTypeAlias { - span?: TSSpan; // schema-v2 precise span - name: string; - signature: string; - comments: TSComment[]; - code: string | null; - aliased_type: string; // the RHS type text - type_parameters: TSTypeParameter[]; - is_exported: boolean; - is_ambient: boolean; - start_line: number; - end_line: number; -} - -// ---------------------------------------------------------------------------------------------- -// Namespace (TS node kind) — recursive container, same shape as Module's declaration buckets -// ---------------------------------------------------------------------------------------------- - -export interface TSNamespace { - span?: TSSpan; // schema-v2 precise span - name: string; - signature: string; - comments: TSComment[]; - classes: Record; - interfaces: Record; - enums: Record; - type_aliases: Record; - functions: Record; - variables: TSVariableDeclaration[]; - namespaces: Record; - is_exported: boolean; - is_ambient: boolean; - start_line: number; - end_line: number; -} - -// ---------------------------------------------------------------------------------------------- -// Module (compilation unit / file) +// Module (compilation unit / file) — a scope holding types, functions, and free bindings // ---------------------------------------------------------------------------------------------- export interface TSModule { - span?: TSSpan; // schema-v2 precise span (whole file) - source?: string | null; // schema-v2: full file text; every node's text slices off this - file_path: string; - module_name: string; // the file key minus extension (== signature prefix) + id: string; // can://// — stamped per-run by assignIds + kind: "module"; + span: TSSpan; // whole file + source: string; // full file text, once; every node's text slices off this imports: TSImport[]; exports: TSExport[]; comments: TSComment[]; - classes: Record; - interfaces: Record; - enums: Record; - type_aliases: Record; - functions: Record; - namespaces: Record; - variables: TSVariableDeclaration[]; - // TS file flags + types: Record; // classes/interfaces/enums/type-aliases/namespaces + functions: Record; // free functions + fields: Record; // module-level const/let/var is_tsx: boolean; is_declaration_file: boolean; - // caching metadata - content_hash: string | null; - last_modified: number | null; - file_size: number | null; + // INTERNAL — caching metadata (stripped from the wire) + content_hash?: string; + last_modified?: number; + file_size?: number; } // ---------------------------------------------------------------------------------------------- -// Call-graph edge (identity-only) +// Call-graph edge (identity-only, provider output; endpoints are signature strings until the +// call-graph-ids pass rewrites them onto can:// ids at L2) // ---------------------------------------------------------------------------------------------- export const CALL_DEP = "CALL_DEP" as const; @@ -359,10 +340,6 @@ export interface TSCallEdge { tags: Record; } -// ---------------------------------------------------------------------------------------------- -// Application (root) -// ---------------------------------------------------------------------------------------------- - // ---------------------------------------------------------------------------------------------- // External (phantom) symbol — a synthetic stub for a call target OUTSIDE the project (an imported // library / Node builtin). Lets the call graph point at external callees (WALA-style phantom @@ -388,13 +365,104 @@ export interface TSSynthesizedCallable { start_column: number; } -export interface TSApplication { +/** + * The analyzer's INTERNAL working set: the live tree plus the signature-keyed provider outputs. + * `finalizeAnalysis` consumes it and assembles the wire (`TSAnalysis`); it is never serialized + * itself. (The program-graph IR travels separately — see AnalysisResult.) + */ +export interface AnalysisInternal { symbol_table: Record; call_graph: TSCallEdge[]; external_symbols: Record; synthesized_callables: Record; - /** Level-3 CFG/PDG/SDG section — present only at `-a 3` (see schema/graphs.ts). */ - program_graphs?: import("./graphs").ProgramGraphs; +} + +// ---------------------------------------------------------------------------------------------- +// The wire: envelope → application root → cross-callable edges (what analysis.json IS, and what +// the Neo4j projection consumes) +// ---------------------------------------------------------------------------------------------- + +export interface TSAnalysis { + schema_version: string; // "2.1.0" + language: string; // "typescript" + max_level: number; // highest level populated; consumers read this, not key-sniffing + k_limit?: number; // access-path depth bound for the L3/L4 dataflow (present at L3+) + analyzer: TSAnalyzer; // which analyzer produced this artifact, and at what version + application: TSApplication; +} + +/** Analyzer identity — lets consumers correlate an `analysis.json` with the tool/version that emitted it. */ +export interface TSAnalyzer { + name: string; // "codeanalyzer-typescript" + version: string; // ANALYZER_VERSION (src/utils/version.ts) +} + +/** The application ROOT node (python's PyApplication): the containment tree + app-scope overlays. */ +export interface TSApplication { + id: string; // can:/// + kind: "application"; + symbol_table: Record; // keyed by project-relative POSIX path (with extension) + call_graph: TSCallGraphEdge[]; // L2 — callable → callable (empty at L1) + param_in: TSParamEdge[]; // L4 (empty until L4) + param_out: TSParamEdge[]; // L4 + // TS-additive (parity): edge endpoints outside the containment tree need an id home. + external_symbols?: Record; // L2 — library call targets, keyed by id + // L2 — 2.1.0 compatibility index: pre-2.1.0 anonymous-callable id → the tree id that replaced + // it. Entries whose key equals their own `id` are the residual fallback nodes for signatures no + // provider could name. + synthesized_callables?: Record; +} + +/** A wire call-graph edge: identity-only, can:// endpoints (l2Callees re-identifies onto these). */ +export interface TSCallGraphEdge { + src: string; + dst: string; + prov: string[]; // provenance, e.g. ["tsc"], ["jelly"] + weight: number; +} + +export interface TSParamEdge { + src: string; + dst: string; + var?: string; +} + +// ---------------------------------------------------------------------------------------------- +// Tree walkers — the one place the containment reach is defined (shared by the id/body/callee +// passes, the call-graph resolver, and the dataflow join). +// ---------------------------------------------------------------------------------------------- + +/** Depth-first over every callable in a module: free functions, type members (class/interface + * accessors and methods, namespace functions), and everything nested inside callables. */ +export function forEachCallable(mod: TSModule, fn: (c: TSCallable) => void): void { + const visitCallable = (c: TSCallable): void => { + fn(c); + for (const nested of Object.values(c.callables ?? {})) visitCallable(nested); + for (const t of Object.values(c.types ?? {})) visitType(t); + }; + const visitType = (t: TSType): void => { + for (const m of Object.values(t.callables ?? {})) visitCallable(m); + for (const f of Object.values(t.functions ?? {})) visitCallable(f); // namespace + for (const nt of Object.values(t.types ?? {})) visitType(nt); // namespace + }; + for (const f of Object.values(mod.functions ?? {})) visitCallable(f); + for (const t of Object.values(mod.types ?? {})) visitType(t); +} + +/** Depth-first over every type node in a module, including types nested inside callables. */ +export function forEachType(mod: TSModule, fn: (t: TSType) => void): void { + const visitType = (t: TSType): void => { + fn(t); + for (const nt of Object.values(t.types ?? {})) visitType(nt); + for (const m of Object.values(t.callables ?? {})) visitCallable(m); + for (const f of Object.values(t.functions ?? {})) visitCallable(f); + }; + const visitCallable = (c: TSCallable): void => { + for (const nested of Object.values(c.callables ?? {})) visitCallable(nested); + for (const t of Object.values(c.types ?? {})) visitType(t); + }; + for (const t of Object.values(mod.types ?? {})) visitType(t); + for (const f of Object.values(mod.functions ?? {})) visitCallable(f); } // ============================================================================================== @@ -407,8 +475,7 @@ export interface TSApplication { */ export function fileKeyOf(absPath: string, projectRoot: string): { fileKey: string; modulePrefix: string } { const rel = toPosix(relativePath(projectRoot, absPath)); - const modulePrefix = stripTsExtension(rel); - return { fileKey: rel, modulePrefix }; + return { fileKey: rel, modulePrefix: modulePrefixOf(rel) }; } /** @@ -424,16 +491,12 @@ export function constructorSignatureOf(classSignature: string): string { return `${classSignature}.constructor`; } -// --- small path helpers (kept dependency-free so schema.ts has no runtime imports) --- +// --- small path helpers (kept dependency-light so schema.ts has no runtime deps beyond ids) --- function toPosix(p: string): string { return p.replace(/\\/g, "/"); } -function stripTsExtension(relPosix: string): string { - return relPosix.replace(/\.d\.ts$/, "").replace(/\.(tsx|ts|jsx|js|mts|cts|mjs|cjs)$/, ""); -} - function relativePath(from: string, to: string): string { const a = toPosix(from).replace(/\/+$/, "").split("/"); const b = toPosix(to).split("/"); diff --git a/src/schema/v2/emit.ts b/src/schema/v2/emit.ts deleted file mode 100644 index 2d9c704..0000000 --- a/src/schema/v2/emit.ts +++ /dev/null @@ -1,436 +0,0 @@ -/** - * v1 in-memory model → schema-v2 `analysis.json`. A pure transform: it reshapes the tree, - * assigns `can://` ids, and renames edge fields — it never re-parses. The parsing/resolution - * guts (buildSymbolTable, call graph, dataflow) are untouched; only this serialization is new. - * - * Level scope: this emits **L1** — the containment tree (modules → types/functions/fields → - * callables → `body` call nodes) with `can://` ids, precise spans, and per-module `source`. - * `call_graph`/`param_in`/`param_out` are emitted empty here and populated by later levels - * (`call_graph` at L2). The `idBySig` map built during the walk is what L2 will use to rewrite - * edges, and doubles as the L1 id-uniqueness gate. - */ - -import * as path from "node:path"; -import type { AnalysisOptions } from "../../options"; -import { ANALYZER_VERSION } from "../../utils/version"; -import type { - TSApplication, - TSCallable, - TSClass, - TSClassAttribute, - TSEnum, - TSInterface, - TSModule, - TSNamespace, - TSSpan, - TSTypeAlias, - TSVariableDeclaration, -} from "../schema"; -import { applyDataflow } from "./dataflow"; -import type { V2Application, V2BodyNode, V2CallEdge, V2Callable, V2External, V2Field, V2Module, V2Node, V2Root, V2Type } from "./model"; - -const LANGUAGE = "typescript"; -const SCHEMA_VERSION = "2.1.0"; -const ANALYZER_NAME = "codeanalyzer-typescript"; -/** Highest analysis level this emitter populates today (L1 tree, L2 call graph, L3/L4 dataflow). */ -const MAX_IMPLEMENTED = 4; - -/** Structural / replaced / cache-metadata keys stripped before carrying language-native attrs. */ -const DROP = new Set([ - "code", - "span", - "path", - "file_path", - "module_name", - "start_line", - "end_line", - "start_column", - "end_column", - "code_start_line", - "bytes", - "methods", - "attributes", - "members", - "properties", - "classes", - "interfaces", - "enums", - "type_aliases", - "namespaces", - "functions", - "variables", - "call_sites", - "inner_callables", - "inner_classes", - "local_variables", - "content_hash", - "last_modified", - "file_size", - "callee_signature", -]); - -/** - * Recursively drop `null`/`undefined` — the canonical convention is "a fact is present or absent; - * there is no null" (the one exception, a call node's `callee: null`, is set explicitly outside - * carry()). Nested nulls (e.g. a parameter's `default_value`, an import's `alias`) go too. - */ -function pruneNulls(v: unknown): unknown { - if (Array.isArray(v)) return v.map(pruneNulls); - if (v && typeof v === "object") { - const out: Record = {}; - for (const [k, val] of Object.entries(v)) { - if (val === null || val === undefined) continue; - out[k] = pruneNulls(val); - } - return out; - } - return v; -} - -/** Copy a v1 node's language-native attributes (everything not structural/replaced), null-pruned. */ -function carry(node: Record): Record { - const out: Record = {}; - for (const [k, v] of Object.entries(node)) { - if (DROP.has(k) || v === null || v === undefined) continue; - out[k] = pruneNulls(v); - } - return out; -} - -/** The containment-path id of a descendant, derived from its dotted signature. */ -function idFromSig(moduleId: string, modulePrefix: string, sig: string): string { - const tail = sig.startsWith(`${modulePrefix}.`) ? sig.slice(modulePrefix.length + 1) : sig; - return `${moduleId}/${tail.split(".").join("/")}`; -} - -/** The map key for a callable/type within its parent: the last signature segment (+ accessor tag). */ -function memberKey(sig: string, accessorKind?: string | null): string { - const seg = sig.split(".").pop() ?? sig; - if (accessorKind === "getter") return `${seg}#get`; - if (accessorKind === "setter") return `${seg}#set`; - return seg; -} - -/** A type's heritage signatures, resolved to `can://` ids once the whole tree walk registers them. */ -interface PendingHeritage { - node: V2Type; - extendsSigs: string[]; // class: the extended base class; interface: extended interface(s) - implementsSigs: string[]; // class only: implemented interfaces -} - -/** State shared across the whole tree walk (edge-rewriting + gating). */ -interface SharedState { - idBySig: Map; - collisions: string[]; - pendingCallees: Array<{ node: V2BodyNode; calleeSig: string | null }>; // backfilled at L2 - pendingHeritage: PendingHeritage[]; // resolved sig→id once the whole tree walk completes - callableBySig: Map; // locates each callable's node for the L3/L4 dataflow pass - level: number; -} - -interface Ctx extends SharedState { - moduleId: string; - modulePrefix: string; -} - -function register(ctx: Ctx, sig: string, id: string): void { - if (ctx.idBySig.has(sig) && ctx.idBySig.get(sig) !== id) ctx.collisions.push(sig); - ctx.idBySig.set(sig, id); -} - -// ---------------------------------------------------------------------------------------------- -// body (L1: call sites → `call` nodes keyed by line:col) -// ---------------------------------------------------------------------------------------------- - -function toBody(c: TSCallable, ctx: Ctx): Record { - const body: Record = {}; - for (const cs of c.call_sites ?? []) { - const span: TSSpan = { - start: [cs.start_line, cs.start_column], - end: [cs.end_line, cs.end_column], - bytes: cs.bytes ?? [0, 0], - }; - let key = `${cs.start_line}:${cs.start_column}`; - for (let k = 2; key in body; k++) key = `${cs.start_line}:${cs.start_column}/${k}`; // chained calls share a start - const node: V2BodyNode = { ...carry(cs as unknown as Record), kind: "call", span, callee: null }; - body[key] = node; - // callee stays null at L1; backfilled to a can:// id at L2 once external/synth ids are homed. - ctx.pendingCallees.push({ node, calleeSig: cs.callee_signature ?? null }); - } - return body; -} - -// ---------------------------------------------------------------------------------------------- -// callable -// ---------------------------------------------------------------------------------------------- - -function toCallable(c: TSCallable, ctx: Ctx): V2Callable { - const id = idFromSig(ctx.moduleId, ctx.modulePrefix, c.signature); - register(ctx, c.signature, id); - const node: V2Callable = { - ...carry(c as unknown as Record), - id, - kind: c.kind, - signature: c.signature, - span: c.span, - body: toBody(c, ctx), - }; - ctx.callableBySig.set(c.signature, node); // for the L3/L4 dataflow pass - const nestedCallables = c.inner_callables ?? {}; - if (Object.keys(nestedCallables).length) { - node.callables = {}; - for (const inner of Object.values(nestedCallables)) node.callables[memberKey(inner.signature, inner.accessor_kind)] = toCallable(inner, ctx); - } - const nestedClasses = c.inner_classes ?? {}; - if (Object.keys(nestedClasses).length) { - node.types = {}; - for (const cls of Object.values(nestedClasses)) node.types[memberKey(cls.signature)] = toClass(cls, ctx); - } - return node; -} - -// ---------------------------------------------------------------------------------------------- -// fields (module vars, class attributes, interface properties, enum members) -// ---------------------------------------------------------------------------------------------- - -function fieldNode(parentId: string, name: string, span: TSSpan | undefined, src: Record): V2Field { - return { ...carry(src), id: `${parentId}/${name}`, kind: "field", span }; -} - -// ---------------------------------------------------------------------------------------------- -// type kinds -// ---------------------------------------------------------------------------------------------- - -function toClass(c: TSClass, ctx: Ctx): V2Type { - const id = idFromSig(ctx.moduleId, ctx.modulePrefix, c.signature); - register(ctx, c.signature, id); - const callables: Record = {}; - for (const m of Object.values(c.methods ?? {})) callables[memberKey(m.signature, m.accessor_kind)] = toCallable(m, ctx); - const fields: Record = {}; - for (const [name, a] of Object.entries(c.attributes ?? {})) - fields[name] = fieldNode(id, name, (a as TSClassAttribute).span, a as unknown as Record); - const node: V2Type = { ...carry(c as unknown as Record), id, kind: "class", signature: c.signature, span: c.span, callables, fields }; - // `base_classes` is the union of extends + implements (schema.ts:231); subtract implements_types - // to recover just the extended base class (0 or 1 — TS classes extend at most one class). - if (c.base_classes.length) { - const extendsSigs = c.base_classes.filter((s) => !c.implements_types.includes(s)); - ctx.pendingHeritage.push({ node, extendsSigs, implementsSigs: c.implements_types }); - } - return node; -} - -function toInterface(i: TSInterface, ctx: Ctx): V2Type { - const id = idFromSig(ctx.moduleId, ctx.modulePrefix, i.signature); - register(ctx, i.signature, id); - const callables: Record = {}; - for (const m of Object.values(i.methods ?? {})) callables[memberKey(m.signature, m.accessor_kind)] = toCallable(m, ctx); - const fields: Record = {}; - for (const [name, p] of Object.entries(i.properties ?? {})) - fields[name] = fieldNode(id, name, (p as TSClassAttribute).span, p as unknown as Record); - const node: V2Type = { ...carry(i as unknown as Record), id, kind: "interface", signature: i.signature, span: i.span, callables, fields }; - // Interface heritage is extends-only (schema.ts:255) — an interface can extend other interfaces - // (or, rarely, a class's instance type), but never "implements". - if (i.base_classes.length) ctx.pendingHeritage.push({ node, extendsSigs: i.base_classes, implementsSigs: [] }); - return node; -} - -function toEnum(e: TSEnum, ctx: Ctx): V2Type { - const id = idFromSig(ctx.moduleId, ctx.modulePrefix, e.signature); - register(ctx, e.signature, id); - const fields: Record = {}; - for (const m of e.members ?? []) fields[m.name] = fieldNode(id, m.name, m.span, m as unknown as Record); - return { ...carry(e as unknown as Record), id, kind: "enum", signature: e.signature, span: e.span, fields }; -} - -function toTypeAlias(t: TSTypeAlias, ctx: Ctx): V2Type { - const id = idFromSig(ctx.moduleId, ctx.modulePrefix, t.signature); - register(ctx, t.signature, id); - return { ...carry(t as unknown as Record), id, kind: "type_alias", signature: t.signature, span: t.span }; -} - -/** Gap A: a namespace is a nested *scope* — same buckets as a module (types/functions/fields). */ -function toNamespace(ns: TSNamespace, ctx: Ctx): V2Type { - const id = idFromSig(ctx.moduleId, ctx.modulePrefix, ns.signature); - register(ctx, ns.signature, id); - const types = collectTypes(ns, ctx); - const functions: Record = {}; - for (const fn of Object.values(ns.functions ?? {})) functions[memberKey(fn.signature, fn.accessor_kind)] = toCallable(fn, ctx); - const fields: Record = {}; - for (const v of ns.variables ?? []) fields[v.name] = fieldNode(id, v.name, v.span, v as unknown as Record); - return { ...carry(ns as unknown as Record), id, kind: "namespace", signature: ns.signature, span: ns.span, types, functions, fields }; -} - -/** Merge a scope's class/interface/enum/type-alias/namespace buckets into one `types{}` map. */ -function collectTypes(scope: { classes?: Record; interfaces?: Record; enums?: Record; type_aliases?: Record; namespaces?: Record }, ctx: Ctx): Record { - const types: Record = {}; - for (const c of Object.values(scope.classes ?? {})) types[memberKey(c.signature)] = toClass(c, ctx); - for (const i of Object.values(scope.interfaces ?? {})) types[memberKey(i.signature)] = toInterface(i, ctx); - for (const e of Object.values(scope.enums ?? {})) types[memberKey(e.signature)] = toEnum(e, ctx); - for (const t of Object.values(scope.type_aliases ?? {})) types[memberKey(t.signature)] = toTypeAlias(t, ctx); - for (const ns of Object.values(scope.namespaces ?? {})) types[memberKey(ns.signature)] = toNamespace(ns, ctx); - return types; -} - -// ---------------------------------------------------------------------------------------------- -// module -// ---------------------------------------------------------------------------------------------- - -function toModule(m: TSModule, moduleId: string, shared: SharedState): V2Module { - const ctx: Ctx = { moduleId, modulePrefix: m.module_name, ...shared }; - const types = collectTypes(m, ctx); - const functions: Record = {}; - for (const fn of Object.values(m.functions ?? {})) functions[memberKey(fn.signature, fn.accessor_kind)] = toCallable(fn, ctx); - const fields: Record = {}; - for (const v of m.variables ?? []) fields[(v as TSVariableDeclaration).name] = fieldNode(moduleId, v.name, v.span, v as unknown as Record); - return { - ...carry(m as unknown as Record), - id: moduleId, - kind: "module", - source: m.source ?? "", - span: m.span, - types, - functions, - fields, - }; -} - -// ---------------------------------------------------------------------------------------------- -// L2 — edge-endpoint id homes (external library targets + first-party anonymous callbacks) -// ---------------------------------------------------------------------------------------------- - -/** External library call targets → `can://…/@external//` ids on the application root. */ -function homeExternals(app: TSApplication, appId: string, idBySig: Map): Record { - const out: Record = {}; - for (const [sig, ext] of Object.entries(app.external_symbols ?? {})) { - const id = `${appId}/@external/${ext.module}/${ext.name}`; - idBySig.set(sig, id); - out[id] = { id, kind: "external", module: ext.module, name: ext.name }; - } - return out; -} - -/** - * The compatibility index for anonymous callables (schema 2.1.0). - * - * Anonymous callables are now real nodes in the containment tree, signed positionally - * (`.`) and reachable by containment. This map is no longer a node - * registry: it maps the **pre-2.1.0 id** of each anonymous callable — `@:`, - * derived from the old `:` signature — onto the tree id that replaced it, - * so a consumer holding an old id can still resolve it. - * - * The old host was the nearest enclosing callable the old rules could name, which is recovered by - * stripping the trailing `` chain. An anonymous callable directly under a module had no - * resolvable old id (the old emitter fell back to an opaque `@synthetic/` key that encoded the - * ambiguous `:` signature, which was not unique across files) — those are - * skipped rather than reproduced. - * - * Any signature the call-graph provider still could not name is homed here too, unchanged, so the - * no-dangling rule holds even if a provider reports a function-like node the tree missed. - */ -function homeSynthesized(app: TSApplication, appId: string, idBySig: Map): Record { - const out: Record = {}; - for (const [sig, id] of [...idBySig.entries()]) { - const m = /^(.*?)((?:\.)+)$/.exec(sig); - if (!m) continue; - const host = idBySig.get(m[1] as string); - if (!host) continue; // module-level anonymous callable — no resolvable pre-2.1.0 id - const last = /$/.exec(sig) as RegExpExecArray; - out[`${host}@${last[1]}:${last[2]}`] = { id, kind: "callable" }; - } - for (const [sig, sc] of Object.entries(app.synthesized_callables ?? {})) { - if (idBySig.has(sig)) continue; // the tree names it now - const m = /^(.*):?$/.exec(sig); - const enclosing = m ? idBySig.get(m[1] as string) : undefined; - const id = m && enclosing ? `${enclosing}@${m[2]}:${m[3]}` : `${appId}/@synthetic/${encodeURIComponent(sig)}`; - idBySig.set(sig, id); - out[id] = { - id, - kind: "callable", - name: sc.name, - path: sc.path, - span: { start: [sc.start_line, sc.start_column], end: [sc.start_line, sc.start_column], bytes: [0, 0] }, - }; - } - return out; -} - -// ---------------------------------------------------------------------------------------------- -// entry point -// ---------------------------------------------------------------------------------------------- - -export interface ToV2Result { - application: V2Application; - idBySig: Map; // signature → can:// id (real callables + externals + synthesized) - collisions: string[]; // signatures that mapped to two distinct ids (L1 id-uniqueness gate) - dangling: string[]; // call-graph endpoints with no id home (L2 no-dangling gate; should be empty) -} - -export function toV2Detailed(app: TSApplication, opts: AnalysisOptions): ToV2Result { - const level = opts.analysisLevel; - const appName = (opts.appName ?? (opts.input ? path.basename(opts.input) : "") ?? "").trim() || "app"; - const appId = `can://${LANGUAGE}/${appName}`; - const idBySig = new Map(); - const collisions: string[] = []; - const pendingCallees: Array<{ node: V2BodyNode; calleeSig: string | null }> = []; - const pendingHeritage: PendingHeritage[] = []; - const callableBySig = new Map(); - const shared: SharedState = { idBySig, collisions, pendingCallees, pendingHeritage, callableBySig, level }; - - // L1 — the containment tree (registers every real callable/type id in idBySig). - const symbol_table: Record = {}; - for (const [fileKey, m] of Object.entries(app.symbol_table)) { - symbol_table[fileKey] = toModule(m, `${appId}/${fileKey}`, shared); - } - const root: V2Root = { id: appId, kind: "application", symbol_table, call_graph: [], param_in: [], param_out: [] }; - - // Resolve heritage sig → can:// id now that every first-party type is registered in idBySig - // (independent of level: types are homed during the unconditional L1 walk above). Unresolved - // (external/library) supertypes are dropped, never nulled — the "resolved-only" rule. - for (const { node, extendsSigs, implementsSigs } of pendingHeritage) { - const extendsIds = extendsSigs.map((s) => idBySig.get(s)).filter((x): x is string => x !== undefined); - const implementsIds = implementsSigs.map((s) => idBySig.get(s)).filter((x): x is string => x !== undefined); - if (extendsIds.length) node.extends_ids = extendsIds; - if (implementsIds.length) node.implements_ids = implementsIds; - } - - // L2 — home the off-tree edge endpoints, backfill `callee`, rewrite the call graph. - const dangling: string[] = []; - if (level >= 2) { - root.external_symbols = homeExternals(app, appId, idBySig); - root.synthesized_callables = homeSynthesized(app, appId, idBySig); - for (const { node, calleeSig } of pendingCallees) { - if (calleeSig) node.callee = idBySig.get(calleeSig) ?? null; - } - root.call_graph = (app.call_graph ?? []) - .map((e): V2CallEdge | null => { - const src = idBySig.get(e.source); - const dst = idBySig.get(e.target); - if (!src) dangling.push(e.source); - if (!dst) dangling.push(e.target); - return src && dst ? { src, dst, prov: e.provenance, weight: e.weight } : null; - }) - .filter((e): e is V2CallEdge => e !== null); - } - - // L3/L4 — grow body{} + cfg/cdg/ddg/summary on callables and param_in/param_out on the app. - let k_limit: number | undefined; - if (level >= 3 && app.program_graphs) { - applyDataflow(root, app, idBySig, callableBySig, level); - k_limit = app.program_graphs.k_limit; - } - - const application: V2Application = { - schema_version: SCHEMA_VERSION, - language: LANGUAGE, - max_level: Math.min(level, MAX_IMPLEMENTED), - ...(k_limit !== undefined ? { k_limit } : {}), - analyzer: { name: ANALYZER_NAME, version: ANALYZER_VERSION }, - application: root, - }; - return { application, idBySig, collisions, dangling }; -} - -/** The default L1 emitter surface: v1 app + options → schema-v2 Application. */ -export function toV2(app: TSApplication, opts: AnalysisOptions): V2Application { - return toV2Detailed(app, opts).application; -} diff --git a/src/schema/v2/index.ts b/src/schema/v2/index.ts deleted file mode 100644 index 645e29b..0000000 --- a/src/schema/v2/index.ts +++ /dev/null @@ -1,4 +0,0 @@ -// Schema v2 — the canonical additive-CPG shape and the v1→v2 emitter (see canonical-schema.md). -export * from "./model"; -export * from "./emit"; -export * from "./dataflow"; diff --git a/src/schema/v2/model.ts b/src/schema/v2/model.ts deleted file mode 100644 index efd5e50..0000000 --- a/src/schema/v2/model.ts +++ /dev/null @@ -1,137 +0,0 @@ -/** - * Schema v2 — the canonical CLDK analysis shape (the "additive CPG"): one scale-free node - * (id / kind / span / children / typed-edge overlays), emitted as `analysis.json`. This file is - * the TypeScript projection of `codeanalyzer-backend/references/canonical-schema.md`; the SDK's - * Pydantic models mirror it. It is produced by `emit.ts` as a transform of the v1 in-memory - * model (`../schema.ts`) — the parsing/resolution guts are untouched; only serialization changes. - * - * Levels are additive: L1 grows the tree to callable depth (+ `call` nodes in `body`); L2 adds - * `call_graph` and backfills `callee`; L3 grows the rest of `body` + `cfg`/`cdg`/`ddg`; L4 adds - * synthetic param vertices + `param_in`/`param_out`/`summary`. This file already declares the - * later-level slots (all optional) so the shape is stable as levels land. - */ - -import type { TSSpan } from "../schema"; - -/** The one universal attribute. `bytes` are char offsets into the owning module's `source`. */ -export type Span = TSSpan; - -// ---------------------------------------------------------------------------------------------- -// Root -// ---------------------------------------------------------------------------------------------- - -export interface V2Application { - schema_version: string; // "2.0.0" - language: string; // "typescript" - max_level: number; // highest level populated; consumers read this, not key-sniffing - k_limit?: number; // access-path depth bound for the L3/L4 dataflow (present at L3+) - analyzer: V2Analyzer; // which analyzer produced this artifact, and at what version - application: V2Root; -} - -/** Analyzer identity — lets consumers correlate an `analysis.json` with the tool/version that emitted it. */ -export interface V2Analyzer { - name: string; // "codeanalyzer-typescript" - version: string; // ANALYZER_VERSION (src/utils/version.ts) -} - -export interface V2Root { - id: string; // can:/// - kind: "application"; - symbol_table: Record; // keyed by project-relative POSIX path (with extension) - call_graph: V2CallEdge[]; // L2 — callable → callable (empty at L1) - param_in: V2ParamEdge[]; // L4 (empty until L4) - param_out: V2ParamEdge[]; // L4 - // TS-additive (parity): edge endpoints outside the containment tree need an id home. - external_symbols?: Record; // L2 — imported/library call targets, keyed by id - // L2 — 2.1.0 compatibility index: pre-2.1.0 anonymous-callable id → the tree id that replaced - // it. Anonymous callables are real nodes in the tree now; entries whose key equals their own - // `id` are the residual fallback nodes for signatures no provider could name. - synthesized_callables?: Record; -} - -/** A call target outside the project (an imported library member / builtin) — an edge endpoint, not a tree node. */ -export interface V2External extends V2Node { - kind: "external"; - module: string; // the import/require specifier, e.g. "node:fs", "express" - name: string; // the called member, e.g. "readFileSync" -} - -// ---------------------------------------------------------------------------------------------- -// Cross-callable edges (application scope) -// ---------------------------------------------------------------------------------------------- - -export interface V2CallEdge { - src: string; // caller callable id - dst: string; // callee callable (or external) id - prov: string[]; // provenance, e.g. ["tsc"], ["jelly"] - weight: number; -} - -export interface V2ParamEdge { - src: string; - dst: string; - var?: string; -} - -// ---------------------------------------------------------------------------------------------- -// Nodes — one scale-free node; language-native attrs ride additively via the index signature. -// ---------------------------------------------------------------------------------------------- - -export interface V2Node { - id: string; - kind: string; - span?: Span; - [attr: string]: unknown; // additive language-native attributes (decorators, is_*, type_parameters, …) -} - -/** A file / compilation unit — a *scope* holding types, functions, and free bindings. */ -export interface V2Module extends V2Node { - kind: "module"; - source: string; // whole file text, once; every node's text slices off this - types: Record; // classes/interfaces/enums/type-aliases/namespaces - functions: Record; // free functions - fields: Record; // module-level const/let/var (Gap B → fields on the scope) -} - -/** - * A type-or-scope node. Classes/interfaces/enums/type-aliases populate `callables`/`fields`; - * a `namespace` (a sub-file scope, Gap A) instead populates `types`/`functions`/`fields`. - */ -export interface V2Type extends V2Node { - kind: "class" | "interface" | "enum" | "type_alias" | "namespace"; - signature: string; // v1 human-readable id (kept; the durable id is `id`) - callables?: Record; // class/interface/enum members - fields?: Record; // class attributes / interface properties / enum members - types?: Record; // namespace: nested types - functions?: Record; // namespace: nested functions - // Heritage: `base_classes`/`implements_types` (carried from v1) stay signature strings — the - // resolved-id projection lives here, additively, for the Neo4j EXTENDS/IMPLEMENTS overlay. - // External/library supertypes that never resolve to a first-party id are dropped, not nulled. - extends_ids?: string[]; // resolved `can://` id(s) of the extended class/interface(s) - implements_ids?: string[]; // resolved `can://` id(s) of implemented interfaces (classes only) -} - -export interface V2Callable extends V2Node { - kind: "function" | "method" | "constructor" | "getter" | "setter" | "arrow" | "function_expression"; - signature: string; - body: Record; // L1: `call` nodes keyed by line:col; L3+: full statements - callables?: Record; // nested callables (closures) — syntactic containment - types?: Record; // nested (local) classes - cfg?: unknown[]; // L3 - cdg?: unknown[]; // L3 - ddg?: unknown[]; // L3→L4 - summary?: unknown[]; // L4 -} - -export interface V2Field extends V2Node { - kind: "field"; -} - -/** A node inside a callable `body` (L1: call sites; L3+: statements; L4: synthetic vertices). */ -export interface V2BodyNode { - kind: string; // "call" | "statement" | "return" | "entry" | "exit" | "formal_in" | … - span?: Span; - callee?: string | null; // on `call` nodes: null at L1, backfilled to a callable id at L2 - [attr: string]: unknown; -} diff --git a/src/semantic_analysis/callGraph.ts b/src/semantic_analysis/callGraph.ts index cb7c206..e20ea08 100644 --- a/src/semantic_analysis/callGraph.ts +++ b/src/semantic_analysis/callGraph.ts @@ -14,11 +14,11 @@ import { CALL_DEP, type TSCallEdge, type TSCallable, - type TSClass, type TSExternalSymbol, type TSModule, - type TSNamespace, type TSSynthesizedCallable, + type TSType, + forEachCallable, } from "../schema"; import { resolveCalleeSignature } from "../schema"; import type { Logger } from "../utils"; @@ -49,16 +49,14 @@ export function buildCallGraph( // these, and gating uses the full (merged) table so a cross-program in-project call resolves. const allSignatures = new Set(); for (const mod of Object.values(symbol_table)) { - const sigs: TSCallable[] = []; - collectModule(mod, sigs); - for (const c of sigs) allSignatures.add(c.signature); + forEachCallable(mod, (c) => allSignatures.add(c.signature)); } // Callables to ITERATE: only this program's modules (all modules when `only` is undefined), so a // multi-program build attributes each call site to the program whose options actually resolve it. const callables: TSCallable[] = []; for (const [key, mod] of Object.entries(symbol_table)) { if (only && !only.has(key)) continue; - collectModule(mod, callables); + forEachCallable(mod, (c) => callables.push(c)); } // 2. Index call/new expression AST nodes by full span so we can match recorded call sites. @@ -135,7 +133,7 @@ export function buildCallGraph( for (const caller of callables) { for (const site of caller.call_sites) { const node = callExprIndex.get( - `${caller.path}#${site.start_line}:${site.start_column}-${site.end_line}:${site.end_column}`, + `${caller.abs_path}#${site.start_line}:${site.start_column}-${site.end_line}:${site.end_column}`, ); if (!node) { unresolved++; @@ -233,24 +231,24 @@ function indexClasses( classMeta: Map, childrenOf: Map>, ): void { - const visitClass = (cl: TSClass): void => { - classMeta.set(cl.signature, { - is_abstract: cl.is_abstract, - methods: new Set(Object.values(cl.methods).map((m) => m.name)), - }); - for (const base of cl.base_classes) { - if (!childrenOf.has(base)) childrenOf.set(base, new Set()); - childrenOf.get(base)!.add(cl.signature); + // RTA reach: module- and namespace-scoped classes only — classes local to a callable body are + // deliberately outside the dispatch universe (the historical reach, kept bit-for-bit). + const visitType = (t: TSType): void => { + if (t.kind === "class") { + classMeta.set(t.signature, { + is_abstract: t.is_abstract ?? false, + methods: new Set(Object.values(t.callables ?? {}).map((m) => m.name)), + }); + for (const base of t.base_classes ?? []) { + if (!childrenOf.has(base)) childrenOf.set(base, new Set()); + childrenOf.get(base)!.add(t.signature); + } + } else if (t.kind === "namespace") { + for (const nt of Object.values(t.types ?? {})) visitType(nt); } - for (const ic of Object.values(cl.inner_classes)) visitClass(ic); - }; - const visitNs = (ns: TSNamespace): void => { - for (const cl of Object.values(ns.classes)) visitClass(cl); - for (const n of Object.values(ns.namespaces)) visitNs(n); }; for (const mod of Object.values(symbol_table)) { - for (const cl of Object.values(mod.classes)) visitClass(cl); - for (const n of Object.values(mod.namespaces)) visitNs(n); + for (const t of Object.values(mod.types)) visitType(t); } } @@ -271,30 +269,3 @@ function indexCallExpressions(project: Project): Map { } return idx; } - -// --- recursive collection of every callable signature in the symbol table --- - -function collectModule(mod: TSModule, out: TSCallable[]): void { - for (const f of Object.values(mod.functions)) collectCallable(f, out); - for (const c of Object.values(mod.classes)) collectClass(c, out); - for (const i of Object.values(mod.interfaces)) for (const m of Object.values(i.methods)) collectCallable(m, out); - for (const ns of Object.values(mod.namespaces)) collectNamespace(ns, out); -} - -function collectNamespace(ns: TSNamespace, out: TSCallable[]): void { - for (const f of Object.values(ns.functions)) collectCallable(f, out); - for (const c of Object.values(ns.classes)) collectClass(c, out); - for (const i of Object.values(ns.interfaces)) for (const m of Object.values(i.methods)) collectCallable(m, out); - for (const n of Object.values(ns.namespaces)) collectNamespace(n, out); -} - -function collectClass(c: TSClass, out: TSCallable[]): void { - for (const m of Object.values(c.methods)) collectCallable(m, out); - for (const ic of Object.values(c.inner_classes)) collectClass(ic, out); -} - -function collectCallable(c: TSCallable, out: TSCallable[]): void { - out.push(c); - for (const ic of Object.values(c.inner_callables)) collectCallable(ic, out); - for (const cl of Object.values(c.inner_classes)) collectClass(cl, out); -} diff --git a/src/syntactic_analysis/builders.ts b/src/syntactic_analysis/builders.ts index f856c13..e87bdd0 100644 --- a/src/syntactic_analysis/builders.ts +++ b/src/syntactic_analysis/builders.ts @@ -1,33 +1,35 @@ /** - * Per-file builders: turn a ts-morph SourceFile into a canonical TSModule. Mirrors the role of - * Python's `build_pymodule_from_file`. ts-morph nodes are accessed via dynamic getters (cast to - * `any`) for brevity and resilience across node kinds; the *returned* objects are strictly typed - * to the schema, which is what the output contract cares about. + * Per-file builders: turn a ts-morph SourceFile into a canonical TSModule — NATIVELY in the + * schema-v2 shape (types{}/functions{}/fields{} buckets, member-keyed maps, span-only containers, + * omit-instead-of-null leaves). Mirrors the role of python's module builder. ts-morph nodes are + * accessed via dynamic getters (cast to `any`) for brevity and resilience across node kinds; the + * *returned* objects are strictly typed to the schema, which is what the output contract cares + * about. + * + * Builders do NOT stamp `can://` ids (ids embed the per-invocation app name; the cached tree must + * stay app-name-free — assignIds.ts stamps them per run) and do NOT build `body{}` (the l1Body + * pass derives it per run from the INTERNAL `call_sites`). */ import { Node, SyntaxKind } from "ts-morph"; import { type TSCallable, type TSCallableKind, type TSCallsite, - type TSClass, - type TSClassAttribute, type TSComment, type TSDecorator, - type TSEnum, type TSExport, + type TSField, type TSImport, - type TSInterface, type TSModule, - type TSNamespace, type TSOverloadSignature, type TSSpan, - type TSTypeAlias, + type TSType, type TSTypeParameter, - type TSVariableDeclaration, constructorSignatureOf, fileKeyOf, } from "../schema"; import { computeSignatureForDecl } from "../schema"; +import { memberKey } from "../schema/ids"; // ---------------------------------------------------------------------------------------------- // dynamic-getter helpers @@ -49,22 +51,22 @@ function isRedundantOverload(node: Node): boolean { return typeof n.getImplementation === "function" ? n.getImplementation() !== undefined : false; } -function clamp(s: string | undefined | null, max = 400): string | null { - if (s == null) return null; +function clamp(s: string | undefined | null, max = 400): string | undefined { + if (s == null) return undefined; return s.length > max ? s.slice(0, max) : s; } -function inferredType(valueNode: Node): string | null { +function inferredType(valueNode: Node): string | undefined { try { const t = (valueNode as unknown as { getType?: () => { getText: (n?: Node) => string } }).getType?.(); - if (!t) return null; + if (!t) return undefined; return clamp(t.getText(valueNode)); } catch { - return null; + return undefined; } } -function returnTypeText(fnNode: Node): string | null { +function returnTypeText(fnNode: Node): string | undefined { const n = fnNode as unknown as { getReturnTypeNode?: () => { getText: () => string } | undefined; getReturnType?: () => { getText: (n?: Node) => string }; @@ -77,7 +79,7 @@ function returnTypeText(fnNode: Node): string | null { } catch { /* unresolved */ } - return null; + return undefined; } function span(node: Node): { start_line: number; end_line: number; start_column: number; end_column: number } { @@ -90,7 +92,7 @@ function span(node: Node): { start_line: number; end_line: number; start_column: /** * schema-v2 precise span: [line, column] endpoints + char offsets into the module source. * `bytes = [getStart(), getEnd()]` are exactly the offsets `Node.getText()` slices, so - * `module.source.slice(bytes[0], bytes[1])` reproduces the node's text (the old `code` field). + * `module.source.slice(bytes[0], bytes[1])` reproduces the node's text. */ function richSpan(node: Node): TSSpan { const sf = node.getSourceFile(); @@ -101,15 +103,7 @@ function richSpan(node: Node): TSSpan { return { start: [sl.line, sl.column], end: [el.line, el.column], bytes: [s, e] }; } -function declLines(node: Node): { start_line: number; end_line: number; code_start_line: number } { - return { - start_line: node.getStartLineNumber(true), - end_line: node.getEndLineNumber(), - code_start_line: node.getStartLineNumber(false), - }; -} - -function accessibilityOf(node: Node): string | null { +function accessibilityOf(node: Node): string | undefined { const mods = (node as unknown as { getModifiers?: () => Node[] }).getModifiers?.() ?? []; for (const m of mods) { const k = m.getKind(); @@ -117,7 +111,7 @@ function accessibilityOf(node: Node): string | null { if (k === SyntaxKind.ProtectedKeyword) return "protected"; if (k === SyntaxKind.PublicKeyword) return "public"; } - return null; + return undefined; } function isExportedDecl(node: Node): boolean { @@ -180,9 +174,10 @@ function decoratorsOf(node: Node): TSDecorator[] { } } } + const qualified = dec.getFullName(); return { name: dec.getName(), - qualified_name: dec.getFullName() ?? null, + ...(qualified != null ? { qualified_name: qualified } : {}), positional_arguments: positional, keyword_arguments: keyword, ...span(d), @@ -199,10 +194,12 @@ function typeParamsOf(node: Node): TSTypeParameter[] { getConstraint?: () => { getText: () => string } | undefined; getDefault?: () => { getText: () => string } | undefined; }; + const constraint = t.getConstraint?.()?.getText(); + const dflt = t.getDefault?.()?.getText(); return { name: t.getName(), - constraint: t.getConstraint?.()?.getText() ?? null, - default: t.getDefault?.()?.getText() ?? null, + ...(constraint != null ? { constraint } : {}), + ...(dflt != null ? { default: dflt } : {}), }; }); } @@ -217,20 +214,24 @@ function buildParam(param: Node): import("../schema").TSCallableParameter { getTypeNode?: () => { getText: () => string } | undefined; getInitializer?: () => { getText: () => string } | undefined; }; + const type = p.getTypeNode?.()?.getText() ?? inferredType(param); + const dflt = p.getInitializer?.()?.getText(); + const accessibility = accessibilityOf(param); return { name: p.getName(), - type: p.getTypeNode?.()?.getText() ?? inferredType(param), - default_value: p.getInitializer?.()?.getText() ?? null, + ...(type != null ? { type } : {}), + ...(dflt != null ? { default_value: dflt } : {}), is_optional: boolOf(param, "isOptional"), is_rest: boolOf(param, "isRestParameter"), is_readonly: boolOf(param, "isReadonly"), - accessibility: accessibilityOf(param), + ...(accessibility != null ? { accessibility } : {}), decorators: decoratorsOf(param), ...span(param), }; } -function buildVariable(vd: Node, scope: TSVariableDeclaration["scope"]): TSVariableDeclaration { +/** A module/namespace `const`/`let`/`var` binding as a `field` node. */ +function buildVariableField(vd: Node, scope: "module" | "namespace"): TSField { const v = vd as unknown as { getName: () => string; getTypeNode?: () => { getText: () => string } | undefined; @@ -239,7 +240,7 @@ function buildVariable(vd: Node, scope: TSVariableDeclaration["scope"]): TSVaria }; const vs = v.getVariableStatement?.(); const kindRaw = String(vs?.getDeclarationKind?.() ?? ""); - const declaration_kind: TSVariableDeclaration["declaration_kind"] = kindRaw.includes("const") + const declaration_kind: TSField["declaration_kind"] = kindRaw.includes("const") ? "const" : kindRaw.includes("let") ? "let" @@ -248,40 +249,46 @@ function buildVariable(vd: Node, scope: TSVariableDeclaration["scope"]): TSVaria : kindRaw.includes("using") ? "using" : "unknown"; + const type = v.getTypeNode?.()?.getText() ?? inferredType(vd); + const initializer = v.getInitializer?.()?.getText(); return { + id: "", + kind: "field", + span: richSpan(vd), name: v.getName(), - type: v.getTypeNode?.()?.getText() ?? inferredType(vd), - initializer: v.getInitializer?.()?.getText() ?? null, - value: null, + ...(type != null ? { type } : {}), + ...(initializer != null ? { initializer } : {}), scope, declaration_kind, is_readonly: declaration_kind === "const", is_exported: vs?.isExported?.() ?? false, - ...span(vd), - span: richSpan(vd), }; } -function buildAttribute(prop: Node): TSClassAttribute { +/** A class property / interface property as a `field` node. */ +function buildAttributeField(prop: Node): TSField { const p = prop as unknown as { getName: () => string; getTypeNode?: () => { getText: () => string } | undefined; getInitializer?: () => { getText: () => string } | undefined; }; + const type = p.getTypeNode?.()?.getText() ?? inferredType(prop); + const initializer = p.getInitializer?.()?.getText(); + const accessibility = accessibilityOf(prop); return { + id: "", + kind: "field", span: richSpan(prop), name: p.getName(), - type: p.getTypeNode?.()?.getText() ?? inferredType(prop), + ...(type != null ? { type } : {}), comments: jsDocsOf(prop), decorators: decoratorsOf(prop), - initializer: p.getInitializer?.()?.getText() ?? null, - accessibility: accessibilityOf(prop), + ...(initializer != null ? { initializer } : {}), + ...(accessibility != null ? { accessibility } : {}), is_static: boolOf(prop, "isStatic"), is_readonly: boolOf(prop, "isReadonly"), is_optional: boolOf(prop, "hasQuestionToken"), is_abstract: boolOf(prop, "isAbstract"), - start_line: prop.getStartLineNumber(true), - end_line: prop.getEndLineNumber(), }; } @@ -289,8 +296,8 @@ function buildCallsite(call: Node): TSCallsite { const isNew = Node.isNewExpression(call); const expr = (call as unknown as { getExpression: () => Node }).getExpression(); let method_name = expr.getText(); - let receiver_expr: string | null = null; - let receiver_type: string | null = null; + let receiver_expr: string | undefined; + let receiver_type: string | undefined; let is_optional_chain = false; if (Node.isPropertyAccessExpression(expr)) { method_name = expr.getName(); @@ -302,14 +309,14 @@ function buildCallsite(call: Node): TSCallsite { const argument_types = args.map((a) => inferredType(a) ?? "unknown"); const typeArgs = (call as unknown as { getTypeArguments?: () => Node[] }).getTypeArguments?.() ?? []; const type_arguments = typeArgs.map((t) => t.getText()); + const return_type = inferredType(call); return { method_name, - receiver_expr, - receiver_type, + ...(receiver_expr != null ? { receiver_expr } : {}), + ...(receiver_type != null ? { receiver_type } : {}), argument_types, type_arguments, - return_type: inferredType(call), - callee_signature: null, + ...(return_type != null ? { return_type } : {}), is_constructor_call: isNew, is_optional_chain, ...span(call), @@ -339,7 +346,6 @@ function namedBoundary(node: Node): Boundary { interface BodyHandlers { onCall: (n: Node) => void; - onLocal: (vd: Node) => void; onNestedCallable: (n: Node) => void; onNestedClass: (n: Node) => void; } @@ -357,7 +363,6 @@ function walkBody(body: Node, h: BodyHandlers): void { } if (b === "skip") return; if (Node.isCallExpression(node) || Node.isNewExpression(node)) h.onCall(node); - if (Node.isVariableDeclaration(node)) h.onLocal(node); node.forEachChild(visit); }; // A concise arrow body can *be* a callable (`() => () => x`). Visiting only the body's children @@ -409,13 +414,16 @@ function computeCC(body: Node): number { function overloadsOf(fnNode: Node): TSOverloadSignature[] { const ovs = (fnNode as unknown as { getOverloads?: () => Node[] }).getOverloads?.(); if (!ovs || !ovs.length) return []; - return ovs.map((o) => ({ - parameters: ((o as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []).map(buildParam), - return_type: returnTypeText(o), - type_parameters: typeParamsOf(o), - start_line: o.getStartLineNumber(true), - end_line: o.getEndLineNumber(), - })); + return ovs.map((o) => { + const return_type = returnTypeText(o); + return { + parameters: ((o as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []).map(buildParam), + ...(return_type != null ? { return_type } : {}), + type_parameters: typeParamsOf(o), + start_line: o.getStartLineNumber(true), + end_line: o.getEndLineNumber(), + }; + }); } /** @@ -446,22 +454,20 @@ export function buildCallable( if (!sig) return null; const call_sites: TSCallsite[] = []; - const local_variables: TSVariableDeclaration[] = []; - const inner_callables: Record = {}; - const inner_classes: Record = {}; + const callables: Record = {}; + const types: Record = {}; const body = (fnNode as unknown as { getBody?: () => Node | undefined }).getBody?.(); if (body) { walkBody(body, { onCall: (n) => call_sites.push(buildCallsite(n)), - onLocal: (vd) => local_variables.push(buildVariable(vd, "function")), onNestedCallable: (n) => { const r = buildNestedCallable(n, root); - if (r) inner_callables[r.sig] = r.callable; + if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; }, onNestedClass: (n) => { const r = buildClass(n, root); - inner_classes[r.sig] = r.cls; + types[memberKey(r.sig)] = r.cls; }, }); } @@ -469,26 +475,23 @@ export function buildCallable( const nameNode = sigNode as unknown as { getName?: () => string | undefined }; const name = Node.isConstructorDeclaration(fnNode) ? "constructor" : (nameNode.getName?.() ?? "(anonymous)"); + const return_type = kind === "constructor" || kind === "setter" ? undefined : returnTypeText(fnNode); + const accessibility = accessibilityOf(fnNode); + const accessor_kind = kind === "getter" ? "getter" : kind === "setter" ? "setter" : undefined; const callable: TSCallable = { + id: "", + kind, span: richSpan(sigNode), name, - path: sigNode.getSourceFile().getFilePath(), signature: sig, comments: jsDocsOf(sigNode), decorators: decoratorsOf(fnNode), parameters: ((fnNode as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []).map(buildParam), type_parameters: typeParamsOf(fnNode), - return_type: kind === "constructor" || kind === "setter" ? null : returnTypeText(fnNode), - code: clamp(sigNode.getText(), 20000), - ...declLines(sigNode), - call_sites, - inner_callables, - inner_classes, - local_variables, + ...(return_type != null ? { return_type } : {}), cyclomatic_complexity: body ? computeCC(body) : 0, - kind, - accessibility: accessibilityOf(fnNode), + ...(accessibility != null ? { accessibility } : {}), is_static: boolOf(fnNode, "isStatic"), is_abstract: boolOf(fnNode, "isAbstract"), is_async: boolOf(fnNode, "isAsync"), @@ -498,9 +501,14 @@ export function buildCallable( is_exported: isExportedDecl(sigNode), is_ambient: isAmbientDecl(sigNode), is_implicit: false, - accessor_kind: kind === "getter" ? "getter" : kind === "setter" ? "setter" : null, + ...(accessor_kind != null ? { accessor_kind } : {}), overload_signatures: overloadsOf(fnNode), + body: {}, + abs_path: sigNode.getSourceFile().getFilePath(), + call_sites, }; + if (Object.keys(callables).length) callable.callables = callables; + if (Object.keys(types).length) callable.types = types; return { sig, callable }; } @@ -509,26 +517,16 @@ function implicitConstructor(classSig: string, filePath: string): { sig: string; return { sig, callable: { + id: "", + kind: "constructor", span: { start: [0, 0], end: [0, 0], bytes: [0, 0] }, // synthetic: no source name: "constructor", - path: filePath, signature: sig, comments: [], decorators: [], parameters: [], type_parameters: [], - return_type: null, - code: null, - start_line: -1, - end_line: -1, - code_start_line: -1, - call_sites: [], - inner_callables: {}, - inner_classes: {}, - local_variables: [], cyclomatic_complexity: 0, - kind: "constructor", - accessibility: null, is_static: false, is_abstract: false, is_async: false, @@ -538,8 +536,10 @@ function implicitConstructor(classSig: string, filePath: string): { sig: string; is_exported: false, is_ambient: false, is_implicit: true, - accessor_kind: null, overload_signatures: [], + body: {}, + abs_path: filePath, + call_sites: [], }, }; } @@ -577,7 +577,7 @@ function resolveHeritage(expr: Node, root: string): string { // type-kind builders // ---------------------------------------------------------------------------------------------- -export function buildClass(cls: Node, root: string): { sig: string; cls: TSClass } { +export function buildClass(cls: Node, root: string): { sig: string; cls: TSType } { const sig = computeSignatureForDecl(cls, root) ?? `${fileKeyOf(cls.getSourceFile().getFilePath(), root).modulePrefix}.(anonymous)`; const filePath = cls.getSourceFile().getFilePath(); const c = cls as unknown as { @@ -591,55 +591,55 @@ export function buildClass(cls: Node, root: string): { sig: string; cls: TSClass getImplements?: () => Node[]; }; - const methods: Record = {}; + const callables: Record = {}; for (const m of c.getMethods()) { if (isRedundantOverload(m)) continue; const r = buildCallable(m, m, "method", root); - if (r) methods[r.sig] = r.callable; + if (r) callables[memberKey(r.sig)] = r.callable; } const ctors = c.getConstructors(); if (ctors.length === 0) { const imp = implicitConstructor(sig, filePath); - methods[imp.sig] = imp.callable; + callables[memberKey(imp.sig)] = imp.callable; } else { for (const ctor of ctors) { if (isRedundantOverload(ctor)) continue; const r = buildCallable(ctor, ctor, "constructor", root); - if (r) methods[r.sig] = r.callable; + if (r) callables[memberKey(r.sig)] = r.callable; } } for (const g of c.getGetAccessors()) { const r = buildCallable(g, g, "getter", root); - if (r) methods[`${r.sig}#get`] = r.callable; + if (r) callables[memberKey(r.sig, "getter")] = r.callable; } for (const s of c.getSetAccessors()) { const r = buildCallable(s, s, "setter", root); - if (r) methods[`${r.sig}#set`] = r.callable; + if (r) callables[memberKey(r.sig, "setter")] = r.callable; } - const attributes: Record = {}; + const fields: Record = {}; for (const p of c.getProperties()) { - attributes[(p as unknown as { getName: () => string }).getName()] = buildAttribute(p); + fields[(p as unknown as { getName: () => string }).getName()] = buildAttributeField(p); } - // parameter properties (constructor(private x: T)) are class fields too + // parameter properties (constructor(private x: T)) are class fields too — spanless on the wire. for (const ctor of ctors) { for (const p of (ctor as unknown as { getParameters: () => Node[] }).getParameters()) { const acc = accessibilityOf(p); if (acc || boolOf(p, "isReadonly")) { const pn = p as unknown as { getName: () => string; getTypeNode?: () => { getText: () => string } | undefined }; - attributes[pn.getName()] = { + const type = pn.getTypeNode?.()?.getText() ?? inferredType(p); + fields[pn.getName()] = { + id: "", + kind: "field", name: pn.getName(), - type: pn.getTypeNode?.()?.getText() ?? inferredType(p), + ...(type != null ? { type } : {}), comments: [], decorators: decoratorsOf(p), - initializer: null, - accessibility: acc, + ...(acc != null ? { accessibility: acc } : {}), is_static: false, is_readonly: boolOf(p, "isReadonly"), is_optional: boolOf(p, "isOptional"), is_abstract: false, - start_line: p.getStartLineNumber(true), - end_line: p.getEndLineNumber(), }; } } @@ -658,28 +658,26 @@ export function buildClass(cls: Node, root: string): { sig: string; cls: TSClass return { sig, cls: { + id: "", + kind: "class", span: richSpan(cls), name: c.getName?.() ?? "(anonymous)", signature: sig, comments: jsDocsOf(cls), - code: clamp(cls.getText(), 20000), decorators: decoratorsOf(cls), base_classes, implements_types, type_parameters: typeParamsOf(cls), - methods, - attributes, - inner_classes: {}, + callables, + fields, is_abstract: boolOf(cls, "isAbstract"), is_exported: isExportedDecl(cls), is_ambient: isAmbientDecl(cls), - start_line: cls.getStartLineNumber(true), - end_line: cls.getEndLineNumber(), }, }; } -export function buildInterface(intf: Node, root: string): { sig: string; intf: TSInterface } { +export function buildInterface(intf: Node, root: string): { sig: string; intf: TSType } { const sig = computeSignatureForDecl(intf, root) ?? `${fileKeyOf(intf.getSourceFile().getFilePath(), root).modulePrefix}.(anonymous)`; const i = intf as unknown as { getName: () => string; @@ -690,14 +688,14 @@ export function buildInterface(intf: Node, root: string): { sig: string; intf: T getConstructSignatures?: () => Node[]; getIndexSignatures?: () => Node[]; }; - const methods: Record = {}; + const callables: Record = {}; for (const m of i.getMethods()) { const r = buildCallable(m, m, "method", root); - if (r) methods[r.sig] = r.callable; + if (r) callables[memberKey(r.sig)] = r.callable; } - const properties: Record = {}; + const fields: Record = {}; for (const p of i.getProperties()) { - properties[(p as unknown as { getName: () => string }).getName()] = buildAttribute(p); + fields[(p as unknown as { getName: () => string }).getName()] = buildAttributeField(p); } const base_classes = (i.getExtends?.() ?? []).map((e) => resolveHeritage(e, root)); const call_signatures = [ @@ -708,78 +706,77 @@ export function buildInterface(intf: Node, root: string): { sig: string; intf: T return { sig, intf: { + id: "", + kind: "interface", span: richSpan(intf), name: i.getName(), signature: sig, comments: jsDocsOf(intf), - code: clamp(intf.getText(), 20000), base_classes, type_parameters: typeParamsOf(intf), - methods, - properties, + callables, + fields, call_signatures, index_signatures, is_exported: isExportedDecl(intf), is_ambient: isAmbientDecl(intf), - start_line: intf.getStartLineNumber(true), - end_line: intf.getEndLineNumber(), }, }; } -export function buildEnum(en: Node, root: string): { sig: string; en: TSEnum } { +export function buildEnum(en: Node, root: string): { sig: string; en: TSType } { const sig = computeSignatureForDecl(en, root) ?? `${fileKeyOf(en.getSourceFile().getFilePath(), root).modulePrefix}.(anonymous)`; const e = en as unknown as { getName: () => string; getMembers: () => Node[]; isConstEnum?: () => boolean }; - const members = e.getMembers().map((m) => { + const fields: Record = {}; + for (const m of e.getMembers()) { const mm = m as unknown as { getName: () => string; getValue?: () => string | number | undefined; getInitializer?: () => { getText: () => string } | undefined; }; const v = mm.getValue?.(); - return { + const value = v !== undefined && v !== null ? String(v) : mm.getInitializer?.()?.getText(); + fields[mm.getName()] = { + id: "", + kind: "field", span: richSpan(m), name: mm.getName(), - value: v !== undefined && v !== null ? String(v) : (mm.getInitializer?.()?.getText() ?? null), - start_line: m.getStartLineNumber(true), - end_line: m.getEndLineNumber(), + ...(value != null ? { value } : {}), }; - }); + } return { sig, en: { + id: "", + kind: "enum", span: richSpan(en), name: e.getName(), signature: sig, comments: jsDocsOf(en), - code: clamp(en.getText(), 20000), - members, + fields, is_const: e.isConstEnum?.() ?? false, is_exported: isExportedDecl(en), is_ambient: isAmbientDecl(en), - start_line: en.getStartLineNumber(true), - end_line: en.getEndLineNumber(), }, }; } -export function buildTypeAlias(ta: Node, root: string): { sig: string; ta: TSTypeAlias } { +export function buildTypeAlias(ta: Node, root: string): { sig: string; ta: TSType } { const sig = computeSignatureForDecl(ta, root) ?? `${fileKeyOf(ta.getSourceFile().getFilePath(), root).modulePrefix}.(anonymous)`; const t = ta as unknown as { getName: () => string; getTypeNode?: () => { getText: () => string } | undefined }; return { sig, ta: { + id: "", + kind: "type_alias", span: richSpan(ta), name: t.getName(), signature: sig, comments: jsDocsOf(ta), - code: clamp(ta.getText(), 20000), aliased_type: t.getTypeNode?.()?.getText() ?? "", type_parameters: typeParamsOf(ta), is_exported: isExportedDecl(ta), is_ambient: isAmbientDecl(ta), - start_line: ta.getStartLineNumber(true), - end_line: ta.getEndLineNumber(), }, }; } @@ -788,17 +785,13 @@ export function buildTypeAlias(ta: Node, root: string): { sig: string; ta: TSTyp // statemented container (Module + Namespace share this) // ---------------------------------------------------------------------------------------------- -interface Buckets { - classes: Record; - interfaces: Record; - enums: Record; - type_aliases: Record; +interface ScopeBuckets { + types: Record; functions: Record; - namespaces: Record; - variables: TSVariableDeclaration[]; + fields: Record; } -function buildStatemented(container: Node, root: string, varScope: TSVariableDeclaration["scope"]): Buckets { +function buildStatemented(container: Node, root: string, varScope: "module" | "namespace"): ScopeBuckets { const c = container as unknown as { getClasses: () => Node[]; getInterfaces: () => Node[]; @@ -808,42 +801,43 @@ function buildStatemented(container: Node, root: string, varScope: TSVariableDec getModules: () => Node[]; getVariableStatements: () => Node[]; }; - const classes: Record = {}; + // One types{} map. Fill order (classes → interfaces → enums → aliases → namespaces) is the + // canonical precedence: on a member-key collision (e.g. class/interface declaration merging) + // the later kind wins, exactly as the historical per-kind bucket merge did. + const types: Record = {}; for (const cl of c.getClasses()) { const r = buildClass(cl, root); - classes[r.sig] = r.cls; + types[memberKey(r.sig)] = r.cls; } - const interfaces: Record = {}; for (const it of c.getInterfaces()) { const r = buildInterface(it, root); - interfaces[r.sig] = r.intf; + types[memberKey(r.sig)] = r.intf; } - const enums: Record = {}; for (const en of c.getEnums()) { const r = buildEnum(en, root); - enums[r.sig] = r.en; + types[memberKey(r.sig)] = r.en; } - const type_aliases: Record = {}; for (const ta of c.getTypeAliases()) { const r = buildTypeAlias(ta, root); - type_aliases[r.sig] = r.ta; + types[memberKey(r.sig)] = r.ta; } const functions: Record = {}; for (const fn of c.getFunctions()) { if (isRedundantOverload(fn)) continue; const r = buildCallable(fn, fn, "function", root); - if (r) functions[r.sig] = r.callable; + if (r) functions[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; } - const variables: TSVariableDeclaration[] = []; + const fields: Record = {}; for (const vs of c.getVariableStatements()) { for (const vd of (vs as unknown as { getDeclarations: () => Node[] }).getDeclarations()) { const init = (vd as unknown as { getInitializer?: () => Node | undefined }).getInitializer?.(); if (init && (Node.isArrowFunction(init) || Node.isFunctionExpression(init))) { const k: TSCallableKind = Node.isArrowFunction(init) ? "arrow" : "function_expression"; const r = buildCallable(vd, init, k, root); - if (r) functions[r.sig] = r.callable; + if (r) functions[memberKey(r.sig)] = r.callable; } else { - variables.push(buildVariable(vd, varScope)); + const f = buildVariableField(vd, varScope); + fields[f.name] = f; } } } @@ -853,28 +847,29 @@ function buildStatemented(container: Node, root: string, varScope: TSVariableDec // loops above already claimed, so nothing is collected twice. walkBody(container, { onCall: () => {}, - onLocal: () => {}, onNestedCallable: (n) => { if (!Node.isArrowFunction(n) && !Node.isFunctionExpression(n)) return; // already bucketed const r = buildCallable(n, n, Node.isArrowFunction(n) ? "arrow" : "function_expression", root); - if (r) functions[r.sig] = r.callable; + if (r) functions[memberKey(r.sig)] = r.callable; }, onNestedClass: () => {}, }); - const namespaces: Record = {}; for (const ns of c.getModules()) { const r = buildNamespace(ns, root); - namespaces[r.sig] = r.ns; + types[memberKey(r.sig)] = r.ns; } - return { classes, interfaces, enums, type_aliases, functions, namespaces, variables }; + return { types, functions, fields }; } -export function buildNamespace(ns: Node, root: string): { sig: string; ns: TSNamespace } { +/** Gap A: a namespace is a nested *scope* — same buckets as a module (types/functions/fields). */ +export function buildNamespace(ns: Node, root: string): { sig: string; ns: TSType } { const sig = computeSignatureForDecl(ns, root) ?? `${fileKeyOf(ns.getSourceFile().getFilePath(), root).modulePrefix}.(anonymous)`; const buckets = buildStatemented(ns, root, "namespace"); return { sig, ns: { + id: "", + kind: "namespace", span: richSpan(ns), name: (ns as unknown as { getName: () => string }).getName(), signature: sig, @@ -882,8 +877,6 @@ export function buildNamespace(ns: Node, root: string): { sig: string; ns: TSNam ...buckets, is_exported: isExportedDecl(ns), is_ambient: isAmbientDecl(ns), - start_line: ns.getStartLineNumber(true), - end_line: ns.getEndLineNumber(), }, }; } @@ -909,21 +902,22 @@ function buildImports(sf: Node): TSImport[] { const def = i.getDefaultImport?.(); const ns = i.getNamespaceImport?.(); const named = i.getNamedImports?.() ?? []; - if (def) out.push({ module, name: def.getText(), alias: null, is_type_only: typeOnly, import_kind: "default", ...s }); + if (def) out.push({ module, name: def.getText(), is_type_only: typeOnly, import_kind: "default", ...s }); if (ns) out.push({ module, name: "*", alias: ns.getText(), is_type_only: typeOnly, import_kind: "namespace", ...s }); for (const ni of named) { const n = ni as unknown as { getName: () => string; getAliasNode?: () => { getText: () => string } | undefined; isTypeOnly?: () => boolean }; + const alias = n.getAliasNode?.()?.getText(); out.push({ module, name: n.getName(), - alias: n.getAliasNode?.()?.getText() ?? null, + ...(alias != null ? { alias } : {}), is_type_only: typeOnly || (n.isTypeOnly?.() ?? false), import_kind: "named", ...s, }); } if (!def && !ns && named.length === 0) { - out.push({ module, name: "", alias: null, is_type_only: typeOnly, import_kind: "side_effect", ...s }); + out.push({ module, name: "", is_type_only: typeOnly, import_kind: "side_effect", ...s }); } } return out; @@ -939,21 +933,37 @@ function buildExports(sf: Node): TSExport[] { getNamespaceExport?: () => { getNameNode?: () => { getText: () => string } } | undefined; getNamedExports?: () => Node[]; }; - const module = e.getModuleSpecifierValue?.() ?? null; + const module = e.getModuleSpecifierValue?.(); const typeOnly = e.isTypeOnly(); const s = span(exp); const nsExp = e.getNamespaceExport?.(); const named = e.getNamedExports?.() ?? []; if (nsExp) { - out.push({ module, name: "*", alias: nsExp.getNameNode?.()?.getText() ?? null, is_type_only: typeOnly, export_kind: module ? "re_export" : "namespace", ...s }); + const alias = nsExp.getNameNode?.()?.getText(); + out.push({ + ...(module != null ? { module } : {}), + name: "*", + ...(alias != null ? { alias } : {}), + is_type_only: typeOnly, + export_kind: module ? "re_export" : "namespace", + ...s, + }); } for (const ne of named) { const n = ne as unknown as { getName: () => string; getAliasNode?: () => { getText: () => string } | undefined }; - out.push({ module, name: n.getName(), alias: n.getAliasNode?.()?.getText() ?? null, is_type_only: typeOnly, export_kind: module ? "re_export" : "named", ...s }); + const alias = n.getAliasNode?.()?.getText(); + out.push({ + ...(module != null ? { module } : {}), + name: n.getName(), + ...(alias != null ? { alias } : {}), + is_type_only: typeOnly, + export_kind: module ? "re_export" : "named", + ...s, + }); } if (!nsExp && named.length === 0 && module) { // `export * from "m"` with no namespace binding - out.push({ module, name: "*", alias: null, is_type_only: typeOnly, export_kind: "re_export", ...s }); + out.push({ module, name: "*", is_type_only: typeOnly, export_kind: "re_export", ...s }); } } return out; @@ -997,30 +1007,20 @@ function collectComments(sf: Node): TSComment[] { export function buildModule(sf: Node, root: string): TSModule { const filePath = sf.getSourceFile().getFilePath(); - const { fileKey, modulePrefix } = fileKeyOf(filePath, root); const buckets = buildStatemented(sf, root, "module"); // schema-v2: retain the whole file text once on the module; every node's text slices off it. const source = (sf as unknown as { getFullText: () => string }).getFullText(); const endLc = sf.getSourceFile().getLineAndColumnAtPos(source.length); return { + id: "", + kind: "module", span: { start: [1, 1], end: [endLc.line, endLc.column], bytes: [0, source.length] }, source, - file_path: fileKey, - module_name: modulePrefix, imports: buildImports(sf), exports: buildExports(sf), comments: collectComments(sf), - classes: buckets.classes, - interfaces: buckets.interfaces, - enums: buckets.enums, - type_aliases: buckets.type_aliases, - functions: buckets.functions, - namespaces: buckets.namespaces, - variables: buckets.variables, + ...buckets, is_tsx: filePath.endsWith(".tsx"), is_declaration_file: (sf as unknown as { isDeclarationFile: () => boolean }).isDeclarationFile(), - content_hash: null, - last_modified: null, - file_size: null, }; } diff --git a/src/utils/cache.ts b/src/utils/cache.ts index 7c1cff3..452c935 100644 --- a/src/utils/cache.ts +++ b/src/utils/cache.ts @@ -1,13 +1,12 @@ import * as fs from "node:fs"; import * as path from "node:path"; -import type { TSCallEdge, TSModule } from "../schema"; +import type { TSModule } from "../schema"; import { sha256 } from "./fs"; import { ANALYZER_VERSION } from "./version"; export interface CacheData { analyzer_version?: string; symbol_table: Record; - call_graph: TSCallEdge[]; } export function cacheFilePath(cacheDir: string): string { diff --git a/src/utils/serialize.ts b/src/utils/serialize.ts index 0a68546..95b9989 100644 --- a/src/utils/serialize.ts +++ b/src/utils/serialize.ts @@ -2,8 +2,7 @@ import * as fs from "node:fs"; import * as path from "node:path"; import { boltWriter, buildSchemaDocument, project, renderCypher } from "../build/neo4j"; import type { AnalysisOptions } from "../options"; -import type { TSApplication } from "../schema"; -import { toV2, toV2Detailed } from "../schema/v2"; +import type { TSAnalysis } from "../schema"; import { Logger } from "./logging"; /** @@ -13,19 +12,18 @@ import { Logger } from "./logging"; * - neo4j: project the IR to a graph. With --neo4j-uri, push incrementally to a live DB over * Bolt; otherwise write a self-contained `/graph.cypher` snapshot. */ -export async function emit(app: TSApplication, opts: AnalysisOptions): Promise { +export async function emit(application: TSAnalysis, opts: AnalysisOptions): Promise { if (opts.emit === "neo4j") { - await emitNeo4j(app, opts); + await emitNeo4j(application, opts); return; } - // schema v2: reshape the v1 model into the canonical additive-CPG tree before serializing. - const out = toV2(app, opts); + // The envelope IS the wire (finalizeAnalysis already stripped the internal fields) — write it. if (opts.output === null) { - process.stdout.write(JSON.stringify(out)); + process.stdout.write(JSON.stringify(application)); return; } fs.mkdirSync(opts.output, { recursive: true }); - fs.writeFileSync(path.join(opts.output, "analysis.json"), JSON.stringify(out)); + fs.writeFileSync(path.join(opts.output, "analysis.json"), JSON.stringify(application)); } /** @@ -42,10 +40,9 @@ export function emitSchema(opts: AnalysisOptions): void { fs.writeFileSync(path.join(opts.output, "schema.json"), doc); } -async function emitNeo4j(app: TSApplication, opts: AnalysisOptions): Promise { - // Second projection of the SAME v2 tree the JSON path emits. --emit neo4j forces full depth - // (cli.ts), so `app` carries the L4 dataflow and the projected graph is the complete CPG. - const { application } = toV2Detailed(app, opts); +async function emitNeo4j(application: TSAnalysis, opts: AnalysisOptions): Promise { + // Second projection of the SAME v2 envelope the JSON path emits. --emit neo4j forces full depth + // (cli.ts), so the envelope carries the L4 dataflow and the projected graph is the complete CPG. const appId = application.application.id; const rows = project(application, appId); diff --git a/test/anonymous-callables.test.ts b/test/anonymous-callables.test.ts index e1a677c..076416e 100644 --- a/test/anonymous-callables.test.ts +++ b/test/anonymous-callables.test.ts @@ -13,7 +13,8 @@ import { describe, expect, test } from "bun:test"; import * as path from "node:path"; import { analyze } from "../src/core"; import type { AnalysisOptions } from "../src/options"; -import { type V2Callable, type V2Module, type V2Node, toV2Detailed } from "../src/schema/v2"; +import type { TSCallable, TSModule } from "../src/schema"; +import type { TSSynthesizedNode } from "../src/schema/homing"; import { project } from "../src/build/neo4j"; const FIXTURE = path.resolve(import.meta.dir, "fixtures/anon-app"); @@ -44,15 +45,15 @@ function options(level: number): AnalysisOptions { } const opts = options(4); -const { application, idBySig, collisions, dangling } = toV2Detailed(await analyze(opts), opts); +const { application, idBySig, collisions, dangling } = await analyze(opts); const root = application.application; -const mod = root.symbol_table["src/routes.ts"] as V2Module; -const fns = mod.functions as Record; +const mod = root.symbol_table["src/routes.ts"] as TSModule; +const fns = mod.functions as Record; -const login = fns["login"] as V2Callable; -const handler = (login.callables ?? {})[""] as V2Callable; +const login = fns["login"] as TSCallable; +const handler = (login.callables ?? {})[""] as TSCallable; -/** Edge lists are typed `unknown[]` on V2Callable until the body-node model lands (roadmap #2). */ +/** Edge lists are typed `unknown[]` on TSCallable until the body-node model lands (roadmap #2). */ type Edge = { src: string; dst: string; var?: string }; const edges = (xs: unknown[] | undefined): Edge[] => (xs ?? []) as Edge[]; @@ -72,30 +73,30 @@ describe("anonymous callables are first-class (issue #92)", () => { }); test("a variable-bound arrow keeps its own name — no segment", () => { - expect((fns["named"] as V2Callable).signature).toBe("src/routes.named"); + expect((fns["named"] as TSCallable).signature).toBe("src/routes.named"); expect(Object.keys(fns)).not.toContain(""); }); test("a bare arrow in a module-level expression statement is materialized", () => { - const h = fns[""] as V2Callable; + const h = fns[""] as TSCallable; expect(h).toBeDefined(); expect(h.kind).toBe("arrow"); expect(edges(h.ddg).some((e) => e.var === "req.query.probe")).toBe(true); }); test("nested anonymous callables chain their segments", () => { - const outer = fns["outer"] as V2Callable; - const first = Object.values(outer.callables ?? {})[0] as V2Callable; - const second = Object.values(first.callables ?? {})[0] as V2Callable; + const outer = fns["outer"] as TSCallable; + const first = Object.values(outer.callables ?? {})[0] as TSCallable; + const second = Object.values(first.callables ?? {})[0] as TSCallable; expect(second.signature.match(//g)).toHaveLength(2); expect(second.id.startsWith(first.id)).toBe(true); }); test("call sites re-anchor from the enclosing callable to the arrow", () => { - const calleeOf = (c: V2Callable): unknown[] => + const calleeOf = (c: TSCallable): unknown[] => Object.values(c.body ?? {}).filter((n) => n.kind === "call").map((n) => n.callee); expect(calleeOf(login)).toEqual([]); - expect(calleeOf(handler)).toEqual([`${(fns["query"] as V2Callable).id}`]); + expect(calleeOf(handler)).toEqual([`${(fns["query"] as TSCallable).id}`]); const srcs = root.call_graph.map((e) => e.src); expect(srcs).toContain(handler.id); @@ -128,7 +129,7 @@ describe("anonymous callables are first-class (issue #92)", () => { }); test("synthesized_callables is a compatibility index onto the tree", () => { - const index = (root.synthesized_callables ?? {}) as Record; + const index = (root.synthesized_callables ?? {}) as Record; expect(index[`${login.id}@2:10`]?.id).toBe(handler.id); // Every entry either points at a tree node or is a residual fallback node keyed by its own id. for (const [key, entry] of Object.entries(index)) { diff --git a/test/dataflow.test.ts b/test/dataflow.test.ts index 155c23a..a539472 100644 --- a/test/dataflow.test.ts +++ b/test/dataflow.test.ts @@ -41,6 +41,7 @@ function options(level: 1 | 2 | 3, cacheDir: string, jobs: number): AnalysisOpti } async function run(level: 1 | 2 | 3, jobs = 1): Promise>> { + // returns the full AnalysisResult — the program-graph IR rides on it, not on the tree const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-dataflow-test-")); try { return await analyze(options(level, cacheDir, jobs)); @@ -49,8 +50,7 @@ async function run(level: 1 | 2 | 3, jobs = 1): Promise { const g = pg.functions[sig]?.cfg; @@ -380,7 +380,7 @@ describe("determinism and gating", () => { test("-a 1 emits no program_graphs section", async () => { const level1 = await run(1); expect(level1.program_graphs).toBeUndefined(); - expect(JSON.stringify(level1)).not.toContain("program_graphs"); + expect(JSON.stringify(level1.application)).not.toContain("program_graphs"); }); test("schema_version and k_limit are stamped", () => { diff --git a/test/external-resolution.test.ts b/test/external-resolution.test.ts index 180915d..86a957e 100644 --- a/test/external-resolution.test.ts +++ b/test/external-resolution.test.ts @@ -48,7 +48,7 @@ function options(): AnalysisOptions { async function run(): Promise { const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-ext-test-")); try { - return await analyze({ ...options(), cacheDir }); + return (await analyze({ ...options(), cacheDir })).internal; } finally { fs.rmSync(cacheDir, { recursive: true, force: true }); } diff --git a/test/multi-tsconfig.test.ts b/test/multi-tsconfig.test.ts index bc73fdd..7084748 100644 --- a/test/multi-tsconfig.test.ts +++ b/test/multi-tsconfig.test.ts @@ -51,7 +51,7 @@ function options(): AnalysisOptions { async function run(): Promise { const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-multitsconfig-test-")); try { - return await analyze({ ...options(), cacheDir }); + return (await analyze({ ...options(), cacheDir })).internal; } finally { fs.rmSync(cacheDir, { recursive: true, force: true }); } diff --git a/test/neo4j-bolt.test.ts b/test/neo4j-bolt.test.ts index dfbdce0..684f7c3 100644 --- a/test/neo4j-bolt.test.ts +++ b/test/neo4j-bolt.test.ts @@ -15,7 +15,7 @@ import neo4j, { type Driver } from "neo4j-driver"; import { type BoltConfig, boltWriter, CONSTRAINTS, INDEXES, project, SCHEMA_VERSION } from "../src/build/neo4j"; import { analyze } from "../src/core"; import type { AnalysisOptions } from "../src/options"; -import { toV2 } from "../src/schema/v2"; +import { finalizeAnalysis } from "../src/schema"; import { Logger } from "../src/utils"; const FIXTURE = path.resolve(import.meta.dir, "fixtures/sample-app"); @@ -93,7 +93,7 @@ containerSuite("neo4j bolt writer", () => { "full push materializes the whole graph + schema", async () => { const opts = optsFor(); - const rows = project(toV2(await analyze(opts), opts)); + const rows = project((await analyze(opts)).application); await boltWriter(rows, cfg, log, true); // Every projected node/edge lands (the fixture has no library deps, so endpoints all resolve). @@ -133,7 +133,7 @@ containerSuite("neo4j bolt writer", () => { "re-pushing identical analysis is idempotent", async () => { const opts = optsFor(); - const rows = project(toV2(await analyze(opts), opts)); + const rows = project((await analyze(opts)).application); await boltWriter(rows, cfg, log, true); expect(await num("MATCH (n) RETURN count(n)")).toBe(rows.nodes.length); expect(await num("MATCH ()-[r]->() RETURN count(r)")).toBe(rows.edges.length); @@ -145,11 +145,12 @@ containerSuite("neo4j bolt writer", () => { "a full run prunes a module whose source vanished", async () => { const opts = optsFor(); - const app = await analyze(opts); + const result = await analyze(opts); + const app = result.internal; const victim = Object.keys(app.symbol_table).sort()[0]; delete app.symbol_table[victim]; - const rows = project(toV2(app, opts)); + const rows = project(finalizeAnalysis(app, result.program_graphs ?? null, opts).application); await boltWriter(rows, cfg, log, true); // The victim's nodes are gone. @@ -182,7 +183,7 @@ containerSuite("neo4j bolt writer", () => { // A full current-version push against the same DB must detect the mismatch and wipe the residue. const opts = optsFor(); - const rows = project(toV2(await analyze(opts), opts)); + const rows = project((await analyze(opts)).application); await boltWriter(rows, cfg, log, true); // Exactly one :Application survives — the fresh v2 one (id set, version bumped). The 1.x app, diff --git a/test/neo4j-schema.test.ts b/test/neo4j-schema.test.ts index 9192a76..02417b7 100644 --- a/test/neo4j-schema.test.ts +++ b/test/neo4j-schema.test.ts @@ -19,7 +19,6 @@ import { } from "../src/build/neo4j"; import { analyze } from "../src/core"; import type { AnalysisOptions } from "../src/options"; -import { toV2Detailed } from "../src/schema/v2"; const FIXTURE = path.resolve(import.meta.dir, "fixtures/dataflow-app"); @@ -34,7 +33,7 @@ async function fixtureRows() { noBuild: true, phantoms: true, callGraphProvider: "union", cacheDir, verbosity: 0, }; try { - return project(toV2Detailed(await analyze(opts), opts).application); + return project((await analyze(opts)).application); } finally { fs.rmSync(cacheDir, { recursive: true, force: true }); } diff --git a/test/schema-v2.test.ts b/test/schema-v2.test.ts index aef3522..eda00ce 100644 --- a/test/schema-v2.test.ts +++ b/test/schema-v2.test.ts @@ -12,8 +12,8 @@ import * as path from "node:path"; import pkg from "../package.json"; import { analyze } from "../src/core"; import type { AnalysisOptions } from "../src/options"; -import type { GraphSelector, TSApplication } from "../src/schema"; -import { type V2Application, type V2Callable, type V2Module, type V2Type, toV2Detailed } from "../src/schema/v2"; +import { forEachCallable, forEachType, type GraphSelector } from "../src/schema"; +import type { AnalysisResult } from "../src/core"; import { type GraphRows, project } from "../src/build/neo4j"; import { tscProvider } from "../src/semantic_analysis"; @@ -44,7 +44,7 @@ function options(): AnalysisOptions { }; } -async function run(): Promise { +async function run(): Promise { const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-test-")); try { return await analyze({ ...options(), cacheDir }); @@ -53,27 +53,28 @@ async function run(): Promise { } } -const v1 = await run(); -const { application: v2, idBySig, collisions } = toV2Detailed(v1, { ...options(), input: FIXTURE }); +const result = await run(); +const v1 = result.internal; +const { application: v2, idBySig, collisions } = result; const root = v2.application; const st = root.symbol_table; /** Every id in the tree, in walk order, for uniqueness + well-formedness checks. */ function allIds(): string[] { const out: string[] = []; - const walkCallable = (c: V2Callable): void => { + const walkCallable = (c: TSCallable): void => { out.push(c.id); for (const cc of Object.values(c.callables ?? {})) walkCallable(cc); for (const t of Object.values(c.types ?? {})) walkType(t); }; - const walkType = (t: V2Type): void => { + const walkType = (t: TSType): void => { out.push(t.id); for (const c of Object.values(t.callables ?? {})) walkCallable(c); for (const f of Object.values(t.fields ?? {})) out.push(f.id); for (const st2 of Object.values(t.types ?? {})) walkType(st2); for (const fn of Object.values(t.functions ?? {})) walkCallable(fn); }; - for (const m of Object.values(st) as V2Module[]) { + for (const m of Object.values(st) as TSModule[]) { out.push(m.id); for (const t of Object.values(m.types)) walkType(t); for (const c of Object.values(m.functions)) walkCallable(c); @@ -121,7 +122,7 @@ describe("schema v2 — L1 identity", () => { }); test("module ids derive from the file key", () => { - for (const [key, m] of Object.entries(st) as [string, V2Module][]) { + for (const [key, m] of Object.entries(st) as [string, TSModule][]) { expect(m.id).toBe(`can://typescript/sample-app/${key}`); expect(m.kind).toBe("module"); } @@ -130,7 +131,7 @@ describe("schema v2 — L1 identity", () => { describe("schema v2 — L1 source & spans (get_method_body)", () => { test("each module carries its full source", () => { - for (const m of Object.values(st) as V2Module[]) expect(typeof m.source).toBe("string"); + for (const m of Object.values(st) as TSModule[]) expect(typeof m.source).toBe("string"); }); test("a callable's span.bytes slices its declaration out of module.source", () => { @@ -159,7 +160,7 @@ describe("schema v2 — L1 tree shape", () => { }); test("a callable's body holds L1 call nodes keyed by line:col with callee null", () => { - const create = st["src/services.ts"].types.UserService.callables?.create as V2Callable; + const create = st["src/services.ts"].types.UserService.callables?.create as TSCallable; const keys = Object.keys(create.body); expect(keys.length).toBeGreaterThan(0); for (const k of keys) expect(k).toMatch(/^\d+:\d+(\/\d+)?$/); @@ -222,21 +223,15 @@ describe("schema v2 — inheritance resolves heritage signatures to can:// ids ( }); }); -describe("schema v2 — L1 superset", () => { - test("every v1 callable/type signature has a v2 id", () => { - const v1sigs = new Set(); - const addType = (t: { signature: string; methods?: Record }): void => { - v1sigs.add(t.signature); - for (const m of Object.values(t.methods ?? {})) v1sigs.add(m.signature); - }; +describe("schema v2 — L1 id registration", () => { + test("every tree callable/type signature has a v2 id", () => { + const sigs = new Set(); for (const m of Object.values(v1.symbol_table)) { - for (const t of Object.values(m.classes)) addType(t); - for (const t of Object.values(m.interfaces)) addType(t); - for (const t of Object.values(m.enums)) v1sigs.add(t.signature); - for (const t of Object.values(m.type_aliases)) v1sigs.add(t.signature); - for (const fn of Object.values(m.functions)) v1sigs.add(fn.signature); + forEachCallable(m, (c) => sigs.add(c.signature)); + forEachType(m, (t) => sigs.add(t.signature)); } - const missing = [...v1sigs].filter((s) => !idBySig.has(s)); + expect(sigs.size).toBeGreaterThan(0); + const missing = [...sigs].filter((s) => !idBySig.has(s)); expect(missing).toEqual([]); }); }); @@ -251,7 +246,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-")); try { - const v1L1 = await analyze({ ...options(), analysisLevel: 1, callGraphProvider: "tsc", cacheDir }); + const v1L1 = (await analyze({ ...options(), analysisLevel: 1, callGraphProvider: "tsc", cacheDir })).internal; expect(spy).not.toHaveBeenCalled(); expect(v1L1.call_graph).toEqual([]); expect(Object.keys(v1L1.external_symbols)).toEqual([]); @@ -266,7 +261,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-l2-")); try { - const v1L2guard = await analyze({ ...options(), analysisLevel: 2, callGraphProvider: "tsc", cacheDir }); + const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, callGraphProvider: "tsc", cacheDir })).internal; expect(spy).toHaveBeenCalledTimes(1); expect(v1L2guard.call_graph.length).toBeGreaterThan(0); } finally { @@ -277,7 +272,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { }); // ---- L2: call graph ------------------------------------------------------------------------- -async function runL2(): Promise { +async function runL2(): Promise { const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l2-")); try { return await analyze({ ...options(), analysisLevel: 2, cacheDir }); @@ -285,8 +280,7 @@ async function runL2(): Promise { fs.rmSync(cacheDir, { recursive: true, force: true }); } } -const v1L2 = await runL2(); -const { application: v2L2, idBySig: idsL2, dangling: danglingL2 } = toV2Detailed(v1L2, { ...options(), analysisLevel: 2, input: FIXTURE }); +const { application: v2L2, idBySig: idsL2, dangling: danglingL2 } = await runL2(); const rootL2 = v2L2.application; const knownIds = new Set(idsL2.values()); @@ -309,7 +303,7 @@ describe("schema v2 — L2 call graph", () => { test("body call-node callees are backfilled to a known id (null → id refinement)", () => { let sawBackfill = false; - const scan = (c: V2Callable): void => { + const scan = (c: TSCallable): void => { for (const b of Object.values(c.body)) { if (b.kind === "call" && b.callee != null) { sawBackfill = true; @@ -341,7 +335,7 @@ const DF_FIXTURE = path.resolve(import.meta.dir, "fixtures/dataflow-app"); function dfOptions(level: 3 | 4): AnalysisOptions { return { ...options(), input: DF_FIXTURE, analysisLevel: level, graphs: level >= 4 ? ["cfg", "dfg", "pdg", "sdg"] : ["cfg", "dfg", "pdg"] }; } -async function runDF(level: 3 | 4): Promise { +async function runDF(level: 3 | 4): Promise { const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), `cants-v2-l${level}-`)); try { return await analyze({ ...dfOptions(level), cacheDir }); @@ -351,14 +345,14 @@ async function runDF(level: 3 | 4): Promise { } /** Every callable node in the tree (module/type/namespace/nested recursion). */ -function allCallables(root: { symbol_table: Record }): V2Callable[] { - const out: V2Callable[] = []; - const wc = (c: V2Callable): void => { +function allCallables(root: { symbol_table: Record }): TSCallable[] { + const out: TSCallable[] = []; + const wc = (c: TSCallable): void => { out.push(c); for (const cc of Object.values(c.callables ?? {})) wc(cc); for (const t of Object.values(c.types ?? {})) wt(t); }; - const wt = (t: V2Type): void => { + const wt = (t: TSType): void => { for (const c of Object.values(t.callables ?? {})) wc(c); for (const st of Object.values(t.types ?? {})) wt(st); for (const fn of Object.values(t.functions ?? {})) wc(fn); @@ -370,8 +364,8 @@ function allCallables(root: { symbol_table: Record }): V2Calla return out; } -const dfL3 = toV2Detailed(await runDF(3), dfOptions(3)).application; -const dfL4 = toV2Detailed(await runDF(4), dfOptions(4)).application; +const dfL3 = (await runDF(3)).application; +const dfL4 = (await runDF(4)).application; /** No intra-callable (bare-local) or cross-callable (canId@local) endpoint may dangle. */ function danglingCount(app: typeof dfL3): number { @@ -441,7 +435,7 @@ describe("schema v2 — L3 intraprocedural dataflow", () => { test("a sampled L3 statement node is source-sliceable (span.bytes reproduces its text)", () => { const mod = dfL3.application.symbol_table["src/flow.ts"]; - const classify = mod.functions.classify as V2Callable; + const classify = mod.functions.classify as TSCallable; const stmt = classify.body["4:3"]; expect(stmt?.kind).toBe("statement"); const [s, e] = (stmt as { span: { bytes: [number, number] } }).span.bytes; @@ -454,7 +448,7 @@ describe("schema v2 — L3 intraprocedural dataflow", () => { // nodes (which slice out just their own text), entry/exit should reproduce the callable's // entire declaration, including its `export` modifier. const mod = dfL3.application.symbol_table["src/flow.ts"]; - const classify = mod.functions.classify as V2Callable; + const classify = mod.functions.classify as TSCallable; const entry = classify.body["@entry"]; const exit = classify.body["@exit"]; expect(entry?.kind).toBe("entry"); @@ -552,7 +546,7 @@ describe("schema v2 — L4 interprocedural SDG", () => { // ---- Real end-to-end monotonicity gate + Neo4j↔JSON count parity (issue #27) ------------------- // // `analyze()` run fresh at each of `-a 1/2/3/4` on dataflow-app (four full runs — the fixture is -// small), then the emitted V2Application at each level is reduced to three KEY-sets — symbol-table +// small), then the emitted TSAnalysis at each level is reduced to three KEY-sets — symbol-table // ids, body-node keys, and edge keys (call_graph / cfg / cdg / ddg / summary / param_in / param_out) // — and every level's set must be a superset of the level below (canonical-schema.md's additive // invariant). Comparing KEYS (not values) means the one sanctioned value mutation, a `call` node's @@ -566,25 +560,24 @@ function optsAt(level: 1 | 2 | 3 | 4): AnalysisOptions { return { ...options(), input: DF_FIXTURE, analysisLevel: level, graphs }; } -async function appAt(level: 1 | 2 | 3 | 4): Promise { +async function appAt(level: 1 | 2 | 3 | 4): Promise { const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-mono-")); try { - const v1run = await analyze({ ...optsAt(level), cacheDir }); - return toV2Detailed(v1run, optsAt(level)).application; + return (await analyze({ ...optsAt(level), cacheDir })).application; } finally { fs.rmSync(cacheDir, { recursive: true, force: true }); } } /** Every symbol-table id — module/type/callable/field — recursing through nested scopes. */ -function symbolIds(app: V2Application): Set { +function symbolIds(app: TSAnalysis): Set { const out = new Set(); - const walkCallable = (c: V2Callable): void => { + const walkCallable = (c: TSCallable): void => { out.add(c.id); for (const cc of Object.values(c.callables ?? {})) walkCallable(cc); for (const t of Object.values(c.types ?? {})) walkType(t); }; - const walkType = (t: V2Type): void => { + const walkType = (t: TSType): void => { out.add(t.id); for (const c of Object.values(t.callables ?? {})) walkCallable(c); for (const f of Object.values(t.fields ?? {})) out.add(f.id); @@ -601,13 +594,13 @@ function symbolIds(app: V2Application): Set { } /** Every type node — classes/interfaces/enums/aliases/namespaces — recursing through nested scopes. */ -function allTypes(app: V2Application): V2Type[] { - const out: V2Type[] = []; - const walkCallable = (c: V2Callable): void => { +function allTypes(app: TSAnalysis): TSType[] { + const out: TSType[] = []; + const walkCallable = (c: TSCallable): void => { for (const cc of Object.values(c.callables ?? {})) walkCallable(cc); for (const t of Object.values(c.types ?? {})) walkType(t); }; - const walkType = (t: V2Type): void => { + const walkType = (t: TSType): void => { out.push(t); for (const c of Object.values(t.callables ?? {})) walkCallable(c); for (const nested of Object.values(t.types ?? {})) walkType(nested); @@ -621,7 +614,7 @@ function allTypes(app: V2Application): V2Type[] { } /** Every body-node key, namespaced by its owning callable id so bare local keys never collide. */ -function bodyKeys(app: V2Application): Set { +function bodyKeys(app: TSAnalysis): Set { const out = new Set(); for (const c of allCallables(app.application)) { for (const k of Object.keys(c.body)) out.add(`${c.id}::${k}`); @@ -630,7 +623,7 @@ function bodyKeys(app: V2Application): Set { } /** Every edge key: call_graph + param_in/param_out (app scope), cfg/cdg/ddg/summary (per callable). */ -function edgeKeys(app: V2Application): Set { +function edgeKeys(app: TSAnalysis): Set { const out = new Set(); const root = app.application; for (const e of root.call_graph) out.add(`call_graph::${e.src}>${e.dst}:${[...e.prov].sort().join(",")}`); @@ -656,7 +649,7 @@ const monoApp1 = await appAt(1); const monoApp2 = await appAt(2); const monoApp3 = await appAt(3); const monoApp4 = await appAt(4); -const monoPairs: Array<[V2Application, V2Application, string]> = [ +const monoPairs: Array<[TSAnalysis, TSAnalysis, string]> = [ [monoApp1, monoApp2, "L1 -> L2"], [monoApp2, monoApp3, "L2 -> L3"], [monoApp3, monoApp4, "L3 -> L4"], @@ -701,7 +694,7 @@ function relCount(rows: GraphRows, type: string): number { } /** Every id materialized as a `:CanNode` row: symbol-table ids + body-node fq ids + external/synth. */ -function canNodeIds(app: V2Application): Set { +function canNodeIds(app: TSAnalysis): Set { const ids = new Set(symbolIds(app)); for (const c of allCallables(app.application)) { for (const k of Object.keys(c.body)) ids.add(k.startsWith("@") ? `${c.id}${k}` : `${c.id}@${k}`); @@ -712,7 +705,7 @@ function canNodeIds(app: V2Application): Set { } /** Every `call` body node whose `callee` resolved to an id — the JSON-side source of RESOLVES_TO. */ -function resolvesToCount(app: V2Application): number { +function resolvesToCount(app: TSAnalysis): number { let n = 0; for (const c of allCallables(app.application)) { for (const bn of Object.values(c.body)) if (typeof bn.callee === "string") n++; diff --git a/test/synthesized-nodes.test.ts b/test/synthesized-nodes.test.ts index 0e13ed3..018cb88 100644 --- a/test/synthesized-nodes.test.ts +++ b/test/synthesized-nodes.test.ts @@ -7,8 +7,7 @@ import { describe, expect, test } from "bun:test"; import { project } from "../src/build/neo4j"; import type { AnalysisOptions } from "../src/options"; -import { CALL_DEP, type TSApplication, type TSCallable, type TSModule, type TSSpan } from "../src/schema"; -import { toV2Detailed } from "../src/schema/v2"; +import { CALL_DEP, type AnalysisInternal, type TSCallable, type TSModule, type TSSpan, finalizeAnalysis } from "../src/schema"; const ANON = "src/x.foo:<3:10>"; const SPAN: TSSpan = { start: [1, 1], end: [5, 1], bytes: [0, 10] }; @@ -16,7 +15,7 @@ const SPAN: TSSpan = { start: [1, 1], end: [5, 1], bytes: [0, 10] }; const callable = (signature: string, name: string): TSCallable => ({ signature, name, kind: "function", span: SPAN, parameters: [], call_sites: [], inner_callables: {}, inner_classes: {} }) as unknown as TSCallable; -const app: TSApplication = { +const app: AnalysisInternal = { symbol_table: { "src/x.ts": { module_name: "src/x", source: "", span: SPAN, @@ -30,7 +29,7 @@ const app: TSApplication = { }; const opts = { appName: "t", input: "", analysisLevel: 2 } as unknown as AnalysisOptions; -const { application, idBySig } = toV2Detailed(app, opts); +const { application, idBySig } = finalizeAnalysis(app, null, opts); const rows = project(application); const fooId = idBySig.get("src/x.foo") as string;