Merge pull request #158 from codellm-devkit/design/issue-157-artifacts-spec

docs(design): spec artifacts/dependencies schema v2 foundation

Author: rahlk

.claude/SCHEMA_DECISIONS.md

@@ -284,3 +284,30 @@ made output load-dependent (#145).
 - Refinement contract unchanged: the linker runs inside the L2 build, so
   `callee: null→id` remains the single sanctioned L1→L2 refinement.
 - Neo4j projection unchanged (`PY_CALLS` carries `prov` as data).
+
+## 2026-08-27 — Artifacts, dependencies, and the `can://artifact/` namespace
+
+Design: `docs/design/specs/2026-08-27-artifacts-and-dependencies-design.md`.
+
+Schema v2 gains non-code coverage: `application.artifacts` (sibling map,
+`symbol_table` stays code-only), `application.dependencies`, and
+`application.unresolved_imports`. All three are L1 data — emitted identically
+at every level, like entrypoints.
+
+- **Artifact ids are language-neutral**: `can://artifact/<app>/<path>`. The
+  first `can://` segment is now a namespace — a language for code nodes, the
+  literal `artifact` for files — so sibling analyzers over the same repo emit
+  the same artifact id (one node in a merged graph). Precondition: `<app>`
+  must agree (`--app-name` pinned for joint analysis).
+- **Dependency `prov` vocabulary** (coined once, parity clause applies):
+  `declared`, `lockfile`, `installed-metadata`, `heuristic`. Deterministic
+  default reads repo files only; `installed-metadata` requires the new
+  `--resolve-installed` flag.
+- **Neo4j**: neutral labels `:Artifact` / `:Package` (no `Py` prefix, shared
+  MERGE targets across analyzers); `:Package.id` is a purl (`pkg:pypi/<name>`).
+  `PY_PROVIDES` joins packages to the existing `:PyExternal` ghost ids, wiring
+  dependencies into the call graph. New edges: `HAS_ARTIFACT`,
+  `DECLARES_DEPENDENCY`, `LOCKS`, `PY_PROVIDES`, `PY_UNRESOLVED_IMPORT`.
+- Capture broad (config files as nodes with `roles`), extract narrow
+  (dependency manifests only this unit). Lock files backfill
+  `locked_version`, never create records; transitive packages out of scope.

CLAUDE.md

@@ -17,6 +17,13 @@ Respect the global `~/.claude/CLAUDE.md` instructions strictly.
   tracked past a global gitignore via `!`-negations in `.gitignore`; keep those
   negations if you touch the ignore file.
 
+## Querying the emitted graph
+
+`docs/skills/analyzing-canpy-graphs/` is a reference skill (tool-neutral, repo-shared)
+for querying the analyzer's own output — vocabulary tables and recipes for entrypoint,
+taint, exit-point, and slicing queries over `analysis.json` and the Neo4j projection.
+Read it before writing Cypher or JSON traversals over schema v2.
+
 ## Schema v2 — the model this analyzer emits
 
 `codeanalyzer-python` emits **canonical schema v2** (`schema_version` `2.0.0`): one

README.md

@@ -576,6 +576,56 @@ runtime (Docker or Podman) and is enabled with an environment variable:
 RUN_CONTAINER_TESTS=1 uv run pytest test/test_neo4j_bolt.py -s
 ```
 
+## Graph query cookbook
+
+Example Cypher over the projected graph (`--emit neo4j`, then load `graph.cypher` or push via Bolt).
+
+```cypher
+// who calls this function? (direct callers)
+MATCH (c:PyCallable)-[:PY_CALLS]->(t:PyCallable {name: "process_payment"})
+RETURN c.id
+
+// every callable that reaches a given library, via the external ghosts
+MATCH (c:PyCallable)-[:PY_CALLS]->(e:PyExternal)
+WHERE e.id CONTAINS "/@external/requests/"
+RETURN DISTINCT c.id
+
+// entrypoints and the frameworks that invoke them
+MATCH (m:PyCallable {is_entrypoint: true})
+RETURN m.id, m.entrypoint_frameworks
+
+// data dependences into one statement (level 3+)
+MATCH (s:PyBodyNode {id: $stmt})<-[d:PY_DDG]-(src:PyBodyNode)
+RETURN src.id, d.var, d.prov
+
+// interprocedural flow through a parameter (level 4)
+MATCH (a:PyBodyNode)-[:PY_PARAM_IN]->(f:PyBodyNode)
+WHERE f.id STARTS WITH "can://python/myapp/src/api.py"
+RETURN a.id, f.id
+```
+
+Landing with #157 (schema v2 artifacts + dependencies, 1.3.0) — not in the current release:
+
+```cypher
+// all container/orchestration configs in the app
+MATCH (a:PyApplication)-[:HAS_ARTIFACT]->(f:Artifact)
+WHERE any(r IN f.roles WHERE r IN ["service-topology", "container-image"])
+RETURN f.id, f.format
+
+// every callable that reaches code from a declared package
+MATCH (c:PyCallable)-[:PY_CALLS]->(:PyExternal)<-[:PY_PROVIDES]-(p:Package {id: "pkg:pypi/requests"})
+RETURN c.id
+
+// undeclared imports (dependency hygiene)
+MATCH (a:PyApplication)-[u:PY_UNRESOLVED_IMPORT]->(e:PyExternal)
+WHERE NOT (e)<-[:PY_PROVIDES]-(:Package)
+RETURN e.id, u.prov
+
+// which lock file pins this package, and to what
+MATCH (f:Artifact)-[l:LOCKS]->(p:Package {id: "pkg:pypi/numpy"})
+RETURN f.id, l.version
+```
+
 ## License
 
 Apache 2.0 — see [LICENSE](./LICENSE).

docs/design/specs/2026-08-27-artifacts-and-dependencies-design.md

@@ -0,0 +1,194 @@
+# Artifacts and Dependencies: Schema v2 Foundation for Non-Code Files
+
+**Date:** 2026-08-27
+**Status:** Approved (design dialogue in-session; decomposition: one issue, one PR)
+**Scope:** codeanalyzer-python, schema v2 additive change (targets 1.3.0)
+
+## Problem
+
+Schema v2 is code-only by construction: `symbol_table` is keyed by `.py` file, and
+nothing represents configuration files, dependency manifests, or the packages an
+application depends on. This blocks three future capabilities — cross-service
+topology, cross-language links, and dependency-aware queries — and leaves basic
+questions ("which packages does this app declare?", "which imports are
+undeclared?") unanswerable from the analysis output.
+
+This spec is unit 1 of a four-unit arc (foundation, cross-service topology,
+cross-language identity, config extractors). It designs the foundation only:
+how a non-code file becomes a node, and dependency manifests as the first
+extracted meaning.
+
+## Locked decisions
+
+### 1. Placement: sibling map, not symbol_table widening
+
+`symbol_table` stays strictly code (`Dict[file, PyModule]`). Non-code files live
+in a parallel map on the application node:
+
+```
+application.artifacts: Dict[str, PyArtifact]   # keyed by repo-relative POSIX path
+```
+
+### 2. Artifact identity is language-neutral
+
+```
+can://artifact/<app>/<repo-relative-path>
+```
+
+The first `can://` segment becomes a namespace: a language (`python`, `java`,
+`typescript`) for code nodes, the literal `artifact` for non-code files. Two
+analyzers over the same monorepo emit the **same id** for the same file — one
+node in a merged graph, never per-language duplicates.
+
+**Precondition for cross-analyzer joins:** `<app>` must agree between analyzers.
+`--app-name` defaults to the input directory name, so analyzers pointed at
+different subdirectories of a monorepo will disagree; joint analysis must pin
+`--app-name` explicitly.
+
+### 3. PyArtifact model
+
+| field | type | notes |
+| --- | --- | --- |
+| `id` | str | `can://artifact/<app>/<path>` |
+| `kind` | `"artifact"` | schema-v2 node discriminant |
+| `format` | str | `toml` \| `yaml` \| `json` \| `ini` \| `requirements` \| `dockerfile` \| `text` |
+| `roles` | List[str] | `dependency-manifest`, `service-topology`, `container-image`, `ci`, `env`, `tool-config`, `unknown` |
+| `size_bytes` | int | |
+| `sha256` | str | |
+| `source` | str | verbatim content, **no size bound** (user decision: do not bound file size yet) |
+| `extraction` | `"full"` \| `"partial"` \| `"none"` | `partial` currently only for dynamic `setup.py` |
+
+Capture is **broad** (every recognized config-shaped file becomes a node);
+extraction is **narrow** (only dependency manifests get an extractor in this
+unit). Later units add extractors — additive edges/records on existing nodes,
+never re-keying.
+
+Discovery is a shipped rules table of filename patterns → `(format, roles)`,
+same mechanism family as `entrypoints/rules.yml`. No user-extension flag yet.
+
+### 4. Dependency model: declared + evidence-tagged binding
+
+```
+application.dependencies: List[PyDependency]
+application.unresolved_imports: List[PyImportBinding]
+```
+
+`PyDependency`: `name` (PEP 503 normalized), `spec`, `kind`
+(`runtime`|`dev`|`optional`|`build`), `extras`, `declared_in` (artifact id),
+`locked_version` (optional), `provides_imports` (top-level import names),
+`prov` (list).
+
+`prov` vocabulary (same idiom as call-edge provenance):
+
+- `declared` — read from a manifest
+- `lockfile` — pinned version from a lock file
+- `installed-metadata` — read from the venv's `.dist-info` (opt-in only)
+- `heuristic` — name-match fallback for import binding
+
+`unresolved_imports` is first-class output: every top-level import the symbol
+table saw that no declared dependency accounts for, with any partial binding
+and its `prov`. This is deliberately the interesting section — it surfaces
+undeclared dependencies instead of silently omitting them.
+
+Lock files never create dependency records; they only backfill
+`locked_version` on declared ones. Transitive (lock-only) packages are
+deliberately skipped — no transitive graph in this unit.
+
+### 5. Determinism: deterministic default, probing opt-in
+
+The default run reads only files in the repo — byte-identical output across
+machines. A new flag `--resolve-installed` additionally probes the venv's
+installed metadata for import→distribution mapping; those records carry
+`prov ["installed-metadata"]`. The CI determinism gate runs the default.
+
+### 6. Extraction targets (this unit)
+
+| manifest | extracted | notes |
+| --- | --- | --- |
+| `requirements*.txt` | declared deps; `-r`/`-c` includes chased | `kind` from filename convention |
+| `pyproject.toml` | PEP 621 `[project.dependencies]` + `optional-dependencies`; Poetry `[tool.poetry.*]`; `[build-system].requires` (`kind: build`) | one parser, three dialects |
+| `setup.py` | `install_requires`/`extras_require` via **static AST only**; literals lifted, never executed | dynamic values → artifact `extraction: "partial"`; imports then surface via `unresolved_imports` |
+| `setup.cfg` | `[options] install_requires`, `extras_require` | ini parse |
+| `Pipfile` / `Pipfile.lock` | declared + pins | |
+| `poetry.lock`, `uv.lock` | `locked_version` backfill | |
+| `environment.yml` | conda deps incl. `pip:` sublist | |
+
+### 7. Neo4j projection
+
+Language-neutral subgraph gets language-neutral labels (no `Py` prefix), so
+sibling analyzers MERGE onto the same nodes:
+
+| label | merge key | properties |
+| --- | --- | --- |
+| `:Artifact` | `id` (`can://artifact/...`) | path, format, roles, sha256, size_bytes, source |
+| `:Package` | `id` = purl (`pkg:pypi/<name>`) | ecosystem, name |
+
+purl as package id is the cross-language join: `pkg:maven/...` and
+`pkg:pypi/...` coexist uniformly.
+
+Edges:
+
+```
+(:PyApplication)-[:HAS_ARTIFACT]->(:Artifact)
+(:Artifact)-[:DECLARES_DEPENDENCY {spec, kind, extras, prov}]->(:Package)
+(:Artifact)-[:LOCKS {version}]->(:Package)
+(:Package)-[:PY_PROVIDES]->(:PyExternal)
+(:PyApplication)-[:PY_UNRESOLVED_IMPORT {prov}]->(:PyExternal)
+```
+
+`PY_PROVIDES` targets the **existing** `:PyExternal` ghosts (same
+`can://python/<app>/@external/<module>` ids the L2 call graph MERGEs on), so
+dependencies join the call graph rather than sit beside it:
+
+```cypher
+MATCH (c:PyCallable)-[:PY_CALLS]->(:PyExternal)<-[:PY_PROVIDES]-(p:Package {id:"pkg:pypi/requests"})
+RETURN c.id
+```
+
+Config-role artifacts get node + roles + source and zero extracted edges this
+unit. New DDL: unique constraints on `Artifact.id` and `Package.id`. Neo4j
+schema version moves additively within 2.x. Full-depth-always rule unchanged.
+
+### 8. Pipeline and CLI
+
+- Artifact scan is **L1 data**: runs at every level, output must not vary with
+  `-a` (same posture as entrypoints). Monotonicity gate holds trivially.
+- Runs after the symbol table (needs module import lists for
+  `unresolved_imports`), before the call graph.
+- New package `codeanalyzer/artifacts/`: `discovery.py` (walk + rules table),
+  `parsers.py` (toml/yaml/ini/requirements/setup.py-AST readers),
+  `dependencies.py` (records, lock backfill, import binding). Walk reuses the
+  symbol table's ignore set, sorted order.
+- CLI: exactly one new flag, `--resolve-installed`. Scan is default-on with no
+  toggle.
+- Not cached: scan cost is trivial; caching would add invalidation surface for
+  nothing.
+
+## Caveats
+
+- `<app>` agreement is a precondition for cross-analyzer artifact joins (see §2).
+- `setup.py` extraction is static-AST only; computed dependency lists are
+  recorded as `extraction: "partial"`, never executed (determinism).
+- Lock-only (transitive) packages are out of scope by decision, not omission.
+- `source` is unbounded by decision; revisit only with measured payload numbers.
+- Import→package binding without `--resolve-installed` relies on
+  `declared` names + `heuristic` matching; `installed-metadata` precision is
+  opt-in and machine-dependent by design.
+
+## Decomposition and release plan
+
+One work-item issue on codeanalyzer-python, closed by one PR; ships in the
+next minor (1.3.0, additive). Sibling analyzers adopt the `can://artifact/`
+namespace, neutral Neo4j labels, and purl ids when their own work starts — no
+epic until a second repo does.
+
+## Definition of done
+
+- `application.artifacts` / `dependencies` / `unresolved_imports` emitted at
+  every level with identical content; monotonicity gate green.
+- All §6 formats parsed on a fixture project carrying every format; prov and
+  purl ids asserted.
+- Neo4j rows for §7 vocabulary via existing row tests; DDL constraints added.
+- Default run byte-identical across two consecutive runs;
+  `--resolve-installed` exercised in one gated test.
+- Full suite green; schema decision recorded in `.claude/SCHEMA_DECISIONS.md`.

docs/skills/analyzing-canpy-graphs/SKILL.md

@@ -0,0 +1,62 @@
+---
+name: analyzing-canpy-graphs
+description: Use when querying codeanalyzer-python (canpy) output — analysis.json or the Neo4j projection — for call-graph, dataflow, taint, entrypoint, exit-point, or slicing questions, or when writing Cypher/JSON traversals over schema v2.
+---
+
+# Analyzing canpy graphs (schema v2)
+
+One additive tree + typed edge overlays, two projections: `analysis.json` and Neo4j.
+Vocabulary is fixed — **never guess a label, property, or key shape**; it is all in
+[references/vocabulary.md](references/vocabulary.md). Query recipes (entrypoints, taint,
+exit points, slicing, both projections) are in [references/recipes.md](references/recipes.md).
+
+## What exists at which level
+
+| `-a` | tree | edges |
+| --- | --- | --- |
+| 1 | callables + `call` body nodes (`callee: null`) + entrypoints | — |
+| 2 | `callee` backfilled | `call_graph` / `PY_CALLS` (prov `jedi`/`defuse`) |
+| 3 | full `body`, `@entry`/`@exit` | `cfg`, `cdg`, `ddg` (prov `ssa`, `reaching-defs`) |
+| 4 | `formal_in/out`, `actual_in/out` vertices | `param_in`, `param_out`, `summary`, ddg widened with prov `points-to` |
+
+Entrypoints (`is_entrypoint`, `entrypoint_frameworks` on callables **and** classes) are
+L1 data — present at every level. Interprocedural anything needs `-a 4`.
+
+## Identity in 20 seconds
+
+- **`can://` ids are opaque** — read fields, never delimiter-split.
+- **LOCAL ids** (body-map keys, intra-callable edge endpoints): `"line:col"`,
+  `"@entry"`, `"@exit"`, `"@formal_in:<i>"`, `"@formal_out"`,
+  `"<callsite-local>/actual_in:<i>"`, `"<callsite-local>/actual_out"`.
+- **GLOBAL ids** (Neo4j `PyBodyNode.id`, `param_in/out` endpoints): `"<callable-id>@<local>"`.
+
+## The four traps (each observed in baseline testing)
+
+1. **Container key asymmetry.** `PyModule.types` and `PyClass.types` are keyed by the
+   **dotted signature** (`"src.flask.sessions.SessionMixin"`); `callables` / `functions`
+   are keyed by the **bare name** (`"permanent"`). Class signatures derive from the
+   module path — a repo with a `src/` layout has `src.`-prefixed signatures.
+2. **Span slicing is bytes, not str.** `span.bytes` are UTF-8 byte offsets:
+   `module.source.encode("utf-8")[lo:hi].decode("utf-8")`. A plain `source[lo:hi]`
+   is silently wrong after the first non-ASCII character in the file.
+3. **Bound your taint walks.** `-[:PY_DDG|…*1..]->` enumerates paths — exponential on
+   real corpora (odoo L4: 4.6M DDG edges). Use the bounded/frontier recipes in
+   references/recipes.md.
+4. **`PY_DDG` is one edge per `(var, prov)`** (internal `_k` discriminant), and ddg
+   `prov` at L4 mixes `ssa`/`reaching-defs`/`points-to` — filter on `prov` when you
+   need only the syntactic subset.
+
+## Exit points, defined
+
+Data leaves a callable through three channels; class-level = union over
+`PY_HAS_METHOD`:
+
+- **return channel** — `body` nodes with `kind: "return"` (L4: `@formal_out`).
+- **external calls** — `PY_CALLS` into `:PyExternal` / `callee` present in
+  `application.external_symbols`.
+- **caller-visible writes** — ddg edges whose `var` is rooted at `self.` or a
+  `global:` path; L4 `summary` edges expose the transitive in→out relation.
+
+Taint is **not a schema section** (provider/client boundary: sources/sinks are the
+SDK's job) — you compose it as reachability over `PY_DDG ∪ PY_PARAM_IN ∪
+PY_PARAM_OUT ∪ PY_SUMMARY`. Recipes file has the exact patterns.