Add VECTORA 0.1.0: repository indexing engine and knowledge-graph explorer

.env.example

@@ -0,0 +1,20 @@
+# VECTORA local engine configuration. Copy to .env or set in your shell.
+# All values are optional; defaults are shown.
+
+# Loopback address and port of the local engine.
+VECTORA_HOST=127.0.0.1
+VECTORA_PORT=4317
+
+# Where VECTORA keeps its database, imported sources and caches.
+# Default: .vectora-data inside the VECTORA folder (uploads and temp files live here too).
+# VECTORA_DATA_DIR=
+
+# Extra allowed browser origins (comma separated). The engine only accepts
+# same-origin requests plus the Vite dev server by default.
+# VECTORA_ALLOWED_ORIGINS=http://localhost:5173
+
+# Log level: fatal | error | warn | info | debug
+VECTORA_LOG_LEVEL=info
+
+# Python interpreter used for Python source analysis.
+# VECTORA_PYTHON=python

.gitattributes

@@ -1,2 +1,3 @@
-# Auto detect text files and perform LF normalization
-* text=auto
+* text=auto eol=lf
+*.wasm binary
+*.png binary

.github/pull_request_template.md

@@ -0,0 +1,11 @@
+## What changed
+
+## Why
+
+## How it was checked
+
+- [ ] `npm run typecheck` and `npm run lint`
+- [ ] `npm test`
+- [ ] `npm run build` and `npm run test:e2e` (for interface changes)
+
+## Notes for the reviewer

.github/workflows/ci.yml

@@ -0,0 +1,42 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  verify:
+    name: Type check, lint, test, build
+    # Windows is the platform VECTORA has been developed and tested on.
+    runs-on: windows-latest
+    timeout-minutes: 25
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 24
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: npm ci --no-audit --no-fund
+      - name: Type check
+        run: npm run typecheck
+      - name: Lint
+        run: npm run lint
+      - name: Unit and integration tests
+        run: npm test
+      - name: Production build
+        run: npm run build
+      - name: Dependency audit
+        run: npm audit --omit=dev

.gitignore

@@ -0,0 +1,16 @@
+node_modules/
+dist/
+coverage/
+.vectora-data/
+test-results/
+playwright-report/
+*.log
+.env
+.env.*
+!.env.example
+.DS_Store
+Thumbs.db
+*.tsbuildinfo
+.bench-tmp/
+.npm-cache/
+.claude/

.npmrc

@@ -0,0 +1 @@
+cache=.npm-cache

.prettierignore

@@ -0,0 +1,10 @@
+node_modules
+dist
+coverage
+samples
+package-lock.json
+*.wasm
+.vectora-data
+test-results
+playwright-report
+docs/LICENSES.md

.prettierrc.json

@@ -0,0 +1,7 @@
+{
+  "semi": true,
+  "singleQuote": true,
+  "trailingComma": "all",
+  "printWidth": 110,
+  "tabWidth": 2
+}

DECISIONS.md

@@ -0,0 +1,92 @@
+# Technical decisions
+
+Decisions made while building VECTORA 0.1, with the reasoning. Newest last.
+
+## D1 — npm workspaces, TypeScript source exports, one `tsconfig`
+
+pnpm and yarn were not installed; npm workspaces need nothing extra. Packages export their TypeScript source (`"exports": "./src/index.ts"`), so there is no per-package build: `tsx` runs the engine in development, Vite and esbuild bundle for release, and one strict `tsconfig.json` type-checks everything.
+
+## D2 — TypeScript 5.9, not 7
+
+The latest `typescript` on the registry is 7.x, the native rewrite, whose programmatic API differs from the Compiler API that the TypeScript adapter depends on (`ts.createProgram`, the type checker, custom module resolution). TypeScript is pinned to `~5.9.3`, the last line with the classic API. Revisit when the 7.x API is stable and documented.
+
+## D3 — `web-tree-sitter` with vendored grammar WASM files
+
+Native Tree-sitter bindings need a compiler toolchain on Windows. `web-tree-sitter` runs everywhere. The widely used `tree-sitter-wasms` package was tested and does **not** load with `web-tree-sitter` 0.27 (ABI mismatch); the WASM files shipped inside the official grammar packages do. Those six files are vendored in `packages/language-adapters/grammars` (MIT, versions listed in `docs/LICENSES.md`) so installation runs no native build scripts.
+
+## D4 — TypeScript Compiler API with the type checker for TypeScript and JavaScript
+
+The brief asks for precise symbol analysis and forbids assuming a call is resolved because a similarly named symbol exists. The type checker gives exactly that: an identifier becomes a relationship only when the checker ties it to a declaration in the repository. Cost: one compiler program per repository, single-threaded (see `docs/PERFORMANCE.md`). Module resolution is implemented by VECTORA against the indexed file list so the compiler never reads `node_modules`.
+
+## D5 — Python analysed by a helper script, not a JavaScript parser
+
+The brief names the Python AST. A small script using the standard-library `ast` module is run with the system interpreter in isolated mode. Trade-off: Python files are not analysed on machines without Python 3; this is detected and shown.
+
+## D6 — Name-based references resolved only with scope evidence
+
+For Python and the Tree-sitter languages there is no type information. The resolver uses, in order: same-file definitions, import bindings, and language visibility rules (same package, included header), and marks the last kind as unverified at confidence 0.7. Ambiguous names and calls on receivers of unknown type are reported as unresolved. This under-reports relationships on purpose; a wrong edge is worse than a missing one.
+
+## D7 — SQLite through `node:sqlite`, no ORM
+
+`better-sqlite3` needs a native binary; `node:sqlite` ships with Node.js 22.13 and later. It is still labelled experimental by Node.js, which is a risk recorded in `KNOWN_ISSUES.md`; the store is one file behind a small class, so swapping the driver is contained. Drizzle was not added: the schema is eight tables with JSON payloads and a handful of statements, and an ORM would add a dependency without removing code.
+
+## D8 — GitHub repositories downloaded as archives; no git commands
+
+Cloning runs git, and git in an untrusted repository can execute configured programs. Downloading the source archive through the REST API (Octokit) avoids executing anything, keeps the token in a header, and reuses the hardened ZIP path. `simple-git` was therefore not needed and was removed. For local folders the branch name is read from `.git/HEAD` as text.
+
+## D9 — Engine serves the web application; loopback only; host and origin checks instead of a token
+
+One process, one origin, so CORS is not needed in production. Because a local service can be reached by any web page in the user's browser, the engine checks `Host` (DNS rebinding) and `Origin`/`Sec-Fetch-Site` (cross-site requests) on every request.
+
+## D10 — Data directory inside the installation folder by default
+
+At the owner's request nothing is written to the system drive. The database, imported sources, uploads and the credential key default to `.vectora-data` in the VECTORA folder; tests and benchmarks use `.bench-tmp`; the npm cache for this project is `.npm-cache`. `VECTORA_DATA_DIR` overrides the location.
+
+## D11 — Sigma.js 3, not 4
+
+Sigma 4 replaced the reducer-based API (node and edge reducers, settings) that the selection, dimming and label behaviour rely on, and its documentation was not sufficient to build against with confidence. Sigma 3.0.3 is the last release of the documented line and works with Graphology 0.26.
+
+## D12 — Visualisation code lives in `apps/web`, not in separate packages
+
+The brief suggests `visualization-2d`, `visualization-3d`, `mindmap` and `ui` packages. The three views are React components tied to the application's store, context and styling; separate packages would add build configuration without a second consumer. The layout engines, which are pure and reusable, are a package (`graph-layout`).
+
+## D13 — Radix primitives and hand-written components instead of generated shadcn/ui components
+
+shadcn/ui is a set of copied components built on Radix. Its default look (rounded cards, soft shadows) is what the design brief rules out, so the components are written directly on Radix (`Dialog`, `Switch`, `Popover`) and `cmdk`, styled with the project's tokens.
+
+## D14 — Custom 3D layout: anchors plus force refinement
+
+A plain 3D force layout does not produce a readable hierarchy. Subsystems are fixed anchors, files are simulated with `d3-force-3d` under a pull towards their anchor, and symbols are placed analytically around their file. This is deterministic, fast for large symbol counts (symbols need no simulation) and gives the centre → regions → contents structure the brief asks for.
+
+## D15 — Labels in 3D as one projected overlay
+
+drei's `Html` creates a React root per label and logged warnings when labels came and went. Labels are plain elements in one overlay, positioned by projecting node positions during rendered frames.
+
+## D16 — Concepts from a fixed vocabulary with evidence
+
+Concept grouping must be based on source evidence and kept apart from facts. Matching path segments, symbol names and imported packages against a fixed vocabulary is transparent: every membership lists exactly why. It is deliberately simple; it will miss domain concepts outside the vocabulary.
+
+## D17 — Question exploration is deterministic
+
+Questions are answered by intent detection and graph queries, so the basic application needs no external service and every statement has a citation. An optional provider adapter (chat-completions format) can rewrite the finished findings as prose; it is off by default and cannot add findings.
+
+## D18 — Credentials encrypted with a key file
+
+AES-256-GCM with a key stored in the data directory. An operating-system keystore would be stronger but needs a native module or platform commands; that is on the roadmap. The limitation is documented in `docs/SECURITY.md`.
+
+## D19 — Not used from the suggested stack
+
+- **chokidar**: no file watching in 0.1; the brief also rules out continuous background work. Re-indexing is on demand.
+- **Comlink**: the two Web Workers have one message type each.
+- **p-limit**: the worker pool bounds concurrency itself.
+- **Drizzle, simple-git**: see D7 and D8.
+- **ELK**: dagre is enough for the tree layout.
+- **Tauri**: plan only, in `docs/ROADMAP.md`.
+
+## D20 — Browser tests use the installed Edge and start the engine without a shell
+
+No browser download is needed (`channel: 'msedge'`). The engine is started from the Playwright global setup with a direct process spawn, because shell-based start-up failed on a machine whose `ComSpec` variable was misconfigured. `npm run build` is a Node script for the same reason.
+
+## D21 — Row-id tables for graph rows
+
+The first schema used `WITHOUT ROWID` tables. Benchmarks showed slow bulk inserts with large JSON rows; ordinary tables with a unique index and multi-row inserts are faster. A migration rebuilds old tables.

IMPLEMENTATION_CHECKLIST.md

@@ -0,0 +1,133 @@
+# Implementation checklist
+
+Status against the master directive. `[x]` done and verified, `[~]` partly done (see note), `[ ]` not done. Notes point to `KNOWN_ISSUES.md` (K) and `DECISIONS.md` (D).
+
+## Phase 0 — Research and architecture
+
+- [x] Inspect workspace and toolchain (empty repository; Node 24.14.1, Python 3.13, no pnpm)
+- [x] Check current versions and compatibility of every dependency (D2, D3, D11)
+- [x] Validate licences (`docs/LICENSES.md`; one package without declared licence, K29)
+- [x] Architecture, design system, decisions documented
+
+## Phase 1 — Foundation
+
+- [x] Monorepo, strict TypeScript, ESLint, Prettier
+- [x] Vite, Tailwind CSS 4, React 19, React Router, TanStack Query, Zustand
+- [x] Typography: Zen Kaku Gothic New, Shippori Mincho, IBM Plex Mono, bundled
+- [x] Light, dark and system themes with centralised tokens
+- [x] Routing and application shell
+- [~] shadcn/ui: Radix primitives used directly with hand-written components (D13)
+
+## Phase 2 — Repository ingestion
+
+- [x] GitHub import: public, private with token, validation, branch selection, subdirectory
+- [x] ZIP import: format validation, nested root detection, structure preview, exclusions
+- [x] Local folder import with folder browser, `.gitignore` and configurable exclusions
+- [x] Import progress with stages
+- [x] File scanning, language detection, manifest and framework detection, entry points
+- [x] Bundled, labelled sample repository
+
+## Phase 3 — Code intelligence
+
+- [x] Tree-sitter (Java, Go, Rust, C, C++, C#)
+- [x] TypeScript Compiler API with type checker (TypeScript, JavaScript)
+- [x] Python `ast`
+- [x] Language-adapter interface: parsing, symbols, relationships, locations, errors, metadata
+- [x] Symbols: classes, functions, methods, interfaces, types, variables, constants, decorators, components, endpoints, models
+- [x] Relationships: IMPORTS, EXPORTS, CALLS, EXTENDS, IMPLEMENTS, USES, INSTANTIATES, ROUTES_TO, READS_FROM, WRITES_TO, DEFINED_IN, BELONGS_TO, CONTAINS, DEPENDS_ON, RELATED_TO
+- [x] Unresolved references reported, never guessed
+- [x] Execution flows: entry points, route handlers, data-access paths, cross-subsystem calls
+- [x] Graph validation, duplicate detection, normalisation, integrity checks
+- [x] Unsupported files do not fail indexing; language coverage displayed
+- [~] Receiver-type inference outside TypeScript (K1, K2); mounted route prefixes (K4)
+
+## Phase 4 — Knowledge graph
+
+- [x] Typed schema: 18 node types, 15 relationship types, evidence, method, confidence
+- [x] Graph engine: adjacency, traversal, shortest path, cycles, impact
+- [x] Persistence in SQLite
+- [x] Community detection (Louvain), subsystems, concepts
+- [x] Incremental indexing with cache and unchanged fast path
+- [x] Search: names, partial matches, paths, qualified names, descriptions, documentation
+
+## Phase 5 — 2D Graph Explorer
+
+- [x] Sigma.js with Graphology, ForceAtlas2 in a worker, no-overlap pass
+- [x] Pan, zoom, fit, minimap, node drag
+- [x] Selection, multi-selection, neighbour highlighting, dimming, inspector
+- [x] Filtering, focus mode, neighbour expansion, community grouping, zoom-dependent labels
+- [x] Search integration
+- [~] Node shapes by type in 2D (glyph, size and label instead; K17)
+
+## Phase 6 — 3D Atlas
+
+- [x] Real 3D scene (Three.js, React Three Fiber, drei)
+- [x] Spatial clusters: repository centre, subsystem regions, files and symbols within, concepts and packages in layers
+- [x] Orbit, pan, zoom, click, double-click, hover, search-to-focus, breadcrumb, reset, focus subsystem, neighbourhood, global view
+- [x] Smooth camera transitions with reduced-motion alternative; exit from focus mode
+- [x] InstancedMesh, BufferGeometry, worker layout, level of detail, label culling, on-demand rendering
+- [x] WebGL context loss handling and 2D fallback
+- [~] Performance verified by measurement in the browser (not measured; K14)
+
+## Phase 7 — Mind Map
+
+- [x] Hierarchy generated from the same graph
+- [x] Expand, collapse, reorganise, search, focus, navigate to graph nodes, inspect source
+- [x] Filter by relationship (overlay)
+- [x] Export SVG and PNG
+
+## Phase 8 — Exploration and documentation
+
+- [x] Inspector: all twelve items, CodeMirror 6 preview, copy actions, focus, expand, resize, collapse
+- [x] Command palette: Ctrl/Cmd+K, Escape, Enter
+- [x] Question exploration with file and line evidence; says so when it cannot verify
+- [x] Provider-independent adapter for optional narrative, off by default (not exercised live; K32)
+- [x] About page: philosophy, how it works, languages, privacy, credits, version, licence, changelog and documentation references
+- [x] Settings: appearance, graph, performance, repository, privacy — persisted and effective
+- [x] Exports: JSON, GraphML, SVG, PNG, Markdown analysis, architecture document; layouts and sessions saved and restored
+- [x] No source text in exports unless requested
+- [x] User-facing copy free of excluded terminology (checked by a test)
+
+## Phase 9 — Testing
+
+- [x] Unit tests: scanning, ignore handling, extraction, resolution, validation, clustering, search, persistence, export
+- [x] Integration tests: ZIP, GitHub (stand-in API), local import, full indexing, graph generation, incremental indexing, workspace persistence
+- [x] Browser tests: landing, import, navigation, graph selection, inspection, mind map, 3D camera, search, settings, export — at desktop, tablet and mobile widths
+- [x] Accessibility checks: keyboard, names and labels, focus visibility, contrast, reduced motion, landmarks, dialogs
+- [x] Security tests: traversal, malicious names, oversized files, invalid archives, credential leakage
+- [x] Benchmarks: 1,000 / 5,000 / 10,000 files; 50,000 symbols and 100,000 relationships
+- [ ] Rendering stress tests, memory-leak checks, WebGL cleanup verification (K14)
+- [ ] Live GitHub import test (K31)
+
+## Phase 10 — Release
+
+- [x] Production build (`dist/web`, `dist/engine`) verified by the browser tests
+- [x] Environment configuration (`.env.example`)
+- [x] Installation and development instructions
+- [x] Versioning, changelog, licence, dependency licence inventory
+- [x] Security notes, error handling, production logging with redaction
+- [x] Dependency audit clean
+- [~] Desktop packaging: plan written, not built (K33)
+- [x] Screenshots and a recorded tour in the README, captured from the real application
+- [x] Continuous integration workflow (type check, lint, tests, build, audit)
+
+## Definition of done
+
+| # | Requirement | Status |
+|---|---|---|
+| 1 | Open VECTORA | Done |
+| 2 | Import an actual repository | Done (local and ZIP verified by tests; GitHub against a stand-in) |
+| 3 | Index its source files | Done |
+| 4 | Extract symbols and dependencies | Done |
+| 5 | Persistent knowledge graph | Done |
+| 6 | Hierarchical mind map | Done |
+| 7 | Functional 2D graph | Done |
+| 8 | Functional 3D graph | Done |
+| 9 | Select nodes | Done |
+| 10 | Inspect original source | Done |
+| 11 | Trace relationships | Done |
+| 12 | Search the repository | Done |
+| 13 | Navigate between the three views | Done |
+| 14 | Save and restore the workspace | Done |
+| 15 | Export the graph | Done |
+| 16 | Light and dark themes | Done |