feat: discover files with git ls-files (respect .gitignore, skip nested worktrees)

AGENTS.md

@@ -92,7 +92,7 @@ MCP tool call → server.py → indexer.py → FileEntry.plugin → tree-sitter
 | File | Responsibility |
 |---|---|
 | `server.py` | FastMCP 3.1.0 server — defines all 23 tools, wires cache + indexer + graph at startup. Language-unaware. |
-| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Skips `.venv`, `node_modules`, `__pycache__`, `.git`, etc. |
+| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Discovers files with `git ls-files` (tracked files always; untracked non-ignored files minus `SKIP_DIRS`; nested worktrees/repos/submodules excluded), falling back to an `os.walk` that prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, etc.) and nested worktrees. |
 | `cache.py` | `.codetree/index.json` — stores pre-computed skeletons with mtime-based invalidation. Language-unaware. |
 | `registry.py` | Maps file extensions → plugin instances. The **only** place languages are registered. |
 
@@ -142,6 +142,8 @@ Each plugin implements:
 |---|---|
 | `test_server.py` | Original 4 MCP tools via FastMCP, output format, line accuracy, cross-language |
 | `test_indexer.py` | Build, skip-dirs, skeleton/symbol/refs/callgraph through indexer layer |
+| `test_file_discovery.py` | git-based discovery (.gitignore, nested worktrees/repos, submodules), walk fallback, stale cache entries |
+| `test_parse_cache.py` | Memoized query compilation and per-thread parse-tree cache |
 | `test_cache.py` | Cache load/save/invalidation |
 | `tests/languages/test_<lang>.py` | Per-language core tests |
 | `tests/languages/test_<lang>_comprehensive.py` | Exhaustive code pattern coverage per language |
@@ -173,7 +175,8 @@ Fixtures in `conftest.py`: `sample_repo` (Python-only), `rich_py_repo` (decorato
 ## tree-sitter 0.25.x API
 
 The tree-sitter Python bindings have breaking changes from older docs:
-- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)`
+- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)` — in plugins, call the memoized `_query(LANGUAGE, "...")` from `languages/base.py` instead (compiling a Query costs more than parsing a file)
+- Wrap module parsers as `_PARSER = CachedParser(Parser(_LANGUAGE))` so repeated calls on the same source reuse the tree
 - Use `QueryCursor(query).matches(node)` not `query.matches(node)`
 - Match captures are `list[Node]` — unwrap with `nodes[0]` or use the shared `_matches()` helper from `languages/base.py`
 - All `.decode()` calls must use `errors="replace"`
@@ -203,6 +206,6 @@ The tree-sitter Python bindings have breaking changes from older docs:
 - Plugin classes: `{Lang}Plugin` (e.g., `PythonPlugin`, `GoPlugin`)
 - Module-level parser/language globals: `_PARSER`, `_LANGUAGE`
 - Skeleton results are deduplicated by `(name, line)` and sorted by line number
-- Indexer `SKIP_DIRS` includes `.venv`, `node_modules`, `__pycache__`, `.git` — without this, crawling `.venv` causes Codex timeout
+- File discovery: in a git work tree, tracked files (`git ls-files --cached`) are always indexed — `.gitignore` decides, so a tracked `build/` or `env/` is indexed — while untracked non-ignored files (`--others --exclude-standard`) also skip `SKIP_DIRS`, so an un-ignored `.venv`/`node_modules` is never crawled. Nested worktrees such as `.claude/worktrees/`, nested repos and submodules never leak in, and `.codetree/` is always excluded. Outside git (no repo, git missing or refusing the repo, root ignored), the walk prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, …) and directories whose `.git` is a file (worktrees, submodules) — without this, crawling `.venv` causes Codex timeout. Only discovered files are re-injected from the cache.
 - FastMCP tool access in tests: `mcp.local_provider._components[f"tool:{name}@"].fn`
 - **Doc sync rule**: When tools are added, removed, or changed, update all 5 doc files: `README.md`, `TOOLS_GUIDE.md`, `LANDING_PAGE.md`, `CLAUDE.md`, `AGENTS.md`

CLAUDE.md

@@ -99,7 +99,7 @@ MCP tool call → server.py → indexer.py → FileEntry.plugin → tree-sitter
 | File | Responsibility |
 |---|---|
 | `server.py` | FastMCP 3.1.0 server — defines the 23 tools, wires cache + indexer + graph at startup. Language-unaware. |
-| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Skips `.venv`, `node_modules`, `__pycache__`, `.git`, etc. |
+| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Discovers files with `git ls-files` (tracked files always; untracked non-ignored files minus `SKIP_DIRS`; nested worktrees/repos/submodules excluded), falling back to an `os.walk` that prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, etc.) and nested worktrees. |
 | `cache.py` | `.codetree/index.json` — stores pre-computed skeletons with mtime-based invalidation. Language-unaware. |
 | `registry.py` | Maps file extensions → plugin instances. The **only** place languages are registered. |
 
@@ -148,6 +148,8 @@ Each plugin implements:
 |---|---|
 | `test_server.py` | Original 4 MCP tools via FastMCP, output format, line accuracy, cross-language |
 | `test_indexer.py` | Build, skip-dirs, skeleton/symbol/refs/callgraph through indexer layer |
+| `test_file_discovery.py` | git-based discovery (.gitignore, nested worktrees/repos, submodules), walk fallback, stale cache entries |
+| `test_parse_cache.py` | Memoized query compilation and per-thread parse-tree cache |
 | `test_cache.py` | Cache load/save/invalidation |
 | `tests/languages/test_<lang>.py` | Per-language core tests |
 | `tests/languages/test_<lang>_comprehensive.py` | Exhaustive code pattern coverage per language |
@@ -181,7 +183,8 @@ Fixtures in `conftest.py`: `sample_repo` (Python-only), `rich_py_repo` (decorato
 ## tree-sitter 0.25.x API
 
 The tree-sitter Python bindings have breaking changes from older docs:
-- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)`
+- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)` — in plugins, call the memoized `_query(LANGUAGE, "...")` from `languages/base.py` instead (compiling a Query costs more than parsing a file)
+- Wrap module parsers as `_PARSER = CachedParser(Parser(_LANGUAGE))` so repeated calls on the same source reuse the tree
 - Use `QueryCursor(query).matches(node)` not `query.matches(node)`
 - Match captures are `list[Node]` — unwrap with `nodes[0]` or use the shared `_matches()` helper from `languages/base.py`
 - All `.decode()` calls must use `errors="replace"`
@@ -211,7 +214,7 @@ The tree-sitter Python bindings have breaking changes from older docs:
 - Plugin classes: `{Lang}Plugin` (e.g., `PythonPlugin`, `GoPlugin`)
 - Module-level parser/language globals: `_PARSER`, `_LANGUAGE`
 - Skeleton results are deduplicated by `(name, line)` and sorted by line number
-- Indexer `SKIP_DIRS` includes `.venv`, `node_modules`, `__pycache__`, `.git` — without this, crawling `.venv` causes Claude Code timeout
+- File discovery: in a git work tree, tracked files (`git ls-files --cached`) are always indexed — `.gitignore` decides, so a tracked `build/` or `env/` is indexed — while untracked non-ignored files (`--others --exclude-standard`) also skip `SKIP_DIRS`, so an un-ignored `.venv`/`node_modules` is never crawled. Nested worktrees such as `.claude/worktrees/`, nested repos and submodules never leak in, and `.codetree/` is always excluded. Outside git (no repo, git missing or refusing the repo, root ignored), the walk prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, …) and directories whose `.git` is a file (worktrees, submodules) — without this, crawling `.venv` causes Claude Code timeout. Only discovered files are re-injected from the cache.
 - FastMCP tool access in tests: `mcp.local_provider._components[f"tool:{name}@"].fn`
 - **Doc sync rule**: When tools are added, removed, or changed, update all 5 doc files: `README.md`, `TOOLS_GUIDE.md`, `LANDING_PAGE.md`, `CLAUDE.md`, `AGENTS.md`
 

README.md

@@ -238,6 +238,19 @@ Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS)
 | **SCIP / LSIF indexers** | Slow builds, complex setup, huge indexes | ~1s startup, JSON cache, zero config |
 | **AST-only tools** | Raw trees are verbose and hard for agents | Pre-structured output designed for agents |
 
+## What Gets Indexed
+
+- **Git repositories:** every file `git ls-files` reports — tracked files plus
+  untracked files that `.gitignore` does not exclude. Tracked directories are
+  indexed even if named `build/`, `dist/` or `env/`; untracked `.venv/`,
+  `node_modules/`, `__pycache__/` and similar are skipped even when nobody
+  ignored them. Nested worktrees (e.g. `.claude/worktrees/`), nested
+  repositories and submodules are not indexed.
+- **Without git** (not a repository, `git` missing, or git refusing the
+  repository, e.g. `safe.directory`): the tree is walked, skipping `.venv`,
+  `node_modules`, `__pycache__`, `.git`, `dist`, `build`, … and nested worktrees.
+- `.codetree/` (codetree's own cache) is never indexed.
+
 ## Architecture
 
 ```

docs/LANDING_PAGE.md

@@ -365,6 +365,7 @@ claude mcp add codetree -- uvx --from mcp-server-codetree codetree --root .
 - **FastMCP** for the MCP protocol — stdio transport, zero network config.
 - **Plugin architecture** — each language is a self-contained class implementing 5 core methods. Adding a language is copying a template.
 - **Smart caching** — `.codetree/index.json` with mtime-based invalidation. Unchanged files skip parsing entirely.
+- **Respects `.gitignore`** — files are discovered via `git ls-files`, so ignored build output and nested git worktrees (e.g. `.claude/worktrees/`) are never indexed.
 - **Lazy call graph** — only built when tools like `find_dead_code` or `get_blast_radius` are first called. Stored in memory, O(1) lookup.
 - **PageRank** — standard algorithm (25 iterations, damping 0.85) for ranking symbol importance by reference count.
 - **Clone detection** — AST normalization (identifiers → `_ID_`, strings → `_STR_`, numbers → `_NUM_`) + SHA-256 hashing. Catches exact copies and renamed-variable copies.