Kevinrob · Oct 1, 2026 · Oct 1, 2026 · Oct 1, 2026 · Oct 1, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -23,7 +23,7 @@ It exposes **23 tools** over MCP:
 | `detect_clones(file_path?, min_lines?)` | Duplicate/near-duplicate functions | `Clone group 1 (2 functions, 12 lines each):` |
 | `search_symbols(query?, type?, parent?, ..., format?)` | Flexible symbol search; `format="compact"` omits doc lines | `calc.py: class Calculator → line 1` |
 | `find_tests(file_path, symbol_name)` | Find test functions for a symbol | `test_calc.py: test_add() → line 3  (name match)` |
-| `index_status()` | Graph index freshness and stats | `{files: 42, symbols: 315, edges: 580}` |
+| `index_status()` | Indexing progress, graph freshness and stats (never blocks) | `{graph_exists, files, symbols, edges, last_indexed_at, status, files_discovered, files_indexed, index_ready, graph_ready, startup_seconds?, error?}` |
 | `get_repository_map(max_items?)` | Compact repo overview for onboarding | `{languages: {py: 20}, hotspots: [...], start_here: [...]}` |
 | `resolve_symbol(query, kind?, path_hint?)` | Disambiguate short name into qualified matches | `calc.py::Calculator.add → line 11` |
 | `search_graph(query?, kind?, file_pattern?)` | Graph search with degree filters and pagination | `{total: 5, results: [...]}` |
@@ -60,7 +60,7 @@ All `file_path` arguments are **relative to the repo root** (e.g., `"src/main.py
 # Activate venv (required before all commands)
 source .venv/bin/activate
 
-# Run all tests (~1058 tests, ~35s)
+# Run all tests (~1200 tests, ~20s)
 pytest
 
 # Run a single test file
@@ -91,8 +91,9 @@ MCP tool call → server.py → indexer.py → FileEntry.plugin → tree-sitter
 
 | File | Responsibility |
 |---|---|
-| `server.py` | FastMCP 3.1.0 server — defines all 23 tools, wires cache + indexer + graph at startup. Language-unaware. |
-| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Skips `.venv`, `node_modules`, `__pycache__`, `.git`, etc. |
+| `server.py` | FastMCP 3.1.0 server — defines all 23 tools; each tool asks `IndexState` for the data it needs (single-file → on-demand parse, repo-wide → full index, graph → SQLite graph). Language-unaware. |
+| `index_state.py` | `IndexState` — owns the index lifecycle: discovery → indexing (skeleton cache) → graph build, in a background thread when run as `codetree` (`create_server(root, background=True)`), synchronously by default (tests). Exposes `files_ready`/`index_ready`/`graph_ready` events, bounded waits (`WAIT_TIMEOUT`, env `CODETREE_WAIT_TIMEOUT`), progress and errors for `index_status`. Exposed to tests as `mcp._codetree_state`. |
+| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Discovers files with `git ls-files` (tracked files always; untracked non-ignored files minus `SKIP_DIRS`; nested worktrees/repos/submodules excluded), falling back to an `os.walk` that prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, etc.) and nested worktrees. |
 | `cache.py` | `.codetree/index.json` — stores pre-computed skeletons with mtime-based invalidation. Language-unaware. |
 | `registry.py` | Maps file extensions → plugin instances. The **only** place languages are registered. |
 
@@ -142,6 +143,9 @@ Each plugin implements:
 |---|---|
 | `test_server.py` | Original 4 MCP tools via FastMCP, output format, line accuracy, cross-language |
 | `test_indexer.py` | Build, skip-dirs, skeleton/symbol/refs/callgraph through indexer layer |
+| `test_file_discovery.py` | git-based discovery (.gitignore, nested worktrees/repos, submodules), walk fallback, stale cache entries |
+| `test_parse_cache.py` | Memoized query compilation and per-thread parse-tree cache |
+| `test_async_startup.py` | Background indexing: non-blocking startup, on-demand single-file tools, "still building" messages, failure handling |
 | `test_cache.py` | Cache load/save/invalidation |
 | `tests/languages/test_<lang>.py` | Per-language core tests |
 | `tests/languages/test_<lang>_comprehensive.py` | Exhaustive code pattern coverage per language |
@@ -173,7 +177,8 @@ Fixtures in `conftest.py`: `sample_repo` (Python-only), `rich_py_repo` (decorato
 ## tree-sitter 0.25.x API
 
 The tree-sitter Python bindings have breaking changes from older docs:
-- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)`
+- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)` — in plugins, call the memoized `_query(LANGUAGE, "...")` from `languages/base.py` instead (compiling a Query costs more than parsing a file)
+- Wrap module parsers as `_PARSER = CachedParser(Parser(_LANGUAGE))` so repeated calls on the same source reuse the tree
 - Use `QueryCursor(query).matches(node)` not `query.matches(node)`
 - Match captures are `list[Node]` — unwrap with `nodes[0]` or use the shared `_matches()` helper from `languages/base.py`
 - All `.decode()` calls must use `errors="replace"`
@@ -203,6 +208,6 @@ The tree-sitter Python bindings have breaking changes from older docs:
 - Plugin classes: `{Lang}Plugin` (e.g., `PythonPlugin`, `GoPlugin`)
 - Module-level parser/language globals: `_PARSER`, `_LANGUAGE`
 - Skeleton results are deduplicated by `(name, line)` and sorted by line number
-- Indexer `SKIP_DIRS` includes `.venv`, `node_modules`, `__pycache__`, `.git` — without this, crawling `.venv` causes Codex timeout
+- File discovery: in a git work tree, tracked files (`git ls-files --cached`) are always indexed — `.gitignore` decides, so a tracked `build/` or `env/` is indexed — while untracked non-ignored files (`--others --exclude-standard`) also skip `SKIP_DIRS`, so an un-ignored `.venv`/`node_modules` is never crawled. Nested worktrees such as `.claude/worktrees/`, nested repos and submodules never leak in, and `.codetree/` is always excluded. Outside git (no repo, git missing or refusing the repo, root ignored), the walk prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, …) and directories whose `.git` is a file (worktrees, submodules) — without this, crawling `.venv` causes Codex timeout. Only discovered files are re-injected from the cache, which also stores each file's `has_errors` flag.
 - FastMCP tool access in tests: `mcp.local_provider._components[f"tool:{name}@"].fn`
 - **Doc sync rule**: When tools are added, removed, or changed, update all 5 doc files: `README.md`, `TOOLS_GUIDE.md`, `LANDING_PAGE.md`, `CLAUDE.md`, `AGENTS.md`
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -30,7 +30,7 @@ It exposes **23 tools** over MCP:
 
 | Tool | Purpose | Returns |
 |------|---------|---------|
-| `index_status()` | Graph index freshness and stats | `{graph_exists, files, symbols, edges, last_indexed_at}` |
+| `index_status()` | Indexing progress, graph freshness and stats (never blocks) | `{graph_exists, files, symbols, edges, last_indexed_at, status, files_discovered, files_indexed, index_ready, graph_ready, startup_seconds?, error?}` |
 | `get_repository_map(max_items?)` | Compact repo overview for agent onboarding | `{languages, entry_points, hotspots, start_here, test_roots, stats}` |
 | `resolve_symbol(query, kind?, path_hint?)` | Disambiguate short symbol names into qualified matches | `{matches: [{qualified_name, name, kind, file, line}]}` |
 | `search_graph(query?, kind?, file_pattern?, ...)` | Structured graph search with pagination and degree filtering | `{total, results: [{qualified_name, kind, in_degree, out_degree}]}` |
@@ -66,7 +66,7 @@ All `file_path` arguments are **relative to the repo root** (e.g., `"src/main.py
 # Activate venv (required before all commands)
 source .venv/bin/activate
 
-# Run all tests (~1058 tests, ~35s)
+# Run all tests (~1200 tests, ~20s)
 pytest
 
 # Run a single test file
@@ -98,8 +98,9 @@ MCP tool call → server.py → indexer.py → FileEntry.plugin → tree-sitter
 
 | File | Responsibility |
 |---|---|
-| `server.py` | FastMCP 3.1.0 server — defines the 23 tools, wires cache + indexer + graph at startup. Language-unaware. |
-| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Skips `.venv`, `node_modules`, `__pycache__`, `.git`, etc. |
+| `server.py` | FastMCP 3.1.0 server — defines the 23 tools; each tool asks `IndexState` for the data it needs (single-file → on-demand parse, repo-wide → full index, graph → SQLite graph). Language-unaware. |
+| `index_state.py` | `IndexState` — owns the index lifecycle: discovery → indexing (skeleton cache) → graph build, in a background thread when run as `codetree` (`create_server(root, background=True)`), synchronously by default (tests). Exposes `files_ready`/`index_ready`/`graph_ready` events, bounded waits (`WAIT_TIMEOUT`, env `CODETREE_WAIT_TIMEOUT`), progress and errors for `index_status`. Exposed to tests as `mcp._codetree_state`. |
+| `indexer.py` | Discovers files, stores a `FileEntry` per file (with its plugin + `has_errors` flag), routes all queries through the stored plugin. Builds a definition index and lazy call graph for dead code, blast radius, and clone detection. Discovers files with `git ls-files` (tracked files always; untracked non-ignored files minus `SKIP_DIRS`; nested worktrees/repos/submodules excluded), falling back to an `os.walk` that prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, etc.) and nested worktrees. |
 | `cache.py` | `.codetree/index.json` — stores pre-computed skeletons with mtime-based invalidation. Language-unaware. |
 | `registry.py` | Maps file extensions → plugin instances. The **only** place languages are registered. |
 
@@ -148,6 +149,9 @@ Each plugin implements:
 |---|---|
 | `test_server.py` | Original 4 MCP tools via FastMCP, output format, line accuracy, cross-language |
 | `test_indexer.py` | Build, skip-dirs, skeleton/symbol/refs/callgraph through indexer layer |
+| `test_file_discovery.py` | git-based discovery (.gitignore, nested worktrees/repos, submodules), walk fallback, stale cache entries |
+| `test_parse_cache.py` | Memoized query compilation and per-thread parse-tree cache |
+| `test_async_startup.py` | Background indexing: non-blocking startup, on-demand single-file tools, "still building" messages, failure handling |
 | `test_cache.py` | Cache load/save/invalidation |
 | `tests/languages/test_<lang>.py` | Per-language core tests |
 | `tests/languages/test_<lang>_comprehensive.py` | Exhaustive code pattern coverage per language |
@@ -181,7 +185,8 @@ Fixtures in `conftest.py`: `sample_repo` (Python-only), `rich_py_repo` (decorato
 ## tree-sitter 0.25.x API
 
 The tree-sitter Python bindings have breaking changes from older docs:
-- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)`
+- Use `Query(LANGUAGE, "...")` not `LANGUAGE.query(...)` — in plugins, call the memoized `_query(LANGUAGE, "...")` from `languages/base.py` instead (compiling a Query costs more than parsing a file)
+- Wrap module parsers as `_PARSER = CachedParser(Parser(_LANGUAGE))` so repeated calls on the same source reuse the tree
 - Use `QueryCursor(query).matches(node)` not `query.matches(node)`
 - Match captures are `list[Node]` — unwrap with `nodes[0]` or use the shared `_matches()` helper from `languages/base.py`
 - All `.decode()` calls must use `errors="replace"`
@@ -211,7 +216,7 @@ The tree-sitter Python bindings have breaking changes from older docs:
 - Plugin classes: `{Lang}Plugin` (e.g., `PythonPlugin`, `GoPlugin`)
 - Module-level parser/language globals: `_PARSER`, `_LANGUAGE`
 - Skeleton results are deduplicated by `(name, line)` and sorted by line number
-- Indexer `SKIP_DIRS` includes `.venv`, `node_modules`, `__pycache__`, `.git` — without this, crawling `.venv` causes Claude Code timeout
+- File discovery: in a git work tree, tracked files (`git ls-files --cached`) are always indexed — `.gitignore` decides, so a tracked `build/` or `env/` is indexed — while untracked non-ignored files (`--others --exclude-standard`) also skip `SKIP_DIRS`, so an un-ignored `.venv`/`node_modules` is never crawled. Nested worktrees such as `.claude/worktrees/`, nested repos and submodules never leak in, and `.codetree/` is always excluded. Outside git (no repo, git missing or refusing the repo, root ignored), the walk prunes `SKIP_DIRS` (`.venv`, `node_modules`, `__pycache__`, `.git`, …) and directories whose `.git` is a file (worktrees, submodules) — without this, crawling `.venv` causes Claude Code timeout. Only discovered files are re-injected from the cache, which also stores each file's `has_errors` flag.
 - FastMCP tool access in tests: `mcp.local_provider._components[f"tool:{name}@"].fn`
 - **Doc sync rule**: When tools are added, removed, or changed, update all 5 doc files: `README.md`, `TOOLS_GUIDE.md`, `LANDING_PAGE.md`, `CLAUDE.md`, `AGENTS.md`
 
@@ -253,13 +258,13 @@ codetree is a Python MCP server that gives coding agents structured code underst
 - Optional: `uv` for faster installation (recommended in README for Quick Start)
 - Lockfile: `.venv/` contains installed packages; no `requirements.txt` or `pyproject.lock` committed
 ## Frameworks
-- FastMCP 3.1.0 (or later `>=2.0.0`) - MCP (Model Context Protocol) server framework
+- FastMCP 3.x (`>=3.0.0`, which runs sync tools in a thread pool — required for background indexing) - MCP (Model Context Protocol) server framework
 - tree-sitter 0.23.0+ - AST parsing library (language-agnostic)
 - pytest (via GitHub Actions workflow, not explicitly in pyproject.toml dependencies but installed in CI)
 - hatchling (build backend)
 ## Key Dependencies
 - tree-sitter (0.23.0+) - Core AST parsing; blocks everything else
-- fastmcp (2.0.0+) - MCP server registration and tool transport
+- fastmcp (3.0.0+) - MCP server registration and tool transport
 - tree-sitter-python, tree-sitter-javascript, tree-sitter-typescript, tree-sitter-go, tree-sitter-rust, tree-sitter-java, tree-sitter-c, tree-sitter-cpp, tree-sitter-ruby
 ## Configuration
 - No explicit environment variables required for normal operation

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -11,7 +11,7 @@ python -m venv .venv
 source .venv/bin/activate
 pip install -e .
 pip install pytest
-pytest  # 999 tests, ~30s
+pytest  # ~1200 tests, ~20s
 ```
 
 ## What to work on

diff --git a/README.md b/README.md
@@ -115,7 +115,7 @@ The agent sees every class, method, and docstring — with line numbers — with
 
 | Tool | Purpose |
 |------|---------|
-| `index_status()` | Graph index freshness and stats |
+| `index_status()` | Indexing progress, graph freshness and stats (never blocks) |
 | `get_repository_map(max_items?)` | Compact repo overview: languages, entry points, hotspots |
 | `resolve_symbol(query, kind?, path_hint?)` | Disambiguate short name into ranked qualified matches |
 | `search_graph(query?, kind?, file_pattern?)` | Graph search with degree filters and pagination |
@@ -238,6 +238,24 @@ Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS)
 | **SCIP / LSIF indexers** | Slow builds, complex setup, huge indexes | ~1s startup, JSON cache, zero config |
 | **AST-only tools** | Raw trees are verbose and hard for agents | Pre-structured output designed for agents |
 
+## What Gets Indexed
+
+- **Git repositories:** every file `git ls-files` reports — tracked files plus
+  untracked files that `.gitignore` does not exclude. Tracked directories are
+  indexed even if named `build/`, `dist/` or `env/`; untracked `.venv/`,
+  `node_modules/`, `__pycache__/` and similar are skipped even when nobody
+  ignored them. Nested worktrees (e.g. `.claude/worktrees/`), nested
+  repositories and submodules are not indexed.
+- **Without git** (not a repository, `git` missing, or git refusing the
+  repository, e.g. `safe.directory`): the tree is walked, skipping `.venv`,
+  `node_modules`, `__pycache__`, `.git`, `dist`, `build`, … and nested worktrees.
+- `.codetree/` (codetree's own cache) is never indexed.
+
+The MCP handshake is answered immediately and indexing runs in the background.
+Single-file tools answer right away; repo-wide and graph tools wait up to 20 s
+for the index (set `CODETREE_WAIT_TIMEOUT` to change it), then report progress.
+`index_status()` never blocks.
+
 ## Architecture
 
 ```
@@ -257,6 +275,7 @@ codetree server (FastMCP)
 | Module | Responsibility |
 |--------|---------------|
 | `server.py` | FastMCP server — defines all 23 tools |
+| `index_state.py` | Index lifecycle: background indexing, readiness, progress for `index_status` |
 | `indexer.py` | File discovery, plugin dispatch, definition index |
 | `cache.py` | Skeleton cache with mtime invalidation |
 | `registry.py` | Maps file extensions to language plugins |
@@ -285,7 +304,7 @@ source .venv/bin/activate
 pip install -e .
 pip install pytest
 
-# Run all tests (~1058 tests, ~35s)
+# Run all tests (~1200 tests, ~20s)
 pytest
 
 # Run a single test file

diff --git a/docs/LANDING_PAGE.md b/docs/LANDING_PAGE.md
@@ -159,7 +159,7 @@ The agent sees every class, method, and docstring — with line numbers — with
 
 | Tool | What it does | Example |
 |------|-------------|---------|
-| `index_status` | Graph index freshness and stats | See how many files, symbols, and edges are indexed |
+| `index_status` | Indexing progress, graph freshness and stats | See indexing progress and how many files, symbols, and edges are indexed |
 | `get_repository_map` | Compact repo overview for agent onboarding | Languages, entry points, hotspots, suggested starting points |
 | `resolve_symbol` | Disambiguate a short name into ranked qualified matches | "add" → `calc.py::Calculator.add`, `math.py::add` |
 | `search_graph` | Flexible graph search with degree filters and pagination | All functions with >5 inbound calls |
@@ -365,6 +365,8 @@ claude mcp add codetree -- uvx --from mcp-server-codetree codetree --root .
 - **FastMCP** for the MCP protocol — stdio transport, zero network config.
 - **Plugin architecture** — each language is a self-contained class implementing 5 core methods. Adding a language is copying a template.
 - **Smart caching** — `.codetree/index.json` with mtime-based invalidation. Unchanged files skip parsing entirely.
+- **Non-blocking startup** — the MCP handshake is answered immediately; indexing runs in a background thread. Single-file tools answer right away (parsing on demand); repo-wide and graph tools wait briefly, then report progress.
+- **Respects `.gitignore`** — files are discovered via `git ls-files`, so ignored build output and nested git worktrees (e.g. `.claude/worktrees/`) are never indexed.
 - **Lazy call graph** — only built when tools like `find_dead_code` or `get_blast_radius` are first called. Stored in memory, O(1) lookup.
 - **PageRank** — standard algorithm (25 iterations, damping 0.85) for ranking symbol importance by reference count.
 - **Clone detection** — AST normalization (identifiers → `_ID_`, strings → `_STR_`, numbers → `_NUM_`) + SHA-256 hashing. Catches exact copies and renamed-variable copies.