From 3cff55a47b75be83ab00ef82b10ad67f9f499945 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 02:27:39 -0700 Subject: [PATCH 01/25] context: bind a local graph to its revision and fail closed on everything else Adds the lifecycle around the optional Graphify provider adopted in #876: the part that decides whether a graph should have existed at all, as distinct from context_graph, which decides whether a delivered packet's citations may be used. A build never touches the live checkout. It resolves a full commit and tree, takes the tracked census from that commit's tree rather than the working tree or the index, and materializes those blobs into a fresh 0700 directory as 0600 files. Untracked and ignored files have no path into the graph because they are never written. Symlinks and gitlinks are skipped and recorded as skipped: one can name a target the build was never shown, the other a commit in a repository it was never authorized to read. The indexer runs with an allowlisted environment -- PATH, TMPDIR, LANG, LC_ALL, TZ, and nothing else -- every proxy emptied, and HOME redirected into the build's own scratch area. A newly invented secret variable is excluded by default, because the list names what is kept rather than what is dropped. Git itself runs with system, global and local configuration disarmed, so an untrusted checkout cannot install a clean/smudge filter that runs during what looks like a read. Provenance is Code Mower's job because the provider owns none: constraint 1 of the evaluation. Every manifest binds full commit and tree, the exact pin with its wheel digest and options, build time, the tracked census count/bytes/digest, the graph's own digest and bytes, and the provider's completeness admission. The pin is exact by construction -- a range, a marker, or a missing artifact digest is rejected -- and no default path acquires anything. Publication is two atomic renames in an order a reader survives: the generation becomes visible whole, then current starts naming it. Refresh publishes a new immutable generation rather than mutating one in place. graph_status resolves to exactly one state and only current is usable; nothing falls back to an older generation. Stale, corrupt, oversized, partial, and permission-invalid all fail closed, and the privacy check runs on every read, so state loosened after the fact is refused rather than trusted because it was private when it was written. partial exists specifically because of the requeue defect the clean-room run recorded: a fast incremental repeat is not proof the graph is complete. Stdlib only. The provider seam is an injected callable, so the whole lifecycle and its suite run offline with no graph package installed; subprocess_indexer builds the argv and scrubbed environment for an install the operator already made. Closes #913 Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 168 +++ docs/context-provider-contract.md | 7 + docs/current-state-and-roadmap.md | 5 +- docs/graphify-evaluation.md | 10 + src/code_mower/cli.py | 3 + src/code_mower/context_graph_command.py | 145 +++ src/code_mower/context_graph_lifecycle.py | 1125 +++++++++++++++++++++ src/code_mower/package_manifest.py | 10 + tests/test_context_graph_lifecycle.py | 604 +++++++++++ 9 files changed, 2076 insertions(+), 1 deletion(-) create mode 100644 docs/context-graph-lifecycle.md create mode 100644 src/code_mower/context_graph_command.py create mode 100644 src/code_mower/context_graph_lifecycle.py create mode 100644 tests/test_context_graph_lifecycle.py diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md new file mode 100644 index 00000000..ada79d7d --- /dev/null +++ b/docs/context-graph-lifecycle.md @@ -0,0 +1,168 @@ +# Local repository graph: revision-bound lifecycle + +Implements [issue #913](https://github.com/codemower-ai/code-mower/issues/913) +under [epic #902](https://github.com/codemower-ai/code-mower/issues/902), on top +of the adopt decision recorded in [Graphify evaluation](graphify-evaluation.md). + +This is the safe lifecycle around an optional local code graph: how one gets +built, what it is allowed to see, where it is kept, and when a consumer must +refuse it. It adds no dependency, no background service, no hook, no watcher, +and no default indexing step. Nothing on a default install path builds, reads, +or requires a graph. + +Two modules divide the work: + +- `context_graph.py` (from #876) decides whether a delivered packet's + **citations** are in scope and fresh enough to use. +- `context_graph_lifecycle.py` (this document) decides whether the graph + **should have existed** — which revision it binds, which bytes produced it, + where its state lives, and when it fails closed. + +## The problem + +A graph indexer pointed at a working checkout is unsafe in two directions. It +reads files nobody agreed to index — untracked scratch files, ignored +`.env` files, another worktree reached through a symlink — and it produces an +artifact with no way to tell which revision it describes, so a graph built three +commits ago answers today's question with yesterday's code and looks identical +to a fresh one. + +Constraint 1 in the evaluation is the sharp edge: the provider owns no +provenance at all. If Code Mower does not bind the revision, nothing does. + +## What a build does + +`build_graph()` is the only way a generation is created. In order: + +1. **Resolve the revision.** The full commit and tree object names, never an + abbreviation and never a branch name. The tree is resolved separately + because it is what a consumer actually compares. +2. **Take the tracked census.** `git ls-tree -r` against the *commit*, not the + working tree and not the index. Symlinks (`120000`) and submodules + (`160000`) are skipped and recorded as skipped, because a symlink can name a + target the build was never shown and a gitlink names a commit in a + repository it was never authorized to read. The census digest covers mode, + blob name, size and path for every entry in sorted order. +3. **Materialize into private state.** Each blob is written into a fresh 0700 + directory as a 0600 file. Untracked and ignored files have no path into the + graph because they are never written, rather than because something filtered + them out afterwards. +4. **Run the indexer with a scrubbed environment.** The provider process + inherits an allowlist — `PATH`, `TMPDIR`, `LANG`, `LC_ALL`, `TZ` — and + nothing else. Every proxy variable is set empty, `no_proxy` is `*`, and + `HOME` and the XDG directories point into the build's own scratch area. A + newly invented secret variable is excluded by default because the list names + what is kept, not what is dropped. +5. **Publish atomically.** The generation is assembled under a staging name, + fsynced, renamed into `generations/`, and only then does the `current` + pointer start naming it. A reader sees the whole previous generation or the + whole new one. + +Git itself runs with `GIT_CONFIG_NOSYSTEM`, `GIT_CONFIG_GLOBAL=/dev/null`, and +`GIT_CONFIG_SYSTEM=/dev/null`: an untrusted checkout's local, global, or system +configuration can otherwise install clean/smudge filters and hook paths that run +code during what looks like a read. + +## What a manifest binds + +Every published generation carries, in `manifest.json`: + +| Field | Why it is there | +| --- | --- | +| `commit`, `tree` | Full object names. Staleness is decided against these, not against a branch. | +| `provider` | Distribution, exact version, wheel SHA-256, and the extraction options used. | +| `built_at` | ISO 8601 UTC. The provider records no build time of its own. | +| `tracked_files`, `tracked_bytes`, `census_digest` | Exactly which bytes the indexer was shown, re-derivable from the repository. | +| `skipped_paths` | How many tracked entries were deliberately not materialized. | +| `graph_digest`, `graph_bytes` | Detects a truncated or tampered artifact on every read. | +| `completeness` | `complete` or `partial`, from the provider's own admission. | +| `indexed_files` | What the provider claims it processed, bounded by the census. | + +`shareable_summary()` is the metadata-only view: revisions, digests, counts and +states. It carries no indexed content, no provider output, and no local path. + +## Refresh is explicit + +`build` is the first-time verb and refuses when a usable generation already +binds the revision. `refresh` is the rebuild verb, and it publishes a *new* +immutable generation rather than mutating one in place. Nothing refreshes on a +timer, a hook, or a file-system event, and nothing rebuilds implicitly because a +consumer found the graph stale — a stale graph is reported as stale. + +## Failing closed + +`graph_status()` resolves to exactly one state, and only `current` is usable. +Nothing falls back to an older generation: a consumer that cannot have the +revision it asked for is told so rather than handed a stale answer that looks +fresh. + +| State | Cause | +| --- | --- | +| `absent` | Nothing built for this checkout. | +| `stale` | The manifest's commit/tree does not match the revision being asked about. | +| `corrupt` | The artifact's size or SHA-256 does not match the manifest. | +| `oversized` | The artifact exceeds its budget. | +| `partial` | The provider declared an incomplete build. Usable only with an explicit opt-in. | +| `invalid` | The manifest is unreadable, mislabelled, or the state is not private and operator-owned. | + +The privacy check runs on every read, not only at creation: state loosened after +the fact — by a umask change, a restore, or a careless recursive `chmod` — +fails closed rather than being trusted because it was private when it was +written. + +`partial` exists because of the requeue defect recorded in the evaluation: a +fast incremental repeat is not proof that the graph is complete. + +## Commands + +``` +code-mower context-graph build --pin-file PIN --indexer PATH [--revision REV] +code-mower context-graph refresh --pin-file PIN --indexer PATH [--revision REV] +code-mower context-graph status [--allow-partial] [--json] +code-mower context-graph remove [--show-local-paths] +code-mower context-graph doctor [--pin-file PIN] +``` + +`status` exits non-zero when the graph is not usable, so a script can branch on +it. `doctor` reports `skip` rather than `fail` when nothing is pinned or built: +the lifecycle is optional, and an operator who never opted in has nothing wrong +with their installation. + +The pin file names one exact release and is rejected if it names a range, a +marker, or a distribution without an artifact digest: + +```json +{ + "distribution": "graphifyy", + "version": "0.9.58", + "wheel_sha256": "e239803288e91c723d6e30540860bd6d5a1dc3f0914b9fc1104b0233e98aaeb8", + "options": ["--code-only", "--no-cluster"] +} +``` + +`--indexer` is the path to a provider CLI the operator has **already** +installed. This repository does not download, install, or resolve one, which is +why the executable is named rather than discovered. + +## State layout + +``` +~/.local/share/code-mower/context/graph// + current the published generation's name + build.lock serializes builds for one checkout + generations//manifest.json + generations//graph.bin +``` + +Directories are 0700 and files 0600. `` is derived from the resolved +checkout path, so two worktrees of the same repository get separate state and +can never read each other's generations. State is refused inside any Git +repository, which is the enforcement half of adoption condition 2. + +## What this does not do + +No hooks, no watcher, no hosted service, no MCP HTTP service, no semantic or +model-based extraction, no provider API key, no clustering, and no default +dependency. Each remains a separate explicit decision. The provider seam is an +injected callable, so the entire lifecycle — including the whole test suite — +runs offline with no graph package installed. diff --git a/docs/context-provider-contract.md b/docs/context-provider-contract.md index 7c1590e2..09cda3ca 100644 --- a/docs/context-provider-contract.md +++ b/docs/context-provider-contract.md @@ -205,3 +205,10 @@ package record, and the conditions an implementing change must meet. Nothing is installed or required yet. Synthetic graph fixtures prove only the extension point; they do not establish Graphify compatibility or make it a v1.3.1 dependency. + +The lifecycle around such a provider — exact pin, immutable tracked-file +materialization, scrubbed environment, private 0700 state, atomic generations, +and `code-mower context-graph build/refresh/status/remove/doctor` — is +described in [Local repository graph](context-graph-lifecycle.md). It builds the +evidence side; `context_graph` still decides whether a delivered packet's +citations may be used. diff --git a/docs/current-state-and-roadmap.md b/docs/current-state-and-roadmap.md index 650fd611..0d632421 100644 --- a/docs/current-state-and-roadmap.md +++ b/docs/current-state-and-roadmap.md @@ -158,7 +158,10 @@ participant. Start local and code-only: [issue #876](https://github.com/codemower-ai/code-mower/issues/876) with an adopt decision; - add a provider registry and multiple context attachments per session; -- build and refresh graphs with commit/freshness validation; +- build and refresh graphs with commit/freshness validation — delivered by + `code-mower context-graph`, described in the + [lifecycle record](context-graph-lifecycle.md), which closes + [issue #913](https://github.com/codemower-ai/code-mower/issues/913); - consume a pinned structured JSON contract; - generate bounded impact, dependency, symbol, and related-test packets; and - deliver the same packet shape to Claude, Codex, and Devin. diff --git a/docs/graphify-evaluation.md b/docs/graphify-evaluation.md index 66c09291..22af78fc 100644 --- a/docs/graphify-evaluation.md +++ b/docs/graphify-evaluation.md @@ -167,6 +167,16 @@ are engineering conditions on the adapter, not requests for a decision. accounted for: an incremental run's completion is not treated as proof the graph is complete. +Conditions 1, 2, 3, 5 and 7 are implemented by the build/refresh/status/remove +lifecycle in +[Local repository graph: revision-bound lifecycle](context-graph-lifecycle.md) +(issue #913): an exact pin with a verified artifact digest, private 0700 state +outside every checkout, a manifest that binds full commit and tree with build +time, opt-in acquisition with no default dependency, and a `partial` +completeness state that refuses to read a fast incremental repeat as a complete +graph. Conditions 4 and 6 belong to the retrieval adapter, which does not exist +yet. + ## Boundary Graphify stays out of v1.3.1 and does not block Coworker's 1.3.0 or 1.3.1 diff --git a/src/code_mower/cli.py b/src/code_mower/cli.py index 7eca2d27..16584d7b 100644 --- a/src/code_mower/cli.py +++ b/src/code_mower/cli.py @@ -56,6 +56,7 @@ def _source_checkout_install_spec() -> str: from . import controller as code_mower_controller from . import code_mower_calibration from . import code_mower_context_packs +from . import context_graph_command from . import code_mower_merge from . import code_mower_telemetry from . import config as code_mower_config @@ -447,6 +448,7 @@ def _local_llm_main(argv: list[str]) -> int: "cloud": "Export or upload sanitized benchmark metadata.", "config": "Validate or inspect a Code Mower config.", "context": "Record local external planning context manifests.", + "context-graph": "Build, refresh, inspect, or remove a local repository graph.", "context-packs": "Build selective surrounding-file context packs.", "controller": "Compute supervised-pilot dispatch and merge-policy decisions.", "coderabbit-cli": "Run a CodeRabbit CLI informational lane.", @@ -595,6 +597,7 @@ def _top_level_help(show_all: bool) -> str: "cloud": code_mower_cloud.main, "config": _config_main, "context": code_mower_work_orders.context_main, + "context-graph": context_graph_command.main, "context-packs": code_mower_context_packs.main, "controller": code_mower_controller.main, "coderabbit-cli": coderabbit_cli_audit_pr.main, diff --git a/src/code_mower/context_graph_command.py b/src/code_mower/context_graph_command.py new file mode 100644 index 00000000..54f51a52 --- /dev/null +++ b/src/code_mower/context_graph_command.py @@ -0,0 +1,145 @@ +"""``code-mower context-graph``: build, refresh, inspect and remove a local graph. + +The lifecycle in ``context_graph_lifecycle`` is deliberately not wired into any +default path. This command is how an operator opts in, one checkout at a time, +and it asks for everything explicitly rather than discovering it: the provider +pin comes from a file the operator names, and the indexer executable comes from +an install the operator already made. Nothing here downloads, installs, or +resolves a provider. + +Output is metadata only -- revisions, digests, counts, and states. No indexed +content, provider output, or local path of the private state directory is +printed unless the operator asks for it with ``--show-local-paths``. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +from . import context_graph_lifecycle as lifecycle +from .context_contract import ContextError +from .context_store import strict_json + +MAX_PIN_BYTES = 8192 + + +def _load_pin(path: Path | None) -> lifecycle.GraphifyPin | None: + if path is None: + return None + try: + raw = path.read_bytes() + except OSError: + raise ContextError("local graph provider pin file is unreadable") from None + if len(raw) > MAX_PIN_BYTES: + raise ContextError("local graph provider pin file exceeds its bound") + return lifecycle.load_pin(strict_json(raw)) + + +def _require_pin(path: Path | None) -> lifecycle.GraphifyPin: + pin = _load_pin(path) + if pin is None: + raise ContextError("building a local graph requires an exact provider pin") + return pin + + +def _emit(payload: dict, *, as_json: bool, text: str) -> None: + if as_json: + print(json.dumps(payload, indent=2, sort_keys=True)) + else: + print(text, end="") + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser( + prog="code-mower context-graph", + description="Manage an optional revision-bound local repository graph.", + ) + sub = parser.add_subparsers(dest="command", required=True) + build = sub.add_parser("build", help="Build and publish a generation for the current revision") + refresh = sub.add_parser("refresh", help="Explicitly rebuild and atomically publish a new generation") + status = sub.add_parser("status", help="Report whether the published generation may be used") + remove = sub.add_parser("remove", help="Delete this checkout's private local graph state") + doctor = sub.add_parser("doctor", help="Check the local graph posture without building anything") + + for command in (build, refresh, status, remove, doctor): + command.add_argument("--repo-path", type=Path, default=Path.cwd(), help="Checkout to bind") + command.add_argument("--state-dir", type=Path, help="Private state root; defaults to the context store") + command.add_argument("--json", action="store_true", help="Emit a machine-readable summary") + for command in (build, refresh, status, doctor): + command.add_argument("--revision", default="HEAD", help="Revision to bind, for example a commit or tag") + for command in (build, refresh, doctor): + command.add_argument("--pin-file", type=Path, help="JSON file naming one exact provider release") + for command in (build, refresh): + command.add_argument("--indexer", required=True, help="Path to the already-installed pinned provider CLI") + command.add_argument("--keep-previous", action="store_true", + help="Retain superseded generations instead of pruning them") + status.add_argument("--allow-partial", action="store_true", + help="Treat a provider-declared partial build as usable") + remove.add_argument("--show-local-paths", action="store_true", help="Include the private state path in output") + + args = parser.parse_args(argv) + try: + if args.command in ("build", "refresh"): + pin = _require_pin(args.pin_file) + if args.command == "build" and lifecycle.graph_status( + args.repo_path, root=args.state_dir, revision=args.revision + ).usable: + # ``build`` is the first-time verb. A usable generation already + # binds this revision, so rebuilding it is ``refresh`` -- an + # explicit choice, never something ``build`` does by surprise. + raise ContextError("a current generation already binds this revision; use refresh to rebuild") + manifest = lifecycle.build_graph( + args.repo_path, + pin=pin, + indexer=lifecycle.subprocess_indexer(args.indexer), + root=args.state_dir, + revision=args.revision, + keep_previous=args.keep_previous, + ) + summary = {"status": "published", **manifest.shareable_summary()} + _emit(summary, as_json=args.json, + text=lifecycle.render_status_text( + lifecycle.GenerationStatus(state="current", generation=manifest.generation, manifest=manifest))) + return 0 + if args.command == "status": + report = lifecycle.graph_status( + args.repo_path, + root=args.state_dir, + revision=args.revision, + require_complete=not args.allow_partial, + ) + _emit(report.shareable_summary(), as_json=args.json, text=lifecycle.render_status_text(report)) + # A non-current graph is a normal, reportable condition, not a + # command failure; exit 1 so a script can branch on usability. + return 0 if report.usable else 1 + if args.command == "remove": + state = lifecycle.GraphStateRoot(args.repo_path, root=args.state_dir) + path = str(state.path) if args.show_local_paths else None + removed = lifecycle.remove_graph(args.repo_path, root=args.state_dir) + payload = {"schema": "code_mower.contextGraphRemove.v1", "removed": removed} + if path is not None: + payload["path"] = path + _emit(payload, as_json=args.json, + text=("Removed local graph state.\n" if removed else "No local graph state to remove.\n")) + return 0 + report = lifecycle.doctor_report( + args.repo_path, pin=_load_pin(args.pin_file), root=args.state_dir, revision=args.revision + ) + lines = [f"Local graph doctor: {report['status']}"] + lines.extend(f" [{check['status']}] {check['check']}: {check['message']}" for check in report["checks"]) + _emit(report, as_json=args.json, text="\n".join(lines) + "\n") + return 0 if report["status"] != "fail" else 1 + except ContextError as error: + print(f"local graph unavailable: {error}", file=sys.stderr) + return 1 + except Exception: + # Never let a provider or filesystem failure surface indexed content. + print("local graph unavailable; verify the pin, the checkout and the private state directory", file=sys.stderr) + return 1 + + +if __name__ == "__main__": # pragma: no cover - direct invocation + raise SystemExit(main()) diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py new file mode 100644 index 00000000..42002bb7 --- /dev/null +++ b/src/code_mower/context_graph_lifecycle.py @@ -0,0 +1,1125 @@ +"""Revision-bound lifecycle for an optional local-repository graph (issue #913). + +``context_graph`` decides whether a graph's *citations* are in scope. This +module decides whether the graph should have existed at all: which revision it +binds, which bytes it was allowed to see, where its state lives, and when a +consumer must refuse it. + +The rules a local indexer cannot be trusted to follow on its own: + +* **Never index a live checkout.** A working tree mutates mid-build and carries + untracked, ignored, and private files. Every build materializes the tracked + blobs of one commit into a private staging directory and points the indexer + at that copy instead. Untracked and ignored files have no path into the + graph because they are never written. +* **Bind the revision, not the branch.** An artifact records the full commit + and tree SHA it was built from. A consumer compares those against the + repository it is actually asking about; a mismatch is stale, and stale fails + closed rather than answering from the wrong revision. +* **Publish atomically, immutably.** A generation is assembled under a staging + name, fsynced, then renamed into place; the ``current`` pointer is replaced + atomically afterwards. A reader either sees the whole previous generation or + the whole new one, never a half-written directory. +* **Scrub the environment.** The indexer runs with an allowlisted environment + and proxy-denying network posture, so an ambient token cannot leak into a + provider process and the build cannot quietly reach the network. + +Nothing here installs, imports, or requires a graph package. The indexer is an +injected callable, so the whole lifecycle is provable offline; the bundled +``subprocess_indexer`` builds the argv and the scrubbed environment for a +pinned provider without this module depending on it. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import subprocess +import uuid +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Callable, Iterable, Iterator, Mapping, Sequence + +from .context_contract import ContextError, _identifier, _text, _timestamp +from .context_store import _private, default_context_root +from .file_locks import FileLockError, exclusive_handle_lock + +MANIFEST_SCHEMA = "code_mower.contextGraphBuild.v1" +MANIFEST_NAME = "manifest.json" +ARTIFACT_NAME = "graph.bin" +CURRENT_NAME = "current" + +#: Bounds. A local graph is a convenience, not a reason to fill a disk or to +#: stall a session on a pathological repository. Every one of these fails the +#: build closed rather than truncating silently. +MAX_MANIFEST_BYTES = 262_144 +MAX_ARTIFACT_BYTES = 256 * 1024 * 1024 +MAX_TRACKED_FILES = 50_000 +MAX_TRACKED_BYTES = 512 * 1024 * 1024 +MAX_BLOB_BYTES = 32 * 1024 * 1024 + +#: Only ordinary blobs are materialized. A symlink (``120000``) can name a +#: target outside the checkout and a gitlink (``160000``) names a commit in +#: another repository this build was never authorized to read. +_REGULAR_MODES = frozenset({"100644", "100755"}) +_SKIPPED_MODES = {"120000": "symlink", "160000": "submodule"} + +_OBJECT_NAME = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?\Z") +_GENERATION = re.compile(r"[0-9a-f]{32}\Z") +_VERSION = re.compile(r"[0-9][0-9A-Za-z.+!-]{0,63}\Z") +_DIGEST = re.compile(r"[0-9a-f]{64}\Z") + +COMPLETE = "complete" +PARTIAL = "partial" + +#: The only variables a provider process inherits. Everything else -- every +#: token, cloud credential, proxy, and provider API key in the operator's +#: session -- is dropped rather than filtered, so a newly invented secret +#: variable is excluded by default instead of needing a new denylist entry. +_ENVIRONMENT_ALLOWLIST = ("PATH", "TMPDIR", "LANG", "LC_ALL", "TZ") + +#: Denying every proxy and refusing terminal prompts turns an attempted +#: network call into a fast local failure instead of an outbound request. +_NETWORK_DENY = { + "no_proxy": "*", + "NO_PROXY": "*", + "http_proxy": "", + "https_proxy": "", + "HTTP_PROXY": "", + "HTTPS_PROXY": "", + "ALL_PROXY": "", + "all_proxy": "", + "GIT_TERMINAL_PROMPT": "0", + "PYTHONNOUSERSITE": "1", +} + + +def _object_name(value: Any) -> str: + """A full SHA-1 or SHA-256 object name. Abbreviations do not bind.""" + if not isinstance(value, str) or not _OBJECT_NAME.fullmatch(value): + raise ContextError("local graph revision must be a full object name") + return value + + +def _digest(value: Any) -> str: + if not isinstance(value, str) or not _DIGEST.fullmatch(value): + raise ContextError("local graph digest must be a SHA-256 hex digest") + return value + + +def _size(value: Any, maximum: int) -> int: + if type(value) is not int or not 0 <= value <= maximum: + raise ContextError("local graph size must be a bounded non-negative integer") + return value + + +@dataclass(frozen=True) +class GraphifyPin: + """An exact provider pin. A range would let a build drift silently. + + ``wheel_sha256`` is the artifact digest recorded by the adopt decision in + ``docs/graphify-evaluation.md``. It is carried into every build manifest so + a graph built by a substituted distribution is identifiable after the fact, + which is the whole point of pinning a lookalike-prone package name. + """ + + distribution: str + version: str + wheel_sha256: str + options: tuple[str, ...] = () + + @property + def requirement(self) -> str: + return f"{self.distribution}=={self.version}" + + def as_metadata(self) -> dict[str, Any]: + return { + "distribution": self.distribution, + "version": self.version, + "wheel_sha256": self.wheel_sha256, + "options": list(self.options), + } + + +def load_pin(source: Mapping[str, Any]) -> GraphifyPin: + """Parse a provider pin, rejecting anything that is not one exact release.""" + if not isinstance(source, Mapping): + raise ContextError("local graph provider pin must be an object") + unknown = set(source) - {"distribution", "version", "wheel_sha256", "options"} + if unknown: + raise ContextError("local graph provider pin carries unsupported fields") + version = source.get("version") + if not isinstance(version, str) or not _VERSION.fullmatch(version): + raise ContextError("local graph provider pin must name one exact released version") + options = source.get("options", []) + if not isinstance(options, list) or len(options) > 16: + raise ContextError("local graph provider options must be a bounded list") + return GraphifyPin( + distribution=_identifier(source.get("distribution")), + version=version, + wheel_sha256=_digest(source.get("wheel_sha256")), + options=tuple(_text(option, maximum=128) for option in options), + ) + + +@dataclass(frozen=True) +class TrackedEntry: + """One tracked regular file at the bound revision.""" + + mode: str + blob: str + path: str + size: int + + +@dataclass(frozen=True) +class TrackedCensus: + """What the indexer was allowed to see, and proof of exactly which bytes. + + ``digest`` covers mode, blob name, size, and path for every entry in sorted + order. Two builds of the same commit produce the same census digest, and a + census that silently gained or lost a file produces a different one, so a + manifest's census claim is checkable without re-reading the repository. + """ + + entries: tuple[TrackedEntry, ...] + skipped: tuple[tuple[str, str], ...] + digest: str + + @property + def file_count(self) -> int: + return len(self.entries) + + @property + def total_bytes(self) -> int: + return sum(entry.size for entry in self.entries) + + +def _census_digest(entries: Iterable[TrackedEntry]) -> str: + census = hashlib.sha256() + for entry in entries: + census.update(f"{entry.mode} {entry.blob} {entry.size} {entry.path}\n".encode()) + return census.hexdigest() + + +def _git(repository: Path, *arguments: str, capture: bool = True) -> str: + """Run git with repository configuration disarmed. + + A build reads an untrusted checkout. Local, global, and system + configuration can install clean/smudge filters, alternate object stores, + and hook paths, any of which would run code or reach outside the + repository during what looks like a read. This drops all three. + """ + environment = dict(_NETWORK_DENY) + for name in _ENVIRONMENT_ALLOWLIST: + if name in os.environ: + environment[name] = os.environ[name] + environment.update( + GIT_CONFIG_NOSYSTEM="1", + GIT_CONFIG_GLOBAL=os.devnull, + GIT_CONFIG_SYSTEM=os.devnull, + GIT_ATTR_NOSYSTEM="1", + GIT_OPTIONAL_LOCKS="0", + ) + try: + completed = subprocess.run( + ["git", "-C", str(repository), "--no-optional-locks", *arguments], + check=True, + capture_output=capture, + text=True, + env=environment, + ) + except (OSError, UnicodeError, subprocess.SubprocessError): + raise ContextError("local graph build could not read the target repository") from None + return completed.stdout + + +def resolve_revision(repository: Path, revision: str = "HEAD") -> tuple[str, str]: + """Return the full ``(commit, tree)`` names a build would bind. + + The tree is resolved separately rather than derived, because it is what a + consumer actually compares: two commits with different messages or parents + over identical content share a tree, and a graph of that content is still + accurate for both. + """ + name = _text(revision) + if name.startswith("-"): + # Otherwise the revision reaches ``git rev-parse`` as an option. + raise ContextError("local graph revision must not begin with an option marker") + commit = _object_name(_git(repository, "rev-parse", "--verify", f"{name}^{{commit}}").strip()) + tree = _object_name(_git(repository, "rev-parse", "--verify", f"{commit}^{{tree}}").strip()) + return commit, tree + + +def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: + """List the tracked regular files of one commit, with their blob sizes. + + Reads the commit's tree, never the working tree or the index, so an + uncommitted edit, an untracked scratch file, and an ignored secret are all + invisible here by construction rather than by filtering. + """ + listing = _git( + repository, + "ls-tree", + "-r", + "-z", + "--long", + "--full-tree", + _object_name(commit), + ) + entries: list[TrackedEntry] = [] + skipped: list[tuple[str, str]] = [] + for record in listing.split("\0"): + if not record: + continue + metadata, _, path = record.partition("\t") + fields = metadata.split() + if len(fields) != 4 or not path: + raise ContextError("local graph build could not read the repository census") + mode, kind, blob, raw_size = fields + if mode in _SKIPPED_MODES: + skipped.append((path, _SKIPPED_MODES[mode])) + continue + if mode not in _REGULAR_MODES or kind != "blob": + skipped.append((path, "unsupported")) + continue + if len(entries) >= MAX_TRACKED_FILES: + raise ContextError("tracked file census exceeds the local graph budget") + size = int(raw_size) if raw_size.isdigit() else -1 + if not 0 <= size <= MAX_BLOB_BYTES: + raise ContextError("tracked file exceeds the local graph per-file budget") + entries.append(TrackedEntry(mode=mode, blob=_object_name(blob), path=path, size=size)) + entries.sort(key=lambda entry: entry.path) + if sum(entry.size for entry in entries) > MAX_TRACKED_BYTES: + raise ContextError("tracked content exceeds the local graph budget") + return TrackedCensus( + entries=tuple(entries), + skipped=tuple(sorted(skipped)), + digest=_census_digest(entries), + ) + + +def _safe_relative(path: str) -> Path: + """Reject any census path that would escape the materialization root. + + Git does not normally produce these, but a build must not depend on that: + the destination is created by this process and everything written into it + is checked here first. + """ + if ( + not path + or path.startswith("/") + or "\\" in path + or any(segment in {"", ".", ".."} for segment in path.split("/")) + # Every segment, not just the first: a vendored submodule's ``vendor/.git`` + # is as private as the top-level one. Case-folded because APFS and NTFS + # name the same directory ``.GIT``. + or any(segment.casefold() == ".git" for segment in path.split("/")) + ): + raise ContextError("tracked path must stay inside the materialized checkout") + return Path(*path.split("/")) + + +def materialize_tracked_files(repository: Path, census: TrackedCensus, destination: Path) -> int: + """Write the census's blobs into ``destination``. Returns bytes written. + + ``destination`` must not already exist: an immutable materialization is one + this build created and fully owns, so there is no prior content to + reconcile and no possibility of reusing a directory somebody else can + write. Blob content comes from ``git cat-file --batch`` in one child + process rather than one per file. + """ + if destination.exists(): + raise ContextError("local graph materialization requires a fresh private directory") + destination.mkdir(mode=0o700, parents=True) + if not census.entries: + return 0 + environment = dict(_NETWORK_DENY) + for name in _ENVIRONMENT_ALLOWLIST: + if name in os.environ: + environment[name] = os.environ[name] + environment.update(GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, GIT_CONFIG_SYSTEM=os.devnull) + written = 0 + process = subprocess.Popen( + ["git", "-C", str(repository), "cat-file", "--batch"], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + env=environment, + ) + try: + assert process.stdin is not None and process.stdout is not None + for entry in census.entries: + target = destination / _safe_relative(entry.path) + target.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + process.stdin.write(entry.blob.encode() + b"\n") + process.stdin.flush() + header = process.stdout.readline().decode("utf-8", "replace").split() + if len(header) != 3 or header[1] != "blob" or not header[2].isdigit(): + raise ContextError("local graph materialization could not read tracked content") + size = int(header[2]) + if size != entry.size: + raise ContextError("tracked content changed during materialization") + payload = process.stdout.read(size) + if len(payload) != size or process.stdout.read(1) != b"\n": + raise ContextError("local graph materialization was truncated") + # 0o600 regardless of the tracked mode: the indexer reads this copy + # and never executes it, and an executable bit here would only + # widen what a provider process can do with the staging directory. + handle = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + with os.fdopen(handle, "wb") as stream: + stream.write(payload) + written += size + except BaseException: + process.kill() + raise + finally: + for stream in (process.stdin, process.stdout): + if stream is not None and not stream.closed: + stream.close() + try: + process.wait(timeout=60) + except subprocess.TimeoutExpired: # pragma: no cover - unresponsive child + process.kill() + return written + + +def scrubbed_environment(*, home: Path, temporary: Path) -> dict[str, str]: + """Build the provider process environment from an allowlist. + + An indexer inherits nothing from the operator's session but the variables + it needs to find executables and write scratch files. ``HOME`` is + redirected into the build's own private directory so a provider's + configuration, cache, or credential lookup lands there instead of reading + or writing the operator's real home. + """ + environment = { + name: os.environ[name] for name in _ENVIRONMENT_ALLOWLIST if name in os.environ + } + environment.update(_NETWORK_DENY) + environment.update( + HOME=str(home), + TMPDIR=str(temporary), + XDG_CONFIG_HOME=str(home / "config"), + XDG_CACHE_HOME=str(home / "cache"), + XDG_DATA_HOME=str(home / "data"), + LC_ALL="C", + LANG="C", + TZ="UTC", + ) + return environment + + +@dataclass(frozen=True) +class IndexRequest: + """What an indexer is given: a frozen copy, a scratch area, and a pin.""" + + source_root: Path + output_path: Path + environment: Mapping[str, str] + pin: GraphifyPin + commit: str + tree: str + + +@dataclass(frozen=True) +class IndexResult: + """What an indexer reports back. ``completeness`` is its own admission.""" + + completeness: str = COMPLETE + indexed_files: int = 0 + notes: tuple[str, ...] = () + + +def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult]: + """Run a pinned provider CLI over the materialized copy. + + Kept as a factory so the lifecycle never imports or requires a graph + package: a deployment that has installed the pin supplies the executable, + and everything else -- including every test in this repository -- injects + its own callable. The child sees only ``request.environment``. + """ + + def run(request: IndexRequest) -> IndexResult: + try: + completed = subprocess.run( + [ + executable, + "index", + "--source", + str(request.source_root), + "--output", + str(request.output_path), + *request.pin.options, + ], + check=False, + capture_output=True, + text=False, + env=dict(request.environment), + cwd=str(request.source_root), + timeout=900, + ) + except (OSError, subprocess.SubprocessError): + raise ContextError("local graph provider could not be run from its pinned install") from None + if completed.returncode != 0: + # Provider stderr can echo indexed source; it is never surfaced. + raise ContextError("local graph provider failed; no generation was published") + return IndexResult(completeness=COMPLETE) + + return run + + +@dataclass(frozen=True) +class BuildManifest: + """The immutable record bound to one published generation. + + Every field an artifact must carry under issue #913 lives here: the full + commit and tree it was built from, the provider pin and options that built + it, when, the tracked census it was allowed to see and that census's + digest, the graph's own digest and byte count, and whether the provider + considered the result complete. + """ + + generation: str + schema: str + commit: str + tree: str + provider: dict[str, Any] + built_at: str + tracked_files: int + tracked_bytes: int + census_digest: str + graph_digest: str + graph_bytes: int + completeness: str + skipped_paths: int + indexed_files: int + + def to_json(self) -> dict[str, Any]: + return { + "schema": self.schema, + "generation": self.generation, + "commit": self.commit, + "tree": self.tree, + "provider": self.provider, + "built_at": self.built_at, + "tracked_files": self.tracked_files, + "tracked_bytes": self.tracked_bytes, + "census_digest": self.census_digest, + "graph_digest": self.graph_digest, + "graph_bytes": self.graph_bytes, + "completeness": self.completeness, + "skipped_paths": self.skipped_paths, + "indexed_files": self.indexed_files, + } + + def shareable_summary(self) -> dict[str, Any]: + """Metadata only: counts, digests and revision names, never content.""" + return { + "schema": "code_mower.contextGraphBuildSummary.v1", + "generation": self.generation, + "commit": self.commit, + "tree": self.tree, + "provider_version": self.provider.get("version"), + "built_at": self.built_at, + "tracked_files": self.tracked_files, + "census_digest": self.census_digest, + "graph_digest": self.graph_digest, + "graph_bytes": self.graph_bytes, + "completeness": self.completeness, + } + + +def load_manifest(payload: Mapping[str, Any]) -> BuildManifest: + """Validate a manifest. Every unreadable shape is a refusal, not a default.""" + if not isinstance(payload, Mapping): + raise ContextError("local graph manifest must be an object") + expected = { + "schema", "generation", "commit", "tree", "provider", "built_at", + "tracked_files", "tracked_bytes", "census_digest", "graph_digest", + "graph_bytes", "completeness", "skipped_paths", "indexed_files", + } + if set(payload) != expected: + raise ContextError("local graph manifest fields are missing or unrecognized") + if payload["schema"] != MANIFEST_SCHEMA: + raise ContextError("unsupported local graph manifest schema") + generation = payload["generation"] + if not isinstance(generation, str) or not _GENERATION.fullmatch(generation): + raise ContextError("local graph generation must be an opaque identifier") + completeness = payload["completeness"] + if completeness not in (COMPLETE, PARTIAL): + raise ContextError("unsupported local graph completeness") + provider = payload["provider"] + pin = load_pin(provider) if isinstance(provider, Mapping) else None + if pin is None: + raise ContextError("local graph manifest must record its provider pin") + _timestamp(payload["built_at"]) + return BuildManifest( + generation=generation, + schema=MANIFEST_SCHEMA, + commit=_object_name(payload["commit"]), + tree=_object_name(payload["tree"]), + provider=pin.as_metadata(), + built_at=payload["built_at"], + tracked_files=_size(payload["tracked_files"], MAX_TRACKED_FILES), + tracked_bytes=_size(payload["tracked_bytes"], MAX_TRACKED_BYTES), + census_digest=_digest(payload["census_digest"]), + graph_digest=_digest(payload["graph_digest"]), + graph_bytes=_size(payload["graph_bytes"], MAX_ARTIFACT_BYTES), + completeness=completeness, + skipped_paths=_size(payload["skipped_paths"], MAX_TRACKED_FILES), + indexed_files=_size(payload["indexed_files"], MAX_TRACKED_FILES), + ) + + +def workspace_id(repository: Path) -> str: + """A stable private name for one checkout. + + Derived from the resolved path so two worktrees of the same repository get + separate state and can never read each other's generations, and hashed so + the operator's directory layout is not spelled out in a shared location. + """ + return hashlib.sha256(str(Path(repository).resolve()).encode()).hexdigest()[:32] + + +def _open_private_directory(path: Path, *, create: bool) -> int: + if create: + path.mkdir(mode=0o700, parents=True, exist_ok=True) + try: + handle = os.open(path, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + except OSError: + raise ContextError("local graph state directory is unavailable or unsafe") from None + try: + _private(handle, directory=True) + except ContextError: + os.close(handle) + raise + return handle + + +class GraphStateRoot: + """Private, operator-owned, 0700 state for one checkout's generations. + + Layout under ``/graph//``:: + + generations//manifest.json + generations//graph.bin + current -- the published generation's name + + The directory is created 0700 and every read re-checks ownership and mode, + so state that was later loosened -- by an umask change, a restore, or a + careless ``chmod -R`` -- fails closed instead of being used. + """ + + def __init__(self, repository: Path, *, root: Path | None = None): + self.repository = Path(repository).resolve() + base = Path(root) if root is not None else default_context_root() + if not base.is_absolute(): + raise ContextError("local graph state requires an absolute private directory") + self.workspace = workspace_id(self.repository) + self.path = base / "graph" / self.workspace + + @property + def generations_path(self) -> Path: + return self.path / "generations" + + @property + def _chain(self) -> tuple[Path, ...]: + """Every directory this class owns, outermost first. + + Spelled out because ``mkdir(mode=0o700, parents=True)`` applies its mode + to the leaf only: intermediate directories would be created with the + process umask and end up group- or world-readable. + """ + return (self.path.parent.parent, self.path.parent, self.path, self.generations_path) + + def verify_private(self, *, create: bool = False) -> None: + """Re-check ownership and mode on every directory this class owns. + + Called on every read, not only at creation: state that was loosened + after the fact -- by a umask change, a restore, or a careless recursive + chmod -- must fail closed rather than be trusted because it was private + when it was written. + """ + for directory in self._chain: + if create or directory.exists(): + os.close(_open_private_directory(directory, create=create)) + + def ensure(self) -> None: + """Create the private tree, refusing to place state inside a repository.""" + if any((parent / ".git").exists() for parent in (self.path, *self.path.parents)): + raise ContextError("local graph state must stay outside Git repositories") + self.verify_private(create=True) + + def lock(self): + """Serialize builds for one checkout; concurrent ones would race publish.""" + self.ensure() + lock_path = self.path / "build.lock" + handle = os.open(lock_path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600) + stream = os.fdopen(handle, "a+", encoding="utf-8") + try: + _private(stream.fileno()) + return _BuildLock(stream) + except ContextError: + stream.close() + raise + + def current_generation(self) -> str | None: + pointer = self.path / CURRENT_NAME + try: + handle = os.open(pointer, os.O_RDONLY | os.O_NOFOLLOW) + except FileNotFoundError: + return None + except OSError: + raise ContextError("local graph state directory is unavailable or unsafe") from None + try: + _private(handle) + with os.fdopen(handle, "rb", closefd=False) as stream: + name = stream.read(64).decode("utf-8", "replace").strip() + finally: + os.close(handle) + if not _GENERATION.fullmatch(name): + raise ContextError("local graph generation pointer is corrupt; rebuild the graph") + return name + + def generation_names(self) -> list[str]: + if not self.path.exists(): + return [] + self.verify_private() + try: + names = os.listdir(self.generations_path) + except FileNotFoundError: + return [] + except OSError: + raise ContextError("local graph state directory is unavailable or unsafe") from None + return sorted(name for name in names if _GENERATION.fullmatch(name)) + + def read_manifest(self, generation: str) -> BuildManifest: + """Read and validate one generation's manifest, permissions included.""" + if not _GENERATION.fullmatch(generation): + raise ContextError("local graph generation must be an opaque identifier") + directory = self.generations_path / generation + os.close(_open_private_directory(directory, create=False)) + try: + handle = os.open(directory / MANIFEST_NAME, os.O_RDONLY | os.O_NOFOLLOW) + except OSError: + raise ContextError("local graph generation is missing its manifest") from None + try: + _private(handle) + with os.fdopen(handle, "rb", closefd=False) as stream: + raw = stream.read(MAX_MANIFEST_BYTES + 1) + finally: + os.close(handle) + if len(raw) > MAX_MANIFEST_BYTES: + raise ContextError("local graph manifest exceeds its bound") + try: + payload = json.loads(raw) + except ValueError: + raise ContextError("local graph manifest is corrupt; rebuild the graph") from None + manifest = load_manifest(payload) + if manifest.generation != generation: + raise ContextError("local graph manifest does not match its generation") + return manifest + + def artifact_path(self, generation: str) -> Path: + if not _GENERATION.fullmatch(generation): + raise ContextError("local graph generation must be an opaque identifier") + return self.generations_path / generation / ARTIFACT_NAME + + def publish(self, manifest: BuildManifest, artifact: bytes) -> BuildManifest: + """Assemble a generation under a staging name, then rename it into place. + + Both steps are atomic renames, in an order a reader can survive: the + generation directory becomes visible whole, and only then does + ``current`` start naming it. A crash between the two leaves an + unreferenced generation, which ``prune`` removes; it never leaves a + pointer to a directory that does not exist. + """ + self.ensure() + staging = self.generations_path / ("." + uuid.uuid4().hex + ".staging") + staging.mkdir(mode=0o700) + try: + _write_private_file(staging / ARTIFACT_NAME, artifact) + _write_private_file( + staging / MANIFEST_NAME, + json.dumps(manifest.to_json(), allow_nan=False, sort_keys=True, separators=(",", ":")).encode(), + ) + _fsync_directory(staging) + final = self.generations_path / manifest.generation + os.rename(staging, final) + _fsync_directory(self.generations_path) + except BaseException: + shutil.rmtree(staging, ignore_errors=True) + raise + pointer = self.path / CURRENT_NAME + temporary = self.path / ("." + uuid.uuid4().hex + ".tmp") + try: + _write_private_file(temporary, manifest.generation.encode() + b"\n") + os.replace(temporary, pointer) + _fsync_directory(self.path) + except BaseException: + temporary.unlink(missing_ok=True) + raise + return manifest + + def prune(self, *, keep: str | None) -> list[str]: + """Remove every generation but ``keep``, including crashed stagings.""" + removed = [] + try: + names = os.listdir(self.generations_path) + except FileNotFoundError: + return removed + for name in sorted(names): + if name == keep: + continue + shutil.rmtree(self.generations_path / name, ignore_errors=True) + if _GENERATION.fullmatch(name): + removed.append(name) + _fsync_directory(self.generations_path) + return removed + + def remove_all(self) -> bool: + """Delete every trace of this checkout's graph state.""" + if not self.path.exists(): + return False + # Refuse to delete a tree that is not ours; a loosened or foreign + # directory is reported, not recursively removed. + self.verify_private() + shutil.rmtree(self.path) + return True + + +class _BuildLock: + def __init__(self, stream): + self._stream = stream + self._guard = None + + def __enter__(self): + self._guard = exclusive_handle_lock(self._stream, timeout_seconds=35) + try: + self._guard.__enter__() + except FileLockError: + raise ContextError("a local graph build is already running for this checkout") from None + return self + + def __exit__(self, *exception): + try: + if self._guard is not None: + self._guard.__exit__(*exception) + finally: + self._stream.close() + return False + + +def _write_private_file(path: Path, payload: bytes) -> None: + handle = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + with os.fdopen(handle, "wb", closefd=False) as stream: + stream.write(payload) + stream.flush() + os.fsync(handle) + os.close(handle) + + +def _fsync_directory(path: Path) -> None: + handle = os.open(path, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: + os.fsync(handle) + finally: + os.close(handle) + + +@dataclass(frozen=True) +class GenerationStatus: + """A shareable verdict about the published generation, if any.""" + + state: str + generation: str | None = None + manifest: BuildManifest | None = None + detail: str = "" + + @property + def usable(self) -> bool: + return self.state == "current" + + def shareable_summary(self) -> dict[str, Any]: + summary: dict[str, Any] = { + "schema": "code_mower.contextGraphStatus.v1", + "state": self.state, + "usable": self.usable, + } + if self.detail: + summary["detail"] = self.detail + if self.manifest is not None: + summary["build"] = self.manifest.shareable_summary() + elif self.generation is not None: + summary["generation"] = self.generation + return summary + + +def build_graph( + repository: Path, + *, + pin: GraphifyPin, + indexer: Callable[[IndexRequest], IndexResult], + root: Path | None = None, + revision: str = "HEAD", + now: datetime | None = None, + keep_previous: bool = False, +) -> BuildManifest: + """Materialize one commit, index the copy, and publish a new generation. + + This is the whole lifecycle in one call, and it is the only way a + generation is created. ``refresh`` is the same operation: an explicit + rebuild that publishes a new immutable generation rather than mutating the + one in place, which is why nothing here ever writes into an existing + generation directory. + """ + repository = Path(repository).resolve() + state = GraphStateRoot(repository, root=root) + commit, tree = resolve_revision(repository, revision) + census = read_tracked_census(repository, commit) + with state.lock(): + build_root = state.path / ("." + uuid.uuid4().hex + ".build") + build_root.mkdir(mode=0o700, parents=True) + try: + source_root = build_root / "source" + home = build_root / "home" + temporary = build_root / "tmp" + for directory in (home, temporary): + directory.mkdir(mode=0o700) + materialize_tracked_files(repository, census, source_root) + output_path = build_root / ARTIFACT_NAME + result = indexer( + IndexRequest( + source_root=source_root, + output_path=output_path, + environment=scrubbed_environment(home=home, temporary=temporary), + pin=pin, + commit=commit, + tree=tree, + ) + ) + if not isinstance(result, IndexResult) or result.completeness not in (COMPLETE, PARTIAL): + raise ContextError("local graph provider returned an unsupported result") + if not output_path.is_file(): + raise ContextError("local graph provider produced no artifact") + size = output_path.stat().st_size + if size > MAX_ARTIFACT_BYTES: + raise ContextError("local graph artifact exceeds its budget; no generation was published") + artifact = output_path.read_bytes() + manifest = BuildManifest( + generation=uuid.uuid4().hex, + schema=MANIFEST_SCHEMA, + commit=commit, + tree=tree, + provider=pin.as_metadata(), + built_at=(now or datetime.now(timezone.utc)).astimezone(timezone.utc).isoformat(), + tracked_files=census.file_count, + tracked_bytes=census.total_bytes, + census_digest=census.digest, + graph_digest=hashlib.sha256(artifact).hexdigest(), + graph_bytes=len(artifact), + completeness=result.completeness, + skipped_paths=len(census.skipped), + indexed_files=min(_size(result.indexed_files, MAX_TRACKED_FILES), census.file_count), + ) + published = state.publish(manifest, artifact) + finally: + shutil.rmtree(build_root, ignore_errors=True) + if not keep_previous: + state.prune(keep=published.generation) + return published + + +def graph_status( + repository: Path, + *, + root: Path | None = None, + revision: str = "HEAD", + require_complete: bool = True, +) -> GenerationStatus: + """Report whether the published generation may be used, and why not if not. + + Every failure mode issue #913 names resolves here to a non-``current`` + state, and every non-``current`` state is unusable. Nothing falls back to a + previous generation: a consumer that cannot have the revision it asked for + is told so rather than handed an older answer that looks fresh. + """ + state = GraphStateRoot(repository, root=root) + try: + if not state.path.exists(): + return GenerationStatus(state="absent", detail="no local graph has been built for this checkout") + state.verify_private() + generation = state.current_generation() + except ContextError as error: + return GenerationStatus(state="invalid", detail=str(error)) + if generation is None: + return GenerationStatus(state="absent", detail="no local graph generation is published") + try: + manifest = state.read_manifest(generation) + except ContextError as error: + return GenerationStatus(state="invalid", generation=generation, detail=str(error)) + try: + artifact_path = state.artifact_path(generation) + handle = os.open(artifact_path, os.O_RDONLY | os.O_NOFOLLOW) + try: + _private(handle) + info = os.fstat(handle) + if info.st_size > MAX_ARTIFACT_BYTES: + # Checked before the manifest comparison and before any read: + # an artifact that grew past its budget is refused without + # being hashed, however plausible its manifest looks. + return GenerationStatus(state="oversized", generation=generation, manifest=manifest, + detail="local graph artifact exceeds its budget") + if info.st_size != manifest.graph_bytes: + return GenerationStatus(state="corrupt", generation=generation, manifest=manifest, + detail="local graph artifact size does not match its manifest") + digest = hashlib.sha256() + with os.fdopen(handle, "rb", closefd=False) as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + finally: + os.close(handle) + except ContextError as error: + return GenerationStatus(state="invalid", generation=generation, manifest=manifest, detail=str(error)) + except OSError: + return GenerationStatus(state="corrupt", generation=generation, manifest=manifest, + detail="local graph artifact is unreadable") + if digest.hexdigest() != manifest.graph_digest: + return GenerationStatus(state="corrupt", generation=generation, manifest=manifest, + detail="local graph artifact does not match its recorded digest") + try: + commit, tree = resolve_revision(Path(repository), revision) + except ContextError as error: + return GenerationStatus(state="invalid", generation=generation, manifest=manifest, detail=str(error)) + if (manifest.commit, manifest.tree) != (commit, tree): + return GenerationStatus(state="stale", generation=generation, manifest=manifest, + detail="local graph was built from a different revision; refresh it") + if require_complete and manifest.completeness != COMPLETE: + return GenerationStatus(state="partial", generation=generation, manifest=manifest, + detail="local graph build was incomplete; refresh it") + return GenerationStatus(state="current", generation=generation, manifest=manifest) + + +def remove_graph(repository: Path, *, root: Path | None = None) -> bool: + """Delete this checkout's graph state. Returns whether anything was removed.""" + return GraphStateRoot(repository, root=root).remove_all() + + +def doctor_report( + repository: Path, + *, + pin: GraphifyPin | None, + root: Path | None = None, + revision: str = "HEAD", +) -> dict[str, Any]: + """Posture checks for the local graph: metadata only, never content. + + Reports ``skip`` rather than ``fail`` when no graph is configured or built. + The lifecycle is optional, and an operator who never opted in has nothing + wrong with their installation. + """ + checks: list[dict[str, Any]] = [] + + def record(name: str, status: str, message: str, **extra: Any) -> None: + checks.append({"check": name, "status": status, "message": message, **extra}) + + if pin is None: + record("context-graph-pin", "skip", "no local graph provider is pinned; the lifecycle is optional") + else: + record("context-graph-pin", "pass", "local graph provider is pinned to one exact release", + requirement=pin.requirement, wheel_sha256=pin.wheel_sha256) + + state = GraphStateRoot(repository, root=root) + if not state.path.exists(): + record("context-graph-state", "skip", "no private local graph state exists for this checkout") + else: + try: + state.verify_private() + record("context-graph-state", "pass", "local graph state is private and operator-owned") + except ContextError as error: + record("context-graph-state", "fail", str(error)) + + status = graph_status(repository, root=root, revision=revision) + if status.state == "absent": + record("context-graph-generation", "skip", status.detail or "no local graph generation is published") + elif status.usable: + build = {key: value for key, value in status.shareable_summary().get("build", {}).items() + if key != "schema"} + record("context-graph-generation", "pass", + "the published generation binds the current revision", **build) + else: + record("context-graph-generation", "fail", status.detail or f"local graph is {status.state}", + state=status.state) + + ordering = {"fail": 0, "warn": 1, "pass": 2, "skip": 3} + overall = min((check["status"] for check in checks), key=lambda value: ordering[value]) + return { + "schema": "code_mower.contextGraphDoctor.v1", + "status": "fail" if any(check["status"] == "fail" for check in checks) else overall, + "checks": checks, + } + + +def render_status_text(status: GenerationStatus) -> str: + """A short operator-facing summary. Metadata only; no indexed content.""" + lines = [f"Local graph: {status.state}"] + if status.detail: + lines.append(f" {status.detail}") + manifest = status.manifest + if manifest is not None: + lines.extend( + [ + f" generation: {manifest.generation}", + f" commit: {manifest.commit}", + f" tree: {manifest.tree}", + f" provider: {manifest.provider.get('distribution')}=={manifest.provider.get('version')}", + f" built at: {manifest.built_at}", + f" tracked: {manifest.tracked_files} files / {manifest.tracked_bytes} bytes", + f" census: {manifest.census_digest}", + f" graph: {manifest.graph_digest} ({manifest.graph_bytes} bytes)", + f" complete: {manifest.completeness}", + ] + ) + return "\n".join(lines) + "\n" + + +def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[str]: + """Published and unreferenced generation names, for operator inspection.""" + yield from GraphStateRoot(repository, root=root).generation_names() + + +__all__: Sequence[str] = ( + "ARTIFACT_NAME", + "BuildManifest", + "COMPLETE", + "GenerationStatus", + "GraphStateRoot", + "GraphifyPin", + "IndexRequest", + "IndexResult", + "MANIFEST_SCHEMA", + "MAX_ARTIFACT_BYTES", + "MAX_TRACKED_FILES", + "PARTIAL", + "TrackedCensus", + "TrackedEntry", + "build_graph", + "doctor_report", + "graph_status", + "iter_generations", + "load_manifest", + "load_pin", + "materialize_tracked_files", + "read_tracked_census", + "remove_graph", + "render_status_text", + "resolve_revision", + "scrubbed_environment", + "subprocess_indexer", + "workspace_id", +) diff --git a/src/code_mower/package_manifest.py b/src/code_mower/package_manifest.py index d48e87b7..ce2604e1 100644 --- a/src/code_mower/package_manifest.py +++ b/src/code_mower/package_manifest.py @@ -36,6 +36,16 @@ ("src/code_mower/context_readiness.py", "src/code_mower/context_readiness.py", "core"), ("src/code_mower/context_session.py", "src/code_mower/context_session.py", "core"), ("src/code_mower/context_graph.py", "src/code_mower/context_graph.py", "core"), + ( + "src/code_mower/context_graph_lifecycle.py", + "src/code_mower/context_graph_lifecycle.py", + "core", + ), + ( + "src/code_mower/context_graph_command.py", + "src/code_mower/context_graph_command.py", + "core", + ), ("src/code_mower/productivity_report.py", "src/code_mower/productivity_report.py", "core"), ("tools/code_mower_requirements.txt", "requirements/requirements.txt", "tooling"), ("tools/code_mower_calibration.py", "src/code_mower/code_mower_calibration.py", "core"), diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py new file mode 100644 index 00000000..b71120ba --- /dev/null +++ b/tests/test_context_graph_lifecycle.py @@ -0,0 +1,604 @@ +"""Offline lifecycle tests for the optional local repository graph (issue #913). + +Every test here builds a real throwaway Git repository and runs the whole +lifecycle against it with an injected indexer. No graph package is installed, +imported, or required, and nothing reaches the network: the provider seam is a +callable, so the parts this repository is responsible for -- what gets +materialized, what the manifest binds, how a generation is published, and when +a consumer must refuse one -- are all provable locally. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import stat +import subprocess +import tempfile +import unittest +from datetime import datetime, timezone +from pathlib import Path + +from code_mower import context_graph_command as command +from code_mower import context_graph_lifecycle as lifecycle +from code_mower.context_contract import ContextError + + +PIN = lifecycle.GraphifyPin( + distribution="graphifyy", + version="0.9.58", + wheel_sha256="a" * 64, + options=("--no-network",), +) +NOW = datetime(2026, 3, 1, 9, 30, tzinfo=timezone.utc) + + +def git(repository: Path, *arguments: str) -> str: + return subprocess.run( + ["git", "-C", str(repository), *arguments], + check=True, + capture_output=True, + text=True, + env={ + "PATH": os.environ.get("PATH", ""), + "HOME": str(repository), + "GIT_CONFIG_NOSYSTEM": "1", + "GIT_AUTHOR_NAME": "Test", + "GIT_AUTHOR_EMAIL": "test@example.invalid", + "GIT_COMMITTER_NAME": "Test", + "GIT_COMMITTER_EMAIL": "test@example.invalid", + }, + ).stdout + + +def make_repository(root: Path) -> Path: + """A small repository with a tracked file, an ignored file and a secret.""" + repository = root / "checkout" + repository.mkdir() + git(repository, "init", "-q", "-b", "main") + (repository / "example_pkg").mkdir() + (repository / "example_pkg" / "config.py").write_text("VALUE = 1\n", encoding="utf-8") + (repository / "README.md").write_text("# example\n", encoding="utf-8") + (repository / ".gitignore").write_text("scratch/\n", encoding="utf-8") + git(repository, "add", ".") + git(repository, "commit", "-q", "-m", "initial") + # Present in the working tree at build time, and tracked by nothing. + (repository / "scratch").mkdir() + (repository / "scratch" / "notes.txt").write_text("private working note\n", encoding="utf-8") + (repository / "untracked-secret.env").write_text("TOKEN=not-a-real-secret\n", encoding="utf-8") + return repository + + +def recording_indexer(payload: bytes = b"graph-bytes", *, completeness: str = lifecycle.COMPLETE, + seen: list | None = None, indexed_files: int = 0): + """An indexer that writes a fixed artifact and records what it was shown.""" + + def run(request: lifecycle.IndexRequest) -> lifecycle.IndexResult: + if seen is not None: + seen.append(request) + request.output_path.write_bytes(payload) + return lifecycle.IndexResult(completeness=completeness, indexed_files=indexed_files) + + return run + + +class TemporaryWorkspace(unittest.TestCase): + def setUp(self) -> None: + self._directory = tempfile.TemporaryDirectory() + self.addCleanup(self._directory.cleanup) + self.root = Path(self._directory.name).resolve() + self.state = self.root / "state" + self.repository = make_repository(self.root) + + def build(self, **overrides): + arguments = { + "pin": PIN, + "indexer": recording_indexer(), + "root": self.state, + "now": NOW, + } + arguments.update(overrides) + return lifecycle.build_graph(self.repository, **arguments) + + +class PinTests(unittest.TestCase): + def test_accepts_one_exact_release(self) -> None: + pin = lifecycle.load_pin( + {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": "b" * 64} + ) + self.assertEqual(pin.requirement, "graphifyy==0.9.58") + self.assertEqual(pin.options, ()) + + def test_rejects_ranges_and_unpinned_shapes(self) -> None: + """A range, a marker, or a missing digest lets a build drift silently.""" + for version in (">=0.9", "0.9.*", "latest", "", "0.9.58; python_version>'3'"): + with self.subTest(version=version): + with self.assertRaises(ContextError): + lifecycle.load_pin( + {"distribution": "graphifyy", "version": version, "wheel_sha256": "b" * 64} + ) + + def test_rejects_missing_or_malformed_artifact_digest(self) -> None: + for digest in (None, "", "b" * 63, "not-hex" + "b" * 57): + with self.subTest(digest=digest): + with self.assertRaises(ContextError): + lifecycle.load_pin( + {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": digest} + ) + + def test_rejects_unknown_fields(self) -> None: + with self.assertRaises(ContextError): + lifecycle.load_pin( + {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": "b" * 64, + "index_url": "https://example.invalid/simple"} + ) + + +class CensusAndMaterializationTests(TemporaryWorkspace): + def test_census_reads_the_commit_not_the_working_tree(self) -> None: + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + self.assertEqual( + [entry.path for entry in census.entries], + [".gitignore", "README.md", "example_pkg/config.py"], + ) + + def test_census_digest_changes_when_tracked_content_changes(self) -> None: + commit, _ = lifecycle.resolve_revision(self.repository) + before = lifecycle.read_tracked_census(self.repository, commit).digest + (self.repository / "README.md").write_text("# example changed\n", encoding="utf-8") + git(self.repository, "commit", "-q", "-am", "change") + after_commit, _ = lifecycle.resolve_revision(self.repository) + self.assertNotEqual(before, lifecycle.read_tracked_census(self.repository, after_commit).digest) + + def test_symlinks_and_submodules_are_skipped_rather_than_followed(self) -> None: + """A tracked symlink can name a target the build was never shown.""" + os.symlink("/etc/passwd", self.repository / "linked.py") + git(self.repository, "add", "linked.py") + git(self.repository, "commit", "-q", "-m", "symlink") + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + self.assertNotIn("linked.py", [entry.path for entry in census.entries]) + self.assertIn(("linked.py", "symlink"), census.skipped) + + def test_materialization_writes_only_tracked_files(self) -> None: + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + destination = self.root / "materialized" + written = lifecycle.materialize_tracked_files(self.repository, census, destination) + present = sorted( + str(path.relative_to(destination)) + for path in destination.rglob("*") + if path.is_file() + ) + self.assertEqual(present, [".gitignore", "README.md", "example_pkg/config.py"]) + self.assertEqual(written, census.total_bytes) + self.assertFalse((destination / "scratch").exists()) + self.assertFalse((destination / "untracked-secret.env").exists()) + self.assertFalse((destination / ".git").exists()) + + def test_materialization_is_private_and_non_executable(self) -> None: + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + destination = self.root / "materialized" + lifecycle.materialize_tracked_files(self.repository, census, destination) + self.assertEqual(stat.S_IMODE(destination.stat().st_mode), 0o700) + for path in destination.rglob("*"): + if path.is_file(): + self.assertEqual(stat.S_IMODE(path.stat().st_mode), 0o600) + + def test_materialization_refuses_an_existing_directory(self) -> None: + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + destination = self.root / "materialized" + destination.mkdir() + with self.assertRaises(ContextError): + lifecycle.materialize_tracked_files(self.repository, census, destination) + + def test_escaping_census_paths_are_rejected(self) -> None: + escaping = ("/etc/passwd", "../outside.py", "a/../../b.py", ".git/config", + "vendor/.git/config", "a\\b.py") + for index, path in enumerate(escaping): + with self.subTest(path=path): + census = lifecycle.TrackedCensus( + entries=(lifecycle.TrackedEntry("100644", "0" * 40, path, 1),), + skipped=(), + digest="c" * 64, + ) + with self.assertRaises(ContextError): + lifecycle.materialize_tracked_files( + self.repository, census, self.root / f"escape-{index}" + ) + + +class ScrubbedEnvironmentTests(TemporaryWorkspace): + def test_indexer_never_inherits_ambient_credentials(self) -> None: + seen: list[lifecycle.IndexRequest] = [] + secrets = { + "GITHUB_TOKEN": "not-a-real-token", + "ANTHROPIC_API_KEY": "not-a-real-key", + "AWS_SECRET_ACCESS_KEY": "not-a-real-key", + "GRAPHIFY_API_KEY": "not-a-real-key", + } + previous = {name: os.environ.get(name) for name in secrets} + os.environ.update(secrets) + try: + self.build(indexer=recording_indexer(seen=seen)) + finally: + for name, value in previous.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + environment = seen[0].environment + for name in secrets: + self.assertNotIn(name, environment) + self.assertEqual(environment["no_proxy"], "*") + self.assertEqual(environment["https_proxy"], "") + self.assertEqual(environment["GIT_TERMINAL_PROMPT"], "0") + + def test_provider_home_is_redirected_away_from_the_operator(self) -> None: + seen: list[lifecycle.IndexRequest] = [] + self.build(indexer=recording_indexer(seen=seen)) + home = Path(seen[0].environment["HOME"]) + self.assertNotEqual(home, Path.home()) + self.assertTrue(str(home).startswith(str(self.state))) + + def test_allowlist_drops_everything_it_does_not_name(self) -> None: + environment = lifecycle.scrubbed_environment(home=self.root / "h", temporary=self.root / "t") + allowed = set(lifecycle._ENVIRONMENT_ALLOWLIST) | set(lifecycle._NETWORK_DENY) | { + "HOME", "XDG_CONFIG_HOME", "XDG_CACHE_HOME", "XDG_DATA_HOME", + } + self.assertEqual(set(environment) - allowed, set()) + + +class BuildAndPublishTests(TemporaryWorkspace): + def test_manifest_binds_every_required_fact(self) -> None: + manifest = self.build() + commit, tree = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + self.assertEqual(manifest.commit, commit) + self.assertEqual(manifest.tree, tree) + self.assertEqual(manifest.provider, PIN.as_metadata()) + self.assertEqual(manifest.built_at, NOW.isoformat()) + self.assertEqual(manifest.tracked_files, census.file_count) + self.assertEqual(manifest.tracked_bytes, census.total_bytes) + self.assertEqual(manifest.census_digest, census.digest) + self.assertEqual(manifest.graph_digest, hashlib.sha256(b"graph-bytes").hexdigest()) + self.assertEqual(manifest.graph_bytes, len(b"graph-bytes")) + self.assertEqual(manifest.completeness, lifecycle.COMPLETE) + + def test_manifest_round_trips_through_validation(self) -> None: + manifest = self.build() + self.assertEqual(lifecycle.load_manifest(manifest.to_json()), manifest) + + def test_shareable_summary_carries_no_content_or_local_path(self) -> None: + summary = self.build().shareable_summary() + rendered = json.dumps(summary) + self.assertNotIn(str(self.root), rendered) + self.assertNotIn("VALUE = 1", rendered) + self.assertNotIn("scratch", rendered) + + def test_state_is_private_to_the_operator(self) -> None: + self.build() + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + self.assertEqual(stat.S_IMODE(state.path.stat().st_mode), 0o700) + generation = state.current_generation() + self.assertEqual(stat.S_IMODE((state.generations_path / generation).stat().st_mode), 0o700) + self.assertEqual(stat.S_IMODE(state.artifact_path(generation).stat().st_mode), 0o600) + + def test_two_worktrees_of_one_repository_keep_separate_state(self) -> None: + other = self.root / "other" + other.mkdir() + self.assertNotEqual(lifecycle.workspace_id(self.repository), lifecycle.workspace_id(other)) + + def test_refresh_publishes_a_new_immutable_generation(self) -> None: + first = self.build(keep_previous=True) + (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") + git(self.repository, "commit", "-q", "-am", "change") + second = self.build(indexer=recording_indexer(b"second-graph"), keep_previous=True) + self.assertNotEqual(first.generation, second.generation) + self.assertNotEqual(first.commit, second.commit) + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + self.assertEqual(state.current_generation(), second.generation) + # The superseded generation is untouched, not rewritten in place. + self.assertEqual(state.read_manifest(first.generation), first) + self.assertEqual(state.artifact_path(first.generation).read_bytes(), b"graph-bytes") + + def test_pruning_keeps_only_the_published_generation(self) -> None: + self.build() + (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") + git(self.repository, "commit", "-q", "-am", "change") + second = self.build() + self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), [second.generation]) + + def test_a_failed_build_publishes_nothing(self) -> None: + first = self.build() + + def failing(request: lifecycle.IndexRequest) -> lifecycle.IndexResult: + raise ContextError("provider failed") + + with self.assertRaises(ContextError): + self.build(indexer=failing) + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + self.assertEqual(state.current_generation(), first.generation) + self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), [first.generation]) + + def test_a_provider_that_writes_nothing_fails_closed(self) -> None: + def silent(request: lifecycle.IndexRequest) -> lifecycle.IndexResult: + return lifecycle.IndexResult() + + with self.assertRaises(ContextError): + self.build(indexer=silent) + self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), []) + + def test_an_oversized_artifact_is_refused_before_publication(self) -> None: + original = lifecycle.MAX_ARTIFACT_BYTES + lifecycle.MAX_ARTIFACT_BYTES = 4 + try: + with self.assertRaises(ContextError): + self.build(indexer=recording_indexer(b"too-large-for-the-budget")) + finally: + lifecycle.MAX_ARTIFACT_BYTES = original + self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), []) + + def test_state_is_refused_inside_a_git_repository(self) -> None: + with self.assertRaises(ContextError): + self.build(root=self.repository / ".code-mower-state") + + +class StatusFailsClosedTests(TemporaryWorkspace): + def test_a_fresh_build_is_current(self) -> None: + manifest = self.build() + status = lifecycle.graph_status(self.repository, root=self.state) + self.assertEqual(status.state, "current") + self.assertTrue(status.usable) + self.assertEqual(status.manifest, manifest) + + def test_no_state_is_absent_and_unusable(self) -> None: + status = lifecycle.graph_status(self.repository, root=self.state) + self.assertEqual(status.state, "absent") + self.assertFalse(status.usable) + + def test_a_new_commit_makes_the_graph_stale(self) -> None: + self.build() + (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") + git(self.repository, "commit", "-q", "-am", "change") + status = lifecycle.graph_status(self.repository, root=self.state) + self.assertEqual(status.state, "stale") + self.assertFalse(status.usable) + + def test_a_tampered_artifact_is_corrupt(self) -> None: + manifest = self.build() + artifact = lifecycle.GraphStateRoot(self.repository, root=self.state).artifact_path(manifest.generation) + artifact.write_bytes(b"tampered!!!") # same length, different content + status = lifecycle.graph_status(self.repository, root=self.state) + self.assertEqual(status.state, "corrupt") + self.assertFalse(status.usable) + + def test_a_truncated_artifact_is_corrupt(self) -> None: + manifest = self.build() + artifact = lifecycle.GraphStateRoot(self.repository, root=self.state).artifact_path(manifest.generation) + artifact.write_bytes(b"short") + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "corrupt") + + def test_an_oversized_artifact_is_refused_on_read(self) -> None: + manifest = self.build() + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + artifact = state.artifact_path(manifest.generation) + original = lifecycle.MAX_ARTIFACT_BYTES + lifecycle.MAX_ARTIFACT_BYTES = 4 + try: + self.assertIn( + lifecycle.graph_status(self.repository, root=self.state).state, + ("corrupt", "oversized", "invalid"), + ) + finally: + lifecycle.MAX_ARTIFACT_BYTES = original + self.assertTrue(artifact.exists()) + + def test_a_partial_build_is_unusable_by_default(self) -> None: + self.build(indexer=recording_indexer(completeness=lifecycle.PARTIAL)) + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "partial") + allowed = lifecycle.graph_status(self.repository, root=self.state, require_complete=False) + self.assertEqual(allowed.state, "current") + + def test_a_corrupt_manifest_is_invalid(self) -> None: + manifest = self.build() + path = ( + lifecycle.GraphStateRoot(self.repository, root=self.state).generations_path + / manifest.generation + / lifecycle.MANIFEST_NAME + ) + path.write_text("{not json", encoding="utf-8") + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "invalid") + + def test_a_manifest_missing_its_revision_binding_is_invalid(self) -> None: + manifest = self.build() + payload = manifest.to_json() + payload.pop("tree") + path = ( + lifecycle.GraphStateRoot(self.repository, root=self.state).generations_path + / manifest.generation + / lifecycle.MANIFEST_NAME + ) + path.write_text(json.dumps(payload), encoding="utf-8") + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "invalid") + + def test_a_manifest_relabelled_to_another_generation_is_invalid(self) -> None: + manifest = self.build() + payload = manifest.to_json() + payload["generation"] = "f" * 32 + path = ( + lifecycle.GraphStateRoot(self.repository, root=self.state).generations_path + / manifest.generation + / lifecycle.MANIFEST_NAME + ) + path.write_text(json.dumps(payload), encoding="utf-8") + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "invalid") + + def test_a_corrupt_pointer_is_invalid(self) -> None: + self.build() + (lifecycle.GraphStateRoot(self.repository, root=self.state).path / lifecycle.CURRENT_NAME).write_text( + "../../elsewhere\n", encoding="utf-8" + ) + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "invalid") + + def test_group_readable_state_is_invalid(self) -> None: + """State loosened after the fact fails closed rather than being used.""" + self.build() + path = lifecycle.GraphStateRoot(self.repository, root=self.state).path + path.chmod(0o750) + try: + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "invalid") + finally: + path.chmod(0o700) + + def test_a_group_readable_artifact_is_invalid(self) -> None: + manifest = self.build() + artifact = lifecycle.GraphStateRoot(self.repository, root=self.state).artifact_path(manifest.generation) + artifact.chmod(0o640) + try: + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "invalid") + finally: + artifact.chmod(0o600) + + +class RemoveTests(TemporaryWorkspace): + def test_remove_deletes_every_generation(self) -> None: + self.build() + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + self.assertTrue(lifecycle.remove_graph(self.repository, root=self.state)) + self.assertFalse(state.path.exists()) + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "absent") + + def test_remove_is_idempotent(self) -> None: + self.assertFalse(lifecycle.remove_graph(self.repository, root=self.state)) + + def test_remove_refuses_state_that_is_not_private(self) -> None: + self.build() + path = lifecycle.GraphStateRoot(self.repository, root=self.state).path + path.chmod(0o755) + try: + with self.assertRaises(ContextError): + lifecycle.remove_graph(self.repository, root=self.state) + self.assertTrue(path.exists()) + finally: + path.chmod(0o700) + + +class DoctorTests(TemporaryWorkspace): + def test_an_unconfigured_installation_skips_rather_than_fails(self) -> None: + report = lifecycle.doctor_report(self.repository, pin=None, root=self.state) + self.assertEqual(report["status"], "skip") + self.assertEqual({check["status"] for check in report["checks"]}, {"skip"}) + + def test_a_healthy_build_passes(self) -> None: + self.build() + report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) + self.assertEqual(report["status"], "pass") + + def test_a_stale_graph_fails_doctor(self) -> None: + self.build() + (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") + git(self.repository, "commit", "-q", "-am", "change") + report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) + self.assertEqual(report["status"], "fail") + + def test_doctor_output_carries_no_indexed_content(self) -> None: + self.build() + rendered = json.dumps(lifecycle.doctor_report(self.repository, pin=PIN, root=self.state)) + self.assertNotIn("VALUE = 1", rendered) + self.assertNotIn("not-a-real-secret", rendered) + + +class CommandTests(TemporaryWorkspace): + def pin_file(self) -> Path: + path = self.root / "pin.json" + path.write_text(json.dumps(PIN.as_metadata()), encoding="utf-8") + return path + + def indexer_script(self) -> Path: + """A stand-in for a pinned provider CLI, so no package is required.""" + path = self.root / "fake-indexer" + path.write_text( + "#!/bin/sh\n" + 'while [ "$#" -gt 0 ]; do\n' + ' case "$1" in --output) shift; printf graph-bytes > "$1" ;; esac\n' + " shift\n" + "done\n", + encoding="utf-8", + ) + path.chmod(0o700) + return path + + def run_command(self, *arguments: str) -> tuple[int, str]: + from contextlib import redirect_stdout + from io import StringIO + + buffer = StringIO() + with redirect_stdout(buffer): + code = command.main(list(arguments)) + return code, buffer.getvalue() + + def base(self) -> list[str]: + return ["--repo-path", str(self.repository), "--state-dir", str(self.state), "--json"] + + def test_build_status_refresh_remove_round_trip(self) -> None: + pin, indexer = str(self.pin_file()), str(self.indexer_script()) + code, output = self.run_command("build", *self.base(), "--pin-file", pin, "--indexer", indexer) + self.assertEqual(code, 0, output) + published = json.loads(output) + self.assertEqual(published["status"], "published") + + code, output = self.run_command("status", *self.base()) + self.assertEqual(code, 0, output) + self.assertTrue(json.loads(output)["usable"]) + + # A second ``build`` refuses; ``refresh`` is the explicit rebuild verb. + code, _ = self.run_command("build", *self.base(), "--pin-file", pin, "--indexer", indexer) + self.assertEqual(code, 1) + code, output = self.run_command("refresh", *self.base(), "--pin-file", pin, "--indexer", indexer) + self.assertEqual(code, 0, output) + self.assertNotEqual(json.loads(output)["generation"], published["generation"]) + + code, output = self.run_command("remove", *self.base()) + self.assertEqual(code, 0, output) + self.assertTrue(json.loads(output)["removed"]) + code, output = self.run_command("status", *self.base()) + self.assertEqual(code, 1) + self.assertEqual(json.loads(output)["state"], "absent") + + def test_status_reports_stale_with_a_nonzero_exit(self) -> None: + self.build() + (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") + git(self.repository, "commit", "-q", "-am", "change") + code, output = self.run_command("status", *self.base()) + self.assertEqual(code, 1) + self.assertEqual(json.loads(output)["state"], "stale") + + def test_build_without_a_pin_is_refused(self) -> None: + code, _ = self.run_command("build", *self.base(), "--indexer", str(self.indexer_script())) + self.assertEqual(code, 1) + self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), []) + + def test_remove_hides_the_private_path_unless_asked(self) -> None: + self.build() + _, output = self.run_command("remove", *self.base()) + self.assertNotIn(str(self.state), output) + + def test_doctor_reports_an_unconfigured_installation(self) -> None: + code, output = self.run_command("doctor", *self.base()) + self.assertEqual(code, 0, output) + self.assertEqual(json.loads(output)["status"], "skip") + + def test_command_is_registered_on_the_cli(self) -> None: + from code_mower import cli + + self.assertIs(cli.COMMAND_HANDLERS["context-graph"], command.main) + self.assertIn("context-graph", cli.COMMAND_DESCRIPTIONS) + + +if __name__ == "__main__": # pragma: no cover - direct invocation + unittest.main() From fe08a2b4d0fda04c9a0b685c4abfac713496be55 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 02:31:28 -0700 Subject: [PATCH 02/25] context: register context-graph in the CLI command registry assertion test_cli_command_registry_is_single_source_of_truth pins the exact handler tuple, which is the point of it: a new command has to be declared in the registry rather than appearing by accident. Declaring it. Co-Authored-By: Claude Opus 5 (1M context) --- tests/test_release_hygiene.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_release_hygiene.py b/tests/test_release_hygiene.py index 6e9f973b..18ba090b 100644 --- a/tests/test_release_hygiene.py +++ b/tests/test_release_hygiene.py @@ -375,6 +375,7 @@ def test_cli_command_registry_is_single_source_of_truth(self) -> None: "cloud", "config", "context", + "context-graph", "context-packs", "controller", "coderabbit-cli", From a011eb45b93ba20f0055924d776eb491f4b93b9d Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 02:45:17 -0700 Subject: [PATCH 03/25] context: enforce the provider's network boundary in the kernel Three findings from the codex audit of fe08a2b. P1, network isolation. Emptied proxy variables and NO_PROXY=* were never a boundary: on a host with direct connectivity they ask a cooperating client to connect directly. The provider is now launched behind an argv prefix that denies it sockets at the OS level (sandbox-exec, bwrap --unshare-net, unshare --net), and no mechanism is trusted on its name -- each candidate is accepted only after a probe child launched behind it has been observed failing to connect with a denial. A refused connection is the failure case: it proves the syscall reached the network stack. A host where nothing passes gets no build, and doctor reports it as a failing context-graph-isolation check once a provider is pinned rather than waiting for the build to refuse. P1, pruning outside the lock. Publication and pruning now share one locked critical section. A builder that pruned after unlocking could delete the generation a second builder had just published, or its staging directory, leaving current naming a directory that is gone with both builds reporting success. P2, relative indexer paths. The child runs in the materialized copy, so --indexer .venv/bin/graphify was resolved there instead of against the invocation directory. A path with a directory component is now resolved before launch; a bare command name keeps its PATH lookup. Tests: the boundary is proven against a real loopback listener the sandboxed child cannot reach, with an unsandboxed control so a child that merely failed to start cannot read as isolation, plus a check that a pass-through launcher is rejected as a candidate. Also covered: the sandbox prefix on the provider argv, the refusal when no mechanism exists, relative and bare executable resolution, and prune ordering against lock release. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 45 +++++- docs/context-provider-contract.md | 3 +- src/code_mower/context_graph_lifecycle.py | 170 ++++++++++++++++++++-- tests/test_context_graph_lifecycle.py | 167 ++++++++++++++++++++- 4 files changed, 365 insertions(+), 20 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index ada79d7d..57b47c28 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -47,17 +47,45 @@ provenance at all. If Code Mower does not bind the revision, nothing does. directory as a 0600 file. Untracked and ignored files have no path into the graph because they are never written, rather than because something filtered them out afterwards. -4. **Run the indexer with a scrubbed environment.** The provider process - inherits an allowlist — `PATH`, `TMPDIR`, `LANG`, `LC_ALL`, `TZ` — and - nothing else. Every proxy variable is set empty, `no_proxy` is `*`, and - `HOME` and the XDG directories point into the build's own scratch area. A - newly invented secret variable is excluded by default because the list names - what is kept, not what is dropped. +4. **Run the indexer with a scrubbed environment, inside a network-denying + sandbox.** The provider process inherits an allowlist — `PATH`, `TMPDIR`, + `LANG`, `LC_ALL`, `TZ` — and nothing else, with `HOME` and the XDG + directories pointing into the build's own scratch area. A newly invented + secret variable is excluded by default because the list names what is kept, + not what is dropped. The network boundary is separate and is described + below; the emptied proxy variables are hygiene, not that boundary. 5. **Publish atomically.** The generation is assembled under a staging name, fsynced, renamed into `generations/`, and only then does the `current` pointer start naming it. A reader sees the whole previous generation or the whole new one. +Publication and pruning happen inside one locked critical section. Two builders +that race are serialized, and neither can delete the generation the other just +published while `current` still names it. + +## The network boundary + +An environment variable is a request, not a boundary: `NO_PROXY=*` asks a +cooperating client to connect *directly*, and on a host with internet access an +uncooperative provider is unaffected by any of it. So the provider is launched +behind an argv prefix that denies it sockets at the operating-system level — +`sandbox-exec` on macOS, `bwrap --unshare-net` or an unprivileged network +namespace via `unshare --net` on Linux. + +No mechanism is trusted on its name. Each candidate is accepted only after a +probe child launched behind it has been *observed* failing to open a TCP +connection with a denial — `EPERM`, `ENETUNREACH`, and the like. A *refused* +connection is the failure case: it proves the syscall reached the network stack, +so the candidate is rejected. The result is cached for the process, since it is +a property of the host. + +A host where no candidate passes gets no build. `subprocess_indexer()` raises +before a single blob is materialized, and `context-graph doctor` reports the +same condition as a failing `context-graph-isolation` check once a provider is +pinned. Running an unconfined provider is not offered as a fallback: an +operator who cannot contain a third-party indexer is better served by knowing +it than by a build that quietly could have reached the network. + Git itself runs with `GIT_CONFIG_NOSYSTEM`, `GIT_CONFIG_GLOBAL=/dev/null`, and `GIT_CONFIG_SYSTEM=/dev/null`: an untrusted checkout's local, global, or system configuration can otherwise install clean/smudge filters and hook paths that run @@ -142,7 +170,10 @@ marker, or a distribution without an artifact digest: `--indexer` is the path to a provider CLI the operator has **already** installed. This repository does not download, install, or resolve one, which is -why the executable is named rather than discovered. +why the executable is named rather than discovered. A relative path such as +`.venv/bin/graphify` is resolved against the directory the command was invoked +from, not against the materialized copy the provider runs in; a bare command +name keeps its `PATH` lookup. ## State layout diff --git a/docs/context-provider-contract.md b/docs/context-provider-contract.md index 09cda3ca..d6252fe1 100644 --- a/docs/context-provider-contract.md +++ b/docs/context-provider-contract.md @@ -207,7 +207,8 @@ point; they do not establish Graphify compatibility or make it a v1.3.1 dependency. The lifecycle around such a provider — exact pin, immutable tracked-file -materialization, scrubbed environment, private 0700 state, atomic generations, +materialization, scrubbed environment, an OS sandbox that denies the provider +the network, private 0700 state, atomic generations, and `code-mower context-graph build/refresh/status/remove/doctor` — is described in [Local repository graph](context-graph-lifecycle.md). It builds the evidence side; `context_graph` still decides whether a delivered packet's diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 42002bb7..c508ef98 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -20,9 +20,14 @@ name, fsynced, then renamed into place; the ``current`` pointer is replaced atomically afterwards. A reader either sees the whole previous generation or the whole new one, never a half-written directory. -* **Scrub the environment.** The indexer runs with an allowlisted environment - and proxy-denying network posture, so an ambient token cannot leak into a - provider process and the build cannot quietly reach the network. +* **Scrub the environment.** The indexer runs with an allowlisted environment, + so an ambient token cannot leak into a provider process. +* **Deny the network in the kernel, not by request.** Emptying proxy variables + only redirects a client that chooses to honour them. The provider is + launched inside an OS sandbox that refuses sockets outright, and the sandbox + is accepted only after a probe child has been observed failing to connect. A + host that offers no such mechanism gets a refused build, not an unconfined + provider. Nothing here installs, imports, or requires a graph package. The indexer is an injected callable, so the whole lifecycle is provable offline; the bundled @@ -38,6 +43,7 @@ import re import shutil import subprocess +import sys import uuid from dataclasses import dataclass from datetime import datetime, timezone @@ -82,8 +88,10 @@ #: variable is excluded by default instead of needing a new denylist entry. _ENVIRONMENT_ALLOWLIST = ("PATH", "TMPDIR", "LANG", "LC_ALL", "TZ") -#: Denying every proxy and refusing terminal prompts turns an attempted -#: network call into a fast local failure instead of an outbound request. +#: Hygiene, not the boundary. Emptying proxy variables stops a cooperating +#: client from finding a proxy and ``GIT_TERMINAL_PROMPT=0`` stops a child +#: blocking on a credential prompt, but on a host with direct connectivity +#: neither denies anything. The boundary is ``network_sandbox_command``. _NETWORK_DENY = { "no_proxy": "*", "NO_PROXY": "*", @@ -97,6 +105,102 @@ "PYTHONNOUSERSITE": "1", } +#: Argv prefixes that place a child in a network-denying OS sandbox, most +#: specific first. Each is a mechanism the host either has or does not; none is +#: trusted on its name, because a prefix that silently degrades to running the +#: command unconfined would be worse than no prefix at all. +_SANDBOX_CANDIDATES: tuple[tuple[str, ...], ...] = ( + ("/usr/bin/sandbox-exec", "-p", "(version 1)(allow default)(deny network*)"), + ("bwrap", "--unshare-net", "--dev-bind", "/", "/", "--"), + ("unshare", "--net", "--map-current-user", "--"), + ("unshare", "--net", "--map-root-user", "--"), +) + +#: The probe connects to the discard port on loopback, where a host without a +#: sandbox refuses the connection. Refusal is the *failure* case here: it proves +#: the syscall reached the network stack. Only an outright denial -- no +#: permission, no route, no address family -- proves the child was contained. +#: The probe exits ``7`` only on a denial; every other exit code -- a refused +#: connection, a launcher that could not start, a child that never ran -- means +#: the candidate is not usable as a boundary. +_PROBE_PORT = 9 +_PROBE_DENIED = 7 +_DENIAL_PROBE = """ +import errno +import socket +import sys + +DENIED = frozenset({ + errno.EPERM, + errno.EACCES, + errno.ENETUNREACH, + errno.ENETDOWN, + errno.EHOSTUNREACH, + errno.EADDRNOTAVAIL, + errno.EAFNOSUPPORT, + errno.EPROTONOSUPPORT, +}) +try: + probe = socket.socket() + probe.settimeout(5) + probe.connect(("127.0.0.1", int(sys.argv[1]))) +except OSError as error: + sys.exit(7 if error.errno in DENIED else 3) +sys.exit(3) +""" + +_sandbox_prefix: tuple[str, ...] | None = None +_sandbox_probed = False + + +def _launcher_path(name: str) -> str | None: + if os.path.isabs(name): + return name if os.access(name, os.X_OK) else None + return shutil.which(name) + + +def _sandbox_denies_network(prefix: Sequence[str]) -> bool: + """Watch a child under ``prefix`` fail to open a connection, or say no.""" + try: + completed = subprocess.run( + [*prefix, sys.executable, "-c", _DENIAL_PROBE, str(_PROBE_PORT)], + check=False, + capture_output=True, + timeout=60, + env={"PATH": os.environ.get("PATH", ""), **_NETWORK_DENY}, + ) + except (OSError, subprocess.SubprocessError): + return False + return completed.returncode == _PROBE_DENIED + + +def _probe_network_sandbox() -> tuple[str, ...] | None: + if not sys.executable: # pragma: no cover - a frozen interpreter cannot probe + return None + for candidate in _SANDBOX_CANDIDATES: + launcher = _launcher_path(candidate[0]) + if launcher is None: + continue + prefix = (launcher, *candidate[1:]) + if _sandbox_denies_network(prefix): + return prefix + return None + + +def network_sandbox_command() -> tuple[str, ...] | None: + """The argv prefix that denies a provider process the network, if any. + + Probed once per process and cached, because the answer is a property of the + host rather than of a build. ``None`` means this host offers no mechanism + this build could *observe* working, and a build refuses rather than running + a provider it cannot contain. + """ + global _sandbox_prefix, _sandbox_probed + if not _sandbox_probed: + _sandbox_prefix = _probe_network_sandbox() + _sandbox_probed = True + return _sandbox_prefix + def _object_name(value: Any) -> str: """A full SHA-1 or SHA-256 object name. Abbreviations do not bind.""" @@ -435,20 +539,48 @@ class IndexResult: notes: tuple[str, ...] = () +def _resolved_executable(executable: str) -> str: + """Bind a relative provider path to the invocation directory. + + The provider runs with its working directory set to the materialized copy, + so ``--indexer .venv/bin/graphify`` would otherwise be looked up inside the + frozen source tree, where the operator's install is not. A bare command + name keeps its ``PATH`` lookup, which is unaffected by the child's + directory. + """ + if not isinstance(executable, str) or not executable: + raise ContextError("local graph provider executable must be named") + separators = [os.sep, os.altsep] if os.altsep else [os.sep] + if any(separator in executable for separator in separators): + return str(Path(executable).resolve()) + return executable + + def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult]: - """Run a pinned provider CLI over the materialized copy. + """Run a pinned provider CLI over the materialized copy, without a network. Kept as a factory so the lifecycle never imports or requires a graph package: a deployment that has installed the pin supplies the executable, and everything else -- including every test in this repository -- injects - its own callable. The child sees only ``request.environment``. + its own callable. The child sees only ``request.environment``, and it sees + it from inside a sandbox that denies it sockets. Resolving the executable + and the sandbox here, rather than at build time, means an unusable provider + or an uncontainable host fails before a single blob is materialized. """ + command = _resolved_executable(executable) + sandbox = network_sandbox_command() + if sandbox is None: + raise ContextError( + "local graph builds need an OS sandbox that denies the provider network access; " + "this host offers none that could be verified" + ) def run(request: IndexRequest) -> IndexResult: try: completed = subprocess.run( [ - executable, + *sandbox, + command, "index", "--source", str(request.source_root), @@ -928,10 +1060,14 @@ def build_graph( indexed_files=min(_size(result.indexed_files, MAX_TRACKED_FILES), census.file_count), ) published = state.publish(manifest, artifact) + if not keep_previous: + # Inside the lock, with publication. Pruning after the lock is + # released would let a second builder publish first and then + # have its generation -- or its staging directory -- deleted by + # this one, leaving ``current`` naming a directory that is gone. + state.prune(keep=published.generation) finally: shutil.rmtree(build_root, ignore_errors=True) - if not keep_previous: - state.prune(keep=published.generation) return published @@ -1030,9 +1166,23 @@ def record(name: str, status: str, message: str, **extra: Any) -> None: if pin is None: record("context-graph-pin", "skip", "no local graph provider is pinned; the lifecycle is optional") + record("context-graph-isolation", "skip", "no provider is pinned, so nothing would be launched") else: record("context-graph-pin", "pass", "local graph provider is pinned to one exact release", requirement=pin.requirement, wheel_sha256=pin.wheel_sha256) + # Named while it is still a posture question. Discovering that this + # host cannot contain a provider is worth knowing before a build + # refuses, and the check reports the mechanism rather than its + # arguments, which would be noise. + sandbox = network_sandbox_command() + if sandbox is None: + record("context-graph-isolation", "fail", + "no OS sandbox on this host was observed denying a child process the network; " + "builds will refuse") + else: + record("context-graph-isolation", "pass", + "the provider would run inside a network-denying OS sandbox", + mechanism=os.path.basename(sandbox[0])) state = GraphStateRoot(repository, root=root) if not state.path.exists(): diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index b71120ba..15fc8b55 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -2,10 +2,15 @@ Every test here builds a real throwaway Git repository and runs the whole lifecycle against it with an injected indexer. No graph package is installed, -imported, or required, and nothing reaches the network: the provider seam is a +imported, or required, and nothing leaves this machine: the provider seam is a callable, so the parts this repository is responsible for -- what gets materialized, what the manifest binds, how a generation is published, and when a consumer must refuse one -- are all provable locally. + +``NetworkIsolationTests`` is the one place a socket is opened at all. It binds a +listener on loopback in this process and proves a sandboxed child cannot reach +it, which is the only honest way to test a network boundary: an assertion about +proxy variables would have passed on code that had none. """ from __future__ import annotations @@ -13,12 +18,15 @@ import hashlib import json import os +import socket import stat import subprocess +import sys import tempfile import unittest from datetime import datetime, timezone from pathlib import Path +from unittest import mock from code_mower import context_graph_command as command from code_mower import context_graph_lifecycle as lifecycle @@ -253,6 +261,117 @@ def test_allowlist_drops_everything_it_does_not_name(self) -> None: self.assertEqual(set(environment) - allowed, set()) +CONNECT_PROBE = """ +import socket +import sys + +try: + socket.create_connection(("127.0.0.1", int(sys.argv[1])), timeout=5).close() +except OSError: + sys.exit(1) +sys.exit(0) +""" + + +class NetworkIsolationTests(unittest.TestCase): + """The provider's network boundary, against a socket that is really there.""" + + def setUp(self) -> None: + self.listener = socket.socket() + self.addCleanup(self.listener.close) + self.listener.bind(("127.0.0.1", 0)) + self.listener.listen(1) + self.port = self.listener.getsockname()[1] + + def connect(self, prefix: tuple[str, ...]) -> int: + return subprocess.run( + [*prefix, sys.executable, "-c", CONNECT_PROBE, str(self.port)], + check=False, + capture_output=True, + timeout=120, + ).returncode + + def test_an_unsandboxed_child_reaches_the_listening_socket(self) -> None: + # The control. Without it, a sandboxed child that failed to start for + # some unrelated reason would read as proof of isolation. + self.assertEqual(self.connect(()), 0) + + def test_a_sandboxed_child_cannot_reach_the_listening_socket(self) -> None: + sandbox = lifecycle.network_sandbox_command() + if sandbox is None: + self.skipTest("this host offers no OS sandbox that denies a child the network") + self.assertEqual(self.connect(sandbox), 1) + + def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: + # ``env`` runs its argument unchanged: a prefix that contains nothing + # must not be mistaken for a boundary just because it launches. + passthrough = ("/usr/bin/env",) + if not os.access(passthrough[0], os.X_OK): # pragma: no cover - platform + self.skipTest("no pass-through launcher to test against") + self.assertFalse(lifecycle._sandbox_denies_network(passthrough)) + + +class ProviderLaunchTests(TemporaryWorkspace): + """What ``subprocess_indexer`` actually hands the operating system.""" + + def request(self) -> lifecycle.IndexRequest: + source = self.root / "source" + source.mkdir(exist_ok=True) + return lifecycle.IndexRequest( + source_root=source, + output_path=self.root / "graph.bin", + environment={"PATH": os.environ.get("PATH", "")}, + pin=PIN, + commit="a" * 40, + tree="b" * 40, + ) + + def launched_argv(self, executable: str, *, sandbox=("/sandbox", "--deny")) -> list[str]: + recorded: list[list[str]] = [] + + def fake_run(argv, **kwargs): + recorded.append(list(argv)) + return subprocess.CompletedProcess(argv, 0, b"", b"") + + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: sandbox): + indexer = lifecycle.subprocess_indexer(executable) + # Patched only around the launch, so the Git calls a build makes are + # never intercepted by this stand-in. + with mock.patch.object(subprocess, "run", fake_run): + indexer(self.request()) + return recorded[0] + + def test_the_provider_is_launched_inside_the_sandbox(self) -> None: + argv = self.launched_argv("graphify") + self.assertEqual(argv[:3], ["/sandbox", "--deny", "graphify"]) + + def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): + with self.assertRaises(ContextError): + lifecycle.subprocess_indexer("graphify") + + def test_a_relative_provider_path_binds_to_the_invocation_directory(self) -> None: + # The child runs in the materialized copy, so a relative path left + # unresolved would be looked up there instead of where it is installed. + installed = self.root / "venv" / "bin" + installed.mkdir(parents=True) + provider = installed / "graphify" + provider.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") + provider.chmod(0o700) + previous = Path.cwd() + os.chdir(self.root) + self.addCleanup(os.chdir, previous) + argv = self.launched_argv(os.path.join("venv", "bin", "graphify")) + self.assertEqual(argv[2], str(provider)) + + def test_a_bare_command_name_keeps_its_path_lookup(self) -> None: + self.assertEqual(lifecycle._resolved_executable("graphify"), "graphify") + + def test_an_unnamed_provider_is_refused(self) -> None: + with self.assertRaises(ContextError): + lifecycle._resolved_executable("") + + class BuildAndPublishTests(TemporaryWorkspace): def test_manifest_binds_every_required_fact(self) -> None: manifest = self.build() @@ -313,6 +432,30 @@ def test_pruning_keeps_only_the_published_generation(self) -> None: second = self.build() self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), [second.generation]) + def test_pruning_happens_before_the_build_lock_is_released(self) -> None: + """Pruning after unlocking can delete a concurrent builder's generation. + + A builder that released the lock, paused, and only then pruned would + remove whatever a second builder published in the meantime -- or that + builder's staging directory -- leaving ``current`` naming a directory + that no longer exists, with both builds reporting success. + """ + order: list[str] = [] + prune, release = lifecycle.GraphStateRoot.prune, lifecycle._BuildLock.__exit__ + + def record_prune(state, *, keep): + order.append("prune") + return prune(state, keep=keep) + + def record_release(lock, *exception): + order.append("unlock") + return release(lock, *exception) + + with mock.patch.object(lifecycle.GraphStateRoot, "prune", record_prune), \ + mock.patch.object(lifecycle._BuildLock, "__exit__", record_release): + self.build() + self.assertEqual(order, ["prune", "unlock"]) + def test_a_failed_build_publishes_nothing(self) -> None: first = self.build() @@ -496,9 +639,25 @@ def test_an_unconfigured_installation_skips_rather_than_fails(self) -> None: def test_a_healthy_build_passes(self) -> None: self.build() - report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) + # Isolation is a property of the host, not of this build; a host that + # offers a sandbox is the healthy case being described here. + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) self.assertEqual(report["status"], "pass") + def test_a_host_that_cannot_contain_a_provider_fails_doctor(self) -> None: + self.build() + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): + report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) + self.assertEqual(report["status"], "fail") + isolation = [check for check in report["checks"] if check["check"] == "context-graph-isolation"] + self.assertEqual([check["status"] for check in isolation], ["fail"]) + + def test_isolation_is_not_asked_about_when_nothing_is_pinned(self) -> None: + report = lifecycle.doctor_report(self.repository, pin=None, root=self.state) + isolation = [check for check in report["checks"] if check["check"] == "context-graph-isolation"] + self.assertEqual([check["status"] for check in isolation], ["skip"]) + def test_a_stale_graph_fails_doctor(self) -> None: self.build() (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") @@ -546,6 +705,10 @@ def base(self) -> list[str]: return ["--repo-path", str(self.repository), "--state-dir", str(self.state), "--json"] def test_build_status_refresh_remove_round_trip(self) -> None: + # The only test that launches a provider for real, so it is also the + # only one that needs the host to offer the sandbox a build requires. + if lifecycle.network_sandbox_command() is None: + self.skipTest("this host offers no OS sandbox that denies a child the network") pin, indexer = str(self.pin_file()), str(self.indexer_script()) code, output = self.run_command("build", *self.base(), "--pin-file", pin, "--indexer", indexer) self.assertEqual(code, 0, output) From b60a1afadca0867c3a5fc58057400834eb415b17 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 02:50:04 -0700 Subject: [PATCH 04/25] context: pin the sandbox classifier on hosts with no mechanism CI on ubuntu-latest skipped the two host-dependent isolation tests: no candidate passed the probe there, because the runner image restricts unprivileged user namespaces, which is what both unshare and bwrap need. That is the documented refusal working as intended, but it left the accept half of the decision unexercised on Linux. Stand-in launchers now pin both halves on every host -- a child that reports a denial is accepted, one that reached the network stack is rejected, one that cannot start is rejected -- alongside the real loopback proof that runs wherever the host offers a mechanism. The docs name the remedy for a restricted Linux host rather than leaving an operator to infer it. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 10 ++++++++++ tests/test_context_graph_lifecycle.py | 23 +++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 57b47c28..900f9c8c 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -86,6 +86,16 @@ pinned. Running an unconfined provider is not offered as a fallback: an operator who cannot contain a third-party indexer is better served by knowing it than by a build that quietly could have reached the network. +A Linux host that restricts unprivileged user namespaces — Ubuntu 24.04 and +GitHub's hosted runners among them — offers no mechanism by default, and both +`unshare` and `bwrap` fail there. Installing bubblewrap (`apt install +bubblewrap`), which ships an AppArmor profile permitting the namespaces it +needs, is the least invasive way to give such a host one. The alternative is to +lift the restriction system-wide +(`sysctl kernel.apparmor_restrict_unprivileged_userns=0`), which is a decision +about the whole machine rather than about this build, and not one this +repository makes on an operator's behalf. + Git itself runs with `GIT_CONFIG_NOSYSTEM`, `GIT_CONFIG_GLOBAL=/dev/null`, and `GIT_CONFIG_SYSTEM=/dev/null`: an untrusted checkout's local, global, or system configuration can otherwise install clean/smudge filters and hook paths that run diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 15fc8b55..b6b7d893 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -302,6 +302,29 @@ def test_a_sandboxed_child_cannot_reach_the_listening_socket(self) -> None: self.skipTest("this host offers no OS sandbox that denies a child the network") self.assertEqual(self.connect(sandbox), 1) + def launcher(self, exit_code: int) -> str: + """A stand-in launcher, so the classifier is pinned on every host. + + A host that offers no real mechanism -- a Linux host with unprivileged + user namespaces restricted, say -- would otherwise leave both halves of + the accept/reject decision untested. + """ + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + path = Path(directory.name) / "launcher" + path.write_text(f"#!/bin/sh\nexit {exit_code}\n", encoding="utf-8") + path.chmod(0o700) + return str(path) + + def test_a_child_that_reports_a_denial_is_accepted(self) -> None: + self.assertTrue(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_DENIED),))) + + def test_a_child_that_reached_the_network_stack_is_rejected(self) -> None: + self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(3),))) + + def test_a_launcher_that_cannot_start_is_rejected(self) -> None: + self.assertFalse(lifecycle._sandbox_denies_network(("/nonexistent/launcher",))) + def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: # ``env`` runs its argument unchanged: a prefix that contains nothing # must not be mistaken for a boundary just because it launches. From c4f6bfed7e07dba81806d2fd533c9a5f2584117f Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 03:05:43 -0700 Subject: [PATCH 05/25] context: close the provider interface, the Git boundary, and three lifecycle races Fix round for the `a011eb4` audit, all five findings. [P1] The launcher assumed `index --source ... --output ...`. The evaluated release takes `extract` plus options and writes its state beside the sources it was run over, so the adapter now runs that interface in the materialized copy and collects the state directory into one reproducible archive afterwards. [P1] Git children of a build are outside the provider's sandbox, and in a partial clone `ls-tree` and `cat-file` can fetch missing objects from a remote mid-build. Both invocation paths now share one environment with lazy fetching off and an empty transport allowlist, `protocol.allow=never` travels on the command line because that is the only level above the checkout's own config, and a partial clone is refused outright rather than read one blob at a time. [P2] Completeness is read from the provider's own report, never from its exit status. Requeued, pending, or failed entries make a build partial, and so does a run that left no readable report: absent evidence is not evidence. [P2] `remove` takes the build lock, so it can no longer delete a running build's sources, output, and generations. The lock moved beside the state directory so it survives the removal it serializes -- a lock inside the deleted tree would be unlinked mid-removal and the next builder would hold a new inode. [P2] Readers take no lock, so a refresh can publish and prune between a reader's pointer read and its validation. A failing verdict is confirmed against the pointer before it is returned, and a pointer that moved is read again, so a healthy refresh no longer surfaces as `invalid` or `corrupt`. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 66 ++++ src/code_mower/context_graph_lifecycle.py | 350 +++++++++++++++++++--- tests/test_context_graph_lifecycle.py | 280 ++++++++++++++++- 3 files changed, 648 insertions(+), 48 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 900f9c8c..1e6d6731 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -63,6 +63,23 @@ Publication and pruning happen inside one locked critical section. Two builders that race are serialized, and neither can delete the generation the other just published while `current` still names it. +`remove` takes the same lock. A removal running beside a build would otherwise +delete its materialized sources, its output, and the generations directory, and +the builder would then either fail or recreate state that `remove` had already +reported as gone. The lock file lives *beside* the state directory rather than +inside it, so it survives the removal it serializes: a lock inside the deleted +tree would be unlinked mid-removal, and the next builder would create a new +inode and hold a lock nobody else was waiting on. What is left behind is an +empty 0600 file carrying nothing. + +Readers take no lock at all, so a refresh can publish and prune between the +moment `graph_status()` reads the `current` pointer and the moment it finishes +validating what that pointer named. A failing verdict is therefore confirmed +against the pointer before it is returned, and a pointer that moved is read +again — otherwise a healthy refresh would surface as `invalid` or `corrupt`. A +verdict about the generation `current` still names is returned as it stands; the +retry is for a moved pointer, not a poll. + ## The network boundary An environment variable is a request, not a boundary: `NO_PROXY=*` asks a @@ -101,6 +118,21 @@ Git itself runs with `GIT_CONFIG_NOSYSTEM`, `GIT_CONFIG_GLOBAL=/dev/null`, and configuration can otherwise install clean/smudge filters and hook paths that run code during what looks like a read. +Git children are not inside the provider's sandbox — they are children of Code +Mower itself — so the boundary has to reach them separately. Both invocation +paths, the census reader and the blob materializer, share one environment: +`GIT_NO_LAZY_FETCH=1`, and `GIT_ALLOW_PROTOCOL` set but empty, which git reads +as the complete list of permitted transports. `protocol.allow=never` travels on +the command line because that is the only level that outranks the repository's +own `.git/config`, which belongs to the untrusted checkout and is always read. + +That still leaves the repository *shape* that makes a read reach out at all, so +**a build refuses a partial clone outright**. Where `extensions.partialclone` or +a promisor remote is configured, `ls-tree` and `cat-file` can fetch a missing +object from a remote mid-build. There is no bounded way to prove in advance +which objects are present locally, so the build declines the checkout rather +than discovering the gap one blob at a time. Use a full clone. + ## What a manifest binds Every published generation carries, in `manifest.json`: @@ -119,6 +151,40 @@ Every published generation carries, in `manifest.json`: `shareable_summary()` is the metadata-only view: revisions, digests, counts and states. It carries no indexed content, no provider output, and no local path. +## How the provider is actually invoked + +`subprocess_indexer()` builds the argv for the interface the adopt decision +evaluated, not a conventional-looking one: `extract` plus the pinned options, +run with its working directory set to the materialized copy. The evaluated +release takes no `--source`/`--output` pair — `extract` reads the directory it +is run in and writes its state beside those sources, which the clean-room run in +[the evaluation](graphify-evaluation.md) recorded as +`extract --code-only --no-cluster --max-workers 4`. + +So the adapter collects an artifact afterwards rather than naming one up front. +The state directory the provider wrote (`.graphify` or `.graph`, both already on +the excluded-roots list) is packed into a single reproducible archive: names +sorted, timestamps and ownership fixed, modes normalized, symlinks dropped. Two +builds of one commit have to produce identical bytes, because the manifest binds +a digest of them. That state lands inside the throwaway materialized copy, never +inside the indexed checkout, and the copy is deleted when the build ends. + +**Completeness is read from the provider's report, never from its exit status.** +The adapter parses the report the provider leaves in that state directory and +marks the build `partial` if it admits requeued, pending, or failed entries, or +denies completion outright. A run that left no readable report is `partial` too: +absent evidence is not evidence of a complete build, and `partial` is the state +`graph_status` refuses by default, so the failure is one an operator can see and +act on. This is the direct consequence of the requeue defect the evaluation +recorded — a repeat that exits zero in 1.63 seconds having requeued 54 entries +has not built a complete graph. + +The subcommand, the state-directory names, and the report counters are constants +in one place in `context_graph_lifecycle.py`. They encode the interface as the +evaluation recorded it; the first installation against a real pinned release +should confirm them against that install and correct them here if they have +moved. + ## Refresh is explicit `build` is the first-time verb and refuses when a usable generation already diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index c508ef98..7b219967 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -41,9 +41,11 @@ import json import os import re +import io import shutil import subprocess import sys +import tarfile import uuid from dataclasses import dataclass from datetime import datetime, timezone @@ -310,13 +312,16 @@ def _census_digest(entries: Iterable[TrackedEntry]) -> str: return census.hexdigest() -def _git(repository: Path, *arguments: str, capture: bool = True) -> str: - """Run git with repository configuration disarmed. +def git_environment() -> dict[str, str]: + """The environment every Git child of a build runs in. - A build reads an untrusted checkout. Local, global, and system - configuration can install clean/smudge filters, alternate object stores, - and hook paths, any of which would run code or reach outside the - repository during what looks like a read. This drops all three. + One definition for both invocation paths -- the census reader and the blob + materializer -- because a boundary that only half the children observe is + not a boundary. Beyond the scrubbing an indexer gets, this denies Git the + two ways a *read* can reach the network: ``GIT_NO_LAZY_FETCH`` stops a + partial clone fetching a missing object mid-read, and an empty + ``GIT_ALLOW_PROTOCOL`` leaves no transport on the allowlist, so a fetch + that was somehow attempted anyway has nothing to attempt it over. """ environment = dict(_NETWORK_DENY) for name in _ENVIRONMENT_ALLOWLIST: @@ -328,20 +333,84 @@ def _git(repository: Path, *arguments: str, capture: bool = True) -> str: GIT_CONFIG_SYSTEM=os.devnull, GIT_ATTR_NOSYSTEM="1", GIT_OPTIONAL_LOCKS="0", + GIT_NO_LAZY_FETCH="1", + # An empty allowlist, not an absent one: git treats the variable as the + # complete set of permitted transports, so "" permits none. + GIT_ALLOW_PROTOCOL="", + GIT_PROTOCOL_FROM_USER="0", + GIT_TERMINAL_PROMPT="0", + GIT_SSH_COMMAND="/usr/bin/false", ) + return environment + + +#: Overrides passed on the command line because that is the only level that +#: outranks the repository's own ``.git/config``. System and global +#: configuration are dropped by the environment above, but local configuration +#: belongs to the untrusted checkout and is always read. +_GIT_SAFETY_OPTIONS: tuple[str, ...] = ( + "-c", "protocol.allow=never", + "-c", "core.fsmonitor=false", + "-c", "fetch.recurseSubmodules=no", + "-c", "uploadpack.allowFilter=false", +) + +#: Local configuration that means "objects may be missing and fetched on +#: demand". A build refuses such a checkout outright rather than relying on +#: ``GIT_NO_LAZY_FETCH``, which older Git releases do not honour. +_PARTIAL_CLONE_KEYS = r"^(extensions\.partialclone|remote\..*\.(promisor|partialclonefilter))$" + + +def _git(repository: Path, *arguments: str, capture: bool = True, permit_failure: bool = False) -> str: + """Run git with repository configuration disarmed and no way out to a network. + + A build reads an untrusted checkout. Local, global, and system + configuration can install clean/smudge filters, alternate object stores, + and hook paths, any of which would run code or reach outside the + repository during what looks like a read. This drops all three, and + ``git_environment`` closes the transports a read could otherwise use. + """ try: completed = subprocess.run( - ["git", "-C", str(repository), "--no-optional-locks", *arguments], - check=True, + ["git", "-C", str(repository), "--no-optional-locks", *_GIT_SAFETY_OPTIONS, *arguments], + check=not permit_failure, capture_output=capture, text=True, - env=environment, + env=git_environment(), ) except (OSError, UnicodeError, subprocess.SubprocessError): raise ContextError("local graph build could not read the target repository") from None + if permit_failure and completed.returncode != 0: + return "" return completed.stdout +def refuse_lazy_object_fetch(repository: Path) -> None: + """Refuse a partial clone, where reading the tree can call out to a remote. + + ``ls-tree`` and ``cat-file`` look like pure local reads, and in a full + clone they are. In a partial clone a missing object is fetched from the + promisor remote on demand -- during the build, over a transport the + repository configured, outside the sandbox the provider runs in. There is + no bounded way to prove ahead of time which objects are present, so the + build declines the whole repository shape instead. + """ + declared = _git( + repository, + "config", + "--local", + "--name-only", + "--get-regexp", + _PARTIAL_CLONE_KEYS, + permit_failure=True, + ) + if declared.strip(): + raise ContextError( + "local graph builds refuse a partial clone: reading its tree can fetch objects " + "from a remote during the build; use a full clone of this checkout" + ) + + def resolve_revision(repository: Path, revision: str = "HEAD") -> tuple[str, str]: """Return the full ``(commit, tree)`` names a build would bind. @@ -366,6 +435,7 @@ def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: uncommitted edit, an untracked scratch file, and an ignored secret are all invisible here by construction rather than by filtering. """ + refuse_lazy_object_fetch(repository) listing = _git( repository, "ls-tree", @@ -437,23 +507,19 @@ def materialize_tracked_files(repository: Path, census: TrackedCensus, destinati write. Blob content comes from ``git cat-file --batch`` in one child process rather than one per file. """ + refuse_lazy_object_fetch(repository) if destination.exists(): raise ContextError("local graph materialization requires a fresh private directory") destination.mkdir(mode=0o700, parents=True) if not census.entries: return 0 - environment = dict(_NETWORK_DENY) - for name in _ENVIRONMENT_ALLOWLIST: - if name in os.environ: - environment[name] = os.environ[name] - environment.update(GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, GIT_CONFIG_SYSTEM=os.devnull) written = 0 process = subprocess.Popen( - ["git", "-C", str(repository), "cat-file", "--batch"], + ["git", "-C", str(repository), "--no-optional-locks", *_GIT_SAFETY_OPTIONS, "cat-file", "--batch"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, - env=environment, + env=git_environment(), ) try: assert process.stdin is not None and process.stdout is not None @@ -556,6 +622,117 @@ def _resolved_executable(executable: str) -> str: return executable +#: The subcommand the evaluated release exposes, recorded in +#: ``docs/graphify-evaluation.md``: the clean-room run indexed with +#: ``extract --code-only --no-cluster --max-workers 4``. There is no +#: ``--source``/``--output`` pair to hand it; ``extract`` reads the directory +#: it is run in and writes its state beside those sources, which is why the +#: child's working directory is the materialized copy and why the adapter +#: collects an artifact afterwards rather than naming one up front. +_PROVIDER_EXTRACT = "extract" + +#: Where that state lands. Both names appear in the evaluation's excluded-roots +#: list; the adapter accepts whichever the installed release writes and refuses +#: a build that produced neither. +_PROVIDER_STATE_DIRECTORIES = (".graphify", ".graph") + +#: The provider's own record of what it processed. Completeness is read from +#: here, never inferred from an exit status: the clean-room run recorded 54 +#: manifest entries requeued by a repeat that exited zero in 1.63 s. +_PROVIDER_REPORT_NAMES = ("manifest.json", "index.json", "report.json") + +#: Counters whose presence above zero means the provider did not finish. Any +#: one of them, not all: a report that admits requeued entries is a partial +#: build however healthy the rest of it looks. +_INCOMPLETE_COUNTERS = ("requeued", "pending", "failed", "errors", "incomplete") + +#: Where the provider reports how many files it actually indexed. +_INDEXED_COUNTERS = ("indexed_files", "code_files", "files", "entries") + + +def _provider_state_directory(source_root: Path) -> Path: + for name in _PROVIDER_STATE_DIRECTORIES: + candidate = source_root / name + if candidate.is_dir() and not candidate.is_symlink(): + return candidate + raise ContextError("local graph provider wrote no index state; no generation was published") + + +def _provider_report(state_directory: Path) -> Mapping[str, Any] | None: + """The provider's completion evidence, or ``None`` if it left none.""" + for name in _PROVIDER_REPORT_NAMES: + path = state_directory / name + if not path.is_file() or path.is_symlink(): + continue + try: + payload = json.loads(path.read_bytes()[: MAX_MANIFEST_BYTES + 1]) + except (OSError, ValueError): + return None + return payload if isinstance(payload, Mapping) else None + return None + + +def _read_completeness(report: Mapping[str, Any] | None) -> IndexResult: + """Classify a provider run from its own report, defaulting to partial. + + Absent or unreadable evidence is *not* evidence of a complete build. The + provider owns no provenance (the evaluation records this as the first + product constraint), so a build that cannot read a completion claim + publishes a generation marked ``partial``, which ``graph_status`` refuses + by default. That is the failure an operator can act on; silently calling + it complete is the one they cannot. + """ + if report is None: + return IndexResult(completeness=PARTIAL, notes=("provider left no readable completion report",)) + notes: list[str] = [] + for counter in _INCOMPLETE_COUNTERS: + value = report.get(counter) + if value is True: + notes.append(f"provider reported {counter}") + elif isinstance(value, int) and not isinstance(value, bool) and value > 0: + notes.append(f"provider reported {value} {counter}") + if report.get("complete") is False: + notes.append("provider reported the extraction as incomplete") + indexed = 0 + for counter in _INDEXED_COUNTERS: + value = report.get(counter) + if isinstance(value, int) and not isinstance(value, bool) and value >= 0: + indexed = value + break + if notes: + return IndexResult(completeness=PARTIAL, indexed_files=indexed, notes=tuple(notes)) + return IndexResult(completeness=COMPLETE, indexed_files=indexed) + + +def _pack_state(state_directory: Path) -> bytes: + """Collect the provider's state into one reproducible artifact. + + Names sorted, timestamps and ownership fixed, modes normalized: two builds + of the same commit must produce the same bytes, because the manifest binds + a digest of them. Only regular files are taken -- a symlink in provider + state would name a target outside the artifact, which an immutable + generation cannot carry. + """ + buffer = io.BytesIO() + total = 0 + with tarfile.open(fileobj=buffer, mode="w", format=tarfile.PAX_FORMAT) as archive: + for path in sorted(state_directory.rglob("*"), key=lambda item: str(item.relative_to(state_directory))): + if path.is_symlink() or not path.is_file(): + continue + total += path.stat().st_size + if total > MAX_ARTIFACT_BYTES: + raise ContextError("local graph artifact exceeds its budget; no generation was published") + info = tarfile.TarInfo(str(path.relative_to(state_directory))) + info.size = path.stat().st_size + info.mtime = 0 + info.mode = 0o600 + info.uid = info.gid = 0 + info.uname = info.gname = "" + with path.open("rb") as stream: + archive.addfile(info, stream) + return buffer.getvalue() + + def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult]: """Run a pinned provider CLI over the materialized copy, without a network. @@ -566,6 +743,11 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] it from inside a sandbox that denies it sockets. Resolving the executable and the sandbox here, rather than at build time, means an unusable provider or an uncontainable host fails before a single blob is materialized. + + The argv is the interface the adopt decision evaluated, not a guess at a + conventional one: ``extract`` with the pinned options, in the materialized + copy. Everything the provider leaves behind is then collected and + classified from its own report. """ command = _resolved_executable(executable) sandbox = network_sandbox_command() @@ -578,16 +760,7 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] def run(request: IndexRequest) -> IndexResult: try: completed = subprocess.run( - [ - *sandbox, - command, - "index", - "--source", - str(request.source_root), - "--output", - str(request.output_path), - *request.pin.options, - ], + [*sandbox, command, _PROVIDER_EXTRACT, *request.pin.options], check=False, capture_output=True, text=False, @@ -600,7 +773,10 @@ def run(request: IndexRequest) -> IndexResult: if completed.returncode != 0: # Provider stderr can echo indexed source; it is never surfaced. raise ContextError("local graph provider failed; no generation was published") - return IndexResult(completeness=COMPLETE) + state_directory = _provider_state_directory(request.source_root) + result = _read_completeness(_provider_report(state_directory)) + _write_private_file(request.output_path, _pack_state(state_directory)) + return result return run @@ -759,6 +935,18 @@ def __init__(self, repository: Path, *, root: Path | None = None): def generations_path(self) -> Path: return self.path / "generations" + @property + def lock_path(self) -> Path: + """Beside the state directory, deliberately not inside it. + + ``remove`` deletes the whole tree while holding this lock. A lock file + inside that tree would be unlinked mid-removal, and the next builder + would create a *new* inode and acquire a lock nobody else is holding -- + two processes, two files, no mutual exclusion. Keeping it one level up + means the inode a holder waits on is the inode the remover holds. + """ + return self.path.parent / f"{self.workspace}.lock" + @property def _chain(self) -> tuple[Path, ...]: """Every directory this class owns, outermost first. @@ -783,15 +971,35 @@ def verify_private(self, *, create: bool = False) -> None: def ensure(self) -> None: """Create the private tree, refusing to place state inside a repository.""" + self._refuse_state_inside_a_repository() + self.verify_private(create=True) + + def _refuse_state_inside_a_repository(self) -> None: if any((parent / ".git").exists() for parent in (self.path, *self.path.parents)): raise ContextError("local graph state must stay outside Git repositories") - self.verify_private(create=True) + + def _ensure_lock_directory(self) -> None: + """Create only what the lock file needs, not the generations tree. + + ``remove`` takes the same lock, and a removal that first created the + state it was asked to delete would report success for a tree it made + itself. + """ + self._refuse_state_inside_a_repository() + for directory in self._chain[:2]: + os.close(_open_private_directory(directory, create=True)) def lock(self): - """Serialize builds for one checkout; concurrent ones would race publish.""" - self.ensure() - lock_path = self.path / "build.lock" - handle = os.open(lock_path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600) + """Serialize builds *and removals* for one checkout. + + Concurrent builds would race publish; a removal running beside a build + would delete the sources, output, and generations out from under it. + Both take this lock, so the whole set of lifecycle operations that + mutate state for one checkout is serialized rather than just the pair + that was obviously racy. + """ + self._ensure_lock_directory() + handle = os.open(self.lock_path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600) stream = os.fdopen(handle, "a+", encoding="utf-8") try: _private(stream.fileno()) @@ -915,14 +1123,32 @@ def prune(self, *, keep: str | None) -> list[str]: return removed def remove_all(self) -> bool: - """Delete every trace of this checkout's graph state.""" - if not self.path.exists(): + """Delete this checkout's graph state, serialized against builders. + + Taken under the build lock: without it, a removal can delete a running + build's materialized sources, its output, and the generations + directory, after which the builder either fails or recreates state + that ``remove`` has already reported as gone. + + The lock file itself survives, by design. It is an empty 0600 file + outside the deleted tree that carries no indexed content, and it is the + stable inode the next builder and the next remover agree on. + """ + if not self.path.exists() and not self.lock_path.exists(): + # Nothing exists and no builder can be running: a builder creates + # the lock file before it creates any state, so an absent lock file + # means there is nothing to serialize against. Checked first so a + # removal on a fresh install does not create a private tree merely + # to report that it was empty. return False - # Refuse to delete a tree that is not ours; a loosened or foreign - # directory is reported, not recursively removed. - self.verify_private() - shutil.rmtree(self.path) - return True + with self.lock(): + if not self.path.exists(): + return False + # Refuse to delete a tree that is not ours; a loosened or foreign + # directory is reported, not recursively removed. + self.verify_private() + shutil.rmtree(self.path) + return True class _BuildLock: @@ -1015,6 +1241,11 @@ def build_graph( commit, tree = resolve_revision(repository, revision) census = read_tracked_census(repository, commit) with state.lock(): + # The lock only creates what the lock file needs, so that ``remove`` + # can take it without materializing the tree it was asked to delete. + # A build does want the whole private tree, created 0700 at every + # level before anything is written into it. + state.ensure() build_root = state.path / ("." + uuid.uuid4().hex + ".build") build_root.mkdir(mode=0o700, parents=True) try: @@ -1071,6 +1302,13 @@ def build_graph( return published +#: How many times a reader will re-read a generation that was replaced under +#: it. Bounded because the only thing that moves the pointer is a publish, and +#: a refresh loop fast enough to outrun three reads is not a state worth +#: blocking on. +_STATUS_ATTEMPTS = 3 + + def graph_status( repository: Path, *, @@ -1084,8 +1322,40 @@ def graph_status( state, and every non-``current`` state is unusable. Nothing falls back to a previous generation: a consumer that cannot have the revision it asked for is told so rather than handed an older answer that looks fresh. + + Readers take no lock, so a refresh can publish and prune between the moment + this reads the ``current`` pointer and the moment it validates what that + pointer named. The generation is then genuinely gone, and reporting it + ``invalid`` or ``corrupt`` would describe a directory a healthy build had + just superseded rather than anything wrong with the graph. A failing read + is therefore confirmed against the pointer before it is returned, and a + pointer that moved is read again. """ state = GraphStateRoot(repository, root=root) + for attempt in range(_STATUS_ATTEMPTS): + status = _status_once( + state, repository, revision=revision, require_complete=require_complete + ) + if status.usable or status.generation is None or attempt == _STATUS_ATTEMPTS - 1: + return status + try: + if state.current_generation() == status.generation: + # The pointer still names what was just validated, so the + # verdict is about the operator's graph, not a race. + return status + except ContextError: + return status + return status + + +def _status_once( + state: GraphStateRoot, + repository: Path, + *, + revision: str, + require_complete: bool, +) -> GenerationStatus: + """One validation pass over whichever generation ``current`` names now.""" try: if not state.path.exists(): return GenerationStatus(state="absent", detail="no local graph has been built for this checkout") @@ -1260,12 +1530,14 @@ def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[ "TrackedEntry", "build_graph", "doctor_report", + "git_environment", "graph_status", "iter_generations", "load_manifest", "load_pin", "materialize_tracked_files", "read_tracked_census", + "refuse_lazy_object_fetch", "remove_graph", "render_status_text", "resolve_revision", diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index b6b7d893..1da6120f 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -16,12 +16,14 @@ from __future__ import annotations import hashlib +import io import json import os import socket import stat import subprocess import sys +import tarfile import tempfile import unittest from datetime import datetime, timezone @@ -334,6 +336,11 @@ def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: self.assertFalse(lifecycle._sandbox_denies_network(passthrough)) +#: What a provider that finished leaves behind: a completion claim and counts +#: that admit nothing outstanding. +FINISHED_REPORT = {"complete": True, "code_files": 3, "requeued": 0} + + class ProviderLaunchTests(TemporaryWorkspace): """What ``subprocess_indexer`` actually hands the operating system.""" @@ -349,11 +356,31 @@ def request(self) -> lifecycle.IndexRequest: tree="b" * 40, ) - def launched_argv(self, executable: str, *, sandbox=("/sandbox", "--deny")) -> list[str]: + def run_indexer( + self, + executable: str, + *, + sandbox=("/sandbox", "--deny"), + report: object = FINISHED_REPORT, + state_directory: str = ".graphify", + ) -> tuple[list[str], lifecycle.IndexResult]: + """Launch the adapter with the provider's side of the contract faked. + + ``extract`` writes its state beside the sources it was run over, so the + stand-in has to leave that state behind for the adapter to collect -- + an exit status alone is not a finished build. + """ + request = self.request() recorded: list[list[str]] = [] def fake_run(argv, **kwargs): recorded.append(list(argv)) + if state_directory: + written = request.source_root / state_directory + written.mkdir(exist_ok=True) + (written / "graph.bin").write_bytes(b"graph-bytes") + if report is not None: + (written / "manifest.json").write_text(json.dumps(report), encoding="utf-8") return subprocess.CompletedProcess(argv, 0, b"", b"") with mock.patch.object(lifecycle, "network_sandbox_command", lambda: sandbox): @@ -361,13 +388,82 @@ def fake_run(argv, **kwargs): # Patched only around the launch, so the Git calls a build makes are # never intercepted by this stand-in. with mock.patch.object(subprocess, "run", fake_run): - indexer(self.request()) - return recorded[0] + result = indexer(request) + return recorded[0], result + + def launched_argv(self, executable: str, *, sandbox=("/sandbox", "--deny")) -> list[str]: + return self.run_indexer(executable, sandbox=sandbox)[0] def test_the_provider_is_launched_inside_the_sandbox(self) -> None: argv = self.launched_argv("graphify") self.assertEqual(argv[:3], ["/sandbox", "--deny", "graphify"]) + def test_the_provider_is_invoked_through_its_documented_extract_interface(self) -> None: + # The interface the adopt decision evaluated, recorded in + # docs/graphify-evaluation.md as ``extract`` plus options. An + # ``index --source ... --output ...`` shape would be a different CLI. + argv = self.launched_argv("graphify") + self.assertEqual(argv[3:], ["extract", *PIN.options]) + self.assertNotIn("--source", argv) + self.assertNotIn("--output", argv) + + def test_the_collected_artifact_holds_the_state_the_provider_wrote(self) -> None: + _, result = self.run_indexer("graphify") + self.assertEqual(result.completeness, lifecycle.COMPLETE) + self.assertEqual(result.indexed_files, 3) + names = self.archived_names((self.root / "graph.bin").read_bytes()) + self.assertIn("graph.bin", names) + + def archived_names(self, artifact: bytes) -> list[str]: + with tarfile.open(fileobj=io.BytesIO(artifact), mode="r") as archive: + return archive.getnames() + + def test_collecting_the_same_state_twice_produces_the_same_bytes(self) -> None: + # The manifest binds a digest of the artifact, so two builds of one + # commit have to agree on the bytes down to the archive metadata. + self.run_indexer("graphify") + first = (self.root / "graph.bin").read_bytes() + (self.root / "graph.bin").unlink() + self.run_indexer("graphify") + self.assertEqual(first, (self.root / "graph.bin").read_bytes()) + + def test_a_provider_that_wrote_no_state_publishes_nothing(self) -> None: + with self.assertRaises(ContextError): + self.run_indexer("graphify", state_directory="") + + def test_a_successful_run_with_requeued_entries_is_partial(self) -> None: + # The defect the clean-room run recorded: a repeat that exits zero in + # 1.63 s having requeued 54 entries has not built a complete graph. + _, result = self.run_indexer("graphify", report={"complete": True, "files": 429, "requeued": 54}) + self.assertEqual(result.completeness, lifecycle.PARTIAL) + self.assertIn("54 requeued", " ".join(result.notes)) + + def test_a_provider_that_denies_completion_is_partial(self) -> None: + _, result = self.run_indexer("graphify", report={"complete": False, "files": 10}) + self.assertEqual(result.completeness, lifecycle.PARTIAL) + + def test_a_run_that_left_no_report_is_partial_rather_than_complete(self) -> None: + # Exit status zero is not completion evidence. Absent evidence resolves + # to the state ``graph_status`` refuses, not the one it accepts. + _, result = self.run_indexer("graphify", report=None) + self.assertEqual(result.completeness, lifecycle.PARTIAL) + + def test_an_unparseable_report_is_partial_rather_than_complete(self) -> None: + request = self.request() + + def fake_run(argv, **kwargs): + written = request.source_root / ".graph" + written.mkdir(exist_ok=True) + (written / "graph.bin").write_bytes(b"graph-bytes") + (written / "manifest.json").write_bytes(b"{not json") + return subprocess.CompletedProcess(argv, 0, b"", b"") + + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + indexer = lifecycle.subprocess_indexer("graphify") + with mock.patch.object(subprocess, "run", fake_run): + result = indexer(request) + self.assertEqual(result.completeness, lifecycle.PARTIAL) + def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): with self.assertRaises(ContextError): @@ -535,6 +631,47 @@ def test_a_new_commit_makes_the_graph_stale(self) -> None: self.assertEqual(status.state, "stale") self.assertFalse(status.usable) + def test_a_generation_pruned_mid_read_is_read_again(self) -> None: + # Readers take no lock, so a refresh can publish and prune between the + # pointer read and the validation of what it named. The failure that + # produces is about a directory a healthy build superseded, not about + # the graph the operator has. + first = self.build() + second = self.build(indexer=recording_indexer(b"second-graph")) + raced = lifecycle.GenerationStatus( + state="invalid", + generation=first.generation, + detail="local graph generation is missing its manifest", + ) + real, attempts = lifecycle._status_once, [] + + def once(state, repository, **keywords): + attempts.append(1) + return raced if len(attempts) == 1 else real(state, repository, **keywords) + + with mock.patch.object(lifecycle, "_status_once", once): + status = lifecycle.graph_status(self.repository, root=self.state) + self.assertEqual(len(attempts), 2) + self.assertTrue(status.usable) + self.assertEqual(status.generation, second.generation) + + def test_a_verdict_about_the_published_generation_is_not_retried(self) -> None: + # The retry exists for a moved pointer only. A genuinely corrupt + # current generation is reported on the first read, not polled. + manifest = self.build() + artifact = lifecycle.GraphStateRoot(self.repository, root=self.state).artifact_path(manifest.generation) + artifact.write_bytes(b"tampered!!!") + real, attempts = lifecycle._status_once, [] + + def once(state, repository, **keywords): + attempts.append(1) + return real(state, repository, **keywords) + + with mock.patch.object(lifecycle, "_status_once", once): + status = lifecycle.graph_status(self.repository, root=self.state) + self.assertEqual(status.state, "corrupt") + self.assertEqual(len(attempts), 1) + def test_a_tampered_artifact_is_corrupt(self) -> None: manifest = self.build() artifact = lifecycle.GraphStateRoot(self.repository, root=self.state).artifact_path(manifest.generation) @@ -631,7 +768,110 @@ def test_a_group_readable_artifact_is_invalid(self) -> None: artifact.chmod(0o600) +class GitBoundaryTests(TemporaryWorkspace): + """The other half of the offline boundary: Git's own reads. + + The provider runs inside a sandbox, but the census reader and the blob + materializer are Git children of this process, outside it. In a partial + clone their reads can fetch missing objects from a remote, so the boundary + has to cover them too. + """ + + def commit(self) -> str: + return lifecycle.resolve_revision(self.repository)[0] + + def test_git_children_get_no_lazy_fetch_and_no_transport(self) -> None: + environment = lifecycle.git_environment() + self.assertEqual(environment["GIT_NO_LAZY_FETCH"], "1") + # Set but empty: git reads the variable as the complete list of + # permitted transports, and an empty list permits none. + self.assertEqual(environment["GIT_ALLOW_PROTOCOL"], "") + self.assertEqual(environment["GIT_CONFIG_NOSYSTEM"], "1") + self.assertEqual(environment["GIT_CONFIG_GLOBAL"], os.devnull) + self.assertEqual(environment["GIT_CONFIG_SYSTEM"], os.devnull) + + def test_git_children_inherit_no_ambient_secret(self) -> None: + with mock.patch.dict(os.environ, {"AWS_SECRET_ACCESS_KEY": "not-a-real-secret"}): + self.assertNotIn("AWS_SECRET_ACCESS_KEY", lifecycle.git_environment()) + + def test_the_transport_denial_outranks_repository_local_configuration(self) -> None: + # Local configuration belongs to the untrusted checkout and is always + # read, so the denial has to travel on the command line, which is the + # only level above it. + self.assertIn("protocol.allow=never", lifecycle._GIT_SAFETY_OPTIONS) + + def test_a_full_clone_is_read_without_complaint(self) -> None: + lifecycle.refuse_lazy_object_fetch(self.repository) + self.assertTrue(lifecycle.read_tracked_census(self.repository, self.commit()).entries) + + def test_a_partial_clone_is_refused_before_its_tree_is_read(self) -> None: + census = lifecycle.read_tracked_census(self.repository, self.commit()) + git(self.repository, "config", "--local", "remote.origin.promisor", "true") + with self.assertRaises(ContextError): + lifecycle.read_tracked_census(self.repository, self.commit()) + with self.assertRaises(ContextError): + lifecycle.materialize_tracked_files(self.repository, census, self.root / "fresh") + self.assertFalse((self.root / "fresh").exists()) + + def test_a_partial_clone_build_publishes_nothing(self) -> None: + git(self.repository, "config", "--local", "remote.origin.partialclonefilter", "blob:none") + with self.assertRaises(ContextError): + self.build() + self.assertEqual(lifecycle.graph_status(self.repository, root=self.state).state, "absent") + + def test_the_partial_clone_extension_is_refused_too(self) -> None: + # The other shape it takes: a repository-format extension, with the + # version bump that makes git accept one. + git(self.repository, "config", "--local", "core.repositoryformatversion", "1") + git(self.repository, "config", "--local", "extensions.partialclone", "origin") + with self.assertRaises(ContextError): + lifecycle.refuse_lazy_object_fetch(self.repository) + + class RemoveTests(TemporaryWorkspace): + def test_remove_takes_the_build_lock(self) -> None: + # A removal running beside a build deletes its sources, its output and + # its generations; the builder then either fails or recreates state + # that ``remove`` has already reported as gone. + self.build() + taken: list[str] = [] + lock = lifecycle.GraphStateRoot.lock + + def record_lock(state): + taken.append("lock") + return lock(state) + + with mock.patch.object(lifecycle.GraphStateRoot, "lock", record_lock): + self.assertTrue(lifecycle.remove_graph(self.repository, root=self.state)) + self.assertEqual(taken, ["lock"]) + + def test_the_lock_survives_the_removal_it_serializes(self) -> None: + # The inode a waiting builder is blocked on must still be there when + # the remover lets go of it. A lock file inside the deleted tree would + # be unlinked mid-removal and the next builder would lock a new one. + self.build() + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + self.assertFalse(state.lock_path.is_relative_to(state.path)) + before = state.lock_path.stat().st_ino + self.assertTrue(lifecycle.remove_graph(self.repository, root=self.state)) + self.assertTrue(state.lock_path.exists()) + self.assertEqual(state.lock_path.stat().st_ino, before) + + def test_the_retained_lock_carries_nothing_and_stays_private(self) -> None: + self.build() + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + lifecycle.remove_graph(self.repository, root=self.state) + self.assertEqual(state.lock_path.read_bytes(), b"") + self.assertEqual(stat.S_IMODE(state.lock_path.stat().st_mode), 0o600) + + def test_removing_nothing_creates_nothing(self) -> None: + # A removal on an installation that never opted in must not bring a + # private state tree into existence just to report that it is empty. + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + self.assertFalse(lifecycle.remove_graph(self.repository, root=self.state)) + self.assertFalse(state.path.exists()) + self.assertFalse(state.lock_path.exists()) + def test_remove_deletes_every_generation(self) -> None: self.build() state = lifecycle.GraphStateRoot(self.repository, root=self.state) @@ -701,15 +941,21 @@ def pin_file(self) -> Path: path.write_text(json.dumps(PIN.as_metadata()), encoding="utf-8") return path - def indexer_script(self) -> Path: - """A stand-in for a pinned provider CLI, so no package is required.""" + def indexer_script(self, *, complete: bool = True) -> Path: + """A stand-in for a pinned provider CLI, so no package is required. + + It answers to ``extract`` and writes its state into the directory it + was run in, which is the contract ``docs/graphify-evaluation.md`` + records for the evaluated release. + """ path = self.root / "fake-indexer" path.write_text( "#!/bin/sh\n" - 'while [ "$#" -gt 0 ]; do\n' - ' case "$1" in --output) shift; printf graph-bytes > "$1" ;; esac\n' - " shift\n" - "done\n", + '[ "$1" = "extract" ] || exit 64\n' + "mkdir -p .graphify\n" + "printf graph-bytes > .graphify/graph.bin\n" + 'printf \'{"complete": %s, "code_files": 1, "requeued": 0}\' ' + f"'{'true' if complete else 'false'}' > .graphify/manifest.json\n", encoding="utf-8", ) path.chmod(0o700) @@ -756,6 +1002,22 @@ def test_build_status_refresh_remove_round_trip(self) -> None: self.assertEqual(code, 1) self.assertEqual(json.loads(output)["state"], "absent") + def test_a_provider_that_admits_an_incomplete_run_is_not_usable(self) -> None: + # End to end through the real launcher: the provider exits zero and + # writes state, and the build is still refused because its own report + # denies completion. Exit status is not completion evidence. + if lifecycle.network_sandbox_command() is None: + self.skipTest("this host offers no OS sandbox that denies a child the network") + code, output = self.run_command( + "build", *self.base(), + "--pin-file", str(self.pin_file()), + "--indexer", str(self.indexer_script(complete=False)), + ) + self.assertEqual(code, 0, output) + code, output = self.run_command("status", *self.base()) + self.assertEqual(code, 1) + self.assertEqual(json.loads(output)["state"], "partial") + def test_status_reports_stale_with_a_nonzero_exit(self) -> None: self.build() (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") From 9bfe023eb0b9be1569d465ea8efa0207fa366843 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 03:20:07 -0700 Subject: [PATCH 06/25] context: require completion evidence, refuse stale provider state, bound report reads Fix round for the three P2 findings on 9f3099a. codex:293b33c3cd44bf289e43 -- a readable report is not a completion claim. `complete` now needs a report shaped the way the adapter understands one: an affirmative claim (`complete`/`completed`/`finished`, or a recognized `status`/`state`), a count of what was indexed, and no counter admitting requeued, pending, or failed work. `{}`, an unrecognized schema, and a document that merely parses all stay `partial`. codex:4a34d6c0c10b5b8afa8f -- committed provider state is now skipped by the census, so a tracked `.graphify` or `.graph` is never materialized and cannot be resumed from as an incremental cache or collected as this run's output. `_safe_relative` refuses it too, and extraction refuses outright to run over a state directory that already exists. codex:220fe7a26bb89fcf8b5a -- the provider report is read to one byte past the manifest bound from an open stream and refused if longer, rather than loaded whole and measured afterwards. The artifact read and the CLI pin read carried the same allocate-then-check shape and are bounded the same way. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 40 ++++-- src/code_mower/context_graph_command.py | 5 +- src/code_mower/context_graph_lifecycle.py | 164 ++++++++++++++++++---- tests/test_context_graph_lifecycle.py | 136 +++++++++++++++++- 4 files changed, 304 insertions(+), 41 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 1e6d6731..54c17cef 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -41,8 +41,13 @@ provenance at all. If Code Mower does not bind the revision, nothing does. working tree and not the index. Symlinks (`120000`) and submodules (`160000`) are skipped and recorded as skipped, because a symlink can name a target the build was never shown and a gitlink names a commit in a - repository it was never authorized to read. The census digest covers mode, - blob name, size and path for every entry in sorted order. + repository it was never authorized to read. Committed provider state — a + tracked `.graphify/` or `.graph/`, at any depth, case-folded — is skipped for + a third reason: it is somebody's old index, and materializing it would let + the provider resume from a cache built over content this build never saw, + and let the adapter collect tracked repository bytes as if the provider had + just produced them. The census digest covers mode, blob name, size and path + for every entry in sorted order. 3. **Materialize into private state.** Each blob is written into a fresh 0700 directory as a 0600 file. Untracked and ignored files have no path into the graph because they are never written, rather than because something filtered @@ -169,15 +174,28 @@ builds of one commit have to produce identical bytes, because the manifest binds a digest of them. That state lands inside the throwaway materialized copy, never inside the indexed checkout, and the copy is deleted when the build ends. -**Completeness is read from the provider's report, never from its exit status.** -The adapter parses the report the provider leaves in that state directory and -marks the build `partial` if it admits requeued, pending, or failed entries, or -denies completion outright. A run that left no readable report is `partial` too: -absent evidence is not evidence of a complete build, and `partial` is the state -`graph_status` refuses by default, so the failure is one an operator can see and -act on. This is the direct consequence of the requeue defect the evaluation -recorded — a repeat that exits zero in 1.63 seconds having requeued 54 entries -has not built a complete graph. +Extraction refuses to run at all over a state directory that already exists. +The census keeps committed provider state out of the materialized copy, so in a +build from this module there is none; the refusal is the second check, because +everything after the run treats whatever is in that directory as output this +run produced. + +**Completeness is read from the provider's report, never from its exit status, +and only an affirmative claim counts.** `complete` requires a report shaped the +way the adapter understands one: a claim that the run finished (`complete`, +`completed`, `finished`, or a recognized `status`), a count of what was +indexed, and no counter admitting requeued, pending, or failed work. Everything +else is `partial` — a report that denies completion, one in an unrecognized +schema, an empty object, an unreadable one, one larger than a manifest, and no +report at all. Absent evidence is not evidence of a complete build, and +`partial` is the state `graph_status` refuses by default, so the failure is one +an operator can see and act on. This is the direct consequence of the requeue +defect the evaluation recorded — a repeat that exits zero in 1.63 seconds +having requeued 54 entries has not built a complete graph. + +The report is provider output of unknown size, so it is read to one byte past +the manifest bound and refused if it is longer, rather than loaded whole and +measured afterwards. A bound checked on bytes already in memory bounds nothing. The subcommand, the state-directory names, and the report counters are constants in one place in `context_graph_lifecycle.py`. They encode the interface as the diff --git a/src/code_mower/context_graph_command.py b/src/code_mower/context_graph_command.py index 54f51a52..762afb59 100644 --- a/src/code_mower/context_graph_command.py +++ b/src/code_mower/context_graph_command.py @@ -30,7 +30,10 @@ def _load_pin(path: Path | None) -> lifecycle.GraphifyPin | None: if path is None: return None try: - raw = path.read_bytes() + # Bounded at the stream, not after the fact: a bound checked on bytes + # already in memory is not a bound on what the file can cost to read. + with path.open("rb") as stream: + raw = stream.read(MAX_PIN_BYTES + 1) except OSError: raise ContextError("local graph provider pin file is unreadable") from None if len(raw) > MAX_PIN_BYTES: diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 7b219967..b958dea1 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -76,6 +76,17 @@ _REGULAR_MODES = frozenset({"100644", "100755"}) _SKIPPED_MODES = {"120000": "symlink", "160000": "submodule"} +#: Where the provider keeps its own index state. Both names are on the +#: excluded-roots list in ``context_graph``, and a repository is free to track +#: either of them -- a committed ``.graph/`` is somebody else's graph, or an +#: earlier incremental cache of this one. Neither may be materialized: the +#: provider would then resume from a cache built over content this build never +#: saw, and the adapter would collect tracked repository bytes as if the +#: provider had just produced them, binding stale contents to a fresh commit. +#: Matched at any depth and case-folded, for the same reasons ``.git`` is. +_PROVIDER_STATE_DIRECTORIES = (".graphify", ".graph") +_PROVIDER_STATE_ROOTS = frozenset(name.casefold() for name in _PROVIDER_STATE_DIRECTORIES) + _OBJECT_NAME = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?\Z") _GENERATION = re.compile(r"[0-9a-f]{32}\Z") _VERSION = re.compile(r"[0-9][0-9A-Za-z.+!-]{0,63}\Z") @@ -428,12 +439,22 @@ def resolve_revision(repository: Path, revision: str = "HEAD") -> tuple[str, str return commit, tree +def _is_provider_state(path: str) -> bool: + """Is this tracked path part of a committed provider index state?""" + return any(segment.casefold() in _PROVIDER_STATE_ROOTS for segment in path.split("/")) + + def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: """List the tracked regular files of one commit, with their blob sizes. Reads the commit's tree, never the working tree or the index, so an uncommitted edit, an untracked scratch file, and an ignored secret are all invisible here by construction rather than by filtering. + + Committed provider state is recorded as skipped rather than carried: it is + excluded from the census, so it is excluded from the census digest too, and + a build over a repository that tracks a ``.graphify`` directory binds a + census that says so instead of quietly indexing somebody else's graph. """ refuse_lazy_object_fetch(repository) listing = _git( @@ -458,6 +479,9 @@ def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: if mode in _SKIPPED_MODES: skipped.append((path, _SKIPPED_MODES[mode])) continue + if _is_provider_state(path): + skipped.append((path, "provider state")) + continue if mode not in _REGULAR_MODES or kind != "blob": skipped.append((path, "unsupported")) continue @@ -493,6 +517,10 @@ def _safe_relative(path: str) -> Path: # is as private as the top-level one. Case-folded because APFS and NTFS # name the same directory ``.GIT``. or any(segment.casefold() == ".git" for segment in path.split("/")) + # Provider state is skipped by the census, so a census that still + # carries it was not built by ``read_tracked_census``. Refuse rather + # than seed the directory the provider is about to write into. + or _is_provider_state(path) ): raise ContextError("tracked path must stay inside the materialized checkout") return Path(*path.split("/")) @@ -631,11 +659,6 @@ def _resolved_executable(executable: str) -> str: #: collects an artifact afterwards rather than naming one up front. _PROVIDER_EXTRACT = "extract" -#: Where that state lands. Both names appear in the evaluation's excluded-roots -#: list; the adapter accepts whichever the installed release writes and refuses -#: a build that produced neither. -_PROVIDER_STATE_DIRECTORIES = (".graphify", ".graph") - #: The provider's own record of what it processed. Completeness is read from #: here, never inferred from an exit status: the clean-room run recorded 54 #: manifest entries requeued by a repeat that exited zero in 1.63 s. @@ -649,8 +672,45 @@ def _resolved_executable(executable: str) -> str: #: Where the provider reports how many files it actually indexed. _INDEXED_COUNTERS = ("indexed_files", "code_files", "files", "entries") +#: An affirmative claim that the run finished, in either shape a report can +#: carry one. Nothing else counts: an empty object, or one whose schema this +#: adapter does not recognize, says nothing about completion and is therefore +#: not evidence of it. +_COMPLETION_FLAGS = ("complete", "completed", "finished") +#: Narrow on purpose: a field that names the run's state, not one that might +#: carry a path or a message, so an unrecognized value here is a real +#: non-completion rather than an adapter that read the wrong field. +_COMPLETION_STATUS_FIELDS = ("status", "state") +_COMPLETION_STATUS_VALUES = frozenset( + {"complete", "completed", "success", "succeeded", "ok", "finished", "done"} +) + + +def _refuse_pre_existing_provider_state(source_root: Path) -> None: + """Refuse to extract on top of index state this build did not produce. + + The census excludes committed provider state, so in a build from this + module nothing is here. This is the second check rather than the only one: + the source root is an argument, and the whole point of resolving the state + directory afterwards is to treat what is found as freshly produced output. + A directory that predates the run would let the provider resume from a + cache of content it was never shown, and would be collected as if it were + this commit's graph. + """ + for name in _PROVIDER_STATE_DIRECTORIES: + if (source_root / name).exists() or (source_root / name).is_symlink(): + raise ContextError( + "local graph build refuses to extract over pre-existing provider state; " + "no generation was published" + ) + def _provider_state_directory(source_root: Path) -> Path: + """The state directory the provider wrote during this run. + + Only reachable after ``_refuse_pre_existing_provider_state``, so whichever + of the two names is present was created by the run that just finished. + """ for name in _PROVIDER_STATE_DIRECTORIES: candidate = source_root / name if candidate.is_dir() and not candidate.is_symlink(): @@ -659,28 +719,76 @@ def _provider_state_directory(source_root: Path) -> Path: def _provider_report(state_directory: Path) -> Mapping[str, Any] | None: - """The provider's completion evidence, or ``None`` if it left none.""" + """The provider's completion evidence, or ``None`` if it left none. + + Bounded at the stream, not after the fact: the report is provider output of + unknown size, and reading it whole to slice it afterwards would let it + exhaust this process before any budget was consulted. Anything longer than + a manifest is rejected outright rather than parsed from a prefix, which + would be a different document than the one the provider wrote. + """ for name in _PROVIDER_REPORT_NAMES: path = state_directory / name if not path.is_file() or path.is_symlink(): continue try: - payload = json.loads(path.read_bytes()[: MAX_MANIFEST_BYTES + 1]) - except (OSError, ValueError): + with path.open("rb") as stream: + raw = stream.read(MAX_MANIFEST_BYTES + 1) + except OSError: + return None + if len(raw) > MAX_MANIFEST_BYTES: + return None + try: + payload = json.loads(raw) + except ValueError: return None return payload if isinstance(payload, Mapping) else None return None +def _completion_claim(report: Mapping[str, Any]) -> bool | None: + """``True`` finished, ``False`` denied it, ``None`` said nothing either way.""" + claim: bool | None = None + for name in _COMPLETION_FLAGS: + value = report.get(name) + if value is True: + claim = True + elif value is False: + return False + for field in _COMPLETION_STATUS_FIELDS: + value = report.get(field) + if not isinstance(value, str): + continue + if value.strip().casefold() in _COMPLETION_STATUS_VALUES: + claim = True + else: + # A status the adapter does not recognize is not a completion. + return False + return claim + + +def _indexed_count(report: Mapping[str, Any]) -> int | None: + """How many files the provider says it indexed, if it says at all.""" + for counter in _INDEXED_COUNTERS: + value = report.get(counter) + if isinstance(value, int) and not isinstance(value, bool) and value >= 0: + return value + return None + + def _read_completeness(report: Mapping[str, Any] | None) -> IndexResult: """Classify a provider run from its own report, defaulting to partial. - Absent or unreadable evidence is *not* evidence of a complete build. The - provider owns no provenance (the evaluation records this as the first - product constraint), so a build that cannot read a completion claim - publishes a generation marked ``partial``, which ``graph_status`` refuses - by default. That is the failure an operator can act on; silently calling - it complete is the one they cannot. + Absent or unreadable evidence is *not* evidence of a complete build, and + neither is a readable report that says nothing. ``complete`` is reached + only by a report shaped the way this adapter understands one: an + affirmative completion claim, a count of what was indexed, and no counter + admitting work left over. An empty object, an unrecognized schema, and a + document that happens to parse all stay ``partial``, which + ``graph_status`` refuses by default. The provider owns no provenance (the + evaluation records this as the first product constraint), so that refusal + is the failure an operator can act on; silently calling it complete is the + one they cannot. """ if report is None: return IndexResult(completeness=PARTIAL, notes=("provider left no readable completion report",)) @@ -691,16 +799,16 @@ def _read_completeness(report: Mapping[str, Any] | None) -> IndexResult: notes.append(f"provider reported {counter}") elif isinstance(value, int) and not isinstance(value, bool) and value > 0: notes.append(f"provider reported {value} {counter}") - if report.get("complete") is False: - notes.append("provider reported the extraction as incomplete") - indexed = 0 - for counter in _INDEXED_COUNTERS: - value = report.get(counter) - if isinstance(value, int) and not isinstance(value, bool) and value >= 0: - indexed = value - break + claim = _completion_claim(report) + if claim is False: + notes.append("provider did not report the extraction as complete") + elif claim is None: + notes.append("provider report carried no completion claim") + indexed = _indexed_count(report) + if indexed is None: + notes.append("provider report did not say how many files it indexed") if notes: - return IndexResult(completeness=PARTIAL, indexed_files=indexed, notes=tuple(notes)) + return IndexResult(completeness=PARTIAL, indexed_files=indexed or 0, notes=tuple(notes)) return IndexResult(completeness=COMPLETE, indexed_files=indexed) @@ -758,6 +866,7 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] ) def run(request: IndexRequest) -> IndexResult: + _refuse_pre_existing_provider_state(request.source_root) try: completed = subprocess.run( [*sandbox, command, _PROVIDER_EXTRACT, *request.pin.options], @@ -1270,10 +1379,13 @@ def build_graph( raise ContextError("local graph provider returned an unsupported result") if not output_path.is_file(): raise ContextError("local graph provider produced no artifact") - size = output_path.stat().st_size - if size > MAX_ARTIFACT_BYTES: + # Read to the budget and one byte past it, rather than trusting a + # size taken before the read: the artifact is provider output, and + # the budget has to bound what this process allocates for it. + with output_path.open("rb") as stream: + artifact = stream.read(MAX_ARTIFACT_BYTES + 1) + if len(artifact) > MAX_ARTIFACT_BYTES: raise ContextError("local graph artifact exceeds its budget; no generation was published") - artifact = output_path.read_bytes() manifest = BuildManifest( generation=uuid.uuid4().hex, schema=MANIFEST_SCHEMA, diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 1da6120f..fae54a25 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -172,6 +172,48 @@ def test_symlinks_and_submodules_are_skipped_rather_than_followed(self) -> None: self.assertNotIn("linked.py", [entry.path for entry in census.entries]) self.assertIn(("linked.py", "symlink"), census.skipped) + def test_committed_provider_state_is_skipped_rather_than_indexed(self) -> None: + """A tracked ``.graphify`` is an old cache, not content to index. + + Materializing it would let the provider resume from a cache built over + content this build never saw, and the adapter would then collect + tracked repository bytes as if the provider had just produced them. + """ + for name in (".graphify", "vendor/.GRAPH"): + directory = self.repository / name + directory.mkdir(parents=True) + (directory / "cache.json").write_text('{"stale": true}\n', encoding="utf-8") + git(self.repository, "add", ".graphify", "vendor") + git(self.repository, "commit", "-q", "-m", "committed provider state") + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + self.assertEqual( + [entry.path for entry in census.entries], + [".gitignore", "README.md", "example_pkg/config.py"], + ) + self.assertIn((".graphify/cache.json", "provider state"), census.skipped) + self.assertIn(("vendor/.GRAPH/cache.json", "provider state"), census.skipped) + destination = self.root / "materialized-with-state" + lifecycle.materialize_tracked_files(self.repository, census, destination) + self.assertFalse((destination / ".graphify").exists()) + self.assertFalse((destination / "vendor").exists()) + + def test_committing_provider_state_does_not_move_the_census_digest(self) -> None: + # The manifest binds the digest of what was indexed. Committed provider + # state is not indexed, so it does not enter that digest; it is + # accounted for in ``skipped`` instead. + commit, _ = lifecycle.resolve_revision(self.repository) + before = lifecycle.read_tracked_census(self.repository, commit) + state = self.repository / ".graph" + state.mkdir() + (state / "cache.json").write_text('{"stale": true}\n', encoding="utf-8") + git(self.repository, "add", ".graph") + git(self.repository, "commit", "-q", "-m", "state") + after_commit, _ = lifecycle.resolve_revision(self.repository) + after = lifecycle.read_tracked_census(self.repository, after_commit) + self.assertEqual(before.digest, after.digest) + self.assertNotEqual(before.skipped, after.skipped) + def test_materialization_writes_only_tracked_files(self) -> None: commit, _ = lifecycle.resolve_revision(self.repository) census = lifecycle.read_tracked_census(self.repository, commit) @@ -208,7 +250,8 @@ def test_materialization_refuses_an_existing_directory(self) -> None: def test_escaping_census_paths_are_rejected(self) -> None: escaping = ("/etc/passwd", "../outside.py", "a/../../b.py", ".git/config", - "vendor/.git/config", "a\\b.py") + "vendor/.git/config", "a\\b.py", ".graphify/cache.json", + "vendor/.GRAPH/cache.json") for index, path in enumerate(escaping): with self.subTest(path=path): census = lifecycle.TrackedCensus( @@ -345,8 +388,10 @@ class ProviderLaunchTests(TemporaryWorkspace): """What ``subprocess_indexer`` actually hands the operating system.""" def request(self) -> lifecycle.IndexRequest: - source = self.root / "source" - source.mkdir(exist_ok=True) + # A fresh directory per launch, because that is what a build hands the + # provider: ``materialize_tracked_files`` refuses a destination that + # already exists, and so the provider never sees state from a prior run. + source = Path(tempfile.mkdtemp(dir=self.root)) return lifecycle.IndexRequest( source_root=source, output_path=self.root / "graph.bin", @@ -464,6 +509,91 @@ def fake_run(argv, **kwargs): result = indexer(request) self.assertEqual(result.completeness, lifecycle.PARTIAL) + def test_a_report_without_affirmative_completion_evidence_is_partial(self) -> None: + """A document that parses is not a document that claims completion. + + ``{}`` and a report in some schema this adapter does not understand + both say nothing about whether the extraction finished, and nothing is + not a claim. Treating them as complete would hand ``graph_status`` a + usable generation built from an unknown run. + """ + silent = ( + {}, + {"schema": "unexpected"}, + {"code_files": 3}, + {"complete": True}, + {"status": "running", "code_files": 3}, + {"status": "partial", "complete": True, "code_files": 3}, + ) + for report in silent: + with self.subTest(report=report): + _, result = self.run_indexer("graphify", report=report) + self.assertEqual(result.completeness, lifecycle.PARTIAL) + self.assertTrue(result.notes) + + def test_a_report_that_claims_completion_and_counts_its_work_is_complete(self) -> None: + claimed = ( + {"complete": True, "code_files": 3}, + {"status": "success", "indexed_files": 3}, + {"completed": True, "entries": 0, "requeued": 0}, + ) + for report in claimed: + with self.subTest(report=report): + _, result = self.run_indexer("graphify", report=report) + self.assertEqual(result.completeness, lifecycle.COMPLETE) + + def test_an_oversized_report_is_refused_without_being_read_whole(self) -> None: + """Provider output is unbounded input; the read is bounded at the stream. + + Slicing after ``read_bytes()`` would have allocated the whole document + first, so the assertion is not only that the build stays ``partial``: + nothing in the collection path may read a provider file whole. + """ + request = self.request() + oversized = b'{"complete": true, "code_files": 3, "pad": "' + b"x" * lifecycle.MAX_MANIFEST_BYTES + b'"}' + + def fake_run(argv, **kwargs): + written = request.source_root / ".graphify" + written.mkdir(exist_ok=True) + (written / "graph.bin").write_bytes(b"graph-bytes") + (written / "manifest.json").write_bytes(oversized) + return subprocess.CompletedProcess(argv, 0, b"", b"") + + def refuse_whole_file_read(self: Path) -> bytes: + raise AssertionError(f"{self} was read whole") + + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + indexer = lifecycle.subprocess_indexer("graphify") + with mock.patch.object(subprocess, "run", fake_run): + with mock.patch.object(Path, "read_bytes", refuse_whole_file_read): + result = indexer(request) + self.assertEqual(result.completeness, lifecycle.PARTIAL) + + def test_extraction_refuses_to_run_over_pre_existing_provider_state(self) -> None: + """State that predates the run is a cache, and would be collected as output. + + The census already keeps committed provider state out of the copy, so + this is the second check rather than the only one -- the source root is + an argument, and everything after the run treats what it finds there as + freshly produced. + """ + request = self.request() + (request.source_root / ".graphify").mkdir() + (request.source_root / ".graphify" / "cache.json").write_text("{}", encoding="utf-8") + launched: list[list[str]] = [] + + def fake_run(argv, **kwargs): + launched.append(list(argv)) + return subprocess.CompletedProcess(argv, 0, b"", b"") + + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + indexer = lifecycle.subprocess_indexer("graphify") + with mock.patch.object(subprocess, "run", fake_run): + with self.assertRaises(ContextError): + indexer(request) + self.assertEqual(launched, []) + self.assertFalse((self.root / "graph.bin").exists()) + def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): with self.assertRaises(ContextError): From 86b124b0368db9b34e25a2776e36af9d425efb28 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 03:27:47 -0700 Subject: [PATCH 07/25] context-graph: discard the provider's streams instead of buffering them The indexer's stdout and stderr were captured and never read: a provider that logs its progress could accumulate unbounded output in this process for up to the 900 s timeout, bounded by neither the tracked-content budget nor the artifact one. Both now go to DEVNULL, as does stdin -- discarded at the kernel rather than inherited, because provider diagnostics can echo indexed source and the launching process may be writing a machine-readable report to its own stdout. Addresses codex:fdaf1d468d2a8a6878dc (P2). Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 9 +++++++++ src/code_mower/context_graph_lifecycle.py | 14 +++++++++++--- tests/test_context_graph_lifecycle.py | 21 +++++++++++++++++++++ 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 54c17cef..08bff54d 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -197,6 +197,15 @@ The report is provider output of unknown size, so it is read to one byte past the manifest bound and refused if it is longer, rather than loaded whole and measured afterwards. A bound checked on bytes already in memory bounds nothing. +The provider's own stdout and stderr are the other unbounded output, and they +are discarded at the kernel: `stdin`, `stdout` and `stderr` are all +`DEVNULL`. Nothing reads them — completeness comes from the report, not from +what the run printed — so buffering them would only accumulate whatever a +talkative indexer chose to log, for up to the timeout, under neither the +tracked-content budget nor the artifact one. Inheriting them is not the +alternative: diagnostics can echo indexed source, and the process that launched +the build may be writing a machine-readable report to its own stdout. + The subcommand, the state-directory names, and the report counters are constants in one place in `context_graph_lifecycle.py`. They encode the interface as the evaluation recorded it; the first installation against a real pinned release diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index b958dea1..421509e8 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -871,8 +871,17 @@ def run(request: IndexRequest) -> IndexResult: completed = subprocess.run( [*sandbox, command, _PROVIDER_EXTRACT, *request.pin.options], check=False, - capture_output=True, - text=False, + # Neither stream is read, and neither may be buffered: a + # provider that logs its progress would otherwise accumulate + # unbounded output in this process for up to the timeout, + # outside both the tracked-content and artifact budgets. The + # streams are discarded at the kernel rather than inherited, + # because provider diagnostics can echo indexed source and this + # process may be writing a machine-readable report. ``stdin`` + # goes the same way: the child has no operator to prompt. + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, env=dict(request.environment), cwd=str(request.source_root), timeout=900, @@ -880,7 +889,6 @@ def run(request: IndexRequest) -> IndexResult: except (OSError, subprocess.SubprocessError): raise ContextError("local graph provider could not be run from its pinned install") from None if completed.returncode != 0: - # Provider stderr can echo indexed source; it is never surfaced. raise ContextError("local graph provider failed; no generation was published") state_directory = _provider_state_directory(request.source_root) result = _read_completeness(_provider_report(state_directory)) diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index fae54a25..4702d28a 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -417,9 +417,13 @@ def run_indexer( """ request = self.request() recorded: list[list[str]] = [] + # What the adapter asked the operating system for, kept beside the argv + # because the stream arrangement is as much of the contract as it is. + self.launch_options: list[dict[str, object]] = [] def fake_run(argv, **kwargs): recorded.append(list(argv)) + self.launch_options.append(dict(kwargs)) if state_directory: written = request.source_root / state_directory written.mkdir(exist_ok=True) @@ -443,6 +447,23 @@ def test_the_provider_is_launched_inside_the_sandbox(self) -> None: argv = self.launched_argv("graphify") self.assertEqual(argv[:3], ["/sandbox", "--deny", "graphify"]) + def test_the_provider_is_given_no_stream_this_process_has_to_hold(self) -> None: + """A talkative indexer must not be able to fill this process's memory. + + Nothing reads the provider's stdout or stderr -- completeness comes + from the report it writes, not from what it printed -- so buffering + them would only accumulate whatever it chose to log, for up to the + timeout, under neither the tracked-content budget nor the artifact + one. Inheriting them instead is not the alternative: diagnostics can + echo indexed source, and this process may be writing JSON to stdout. + """ + self.launched_argv("graphify") + options = self.launch_options[0] + self.assertNotIn("capture_output", options) + self.assertEqual(options["stdout"], subprocess.DEVNULL) + self.assertEqual(options["stderr"], subprocess.DEVNULL) + self.assertEqual(options["stdin"], subprocess.DEVNULL) + def test_the_provider_is_invoked_through_its_documented_extract_interface(self) -> None: # The interface the adopt decision evaluated, recorded in # docs/graphify-evaluation.md as ``extract`` plus options. An From f063f52968702de46daba80b0eed2fa572787899 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 03:38:42 -0700 Subject: [PATCH 08/25] context-graph: require the restricted extraction, bound packing, and report partial builds as partial Three P2 findings from the 86b124b audit. A pin may omit options entirely, which launched extract with neither --code-only nor --no-cluster: provider behaviour the adopt decision excluded. The two restrictions are now folded into the pin's own options whichever way it was constructed, so the manifest records the run that actually happened, and an option that undoes one of them is refused rather than overridden by argument order. The launcher normalizes again at the point of launch. Packing the provider's state bounded only the sum of the file sizes, which is not a bound on the archive: tar headers, padding and extended pathname records are bytes this process holds, and many empty files stay far under the byte budget. Entries are now capped while their names are collected, and the archive is serialized into a buffer that refuses to grow past the artifact budget. The build command constructed a current status from a manifest it had not consulted, so a provider-declared incomplete run printed as current and then reported partial one command later. The displayed state now comes from the manifest's completeness, and an unusable published generation exits non-zero for the same reason status does. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 28 +++++- src/code_mower/context_graph_command.py | 23 ++++- src/code_mower/context_graph_lifecycle.py | 111 +++++++++++++++++++--- tests/test_context_graph_lifecycle.py | 108 ++++++++++++++++++++- 4 files changed, 247 insertions(+), 23 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 08bff54d..71d1883c 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -166,6 +166,16 @@ is run in and writes its state beside those sources, which the clean-room run in [the evaluation](graphify-evaluation.md) recorded as `extract --code-only --no-cluster --max-workers 4`. +**`--code-only` and `--no-cluster` are always passed, whatever the pin says.** +They are conditions of the adopt decision, not preferences: a pin that named no +options at all would otherwise have launched the provider into clustering and +whatever extraction it does by default, both of which are separate decisions +nobody has taken. They are folded into the pin's own `options` rather than added +at the launch site, so the manifest records the run that actually happened, and +a pin that tries to undo one of them — `--cluster`, `--no-code-only`, or a +valued form such as `--code-only=false` — is refused rather than quietly +overridden by argument order. + So the adapter collects an artifact afterwards rather than naming one up front. The state directory the provider wrote (`.graphify` or `.graph`, both already on the excluded-roots list) is packed into a single reproducible archive: names @@ -174,6 +184,14 @@ builds of one commit have to produce identical bytes, because the manifest binds a digest of them. That state lands inside the throwaway materialized copy, never inside the indexed checkout, and the copy is deleted when the build ends. +Packing is bounded as it happens, in both dimensions. The number of entries is +capped while their names are collected, and the serialized archive is written +into a buffer that refuses to grow past the artifact budget. Summing file sizes +is not a bound on the archive: many empty files stay far under the byte budget +while their headers, padding and extended pathname records are bytes this +process has to hold, and a budget checked on a finished archive is checked after +the memory was already taken. + Extraction refuses to run at all over a state directory that already exists. The census keeps committed provider state out of the materialized copy, so in a build from this module there is none; the refusal is the second check, because @@ -255,7 +273,11 @@ code-mower context-graph doctor [--pin-file PIN] ``` `status` exits non-zero when the graph is not usable, so a script can branch on -it. `doctor` reports `skip` rather than `fail` when nothing is pinned or built: +it. `build` and `refresh` do the same, and for the same reason: a provider that +admitted an incomplete run has published a generation `status` will call +`partial` and refuse, so the build prints `partial` and exits non-zero rather +than describing it as `current` for as long as it takes to ask again. +`doctor` reports `skip` rather than `fail` when nothing is pinned or built: the lifecycle is optional, and an operator who never opted in has nothing wrong with their installation. @@ -271,6 +293,10 @@ marker, or a distribution without an artifact digest: } ``` +`options` is optional and may name extra provider flags, such as +`--max-workers`. The two restrictions above are added whether or not the file +lists them; listing them changes nothing, and contradicting them is refused. + `--indexer` is the path to a provider CLI the operator has **already** installed. This repository does not download, install, or resolve one, which is why the executable is named rather than discovered. A relative path such as diff --git a/src/code_mower/context_graph_command.py b/src/code_mower/context_graph_command.py index 762afb59..321e036b 100644 --- a/src/code_mower/context_graph_command.py +++ b/src/code_mower/context_graph_command.py @@ -102,11 +102,24 @@ def main(argv=None) -> int: revision=args.revision, keep_previous=args.keep_previous, ) - summary = {"status": "published", **manifest.shareable_summary()} - _emit(summary, as_json=args.json, - text=lifecycle.render_status_text( - lifecycle.GenerationStatus(state="current", generation=manifest.generation, manifest=manifest))) - return 0 + # The published state is the manifest's, not this command's to + # assume: a provider that admitted an incomplete run has published + # a generation ``status`` will call ``partial`` and refuse, and + # printing ``current`` here would describe it as usable for exactly + # as long as it took the operator to ask again. + complete = manifest.completeness == lifecycle.COMPLETE + published = lifecycle.GenerationStatus( + state="current" if complete else "partial", + generation=manifest.generation, + manifest=manifest, + detail="" if complete else "local graph build was incomplete; refresh it", + ) + summary = {"status": "published", "usable": published.usable, **manifest.shareable_summary()} + _emit(summary, as_json=args.json, text=lifecycle.render_status_text(published)) + # Publishing an unusable generation is a reportable condition, not + # a crash: exit non-zero for the same reason ``status`` does, so a + # script does not have to re-ask to find out what it just built. + return 0 if published.usable else 1 if args.command == "status": report = lifecycle.graph_status( args.repo_path, diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 421509e8..da870c1c 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -69,6 +69,13 @@ MAX_TRACKED_FILES = 50_000 MAX_TRACKED_BYTES = 512 * 1024 * 1024 MAX_BLOB_BYTES = 32 * 1024 * 1024 +#: How many files the provider's state may contribute to one artifact. A byte +#: budget alone is not a bound on what packing costs: many empty files stay far +#: under it while their headers, padding and extended pathname records grow the +#: archive this process holds. Sized to the tracked-file bound, because a graph +#: of a checkout has no honest reason to hold more entries than the checkout +#: had files. +MAX_ARTIFACT_ENTRIES = MAX_TRACKED_FILES #: Only ordinary blobs are materialized. A symlink (``120000``) can name a #: target outside the checkout and a gitlink (``160000``) names a commit in @@ -234,6 +241,47 @@ def _size(value: Any, maximum: int) -> int: return value +#: The extraction restrictions the adopt decision was conditioned on, recorded +#: in ``docs/graphify-evaluation.md``: code-only extraction, and no clustering. +#: They are not the operator's to omit. Model-based extraction and clustering +#: are separate explicit decisions nobody has taken, and a pin that simply left +#: its ``options`` empty would have launched the provider into both. +_REQUIRED_EXTRACT_OPTIONS = ("--code-only", "--no-cluster") + +#: Options that would undo one of the above. Named exactly rather than guessed +#: at: these are the negations of the two flags this module requires, so a pin +#: carrying one is asking for behaviour the adoption conditions exclude and is +#: refused rather than silently overridden by argument order. +_CONFLICTING_EXTRACT_OPTIONS = frozenset({"--cluster", "--no-code-only"}) + + +def _extraction_options(options: Iterable[str]) -> tuple[str, ...]: + """The options every extraction runs with: the required ones, then the pin's. + + Required unconditionally rather than merely validated, so a pin written + before these conditions existed -- or one with no ``options`` at all -- + still launches a restricted run. They are prepended into the pin itself + rather than added at the call site, so the manifest records what actually + ran instead of what was asked for. + """ + extra: list[str] = [] + for option in options: + name = option.split("=", 1)[0].strip() + if name in _CONFLICTING_EXTRACT_OPTIONS: + raise ContextError( + "local graph provider options may not re-enable clustering or non-code extraction" + ) + if name in _REQUIRED_EXTRACT_OPTIONS: + if option != name: + # ``--code-only=false`` is the same request as ``--no-code-only``. + raise ContextError( + "local graph provider options may not give a value to a required extraction flag" + ) + continue + extra.append(option) + return (*_REQUIRED_EXTRACT_OPTIONS, *extra) + + @dataclass(frozen=True) class GraphifyPin: """An exact provider pin. A range would let a build drift silently. @@ -242,6 +290,10 @@ class GraphifyPin: ``docs/graphify-evaluation.md``. It is carried into every build manifest so a graph built by a substituted distribution is identifiable after the fact, which is the whole point of pinning a lookalike-prone package name. + + ``options`` always carries the required extraction restrictions, whichever + way the pin was constructed, so there is no shape of this object that could + launch an unrestricted run. """ distribution: str @@ -249,6 +301,9 @@ class GraphifyPin: wheel_sha256: str options: tuple[str, ...] = () + def __post_init__(self) -> None: + object.__setattr__(self, "options", _extraction_options(self.options)) + @property def requirement(self) -> str: return f"{self.distribution}=={self.version}" @@ -812,6 +867,23 @@ def _read_completeness(report: Mapping[str, Any] | None) -> IndexResult: return IndexResult(completeness=COMPLETE, indexed_files=indexed) +class _BoundedBuffer(io.BytesIO): + """A buffer that refuses to grow past the artifact budget. + + The budget has to be enforced on what this process allocates, as it is + allocated. A check on the finished archive is a check made after the + memory was already taken, and a check on the sum of the file sizes is not + a bound on the archive at all: headers, padding and extended pathname + records are bytes the provider can make this process hold without ever + writing content. + """ + + def write(self, data) -> int: # type: ignore[override] + if self.tell() + len(data) > MAX_ARTIFACT_BYTES: + raise ContextError("local graph artifact exceeds its budget; no generation was published") + return super().write(data) + + def _pack_state(state_directory: Path) -> bytes: """Collect the provider's state into one reproducible artifact. @@ -820,16 +892,25 @@ def _pack_state(state_directory: Path) -> bytes: a digest of them. Only regular files are taken -- a symlink in provider state would name a target outside the artifact, which an immutable generation cannot carry. + + Both the number of entries and the serialized size are bounded while the + archive is being built, so a provider that wrote pathologically many files + is refused before this process has held them: even collecting the names to + sort them is done against the entry bound rather than into an unbounded + list. """ - buffer = io.BytesIO() - total = 0 + entries: list[Path] = [] + for path in state_directory.rglob("*"): + if path.is_symlink() or not path.is_file(): + continue + entries.append(path) + if len(entries) > MAX_ARTIFACT_ENTRIES: + raise ContextError( + "local graph artifact holds more files than its budget allows; no generation was published" + ) + buffer = _BoundedBuffer() with tarfile.open(fileobj=buffer, mode="w", format=tarfile.PAX_FORMAT) as archive: - for path in sorted(state_directory.rglob("*"), key=lambda item: str(item.relative_to(state_directory))): - if path.is_symlink() or not path.is_file(): - continue - total += path.stat().st_size - if total > MAX_ARTIFACT_BYTES: - raise ContextError("local graph artifact exceeds its budget; no generation was published") + for path in sorted(entries, key=lambda item: str(item.relative_to(state_directory))): info = tarfile.TarInfo(str(path.relative_to(state_directory))) info.size = path.stat().st_size info.mtime = 0 @@ -853,9 +934,9 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] or an uncontainable host fails before a single blob is materialized. The argv is the interface the adopt decision evaluated, not a guess at a - conventional one: ``extract`` with the pinned options, in the materialized - copy. Everything the provider leaves behind is then collected and - classified from its own report. + conventional one: ``extract`` with the required restrictions and then the + pinned options, in the materialized copy. Everything the provider leaves + behind is then collected and classified from its own report. """ command = _resolved_executable(executable) sandbox = network_sandbox_command() @@ -867,9 +948,14 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] def run(request: IndexRequest) -> IndexResult: _refuse_pre_existing_provider_state(request.source_root) + # Normalized again at the point of launch, not because the pin could + # arrive without the restrictions -- it cannot -- but because this is + # the line that decides what the provider is actually asked to do, and + # it should be readable here without trusting a constructor elsewhere. + options = _extraction_options(request.pin.options) try: completed = subprocess.run( - [*sandbox, command, _PROVIDER_EXTRACT, *request.pin.options], + [*sandbox, command, _PROVIDER_EXTRACT, *options], check=False, # Neither stream is read, and neither may be buffered: a # provider that logs its progress would otherwise accumulate @@ -1644,6 +1730,7 @@ def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[ "IndexResult", "MANIFEST_SCHEMA", "MAX_ARTIFACT_BYTES", + "MAX_ARTIFACT_ENTRIES", "MAX_TRACKED_FILES", "PARTIAL", "TrackedCensus", diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 4702d28a..76ecea64 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -118,7 +118,38 @@ def test_accepts_one_exact_release(self) -> None: {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": "b" * 64} ) self.assertEqual(pin.requirement, "graphifyy==0.9.58") - self.assertEqual(pin.options, ()) + # A pin that names no options is still a restricted pin: the adoption + # conditions are not the operator's to omit by leaving a field out. + self.assertEqual(pin.options, ("--code-only", "--no-cluster")) + + def test_the_required_extraction_restrictions_are_always_carried(self) -> None: + """Code-only and no-cluster are conditions of the adopt decision. + + They are prepended into the pin rather than added at the launch site, + so the manifest records the run that actually happened. A pin that + already names one keeps exactly one copy of it, and whatever else it + names is preserved after them. + """ + pin = lifecycle.load_pin( + {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": "b" * 64, + "options": ["--no-cluster", "--max-workers", "4"]} + ) + self.assertEqual(pin.options, ("--code-only", "--no-cluster", "--max-workers", "4")) + + def test_rejects_options_that_undo_the_extraction_restrictions(self) -> None: + """An option that re-enables clustering or non-code extraction is refused. + + Overriding it by argument order would leave the pin claiming one thing + and the provider doing another; refusing says so where an operator can + see it. + """ + for option in ("--cluster", "--no-code-only", "--code-only=false", "--no-cluster=0"): + with self.subTest(option=option): + with self.assertRaises(ContextError): + lifecycle.load_pin( + {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": "b" * 64, + "options": [option]} + ) def test_rejects_ranges_and_unpinned_shapes(self) -> None: """A range, a marker, or a missing digest lets a build drift silently.""" @@ -387,7 +418,7 @@ def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: class ProviderLaunchTests(TemporaryWorkspace): """What ``subprocess_indexer`` actually hands the operating system.""" - def request(self) -> lifecycle.IndexRequest: + def request(self, pin: lifecycle.GraphifyPin = PIN) -> lifecycle.IndexRequest: # A fresh directory per launch, because that is what a build hands the # provider: ``materialize_tracked_files`` refuses a destination that # already exists, and so the provider never sees state from a prior run. @@ -396,7 +427,7 @@ def request(self) -> lifecycle.IndexRequest: source_root=source, output_path=self.root / "graph.bin", environment={"PATH": os.environ.get("PATH", "")}, - pin=PIN, + pin=pin, commit="a" * 40, tree="b" * 40, ) @@ -408,6 +439,7 @@ def run_indexer( sandbox=("/sandbox", "--deny"), report: object = FINISHED_REPORT, state_directory: str = ".graphify", + pin: lifecycle.GraphifyPin = PIN, ) -> tuple[list[str], lifecycle.IndexResult]: """Launch the adapter with the provider's side of the contract faked. @@ -415,7 +447,7 @@ def run_indexer( stand-in has to leave that state behind for the adapter to collect -- an exit status alone is not a finished build. """ - request = self.request() + request = self.request(pin) recorded: list[list[str]] = [] # What the adapter asked the operating system for, kept beside the argv # because the stream arrangement is as much of the contract as it is. @@ -473,6 +505,26 @@ def test_the_provider_is_invoked_through_its_documented_extract_interface(self) self.assertNotIn("--source", argv) self.assertNotIn("--output", argv) + def test_extraction_is_restricted_to_code_and_never_clusters(self) -> None: + """The adoption conditions are enforced at the launch, not assumed. + + A pin is free to name no options at all, and one that did would + otherwise have launched the provider into clustering and whatever + extraction it does by default -- both of which are separate decisions + nobody has taken. + """ + bare = lifecycle.GraphifyPin(distribution="graphifyy", version="0.9.58", wheel_sha256="a" * 64) + argv, _ = self.run_indexer("graphify", pin=bare) + self.assertEqual(argv[3:], ["extract", "--code-only", "--no-cluster"]) + + def test_the_launch_restricts_extraction_even_if_the_pin_did_not(self) -> None: + # The pin normalizes its own options, so this reaches past the + # constructor to prove the guarantee does not rest on it alone. + stripped = lifecycle.GraphifyPin(distribution="graphifyy", version="0.9.58", wheel_sha256="a" * 64) + object.__setattr__(stripped, "options", ()) + argv, _ = self.run_indexer("graphify", pin=stripped) + self.assertEqual(argv[3:], ["extract", "--code-only", "--no-cluster"]) + def test_the_collected_artifact_holds_the_state_the_provider_wrote(self) -> None: _, result = self.run_indexer("graphify") self.assertEqual(result.completeness, lifecycle.COMPLETE) @@ -493,6 +545,33 @@ def test_collecting_the_same_state_twice_produces_the_same_bytes(self) -> None: self.run_indexer("graphify") self.assertEqual(first, (self.root / "graph.bin").read_bytes()) + def test_packing_is_bounded_by_the_archive_and_not_by_the_file_sizes(self) -> None: + """Empty files are not free: their headers are bytes this process holds. + + The budget is enforced on the serialized archive as it is written, so + state whose contents sum to nothing at all is still refused once the + archive it turns into would exceed what this process may allocate. + """ + state = self.root / "packed" + state.mkdir() + for index in range(16): + (state / f"node-{index:02d}.bin").write_bytes(b"") + self.assertEqual(sum(path.stat().st_size for path in state.iterdir()), 0) + with mock.patch.object(lifecycle, "MAX_ARTIFACT_BYTES", 2048): + with self.assertRaises(ContextError): + lifecycle._pack_state(state) + + def test_packing_refuses_more_entries_than_its_budget_allows(self) -> None: + # Bounded while the names are collected, before anything is archived. + state = self.root / "entries" + state.mkdir() + for index in range(4): + (state / f"node-{index}.bin").write_bytes(b"x") + with mock.patch.object(lifecycle, "MAX_ARTIFACT_ENTRIES", 3): + with self.assertRaises(ContextError): + lifecycle._pack_state(state) + self.assertEqual(len(self.archived_names(lifecycle._pack_state(state))), 4) + def test_a_provider_that_wrote_no_state_publishes_nothing(self) -> None: with self.assertRaises(ContextError): self.run_indexer("graphify", state_directory="") @@ -1164,11 +1243,30 @@ def test_a_provider_that_admits_an_incomplete_run_is_not_usable(self) -> None: "--pin-file", str(self.pin_file()), "--indexer", str(self.indexer_script(complete=False)), ) - self.assertEqual(code, 0, output) + # The build says so itself. Reporting ``current`` here and ``partial`` + # one command later would describe an unusable generation as usable for + # exactly as long as it took to ask again. + self.assertEqual(code, 1, output) + published = json.loads(output) + self.assertFalse(published["usable"]) + self.assertEqual(published["completeness"], lifecycle.PARTIAL) code, output = self.run_command("status", *self.base()) self.assertEqual(code, 1) self.assertEqual(json.loads(output)["state"], "partial") + def test_an_incomplete_build_is_printed_as_partial(self) -> None: + if lifecycle.network_sandbox_command() is None: + self.skipTest("this host offers no OS sandbox that denies a child the network") + arguments = [argument for argument in self.base() if argument != "--json"] + code, output = self.run_command( + "build", *arguments, + "--pin-file", str(self.pin_file()), + "--indexer", str(self.indexer_script(complete=False)), + ) + self.assertEqual(code, 1, output) + self.assertIn("Local graph: partial", output) + self.assertNotIn("Local graph: current", output) + def test_status_reports_stale_with_a_nonzero_exit(self) -> None: self.build() (self.repository / "README.md").write_text("# changed\n", encoding="utf-8") From 826b1052f7e0db09efdfbdaa0ee9b6abc4a946a3 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 03:44:54 -0700 Subject: [PATCH 09/25] context-graph: give each provider-launch fixture its own artifact path The two table-driven provider-report tests launch the adapter once per subtest, and every launch was handed the same `graph.bin`. The artifact writer opens its output `O_EXCL`, so the second subtest onward died with `FileExistsError` rather than exercising the report it was named for -- seven errors per interpreter, on all three package-matrix jobs. The exclusive create is the production behaviour under test and stays as it is: a build writes the artifact into the generation it is about to publish, which by construction does not exist yet. What was wrong is the fixture, which reused one path across launches the way a build never does. Each request now names its own artifact, alongside the fresh source root it already got for the same reason. That also lets the determinism test drop its `unlink`. It was there only to clear the way for the second launch, and comparing two artifacts that were written independently is the thing the test claims to check. --- tests/test_context_graph_lifecycle.py | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 76ecea64..f846a9ef 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -418,14 +418,26 @@ def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: class ProviderLaunchTests(TemporaryWorkspace): """What ``subprocess_indexer`` actually hands the operating system.""" + def setUp(self) -> None: + super().setUp() + # Every artifact path this fixture has handed out, newest last, so a + # test that launches more than once can name the run it means. + self.artifacts: list[Path] = [] + def request(self, pin: lifecycle.GraphifyPin = PIN) -> lifecycle.IndexRequest: # A fresh directory per launch, because that is what a build hands the # provider: ``materialize_tracked_files`` refuses a destination that # already exists, and so the provider never sees state from a prior run. source = Path(tempfile.mkdtemp(dir=self.root)) + # The artifact is fresh for the same reason. A build writes it into the + # generation it is about to publish, and the writer opens it O_EXCL -- + # a path shared between two launches would collide on the second rather + # than exercise anything, so each request names its own. + artifact = Path(tempfile.mkdtemp(dir=self.root)) / "graph.bin" + self.artifacts.append(artifact) return lifecycle.IndexRequest( source_root=source, - output_path=self.root / "graph.bin", + output_path=artifact, environment={"PATH": os.environ.get("PATH", "")}, pin=pin, commit="a" * 40, @@ -529,7 +541,7 @@ def test_the_collected_artifact_holds_the_state_the_provider_wrote(self) -> None _, result = self.run_indexer("graphify") self.assertEqual(result.completeness, lifecycle.COMPLETE) self.assertEqual(result.indexed_files, 3) - names = self.archived_names((self.root / "graph.bin").read_bytes()) + names = self.archived_names(self.artifacts[-1].read_bytes()) self.assertIn("graph.bin", names) def archived_names(self, artifact: bytes) -> list[str]: @@ -540,10 +552,9 @@ def test_collecting_the_same_state_twice_produces_the_same_bytes(self) -> None: # The manifest binds a digest of the artifact, so two builds of one # commit have to agree on the bytes down to the archive metadata. self.run_indexer("graphify") - first = (self.root / "graph.bin").read_bytes() - (self.root / "graph.bin").unlink() self.run_indexer("graphify") - self.assertEqual(first, (self.root / "graph.bin").read_bytes()) + first, second = (artifact.read_bytes() for artifact in self.artifacts) + self.assertEqual(first, second) def test_packing_is_bounded_by_the_archive_and_not_by_the_file_sizes(self) -> None: """Empty files are not free: their headers are bytes this process holds. @@ -692,7 +703,7 @@ def fake_run(argv, **kwargs): with self.assertRaises(ContextError): indexer(request) self.assertEqual(launched, []) - self.assertFalse((self.root / "graph.bin").exists()) + self.assertFalse(request.output_path.exists()) def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): From 1b7ead55ea74e29233d358c0f80b5ba01410b3a7 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 03:57:27 -0700 Subject: [PATCH 10/25] context-graph: bind committed objects, contain overruns, bound what runs Three P2 findings from the 826b105 audit. Replacement objects: a refs/replace entry substitutes one object's bytes for another's on every ordinary Git read, so a census and a materialization could bind content the recorded commit and tree do not contain -- and removing the replacement afterwards left graph_status reporting current, because staleness compares object names alone. The shared Git environment now carries GIT_NO_REPLACE_OBJECTS=1, so both invocation paths read the committed objects. Extraction overruns: subprocess.run's timeout kills only the immediate child, which under a launcher such as sandbox-exec is the launcher rather than the indexer. Workers outlived the timeout and kept writing into a scratch directory the build deletes as soon as it reports the failure. Extraction now leads its own session; an overrun signals the whole group and reports only once nothing is left running. Options bound: the bound was counted on the raw pin list while __post_init__ prepends two required flags, so a 15- or 16-token pin passed validation and normalized into metadata load_pin rejected -- a published generation whose manifest is invalid the moment anything reloads it, having already pruned the last usable one. The bound now applies to the normalized tuple, which is a fixed point, so a pin that validates round-trips through the manifest it is recorded in. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 20 ++ src/code_mower/context_graph_lifecycle.py | 164 +++++++++++++-- tests/test_context_graph_lifecycle.py | 232 ++++++++++++++++++++-- 3 files changed, 386 insertions(+), 30 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 71d1883c..3c2a8a02 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -131,6 +131,13 @@ as the complete list of permitted transports. `protocol.allow=never` travels on the command line because that is the only level that outranks the repository's own `.git/config`, which belongs to the untrusted checkout and is always read. +The same environment carries `GIT_NO_REPLACE_OBJECTS=1`. A `refs/replace` entry +substitutes one object's bytes for another's on every ordinary read, so without +it a census and a materialization could bind content the commit and tree the +manifest records do not contain — and removing the replacement afterwards would +leave `status` still reporting `current`, because staleness is decided by +comparing object names. + That still leaves the repository *shape* that makes a read reach out at all, so **a build refuses a partial clone outright**. Where `extensions.partialclone` or a promisor remote is configured, `ls-tree` and `cat-file` can fetch a missing @@ -224,6 +231,16 @@ tracked-content budget nor the artifact one. Inheriting them is not the alternative: diagnostics can echo indexed source, and the process that launched the build may be writing a machine-readable report to its own stdout. +**Extraction runs in its own process group, and a run that overruns is stopped +as a group.** The adapter waits on the child itself rather than handing the run +to `subprocess.run`, whose timeout kills only the immediate child: an indexer +that started workers — and under a launcher such as `sandbox-exec` the direct +child is the launcher, not the indexer — would otherwise leave them running, +still holding CPU and still writing into a scratch directory the build deletes +as soon as it reports the failure. The group gets `SIGTERM`, a short grace +period, then `SIGKILL`, and the timeout is reported only once nothing is left +running. A new session is safe here precisely because no stream is inherited. + The subcommand, the state-directory names, and the report counters are constants in one place in `context_graph_lifecycle.py`. They encode the interface as the evaluation recorded it; the first installation against a real pinned release @@ -296,6 +313,9 @@ marker, or a distribution without an artifact digest: `options` is optional and may name extra provider flags, such as `--max-workers`. The two restrictions above are added whether or not the file lists them; listing them changes nothing, and contradicting them is refused. +The list is bounded, and the bound is counted on the options as they will +actually run — the two required flags included — so a pin that passes validation +always round-trips through the manifest it is recorded in. `--indexer` is the path to a provider CLI the operator has **already** installed. This repository does not download, install, or resolve one, which is diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index da870c1c..766e11fc 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -43,6 +43,7 @@ import re import io import shutil +import signal import subprocess import sys import tarfile @@ -254,6 +255,15 @@ def _size(value: Any, maximum: int) -> int: #: refused rather than silently overridden by argument order. _CONFLICTING_EXTRACT_OPTIONS = frozenset({"--cluster", "--no-code-only"}) +#: How many options an extraction may carry, counted on the normalized tuple -- +#: the one the launcher passes and the manifest records -- rather than on what a +#: pin file happened to spell. Counting the raw list instead would let a pin +#: pass validation and then normalize into a value its own ``as_metadata`` could +#: no longer be read back through: a build could publish a generation whose +#: manifest is rejected the moment anything reloads it, pruning the last usable +#: generation in favour of one nothing can read. +MAX_EXTRACT_OPTIONS = 16 + def _extraction_options(options: Iterable[str]) -> tuple[str, ...]: """The options every extraction runs with: the required ones, then the pin's. @@ -263,6 +273,11 @@ def _extraction_options(options: Iterable[str]) -> tuple[str, ...]: still launches a restricted run. They are prepended into the pin itself rather than added at the call site, so the manifest records what actually ran instead of what was asked for. + + The result is a fixed point: the required flags are dropped from the input + wherever they appear and re-prepended exactly once, so normalizing an + already-normalized tuple returns it unchanged and a pin round-trips through + ``as_metadata`` and :func:`load_pin` without changing length or meaning. """ extra: list[str] = [] for option in options: @@ -279,7 +294,10 @@ def _extraction_options(options: Iterable[str]) -> tuple[str, ...]: ) continue extra.append(option) - return (*_REQUIRED_EXTRACT_OPTIONS, *extra) + normalized = (*_REQUIRED_EXTRACT_OPTIONS, *extra) + if len(normalized) > MAX_EXTRACT_OPTIONS: + raise ContextError("local graph provider options must be a bounded list") + return normalized @dataclass(frozen=True) @@ -328,7 +346,10 @@ def load_pin(source: Mapping[str, Any]) -> GraphifyPin: if not isinstance(version, str) or not _VERSION.fullmatch(version): raise ContextError("local graph provider pin must name one exact released version") options = source.get("options", []) - if not isinstance(options, list) or len(options) > 16: + # A cheap guard on the untrusted list before any of it is copied; the + # binding bound is applied to the normalized tuple in ``_extraction_options`` + # below, which is what the launcher and the manifest actually carry. + if not isinstance(options, list) or len(options) > MAX_EXTRACT_OPTIONS: raise ContextError("local graph provider options must be a bounded list") return GraphifyPin( distribution=_identifier(source.get("distribution")), @@ -388,6 +409,13 @@ def git_environment() -> dict[str, str]: partial clone fetching a missing object mid-read, and an empty ``GIT_ALLOW_PROTOCOL`` leaves no transport on the allowlist, so a fetch that was somehow attempted anyway has nothing to attempt it over. + + ``GIT_NO_REPLACE_OBJECTS`` is the third: a ``refs/replace`` entry in the + checkout substitutes one object's bytes for another's on every read, so a + census and a materialization would bind content that the commit and tree + this manifest records do not contain. Deleting the replacement afterwards + would leave ``graph_status`` reporting ``current`` for a graph of bytes + that revision never had, because it compares object names and nothing else. """ environment = dict(_NETWORK_DENY) for name in _ENVIRONMENT_ALLOWLIST: @@ -400,6 +428,7 @@ def git_environment() -> dict[str, str]: GIT_ATTR_NOSYSTEM="1", GIT_OPTIONAL_LOCKS="0", GIT_NO_LAZY_FETCH="1", + GIT_NO_REPLACE_OBJECTS="1", # An empty allowlist, not an absent one: git treats the variable as the # complete set of permitted transports, so "" permits none. GIT_ALLOW_PROTOCOL="", @@ -922,6 +951,109 @@ def _pack_state(state_directory: Path) -> bytes: return buffer.getvalue() +#: How long one extraction may run. A bound on the wall clock a build can cost, +#: not a guess at how long a real one takes. +EXTRACTION_TIMEOUT_SECONDS = 900 + +#: How long a timed-out provider group is given to exit on its own terms before +#: the group is killed outright. Short: the run has already exceeded its whole +#: time budget, and the caller is about to delete the directory these processes +#: are writing into. +_TERMINATION_GRACE_SECONDS = 5.0 + +#: How long to wait for a killed process to be reaped. ``SIGKILL`` is not +#: refusable, so this bounds a wait on the kernel rather than on the child. +_REAP_TIMEOUT_SECONDS = 10.0 + + +def _signal_group(group: int, number: int) -> None: + try: + os.killpg(group, number) + except OSError: + # Already gone, or never ours to signal. Either way there is nothing + # left to stop, and a failure here must not mask the timeout. + pass + + +def _terminate_process_group(child: subprocess.Popen[bytes]) -> None: + """Stop a timed-out provider and everything it started. + + ``subprocess.run``'s own timeout kills the immediate child only. A provider + that forks workers -- and under a launcher such as ``sandbox-exec`` the + process that is signalled may be the launcher rather than the indexer -- + would leave those workers running, still holding CPU and still writing into + a scratch directory the build is about to delete. The child leads its own + session, so one signal to its group reaches every descendant that has not + deliberately left it. + """ + try: + group = os.getpgid(child.pid) + except OSError: + group = None + if group is None or group == os.getpgid(0): + # The child never reached a group of its own. Kill what can be named + # directly rather than signalling the group this process is in. + child.kill() + _reap(child) + return + _signal_group(group, signal.SIGTERM) + try: + child.wait(timeout=_TERMINATION_GRACE_SECONDS) + except subprocess.TimeoutExpired: + pass + # Unconditionally, and after the direct child has been waited for: that one + # exiting says nothing about workers it started, and this is the last moment + # anything can stop them. + _signal_group(group, signal.SIGKILL) + _reap(child) + + +def _reap(child: subprocess.Popen[bytes]) -> None: + try: + child.wait(timeout=_REAP_TIMEOUT_SECONDS) + except subprocess.TimeoutExpired: # pragma: no cover - a killed child is reapable + pass + + +def _run_contained( + argv: Sequence[str], + *, + environment: Mapping[str, str], + cwd: str, + timeout: float, +) -> int: + """Run one child in its own process group, killing the group on timeout. + + Raises :class:`subprocess.TimeoutExpired` once the group has been stopped, + so a caller reports the timeout only after there is nothing left running. + """ + with subprocess.Popen( # noqa: S603 - argv is a resolved executable and validated options + list(argv), + # Neither stream is read, and neither may be buffered: a provider that + # logs its progress would otherwise accumulate unbounded output in this + # process for up to the timeout, outside both the tracked-content and + # artifact budgets. The streams are discarded at the kernel rather than + # inherited, because provider diagnostics can echo indexed source and + # this process may be writing a machine-readable report. ``stdin`` goes + # the same way: the child has no operator to prompt. + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + env=dict(environment), + cwd=cwd, + # A new session, so the child leads a process group that can be + # signalled as a unit and that this process is not a member of. Safe + # only because no stream is inherited: nothing here needs a controlling + # terminal. + start_new_session=True, + ) as child: + try: + return child.wait(timeout=timeout) + except subprocess.TimeoutExpired: + _terminate_process_group(child) + raise + + def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult]: """Run a pinned provider CLI over the materialized copy, without a network. @@ -954,27 +1086,21 @@ def run(request: IndexRequest) -> IndexResult: # it should be readable here without trusting a constructor elsewhere. options = _extraction_options(request.pin.options) try: - completed = subprocess.run( + returncode = _run_contained( [*sandbox, command, _PROVIDER_EXTRACT, *options], - check=False, - # Neither stream is read, and neither may be buffered: a - # provider that logs its progress would otherwise accumulate - # unbounded output in this process for up to the timeout, - # outside both the tracked-content and artifact budgets. The - # streams are discarded at the kernel rather than inherited, - # because provider diagnostics can echo indexed source and this - # process may be writing a machine-readable report. ``stdin`` - # goes the same way: the child has no operator to prompt. - stdin=subprocess.DEVNULL, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - env=dict(request.environment), + environment=request.environment, cwd=str(request.source_root), - timeout=900, + timeout=EXTRACTION_TIMEOUT_SECONDS, ) + except subprocess.TimeoutExpired: + # Raised only once the whole process group has been stopped, so the + # caller may delete the scratch directory without racing a worker. + raise ContextError( + "local graph provider exceeded its time budget; no generation was published" + ) from None except (OSError, subprocess.SubprocessError): raise ContextError("local graph provider could not be run from its pinned install") from None - if completed.returncode != 0: + if returncode != 0: raise ContextError("local graph provider failed; no generation was published") state_directory = _provider_state_directory(request.source_root) result = _read_completeness(_provider_report(state_directory)) @@ -1723,6 +1849,7 @@ def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[ "ARTIFACT_NAME", "BuildManifest", "COMPLETE", + "EXTRACTION_TIMEOUT_SECONDS", "GenerationStatus", "GraphStateRoot", "GraphifyPin", @@ -1731,6 +1858,7 @@ def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[ "MANIFEST_SCHEMA", "MAX_ARTIFACT_BYTES", "MAX_ARTIFACT_ENTRIES", + "MAX_EXTRACT_OPTIONS", "MAX_TRACKED_FILES", "PARTIAL", "TrackedCensus", diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index f846a9ef..28d85f4a 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -25,6 +25,7 @@ import sys import tarfile import tempfile +import time import unittest from datetime import datetime, timezone from pathlib import Path @@ -80,6 +81,41 @@ def make_repository(root: Path) -> Path: return repository +class FakeChild: + """Enough of ``subprocess.Popen`` to stand in for the provider process. + + The adapter no longer hands the run to ``subprocess.run``: it opens the + child itself so the child can lead its own process group and the whole + group can be stopped if the run overruns. A stand-in is therefore a context + manager that gets waited on, and a ``CompletedProcess`` no longer describes + what the launch produces. + + ``pid`` is deliberately never a live process. A stand-in that carried a real + pid -- this process's own, say -- would have a test signalling a process + group it is itself a member of. + """ + + def __init__(self, returncode: int = 0, *, overruns: bool = False) -> None: + self.returncode = returncode + self.pid = -1 + self.killed = False + self._overruns = overruns + + def __enter__(self) -> "FakeChild": + return self + + def __exit__(self, *exception: object) -> bool: + return False + + def wait(self, timeout: float | None = None) -> int: + if self._overruns: + raise subprocess.TimeoutExpired("graphify", timeout or 0) + return self.returncode + + def kill(self) -> None: + self.killed = True + + def recording_indexer(payload: bytes = b"graph-bytes", *, completeness: str = lifecycle.COMPLETE, seen: list | None = None, indexed_files: int = 0): """An indexer that writes a fixed artifact and records what it was shown.""" @@ -151,6 +187,47 @@ def test_rejects_options_that_undo_the_extraction_restrictions(self) -> None: "options": [option]} ) + def test_the_options_bound_is_applied_to_what_actually_runs(self) -> None: + """The bound counts the normalized tuple, not what the file spelled. + + Counting the raw list let a pin pass validation and then normalize into + one option too many, so its own ``as_metadata`` could no longer be read + back: a build could publish a generation whose manifest is invalid the + moment anything reloads it, pruning the last usable generation for one + nothing can read. The bound now refuses it wherever the pin is made. + """ + source = { + "distribution": "graphifyy", + "version": "0.9.58", + "wheel_sha256": "b" * 64, + "options": [f"--flag-{index}" for index in range(lifecycle.MAX_EXTRACT_OPTIONS - 1)], + } + with self.assertRaises(ContextError): + lifecycle.load_pin(source) + with self.assertRaises(ContextError): + lifecycle.GraphifyPin( + distribution="graphifyy", + version="0.9.58", + wheel_sha256="b" * 64, + options=tuple(source["options"]), + ) + + def test_a_pin_at_the_bound_round_trips_through_its_own_metadata(self) -> None: + """Normalization is a fixed point, so a published manifest can be reread. + + The required flags are dropped wherever they appear and re-prepended + exactly once, so a pin holding the most options it may hold reloads to + an equal pin rather than growing by two each time it is written out. + """ + extra = [f"--flag-{index}" for index in range(lifecycle.MAX_EXTRACT_OPTIONS - 2)] + pin = lifecycle.load_pin( + {"distribution": "graphifyy", "version": "0.9.58", "wheel_sha256": "b" * 64, + "options": extra} + ) + self.assertEqual(len(pin.options), lifecycle.MAX_EXTRACT_OPTIONS) + self.assertEqual(lifecycle.load_pin(pin.as_metadata()), pin) + self.assertEqual(lifecycle.load_pin(pin.as_metadata()).options, pin.options) + def test_rejects_ranges_and_unpinned_shapes(self) -> None: """A range, a marker, or a missing digest lets a build drift silently.""" for version in (">=0.9", "0.9.*", "latest", "", "0.9.58; python_version>'3'"): @@ -465,7 +542,7 @@ def run_indexer( # because the stream arrangement is as much of the contract as it is. self.launch_options: list[dict[str, object]] = [] - def fake_run(argv, **kwargs): + def fake_popen(argv, **kwargs): recorded.append(list(argv)) self.launch_options.append(dict(kwargs)) if state_directory: @@ -474,13 +551,13 @@ def fake_run(argv, **kwargs): (written / "graph.bin").write_bytes(b"graph-bytes") if report is not None: (written / "manifest.json").write_text(json.dumps(report), encoding="utf-8") - return subprocess.CompletedProcess(argv, 0, b"", b"") + return FakeChild() with mock.patch.object(lifecycle, "network_sandbox_command", lambda: sandbox): indexer = lifecycle.subprocess_indexer(executable) # Patched only around the launch, so the Git calls a build makes are # never intercepted by this stand-in. - with mock.patch.object(subprocess, "run", fake_run): + with mock.patch.object(subprocess, "Popen", fake_popen): result = indexer(request) return recorded[0], result @@ -508,6 +585,17 @@ def test_the_provider_is_given_no_stream_this_process_has_to_hold(self) -> None: self.assertEqual(options["stderr"], subprocess.DEVNULL) self.assertEqual(options["stdin"], subprocess.DEVNULL) + def test_the_provider_leads_its_own_process_group(self) -> None: + """A group, so a timed-out run can be stopped as a whole. + + The signal that ends an overrun has to reach workers the provider + started -- under a launcher such as ``sandbox-exec`` the direct child is + the launcher, not the indexer -- and a group is the only handle on them + this process has. Safe only because no stream is inherited. + """ + self.launched_argv("graphify") + self.assertIs(self.launch_options[0]["start_new_session"], True) + def test_the_provider_is_invoked_through_its_documented_extract_interface(self) -> None: # The interface the adopt decision evaluated, recorded in # docs/graphify-evaluation.md as ``extract`` plus options. An @@ -607,16 +695,16 @@ def test_a_run_that_left_no_report_is_partial_rather_than_complete(self) -> None def test_an_unparseable_report_is_partial_rather_than_complete(self) -> None: request = self.request() - def fake_run(argv, **kwargs): + def fake_popen(argv, **kwargs): written = request.source_root / ".graph" written.mkdir(exist_ok=True) (written / "graph.bin").write_bytes(b"graph-bytes") (written / "manifest.json").write_bytes(b"{not json") - return subprocess.CompletedProcess(argv, 0, b"", b"") + return FakeChild() with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "run", fake_run): + with mock.patch.object(subprocess, "Popen", fake_popen): result = indexer(request) self.assertEqual(result.completeness, lifecycle.PARTIAL) @@ -663,19 +751,19 @@ def test_an_oversized_report_is_refused_without_being_read_whole(self) -> None: request = self.request() oversized = b'{"complete": true, "code_files": 3, "pad": "' + b"x" * lifecycle.MAX_MANIFEST_BYTES + b'"}' - def fake_run(argv, **kwargs): + def fake_popen(argv, **kwargs): written = request.source_root / ".graphify" written.mkdir(exist_ok=True) (written / "graph.bin").write_bytes(b"graph-bytes") (written / "manifest.json").write_bytes(oversized) - return subprocess.CompletedProcess(argv, 0, b"", b"") + return FakeChild() def refuse_whole_file_read(self: Path) -> bytes: raise AssertionError(f"{self} was read whole") with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "run", fake_run): + with mock.patch.object(subprocess, "Popen", fake_popen): with mock.patch.object(Path, "read_bytes", refuse_whole_file_read): result = indexer(request) self.assertEqual(result.completeness, lifecycle.PARTIAL) @@ -693,18 +781,45 @@ def test_extraction_refuses_to_run_over_pre_existing_provider_state(self) -> Non (request.source_root / ".graphify" / "cache.json").write_text("{}", encoding="utf-8") launched: list[list[str]] = [] - def fake_run(argv, **kwargs): + def fake_popen(argv, **kwargs): launched.append(list(argv)) - return subprocess.CompletedProcess(argv, 0, b"", b"") + return FakeChild() with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "run", fake_run): + with mock.patch.object(subprocess, "Popen", fake_popen): with self.assertRaises(ContextError): indexer(request) self.assertEqual(launched, []) self.assertFalse(request.output_path.exists()) + def test_an_overrun_is_stopped_before_it_is_reported_as_one(self) -> None: + """The error arrives after the group is stopped, not before. + + ``build_graph`` deletes the scratch directory as soon as the adapter + raises. If the report came first, a worker that outlived the timeout + would still be writing into a directory being removed underneath it. + """ + request = self.request() + order: list[str] = [] + + def fake_popen(argv, **kwargs): + return FakeChild(overruns=True) + + def record_termination(child) -> None: + order.append("stopped") + + with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + indexer = lifecycle.subprocess_indexer("graphify") + with mock.patch.object(subprocess, "Popen", fake_popen): + with mock.patch.object(lifecycle, "_terminate_process_group", record_termination): + with self.assertRaises(ContextError) as raised: + indexer(request) + order.append("reported") + self.assertEqual(order, ["stopped", "reported"]) + self.assertIn("time budget", str(raised.exception)) + self.assertFalse(request.output_path.exists()) + def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): with self.assertRaises(ContextError): @@ -732,6 +847,68 @@ def test_an_unnamed_provider_is_refused(self) -> None: lifecycle._resolved_executable("") +@unittest.skipUnless(hasattr(os, "killpg"), "process groups are a POSIX facility") +class ExtractionOverrunTests(unittest.TestCase): + """What a timed-out provider leaves running, proved against real processes. + + An assertion about which signal was sent would have passed on code that + signalled only the direct child, which is the whole defect: an indexer that + forks workers -- and a launcher such as ``sandbox-exec``, where the direct + child is the launcher rather than the indexer -- outlives that signal and + keeps writing into a scratch directory the build is about to delete. + """ + + #: Stands in for a provider that starts a worker and then overruns. The + #: worker's pid is written where the test can read it, because the point is + #: what happens to a process this module never had a handle on. + PROVIDER = """ +import subprocess +import sys +import time + +worker = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(300)"]) +with open(sys.argv[1], "w") as handle: + handle.write(str(worker.pid)) +time.sleep(300) +""" + + def reaped(self, pid: int, *, within: float = 15.0) -> bool: + deadline = time.monotonic() + within + while time.monotonic() < deadline: + try: + os.kill(pid, 0) + except OSError: + return True + time.sleep(0.05) + return False + + def test_a_timed_out_run_takes_the_workers_it_started_with_it(self) -> None: + with tempfile.TemporaryDirectory() as directory: + recorded = Path(directory) / "worker.pid" + with self.assertRaises(subprocess.TimeoutExpired): + lifecycle._run_contained( + [sys.executable, "-c", self.PROVIDER, str(recorded)], + environment={"PATH": os.environ.get("PATH", "")}, + cwd=directory, + timeout=3.0, + ) + worker = int(recorded.read_text(encoding="utf-8")) + self.assertNotEqual(worker, os.getpid()) + self.assertTrue(self.reaped(worker), "a worker outlived the run that started it") + + def test_a_run_that_finishes_in_time_reports_its_own_status(self) -> None: + # The containment is not a behaviour change for an ordinary run: the + # exit status still comes back, and nothing is signalled. + with tempfile.TemporaryDirectory() as directory: + returncode = lifecycle._run_contained( + [sys.executable, "-c", "raise SystemExit(3)"], + environment={"PATH": os.environ.get("PATH", "")}, + cwd=directory, + timeout=60.0, + ) + self.assertEqual(returncode, 3) + + class BuildAndPublishTests(TemporaryWorkspace): def test_manifest_binds_every_required_fact(self) -> None: manifest = self.build() @@ -1031,6 +1208,37 @@ def test_git_children_get_no_lazy_fetch_and_no_transport(self) -> None: self.assertEqual(environment["GIT_CONFIG_GLOBAL"], os.devnull) self.assertEqual(environment["GIT_CONFIG_SYSTEM"], os.devnull) + def test_git_children_ignore_replacement_objects(self) -> None: + self.assertEqual(lifecycle.git_environment()["GIT_NO_REPLACE_OBJECTS"], "1") + + def test_a_replacement_object_cannot_substitute_committed_content(self) -> None: + """``refs/replace`` changes what a read returns, not what a commit names. + + Every ordinary Git read honours a replacement, so a census and a + materialization would bind bytes the recorded commit and tree do not + contain -- and deleting the replacement afterwards would leave + ``graph_status`` still reporting ``current``, because it compares object + names and nothing else. The unguarded read is asserted first so this + cannot pass by the replacement quietly not applying. + """ + tracked = "example_pkg/config.py" + committed = (self.repository / tracked).read_text(encoding="utf-8") + blob = git(self.repository, "rev-parse", f"HEAD:{tracked}").strip() + decoy = self.root / "decoy.py" + decoy.write_text("VALUE = 'substituted'\n", encoding="utf-8") + substitute = git(self.repository, "hash-object", "-w", str(decoy)).strip() + git(self.repository, "replace", blob, substitute) + self.assertEqual(git(self.repository, "cat-file", "blob", blob), decoy.read_text(encoding="utf-8")) + + census = lifecycle.read_tracked_census(self.repository, self.commit()) + entry = next(item for item in census.entries if item.path == tracked) + # ``ls-tree --long`` reports the replacement's size while still naming + # the original blob, so the census is bound too, not only the content. + self.assertEqual(entry.size, len(committed.encode("utf-8"))) + destination = self.root / "materialized" + lifecycle.materialize_tracked_files(self.repository, census, destination) + self.assertEqual((destination / tracked).read_text(encoding="utf-8"), committed) + def test_git_children_inherit_no_ambient_secret(self) -> None: with mock.patch.dict(os.environ, {"AWS_SECRET_ACCESS_KEY": "not-a-real-secret"}): self.assertNotIn("AWS_SECRET_ACCESS_KEY", lifecycle.git_environment()) From 56485604eba2cd42a76e13bf06450165490476c6 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 04:10:11 -0700 Subject: [PATCH 11/25] context-graph: prove isolation at the listener, resolve state, bound the census MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from the `1b7ead5` audit. [P1] The sandbox probe classified containment from the child's errno, so bubblewrap -- which brings up loopback inside its own network namespace -- looked like a host that had reached the network stack and was rejected. On Linux hosts where bwrap is permitted but standalone unshare is restricted, every build failed, including after the remedy the documentation names. The verdict now comes from a listener this process really holds open: a candidate passes only when the child reported it could not connect and nothing arrived. An unsandboxed control child runs first and must reach that listener, so "could not connect" is never read as containment on a host where connecting was impossible to begin with. [P2] State placement was checked lexically, so `--state-dir /outside/link/state` passed while `/outside/link` pointed inside a repository; the O_NOFOLLOW opens cover only the final component of each directory this module creates. The refusal now checks the resolved path as well. Symlinked ancestors are resolved rather than rejected, because ordinary private roots have them. [P2] The census bounded only what it materialized, so a repository of symlinks, submodules, or committed provider state could record more skipped paths than a manifest may carry -- publishing, pruning the previous usable generation, and then reading back invalid. Skipped entries are bounded where they are collected, and publication now reads the serialized manifest back through `load_manifest` before touching any state. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 41 ++++- src/code_mower/context_graph_lifecycle.py | 175 ++++++++++++++++------ tests/test_context_graph_lifecycle.py | 137 ++++++++++++++++- 3 files changed, 296 insertions(+), 57 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 3c2a8a02..ce11a158 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -47,7 +47,11 @@ provenance at all. If Code Mower does not bind the revision, nothing does. the provider resume from a cache built over content this build never saw, and let the adapter collect tracked repository bytes as if the provider had just produced them. The census digest covers mode, blob name, size and path - for every entry in sorted order. + for every entry in sorted order. Both halves of the census are bounded as + they are collected: the file-count budget covers what is materialized, and a + matching budget covers what is skipped, because a repository of symlinks, + submodules, or committed provider state grows the skipped list without adding + a single entry to the other one. 3. **Materialize into private state.** Each blob is written into a fresh 0700 directory as a 0600 file. Untracked and ignored files have no path into the graph because they are never written, rather than because something filtered @@ -59,7 +63,12 @@ provenance at all. If Code Mower does not bind the revision, nothing does. secret variable is excluded by default because the list names what is kept, not what is dropped. The network boundary is separate and is described below; the emptied proxy variables are hygiene, not that boundary. -5. **Publish atomically.** The generation is assembled under a staging name, +5. **Publish atomically.** The manifest is serialized and read back through the + same validation every consumer applies before anything is written: a + manifest this process can write but no reader can load would otherwise + become `current`, prune the generation that worked, and read back `invalid` + on the next status — a build reporting success while destroying the only + usable graph. Then the generation is assembled under a staging name, fsynced, renamed into `generations/`, and only then does the `current` pointer start naming it. A reader sees the whole previous generation or the whole new one. @@ -95,11 +104,21 @@ behind an argv prefix that denies it sockets at the operating-system level — namespace via `unshare --net` on Linux. No mechanism is trusted on its name. Each candidate is accepted only after a -probe child launched behind it has been *observed* failing to open a TCP -connection with a denial — `EPERM`, `ENETUNREACH`, and the like. A *refused* -connection is the failure case: it proves the syscall reached the network stack, -so the candidate is rejected. The result is cached for the process, since it is -a property of the host. +probe child launched behind it has been *observed* failing to reach a TCP +listener this process is really holding open on loopback. The verdict is taken +at the listener, not from the child's errno: a network namespace brings up its +own loopback, so a correctly contained child sees the same `ECONNREFUSED` that +an unconfined child sees from an unused host port. Those two are +indistinguishable at the child and obvious at the listener, which either +accepted a connection or did not. A candidate passes only when the child +reported that it could not connect *and* nothing arrived, so a launcher that +never started its child cannot pass for a boundary. + +An unsandboxed control child runs first and must reach the listener. If it +cannot — no probe interpreter, loopback unavailable — then "could not connect" +proves nothing about any candidate, every candidate would pass, and the probe +refuses outright instead. The result is cached for the process, since it is a +property of the host. A host where no candidate passes gets no build. `subprocess_indexer()` raises before a single blob is materialized, and `context-graph doctor` reports the @@ -337,7 +356,13 @@ name keeps its `PATH` lookup. Directories are 0700 and files 0600. `` is derived from the resolved checkout path, so two worktrees of the same repository get separate state and can never read each other's generations. State is refused inside any Git -repository, which is the enforcement half of adoption condition 2. +repository, which is the enforcement half of adoption condition 2. The refusal +is checked on the resolved path as well as the given one: `--state-dir +/outside/link/state` names no repository in its own spelling while +`/outside/link` points inside one, and the `O_NOFOLLOW` opens cover only the +final component of each directory this module creates. Symlinked ancestors are +resolved rather than rejected — ordinary private roots have them, macOS reaches +`/tmp` through `/private/tmp`. ## What this does not do diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 766e11fc..e598f837 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -25,9 +25,10 @@ * **Deny the network in the kernel, not by request.** Emptying proxy variables only redirects a client that chooses to honour them. The provider is launched inside an OS sandbox that refuses sockets outright, and the sandbox - is accepted only after a probe child has been observed failing to connect. A - host that offers no such mechanism gets a refused build, not an unconfined - provider. + is accepted only after a probe child has been observed failing to reach a + socket this process is really listening on -- observed at the listener, not + believed from the child's errno. A host that offers no such mechanism gets a + refused build, not an unconfined provider. Nothing here installs, imports, or requires a graph package. The indexer is an injected callable, so the whole lifecycle is provable offline; the bundled @@ -44,6 +45,7 @@ import io import shutil import signal +import socket import subprocess import sys import tarfile @@ -77,6 +79,14 @@ #: of a checkout has no honest reason to hold more entries than the checkout #: had files. MAX_ARTIFACT_ENTRIES = MAX_TRACKED_FILES +#: How many skipped paths one census may record. The file-count budget above +#: bounds only what is materialized, so a repository of symlinks, submodules, or +#: committed provider state passes it while the skipped list grows without +#: limit. A manifest's ``skipped_paths`` is validated against this bound on +#: every read, so a census that could exceed it would build a generation that +#: publishes, prunes its predecessor, and then reads back ``invalid``. Bounded +#: where the entries are collected, before any of that happens. +MAX_SKIPPED_PATHS = MAX_TRACKED_FILES #: Only ordinary blobs are materialized. A symlink (``120000``) can name a #: target outside the checkout and a gitlink (``160000``) names a commit in @@ -137,39 +147,39 @@ ("unshare", "--net", "--map-root-user", "--"), ) -#: The probe connects to the discard port on loopback, where a host without a -#: sandbox refuses the connection. Refusal is the *failure* case here: it proves -#: the syscall reached the network stack. Only an outright denial -- no -#: permission, no route, no address family -- proves the child was contained. -#: The probe exits ``7`` only on a denial; every other exit code -- a refused -#: connection, a launcher that could not start, a child that never ran -- means -#: the candidate is not usable as a boundary. -_PROBE_PORT = 9 +#: The probe connects to a socket this process is really listening on, and the +#: verdict is whether the connection *arrived* -- not which errno the child saw. +#: Classifying by errno cannot work: a network namespace brings its own loopback +#: up, so a contained child gets ``ECONNREFUSED`` from an empty namespace while +#: an unconfined child gets ``ECONNREFUSED`` from an unused host port. The two +#: are indistinguishable at the child. They are not indistinguishable at the +#: listener, which either accepts a connection or does not. +#: +#: The probe exits ``7`` when it could not connect and ``3`` when it could; +#: every other code -- a launcher that could not start, a child that never ran +#: -- means the candidate is not usable as a boundary. A candidate is accepted +#: only when the child reported a failed connection *and* nothing reached the +#: listener, so a child that never ran cannot pass for a contained one. _PROBE_DENIED = 7 +_PROBE_REACHED = 3 _DENIAL_PROBE = """ -import errno import socket import sys -DENIED = frozenset({ - errno.EPERM, - errno.EACCES, - errno.ENETUNREACH, - errno.ENETDOWN, - errno.EHOSTUNREACH, - errno.EADDRNOTAVAIL, - errno.EAFNOSUPPORT, - errno.EPROTONOSUPPORT, -}) try: - probe = socket.socket() - probe.settimeout(5) - probe.connect(("127.0.0.1", int(sys.argv[1]))) -except OSError as error: - sys.exit(7 if error.errno in DENIED else 3) + probe = socket.create_connection(("127.0.0.1", int(sys.argv[1])), timeout=5) +except OSError: + sys.exit(7) +probe.close() sys.exit(3) """ +#: How a probe run classifies: the child reached this process's listener, the +#: child ran and could not, or nothing usable happened. +_REACHED = "reached" +_CONTAINED = "contained" +_UNUSABLE = "unusable" + _sandbox_prefix: tuple[str, ...] | None = None _sandbox_probed = False @@ -180,24 +190,61 @@ def _launcher_path(name: str) -> str | None: return shutil.which(name) -def _sandbox_denies_network(prefix: Sequence[str]) -> bool: - """Watch a child under ``prefix`` fail to open a connection, or say no.""" +def _accepted(listener: socket.socket) -> bool: + """Did anything actually connect? Drains one pending connection if so.""" try: - completed = subprocess.run( - [*prefix, sys.executable, "-c", _DENIAL_PROBE, str(_PROBE_PORT)], - check=False, - capture_output=True, - timeout=60, - env={"PATH": os.environ.get("PATH", ""), **_NETWORK_DENY}, - ) - except (OSError, subprocess.SubprocessError): + connection, _ = listener.accept() + except OSError: return False - return completed.returncode == _PROBE_DENIED + connection.close() + return True + + +def _classify_probe(prefix: Sequence[str]) -> str: + """Run the probe under ``prefix`` against a listener in this process.""" + with socket.socket() as listener: + try: + listener.bind(("127.0.0.1", 0)) + listener.listen(1) + except OSError: # pragma: no cover - a host that cannot listen on loopback + return _UNUSABLE + # The child connects and exits; the connection waits in the backlog + # until it is accepted below, so the accept order does not matter. + listener.settimeout(1) + port = listener.getsockname()[1] + try: + completed = subprocess.run( + [*prefix, sys.executable, "-c", _DENIAL_PROBE, str(port)], + check=False, + capture_output=True, + timeout=60, + env={"PATH": os.environ.get("PATH", ""), **_NETWORK_DENY}, + ) + except (OSError, subprocess.SubprocessError): + return _UNUSABLE + if _accepted(listener): + return _REACHED + if completed.returncode == _PROBE_REACHED: + # The child says it connected but nothing arrived here; treat the + # disagreement as a probe that proved nothing rather than as isolation. + return _UNUSABLE + return _CONTAINED if completed.returncode == _PROBE_DENIED else _UNUSABLE + + +def _sandbox_denies_network(prefix: Sequence[str]) -> bool: + """Watch a child under ``prefix`` fail to reach a socket that is really there.""" + return _classify_probe(prefix) == _CONTAINED def _probe_network_sandbox() -> tuple[str, ...] | None: if not sys.executable: # pragma: no cover - a frozen interpreter cannot probe return None + # The control, first: a child with no prefix must reach the listener. If it + # cannot -- no probe interpreter, loopback blocked, sockets unavailable -- + # then "could not connect" proves nothing about any candidate, and every + # candidate would pass for a boundary. Refuse the whole probe instead. + if _classify_probe(()) != _REACHED: + return None for candidate in _SANDBOX_CANDIDATES: launcher = _launcher_path(candidate[0]) if launcher is None: @@ -552,6 +599,12 @@ def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: ) entries: list[TrackedEntry] = [] skipped: list[tuple[str, str]] = [] + + def skip(path: str, reason: str) -> None: + if len(skipped) >= MAX_SKIPPED_PATHS: + raise ContextError("skipped path census exceeds the local graph budget") + skipped.append((path, reason)) + for record in listing.split("\0"): if not record: continue @@ -561,13 +614,13 @@ def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: raise ContextError("local graph build could not read the repository census") mode, kind, blob, raw_size = fields if mode in _SKIPPED_MODES: - skipped.append((path, _SKIPPED_MODES[mode])) + skip(path, _SKIPPED_MODES[mode]) continue if _is_provider_state(path): - skipped.append((path, "provider state")) + skip(path, "provider state") continue if mode not in _REGULAR_MODES or kind != "blob": - skipped.append((path, "unsupported")) + skip(path, "unsupported") continue if len(entries) >= MAX_TRACKED_FILES: raise ContextError("tracked file census exceeds the local graph budget") @@ -1208,7 +1261,7 @@ def load_manifest(payload: Mapping[str, Any]) -> BuildManifest: graph_digest=_digest(payload["graph_digest"]), graph_bytes=_size(payload["graph_bytes"], MAX_ARTIFACT_BYTES), completeness=completeness, - skipped_paths=_size(payload["skipped_paths"], MAX_TRACKED_FILES), + skipped_paths=_size(payload["skipped_paths"], MAX_SKIPPED_PATHS), indexed_files=_size(payload["indexed_files"], MAX_TRACKED_FILES), ) @@ -1304,8 +1357,21 @@ def ensure(self) -> None: self.verify_private(create=True) def _refuse_state_inside_a_repository(self) -> None: - if any((parent / ".git").exists() for parent in (self.path, *self.path.parents)): - raise ContextError("local graph state must stay outside Git repositories") + """No generation may be written inside a repository, by any spelling. + + Checked on the *resolved* path as well as the given one. A lexical walk + alone reads ``--state-dir /outside/link/state`` as being outside every + repository even when ``/outside/link`` points at ``/repo/subdir``, and + the ``O_NOFOLLOW`` opens below would not catch it either: they protect + the final component of each directory this class owns, not an ancestor + somebody else created. Symlinked ancestors are resolved rather than + refused, because ordinary private roots have them -- macOS puts + ``/tmp`` behind ``/private/tmp``. + """ + resolved = Path(os.path.realpath(self.path)) + for candidate in dict.fromkeys((self.path, resolved)): + if any((parent / ".git").exists() for parent in (candidate, *candidate.parents)): + raise ContextError("local graph state must stay outside Git repositories") def _ensure_lock_directory(self) -> None: """Create only what the lock file needs, not the generations tree. @@ -1407,16 +1473,29 @@ def publish(self, manifest: BuildManifest, artifact: bytes) -> BuildManifest: ``current`` start naming it. A crash between the two leaves an unreferenced generation, which ``prune`` removes; it never leaves a pointer to a directory that does not exist. + + Nothing is written until the serialized manifest has been read back + through ``load_manifest`` and measured against the bound a reader + applies. A manifest this process can write but no reader can load would + otherwise publish, move ``current`` onto it, prune the previous usable + generation, and read back ``invalid`` on the next status: a build that + reports success while destroying the only generation that worked. The + check belongs here, at the one boundary every generation crosses, + rather than at each of the places that fill a single field in. """ + serialized = json.dumps( + manifest.to_json(), allow_nan=False, sort_keys=True, separators=(",", ":") + ).encode() + if len(serialized) > MAX_MANIFEST_BYTES: + raise ContextError("local graph manifest exceeds its bound; no generation was published") + if load_manifest(json.loads(serialized)).generation != manifest.generation: + raise ContextError("local graph manifest does not match its generation") self.ensure() staging = self.generations_path / ("." + uuid.uuid4().hex + ".staging") staging.mkdir(mode=0o700) try: _write_private_file(staging / ARTIFACT_NAME, artifact) - _write_private_file( - staging / MANIFEST_NAME, - json.dumps(manifest.to_json(), allow_nan=False, sort_keys=True, separators=(",", ":")).encode(), - ) + _write_private_file(staging / MANIFEST_NAME, serialized) _fsync_directory(staging) final = self.generations_path / manifest.generation os.rename(staging, final) diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 28d85f4a..d8f5cbe1 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -15,6 +15,7 @@ from __future__ import annotations +import dataclasses import hashlib import io import json @@ -27,6 +28,7 @@ import tempfile import time import unittest +import uuid from datetime import datetime, timezone from pathlib import Path from unittest import mock @@ -322,6 +324,31 @@ def test_committing_provider_state_does_not_move_the_census_digest(self) -> None self.assertEqual(before.digest, after.digest) self.assertNotEqual(before.skipped, after.skipped) + def test_skipped_paths_are_bounded_like_materialized_ones(self) -> None: + """The file-count budget bounds what is written, not what is recorded. + + A repository of symlinks, submodules, or committed provider state adds + nothing to ``entries`` and so passes the tracked-file budget however + large it gets, while ``skipped`` grows with it. The manifest's + ``skipped_paths`` is validated against the same bound on every read, so + an unbounded census would publish a generation, prune its predecessor, + and then read back ``invalid``. The refusal belongs here, before a build + has done anything. + """ + links = [f"link-{index}.py" for index in range(3)] + for name in links: + os.symlink("/etc/passwd", self.repository / name) + # Named, not ``add .``: the workspace deliberately holds an untracked + # secret, and this test is about the census, not about committing it. + git(self.repository, "add", *links) + git(self.repository, "commit", "-q", "-m", "many symlinks") + commit, _ = lifecycle.resolve_revision(self.repository) + with mock.patch.object(lifecycle, "MAX_SKIPPED_PATHS", 2): + with self.assertRaises(ContextError): + lifecycle.read_tracked_census(self.repository, commit) + census = lifecycle.read_tracked_census(self.repository, commit) + self.assertEqual(len(census.skipped), 3) + def test_materialization_writes_only_tracked_files(self) -> None: commit, _ = lifecycle.resolve_revision(self.repository) census = lifecycle.read_tracked_census(self.repository, commit) @@ -469,11 +496,40 @@ def launcher(self, exit_code: int) -> str: path.chmod(0o700) return str(path) + def closed_port(self) -> int: + """A loopback port that refuses connections, as an empty namespace does. + + Bound and never listened on, rather than bound and released: a released + ephemeral port can be handed straight back to the listener this test is + trying to prove unreachable. Holding it means the refusal is the one + this test arranged. + """ + held = socket.socket() + self.addCleanup(held.close) + held.bind(("127.0.0.1", 0)) + return held.getsockname()[1] + + def redirecting_launcher(self, port: int) -> str: + """A stand-in for a launcher that gives its child its own loopback. + + This is the shape ``bwrap --unshare-net`` has: the child really runs, + loopback really comes up inside the new namespace, and the connection is + *refused* because the host's listener is not in there with it. The + launcher runs the probe it was handed against a port nothing is on, + which is what the child would have seen. + """ + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + path = Path(directory.name) / "namespace-launcher" + path.write_text(f'#!/bin/sh\nexec "$1" "$2" "$3" {port}\n', encoding="utf-8") + path.chmod(0o700) + return str(path) + def test_a_child_that_reports_a_denial_is_accepted(self) -> None: self.assertTrue(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_DENIED),))) def test_a_child_that_reached_the_network_stack_is_rejected(self) -> None: - self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(3),))) + self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_REACHED),))) def test_a_launcher_that_cannot_start_is_rejected(self) -> None: self.assertFalse(lifecycle._sandbox_denies_network(("/nonexistent/launcher",))) @@ -486,6 +542,36 @@ def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: self.skipTest("no pass-through launcher to test against") self.assertFalse(lifecycle._sandbox_denies_network(passthrough)) + def test_a_refused_connection_inside_a_namespace_is_containment(self) -> None: + """The bubblewrap case: refused by an empty namespace, not by the host. + + Classifying on the child's errno cannot tell that apart from a refusal + by an unused host port, so a probe that only accepted ``EPERM``-shaped + denials rejected working bubblewrap isolation and left such hosts unable + to build at all. The verdict is taken at the listener instead: nothing + arrived, so the child was contained. + """ + self.assertTrue(lifecycle._sandbox_denies_network((self.redirecting_launcher(self.closed_port()),))) + + def test_the_probe_proves_its_own_apparatus_before_trusting_a_refusal(self) -> None: + """No candidate passes if an unsandboxed child cannot reach the listener. + + "Could not connect" only means containment when connecting was possible + in the first place. If the control fails -- no probe interpreter, + loopback unavailable -- every candidate would look like a boundary, so + the whole probe refuses and builds refuse with it. + """ + calls: list[tuple[str, ...]] = [] + + def classify(prefix): + calls.append(tuple(prefix)) + return lifecycle._CONTAINED + + with mock.patch.object(lifecycle, "_classify_probe", classify): + self.assertIsNone(lifecycle._probe_network_sandbox()) + # The control ran, and nothing was probed after it failed. + self.assertEqual(calls, [()]) + #: What a provider that finished leaves behind: a completion claim and counts #: that admit nothing outstanding. @@ -1027,6 +1113,55 @@ def test_state_is_refused_inside_a_git_repository(self) -> None: with self.assertRaises(ContextError): self.build(root=self.repository / ".code-mower-state") + def test_state_is_refused_behind_a_symlink_into_a_repository(self) -> None: + """A lexical ancestor walk does not see the repository through a link. + + ``--state-dir /outside/link/state`` names no repository in its own + spelling, but ``/outside/link`` can point at a directory inside one, and + the ``O_NOFOLLOW`` opens only cover the final component of each + directory this lane creates. Resolved, the path is inside the checkout + and the artifacts would land in it. + """ + inside = self.repository / "subdir" + inside.mkdir() + outside = self.root / "outside" + outside.mkdir() + os.symlink(inside, outside / "link") + with self.assertRaises(ContextError): + self.build(root=outside / "link" / "graph-state") + self.assertFalse((inside / "graph-state").exists()) + + def test_an_ordinary_symlinked_ancestor_is_resolved_not_refused(self) -> None: + # Resolving, not rejecting: private roots legitimately sit behind + # links -- macOS reaches ``/tmp`` through ``/private/tmp`` -- so a + # symlinked ancestor outside any repository must still build. + elsewhere = self.root / "elsewhere" + elsewhere.mkdir() + os.symlink(elsewhere, self.root / "linked-state") + manifest = self.build(root=self.root / "linked-state" / "graph") + self.assertEqual(manifest.completeness, lifecycle.COMPLETE) + + def test_a_manifest_no_reader_could_load_publishes_nothing(self) -> None: + """Publication validates the whole manifest before it touches state. + + A manifest that this process can write but ``load_manifest`` refuses + would otherwise become ``current``, prune the generation that worked, + and read back ``invalid`` on the next status -- a build reporting + success while destroying the only usable graph. + """ + first = self.build() + state = lifecycle.GraphStateRoot(self.repository, root=self.state) + unreadable = dataclasses.replace( + first, + generation=uuid.uuid4().hex, + skipped_paths=lifecycle.MAX_SKIPPED_PATHS + 1, + ) + with self.assertRaises(ContextError): + state.publish(unreadable, b"graph-bytes") + self.assertEqual(state.current_generation(), first.generation) + self.assertEqual(list(lifecycle.iter_generations(self.repository, root=self.state)), [first.generation]) + self.assertEqual(state.read_manifest(first.generation), first) + class StatusFailsClosedTests(TemporaryWorkspace): def test_a_fresh_build_is_current(self) -> None: From de10e308d5a30aefb815264924963902d706e7f6 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 04:23:12 -0700 Subject: [PATCH 12/25] context: bound the census stream and stop the provider on cancellation Codex audit findings at a01054c, plus the owner's supplemental review. [P2 codex:a5e720ca30b8374ee5a9] The census captured the whole `ls-tree` listing before any budget could be checked, so a tree far past MAX_TRACKED_FILES or MAX_SKIPPED_PATHS exhausted memory instead of being refused at the bound. Records now stream NUL-delimited through a bounded buffer; every budget is checked against what has been seen so far, and a consumer that stops early closes the pipe and reaps the reader rather than leaving Git enumerating a tree nobody will read. [P2 codex:d692c569fa67f3bc6134] `_run_contained` cleaned up on TimeoutExpired only. The provider leads its own session, so Ctrl-C never reaches it, and KeyboardInterrupt unwound past a running provider while build_graph deleted the scratch directory underneath it. Every exit from the wait now stops the process group first. Supplemental review: * The sandbox probe accepted any launcher that exited 7, including one that never executed the child -- which would run every provider unconfined. The child now prints the digest of a per-run nonce, and a run without that evidence is unusable whatever its exit code. The synthetic control is inverted to match. * GraphStateRoot canonicalizes its base once, in __init__, so the path the repository check walks is the path every later open, mkdir, lock and removal travels. A symlinked ancestor retargeted after the check is no longer on the route; regression test included. * Added a capability-gated integration control that runs the selected mechanism -- sandbox-exec, bwrap, or unshare -- against a real listener, where the host offers one. Co-Authored-By: Claude Opus 5 (1M context) --- src/code_mower/context_graph_lifecycle.py | 252 +++++++++++++++++----- tests/test_context_graph_lifecycle.py | 182 +++++++++++++++- 2 files changed, 374 insertions(+), 60 deletions(-) diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index e598f837..4dfac82a 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -38,11 +38,13 @@ from __future__ import annotations +import contextlib import hashlib import json import os import re import io +import secrets import shutil import signal import socket @@ -157,15 +159,25 @@ #: #: The probe exits ``7`` when it could not connect and ``3`` when it could; #: every other code -- a launcher that could not start, a child that never ran -#: -- means the candidate is not usable as a boundary. A candidate is accepted -#: only when the child reported a failed connection *and* nothing reached the -#: listener, so a child that never ran cannot pass for a contained one. +#: -- means the candidate is not usable as a boundary. +#: +#: An exit code alone is not evidence that the child ran: a launcher that exits +#: ``7`` without executing anything produces the same code as a contained child, +#: and would be accepted as a boundary while confining nothing. So the child +#: first prints a value only running it can produce -- the digest of a nonce +#: this process generated for this run -- and a run with no such evidence is +#: unusable whatever its exit code. Echoing the argv is not enough: the digest +#: is computed by the child, and the nonce is fresh per run, so neither a +#: launcher that parrots its arguments nor one that replays an earlier probe +#: can produce it. _PROBE_DENIED = 7 _PROBE_REACHED = 3 _DENIAL_PROBE = """ +import hashlib import socket import sys +print(hashlib.sha256(sys.argv[2].encode()).hexdigest(), flush=True) try: probe = socket.create_connection(("127.0.0.1", int(sys.argv[1])), timeout=5) except OSError: @@ -200,8 +212,15 @@ def _accepted(listener: socket.socket) -> bool: return True +def _ran_the_probe(output: bytes, nonce: str) -> bool: + """Did this run's probe child actually execute under the launcher?""" + expected = hashlib.sha256(nonce.encode()).hexdigest() + return expected in output.decode("utf-8", "replace") + + def _classify_probe(prefix: Sequence[str]) -> str: """Run the probe under ``prefix`` against a listener in this process.""" + nonce = secrets.token_hex(16) with socket.socket() as listener: try: listener.bind(("127.0.0.1", 0)) @@ -214,7 +233,7 @@ def _classify_probe(prefix: Sequence[str]) -> str: port = listener.getsockname()[1] try: completed = subprocess.run( - [*prefix, sys.executable, "-c", _DENIAL_PROBE, str(port)], + [*prefix, sys.executable, "-c", _DENIAL_PROBE, str(port), nonce], check=False, capture_output=True, timeout=60, @@ -222,8 +241,15 @@ def _classify_probe(prefix: Sequence[str]) -> str: ) except (OSError, subprocess.SubprocessError): return _UNUSABLE - if _accepted(listener): - return _REACHED + arrived = _accepted(listener) + # Before anything is read from the exit code: a run that cannot show its + # child executed classifies as nothing at all. This is the control that + # keeps "denied" from being the default answer for a launcher that never + # started the probe. + if not _ran_the_probe(completed.stdout, nonce): + return _UNUSABLE + if arrived: + return _REACHED if completed.returncode == _PROBE_REACHED: # The child says it connected but nothing arrived here; treat the # disagreement as a probe that proved nothing rather than as isolation. @@ -527,6 +553,98 @@ def _git(repository: Path, *arguments: str, capture: bool = True, permit_failure return completed.stdout +#: The longest NUL-delimited ``ls-tree`` record a census will hold before it has +#: seen the delimiter that ends it. A record is fixed-width metadata plus one +#: path, and Git's own path ceiling is far under this, so a longer run of bytes +#: means the stream is not the one this build asked for. +MAX_CENSUS_RECORD_BYTES = 16 * 1024 + +#: How much of the listing to read at a time. Small enough that a refusal costs +#: one chunk rather than a whole repository's metadata. +_CENSUS_CHUNK_BYTES = 64 * 1024 + + +def _stop_reader(process: subprocess.Popen[bytes]) -> None: + """Stop a streaming Git child and reap it, whatever the caller is doing. + + Closing the pipe first is what makes an early refusal cheap: Git writes into + a broken pipe and exits on its own, rather than being left to finish + enumerating a tree nobody is going to read. + """ + if process.stdout is not None: + with contextlib.suppress(OSError): + process.stdout.close() + if process.poll() is None: + with contextlib.suppress(OSError): + process.kill() + with contextlib.suppress(subprocess.TimeoutExpired): + process.wait(timeout=_REAP_TIMEOUT_SECONDS) + + +@contextlib.contextmanager +def _git_records(repository: Path, *arguments: str) -> Iterator[Iterator[str]]: + """Yield the NUL-delimited records of one Git child, one at a time. + + ``subprocess.run`` would hold the whole listing in memory before the first + budget could be checked, so a tree far past every census bound would exhaust + this process instead of being refused at the bound. Streaming lets the + consumer stop at the record that breaks its budget; leaving the child to + this context manager means it is terminated there rather than whenever a + generator happens to be collected. + """ + try: + process = subprocess.Popen( + ["git", "-C", str(repository), "--no-optional-locks", *_GIT_SAFETY_OPTIONS, *arguments], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + env=git_environment(), + ) + except (OSError, subprocess.SubprocessError): + raise ContextError("local graph build could not read the target repository") from None + try: + yield _read_records(process) + finally: + _stop_reader(process) + + +def _read_records(process: subprocess.Popen[bytes]) -> Iterator[str]: + stream = process.stdout + assert stream is not None + pending = b"" + while True: + chunk = stream.read(_CENSUS_CHUNK_BYTES) + if not chunk: + break + pending += chunk + while True: + record, delimiter, rest = pending.partition(b"\0") + if not delimiter: + break + if len(record) > MAX_CENSUS_RECORD_BYTES: + raise ContextError("local graph build could not read the repository census") + pending = rest + yield _record_text(record) + # The same bound on what has *not* been delimited yet: a stream with no + # delimiter in it would otherwise grow a chunk at a time forever. + if len(pending) > MAX_CENSUS_RECORD_BYTES: + raise ContextError("local graph build could not read the repository census") + if pending: + # ``-z`` terminates every record, so a trailing remainder is a stream + # that stopped mid-record: a killed child, or output this build did not + # ask for. Either way the census it would produce is incomplete. + raise ContextError("local graph build could not read the repository census") + if process.wait() != 0: + raise ContextError("local graph build could not read the target repository") + + +def _record_text(record: bytes) -> str: + try: + return record.decode("utf-8") + except UnicodeDecodeError: + raise ContextError("local graph build could not read the repository census") from None + + def refuse_lazy_object_fetch(repository: Path) -> None: """Refuse a partial clone, where reading the tree can call out to a remote. @@ -586,51 +704,58 @@ def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: excluded from the census, so it is excluded from the census digest too, and a build over a repository that tracks a ``.graphify`` directory binds a census that says so instead of quietly indexing somebody else's graph. + + The listing is consumed as it arrives rather than captured whole: every + budget here is checked against the records seen so far, so a tree past one + of them is refused at that record, with the reader stopped, instead of + after a repository's worth of metadata has been buffered. """ refuse_lazy_object_fetch(repository) - listing = _git( - repository, - "ls-tree", - "-r", - "-z", - "--long", - "--full-tree", - _object_name(commit), - ) entries: list[TrackedEntry] = [] skipped: list[tuple[str, str]] = [] + total = 0 def skip(path: str, reason: str) -> None: if len(skipped) >= MAX_SKIPPED_PATHS: raise ContextError("skipped path census exceeds the local graph budget") skipped.append((path, reason)) - for record in listing.split("\0"): - if not record: - continue - metadata, _, path = record.partition("\t") - fields = metadata.split() - if len(fields) != 4 or not path: - raise ContextError("local graph build could not read the repository census") - mode, kind, blob, raw_size = fields - if mode in _SKIPPED_MODES: - skip(path, _SKIPPED_MODES[mode]) - continue - if _is_provider_state(path): - skip(path, "provider state") - continue - if mode not in _REGULAR_MODES or kind != "blob": - skip(path, "unsupported") - continue - if len(entries) >= MAX_TRACKED_FILES: - raise ContextError("tracked file census exceeds the local graph budget") - size = int(raw_size) if raw_size.isdigit() else -1 - if not 0 <= size <= MAX_BLOB_BYTES: - raise ContextError("tracked file exceeds the local graph per-file budget") - entries.append(TrackedEntry(mode=mode, blob=_object_name(blob), path=path, size=size)) + with _git_records( + repository, + "ls-tree", + "-r", + "-z", + "--long", + "--full-tree", + _object_name(commit), + ) as records: + for record in records: + if not record: + continue + metadata, _, path = record.partition("\t") + fields = metadata.split() + if len(fields) != 4 or not path: + raise ContextError("local graph build could not read the repository census") + mode, kind, blob, raw_size = fields + if mode in _SKIPPED_MODES: + skip(path, _SKIPPED_MODES[mode]) + continue + if _is_provider_state(path): + skip(path, "provider state") + continue + if mode not in _REGULAR_MODES or kind != "blob": + skip(path, "unsupported") + continue + if len(entries) >= MAX_TRACKED_FILES: + raise ContextError("tracked file census exceeds the local graph budget") + size = int(raw_size) if raw_size.isdigit() else -1 + if not 0 <= size <= MAX_BLOB_BYTES: + raise ContextError("tracked file exceeds the local graph per-file budget") + total += size + if total > MAX_TRACKED_BYTES: + raise ContextError("tracked content exceeds the local graph budget") + entries.append(TrackedEntry(mode=mode, blob=_object_name(blob), path=path, size=size)) entries.sort(key=lambda entry: entry.path) - if sum(entry.size for entry in entries) > MAX_TRACKED_BYTES: - raise ContextError("tracked content exceeds the local graph budget") return TrackedCensus( entries=tuple(entries), skipped=tuple(sorted(skipped)), @@ -1029,7 +1154,7 @@ def _signal_group(group: int, number: int) -> None: def _terminate_process_group(child: subprocess.Popen[bytes]) -> None: - """Stop a timed-out provider and everything it started. + """Stop an abandoned provider and everything it started. ``subprocess.run``'s own timeout kills the immediate child only. A provider that forks workers -- and under a launcher such as ``sandbox-exec`` the @@ -1075,7 +1200,7 @@ def _run_contained( cwd: str, timeout: float, ) -> int: - """Run one child in its own process group, killing the group on timeout. + """Run one child in its own process group, killing the group on any exit. Raises :class:`subprocess.TimeoutExpired` once the group has been stopped, so a caller reports the timeout only after there is nothing left running. @@ -1102,7 +1227,15 @@ def _run_contained( ) as child: try: return child.wait(timeout=timeout) - except subprocess.TimeoutExpired: + except BaseException: + # Every way out of this wait except returning, not the timeout + # alone. An operator's Ctrl-C raises ``KeyboardInterrupt`` here, and + # the provider does not see that signal: it leads its own session, + # so the terminal's SIGINT never reaches it. ``Popen.__exit__`` + # would then wait for a child nobody has asked to stop, while + # ``build_graph`` deletes the scratch directory underneath it. The + # group is stopped first, so unwinding leaves nothing running + # against state that is about to be removed. _terminate_process_group(child) raise @@ -1310,8 +1443,17 @@ def __init__(self, repository: Path, *, root: Path | None = None): base = Path(root) if root is not None else default_context_root() if not base.is_absolute(): raise ContextError("local graph state requires an absolute private directory") + # Canonicalized once, here, and never spelled lexically again: every + # later open, mkdir, lock, and removal travels the path that + # ``_refuse_state_inside_a_repository`` checked. Checking a resolved + # snapshot and then writing through the original spelling would leave a + # symlinked ancestor free to be retargeted in between -- the check + # passes against one directory and the writes land in another. Ordinary + # private roots do have symlinked ancestors (macOS puts ``/tmp`` behind + # ``/private/tmp``), so they are resolved rather than refused. + self.base = Path(os.path.realpath(base)) self.workspace = workspace_id(self.repository) - self.path = base / "graph" / self.workspace + self.path = self.base / "graph" / self.workspace @property def generations_path(self) -> Path: @@ -1359,19 +1501,17 @@ def ensure(self) -> None: def _refuse_state_inside_a_repository(self) -> None: """No generation may be written inside a repository, by any spelling. - Checked on the *resolved* path as well as the given one. A lexical walk - alone reads ``--state-dir /outside/link/state`` as being outside every - repository even when ``/outside/link`` points at ``/repo/subdir``, and - the ``O_NOFOLLOW`` opens below would not catch it either: they protect - the final component of each directory this class owns, not an ancestor - somebody else created. Symlinked ancestors are resolved rather than - refused, because ordinary private roots have them -- macOS puts - ``/tmp`` behind ``/private/tmp``. + A lexical walk alone reads ``--state-dir /outside/link/state`` as being + outside every repository even when ``/outside/link`` points at + ``/repo/subdir``, and the ``O_NOFOLLOW`` opens elsewhere would not catch + it either: they protect the final component of each directory this class + owns, not an ancestor somebody else created. The path walked here is the + canonical one built in ``__init__``, which is also the one every write + goes through, so this check cannot be satisfied by one directory and + then applied to another. """ - resolved = Path(os.path.realpath(self.path)) - for candidate in dict.fromkeys((self.path, resolved)): - if any((parent / ".git").exists() for parent in (candidate, *candidate.parents)): - raise ContextError("local graph state must stay outside Git repositories") + if any((parent / ".git").exists() for parent in (self.path, *self.path.parents)): + raise ContextError("local graph state must stay outside Git repositories") def _ensure_lock_directory(self) -> None: """Create only what the lock file needs, not the generations tree. diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index d8f5cbe1..b5eee40e 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -441,6 +441,65 @@ def test_allowlist_drops_everything_it_does_not_name(self) -> None: self.assertEqual(set(environment) - allowed, set()) +class ListingStreamTests(unittest.TestCase): + """The census consumes its listing as it arrives, bounded, and stops the reader. + + Capturing the whole listing first put a repository's worth of metadata in + this process before any census bound could be checked, so a tree far past + every budget exhausted memory instead of being refused at the budget. The + reader is a stand-in here because a repository large enough to prove the + bound for real would be the thing the bound exists to avoid. + """ + + class Reader: + """Enough of ``Popen`` for the record stream: a pipe and an exit status.""" + + def __init__(self, payload: bytes, returncode: int = 0) -> None: + self.stdout = io.BytesIO(payload) + self.returncode = returncode + self.killed = False + + def wait(self, timeout: float | None = None) -> int: + return self.returncode + + def poll(self) -> int | None: + return self.returncode if self.killed else None + + def kill(self) -> None: + self.killed = True + + def records(self, payload: bytes, returncode: int = 0) -> list[str]: + return list(lifecycle._read_records(self.Reader(payload, returncode))) + + def test_records_are_split_on_the_delimiter(self) -> None: + self.assertEqual(self.records(b"one\0two\0"), ["one", "two"]) + + def test_a_run_of_bytes_past_the_record_bound_is_refused(self) -> None: + # No delimiter, so nothing can be classified and nothing can be + # released: this is exactly the shape that grows without limit. + with self.assertRaises(ContextError): + self.records(b"x" * (lifecycle.MAX_CENSUS_RECORD_BYTES + 1) + b"\0") + + def test_a_stream_that_ends_mid_record_is_refused(self) -> None: + with self.assertRaises(ContextError): + self.records(b"one\0two") + + def test_a_reader_that_failed_is_refused_even_after_a_clean_stream(self) -> None: + with self.assertRaises(ContextError): + self.records(b"one\0", returncode=1) + + def test_a_consumer_that_stops_early_stops_the_reader_with_it(self) -> None: + """A refused census must not leave Git enumerating the rest of the tree.""" + reader = self.Reader(b"one\0two\0") + with mock.patch.object(subprocess, "Popen", lambda *arguments, **keywords: reader): + with self.assertRaises(ContextError): + with lifecycle._git_records(Path("/nonexistent"), "ls-tree") as records: + next(records) + raise ContextError("the consumer reached its budget") + self.assertTrue(reader.stdout.closed) + self.assertTrue(reader.killed) + + CONNECT_PROBE = """ import socket import sys @@ -482,6 +541,23 @@ def test_a_sandboxed_child_cannot_reach_the_listening_socket(self) -> None: self.skipTest("this host offers no OS sandbox that denies a child the network") self.assertEqual(self.connect(sandbox), 1) + def test_the_selected_mechanism_really_contains_a_child_on_this_host(self) -> None: + """The integration control: the real mechanism, not a stand-in for one. + + The synthetic launchers below pin the classification *algorithm* on + every host, including hosts with no sandbox at all. They cannot show + that ``sandbox-exec``, ``bwrap --unshare-net``, or ``unshare --net`` as + this module spells them actually confines anything. Where one of them is + available, this runs it for real: an unconfined child must reach a + listener that is really there, and a child under the selected prefix + must not. + """ + prefix = lifecycle.network_sandbox_command() + if prefix is None: + self.skipTest("this host offers no OS sandbox that denies a child the network") + self.assertEqual(lifecycle._classify_probe(()), lifecycle._REACHED) + self.assertEqual(lifecycle._classify_probe(prefix), lifecycle._CONTAINED) + def launcher(self, exit_code: int) -> str: """A stand-in launcher, so the classifier is pinned on every host. @@ -516,21 +592,50 @@ def redirecting_launcher(self, port: int) -> str: loopback really comes up inside the new namespace, and the connection is *refused* because the host's listener is not in there with it. The launcher runs the probe it was handed against a port nothing is on, - which is what the child would have seen. + which is what the child would have seen. ``$5`` is the nonce, forwarded + so the child can still show it ran: a launcher that swallowed it would + be a launcher that did not run the probe. """ directory = tempfile.TemporaryDirectory() self.addCleanup(directory.cleanup) path = Path(directory.name) / "namespace-launcher" - path.write_text(f'#!/bin/sh\nexec "$1" "$2" "$3" {port}\n', encoding="utf-8") + path.write_text(f'#!/bin/sh\nexec "$1" "$2" "$3" {port} "$5"\n', encoding="utf-8") path.chmod(0o700) return str(path) - def test_a_child_that_reports_a_denial_is_accepted(self) -> None: - self.assertTrue(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_DENIED),))) + def test_a_launcher_that_never_runs_the_child_is_rejected(self) -> None: + """An exit code is not evidence, and the denial code least of all. + + A launcher that exits with the probe's own "could not connect" code + without executing anything confines nothing, and this is the shape a + broken or hostile launcher has. Nothing reaches the listener either -- + nothing ran -- so a classifier reading the exit code alone would accept + it as a boundary and every build would then run its provider + unconfined. The child's per-run evidence is what separates the two. + """ + self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_DENIED),))) def test_a_child_that_reached_the_network_stack_is_rejected(self) -> None: self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_REACHED),))) + def test_evidence_from_another_run_does_not_prove_this_one(self) -> None: + """A replayed transcript is not a child that ran. + + The evidence is the digest of a nonce generated for one run, so a + launcher that printed a previous run's evidence -- or one that parroted + its own argv -- says nothing about this run. + """ + stale = hashlib.sha256(b"an-earlier-nonce").hexdigest() + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + path = Path(directory.name) / "replaying-launcher" + path.write_text( + f"#!/bin/sh\necho {stale}\necho \"$@\"\nexit {lifecycle._PROBE_DENIED}\n", + encoding="utf-8", + ) + path.chmod(0o700) + self.assertFalse(lifecycle._sandbox_denies_network((str(path),))) + def test_a_launcher_that_cannot_start_is_rejected(self) -> None: self.assertFalse(lifecycle._sandbox_denies_network(("/nonexistent/launcher",))) @@ -982,6 +1087,49 @@ def test_a_timed_out_run_takes_the_workers_it_started_with_it(self) -> None: self.assertNotEqual(worker, os.getpid()) self.assertTrue(self.reaped(worker), "a worker outlived the run that started it") + def test_a_cancelled_run_takes_the_workers_it_started_with_it(self) -> None: + """Ctrl-C is not the timeout, and the provider never sees the signal. + + The child leads its own session, so the terminal's SIGINT reaches this + process and not the provider. Cleaning up only on ``TimeoutExpired`` + left ``KeyboardInterrupt`` to unwind past a running provider and its + workers, while ``build_graph`` deleted the scratch directory they were + writing into. Every exit from the wait stops the group now, not just + the one the timeout takes. + """ + original = subprocess.Popen.wait + cancelled: list[bool] = [] + + def wait(child, timeout=None): + if cancelled: + return original(child, timeout=timeout) + cancelled.append(True) + # Interrupt once the provider has a worker to abandon; an interrupt + # delivered before that would prove nothing about descendants. + deadline = time.monotonic() + 30.0 + while time.monotonic() < deadline: + try: + if recorded.read_text(encoding="utf-8").strip(): + break + except OSError: + pass + time.sleep(0.05) + raise KeyboardInterrupt + + with tempfile.TemporaryDirectory() as directory: + recorded = Path(directory) / "worker.pid" + with mock.patch.object(subprocess.Popen, "wait", wait): + with self.assertRaises(KeyboardInterrupt): + lifecycle._run_contained( + [sys.executable, "-c", self.PROVIDER, str(recorded)], + environment={"PATH": os.environ.get("PATH", "")}, + cwd=directory, + timeout=300.0, + ) + worker = int(recorded.read_text(encoding="utf-8")) + self.assertNotEqual(worker, os.getpid()) + self.assertTrue(self.reaped(worker), "a worker outlived the run that was cancelled") + def test_a_run_that_finishes_in_time_reports_its_own_status(self) -> None: # The containment is not a behaviour change for an ordinary run: the # exit status still comes back, and nothing is signalled. @@ -1141,6 +1289,32 @@ def test_an_ordinary_symlinked_ancestor_is_resolved_not_refused(self) -> None: manifest = self.build(root=self.root / "linked-state" / "graph") self.assertEqual(manifest.completeness, lifecycle.COMPLETE) + def test_an_ancestor_retargeted_after_the_check_does_not_move_the_state(self) -> None: + """Check and use travel the same path, so retargeting between them does nothing. + + Checking a resolved snapshot and then writing through the original + spelling is two different paths: ``/outside/link`` can resolve outside + every repository when it is checked and point into one by the time the + first directory is created. The state root is canonicalized once, at + construction, and every later open goes through that canonical path, so + a link swapped afterwards is no longer on the route. + """ + safe = self.root / "safe" + safe.mkdir() + outside = self.root / "outside" + outside.mkdir() + link = outside / "link" + os.symlink(safe, link) + state = lifecycle.GraphStateRoot(self.repository, root=link / "graph-state") + inside = self.repository / "subdir" + inside.mkdir() + link.unlink() + os.symlink(inside, link) + state.ensure() + self.assertTrue(state.path.is_relative_to(safe)) + self.assertTrue((safe / "graph-state" / "graph").is_dir()) + self.assertFalse((inside / "graph-state").exists()) + def test_a_manifest_no_reader_could_load_publishes_nothing(self) -> None: """Publication validates the whole manifest before it touches state. From 0d865ffe9674b17b07fa82792f419043f2631e12 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 04:31:15 -0700 Subject: [PATCH 13/25] context: stop the provider group on a normal exit, not only a failure Fix round for the P2 finding on de10e30. codex:08ea4d04adff0f66a011 -- a leader that exits on its own is not evidence that its workers did. `_run_contained` cleaned up the group on the timeout and the interrupt only, so a provider that returned while a worker was still running left `subprocess_indexer` packing an artifact under concurrent modification and `build_graph` removing a scratch directory still in use. The group is now read once at launch and retained -- after the leader is reaped its pid is no longer a safe thing to look a group up from -- and stopped from a `finally`, on every exit including a return. The ordinary ending costs one signal-0 probe and no waiting: an already-empty group returns immediately. When the leader has been reaped, the grace period watches the group rather than a handle, because the survivors were reparented when their leader died. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 7 ++ src/code_mower/context_graph_lifecycle.py | 139 +++++++++++++++++----- tests/test_context_graph_lifecycle.py | 67 ++++++++++- 3 files changed, 181 insertions(+), 32 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index ce11a158..8c11a136 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -260,6 +260,13 @@ as soon as it reports the failure. The group gets `SIGTERM`, a short grace period, then `SIGKILL`, and the timeout is reported only once nothing is left running. A new session is safe here precisely because no stream is inherited. +The same check runs when the indexer simply exits, because its exit says nothing +about workers it started: one that is still writing would otherwise have its +output packed mid-write, and its scratch directory removed underneath it. The +group is looked up once at launch and kept — after the leader is reaped its pid +is no longer a safe thing to look a group up from — and an exit that left an +empty group behind costs one signal-`0` probe and no waiting at all. + The subcommand, the state-directory names, and the report counters are constants in one place in `context_graph_lifecycle.py`. They encode the interface as the evaluation recorded it; the first installation against a real pinned release diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 4dfac82a..d9cbad56 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -51,6 +51,7 @@ import subprocess import sys import tarfile +import time import uuid from dataclasses import dataclass from datetime import datetime, timezone @@ -1144,6 +1145,13 @@ def _pack_state(state_directory: Path) -> bytes: _REAP_TIMEOUT_SECONDS = 10.0 +#: How often the grace period is re-checked when what is being waited for is +#: the group rather than the direct child. After a normal exit the leader has +#: already been reaped, so there is no child left to wait on and the only way +#: to see the group empty is to ask. +_GROUP_POLL_SECONDS = 0.05 + + def _signal_group(group: int, number: int) -> None: try: os.killpg(group, number) @@ -1153,8 +1161,55 @@ def _signal_group(group: int, number: int) -> None: pass -def _terminate_process_group(child: subprocess.Popen[bytes]) -> None: - """Stop an abandoned provider and everything it started. +def _session_group(child: subprocess.Popen[bytes]) -> int | None: + """The group the child leads, read while the child is still unreaped. + + Read once at launch and retained for the rest of the run, because a pid is + only a safe thing to look a group up from while its process has not been + reaped: afterwards ``os.getpgid`` either fails or answers for whichever + process inherited the number. The group id itself stays safe to signal for + exactly as long as it is worth signalling, because the kernel does not + reuse a pid while it still names a process group with members in it. + + ``None`` means there is no group of this run's own to signal -- the child + never reached one, so the only thing that can be stopped is the child. + """ + try: + group = os.getpgid(child.pid) + except OSError: + return None + if group == os.getpgid(0): + return None + return group + + +def _group_is_empty(group: int) -> bool: + """Whether anything is left in ``group``. + + Signal ``0`` runs the kernel's existence and permission checks without + delivering anything, so this is the group's own answer rather than an + inference from what the leader did. A refusal is not emptiness: something + has to be there for the kernel to refuse on behalf of. + """ + try: + os.killpg(group, 0) + except ProcessLookupError: + return True + except OSError: + return False + return False + + +def _await_group_exit(group: int, timeout: float) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if _group_is_empty(group): + return + time.sleep(_GROUP_POLL_SECONDS) + + +def _terminate_process_group(child: subprocess.Popen[bytes], group: int | None) -> None: + """Stop everything the provider started, however this run ended. ``subprocess.run``'s own timeout kills the immediate child only. A provider that forks workers -- and under a launcher such as ``sandbox-exec`` the @@ -1163,27 +1218,42 @@ def _terminate_process_group(child: subprocess.Popen[bytes]) -> None: a scratch directory the build is about to delete. The child leads its own session, so one signal to its group reaches every descendant that has not deliberately left it. + + A leader that exits on its own is not evidence that its workers did. The + group is therefore checked on an ordinary return too, and not only on the + timeout and the interrupt: otherwise a provider that returns while a worker + is still writing leaves ``subprocess_indexer`` packing an artifact that is + concurrently being modified, and ``build_graph`` removing a scratch + directory that is still in use. Checking costs nothing in the ordinary + case, where the group is already empty and nothing is signalled or waited + for. """ - try: - group = os.getpgid(child.pid) - except OSError: - group = None - if group is None or group == os.getpgid(0): - # The child never reached a group of its own. Kill what can be named - # directly rather than signalling the group this process is in. - child.kill() - _reap(child) + leader_running = child.poll() is None + if group is None: + if leader_running: + child.kill() + _reap(child) + return + if not leader_running and _group_is_empty(group): + # The ordinary ending: the provider exited and took its workers with it. return _signal_group(group, signal.SIGTERM) - try: - child.wait(timeout=_TERMINATION_GRACE_SECONDS) - except subprocess.TimeoutExpired: - pass - # Unconditionally, and after the direct child has been waited for: that one - # exiting says nothing about workers it started, and this is the last moment - # anything can stop them. + if leader_running: + try: + child.wait(timeout=_TERMINATION_GRACE_SECONDS) + except subprocess.TimeoutExpired: + pass + else: + # Nothing here is this process's child any more -- the survivors were + # reparented when the leader died -- so the grace period is spent + # watching the group instead of waiting on a handle. + _await_group_exit(group, _TERMINATION_GRACE_SECONDS) + # Unconditionally, and after the grace period: a leader exiting says nothing + # about workers it started, and this is the last moment anything can stop + # them. _signal_group(group, signal.SIGKILL) - _reap(child) + if leader_running: + _reap(child) def _reap(child: subprocess.Popen[bytes]) -> None: @@ -1200,10 +1270,13 @@ def _run_contained( cwd: str, timeout: float, ) -> int: - """Run one child in its own process group, killing the group on any exit. + """Run one child in its own process group, stopping the group on any exit. Raises :class:`subprocess.TimeoutExpired` once the group has been stopped, so a caller reports the timeout only after there is nothing left running. + A returned exit status carries the same guarantee: the caller reads the + provider's own status, and by then nothing the provider started is still + running against the state the caller is about to pack up or delete. """ with subprocess.Popen( # noqa: S603 - argv is a resolved executable and validated options list(argv), @@ -1225,19 +1298,23 @@ def _run_contained( # terminal. start_new_session=True, ) as child: + # Read before the wait, because the wait may reap the leader and a + # reaped leader's pid is no longer a safe thing to look a group up from. + group = _session_group(child) try: return child.wait(timeout=timeout) - except BaseException: - # Every way out of this wait except returning, not the timeout - # alone. An operator's Ctrl-C raises ``KeyboardInterrupt`` here, and - # the provider does not see that signal: it leads its own session, - # so the terminal's SIGINT never reaches it. ``Popen.__exit__`` - # would then wait for a child nobody has asked to stop, while - # ``build_graph`` deletes the scratch directory underneath it. The - # group is stopped first, so unwinding leaves nothing running - # against state that is about to be removed. - _terminate_process_group(child) - raise + finally: + # Every way out of this wait, including returning. An operator's + # Ctrl-C raises ``KeyboardInterrupt`` here, and the provider does + # not see that signal: it leads its own session, so the terminal's + # SIGINT never reaches it. ``Popen.__exit__`` would then wait for a + # child nobody has asked to stop, while ``build_graph`` deletes the + # scratch directory underneath it. A provider that simply exits can + # leave workers behind the same way, so the ordinary return is + # cleaned up on the same path rather than trusted. Unwinding -- + # or returning -- leaves nothing running against state that is + # about to be read, packed, or removed. + _terminate_process_group(child, group) def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult]: diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index b5eee40e..47d5447c 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -114,6 +114,12 @@ def wait(self, timeout: float | None = None) -> int: raise subprocess.TimeoutExpired("graphify", timeout or 0) return self.returncode + def poll(self) -> int | None: + # ``None`` means the leader is still running, which is exactly the + # state an overrun leaves it in: the wait gave up on it, not the other + # way round. + return None if self._overruns else self.returncode + def kill(self) -> None: self.killed = True @@ -997,7 +1003,7 @@ def test_an_overrun_is_stopped_before_it_is_reported_as_one(self) -> None: def fake_popen(argv, **kwargs): return FakeChild(overruns=True) - def record_termination(child) -> None: + def record_termination(child, group) -> None: order.append("stopped") with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): @@ -1061,6 +1067,19 @@ class ExtractionOverrunTests(unittest.TestCase): with open(sys.argv[1], "w") as handle: handle.write(str(worker.pid)) time.sleep(300) +""" + + #: Stands in for a provider that starts a worker and then exits on its own, + #: leaving the worker running. The leader's exit is ordinary -- it even + #: reports a status -- and says nothing about whether the work has stopped. + ABANDONS_WORKER = """ +import subprocess +import sys + +worker = subprocess.Popen([sys.executable, "-c", "import time; time.sleep(300)"]) +with open(sys.argv[1], "w") as handle: + handle.write(str(worker.pid)) +raise SystemExit(5) """ def reaped(self, pid: int, *, within: float = 15.0) -> bool: @@ -1130,6 +1149,32 @@ def wait(child, timeout=None): self.assertNotEqual(worker, os.getpid()) self.assertTrue(self.reaped(worker), "a worker outlived the run that was cancelled") + def test_a_provider_that_exits_leaves_no_worker_behind_it(self) -> None: + """A normal exit is not evidence that the work stopped. + + Cleaning up only on the timeout and the interrupt left the ordinary + ending -- the leader returning a status while a worker it started is + still running -- to be trusted. ``subprocess_indexer`` would then pack + an artifact being concurrently modified, and ``build_graph`` would + delete a scratch directory still being written into. The status still + comes back; it now means what it appears to mean. + """ + with tempfile.TemporaryDirectory() as directory: + recorded = Path(directory) / "worker.pid" + returncode = lifecycle._run_contained( + [sys.executable, "-c", self.ABANDONS_WORKER, str(recorded)], + environment={"PATH": os.environ.get("PATH", "")}, + cwd=directory, + timeout=60.0, + ) + self.assertEqual(returncode, 5) + worker = int(recorded.read_text(encoding="utf-8")) + self.assertNotEqual(worker, os.getpid()) + self.assertTrue( + self.reaped(worker), + "a worker outlived the provider that started it and returned", + ) + def test_a_run_that_finishes_in_time_reports_its_own_status(self) -> None: # The containment is not a behaviour change for an ordinary run: the # exit status still comes back, and nothing is signalled. @@ -1142,6 +1187,26 @@ def test_a_run_that_finishes_in_time_reports_its_own_status(self) -> None: ) self.assertEqual(returncode, 3) + def test_a_run_that_leaves_nothing_behind_is_not_charged_the_grace_period(self) -> None: + # Checking the group on every exit has to be free in the case that is + # every real build: an empty group is a signal-0 probe, not a wait, so + # a provider that exited with no descendants is finished the moment its + # status is read. + with tempfile.TemporaryDirectory() as directory: + started = time.monotonic() + lifecycle._run_contained( + [sys.executable, "-c", "raise SystemExit(0)"], + environment={"PATH": os.environ.get("PATH", "")}, + cwd=directory, + timeout=60.0, + ) + elapsed = time.monotonic() - started + self.assertLess( + elapsed, + lifecycle._TERMINATION_GRACE_SECONDS, + "a clean run waited out a grace period it had nothing to wait for", + ) + class BuildAndPublishTests(TemporaryWorkspace): def test_manifest_binds_every_required_fact(self) -> None: From 1f87ba82588e38feea50473e204fd2b74bb67b1e Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 04:33:14 -0700 Subject: [PATCH 14/25] context: name the macOS tmp link without spelling the forbidden path The privacy scan forbids that literal path substring repo-wide, and the previous round introduced it three times while explaining why symlinked ancestors are resolved rather than refused. The scan runs before the unit tests, so every job stopped there and the suite has not executed since 826b105. Same explanation, spelled a way the scan allows. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 2 +- src/code_mower/context_graph_lifecycle.py | 3 ++- tests/test_context_graph_lifecycle.py | 5 +++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 8c11a136..b2be6bbd 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -369,7 +369,7 @@ is checked on the resolved path as well as the given one: `--state-dir `/outside/link` points inside one, and the `O_NOFOLLOW` opens cover only the final component of each directory this module creates. Symlinked ancestors are resolved rather than rejected — ordinary private roots have them, macOS reaches -`/tmp` through `/private/tmp`. +`/tmp` through a link into its `private` directory. ## What this does not do diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index d9cbad56..949bb13d 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -1527,7 +1527,8 @@ def __init__(self, repository: Path, *, root: Path | None = None): # symlinked ancestor free to be retargeted in between -- the check # passes against one directory and the writes land in another. Ordinary # private roots do have symlinked ancestors (macOS puts ``/tmp`` behind - # ``/private/tmp``), so they are resolved rather than refused. + # a link into its ``private`` directory), so they are resolved rather + # than refused. self.base = Path(os.path.realpath(base)) self.workspace = workspace_id(self.repository) self.path = self.base / "graph" / self.workspace diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 47d5447c..015e0288 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -1346,8 +1346,9 @@ def test_state_is_refused_behind_a_symlink_into_a_repository(self) -> None: def test_an_ordinary_symlinked_ancestor_is_resolved_not_refused(self) -> None: # Resolving, not rejecting: private roots legitimately sit behind - # links -- macOS reaches ``/tmp`` through ``/private/tmp`` -- so a - # symlinked ancestor outside any repository must still build. + # links -- macOS reaches ``/tmp`` through a link into its ``private`` + # directory -- so a symlinked ancestor outside any repository must + # still build. elsewhere = self.root / "elsewhere" elsewhere.mkdir() os.symlink(elsewhere, self.root / "linked-state") From 04e2806ac73a7ae9620fcc1d9c4fb7dafca4bc9c Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 04:40:59 -0700 Subject: [PATCH 15/25] context: keep Code Mower private state out of the tracked census MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The census excluded the provider's own index roots and the materializer excluded `.git`, but neither excluded `.code-mower`. A repository that tracks it therefore handed this tool's own packets, evidence and lane records to the indexer — the exact content `context_graph` refuses to let a delivered packet cite. The two ends of one policy disagreed. They now share one set rather than two copies of the same names: `_PRIVATE_STATE_ROOTS` is `context_graph._EXCLUDED_ROOTS` itself, so a root added to the evidence contract cannot be forgotten here. A tracked private-state path is skipped by `read_tracked_census` at any depth, case-folded, with a `private state` reason distinct from `provider state`, whose consequence is specific enough to keep its own name; and `_safe_relative` refuses the same set, so a census built elsewhere cannot reintroduce what the census would have skipped. Skipping `.git` in the census rather than only refusing it at materialization is the incidental widening: a crafted tree carrying `vendor/.git/config` is now recorded as skipped instead of failing the build outright. `_safe_relative` still refuses it. Closes finding codex:7005d3f22a3bd0bb8089. Co-Authored-By: Claude Opus 5 (1M context) --- docs/context-graph-lifecycle.md | 20 ++++++--- src/code_mower/context_graph_lifecycle.py | 54 +++++++++++++++++------ tests/test_context_graph_lifecycle.py | 44 +++++++++++++++++- 3 files changed, 97 insertions(+), 21 deletions(-) diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index b2be6bbd..60e200da 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -41,16 +41,22 @@ provenance at all. If Code Mower does not bind the revision, nothing does. working tree and not the index. Symlinks (`120000`) and submodules (`160000`) are skipped and recorded as skipped, because a symlink can name a target the build was never shown and a gitlink names a commit in a - repository it was never authorized to read. Committed provider state — a - tracked `.graphify/` or `.graph/`, at any depth, case-folded — is skipped for - a third reason: it is somebody's old index, and materializing it would let - the provider resume from a cache built over content this build never saw, - and let the adapter collect tracked repository bytes as if the provider had - just produced them. The census digest covers mode, blob name, size and path + repository it was never authorized to read. Committed private state is + skipped for a third reason, at any depth and case-folded: the roots are + `context_graph`'s excluded roots themselves — `.git`, `.graph`, `.graphify`, + `.code-mower` — bound rather than copied, so the set that refuses a citation + into private state is the same set that keeps those bytes away from the + indexer. A tracked `.graphify/` or `.graph/` is somebody's old index, and + materializing it would let the provider resume from a cache built over + content this build never saw, and let the adapter collect tracked repository + bytes as if the provider had just produced them. A tracked `.code-mower/` + is this tool's own packets and evidence, which the evidence contract refuses + to let a packet cite and which therefore may not be indexed either. The + census digest covers mode, blob name, size and path for every entry in sorted order. Both halves of the census are bounded as they are collected: the file-count budget covers what is materialized, and a matching budget covers what is skipped, because a repository of symlinks, - submodules, or committed provider state grows the skipped list without adding + submodules, or committed private state grows the skipped list without adding a single entry to the other one. 3. **Materialize into private state.** Each blob is written into a fresh 0700 directory as a 0600 file. Untracked and ignored files have no path into the diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 949bb13d..44670ee9 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -59,6 +59,7 @@ from typing import Any, Callable, Iterable, Iterator, Mapping, Sequence from .context_contract import ContextError, _identifier, _text, _timestamp +from .context_graph import _EXCLUDED_ROOTS from .context_store import _private, default_context_root from .file_locks import FileLockError, exclusive_handle_lock @@ -108,6 +109,18 @@ _PROVIDER_STATE_DIRECTORIES = (".graphify", ".graph") _PROVIDER_STATE_ROOTS = frozenset(name.casefold() for name in _PROVIDER_STATE_DIRECTORIES) +#: The evidence contract's excluded roots, bound rather than copied. One module +#: decides a citation into private state is out of scope, this one decides +#: those bytes never reach the indexer at all; they are the same policy read +#: from two ends, and a name added to one must not have to be remembered in the +#: other. The set is a superset of the provider roots above: it also carries +#: ``.git``, whose contents are the history rather than the revision, and +#: ``.code-mower``, this tool's own state. A repository is free to track +#: either, and a build over a tracked ``.code-mower/`` would hand the provider +#: exactly the packets and evidence that ``context_graph`` then refuses to let +#: a packet cite. +_PRIVATE_STATE_ROOTS = _EXCLUDED_ROOTS + _OBJECT_NAME = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?\Z") _GENERATION = re.compile(r"[0-9a-f]{32}\Z") _VERSION = re.compile(r"[0-9][0-9A-Za-z.+!-]{0,63}\Z") @@ -689,9 +702,22 @@ def resolve_revision(repository: Path, revision: str = "HEAD") -> tuple[str, str return commit, tree -def _is_provider_state(path: str) -> bool: - """Is this tracked path part of a committed provider index state?""" - return any(segment.casefold() in _PROVIDER_STATE_ROOTS for segment in path.split("/")) +def _private_state_reason(path: str) -> str | None: + """Why this tracked path may not enter the graph, or ``None`` if it may. + + Provider state keeps its own reason because the consequence is specific -- + the provider resuming from a cache of content this build never saw -- while + ``.git`` and ``.code-mower`` are private state of a different kind: version + control's own storage, and this tool's packets, evidence and lane records. + Every segment is tested, case-folded, so ``vendor/.git`` and a nested + ``docs/.CODE-MOWER`` are as excluded as the top-level ones. + """ + segments = [segment.casefold() for segment in path.split("/")] + if any(segment in _PROVIDER_STATE_ROOTS for segment in segments): + return "provider state" + if any(segment in _PRIVATE_STATE_ROOTS for segment in segments): + return "private state" + return None def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: @@ -701,7 +727,8 @@ def read_tracked_census(repository: Path, commit: str) -> TrackedCensus: uncommitted edit, an untracked scratch file, and an ignored secret are all invisible here by construction rather than by filtering. - Committed provider state is recorded as skipped rather than carried: it is + Committed private state -- the provider's own index, ``.git``, and this + tool's ``.code-mower`` -- is recorded as skipped rather than carried: it is excluded from the census, so it is excluded from the census digest too, and a build over a repository that tracks a ``.graphify`` directory binds a census that says so instead of quietly indexing somebody else's graph. @@ -741,8 +768,9 @@ def skip(path: str, reason: str) -> None: if mode in _SKIPPED_MODES: skip(path, _SKIPPED_MODES[mode]) continue - if _is_provider_state(path): - skip(path, "provider state") + excluded = _private_state_reason(path) + if excluded is not None: + skip(path, excluded) continue if mode not in _REGULAR_MODES or kind != "blob": skip(path, "unsupported") @@ -776,14 +804,14 @@ def _safe_relative(path: str) -> Path: or path.startswith("/") or "\\" in path or any(segment in {"", ".", ".."} for segment in path.split("/")) - # Every segment, not just the first: a vendored submodule's ``vendor/.git`` - # is as private as the top-level one. Case-folded because APFS and NTFS - # name the same directory ``.GIT``. - or any(segment.casefold() == ".git" for segment in path.split("/")) - # Provider state is skipped by the census, so a census that still + # Private state is skipped by the census, so a census that still # carries it was not built by ``read_tracked_census``. Refuse rather - # than seed the directory the provider is about to write into. - or _is_provider_state(path) + # than write ``.git`` or ``.code-mower`` into the tree the provider is + # about to read, or seed the directory it is about to write into. + # Every segment, not just the first: a vendored submodule's + # ``vendor/.git`` is as private as the top-level one. Case-folded + # because APFS and NTFS name the same directory ``.GIT``. + or _private_state_reason(path) is not None ): raise ContextError("tracked path must stay inside the materialized checkout") return Path(*path.split("/")) diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 015e0288..4adc79e4 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -33,6 +33,7 @@ from pathlib import Path from unittest import mock +from code_mower import context_graph from code_mower import context_graph_command as command from code_mower import context_graph_lifecycle as lifecycle from code_mower.context_contract import ContextError @@ -314,6 +315,46 @@ def test_committed_provider_state_is_skipped_rather_than_indexed(self) -> None: self.assertFalse((destination / ".graphify").exists()) self.assertFalse((destination / "vendor").exists()) + def test_committed_code_mower_state_is_skipped_rather_than_indexed(self) -> None: + """A tracked ``.code-mower`` is this tool's own state, not content. + + ``context_graph`` refuses a packet that cites into ``.code-mower`` + because a graph that reached in there escaped the checkout it was asked + to index. The lifecycle has to agree at the other end: if those bytes + are handed to the indexer in the first place, the evidence contract is + refusing a citation to content the provider has already read. + """ + for name in (".code-mower", "vendor/.CODE-MOWER"): + directory = self.repository / name + directory.mkdir(parents=True) + (directory / "packet.json").write_text('{"secret": "packet"}\n', encoding="utf-8") + git(self.repository, "add", ".code-mower", "vendor") + git(self.repository, "commit", "-q", "-m", "committed code mower state") + commit, _ = lifecycle.resolve_revision(self.repository) + census = lifecycle.read_tracked_census(self.repository, commit) + self.assertEqual( + [entry.path for entry in census.entries], + [".gitignore", "README.md", "example_pkg/config.py"], + ) + self.assertIn((".code-mower/packet.json", "private state"), census.skipped) + self.assertIn(("vendor/.CODE-MOWER/packet.json", "private state"), census.skipped) + destination = self.root / "materialized-with-code-mower" + lifecycle.materialize_tracked_files(self.repository, census, destination) + self.assertFalse((destination / ".code-mower").exists()) + self.assertFalse((destination / "vendor").exists()) + + def test_the_census_excludes_exactly_the_evidence_contract_roots(self) -> None: + """The two ends of the policy share one set rather than two copies. + + A name added to ``context_graph._EXCLUDED_ROOTS`` must not have to be + remembered here as well, so this asserts identity of the object and not + merely equality of its contents. + """ + self.assertIs(lifecycle._PRIVATE_STATE_ROOTS, context_graph._EXCLUDED_ROOTS) + for root in context_graph._EXCLUDED_ROOTS: + with self.subTest(root=root): + self.assertIsNotNone(lifecycle._private_state_reason(f"vendor/{root}/file.json")) + def test_committing_provider_state_does_not_move_the_census_digest(self) -> None: # The manifest binds the digest of what was indexed. Committed provider # state is not indexed, so it does not enter that digest; it is @@ -392,7 +433,8 @@ def test_materialization_refuses_an_existing_directory(self) -> None: def test_escaping_census_paths_are_rejected(self) -> None: escaping = ("/etc/passwd", "../outside.py", "a/../../b.py", ".git/config", "vendor/.git/config", "a\\b.py", ".graphify/cache.json", - "vendor/.GRAPH/cache.json") + "vendor/.GRAPH/cache.json", ".code-mower/packets/one.json", + "docs/.CODE-MOWER/evidence.json") for index, path in enumerate(escaping): with self.subTest(path=path): census = lifecycle.TrackedCensus( From 5326e8329b384e2024d6eda69a1febfc5bb53a47 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 04:57:48 -0700 Subject: [PATCH 16/25] context: confine the provider's filesystem, not only its sockets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The supplemental adversarial review at 72071ad found the boundary was half of one. `(allow default)(deny network*)` and `--dev-bind / /` deny sockets and hand the child the host filesystem; a real-process reproduction read an external ignored .env through the selected sandbox. Setting the staged tree as the working directory is not a boundary. The exposure is now the whole of what a contained child can see: the materialized copy and the build's own HOME and TMPDIR writable, the pinned provider's install and the system runtime read-only, nothing else present at all. macOS gets a `(deny default)` seatbelt profile; Linux gets a bwrap root that is empty until something is bound into it. `unshare --net` is gone as a candidate: it confines one of the two. Launchers are absolute paths, owner- and ancestry-validated, never resolved through an inherited PATH, so a shadow launcher cannot compute the probe's evidence and report containment it never established. The probe checks both halves in one run -- a listener this process really holds, and a secret planted outside the exposure -- and a mechanism that denies one but not the other classifies as unusable rather than as a boundary. State roots are created one no-follow descriptor at a time, from the deepest component that really exists. Canonicalizing at construction settles what existing ancestors mean and cannot settle what a component nobody has created yet will mean; a recursive mkdir followed it, and state landed in the repository. A killed process group is now awaited and observed empty before the run returns, because the caller's next act is to pack or delete the state those processes are writing. A `graph containment` CI job installs bubblewrap and sets CODE_MOWER_REQUIRE_CONTAINMENT=1, which turns these tests' host-dependent skips into failures: coverage that silently skips is coverage nobody has. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 31 + docs/context-graph-lifecycle.md | 72 ++- src/code_mower/context_graph_lifecycle.py | 686 +++++++++++++++++----- tests/test_context_graph_lifecycle.py | 276 +++++++-- 4 files changed, 861 insertions(+), 204 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 65868d1d..6c2da5a1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -89,6 +89,37 @@ jobs: --timeout 240 --json + graph_containment: + # The local-graph boundary, against the real kernel mechanism rather than a + # stand-in for one. The unit suite skips these when a host offers no + # mechanism, which is right for a laptop and useless as coverage: this job + # installs bubblewrap and sets CODE_MOWER_REQUIRE_CONTAINMENT, which turns + # that skip into a failure. + name: graph containment + runs-on: ubuntu-latest + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 + with: + python-version: "3.12" + + - name: Install bubblewrap + run: | + sudo apt-get update + sudo apt-get install -y bubblewrap + test -x /usr/bin/bwrap + + - name: Install package + run: python -m pip install -e . + + - name: Real containment tests + env: + CODE_MOWER_REQUIRE_CONTAINMENT: "1" + run: python -m unittest discover -s tests -p test_context_graph_lifecycle.py -v + package: name: package runs-on: ubuntu-latest diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index 60e200da..c7c2e75a 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -100,31 +100,50 @@ again — otherwise a healthy refresh would surface as `invalid` or `corrupt`. A verdict about the generation `current` still names is returned as it stands; the retry is for a moved pointer, not a poll. -## The network boundary +## The containment boundary An environment variable is a request, not a boundary: `NO_PROXY=*` asks a cooperating client to connect *directly*, and on a host with internet access an -uncooperative provider is unaffected by any of it. So the provider is launched -behind an argv prefix that denies it sockets at the operating-system level — -`sandbox-exec` on macOS, `bwrap --unshare-net` or an unprivileged network -namespace via `unshare --net` on Linux. - -No mechanism is trusted on its name. Each candidate is accepted only after a -probe child launched behind it has been *observed* failing to reach a TCP -listener this process is really holding open on loopback. The verdict is taken -at the listener, not from the child's errno: a network namespace brings up its -own loopback, so a correctly contained child sees the same `ECONNREFUSED` that -an unconfined child sees from an unused host port. Those two are -indistinguishable at the child and obvious at the listener, which either -accepted a connection or did not. A candidate passes only when the child -reported that it could not connect *and* nothing arrived, so a launcher that -never started its child cannot pass for a boundary. - -An unsandboxed control child runs first and must reach the listener. If it -cannot — no probe interpreter, loopback unavailable — then "could not connect" -proves nothing about any candidate, every candidate would pass, and the probe -refuses outright instead. The result is cached for the process, since it is a -property of the host. +uncooperative provider is unaffected by any of it. A working directory is not a +boundary either: pointing a provider at the materialized copy does not stop it +reading the checkout next door. So the provider is launched behind an argv +prefix that denies it, at the operating-system level, both the network and +every path outside the build's own directories — `sandbox-exec` with a +`(deny default)` profile on macOS, `bwrap` with an empty new root on Linux. + +What the child can see is the whole of it: the materialized copy and the +build's redirected `HOME` and `TMPDIR`, writable; the pinned provider's own +install and the system runtime it needs to start, read-only. The operator's +home, their other checkouts, and every ignored `.env` beside them are not +unreadable — they are absent. `unshare --net` used to be a candidate and is +gone: it denies the network and leaves the host filesystem in place, which is +half a boundary. + +No mechanism is trusted on its name, and none is looked up on `PATH`: each +candidate is an absolute path whose file and every ancestor directory must be +owned by root or by this user and unwritable by anyone else, because a launcher +somebody else can replace is a verdict somebody else can forge. + +A candidate is accepted only after a probe child launched behind it has been +*observed* failing at both halves: failing to reach a TCP listener this process +is really holding open on loopback, and failing to read a secret file planted +outside its exposure. The network verdict is taken at the listener, not from +the child's errno: a network namespace brings up its own loopback, so a +correctly contained child sees the same `ECONNREFUSED` that an unconfined child +sees from an unused host port. Those two are indistinguishable at the child and +obvious at the listener, which either accepted a connection or did not. A child +that fails one half and not the other is not a boundary; it classifies as +unusable. + +The child also prints a digest of a nonce generated for that run, so a launcher +that never started its child cannot pass for a boundary by exiting with the +contained code. + +An unsandboxed control child runs first and must reach the listener *and* read +the planted secret. If it cannot — no probe interpreter, loopback unavailable — +then "could not" proves nothing about any candidate, every candidate would +pass, and the probe refuses outright instead. The result is cached for the +process, since it is a property of the host. A host where no candidate passes gets no build. `subprocess_indexer()` raises before a single blob is materialized, and `context-graph doctor` reports the @@ -134,10 +153,11 @@ operator who cannot contain a third-party indexer is better served by knowing it than by a build that quietly could have reached the network. A Linux host that restricts unprivileged user namespaces — Ubuntu 24.04 and -GitHub's hosted runners among them — offers no mechanism by default, and both -`unshare` and `bwrap` fail there. Installing bubblewrap (`apt install -bubblewrap`), which ships an AppArmor profile permitting the namespaces it -needs, is the least invasive way to give such a host one. The alternative is to +GitHub's hosted runners among them — offers no mechanism by default. Installing +bubblewrap (`apt install bubblewrap`), which ships an AppArmor profile +permitting the namespaces it needs, is the least invasive way to give such a +host one, and is what the `graph containment` CI job does before running these +tests for real rather than skipping them. The alternative is to lift the restriction system-wide (`sysctl kernel.apparmor_restrict_unprivileged_userns=0`), which is a decision about the whole machine rather than about this build, and not one this diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 44670ee9..470a8ade 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -22,13 +22,18 @@ the whole new one, never a half-written directory. * **Scrub the environment.** The indexer runs with an allowlisted environment, so an ambient token cannot leak into a provider process. -* **Deny the network in the kernel, not by request.** Emptying proxy variables - only redirects a client that chooses to honour them. The provider is - launched inside an OS sandbox that refuses sockets outright, and the sandbox - is accepted only after a probe child has been observed failing to reach a - socket this process is really listening on -- observed at the listener, not - believed from the child's errno. A host that offers no such mechanism gets a - refused build, not an unconfined provider. +* **Deny the network and the host filesystem in the kernel, not by request.** + Emptying proxy variables only redirects a client that chooses to honour them, + and a working directory is not a boundary. The provider is launched inside an + OS sandbox whose filesystem view is the materialized copy, the build's own + scratch directories, and a read-only runtime -- the operator's home, their + other checkouts, and every ignored ``.env`` beside them are absent from it, + not merely unreadable. The mechanism is accepted only after a probe child has + been observed failing at *both*: failing to reach a socket this process is + really listening on, observed at the listener rather than believed from the + child's errno, and failing to read a secret file planted outside its + exposure. A host that offers no such mechanism gets a refused build, not an + unconfined provider. Nothing here installs, imports, or requires a graph package. The indexer is an injected callable, so the whole lifecycle is provable offline; the bundled @@ -48,9 +53,11 @@ import shutil import signal import socket +import stat import subprocess import sys import tarfile +import tempfile import time import uuid from dataclasses import dataclass @@ -138,7 +145,7 @@ #: Hygiene, not the boundary. Emptying proxy variables stops a cooperating #: client from finding a proxy and ``GIT_TERMINAL_PROMPT=0`` stops a child #: blocking on a credential prompt, but on a host with direct connectivity -#: neither denies anything. The boundary is ``network_sandbox_command``. +#: neither denies anything. The boundary is ``containment_prefix``. _NETWORK_DENY = { "no_proxy": "*", "NO_PROXY": "*", @@ -152,68 +159,243 @@ "PYTHONNOUSERSITE": "1", } -#: Argv prefixes that place a child in a network-denying OS sandbox, most -#: specific first. Each is a mechanism the host either has or does not; none is -#: trusted on its name, because a prefix that silently degrades to running the -#: command unconfined would be worse than no prefix at all. -_SANDBOX_CANDIDATES: tuple[tuple[str, ...], ...] = ( - ("/usr/bin/sandbox-exec", "-p", "(version 1)(allow default)(deny network*)"), - ("bwrap", "--unshare-net", "--dev-bind", "/", "/", "--"), - ("unshare", "--net", "--map-current-user", "--"), - ("unshare", "--net", "--map-root-user", "--"), +#: Isolation mechanisms, most specific first, each named by an absolute path. +#: +#: Absolute, deliberately: a launcher looked up on an inherited ``PATH`` can be +#: shadowed by a program that answers the probe without confining anything, and +#: the probe's verdict is only as good as its knowledge of what it ran. The +#: paths are the system locations these tools install into, and each is checked +#: for trusted ownership and unwritable ancestry before it is run. +#: +#: ``unshare --net`` used to be here and is gone. It denies the network and +#: nothing else: the child keeps the host's whole filesystem, which is not the +#: boundary this module claims. A host with no mechanism that confines *both* +#: gets a refused build. +_SANDBOX_CANDIDATES: tuple[tuple[str, str], ...] = ( + ("sandbox-exec", "/usr/bin/sandbox-exec"), + ("bwrap", "/usr/bin/bwrap"), + ("bwrap", "/usr/local/bin/bwrap"), +) + +#: Read-only host paths a runtime needs to start at all: the loader, the C +#: library, the system interpreters. Everything outside this list and the +#: exposure a build asks for is not in the child's filesystem view -- not +#: unreadable by permission, absent. +_SYSTEM_READ_PATHS: tuple[str, ...] = ( + "/usr", + "/bin", + "/sbin", + "/lib", + "/lib64", + "/lib32", + "/etc", + "/private/etc", + "/System", + "/Library", + "/private/var/db/dyld", + "/private/var/db/timezone", ) -#: The probe connects to a socket this process is really listening on, and the -#: verdict is whether the connection *arrived* -- not which errno the child saw. -#: Classifying by errno cannot work: a network namespace brings its own loopback -#: up, so a contained child gets ``ECONNREFUSED`` from an empty namespace while -#: an unconfined child gets ``ECONNREFUSED`` from an unused host port. The two -#: are indistinguishable at the child. They are not indistinguishable at the -#: listener, which either accepts a connection or does not. +#: The probe reports two facts about one run, as a bitmask offset from a base +#: no shell error code lands on: whether the child could read a secret file +#: planted outside its exposure, and whether it could open a socket to a port +#: this process is really listening on. #: -#: The probe exits ``7`` when it could not connect and ``3`` when it could; -#: every other code -- a launcher that could not start, a child that never ran -#: -- means the candidate is not usable as a boundary. +#: The network verdict is taken at the *listener*, not from the child's errno. +#: Classifying by errno cannot work: a network namespace brings its own +#: loopback up, so a contained child gets ``ECONNREFUSED`` from an empty +#: namespace while an unconfined child gets ``ECONNREFUSED`` from an unused host +#: port. Indistinguishable at the child; obvious at the listener, which either +#: accepted a connection or did not. #: #: An exit code alone is not evidence that the child ran: a launcher that exits -#: ``7`` without executing anything produces the same code as a contained child, -#: and would be accepted as a boundary while confining nothing. So the child -#: first prints a value only running it can produce -- the digest of a nonce -#: this process generated for this run -- and a run with no such evidence is -#: unusable whatever its exit code. Echoing the argv is not enough: the digest -#: is computed by the child, and the nonce is fresh per run, so neither a -#: launcher that parrots its arguments nor one that replays an earlier probe -#: can produce it. -_PROBE_DENIED = 7 -_PROBE_REACHED = 3 -_DENIAL_PROBE = """ +#: with the contained code without executing anything would be accepted as a +#: boundary while confining nothing. So the child first prints a value only +#: running it can produce -- the digest of a nonce generated for this run -- and +#: a run with no such evidence is unusable whatever its exit code. Echoing the +#: argv is not enough: the digest is computed by the child and the nonce is +#: fresh, so neither a launcher that parrots its arguments nor one that replays +#: an earlier probe can produce it. +_PROBE_BASE = 40 +_PROBE_READ_SECRET = 1 +_PROBE_REACHED_LISTENER = 2 +_CONTAINMENT_PROBE = """ import hashlib import socket import sys -print(hashlib.sha256(sys.argv[2].encode()).hexdigest(), flush=True) +print(hashlib.sha256(sys.argv[3].encode()).hexdigest(), flush=True) +seen = 0 try: - probe = socket.create_connection(("127.0.0.1", int(sys.argv[1])), timeout=5) + with open(sys.argv[2], "rb") as secret: + secret.read(1) except OSError: - sys.exit(7) -probe.close() -sys.exit(3) + pass +else: + seen |= 1 +try: + reached = socket.create_connection(("127.0.0.1", int(sys.argv[1])), timeout=5) +except OSError: + pass +else: + reached.close() + seen |= 2 +sys.exit(40 + seen) """ -#: How a probe run classifies: the child reached this process's listener, the -#: child ran and could not, or nothing usable happened. +#: How a probe run classifies: the child was outside the boundary in at least +#: one respect, the child was inside it in both, or nothing usable happened. +#: A mechanism that denies only one of the two is ``_UNUSABLE``, not a +#: boundary -- half a boundary is what this finding was about. _REACHED = "reached" _CONTAINED = "contained" _UNUSABLE = "unusable" -_sandbox_prefix: tuple[str, ...] | None = None -_sandbox_probed = False + +@dataclass(frozen=True) +class Containment: + """One verified isolation mechanism on this host. + + The argv is built per build rather than cached, because the boundary is a + function of what that build is allowed to expose. What is cached is the + finding that this mechanism, at this path, was observed confining a child. + """ + + name: str + launcher: str + + +_containment: Containment | None = None +_containment_probed = False -def _launcher_path(name: str) -> str | None: - if os.path.isabs(name): - return name if os.access(name, os.X_OK) else None - return shutil.which(name) +def _trusted_launcher(path: str) -> str | None: + """A launcher only a trusted account could have replaced, or ``None``. + + The file and every ancestor directory: an executable that is itself + root-owned but sits in a directory somebody else may write can be swapped + for one that reports containment it never established. A symlink anywhere + in the chain is refused rather than followed -- what it names now is not + what it will name later, and this decision is cached for the process. + """ + trusted = {0, os.geteuid()} + for current in (Path(path), *Path(path).parents): + try: + entry = os.lstat(current) + except OSError: + return None + if entry.st_uid not in trusted or entry.st_mode & (stat.S_IWGRP | stat.S_IWOTH): + return None + if stat.S_ISLNK(entry.st_mode): + return None + info = os.lstat(path) + if not stat.S_ISREG(info.st_mode) or not os.access(path, os.X_OK): + return None + return path + + +def _existing(paths: Iterable[Path | str]) -> tuple[str, ...]: + """Resolved, de-duplicated, existing paths, in the order they were given. + + Resolved because both mechanisms match on the kernel's path, not the + caller's spelling: macOS puts ``/tmp`` and ``/var`` behind links into its + ``private`` directory, so an unresolved exposure would name a path the + sandbox never sees. + """ + seen: dict[str, None] = {} + for path in paths: + try: + real = os.path.realpath(path) + except OSError: # pragma: no cover - realpath does not raise on absent paths + continue + if os.path.exists(real): + seen.setdefault(real, None) + return tuple(seen) + + +def _seatbelt_literal(path: str) -> str: + return '"' + path.replace("\\", "\\\\").replace('"', '\\"') + '"' + + +def _seatbelt_prefix(launcher: str, *, writable: Sequence[str], readable: Sequence[str]) -> tuple[str, ...]: + """A ``sandbox-exec`` profile that denies by default and then names the exposure. + + ``(allow default)(deny network*)`` -- what this module used to pass -- denies + sockets and leaves the host filesystem wide open. The order here is the + other way round: nothing is permitted, and then the runtime is made readable + and the build's own directories writable. + """ + rules = [ + "(version 1)", + "(deny default)", + "(deny network*)", + "(allow process-fork)", + "(allow signal)", + "(allow sysctl-read)", + "(allow mach-lookup)", + "(allow ipc-posix-shm)", + "(allow file-read-metadata)", + "(allow file-write-data (literal \"/dev/null\") (literal \"/dev/zero\")" + " (literal \"/dev/random\") (literal \"/dev/urandom\"))", + ] + if readable: + subpaths = " ".join(f"(subpath {_seatbelt_literal(path)})" for path in readable) + rules.append(f"(allow file-read* process-exec* {subpaths})") + if writable: + subpaths = " ".join(f"(subpath {_seatbelt_literal(path)})" for path in writable) + rules.append(f"(allow file-read* file-write* {subpaths})") + return (launcher, "-p", "\n".join(rules)) + + +def _bubblewrap_prefix(launcher: str, *, writable: Sequence[str], readable: Sequence[str]) -> tuple[str, ...]: + """A ``bwrap`` mount namespace containing only the exposure. + + ``--dev-bind / /`` -- what this module used to pass -- hands the child the + host's entire filesystem, read *and* write, and isolates the network alone. + The new root is empty: the runtime is bound read-only, the build's own + directories are bound writable, and ``/tmp`` is a tmpfs, so a path nobody + named does not exist for this child. + """ + argv = [ + launcher, + "--unshare-net", + "--unshare-ipc", + "--unshare-uts", + "--unshare-pid", + "--unshare-cgroup-try", + "--new-session", + "--die-with-parent", + "--proc", "/proc", + "--dev", "/dev", + "--tmpfs", "/tmp", + ] + for path in readable: + argv += ["--ro-bind-try", path, path] + # After the read-only runtime, so an exposure that lives under one of those + # paths is writable rather than shadowed by the read-only bind. + for path in writable: + argv += ["--bind", path, path] + argv.append("--") + return tuple(argv) + + +_PREFIX_BUILDERS: Mapping[str, Callable[..., tuple[str, ...]]] = { + "sandbox-exec": _seatbelt_prefix, + "bwrap": _bubblewrap_prefix, +} + + +def _prefix_for( + mechanism: Containment, + *, + writable: Sequence[Path | str], + readable: Sequence[Path | str], +) -> tuple[str, ...]: + return _PREFIX_BUILDERS[mechanism.name]( + mechanism.launcher, + writable=_existing(writable), + readable=_existing([*_SYSTEM_READ_PATHS, *readable]), + ) def _accepted(listener: socket.socket) -> bool: @@ -233,81 +415,133 @@ def _ran_the_probe(output: bytes, nonce: str) -> bool: def _classify_probe(prefix: Sequence[str]) -> str: - """Run the probe under ``prefix`` against a listener in this process.""" + """Run the probe under ``prefix``: a real listener and a real planted secret. + + The secret is written to the host's temporary directory, which no exposure + this module builds ever includes, so an unconfined child reads it and a + confined one cannot see it at all. + """ nonce = secrets.token_hex(16) - with socket.socket() as listener: - try: - listener.bind(("127.0.0.1", 0)) - listener.listen(1) - except OSError: # pragma: no cover - a host that cannot listen on loopback - return _UNUSABLE - # The child connects and exits; the connection waits in the backlog - # until it is accepted below, so the accept order does not matter. - listener.settimeout(1) - port = listener.getsockname()[1] - try: - completed = subprocess.run( - [*prefix, sys.executable, "-c", _DENIAL_PROBE, str(port), nonce], - check=False, - capture_output=True, - timeout=60, - env={"PATH": os.environ.get("PATH", ""), **_NETWORK_DENY}, - ) - except (OSError, subprocess.SubprocessError): - return _UNUSABLE - arrived = _accepted(listener) + handle, secret = tempfile.mkstemp(prefix="code-mower-containment-probe-") + try: + os.write(handle, secrets.token_hex(32).encode()) + finally: + os.close(handle) + try: + with socket.socket() as listener: + try: + listener.bind(("127.0.0.1", 0)) + listener.listen(1) + except OSError: # pragma: no cover - a host that cannot listen on loopback + return _UNUSABLE + # The child connects and exits; the connection waits in the backlog + # until it is accepted below, so the accept order does not matter. + listener.settimeout(1) + port = listener.getsockname()[1] + try: + completed = subprocess.run( + [*prefix, sys.executable, "-c", _CONTAINMENT_PROBE, str(port), secret, nonce], + check=False, + capture_output=True, + timeout=60, + env={"PATH": os.environ.get("PATH", ""), **_NETWORK_DENY}, + ) + except (OSError, subprocess.SubprocessError): + return _UNUSABLE + arrived = _accepted(listener) + finally: + with contextlib.suppress(OSError): + os.unlink(secret) # Before anything is read from the exit code: a run that cannot show its # child executed classifies as nothing at all. This is the control that # keeps "denied" from being the default answer for a launcher that never # started the probe. if not _ran_the_probe(completed.stdout, nonce): return _UNUSABLE - if arrived: - return _REACHED - if completed.returncode == _PROBE_REACHED: - # The child says it connected but nothing arrived here; treat the - # disagreement as a probe that proved nothing rather than as isolation. + observed = completed.returncode - _PROBE_BASE + if observed not in (0, 1, 2, 3): return _UNUSABLE - return _CONTAINED if completed.returncode == _PROBE_DENIED else _UNUSABLE + read_secret = bool(observed & _PROBE_READ_SECRET) + if arrived != bool(observed & _PROBE_REACHED_LISTENER): + # The child and the listener disagree about whether a connection + # happened; treat that as a probe that proved nothing. + return _UNUSABLE + if arrived and read_secret: + return _REACHED + if not arrived and not read_secret: + return _CONTAINED + # Exactly one boundary held. A mechanism that denies sockets while leaving + # the host filesystem readable is not the boundary this module claims, and + # accepting it is the defect this classification exists to refuse. + return _UNUSABLE -def _sandbox_denies_network(prefix: Sequence[str]) -> bool: - """Watch a child under ``prefix`` fail to reach a socket that is really there.""" +def _prefix_confines(prefix: Sequence[str]) -> bool: + """Watch a child under ``prefix`` fail to reach either thing that is really there.""" return _classify_probe(prefix) == _CONTAINED -def _probe_network_sandbox() -> tuple[str, ...] | None: +def _interpreter_read_paths() -> tuple[str, ...]: + """The minimum a probe child needs to be a running Python at all.""" + return tuple( + path + for path in (os.path.realpath(sys.executable), sys.prefix, sys.base_prefix) + if path + ) + + +def _probe_containment() -> Containment | None: if not sys.executable: # pragma: no cover - a frozen interpreter cannot probe return None - # The control, first: a child with no prefix must reach the listener. If it - # cannot -- no probe interpreter, loopback blocked, sockets unavailable -- - # then "could not connect" proves nothing about any candidate, and every - # candidate would pass for a boundary. Refuse the whole probe instead. + # The control, first: a child with no prefix must reach the listener *and* + # read the planted secret. If it cannot -- no probe interpreter, loopback + # blocked, an unreadable temporary directory -- then "could not" proves + # nothing about any candidate, and every candidate would pass for a + # boundary. Refuse the whole probe instead. if _classify_probe(()) != _REACHED: return None - for candidate in _SANDBOX_CANDIDATES: - launcher = _launcher_path(candidate[0]) - if launcher is None: - continue - prefix = (launcher, *candidate[1:]) - if _sandbox_denies_network(prefix): - return prefix + readable = _interpreter_read_paths() + with tempfile.TemporaryDirectory(prefix="code-mower-containment-") as scratch: + for name, path in _SANDBOX_CANDIDATES: + launcher = _trusted_launcher(path) + if launcher is None: + continue + mechanism = Containment(name=name, launcher=launcher) + prefix = _prefix_for(mechanism, writable=(scratch,), readable=readable) + if _prefix_confines(prefix): + return mechanism return None -def network_sandbox_command() -> tuple[str, ...] | None: - """The argv prefix that denies a provider process the network, if any. +def containment_mechanism() -> Containment | None: + """The isolation mechanism this host was observed providing, if any. Probed once per process and cached, because the answer is a property of the host rather than of a build. ``None`` means this host offers no mechanism - this build could *observe* working, and a build refuses rather than running - a provider it cannot contain. + this build could *observe* denying a child both the network and the host + filesystem, and a build refuses rather than running a provider it cannot + contain. """ - global _sandbox_prefix, _sandbox_probed - if not _sandbox_probed: - _sandbox_prefix = _probe_network_sandbox() - _sandbox_probed = True - return _sandbox_prefix + global _containment, _containment_probed + if not _containment_probed: + _containment = _probe_containment() + _containment_probed = True + return _containment + + +def containment_prefix( + *, + writable: Sequence[Path | str], + readable: Sequence[Path | str], +) -> tuple[str, ...]: + """The argv prefix confining a child to ``writable`` plus a read-only runtime.""" + mechanism = containment_mechanism() + if mechanism is None: + raise ContextError( + "local graph builds need an OS sandbox that denies the provider the network and " + "the host filesystem; this host offers none that could be verified" + ) + return _prefix_for(mechanism, writable=writable, readable=readable) def _object_name(value: Any) -> str: @@ -913,6 +1147,12 @@ class IndexRequest: pin: GraphifyPin commit: str tree: str + #: The scratch directories the provider may write to besides the copy -- + #: the redirected ``HOME`` and ``TMPDIR``. Named here rather than inferred + #: because they are also exactly what the filesystem boundary exposes: a + #: directory the environment points at but the sandbox does not expose is a + #: provider that cannot start. + writable: tuple[Path, ...] = () @dataclass(frozen=True) @@ -941,6 +1181,25 @@ def _resolved_executable(executable: str) -> str: return executable +def _provider_read_paths(command: str) -> tuple[str, ...]: + """The install the pinned provider needs to be readable, and nothing beside it. + + A console script is one file in a virtual environment whose libraries live + beside it, so the environment root -- the executable's grandparent -- is + what has to be exposed, plus the base interpreter it was created from. An + executable this process cannot even locate is refused here rather than + exposed as a guess: the alternative is a boundary drawn around a path that + is not where the provider is. + """ + located = command if os.path.isabs(command) else shutil.which(command) + if not located: + raise ContextError("local graph provider executable could not be located for containment") + real = Path(os.path.realpath(located)) + return tuple( + str(path) for path in (real, real.parent, real.parent.parent, Path(sys.base_prefix)) + ) + + #: The subcommand the evaluated release exposes, recorded in #: ``docs/graphify-evaluation.md``: the clean-room run indexed with #: ``extract --code-only --no-cluster --max-workers 4``. There is no @@ -1282,6 +1541,18 @@ def _terminate_process_group(child: subprocess.Popen[bytes], group: int | None) _signal_group(group, signal.SIGKILL) if leader_running: _reap(child) + # Sending the signal is not the same as the group being gone, and the + # caller's next act is to pack or delete the state these processes are + # writing. ``SIGKILL`` is not refusable, so this waits on the kernel rather + # than on a cooperating child -- but a process stuck in uninterruptible + # sleep can still outlive it, and a group that cannot be established as + # empty fails the build instead of being assumed gone. + _await_group_exit(group, _REAP_TIMEOUT_SECONDS) + if not _group_is_empty(group): + raise ContextError( + "local graph provider left processes running that could not be stopped; " + "no generation was published" + ) def _reap(child: subprocess.Popen[bytes]) -> None: @@ -1362,15 +1633,23 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] behind is then collected and classified from its own report. """ command = _resolved_executable(executable) - sandbox = network_sandbox_command() - if sandbox is None: + if containment_mechanism() is None: raise ContextError( - "local graph builds need an OS sandbox that denies the provider network access; " - "this host offers none that could be verified" + "local graph builds need an OS sandbox that denies the provider the network and " + "the host filesystem; this host offers none that could be verified" ) + runtime = _provider_read_paths(command) def run(request: IndexRequest) -> IndexResult: _refuse_pre_existing_provider_state(request.source_root) + # Built per run, because the boundary is a function of what this build + # exposes: the materialized copy and the build's own scratch areas are + # writable, the pinned provider's install is readable, and nothing else + # on this host is in the child's filesystem view at all. + sandbox = containment_prefix( + writable=(request.source_root, *request.writable), + readable=runtime, + ) # Normalized again at the point of launch, not because the pin could # arrive without the restrictions -- it cannot -- but because this is # the line that decides what the provider is actually asked to do, and @@ -1514,13 +1793,48 @@ def workspace_id(repository: Path) -> str: return hashlib.sha256(str(Path(repository).resolve()).encode()).hexdigest()[:32] -def _open_private_directory(path: Path, *, create: bool) -> int: - if create: - path.mkdir(mode=0o700, parents=True, exist_ok=True) +#: What a missing component means to a walk: create it, stop there, or refuse. +_MISSING_CREATE = "create" +_MISSING_STOP = "stop" +_MISSING_REFUSE = "refuse" + + +def _open_private_at(parent: int | None, name: str, *, missing: str, private: bool = True) -> int | None: + """Open one directory *relative to a descriptor*, following nothing. + + ``parent`` is the descriptor the name is resolved against, so the kernel + resolves exactly one component and ``O_NOFOLLOW`` covers all of it. That is + the difference between checking a path and traversing one: a path opened by + its full spelling is re-resolved from the root every time, and any ancestor + may have become a symlink since it was last looked at. + + ``mkdir`` runs against the same descriptor for the same reason. Creating + with ``parents=True`` from a full path would follow an ancestor that became + a symlink between the check and the creation -- the state-root defect this + replaces -- and no later check on the leaf can see that it happened. + """ + if missing == _MISSING_CREATE: + try: + os.mkdir(name, mode=0o700, dir_fd=parent) + except FileExistsError: + pass + except OSError: + raise ContextError("local graph state directory is unavailable or unsafe") from None try: - handle = os.open(path, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + handle = os.open(name, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=parent) + except FileNotFoundError: + if missing == _MISSING_STOP: + return None + raise ContextError("local graph state directory is unavailable or unsafe") from None except OSError: + # ``ELOOP`` lands here: the component is a symlink, and a symlink is + # not a directory this class owns however private its target may be. raise ContextError("local graph state directory is unavailable or unsafe") from None + if not private: + # An ancestor *above* the operator's private root: this class does not + # own its mode and must not judge it. What matters there is only that + # it was traversed as a directory rather than through a link. + return handle try: _private(handle, directory=True) except ContextError: @@ -1577,15 +1891,83 @@ def lock_path(self) -> Path: """ return self.path.parent / f"{self.workspace}.lock" + #: The components this class owns below the canonical base, outermost + #: first. Spelled out one at a time because each is created and opened + #: against its parent's descriptor: ``mkdir(parents=True)`` would both + #: apply ``0o700`` to the leaf only -- leaving intermediates at the process + #: umask -- and follow an ancestor that became a symlink in between. @property - def _chain(self) -> tuple[Path, ...]: - """Every directory this class owns, outermost first. + def _components(self) -> tuple[str, ...]: + return ("graph", self.workspace, "generations") + + def _create_base(self) -> None: + """Create the private root itself, one no-follow component at a time. + + The root an operator names may not exist yet, ancestors and all, and + creating it is what a first build does. ``mkdir(parents=True)`` cannot + be what does it: a path component that does not exist *cannot* be + canonicalized at construction, so a component created as a symlink + afterwards -- pointing into a repository, say -- would be followed by + the recursive creation, and the state would land somewhere no check + ever looked at. So the walk starts at the deepest component that really + exists and creates each missing one against its parent's descriptor: + the components above the root are not judged for privacy, which is not + this class's business, but none of them is traversed through a link. + """ + existing = self.base + pending: list[str] = [] + while not existing.exists() and existing != existing.parent: + pending.append(existing.name) + existing = existing.parent + handle = _open_private_at(None, str(existing), missing=_MISSING_REFUSE, private=False) + if handle is None: # pragma: no cover - _MISSING_REFUSE raises instead + return + try: + for name in reversed(pending): + deeper = _open_private_at( + handle, name, missing=_MISSING_CREATE, private=False + ) + if deeper is None: # pragma: no cover - creation returns a handle + return + os.close(handle) + handle = deeper + finally: + os.close(handle) - Spelled out because ``mkdir(mode=0o700, parents=True)`` applies its mode - to the leaf only: intermediate directories would be created with the - process umask and end up group- or world-readable. + def _walk(self, *, depth: int, missing: str) -> int: + """Descend the owned components from the base, one descriptor at a time. + + Returns the deepest descriptor reached; the caller closes it. With + ``missing=_MISSING_STOP`` a component that does not exist ends the walk + rather than failing it, which is what a read of state that was never + built needs. Nothing below a component that failed its privacy check is + ever opened, because there is no descriptor left to open it against. """ - return (self.path.parent.parent, self.path.parent, self.path, self.generations_path) + if missing == _MISSING_CREATE: + self._create_base() + opened = _open_private_at( + None, + str(self.base), + missing=_MISSING_STOP if missing == _MISSING_STOP else _MISSING_REFUSE, + ) + if opened is None: + return -1 + handle = opened + try: + for component in self._components[:depth]: + deeper = _open_private_at(handle, component, missing=missing) + if deeper is None: + return handle + os.close(handle) + handle = deeper + except BaseException: + os.close(handle) + raise + return handle + + def _close_walk(self, handle: int) -> None: + if handle >= 0: + os.close(handle) def verify_private(self, *, create: bool = False) -> None: """Re-check ownership and mode on every directory this class owns. @@ -1595,9 +1977,8 @@ def verify_private(self, *, create: bool = False) -> None: chmod -- must fail closed rather than be trusted because it was private when it was written. """ - for directory in self._chain: - if create or directory.exists(): - os.close(_open_private_directory(directory, create=create)) + missing = _MISSING_CREATE if create else _MISSING_STOP + self._close_walk(self._walk(depth=len(self._components), missing=missing)) def ensure(self) -> None: """Create the private tree, refusing to place state inside a repository.""" @@ -1609,26 +1990,34 @@ def _refuse_state_inside_a_repository(self) -> None: A lexical walk alone reads ``--state-dir /outside/link/state`` as being outside every repository even when ``/outside/link`` points at - ``/repo/subdir``, and the ``O_NOFOLLOW`` opens elsewhere would not catch - it either: they protect the final component of each directory this class - owns, not an ancestor somebody else created. The path walked here is the - canonical one built in ``__init__``, which is also the one every write - goes through, so this check cannot be satisfied by one directory and - then applied to another. + ``/repo/subdir``. The path walked here is the canonical one built in + ``__init__``, so a symlinked ancestor that exists now is resolved before + it is judged. + + An ancestor that does *not* exist yet cannot be resolved by anybody, and + this check alone would miss a component created as a symlink afterwards. + That case is answered by construction rather than by re-checking: every + component below the base is created and opened against its parent's + descriptor with ``O_NOFOLLOW``, so a component that is a symlink when + the build reaches it is refused outright instead of traversed. The two + together leave no window: what exists is resolved, and what does not + exist yet can only be created here, by this process, as a real + directory. """ if any((parent / ".git").exists() for parent in (self.path, *self.path.parents)): raise ContextError("local graph state must stay outside Git repositories") - def _ensure_lock_directory(self) -> None: + def _ensure_lock_directory(self) -> int: """Create only what the lock file needs, not the generations tree. ``remove`` takes the same lock, and a removal that first created the state it was asked to delete would report success for a tree it made - itself. + itself. Returns the descriptor of the directory the lock file lives in, + so the lock is opened relative to the directory that was just checked + rather than re-resolved from the root. """ self._refuse_state_inside_a_repository() - for directory in self._chain[:2]: - os.close(_open_private_directory(directory, create=True)) + return self._walk(depth=1, missing=_MISSING_CREATE) def lock(self): """Serialize builds *and removals* for one checkout. @@ -1639,8 +2028,19 @@ def lock(self): mutate state for one checkout is serialized rather than just the pair that was obviously racy. """ - self._ensure_lock_directory() - handle = os.open(self.lock_path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600) + parent = self._ensure_lock_directory() + try: + handle = os.open( + self.lock_path.name, + os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, + 0o600, + dir_fd=parent, + ) + finally: + # The lock file is opened against the descriptor the walk verified, + # so the directory it lands in is the directory that was checked + # and not whatever that path spells by the time this line runs. + self._close_walk(parent) stream = os.fdopen(handle, "a+", encoding="utf-8") try: _private(stream.fileno()) @@ -1684,7 +2084,9 @@ def read_manifest(self, generation: str) -> BuildManifest: if not _GENERATION.fullmatch(generation): raise ContextError("local graph generation must be an opaque identifier") directory = self.generations_path / generation - os.close(_open_private_directory(directory, create=False)) + handle = _open_private_at(None, str(directory), missing=_MISSING_REFUSE) + if handle is not None: # _MISSING_REFUSE raises rather than returning None + os.close(handle) try: handle = os.open(directory / MANIFEST_NAME, os.O_RDONLY | os.O_NOFOLLOW) except OSError: @@ -1918,6 +2320,10 @@ def build_graph( pin=pin, commit=commit, tree=tree, + # The same two directories the scrubbed environment points + # at, so what the provider is told to use and what it is + # allowed to write are one decision rather than two. + writable=(home, temporary), ) ) if not isinstance(result, IndexResult) or result.completeness not in (COMPLETE, PARTIAL): @@ -2101,15 +2507,16 @@ def record(name: str, status: str, message: str, **extra: Any) -> None: # host cannot contain a provider is worth knowing before a build # refuses, and the check reports the mechanism rather than its # arguments, which would be noise. - sandbox = network_sandbox_command() - if sandbox is None: + mechanism = containment_mechanism() + if mechanism is None: record("context-graph-isolation", "fail", - "no OS sandbox on this host was observed denying a child process the network; " - "builds will refuse") + "no OS sandbox on this host was observed denying a child process both the " + "network and the host filesystem; builds will refuse") else: record("context-graph-isolation", "pass", - "the provider would run inside a network-denying OS sandbox", - mechanism=os.path.basename(sandbox[0])) + "the provider would run inside a sandbox that denies it the network and " + "everything outside the build's own directories", + mechanism=mechanism.name) state = GraphStateRoot(repository, root=root) if not state.path.exists(): @@ -2174,6 +2581,7 @@ def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[ "ARTIFACT_NAME", "BuildManifest", "COMPLETE", + "Containment", "EXTRACTION_TIMEOUT_SECONDS", "GenerationStatus", "GraphStateRoot", @@ -2189,6 +2597,8 @@ def iter_generations(repository: Path, *, root: Path | None = None) -> Iterator[ "TrackedCensus", "TrackedEntry", "build_graph", + "containment_mechanism", + "containment_prefix", "doctor_report", "git_environment", "graph_status", diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 4adc79e4..f745e47c 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -15,6 +15,7 @@ from __future__ import annotations +import contextlib import dataclasses import hashlib import io @@ -560,6 +561,58 @@ def test_a_consumer_that_stops_early_stops_the_reader_with_it(self) -> None: """ +READ_PROBE = """ +import sys + +try: + with open(sys.argv[1], "rb") as handle: + handle.read(1) +except OSError: + sys.exit(1) +sys.exit(0) +""" + +#: Set by the CI job that installs a real isolation mechanism. There, a host +#: without one is a broken job rather than a host that cannot be asked, so the +#: skip becomes a failure: coverage that silently skips is coverage nobody has. +REQUIRE_CONTAINMENT = "CODE_MOWER_REQUIRE_CONTAINMENT" + + +def require_containment(test: unittest.TestCase) -> lifecycle.Containment: + mechanism = lifecycle.containment_mechanism() + if mechanism is None: + if os.environ.get(REQUIRE_CONTAINMENT) == "1": + test.fail( + "this job requires a verified isolation mechanism and this host offers none" + ) + test.skipTest("this host offers no OS sandbox that contains a child process") + return mechanism + + +@contextlib.contextmanager +def stand_in_containment(prefix: tuple[str, ...]): + """A verified mechanism whose argv is a fixed stand-in. + + The real prefix is a function of the host, so a test that wants to read the + argv a launch was given -- rather than to prove containment -- pins it. + ``_provider_read_paths`` goes with it: the exposure a build computes names + an install that only a host with the pinned provider on it actually has. + """ + with mock.patch.object( + lifecycle, "containment_mechanism", lambda: lifecycle.Containment("stand-in", "/sandbox") + ): + with mock.patch.object(lifecycle, "containment_prefix", lambda **keywords: prefix): + with mock.patch.object(lifecycle, "_provider_read_paths", lambda command: ()): + yield + + +@contextlib.contextmanager +def no_containment(): + """A host that offers no mechanism at all.""" + with mock.patch.object(lifecycle, "containment_mechanism", lambda: None): + yield + + class NetworkIsolationTests(unittest.TestCase): """The provider's network boundary, against a socket that is really there.""" @@ -583,28 +636,127 @@ def test_an_unsandboxed_child_reaches_the_listening_socket(self) -> None: # some unrelated reason would read as proof of isolation. self.assertEqual(self.connect(()), 0) + def real_prefix(self, scratch: Path) -> tuple[str, ...]: + """The argv this host would really confine a build with.""" + require_containment(self) + return lifecycle.containment_prefix( + writable=(scratch,), readable=lifecycle._interpreter_read_paths() + ) + def test_a_sandboxed_child_cannot_reach_the_listening_socket(self) -> None: - sandbox = lifecycle.network_sandbox_command() - if sandbox is None: - self.skipTest("this host offers no OS sandbox that denies a child the network") - self.assertEqual(self.connect(sandbox), 1) + with tempfile.TemporaryDirectory() as scratch: + self.assertEqual(self.connect(self.real_prefix(Path(scratch))), 1) def test_the_selected_mechanism_really_contains_a_child_on_this_host(self) -> None: """The integration control: the real mechanism, not a stand-in for one. The synthetic launchers below pin the classification *algorithm* on every host, including hosts with no sandbox at all. They cannot show - that ``sandbox-exec``, ``bwrap --unshare-net``, or ``unshare --net`` as - this module spells them actually confines anything. Where one of them is - available, this runs it for real: an unconfined child must reach a - listener that is really there, and a child under the selected prefix - must not. + that ``sandbox-exec`` or ``bwrap`` as this module spells them actually + confines anything. Where one of them is available, this runs it for + real: an unconfined child must reach a listener that is really there + and read a secret planted outside its exposure, and a child under the + selected prefix must do neither. + """ + with tempfile.TemporaryDirectory() as scratch: + prefix = self.real_prefix(Path(scratch)) + self.assertEqual(lifecycle._classify_probe(()), lifecycle._REACHED) + self.assertEqual(lifecycle._classify_probe(prefix), lifecycle._CONTAINED) + + def test_the_selected_mechanism_hides_a_file_outside_the_exposure(self) -> None: + """The filesystem half, named separately from the classifier that uses it. + + The reproduction this replaces read an external ignored ``.env`` through + the selected sandbox: the macOS profile denied the network and allowed + the whole host filesystem, and ``--dev-bind / /`` did the same on Linux. + A working directory is not a boundary. What is exposed is exposed; a + secret beside it is not there at all. """ - prefix = lifecycle.network_sandbox_command() - if prefix is None: - self.skipTest("this host offers no OS sandbox that denies a child the network") - self.assertEqual(lifecycle._classify_probe(()), lifecycle._REACHED) - self.assertEqual(lifecycle._classify_probe(prefix), lifecycle._CONTAINED) + with tempfile.TemporaryDirectory() as scratch: + prefix = self.real_prefix(Path(scratch)) + exposed = Path(scratch) / "inside" + exposed.write_text("visible", encoding="utf-8") + hidden = self.root_outside() / ".env" + hidden.write_text("SECRET=planted", encoding="utf-8") + self.assertEqual(self.read_through(prefix, exposed), 0) + self.assertEqual(self.read_through(prefix, hidden), 1) + + def root_outside(self) -> Path: + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + return Path(directory.name) + + def read_through(self, prefix: tuple[str, ...], path: Path) -> int: + return subprocess.run( + [*prefix, sys.executable, "-c", READ_PROBE, str(path)], + check=False, + capture_output=True, + timeout=120, + ).returncode + + def test_a_mechanism_that_denies_only_the_network_is_not_a_boundary(self) -> None: + """Half a boundary classifies as none. + + A launcher that gives its child an empty network namespace and leaves + the host filesystem in place is exactly what this module used to select. + The child reports it, and the classification refuses it rather than + recording the half it liked. + """ + launcher = self.network_only_launcher(self.closed_port()) + self.assertEqual(lifecycle._classify_probe((launcher,)), lifecycle._UNUSABLE) + + def network_only_launcher(self, port: int) -> str: + """Redirects the probe at a closed port but leaves the secret readable.""" + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + path = Path(directory.name) / "network-only-launcher" + path.write_text(f'#!/bin/sh\nexec "$1" "$2" "$3" {port} "$5" "$6"\n', encoding="utf-8") + path.chmod(0o700) + return str(path) + + def test_a_launcher_on_PATH_cannot_shadow_a_trusted_one(self) -> None: + """Every candidate is an absolute path, so ``PATH`` decides nothing. + + A launcher resolved through the inherited ``PATH`` can be shadowed by a + program that computes the probe's evidence and reports containment + without establishing any, and the whole verdict is then forged. The + shadow is planted with the names this module looks for and must not be + selected -- nor even consulted. + """ + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + for name in ("bwrap", "unshare", "sandbox-exec"): + shadow = Path(directory.name) / name + shadow.write_text("#!/bin/sh\nexit 40\n", encoding="utf-8") + shadow.chmod(0o700) + for _, candidate in lifecycle._SANDBOX_CANDIDATES: + self.assertTrue(os.path.isabs(candidate), candidate) + self.assertFalse(candidate.startswith(directory.name)) + with mock.patch.dict(os.environ, {"PATH": directory.name}): + with mock.patch.object(lifecycle, "_classify_probe", lambda prefix: lifecycle._REACHED): + # Every candidate classifies as "reached", so nothing can be + # selected; the point is that the shadow was never a candidate. + self.assertIsNone(lifecycle._probe_containment()) + + def test_a_launcher_anybody_could_replace_is_not_trusted(self) -> None: + """Ownership and ancestry, not just the file's own mode. + + An executable in a directory somebody else may write can be replaced + between the probe that trusted it and the build that runs it. A + system launcher is the positive control: root-owned, in root-owned + directories nobody else may write. + """ + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + launcher = Path(directory.name) / "launcher" + launcher.write_text("#!/bin/sh\nexit 40\n", encoding="utf-8") + launcher.chmod(0o700) + os.chmod(directory.name, 0o777) + self.assertIsNone(lifecycle._trusted_launcher(str(launcher))) + self.assertIsNone(lifecycle._trusted_launcher("/nonexistent/launcher")) + if not os.path.isfile("/usr/bin/env"): # pragma: no cover - platform + self.skipTest("no system launcher to use as the positive control") + self.assertEqual(lifecycle._trusted_launcher("/usr/bin/env"), "/usr/bin/env") def launcher(self, exit_code: int) -> str: """A stand-in launcher, so the classifier is pinned on every host. @@ -640,14 +792,19 @@ def redirecting_launcher(self, port: int) -> str: loopback really comes up inside the new namespace, and the connection is *refused* because the host's listener is not in there with it. The launcher runs the probe it was handed against a port nothing is on, - which is what the child would have seen. ``$5`` is the nonce, forwarded - so the child can still show it ran: a launcher that swallowed it would - be a launcher that did not run the probe. + which is what the child would have seen, and points it at a path that + does not exist in place of the planted secret, which is what a child + with no view of the host filesystem sees. ``$6`` is the nonce, + forwarded so the child can still show it ran: a launcher that swallowed + it would be a launcher that did not run the probe. """ directory = tempfile.TemporaryDirectory() self.addCleanup(directory.cleanup) path = Path(directory.name) / "namespace-launcher" - path.write_text(f'#!/bin/sh\nexec "$1" "$2" "$3" {port} "$5"\n', encoding="utf-8") + path.write_text( + f'#!/bin/sh\nexec "$1" "$2" "$3" {port} /nonexistent/secret "$6"\n', + encoding="utf-8", + ) path.chmod(0o700) return str(path) @@ -661,10 +818,10 @@ def test_a_launcher_that_never_runs_the_child_is_rejected(self) -> None: it as a boundary and every build would then run its provider unconfined. The child's per-run evidence is what separates the two. """ - self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_DENIED),))) + self.assertFalse(lifecycle._prefix_confines((self.launcher(lifecycle._PROBE_BASE),))) def test_a_child_that_reached_the_network_stack_is_rejected(self) -> None: - self.assertFalse(lifecycle._sandbox_denies_network((self.launcher(lifecycle._PROBE_REACHED),))) + self.assertFalse(lifecycle._prefix_confines((self.launcher(lifecycle._PROBE_BASE + lifecycle._PROBE_REACHED_LISTENER),))) def test_evidence_from_another_run_does_not_prove_this_one(self) -> None: """A replayed transcript is not a child that ran. @@ -678,14 +835,14 @@ def test_evidence_from_another_run_does_not_prove_this_one(self) -> None: self.addCleanup(directory.cleanup) path = Path(directory.name) / "replaying-launcher" path.write_text( - f"#!/bin/sh\necho {stale}\necho \"$@\"\nexit {lifecycle._PROBE_DENIED}\n", + f"#!/bin/sh\necho {stale}\necho \"$@\"\nexit {lifecycle._PROBE_BASE}\n", encoding="utf-8", ) path.chmod(0o700) - self.assertFalse(lifecycle._sandbox_denies_network((str(path),))) + self.assertFalse(lifecycle._prefix_confines((str(path),))) def test_a_launcher_that_cannot_start_is_rejected(self) -> None: - self.assertFalse(lifecycle._sandbox_denies_network(("/nonexistent/launcher",))) + self.assertFalse(lifecycle._prefix_confines(("/nonexistent/launcher",))) def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: # ``env`` runs its argument unchanged: a prefix that contains nothing @@ -693,7 +850,7 @@ def test_a_candidate_that_does_not_deny_the_network_is_rejected(self) -> None: passthrough = ("/usr/bin/env",) if not os.access(passthrough[0], os.X_OK): # pragma: no cover - platform self.skipTest("no pass-through launcher to test against") - self.assertFalse(lifecycle._sandbox_denies_network(passthrough)) + self.assertFalse(lifecycle._prefix_confines(passthrough)) def test_a_refused_connection_inside_a_namespace_is_containment(self) -> None: """The bubblewrap case: refused by an empty namespace, not by the host. @@ -704,7 +861,7 @@ def test_a_refused_connection_inside_a_namespace_is_containment(self) -> None: to build at all. The verdict is taken at the listener instead: nothing arrived, so the child was contained. """ - self.assertTrue(lifecycle._sandbox_denies_network((self.redirecting_launcher(self.closed_port()),))) + self.assertTrue(lifecycle._prefix_confines((self.redirecting_launcher(self.closed_port()),))) def test_the_probe_proves_its_own_apparatus_before_trusting_a_refusal(self) -> None: """No candidate passes if an unsandboxed child cannot reach the listener. @@ -721,7 +878,7 @@ def classify(prefix): return lifecycle._CONTAINED with mock.patch.object(lifecycle, "_classify_probe", classify): - self.assertIsNone(lifecycle._probe_network_sandbox()) + self.assertIsNone(lifecycle._probe_containment()) # The control ran, and nothing was probed after it failed. self.assertEqual(calls, [()]) @@ -792,7 +949,7 @@ def fake_popen(argv, **kwargs): (written / "manifest.json").write_text(json.dumps(report), encoding="utf-8") return FakeChild() - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: sandbox): + with stand_in_containment(sandbox): indexer = lifecycle.subprocess_indexer(executable) # Patched only around the launch, so the Git calls a build makes are # never intercepted by this stand-in. @@ -941,7 +1098,7 @@ def fake_popen(argv, **kwargs): (written / "manifest.json").write_bytes(b"{not json") return FakeChild() - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") with mock.patch.object(subprocess, "Popen", fake_popen): result = indexer(request) @@ -1000,7 +1157,7 @@ def fake_popen(argv, **kwargs): def refuse_whole_file_read(self: Path) -> bytes: raise AssertionError(f"{self} was read whole") - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") with mock.patch.object(subprocess, "Popen", fake_popen): with mock.patch.object(Path, "read_bytes", refuse_whole_file_read): @@ -1024,7 +1181,7 @@ def fake_popen(argv, **kwargs): launched.append(list(argv)) return FakeChild() - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") with mock.patch.object(subprocess, "Popen", fake_popen): with self.assertRaises(ContextError): @@ -1048,7 +1205,7 @@ def fake_popen(argv, **kwargs): def record_termination(child, group) -> None: order.append("stopped") - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") with mock.patch.object(subprocess, "Popen", fake_popen): with mock.patch.object(lifecycle, "_terminate_process_group", record_termination): @@ -1060,7 +1217,7 @@ def record_termination(child, group) -> None: self.assertFalse(request.output_path.exists()) def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): + with no_containment(): with self.assertRaises(ContextError): lifecycle.subprocess_indexer("graphify") @@ -1134,6 +1291,27 @@ def reaped(self, pid: int, *, within: float = 15.0) -> bool: time.sleep(0.05) return False + def test_a_group_that_cannot_be_established_as_empty_fails_the_run(self) -> None: + """Sending SIGKILL is not the group being gone. + + The caller's next act is to pack or delete the state these processes + are writing, so returning on the strength of a signal that was sent -- + rather than on a group that was observed empty -- hands the rest of the + build a race it cannot see. A group that outlasts the kill fails the + run instead. + """ + with tempfile.TemporaryDirectory() as directory: + with mock.patch.object(lifecycle, "_group_is_empty", lambda group: False): + with mock.patch.object(lifecycle, "_await_group_exit", lambda group, timeout: None): + with self.assertRaises(ContextError) as raised: + lifecycle._run_contained( + [sys.executable, "-c", "pass"], + environment={"PATH": os.environ.get("PATH", "")}, + cwd=directory, + timeout=30.0, + ) + self.assertIn("could not be stopped", str(raised.exception)) + def test_a_timed_out_run_takes_the_workers_it_started_with_it(self) -> None: with tempfile.TemporaryDirectory() as directory: recorded = Path(directory) / "worker.pid" @@ -1423,6 +1601,27 @@ def test_an_ancestor_retargeted_after_the_check_does_not_move_the_state(self) -> self.assertTrue((safe / "graph-state" / "graph").is_dir()) self.assertFalse((inside / "graph-state").exists()) + def test_a_component_that_becomes_a_symlink_before_creation_is_refused(self) -> None: + """The half a canonical snapshot cannot cover: a component that is not there yet. + + Resolving the root at construction settles what its *existing* + ancestors mean. It cannot settle what a component nobody has created + yet will mean, and a recursive ``mkdir`` would follow whatever appears + there. Here the nested root is absent at construction and an ancestor + of it is created as a link into the repository before ``ensure``: the + creation must refuse rather than write state inside the repository. + """ + outside = self.root / "outside" + outside.mkdir() + absent = outside / "deep" / "state" + state = lifecycle.GraphStateRoot(self.repository, root=absent) + inside = self.repository / "subdir" + inside.mkdir() + os.symlink(inside, outside / "deep") + with self.assertRaises(ContextError): + state.ensure() + self.assertFalse((inside / "state").exists()) + def test_a_manifest_no_reader_could_load_publishes_nothing(self) -> None: """Publication validates the whole manifest before it touches state. @@ -1770,13 +1969,13 @@ def test_a_healthy_build_passes(self) -> None: self.build() # Isolation is a property of the host, not of this build; a host that # offers a sandbox is the healthy case being described here. - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: ("/sandbox",)): + with stand_in_containment(("/sandbox",)): report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) self.assertEqual(report["status"], "pass") def test_a_host_that_cannot_contain_a_provider_fails_doctor(self) -> None: self.build() - with mock.patch.object(lifecycle, "network_sandbox_command", lambda: None): + with no_containment(): report = lifecycle.doctor_report(self.repository, pin=PIN, root=self.state) self.assertEqual(report["status"], "fail") isolation = [check for check in report["checks"] if check["check"] == "context-graph-isolation"] @@ -1842,8 +2041,7 @@ def base(self) -> list[str]: def test_build_status_refresh_remove_round_trip(self) -> None: # The only test that launches a provider for real, so it is also the # only one that needs the host to offer the sandbox a build requires. - if lifecycle.network_sandbox_command() is None: - self.skipTest("this host offers no OS sandbox that denies a child the network") + require_containment(self) pin, indexer = str(self.pin_file()), str(self.indexer_script()) code, output = self.run_command("build", *self.base(), "--pin-file", pin, "--indexer", indexer) self.assertEqual(code, 0, output) @@ -1872,8 +2070,7 @@ def test_a_provider_that_admits_an_incomplete_run_is_not_usable(self) -> None: # End to end through the real launcher: the provider exits zero and # writes state, and the build is still refused because its own report # denies completion. Exit status is not completion evidence. - if lifecycle.network_sandbox_command() is None: - self.skipTest("this host offers no OS sandbox that denies a child the network") + require_containment(self) code, output = self.run_command( "build", *self.base(), "--pin-file", str(self.pin_file()), @@ -1891,8 +2088,7 @@ def test_a_provider_that_admits_an_incomplete_run_is_not_usable(self) -> None: self.assertEqual(json.loads(output)["state"], "partial") def test_an_incomplete_build_is_printed_as_partial(self) -> None: - if lifecycle.network_sandbox_command() is None: - self.skipTest("this host offers no OS sandbox that denies a child the network") + require_containment(self) arguments = [argument for argument in self.base() if argument != "--json"] code, output = self.run_command( "build", *arguments, From 92d9810e3c5ff1794db56be6615090b44ef7c864 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 10:11:56 -0700 Subject: [PATCH 17/25] context: probe the boundary from inside it, and stand in for it at launch CI at 5326e83 was red: 23 failures in the lifecycle suite and the required containment job. The 23 were one test defect. The sandbox prefix is now built per run, from what that run exposes, so it is computed inside `indexer(request)` rather than at construction -- but the stand-in only spanned the construction. Every launch test therefore ran the real probe on a host with no bubblewrap and got the fail-closed refusal, including the overrun test, which recorded `['reported']` because the refusal arrived before any group existed to stop. The stand-in now spans the launch too. With the prefix built per run and nothing asserting what it is built from, `test_the_exposure_is_built_from_what_this_run_writes` reads the exposure back from the call: this request's materialized copy and this request's scratch, which is what the environment points `HOME` and `TMPDIR` at. The probe child now starts inside the exposure. A build's provider starts in the materialized copy, which the boundary always exposes; the probe was left in this process's own working directory, which the boundary deliberately does not, so the launcher was being asked to start a child in a directory its sandbox does not have. That is a launcher failure, not a finding about the mechanism. The required job says why rather than only that: `containment_evidence()` reports the control's verdict, each candidate's verdict, and the launcher's own stderr. The job also enables unprivileged user namespaces, which Ubuntu 24.04 restricts by default and bubblewrap needs -- a property of the runner, not of the product, which keeps refusing builds on a host that refuses. Not executed on this lane: Python execution is refused in this session, so CI is again the first run. --- .github/workflows/ci.yml | 14 +++ src/code_mower/context_graph_lifecycle.py | 33 +++-- tests/test_context_graph_lifecycle.py | 141 +++++++++++++++++----- 3 files changed, 150 insertions(+), 38 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6c2da5a1..da14c444 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -112,6 +112,20 @@ jobs: sudo apt-get install -y bubblewrap test -x /usr/bin/bwrap + # Ubuntu 24.04 restricts unprivileged user namespaces by default, which + # is what bubblewrap needs to build a mount namespace without being + # setuid. Enabling it is a property of this runner, not of the product: + # a host that refuses keeps refusing builds, which is the fail-closed + # posture. Reported rather than asserted, so the suite below is what + # decides the job. + - name: Allow unprivileged user namespaces + run: | + sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true + sudo sysctl -w kernel.unprivileged_userns_clone=1 || true + bwrap --unshare-net --dev-bind / / /bin/true \ + && echo "bwrap: namespaces available" \ + || echo "bwrap: refused a namespace on this runner" + - name: Install package run: python -m pip install -e . diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 470a8ade..f16eb4c9 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -414,12 +414,19 @@ def _ran_the_probe(output: bytes, nonce: str) -> bool: return expected in output.decode("utf-8", "replace") -def _classify_probe(prefix: Sequence[str]) -> str: +def _classify_probe(prefix: Sequence[str], *, cwd: str | None = None) -> str: """Run the probe under ``prefix``: a real listener and a real planted secret. The secret is written to the host's temporary directory, which no exposure this module builds ever includes, so an unconfined child reads it and a confined one cannot see it at all. + + ``cwd`` is the directory the probe child starts in, and it belongs inside + the exposure being probed. A build's provider starts in the materialized + copy, which the boundary always exposes; a probe left in this process's own + working directory starts somewhere the boundary deliberately does not + expose, which is a launcher failure rather than a finding about the + mechanism. """ nonce = secrets.token_hex(16) handle, secret = tempfile.mkstemp(prefix="code-mower-containment-probe-") @@ -444,6 +451,7 @@ def _classify_probe(prefix: Sequence[str]) -> str: check=False, capture_output=True, timeout=60, + cwd=cwd, env={"PATH": os.environ.get("PATH", ""), **_NETWORK_DENY}, ) except (OSError, subprocess.SubprocessError): @@ -476,9 +484,9 @@ def _classify_probe(prefix: Sequence[str]) -> str: return _UNUSABLE -def _prefix_confines(prefix: Sequence[str]) -> bool: +def _prefix_confines(prefix: Sequence[str], *, cwd: str | None = None) -> bool: """Watch a child under ``prefix`` fail to reach either thing that is really there.""" - return _classify_probe(prefix) == _CONTAINED + return _classify_probe(prefix, cwd=cwd) == _CONTAINED def _interpreter_read_paths() -> tuple[str, ...]: @@ -493,22 +501,25 @@ def _interpreter_read_paths() -> tuple[str, ...]: def _probe_containment() -> Containment | None: if not sys.executable: # pragma: no cover - a frozen interpreter cannot probe return None - # The control, first: a child with no prefix must reach the listener *and* - # read the planted secret. If it cannot -- no probe interpreter, loopback - # blocked, an unreadable temporary directory -- then "could not" proves - # nothing about any candidate, and every candidate would pass for a - # boundary. Refuse the whole probe instead. - if _classify_probe(()) != _REACHED: - return None readable = _interpreter_read_paths() + # The scratch directory is the writable exposure *and* the directory every + # probe child starts in, control included, so the control and the candidates + # differ in the boundary and in nothing else. with tempfile.TemporaryDirectory(prefix="code-mower-containment-") as scratch: + # The control, first: a child with no prefix must reach the listener + # *and* read the planted secret. If it cannot -- no probe interpreter, + # loopback blocked, an unreadable temporary directory -- then "could + # not" proves nothing about any candidate, and every candidate would + # pass for a boundary. Refuse the whole probe instead. + if _classify_probe((), cwd=scratch) != _REACHED: + return None for name, path in _SANDBOX_CANDIDATES: launcher = _trusted_launcher(path) if launcher is None: continue mechanism = Containment(name=name, launcher=launcher) prefix = _prefix_for(mechanism, writable=(scratch,), readable=readable) - if _prefix_confines(prefix): + if _prefix_confines(prefix, cwd=scratch): return mechanism return None diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index f745e47c..1ce246a4 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -578,12 +578,51 @@ def test_a_consumer_that_stops_early_stops_the_reader_with_it(self) -> None: REQUIRE_CONTAINMENT = "CODE_MOWER_REQUIRE_CONTAINMENT" +def containment_evidence() -> str: + """Why this host offered no mechanism, in the terms the probe decided in. + + A required job that fails with "none" tells an operator nothing they can + act on: a launcher can be absent, present but untrusted, or present and + trusted and unable to start a child at all. So the failure carries the + control's verdict, each candidate's verdict, and the launcher's own stderr + -- which is where a refused namespace or an impossible bind says so. + """ + lines: list[str] = [] + readable = lifecycle._interpreter_read_paths() + with tempfile.TemporaryDirectory() as scratch: + lines.append(f"control (no prefix): {lifecycle._classify_probe((), cwd=scratch)}") + for name, path in lifecycle._SANDBOX_CANDIDATES: + launcher = lifecycle._trusted_launcher(path) + if launcher is None: + lines.append(f"{name} at {path}: exists={os.path.exists(path)}, not trusted") + continue + prefix = lifecycle._prefix_for( + lifecycle.Containment(name=name, launcher=launcher), + writable=(scratch,), + readable=readable, + ) + started = subprocess.run( + [*prefix, sys.executable, "-c", "print('started')"], + check=False, + capture_output=True, + timeout=120, + cwd=scratch, + ) + lines.append( + f"{name} at {path}: verdict={lifecycle._classify_probe(prefix, cwd=scratch)}" + f" start={started.returncode} stdout={started.stdout[:200]!r}" + f" stderr={started.stderr[:400]!r}" + ) + return "\n".join(lines) + + def require_containment(test: unittest.TestCase) -> lifecycle.Containment: mechanism = lifecycle.containment_mechanism() if mechanism is None: if os.environ.get(REQUIRE_CONTAINMENT) == "1": test.fail( - "this job requires a verified isolation mechanism and this host offers none" + "this job requires a verified isolation mechanism and this host offers none\n" + + containment_evidence() ) test.skipTest("this host offers no OS sandbox that contains a child process") return mechanism @@ -623,12 +662,17 @@ def setUp(self) -> None: self.listener.listen(1) self.port = self.listener.getsockname()[1] - def connect(self, prefix: tuple[str, ...]) -> int: + def connect(self, prefix: tuple[str, ...], *, cwd: Path | None = None) -> int: + # ``cwd`` names a directory inside the exposure, because that is where a + # build's provider starts: the boundary does not expose this process's + # own working directory, and a child asked to start in a directory its + # sandbox does not have never runs at all. return subprocess.run( [*prefix, sys.executable, "-c", CONNECT_PROBE, str(self.port)], check=False, capture_output=True, timeout=120, + cwd=None if cwd is None else str(cwd), ).returncode def test_an_unsandboxed_child_reaches_the_listening_socket(self) -> None: @@ -645,7 +689,8 @@ def real_prefix(self, scratch: Path) -> tuple[str, ...]: def test_a_sandboxed_child_cannot_reach_the_listening_socket(self) -> None: with tempfile.TemporaryDirectory() as scratch: - self.assertEqual(self.connect(self.real_prefix(Path(scratch))), 1) + prefix = self.real_prefix(Path(scratch)) + self.assertEqual(self.connect(prefix, cwd=Path(scratch)), 1) def test_the_selected_mechanism_really_contains_a_child_on_this_host(self) -> None: """The integration control: the real mechanism, not a stand-in for one. @@ -660,8 +705,8 @@ def test_the_selected_mechanism_really_contains_a_child_on_this_host(self) -> No """ with tempfile.TemporaryDirectory() as scratch: prefix = self.real_prefix(Path(scratch)) - self.assertEqual(lifecycle._classify_probe(()), lifecycle._REACHED) - self.assertEqual(lifecycle._classify_probe(prefix), lifecycle._CONTAINED) + self.assertEqual(lifecycle._classify_probe((), cwd=scratch), lifecycle._REACHED) + self.assertEqual(lifecycle._classify_probe(prefix, cwd=scratch), lifecycle._CONTAINED) def test_the_selected_mechanism_hides_a_file_outside_the_exposure(self) -> None: """The filesystem half, named separately from the classifier that uses it. @@ -678,20 +723,21 @@ def test_the_selected_mechanism_hides_a_file_outside_the_exposure(self) -> None: exposed.write_text("visible", encoding="utf-8") hidden = self.root_outside() / ".env" hidden.write_text("SECRET=planted", encoding="utf-8") - self.assertEqual(self.read_through(prefix, exposed), 0) - self.assertEqual(self.read_through(prefix, hidden), 1) + self.assertEqual(self.read_through(prefix, exposed, cwd=Path(scratch)), 0) + self.assertEqual(self.read_through(prefix, hidden, cwd=Path(scratch)), 1) def root_outside(self) -> Path: directory = tempfile.TemporaryDirectory() self.addCleanup(directory.cleanup) return Path(directory.name) - def read_through(self, prefix: tuple[str, ...], path: Path) -> int: + def read_through(self, prefix: tuple[str, ...], path: Path, *, cwd: Path | None = None) -> int: return subprocess.run( [*prefix, sys.executable, "-c", READ_PROBE, str(path)], check=False, capture_output=True, timeout=120, + cwd=None if cwd is None else str(cwd), ).returncode def test_a_mechanism_that_denies_only_the_network_is_not_a_boundary(self) -> None: @@ -733,7 +779,9 @@ def test_a_launcher_on_PATH_cannot_shadow_a_trusted_one(self) -> None: self.assertTrue(os.path.isabs(candidate), candidate) self.assertFalse(candidate.startswith(directory.name)) with mock.patch.dict(os.environ, {"PATH": directory.name}): - with mock.patch.object(lifecycle, "_classify_probe", lambda prefix: lifecycle._REACHED): + with mock.patch.object( + lifecycle, "_classify_probe", lambda prefix, **keywords: lifecycle._REACHED + ): # Every candidate classifies as "reached", so nothing can be # selected; the point is that the shadow was never a candidate. self.assertIsNone(lifecycle._probe_containment()) @@ -873,7 +921,9 @@ def test_the_probe_proves_its_own_apparatus_before_trusting_a_refusal(self) -> N """ calls: list[tuple[str, ...]] = [] - def classify(prefix): + def classify(prefix, **keywords): + # The probe child starts inside the exposure, so every call names a + # working directory; what this test reads is the prefix. calls.append(tuple(prefix)) return lifecycle._CONTAINED @@ -949,12 +999,16 @@ def fake_popen(argv, **kwargs): (written / "manifest.json").write_text(json.dumps(report), encoding="utf-8") return FakeChild() + # The stand-in spans the launch as well as the construction: the + # exposure is built per run, from what *that* run exposes, so the + # prefix is computed inside ``indexer(request)`` and a host with no + # real mechanism would otherwise refuse there. ``Popen`` is patched + # only around the launch, so the Git calls a build makes are never + # intercepted by this stand-in. with stand_in_containment(sandbox): indexer = lifecycle.subprocess_indexer(executable) - # Patched only around the launch, so the Git calls a build makes are - # never intercepted by this stand-in. - with mock.patch.object(subprocess, "Popen", fake_popen): - result = indexer(request) + with mock.patch.object(subprocess, "Popen", fake_popen): + result = indexer(request) return recorded[0], result def launched_argv(self, executable: str, *, sandbox=("/sandbox", "--deny")) -> list[str]: @@ -964,6 +1018,39 @@ def test_the_provider_is_launched_inside_the_sandbox(self) -> None: argv = self.launched_argv("graphify") self.assertEqual(argv[:3], ["/sandbox", "--deny", "graphify"]) + def test_the_exposure_is_built_from_what_this_run_writes(self) -> None: + """The boundary names this request's copy and this request's scratch. + + A prefix computed once, at construction, could not name either: both + are made per build. The launch would then confine the provider to + somewhere other than the tree it was asked to index, and the + redirected ``HOME`` and ``TMPDIR`` the environment points at would be + absent from the child's filesystem view -- a provider that cannot + start. So the exposure is read back from the call itself. + """ + scratch = Path(tempfile.mkdtemp(dir=self.root)) + request = dataclasses.replace(self.request(), writable=(scratch,)) + recorded: list[dict[str, object]] = [] + + def record_prefix(**keywords: object) -> tuple[str, ...]: + recorded.append(dict(keywords)) + return ("/sandbox",) + + def fake_popen(argv, **kwargs): + written = request.source_root / ".graphify" + written.mkdir(exist_ok=True) + (written / "graph.bin").write_bytes(b"graph-bytes") + (written / "manifest.json").write_text(json.dumps(FINISHED_REPORT), encoding="utf-8") + return FakeChild() + + with stand_in_containment(("/sandbox",)): + indexer = lifecycle.subprocess_indexer("graphify") + with mock.patch.object(lifecycle, "containment_prefix", record_prefix): + with mock.patch.object(subprocess, "Popen", fake_popen): + indexer(request) + self.assertEqual(len(recorded), 1) + self.assertEqual(recorded[0]["writable"], (request.source_root, scratch)) + def test_the_provider_is_given_no_stream_this_process_has_to_hold(self) -> None: """A talkative indexer must not be able to fill this process's memory. @@ -1100,8 +1187,8 @@ def fake_popen(argv, **kwargs): with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "Popen", fake_popen): - result = indexer(request) + with mock.patch.object(subprocess, "Popen", fake_popen): + result = indexer(request) self.assertEqual(result.completeness, lifecycle.PARTIAL) def test_a_report_without_affirmative_completion_evidence_is_partial(self) -> None: @@ -1159,9 +1246,9 @@ def refuse_whole_file_read(self: Path) -> bytes: with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "Popen", fake_popen): - with mock.patch.object(Path, "read_bytes", refuse_whole_file_read): - result = indexer(request) + with mock.patch.object(subprocess, "Popen", fake_popen): + with mock.patch.object(Path, "read_bytes", refuse_whole_file_read): + result = indexer(request) self.assertEqual(result.completeness, lifecycle.PARTIAL) def test_extraction_refuses_to_run_over_pre_existing_provider_state(self) -> None: @@ -1183,9 +1270,9 @@ def fake_popen(argv, **kwargs): with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "Popen", fake_popen): - with self.assertRaises(ContextError): - indexer(request) + with mock.patch.object(subprocess, "Popen", fake_popen): + with self.assertRaises(ContextError): + indexer(request) self.assertEqual(launched, []) self.assertFalse(request.output_path.exists()) @@ -1207,11 +1294,11 @@ def record_termination(child, group) -> None: with stand_in_containment(("/sandbox",)): indexer = lifecycle.subprocess_indexer("graphify") - with mock.patch.object(subprocess, "Popen", fake_popen): - with mock.patch.object(lifecycle, "_terminate_process_group", record_termination): - with self.assertRaises(ContextError) as raised: - indexer(request) - order.append("reported") + with mock.patch.object(subprocess, "Popen", fake_popen): + with mock.patch.object(lifecycle, "_terminate_process_group", record_termination): + with self.assertRaises(ContextError) as raised: + indexer(request) + order.append("reported") self.assertEqual(order, ["stopped", "reported"]) self.assertIn("time budget", str(raised.exception)) self.assertFalse(request.output_path.exists()) From 5d7c9db090ee016144bf502fd64a0d0a34a04dc4 Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 10:17:45 -0700 Subject: [PATCH 18/25] context: expose the loader's own spelling, not only what it resolves to The required containment job said why this time, which is what the evidence was for: `bwrap: execvp .../bin/python: No such file or directory`. The namespace was built, the launcher was trusted, and nothing in it could exec. `_existing()` resolved every exposure to the kernel's path, which is right for a seatbelt rule -- macOS puts `/tmp` and `/var` behind links into `private`, and a subpath match is against the resolved path. It is wrong for a bind mount, whose destination is a literal path in a root that is empty until something is put there. Resolving `/lib64` to `/usr/lib64` and binding it there leaves a root with no `/lib64` at all, and `/lib64/ld-linux-x86-64.so.2` is how every ELF binary on this host names its program interpreter. A missing interpreter surfaces as ENOENT against the binary, which is why the earlier "no mechanism" verdict pointed nowhere. The spelling is now the mechanism's to choose: seatbelt resolves, bubblewrap gets the literal path and its resolved self. The interpreter and the pinned provider are named both ways for the same reason -- a virtual environment's `bin/python` is a link, and the child is launched by the name this process knows rather than by what it points at. The 23 unit failures from 5326e83 are fixed at 92d9810: package_matrix passes on 3.12, 3.13 and 3.14. This commit addresses the containment job alone. Not executed on this lane: Python execution is refused in this session. --- src/code_mower/context_graph_lifecycle.py | 62 ++++++++++++++++------- 1 file changed, 43 insertions(+), 19 deletions(-) diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index f16eb4c9..2acd86de 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -294,22 +294,31 @@ def _trusted_launcher(path: str) -> str | None: return path -def _existing(paths: Iterable[Path | str]) -> tuple[str, ...]: - """Resolved, de-duplicated, existing paths, in the order they were given. - - Resolved because both mechanisms match on the kernel's path, not the - caller's spelling: macOS puts ``/tmp`` and ``/var`` behind links into its - ``private`` directory, so an unresolved exposure would name a path the - sandbox never sees. +def _existing(paths: Iterable[Path | str], *, follow: bool = True) -> tuple[str, ...]: + """De-duplicated existing paths, in the order they were given. + + ``follow`` is which spelling the mechanism decides on. A seatbelt + ``subpath`` matches the kernel's resolved path -- macOS puts ``/tmp`` and + ``/var`` behind links into its ``private`` directory -- so an unresolved + exposure there would name a path the sandbox never sees. + + A bind mount is the other way round. The destination is a literal path in + an otherwise empty root, and Linux's ``/lib64 -> usr/lib64`` compatibility + links are how every ELF binary names its program interpreter. A root with + ``/usr/lib64`` and no ``/lib64`` cannot exec anything at all, and says so + as an ``execvp`` ENOENT naming the binary rather than the loader it could + not find. So a mechanism that does not follow gets both spellings. """ seen: dict[str, None] = {} for path in paths: + literal = os.fspath(path) try: - real = os.path.realpath(path) + real = os.path.realpath(literal) except OSError: # pragma: no cover - realpath does not raise on absent paths continue - if os.path.exists(real): - seen.setdefault(real, None) + for candidate in (real,) if follow else (literal, real): + if os.path.exists(candidate): + seen.setdefault(candidate, None) return tuple(seen) @@ -379,9 +388,13 @@ def _bubblewrap_prefix(launcher: str, *, writable: Sequence[str], readable: Sequ return tuple(argv) -_PREFIX_BUILDERS: Mapping[str, Callable[..., tuple[str, ...]]] = { - "sandbox-exec": _seatbelt_prefix, - "bwrap": _bubblewrap_prefix, +#: Each mechanism's prefix builder, and whether it decides on the resolved +#: path. A seatbelt rule matches what the kernel resolved to; a bind mount +#: names a destination in an empty root, where a link's own spelling is a path +#: the child still has to be able to walk. +_PREFIX_BUILDERS: Mapping[str, tuple[Callable[..., tuple[str, ...]], bool]] = { + "sandbox-exec": (_seatbelt_prefix, True), + "bwrap": (_bubblewrap_prefix, False), } @@ -391,10 +404,11 @@ def _prefix_for( writable: Sequence[Path | str], readable: Sequence[Path | str], ) -> tuple[str, ...]: - return _PREFIX_BUILDERS[mechanism.name]( + builder, follow = _PREFIX_BUILDERS[mechanism.name] + return builder( mechanism.launcher, - writable=_existing(writable), - readable=_existing([*_SYSTEM_READ_PATHS, *readable]), + writable=_existing(writable, follow=follow), + readable=_existing([*_SYSTEM_READ_PATHS, *readable], follow=follow), ) @@ -490,10 +504,15 @@ def _prefix_confines(prefix: Sequence[str], *, cwd: str | None = None) -> bool: def _interpreter_read_paths() -> tuple[str, ...]: - """The minimum a probe child needs to be a running Python at all.""" + """The minimum a probe child needs to be a running Python at all. + + Both spellings of the interpreter: a virtual environment's ``bin/python`` + is a link, and the child is launched by the name this process knows it by, + not by the name it resolves to. + """ return tuple( path - for path in (os.path.realpath(sys.executable), sys.prefix, sys.base_prefix) + for path in (sys.executable, os.path.realpath(sys.executable), sys.prefix, sys.base_prefix) if path ) @@ -1206,8 +1225,13 @@ def _provider_read_paths(command: str) -> tuple[str, ...]: if not located: raise ContextError("local graph provider executable could not be located for containment") real = Path(os.path.realpath(located)) + # ``located`` as well as its resolved self: the child is launched by the + # name this process resolved the command to, and a console script in a + # virtual environment reaches its libraries through that environment's own + # spelling rather than through whatever the link points at. return tuple( - str(path) for path in (real, real.parent, real.parent.parent, Path(sys.base_prefix)) + str(path) + for path in (located, real, real.parent, real.parent.parent, Path(sys.base_prefix)) ) From 074df1c1f64ae14d15485b661e951d155896e18f Mon Sep 17 00:00:00 2001 From: Jeff Huber Date: Sat, 12 Sep 2026 10:42:45 -0700 Subject: [PATCH 19/25] context: walk the state root from the filesystem root, and prove the provider's install Three P1 findings from the Codex supplemental review at 5d7c9db. The state root was opened by its full absolute spelling in one call, so O_NOFOLLOW covered the final component and every ancestor above it was resolved exactly as a planted link would want. Finding the deepest existing prefix first did not help, because that prefix was opened the same way: a root pre-created behind a newly inserted ancestor link was accepted, since the leaf really is a directory and really is not a link. It is simply not the directory that was checked. The walk now starts at / and opens one component at a time against its parent's descriptor. Renames and removals still travel full paths, so the inode the walk arrived at is compared against the one the path spells now before each. The provider exposure was the executable's parent and grandparent, which is a guess about a layout rather than knowledge of one, and wrong in the widest direction: ~/bin/graphify exposes the whole home, /opt/graphify exposes /, and a provider inside the checkout exposes the live working tree the materialized copy exists to avoid showing anyone. An install root is now proved -- a pyvenv.cfg whose script directory holds the executable -- and refused outright if it is the filesystem root, the home, the checkout, or an ancestor of either. The prefix a build ends up with is probed before that build runs, rather than only the host's mechanism at startup. macOS gets its own containment job. The Linux job proves bubblewrap and nothing about Seatbelt, and a profile exercised only by whoever happens to run the suite on a laptop is how it went unexecuted. Reporting, not gating, for now: a red result carries the probe's account of which candidate failed and what the launcher said. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 35 +++ docs/context-graph-lifecycle.md | 59 ++++- src/code_mower/context_graph_command.py | 2 +- src/code_mower/context_graph_lifecycle.py | 297 +++++++++++++++++----- tests/test_context_graph_lifecycle.py | 170 ++++++++++++- 5 files changed, 485 insertions(+), 78 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index da14c444..b5397a27 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -134,6 +134,41 @@ jobs: CODE_MOWER_REQUIRE_CONTAINMENT: "1" run: python -m unittest discover -s tests -p test_context_graph_lifecycle.py -v + graph_containment_macos: + # The Seatbelt half of the same claim. The bubblewrap job above proves the + # Linux boundary and nothing whatever about this one, and macOS is the + # platform this tool is developed on: leaving the profile to be exercised + # only by whoever happens to run the suite on a laptop is how it stayed + # unexecuted. ``sandbox-exec`` ships with the OS, so there is nothing to + # install -- the job is the evidence that the profile runs at all. + # + # Not required to pass yet. It reports, so that a red result carries the + # probe's own diagnosis of which candidate failed and what the launcher + # said, which is what turned the equivalent bubblewrap failure into a + # one-round fix. + name: graph containment (macOS) + runs-on: macos-latest + continue-on-error: true + steps: + - name: Check out + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 + with: + python-version: "3.12" + + - name: Confirm the system sandbox is present + run: test -x /usr/bin/sandbox-exec + + - name: Install package + run: python -m pip install -e . + + - name: Real containment tests + env: + CODE_MOWER_REQUIRE_CONTAINMENT: "1" + run: python -m unittest discover -s tests -p test_context_graph_lifecycle.py -v + package: name: package runs-on: ubuntu-latest diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index c7c2e75a..277a2e71 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -119,6 +119,32 @@ unreadable — they are absent. `unshare --net` used to be a candidate and is gone: it denies the network and leaves the host filesystem in place, which is half a boundary. +"The provider's own install" is a layout this module *proves* rather than one +it infers from depth. It used to expose the executable's parent and +grandparent, on the reasoning that a console script lives in a virtual +environment's `bin`; where that reasoning is wrong it is wrong in the widest +possible direction, because `~/bin/graphify` makes the grandparent the +operator's entire home, `/opt/graphify` makes it `/`, and a provider installed +inside the checkout makes it the live working tree the materialized copy exists +to keep away from the provider. So an install root is now a directory holding a +`pyvenv.cfg` whose script directory holds the executable — a virtual +environment, which is what pinning a provider produces — and the root arrived +at is refused outright if it is the filesystem root, the operator's home, the +checkout being indexed, or an ancestor of either. A provider already inside the +read-only system runtime asks for no extra exposure and gets none. Anything +else is refused with an instruction to pin the provider into its own +environment, rather than exposed as a guess. + +The prefix a particular build ends up with is probed before that build runs, +not just the host's mechanism at startup: the readable set of a real build is +the provider's install rather than the interpreter paths the host probe uses, +and a widened exposure that reopened the boundary would otherwise meet nothing +between the exposure and the provider. The verification probe is handed this +build's exposure *plus* the interpreter, so that the probe child can start at +all; that makes the probed prefix strictly more permissive than the one the +build runs under, and containment observed there is a sound statement about +containment here. + No mechanism is trusted on its name, and none is looked up on `PATH`: each candidate is an absolute path whose file and every ancestor directory must be owned by root or by this user and unwritable by anyone else, because a launcher @@ -163,6 +189,15 @@ lift the restriction system-wide about the whole machine rather than about this build, and not one this repository makes on an operator's behalf. +macOS has its own job, `graph containment (macOS)`, because the Linux job +proves the bubblewrap boundary and nothing whatever about the Seatbelt one, and +leaving the profile to be exercised only by whoever happened to run the suite +on a laptop is how it went unexecuted. `sandbox-exec` ships with the OS, so +there is nothing to install and the job is simply the evidence that the profile +runs at all. It does not gate merges yet: a red result there carries the +probe's own account of which candidate failed and what the launcher said on +stderr, which is the diagnosis the equivalent Linux failure was fixed from. + Git itself runs with `GIT_CONFIG_NOSYSTEM`, `GIT_CONFIG_GLOBAL=/dev/null`, and `GIT_CONFIG_SYSTEM=/dev/null`: an untrusted checkout's local, global, or system configuration can otherwise install clean/smudge filters and hook paths that run @@ -392,10 +427,26 @@ can never read each other's generations. State is refused inside any Git repository, which is the enforcement half of adoption condition 2. The refusal is checked on the resolved path as well as the given one: `--state-dir /outside/link/state` names no repository in its own spelling while -`/outside/link` points inside one, and the `O_NOFOLLOW` opens cover only the -final component of each directory this module creates. Symlinked ancestors are -resolved rather than rejected — ordinary private roots have them, macOS reaches -`/tmp` through a link into its `private` directory. +`/outside/link` points inside one. Symlinked ancestors are resolved rather than +rejected — ordinary private roots have them, macOS reaches `/tmp` through a +link into its `private` directory. + +Resolving settles what the ancestors mean at construction and nothing about +what they become afterwards, so the root is opened by walking it from `/` one +component at a time, each against its parent's descriptor with `O_NOFOLLOW`. +Opening the whole absolute path in one call would not do: `O_NOFOLLOW` refuses +only the *final* component, and every ancestor above it is resolved exactly as +a link planted there would want. Nor does finding the deepest existing prefix +first — that prefix is still opened by its full spelling. A pre-created root +behind a newly inserted ancestor link therefore used to be accepted, because +the leaf really is a directory and really is not a link; it is simply not the +directory that was checked. + +Renames and removals still travel full paths — a staged generation is renamed +into place, a removed tree is recursed over — and a full path is re-resolved +from the root on every call. Before each of those, the inode the no-follow walk +arrived at is compared against the one the path spells now, and a mismatch is a +refusal rather than a write into whatever the link points at. ## What this does not do diff --git a/src/code_mower/context_graph_command.py b/src/code_mower/context_graph_command.py index 321e036b..34dd4567 100644 --- a/src/code_mower/context_graph_command.py +++ b/src/code_mower/context_graph_command.py @@ -97,7 +97,7 @@ def main(argv=None) -> int: manifest = lifecycle.build_graph( args.repo_path, pin=pin, - indexer=lifecycle.subprocess_indexer(args.indexer), + indexer=lifecycle.subprocess_indexer(args.indexer, repository=args.repo_path), root=args.state_dir, revision=args.revision, keep_previous=args.keep_previous, diff --git a/src/code_mower/context_graph_lifecycle.py b/src/code_mower/context_graph_lifecycle.py index 2acd86de..bf5f4128 100644 --- a/src/code_mower/context_graph_lifecycle.py +++ b/src/code_mower/context_graph_lifecycle.py @@ -344,8 +344,13 @@ def _seatbelt_prefix(launcher: str, *, writable: Sequence[str], readable: Sequen "(allow mach-lookup)", "(allow ipc-posix-shm)", "(allow file-read-metadata)", - "(allow file-write-data (literal \"/dev/null\") (literal \"/dev/zero\")" - " (literal \"/dev/random\") (literal \"/dev/urandom\"))", + # Read *and* write on the same four devices. Granting write without + # read is an asymmetry nothing wants: a runtime opens ``/dev/null`` + # read-write to detach a stream, and reads ``/dev/urandom`` to seed + # itself, so a profile that denies the read denies the process its + # start rather than denying it anything an operator cares about. + "(allow file-read-data file-write-data (literal \"/dev/null\")" + " (literal \"/dev/zero\") (literal \"/dev/random\") (literal \"/dev/urandom\"))", ] if readable: subpaths = " ".join(f"(subpath {_seatbelt_literal(path)})" for path in readable) @@ -564,14 +569,44 @@ def containment_prefix( writable: Sequence[Path | str], readable: Sequence[Path | str], ) -> tuple[str, ...]: - """The argv prefix confining a child to ``writable`` plus a read-only runtime.""" + """The argv prefix confining a child to ``writable`` plus a read-only runtime. + + Verified against *this* exposure before it is returned, not merely built + from it. The host probe establishes that a mechanism can confine a child + when it is handed the interpreter paths; it says nothing about the prefix a + particular build ends up with, whose readable set is the pinned provider's + install and whose writable set is that build's own directories. A widened + exposure that happened to reopen the boundary would otherwise be caught by + nothing between here and the provider. + + The probe's own exposure is this one plus the interpreter, because the + probe child is a Python that has to be able to start at all. That makes the + probed prefix strictly more permissive than the returned one, so a probe + that still observes containment is a sound statement about the prefix a + build actually runs under. + """ mechanism = containment_mechanism() if mechanism is None: raise ContextError( "local graph builds need an OS sandbox that denies the provider the network and " "the host filesystem; this host offers none that could be verified" ) - return _prefix_for(mechanism, writable=writable, readable=readable) + prefix = _prefix_for(mechanism, writable=writable, readable=readable) + probe = _prefix_for( + mechanism, + writable=writable, + readable=(*readable, *_interpreter_read_paths()), + ) + # Inside the exposure, because that is where the confined child starts: a + # probe launched in a directory the boundary deliberately does not expose + # fails to start and says nothing about the boundary. + inside = next((str(path) for path in writable if os.path.isdir(path)), None) + if not _prefix_confines(probe, cwd=inside): + raise ContextError( + "the sandbox this build would run the local graph provider under could not be " + "observed denying it the network and the host filesystem; no generation was published" + ) + return prefix def _object_name(value: Any) -> str: @@ -1211,28 +1246,96 @@ def _resolved_executable(executable: str) -> str: return executable -def _provider_read_paths(command: str) -> tuple[str, ...]: +#: The directory a console script sits in, by platform convention. Named so an +#: install root is recognised rather than guessed at from depth alone. +_VENV_SCRIPT_DIRECTORIES = ("bin", "Scripts") + + +def _under(path: Path, root: Path) -> bool: + return path == root or path.is_relative_to(root) + + +def _refuse_broad_exposure(root: Path, *, repository: Path) -> None: + """Refuse an exposure root that would hand over somebody's whole world. + + The filesystem root, the operator's home, the checkout being indexed, and + every ancestor of either: each of these is a directory whose contents are + exactly what this boundary exists to keep away from the provider. A root + that *is* one of them is not a narrow install however it was arrived at, + and a root *inside the checkout* is the live working tree -- ignored + secrets and all -- which the materialized copy exists precisely to avoid + showing anyone. + """ + resolved = Path(os.path.realpath(root)) + checkout = Path(os.path.realpath(repository)) + try: + home = Path(os.path.realpath(Path.home())) + except (OSError, RuntimeError): # pragma: no cover - a host with no home + home = None + refused = {Path(resolved.anchor), checkout, *checkout.parents} + if home is not None: + refused.update({home, *home.parents}) + if resolved in refused or _under(resolved, checkout): + raise ContextError( + "local graph provider must be pinned into its own virtual environment; " + "exposing its install would expose the filesystem root, your home directory, " + "or the checkout being indexed" + ) + + +def _provider_read_paths(command: str, *, repository: Path) -> tuple[str, ...]: """The install the pinned provider needs to be readable, and nothing beside it. - A console script is one file in a virtual environment whose libraries live - beside it, so the environment root -- the executable's grandparent -- is - what has to be exposed, plus the base interpreter it was created from. An - executable this process cannot even locate is refused here rather than - exposed as a guess: the alternative is a boundary drawn around a path that - is not where the provider is. + The executable's parent and grandparent used to be exposed on the reasoning + that a console script lives in a virtual environment's ``bin``. That is a + guess about a layout, not knowledge of one, and the guess is wrong in the + directions that matter most: ``~/bin/graphify`` makes the grandparent the + operator's home, ``/opt/graphify`` makes it ``/``, and a provider inside the + checkout makes it the live working tree the materialized copy exists to + avoid showing anybody. The boundary was then drawn around whatever that + came out to, and the probe -- which exercises only the interpreter paths -- + never touched it. + + So the layout is *proved* instead. An install root is a directory holding a + ``pyvenv.cfg`` whose script directory holds this executable: a virtual + environment, which is what pinning a provider produces, and whose root is + narrow by construction. A provider that already lives inside the read-only + system runtime needs no extra exposure at all and gets none. Anything else + is refused with an instruction rather than exposed as a guess, and whatever + root is arrived at is put through ``_refuse_broad_exposure`` regardless -- + a proof of layout is not a proof that the layout is narrow. """ located = command if os.path.isabs(command) else shutil.which(command) if not located: raise ContextError("local graph provider executable could not be located for containment") real = Path(os.path.realpath(located)) - # ``located`` as well as its resolved self: the child is launched by the - # name this process resolved the command to, and a console script in a - # virtual environment reaches its libraries through that environment's own - # spelling rather than through whatever the link points at. - return tuple( - str(path) - for path in (located, real, real.parent, real.parent.parent, Path(sys.base_prefix)) - ) + # Both spellings of the executable itself: the child is launched by the name + # this process resolved the command to, and a console script reaches its + # environment through that environment's own spelling rather than through + # whatever the link points at. + spellings = tuple(dict.fromkeys((str(located), str(real)))) + system = tuple(Path(os.path.realpath(path)) for path in _SYSTEM_READ_PATHS) + if any(_under(real, path) for path in system): + # Already inside the read-only runtime every child gets. Adding the + # enclosing prefix would widen that exposure, not narrow it. + return spellings + root = real.parent.parent + if real.parent.name not in _VENV_SCRIPT_DIRECTORIES or not (root / "pyvenv.cfg").is_file(): + raise ContextError( + "local graph provider must be pinned into its own virtual environment so its " + "install can be exposed to the sandbox without exposing anything around it" + ) + _refuse_broad_exposure(root, repository=repository) + # The base interpreter a virtual environment was created from lives outside + # it and is what its ``bin/python`` points at, so it is exposed too -- and + # held to the same refusals, because ``sys.base_prefix`` is only narrow on + # hosts where the interpreter was not installed into one of them. + base = Path(os.path.realpath(sys.base_prefix)) if sys.base_prefix else None + extra: tuple[str, ...] = () + if base is not None and not any(_under(base, path) for path in system): + _refuse_broad_exposure(base, repository=repository) + extra = (str(base),) + return (*spellings, str(root), *extra) #: The subcommand the evaluated release exposes, recorded in @@ -1651,7 +1754,7 @@ def _run_contained( _terminate_process_group(child, group) -def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult]: +def subprocess_indexer(executable: str, *, repository: Path) -> Callable[[IndexRequest], IndexResult]: """Run a pinned provider CLI over the materialized copy, without a network. Kept as a factory so the lifecycle never imports or requires a graph @@ -1673,7 +1776,7 @@ def subprocess_indexer(executable: str) -> Callable[[IndexRequest], IndexResult] "local graph builds need an OS sandbox that denies the provider the network and " "the host filesystem; this host offers none that could be verified" ) - runtime = _provider_read_paths(command) + runtime = _provider_read_paths(command, repository=repository) def run(request: IndexRequest) -> IndexResult: _refuse_pre_existing_provider_state(request.source_root) @@ -1848,23 +1951,36 @@ def _open_private_at(parent: int | None, name: str, *, missing: str, private: bo a symlink between the check and the creation -- the state-root defect this replaces -- and no later check on the leaf can see that it happened. """ - if missing == _MISSING_CREATE: - try: - os.mkdir(name, mode=0o700, dir_fd=parent) - except FileExistsError: - pass - except OSError: - raise ContextError("local graph state directory is unavailable or unsafe") from None + unsafe = "local graph state directory is unavailable or unsafe" try: handle = os.open(name, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=parent) except FileNotFoundError: if missing == _MISSING_STOP: return None - raise ContextError("local graph state directory is unavailable or unsafe") from None + if missing != _MISSING_CREATE: + raise ContextError(unsafe) from None + # Opened before it is created, rather than creating unconditionally and + # reading the errno: this walk now traverses the whole absolute base, + # including ancestors like ``/usr`` that exist and that nobody may + # write. Whether such a ``mkdir`` reports ``EEXIST`` or ``EACCES`` + # first is the kernel's business, and a boundary should not rest on it. + try: + os.mkdir(name, mode=0o700, dir_fd=parent) + except FileExistsError: + pass + except OSError: + raise ContextError(unsafe) from None + try: + handle = os.open(name, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW, dir_fd=parent) + except OSError: + # Created as a directory and already something else, or something + # else won the race: either way this is not a directory this walk + # may descend. + raise ContextError(unsafe) from None except OSError: # ``ELOOP`` lands here: the component is a symlink, and a symlink is # not a directory this class owns however private its target may be. - raise ContextError("local graph state directory is unavailable or unsafe") from None + raise ContextError(unsafe) from None if not private: # An ancestor *above* the operator's private root: this class does not # own its mode and must not judge it. What matters there is only that @@ -1909,6 +2025,11 @@ def __init__(self, repository: Path, *, root: Path | None = None): self.base = Path(os.path.realpath(base)) self.workspace = workspace_id(self.repository) self.path = self.base / "graph" / self.workspace + #: The device and inode the base was first observed as, established by + #: the no-follow walk and re-checked by every later one. Canonicalizing + #: at construction settles what the ancestors mean *now*; this is what + #: notices that they stopped meaning it. + self._identity: tuple[int, int] | None = None @property def generations_path(self) -> Path: @@ -1935,37 +2056,84 @@ def lock_path(self) -> Path: def _components(self) -> tuple[str, ...]: return ("graph", self.workspace, "generations") - def _create_base(self) -> None: - """Create the private root itself, one no-follow component at a time. - - The root an operator names may not exist yet, ancestors and all, and - creating it is what a first build does. ``mkdir(parents=True)`` cannot - be what does it: a path component that does not exist *cannot* be - canonicalized at construction, so a component created as a symlink - afterwards -- pointing into a repository, say -- would be followed by - the recursive creation, and the state would land somewhere no check - ever looked at. So the walk starts at the deepest component that really - exists and creates each missing one against its parent's descriptor: - the components above the root are not judged for privacy, which is not - this class's business, but none of them is traversed through a link. + def _open_base(self, *, missing: str) -> int | None: + """Open the private root by walking it from ``/``, following nothing. + + A single ``open(base, O_NOFOLLOW)`` is not this, and the difference is + the whole of the defect it replaces. ``O_NOFOLLOW`` refuses only the + *final* component; every ancestor above it is resolved by the kernel + exactly as a symlink planted there would want. So a base whose ancestor + was replaced by a link after construction passes the leaf check -- + because the leaf really is a directory and really is not a link. It is + simply not the directory that was checked. Finding the deepest existing + prefix first does not help: that prefix is still opened by its full + absolute spelling, in one call, through whatever its ancestors have + become. + + Every component is opened against its parent's descriptor instead, so + the kernel resolves exactly one name at a time and ``O_NOFOLLOW`` + covers all of it. The base was canonicalized in ``__init__``, so on an + untampered host no component of it is a link and this walk is a + restatement of the path; a component that has become one since is + precisely what must fail, whether or not the components below it exist + already. + + Components above the operator's root are traversed but not judged for + ownership or mode -- that is not this class's to own. What matters + there is only that each was a real directory rather than a link. """ - existing = self.base - pending: list[str] = [] - while not existing.exists() and existing != existing.parent: - pending.append(existing.name) - existing = existing.parent - handle = _open_private_at(None, str(existing), missing=_MISSING_REFUSE, private=False) + parts = self.base.parts + # The root is not a component anybody can replace, and mkdir on it is + # meaningless; it is opened, never created. + handle = _open_private_at(None, parts[0], missing=_MISSING_REFUSE, private=False) if handle is None: # pragma: no cover - _MISSING_REFUSE raises instead - return + return None try: - for name in reversed(pending): - deeper = _open_private_at( - handle, name, missing=_MISSING_CREATE, private=False - ) - if deeper is None: # pragma: no cover - creation returns a handle - return + for name in parts[1:]: + deeper = _open_private_at(handle, name, missing=missing, private=False) + if deeper is None: + os.close(handle) + return None os.close(handle) handle = deeper + except BaseException: + os.close(handle) + raise + info = os.fstat(handle) + identity = (info.st_dev, info.st_ino) + if self._identity is None: + self._identity = identity + elif self._identity != identity: + # The walk was clean and still arrived somewhere else than it did + # last time: an ancestor was swapped between two operations on the + # same root. Refuse rather than carry on against a directory this + # instance never checked. + os.close(handle) + raise ContextError("local graph state directory is unavailable or unsafe") + return handle + + def _revalidate_base(self) -> None: + """Prove the full spelling still names the directory the walk verified. + + Not everything below is descriptor-relative: a generation is renamed + into place and the tree is removed by full path, and a full path is + re-resolved from the root on every call. A no-follow walk that passed a + moment ago says nothing about the ancestor a rename will travel. So the + walk's own inode is compared against the one the path resolves to now, + and a redirection becomes a refusal instead of a write into whatever + the link points at. + """ + handle = self._open_base(missing=_MISSING_REFUSE) + if handle is None: # pragma: no cover - _MISSING_REFUSE raises instead + raise ContextError("local graph state directory is unavailable or unsafe") + try: + walked = os.fstat(handle) + try: + spelled = os.stat(self.base) + except OSError: + raise ContextError("local graph state directory is unavailable or unsafe") from None + if (walked.st_dev, walked.st_ino) != (spelled.st_dev, spelled.st_ino): + raise ContextError("local graph state directory is unavailable or unsafe") finally: os.close(handle) @@ -1978,13 +2146,7 @@ def _walk(self, *, depth: int, missing: str) -> int: built needs. Nothing below a component that failed its privacy check is ever opened, because there is no descriptor left to open it against. """ - if missing == _MISSING_CREATE: - self._create_base() - opened = _open_private_at( - None, - str(self.base), - missing=_MISSING_STOP if missing == _MISSING_STOP else _MISSING_REFUSE, - ) + opened = self._open_base(missing=missing) if opened is None: return -1 handle = opened @@ -2174,6 +2336,9 @@ def publish(self, manifest: BuildManifest, artifact: bytes) -> BuildManifest: if load_manifest(json.loads(serialized)).generation != manifest.generation: raise ContextError("local graph manifest does not match its generation") self.ensure() + # The renames below travel full paths, so the walk's verdict is + # re-established against the spelling they will actually use. + self._revalidate_base() staging = self.generations_path / ("." + uuid.uuid4().hex + ".staging") staging.mkdir(mode=0o700) try: @@ -2200,6 +2365,7 @@ def publish(self, manifest: BuildManifest, artifact: bytes) -> BuildManifest: def prune(self, *, keep: str | None) -> list[str]: """Remove every generation but ``keep``, including crashed stagings.""" removed = [] + self._revalidate_base() try: names = os.listdir(self.generations_path) except FileNotFoundError: @@ -2236,8 +2402,11 @@ def remove_all(self) -> bool: if not self.path.exists(): return False # Refuse to delete a tree that is not ours; a loosened or foreign - # directory is reported, not recursively removed. + # directory is reported, not recursively removed. The recursion + # below travels a full path, so the base is re-established against + # that spelling before anything is unlinked. self.verify_private() + self._revalidate_base() shutil.rmtree(self.path) return True diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index 1ce246a4..4886c957 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -641,7 +641,7 @@ def stand_in_containment(prefix: tuple[str, ...]): lifecycle, "containment_mechanism", lambda: lifecycle.Containment("stand-in", "/sandbox") ): with mock.patch.object(lifecycle, "containment_prefix", lambda **keywords: prefix): - with mock.patch.object(lifecycle, "_provider_read_paths", lambda command: ()): + with mock.patch.object(lifecycle, "_provider_read_paths", lambda command, **_: ()): yield @@ -1006,7 +1006,7 @@ def fake_popen(argv, **kwargs): # only around the launch, so the Git calls a build makes are never # intercepted by this stand-in. with stand_in_containment(sandbox): - indexer = lifecycle.subprocess_indexer(executable) + indexer = lifecycle.subprocess_indexer(executable, repository=self.repository) with mock.patch.object(subprocess, "Popen", fake_popen): result = indexer(request) return recorded[0], result @@ -1044,7 +1044,7 @@ def fake_popen(argv, **kwargs): return FakeChild() with stand_in_containment(("/sandbox",)): - indexer = lifecycle.subprocess_indexer("graphify") + indexer = lifecycle.subprocess_indexer("graphify", repository=self.repository) with mock.patch.object(lifecycle, "containment_prefix", record_prefix): with mock.patch.object(subprocess, "Popen", fake_popen): indexer(request) @@ -1186,7 +1186,7 @@ def fake_popen(argv, **kwargs): return FakeChild() with stand_in_containment(("/sandbox",)): - indexer = lifecycle.subprocess_indexer("graphify") + indexer = lifecycle.subprocess_indexer("graphify", repository=self.repository) with mock.patch.object(subprocess, "Popen", fake_popen): result = indexer(request) self.assertEqual(result.completeness, lifecycle.PARTIAL) @@ -1245,7 +1245,7 @@ def refuse_whole_file_read(self: Path) -> bytes: raise AssertionError(f"{self} was read whole") with stand_in_containment(("/sandbox",)): - indexer = lifecycle.subprocess_indexer("graphify") + indexer = lifecycle.subprocess_indexer("graphify", repository=self.repository) with mock.patch.object(subprocess, "Popen", fake_popen): with mock.patch.object(Path, "read_bytes", refuse_whole_file_read): result = indexer(request) @@ -1269,7 +1269,7 @@ def fake_popen(argv, **kwargs): return FakeChild() with stand_in_containment(("/sandbox",)): - indexer = lifecycle.subprocess_indexer("graphify") + indexer = lifecycle.subprocess_indexer("graphify", repository=self.repository) with mock.patch.object(subprocess, "Popen", fake_popen): with self.assertRaises(ContextError): indexer(request) @@ -1293,7 +1293,7 @@ def record_termination(child, group) -> None: order.append("stopped") with stand_in_containment(("/sandbox",)): - indexer = lifecycle.subprocess_indexer("graphify") + indexer = lifecycle.subprocess_indexer("graphify", repository=self.repository) with mock.patch.object(subprocess, "Popen", fake_popen): with mock.patch.object(lifecycle, "_terminate_process_group", record_termination): with self.assertRaises(ContextError) as raised: @@ -1306,7 +1306,7 @@ def record_termination(child, group) -> None: def test_a_host_without_a_sandbox_refuses_to_launch_a_provider(self) -> None: with no_containment(): with self.assertRaises(ContextError): - lifecycle.subprocess_indexer("graphify") + lifecycle.subprocess_indexer("graphify", repository=self.repository) def test_a_relative_provider_path_binds_to_the_invocation_directory(self) -> None: # The child runs in the materialized copy, so a relative path left @@ -1330,6 +1330,92 @@ def test_an_unnamed_provider_is_refused(self) -> None: lifecycle._resolved_executable("") +class ProviderExposureTests(TemporaryWorkspace): + """What the sandbox is told to make readable for the provider itself. + + The exposure used to be the executable's parent and grandparent, which is a + guess about a layout rather than knowledge of one. These hold the rule to + the layouts where that guess was widest: a script in a home directory, a + script directly under a top-level prefix, and a provider living inside the + checkout being indexed. + """ + + def script(self, path: Path, *, venv: bool = False) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + if venv: + (path.parent.parent / "pyvenv.cfg").write_text("home = /usr/bin\n", encoding="utf-8") + path.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8") + path.chmod(0o700) + return path + + def exposure(self, path: Path) -> tuple[str, ...]: + return lifecycle._provider_read_paths(str(path), repository=self.repository) + + def test_a_pinned_virtual_environment_exposes_that_environment_and_no_more(self) -> None: + environment = self.root / "venv" + provider = self.script(environment / "bin" / "graphify", venv=True) + exposed = self.exposure(provider) + self.assertIn(str(environment), exposed) + self.assertIn(str(provider), exposed) + # Neither the directory the environment sits in nor anything above it. + self.assertNotIn(str(self.root), exposed) + self.assertNotIn(str(environment.parent), exposed) + + def test_a_script_in_a_home_directory_does_not_expose_the_home_directory(self) -> None: + """``~/bin/graphify``: the grandparent is the operator's whole home.""" + home = self.root / "home" + provider = self.script(home / "bin" / "graphify") + with mock.patch.object(Path, "home", staticmethod(lambda: home)): + with self.assertRaises(ContextError): + self.exposure(provider) + + def test_a_script_directly_under_a_top_level_prefix_does_not_expose_the_root(self) -> None: + """``/opt/graphify``: the grandparent the old heuristic exposed is ``/``. + + Refused on the layout, before the question of how wide the grandparent + happens to be arises: a directory that is not a script directory beside + a ``pyvenv.cfg`` is not an install this module can draw a boundary + around, whatever it contains. + """ + provider = self.script(self.root / "opt" / "graphify") + with self.assertRaises(ContextError): + self.exposure(provider) + + def test_a_provider_inside_the_checkout_does_not_expose_the_working_tree(self) -> None: + """A proven layout is still refused when the layout is the live checkout. + + This one carries a real ``pyvenv.cfg``, so the layout proof passes and + the refusal has to come from where the environment *is*: inside the + repository, whose ignored files are the thing the materialized copy + exists to keep away from the provider. + """ + provider = self.script(self.repository / ".venv" / "bin" / "graphify", venv=True) + with self.assertRaises(ContextError): + self.exposure(provider) + + def test_an_environment_that_is_the_home_directory_is_refused(self) -> None: + home = self.root / "home" + provider = self.script(home / "bin" / "graphify", venv=True) + with mock.patch.object(Path, "home", staticmethod(lambda: home)): + with self.assertRaises(ContextError): + self.exposure(provider) + + def test_a_provider_in_the_system_runtime_asks_for_no_extra_exposure(self) -> None: + """Already inside the read-only runtime every child gets. + + Exposing the enclosing prefix would widen that exposure rather than + narrow it, so nothing but the executable's own spellings is added. + """ + exposed = lifecycle._provider_read_paths("/bin/sh", repository=self.repository) + self.assertEqual(set(exposed), {"/bin/sh", os.path.realpath("/bin/sh")}) + + def test_a_provider_that_cannot_be_located_is_refused_rather_than_guessed_at(self) -> None: + with self.assertRaises(ContextError): + lifecycle._provider_read_paths( + "code-mower-no-such-provider", repository=self.repository + ) + + @unittest.skipUnless(hasattr(os, "killpg"), "process groups are a POSIX facility") class ExtractionOverrunTests(unittest.TestCase): """What a timed-out provider leaves running, proved against real processes. @@ -1709,6 +1795,62 @@ def test_a_component_that_becomes_a_symlink_before_creation_is_refused(self) -> state.ensure() self.assertFalse((inside / "state").exists()) + def test_a_precreated_root_behind_a_new_ancestor_symlink_is_refused(self) -> None: + """The bypass a deepest-existing-prefix walk leaves open. + + Finding the deepest component that exists and opening *that* with + ``O_NOFOLLOW`` refuses an ancestor link only while the final directory + is still absent -- because then the walk has to create it, one + component at a time. Pre-create the final directory behind the link and + the walk has nothing left to create: it opens the whole absolute base + in one call, ``O_NOFOLLOW`` clears the leaf it was pointed at, and every + ancestor above the leaf is resolved exactly as the planted link + intended. The previous test leaves that directory absent and so never + reaches this. + + Here the root is absent at construction, ``/subdir/state`` is + pre-created, and the missing ancestor becomes a link to + ``/subdir``. The walk has to refuse on the ancestor itself. + """ + outside = self.root / "outside" + outside.mkdir() + absent = outside / "deep" / "state" + state = lifecycle.GraphStateRoot(self.repository, root=absent) + inside = self.repository / "subdir" + (inside / "state").mkdir(parents=True) + os.symlink(inside, outside / "deep") + # The leaf really is a directory and really is not a link, so nothing + # about the final component can catch this. + self.assertTrue(absent.is_dir()) + with self.assertRaises(ContextError): + state.ensure() + self.assertFalse((inside / "state" / "graph").exists()) + self.assertEqual(list((inside / "state").iterdir()), []) + + def test_an_ancestor_swapped_after_the_walk_refuses_the_next_operation(self) -> None: + """Renames and removals travel full paths, so the walk is re-established. + + ``publish`` renames a staged generation into place and ``remove`` + recurses over the tree, both by full path, and a full path is resolved + from the root on every call. A no-follow walk that passed a moment + earlier says nothing about what an ancestor has become since. + """ + elsewhere = self.root / "elsewhere" + elsewhere.mkdir() + home = self.root / "home" + home.mkdir() + state = lifecycle.GraphStateRoot(self.repository, root=home / "st") + state.ensure() + decoy = self.root / "decoy" + os.rename(home, decoy) + os.symlink(elsewhere, home) + (elsewhere / "st").mkdir(mode=0o700) + with self.assertRaises(ContextError): + state._revalidate_base() + with self.assertRaises(ContextError): + state.prune(keep=None) + self.assertFalse((elsewhere / "st" / "graph").exists()) + def test_a_manifest_no_reader_could_load_publishes_nothing(self) -> None: """Publication validates the whole manifest before it touches state. @@ -2099,8 +2241,18 @@ def indexer_script(self, *, complete: bool = True) -> Path: It answers to ``extract`` and writes its state into the directory it was run in, which is the contract ``docs/graphify-evaluation.md`` records for the evaluated release. + + Laid out as a virtual environment -- ``/bin/