From 7974f7d3e41cf5b4a91306f1177b52c9d19bd864 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Tue, 27 Oct 2026 16:40:55 +0800 Subject: [PATCH 1/2] rebase: reap attempt-scoped test chains and give module tests a real budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v0.33 campaign (run-20261024-235522-d35da7) blocked at the test loop with GpuLockTimeout. Module agents had started pytest chains via setsid; those outlived their sessions, kept the run's GPU lock for ~7 hours after the engine finished wave 1, and the lock's liveness check (kill -0) also treated an unreaped zombie owner as alive. - rebase_engine/processes.py: cleanup_run_processes() kills same-user processes carrying BOTH run markers (IMX_RUN_ID + IMX_LOG_DIR), keyed by process birth time so a reused PID is never a bystander kill. Called when a module/debug attempt ends (including normal return) and around the checkout-lock finalizer. - testing/process_tree._alive(): a Z/X state in /proc//stat is dead; gpu_lock reuses it so a zombie can no longer hold the mutex. - Module test budget: adapters declare rebase.testing.module_timeout_sec (vllm-omni: 21600 — the offline inference suite needs ~4h, so the old 7200s wrapper could never reach a terminal result). The value flows to TEST_TIMEOUT_SEC in the agent shell env, the prompt template ({MODULE_TEST_TIMEOUT_SEC}) and the harness budget note. - Settings.rebase_harness_timeout_sec replaces the hard-coded 7200s session budget passed to ModuleRunConfig. - providers/zcode: write mcp.servers.*.timeoutMs from the session budget; ZCode's 30s default abandoned bridge tool calls that were still running. - Prompt template: drop the obsolete 30s-polling / detached-chain guidance and tell agents the attempt's background processes are reaped on return. Tests: test_rebase_process_lifecycle.py (real zombie owner, detached orphan reaped while other runs' processes survive), zcode timeoutMs test, harness stub gains the new field. Co-authored-by: Cursor --- adapters/vllm_omni/manifest.yaml | 4 ++ .../templates/module_rebase.live.prompt.tmpl | 17 +++-- src/infermatrix_copilot/config.py | 3 +- .../engine/steps/rebase_v3.py | 20 +++++- src/infermatrix_copilot/providers/zcode.py | 10 ++- .../rebase_engine/module_rebase.py | 14 ++++- .../rebase_engine/processes.py | 40 ++++++++++++ .../rebase_engine/prompt_builder.py | 2 + src/infermatrix_copilot/testing/gpu_lock.py | 9 +-- .../testing/process_tree.py | 6 ++ test/test_provider_zcode.py | 12 ++++ test/test_rebase_harness_backend.py | 4 +- test/test_rebase_process_lifecycle.py | 62 +++++++++++++++++++ 13 files changed, 182 insertions(+), 21 deletions(-) create mode 100644 src/infermatrix_copilot/rebase_engine/processes.py create mode 100644 test/test_rebase_process_lifecycle.py diff --git a/adapters/vllm_omni/manifest.yaml b/adapters/vllm_omni/manifest.yaml index a74177d73..94868f04f 100644 --- a/adapters/vllm_omni/manifest.yaml +++ b/adapters/vllm_omni/manifest.yaml @@ -328,6 +328,10 @@ rebase: - [/distributed/, scheduler] - [/benchmarks/, benchmarks] testing: + # TEST_TIMEOUT_SEC for the module agents' required pytest commands. The + # full offline inference suite takes about four hours on the shared host, + # so the previous 7200s default could never produce a terminal result. + module_timeout_sec: 21600 # root-level artifacts tests leak between runs (test_artifact_cleanup.sh # TEST_ARTIFACT_GLOBS, verbatim) — gitignored upstream but pile up on disk artifact_globs: ["audio_chatcmpl-*.wav", "audio_[0-9]*ch_*.wav", diff --git a/adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl b/adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl index b9ae783b4..9263ced77 100644 --- a/adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl +++ b/adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl @@ -204,11 +204,18 @@ On its own line (parser-friendly): ## Verification (must all pass) **Module pytest wrapper:** Every verification line uses `imx-omni-pytest …`. **Do not replace with bare `pytest`.** It applies `test_watchdog.sh`, `TEST_TIMEOUT_SEC`, logs under `\${LOG_DIR}/tests/module_*.log`, and shared GPU lock for **every** pytest line from Phase-2 sessions. -**Long-running shell waits — avoid CLI auth-token expiry:** A single `shellToolCall` blocking idle for >~120 seconds crashes mid-run with `Invalid API key — Please run /login`. The wrapper retries but each retry restarts your session. -- **Do not** issue `sleep 480 && tail …` to wait for long tests. Use `imx-omni-pytest …` (has its own watchdog + log streaming). -- For background tasks, prefer **short polled checks** (`for i in 1 2 3 …; do sleep 30; ; done`). -- Cap per-call sleep at 60 s when waiting on a known-running pid; loop with status checks. -- **Thinking stalls are equally dangerous.** Produce incremental output (file writes, short commands, debug memory entries) to maintain heartbeat visibility. Silent agents are timed out after 7200s with `No code changes from debug agents.` +**Long-running verification:** The MCP client timeout follows the harness +session budget; it is not a fixed 30-second limit. Use the module pytest wrapper +with `TEST_TIMEOUT_SEC={MODULE_TEST_TIMEOUT_SEC}` and allow terminal completion. +For short observation polls, guard missing paths before invoking commands that +would otherwise read stdin. Do not duplicate a live test or confuse an +observation timeout with a test result. +- All commands, assertions, selected scope and granted GPU geometry remain required. +- The engine cleans up this attempt's background processes when the attempt ends, + including after a normal return. Never return while verification is still live + or leave a detached chain for another module to inherit. +- If background verification is needed, redirect all three standard streams, + record the wrapper PID, poll it, and collect its terminal outcome before returning. Run: {IMPORT_CHECK} diff --git a/src/infermatrix_copilot/config.py b/src/infermatrix_copilot/config.py index 9fe245cde..7680cbbd9 100644 --- a/src/infermatrix_copilot/config.py +++ b/src/infermatrix_copilot/config.py @@ -7,7 +7,7 @@ from typing import Annotated, Literal from urllib.parse import urlparse -from pydantic import field_validator, model_validator +from pydantic import Field, field_validator, model_validator from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict _PACKAGE_ROOT = Path(__file__).resolve().parent @@ -256,6 +256,7 @@ class Settings(BaseSettings): # is meaningless to a harness CLI, so it is never forwarded there; empty # falls back to `strict_backend_model`, then the harness's own default. rebase_backend_model: str = "" + rebase_harness_timeout_sec: int = Field(default=7200, gt=0) # v3 remote CI (rebase_engine/ci_loop.run_ci_rounds) — neutral knobs; # pipeline identities live in the adapter. Defaults are the parent's. rebase_ci_retries: int = 2 # push/rebuild rounds after the first diff --git a/src/infermatrix_copilot/engine/steps/rebase_v3.py b/src/infermatrix_copilot/engine/steps/rebase_v3.py index 5502c0907..6855000ad 100644 --- a/src/infermatrix_copilot/engine/steps/rebase_v3.py +++ b/src/infermatrix_copilot/engine/steps/rebase_v3.py @@ -185,6 +185,9 @@ def _agent_shell_env(ctx: StepContext, manifest: dict, repo_root: str, # with the run identity and merges the runtime noise overlay _state = getattr(ctx, "state", None) or {} env["IMX_RUN_ID"] = str(_state.get("run_id", "") or ctx.run_dir.name) + env["TEST_TIMEOUT_SEC"] = str( + ((manifest.get("rebase") or {}).get("testing") or {}) + .get("module_timeout_sec", 7200)) _settings = getattr(ctx, "settings", None) if _settings is not None: from ...memory.paths import KnowledgePaths as _KP2 @@ -689,11 +692,17 @@ def _ensure_checkout_locks(ctx: StepContext, manifest: dict, held.append(lock) _HELD_LOCKS[key] = held from ..lifecycle import register_finalizer + from ...rebase_engine.processes import cleanup_run_processes + + cleanup_run_processes(ctx.run_dir, ctx.run_dir.name) async def _release_locks(_outcome, _key=key) -> None: - for h in _HELD_LOCKS.pop(_key, []): - h.release() - _drop_serial_lock(_key) + try: + cleanup_run_processes(ctx.run_dir, ctx.run_dir.name) + finally: + for h in _HELD_LOCKS.pop(_key, []): + h.release() + _drop_serial_lock(_key) register_finalizer(ctx.run_dir, _release_locks) ctx.trace.record("checkout_locks_acquired", @@ -1165,6 +1174,9 @@ async def _run_debug_agent(ctx: StepContext, manifest: dict, module: str, except Exception as exc: # noqa: BLE001 - a debug crash is STRUCTURAL ctx.trace.record("debug_attempt_error", slug=slug, error=str(exc)) return f"error:debug agent crashed: {exc}" + finally: + from ...rebase_engine.processes import cleanup_run_processes + cleanup_run_processes(ctx.run_dir, ctx.run_dir.name) if result.get("done"): return "done" # done=False from the loop is ALWAYS machinery/budget (fatal auth, @@ -1804,6 +1816,8 @@ async def _v3_module_rebase(ctx: StepContext) -> StepResult: # harness delegation: "api" (default) keeps the in-process loop backend=getattr(ctx.settings, "rebase_backend", "api") or "api", backend_model=getattr(ctx.settings, "rebase_backend_model", "") or "", + harness_timeout_s=ctx.settings.rebase_harness_timeout_sec, + test_timeout_sec=int(paths.env["TEST_TIMEOUT_SEC"]), settings=ctx.settings, manifest_path=str(adapter_dir / "manifest.yaml"), paths_spec={k: val for k, val in _dc_asdict(paths).items() diff --git a/src/infermatrix_copilot/providers/zcode.py b/src/infermatrix_copilot/providers/zcode.py index b423d38d3..33093247b 100644 --- a/src/infermatrix_copilot/providers/zcode.py +++ b/src/infermatrix_copilot/providers/zcode.py @@ -200,13 +200,18 @@ def auth_gap(self) -> str | None: # -- process plumbing ---------------------------------------------------- @staticmethod - def _write_mcp_config(session: Path, spec_path: Path) -> None: + def _write_mcp_config(session: Path, spec_path: Path, *, + timeout_s: float = 30.0) -> None: package_root = Path(__file__).resolve().parents[2] config = session / ".zcode" / "config.json" config.parent.mkdir() config.write_text(json.dumps({"features": {"memory": False}, "memory": {"use": False}, "mcp": {"servers": {_BRIDGE_SERVER: { "type": "stdio", + # ZCode otherwise abandons tools after 30s, even while a valid + # test or nested plan review is still running in the bridge. + # The session deadline remains the outer bound for execution. + "timeoutMs": max(1, int(timeout_s * 1000)), "command": sys.executable, "args": ["-m", "infermatrix_copilot.tool_bridge", "--spec", str(spec_path)], @@ -608,7 +613,8 @@ def run_session(self, req: AgentSessionRequest) -> AgentOutcome: session = Path(tempfile.mkdtemp(prefix="imc-zcode-session-")) try: if req.bridge_spec_path is not None: - self._write_mcp_config(session, req.bridge_spec_path) + self._write_mcp_config(session, req.bridge_spec_path, + timeout_s=req.timeout_s) header = "" if req.scope.root: header = (f"The repository checkout is at {req.scope.root}. " diff --git a/src/infermatrix_copilot/rebase_engine/module_rebase.py b/src/infermatrix_copilot/rebase_engine/module_rebase.py index f1ff2da87..b85748c4c 100644 --- a/src/infermatrix_copilot/rebase_engine/module_rebase.py +++ b/src/infermatrix_copilot/rebase_engine/module_rebase.py @@ -59,6 +59,7 @@ class ModuleRunConfig: # the harness HAS one (claude --max-turns); cursor/codex have none, so the # real bound there is this timeout plus the prompt's budget discipline. harness_timeout_s: float = 7200.0 + test_timeout_sec: int = 7200 max_debug_retries: int = 3 plan_review_max_rounds: int = 2 model_aliases: Mapping[str, str] | None = None @@ -180,6 +181,13 @@ async def _harness_attempt(prompt: str, *, module: str, config, from ..tool_bridge import write_bridge_spec run_dir = Path(config.log_dir) + prompt += (f"\n\nCurrent operator-selected budgets: this harness session has " + f"{config.harness_timeout_s:g} seconds; required module test " + f"commands have TEST_TIMEOUT_SEC={config.test_timeout_sec}. " + "These replace older attempt budgets. All test commands, " + "assertions and model geometry remain required. Background " + "tests belong to this attempt and will be stopped when it ends; " + "do not leave a detached chain running after returning.") transport = transport_for_id(config.settings, config.backend) spec_path = write_bridge_spec( run_dir=run_dir, step_name=f"rebase.module.{module}", scope=scope, @@ -266,7 +274,8 @@ async def rebase_module( plan_review_max_rounds=config.plan_review_max_rounds, broken_imports=broken_imports, module_test_plan=module_test_plan, adaptive_guidance=guidance, live=True, - baseline_ref=config.baseline_ref) + baseline_ref=config.baseline_ref, + test_timeout_sec=config.test_timeout_sec) substate.update({"modules": {module: {"status": "running"}}}) agent_log = str(Path(config.log_dir) / "agents" / f"module-{module}.log") @@ -291,6 +300,9 @@ async def _attempt(p: str, *, require_plan_review: bool = True) -> dict: except Exception as exc: # noqa: BLE001 - failure is substate data return {"done": False, "text": f"agent loop error: {exc}", "turns": 0, "plan_done": False} + finally: + from .processes import cleanup_run_processes + cleanup_run_processes(Path(config.log_dir), substate.run_id) # parent parity: an incomplete first run gets up to max_debug_retries # follow-up attempts with the debug prompt built from the failure text. diff --git a/src/infermatrix_copilot/rebase_engine/processes.py b/src/infermatrix_copilot/rebase_engine/processes.py new file mode 100644 index 000000000..c14e75b0d --- /dev/null +++ b/src/infermatrix_copilot/rebase_engine/processes.py @@ -0,0 +1,40 @@ +"""Reap run-owned background tests, including reparented setsid chains.""" +from __future__ import annotations + +import os +from pathlib import Path + +from ..testing.process_tree import _is_ancestor_of_self, _start_time, kill_tree + + +def cleanup_run_processes(run_dir: Path, run_id: str) -> list[int]: + """Only same-user processes carrying BOTH exact run markers are owned. + + Descendant snapshots cannot find a shell that daemonized between samples. + The inherited markers survive that reparenting. Record process birth times + before killing so a reused PID cannot turn cleanup into a bystander kill. + """ + if not run_id: + return [] + root = Path(run_dir).resolve() + identity: dict[int, int] = {} + for entry in Path('/proc').iterdir(): + if not entry.name.isdigit(): + continue + pid = int(entry.name) + try: + if entry.stat().st_uid != os.getuid(): + continue + born = _start_time(pid) + env = dict(item.split(b'=', 1) for item in + (entry / 'environ').read_bytes().split(b'\0') if b'=' in item) + if env.get(b'IMX_RUN_ID') != os.fsencode(run_id): + continue + log_dir = env.get(b'IMX_LOG_DIR') + if not log_dir or Path(os.fsdecode(log_dir)).resolve() != root: + continue + if born is not None and not _is_ancestor_of_self(pid): + identity[pid] = born + except (OSError, ValueError): + continue + return kill_tree(sorted(identity), identity=identity) if identity else [] diff --git a/src/infermatrix_copilot/rebase_engine/prompt_builder.py b/src/infermatrix_copilot/rebase_engine/prompt_builder.py index 30c9ee993..e54d7dbda 100644 --- a/src/infermatrix_copilot/rebase_engine/prompt_builder.py +++ b/src/infermatrix_copilot/rebase_engine/prompt_builder.py @@ -213,6 +213,7 @@ def build_module_prompt( adaptive_guidance: str = "", live: bool = False, baseline_ref: str = "origin/main", + test_timeout_sec: int = 7200, run_git: Callable[[list[str], str], str] | None = None, ) -> str: """Byte-parity render of the parent's `build_module_prompt`. `script_dir` @@ -261,6 +262,7 @@ def build_module_prompt( vars_map = { "MODULE_NAME": module, "MODULE_KEY": module, + "MODULE_TEST_TIMEOUT_SEC": str(test_timeout_sec), "VLLM_PATH": vllm_path, "OMNI_PATH": omni_path, "VLLM_COMMIT": vllm_commit, diff --git a/src/infermatrix_copilot/testing/gpu_lock.py b/src/infermatrix_copilot/testing/gpu_lock.py index c520f37cb..e3a0937fc 100644 --- a/src/infermatrix_copilot/testing/gpu_lock.py +++ b/src/infermatrix_copilot/testing/gpu_lock.py @@ -161,13 +161,8 @@ def visible_devices(spec: str) -> list[str]: def _pid_alive(pid: int) -> bool: - try: - os.kill(pid, 0) - return True - except ProcessLookupError: - return False - except PermissionError: # exists, other user - return True + from .process_tree import _alive + return _alive(pid) # -- nvidia-smi hygiene (injectable for tests) --------------------------------- diff --git a/src/infermatrix_copilot/testing/process_tree.py b/src/infermatrix_copilot/testing/process_tree.py index 5965f8ad2..2fd706eab 100644 --- a/src/infermatrix_copilot/testing/process_tree.py +++ b/src/infermatrix_copilot/testing/process_tree.py @@ -61,6 +61,12 @@ def _is_ancestor_of_self(pid: int) -> bool: def _alive(pid: int) -> bool: try: os.kill(pid, 0) + try: + with open(f"/proc/{pid}/stat") as stat: + if stat.read().rsplit(")", 1)[1].split()[0] in ("Z", "X"): + return False + except (OSError, IndexError): + pass return True except ProcessLookupError: return False diff --git a/test/test_provider_zcode.py b/test/test_provider_zcode.py index 2599c12ad..b9c9276b3 100644 --- a/test/test_provider_zcode.py +++ b/test/test_provider_zcode.py @@ -166,6 +166,18 @@ def test_run_session_plan_mode_bridge_and_parse(tmp_path, monkeypatch): assert session["served_model"] == "GLM-5.3-Flash" +def test_bridge_timeout_covers_long_running_module_tools(tmp_path): + transport = _transport(tmp_path) + req = _request(tmp_path) + req.timeout_s = 7200 + transport.run_session(req) + server = json.loads(_capture(tmp_path)["config"])["mcp"]["servers"][ + "infermatrix-tools"] + # ZCode defaults to 30 seconds without this setting, abandoning valid + # test and plan-review calls while the bridge continues executing them. + assert server.get("timeoutMs", 30000) == 7_200_000 + + def test_tool_outside_allowlist_is_an_audit_refusal(tmp_path): transport = _transport(tmp_path) (tmp_path / "bin" / "rogue").write_text("", encoding="utf-8") diff --git a/test/test_rebase_harness_backend.py b/test/test_rebase_harness_backend.py index c0ae1b3b8..4e1549c07 100644 --- a/test/test_rebase_harness_backend.py +++ b/test/test_rebase_harness_backend.py @@ -216,8 +216,8 @@ def run_session(self, req): cfg = _types.SimpleNamespace( log_dir="/tmp", backend="cursor", backend_model="", settings=None, model="deepseek-flash", script_dir="/tmp", manifest_path="", - repo="r", max_turns=10, harness_timeout_s=1.0, paths_spec={}, - state_slice={}, baseline_ref="origin/main") + repo="r", max_turns=10, harness_timeout_s=1.0, test_timeout_sec=7200, + paths_spec={}, state_slice={}, baseline_ref="origin/main") scope = ToolScope(name="s", allowed_tools=frozenset(), path_scope=None, read_only=True, root="/tmp") asyncio.run(_harness_attempt( diff --git a/test/test_rebase_process_lifecycle.py b/test/test_rebase_process_lifecycle.py new file mode 100644 index 000000000..48a955252 --- /dev/null +++ b/test/test_rebase_process_lifecycle.py @@ -0,0 +1,62 @@ +"""Real CPU processes exercise detached test ownership and zombie locks.""" +import os +from pathlib import Path +import subprocess +import sys +import time + +import pytest + +from infermatrix_copilot.rebase_engine.processes import cleanup_run_processes +from infermatrix_copilot.testing.gpu_lock import GpuLock +from infermatrix_copilot.testing.process_tree import _alive, kill_tree + +pytestmark = pytest.mark.skipif(not Path('/proc/self/stat').exists(), reason='requires procfs') + + +def test_zombie_owner_does_not_hold_gpu_mutex(tmp_path): + child = subprocess.Popen([sys.executable, '-c', 'pass']) + try: + deadline = time.monotonic() + 5 + while time.monotonic() < deadline: + state = Path(f'/proc/{child.pid}/stat').read_text().rsplit(')', 1)[1].split()[0] + if state == 'Z': + break + time.sleep(0.01) + else: + pytest.fail('child did not become an unreaped zombie') + # kill(pid, 0) still succeeds: this was the old lock's liveness test. + os.kill(child.pid, 0) + tmp_path.joinpath('lock').write_text(str(child.pid)) + with GpuLock(tmp_path, poll_sec=0.01, timeout_sec=0.2): + assert tmp_path.joinpath('lock').read_text() == str(os.getpid()) + assert not _alive(child.pid) + finally: + child.wait(timeout=5) + + +def test_cleanup_reaches_detached_orphan_and_preserves_other_runs(tmp_path): + run = tmp_path / 'run' + run.mkdir() + env = dict(os.environ, IMX_RUN_ID='fixture-run', IMX_LOG_DIR=str(run)) + launcher = subprocess.Popen([ + sys.executable, '-c', + 'import subprocess,sys; p=subprocess.Popen([sys.executable,"-c","import time;time.sleep(120)"], ' + 'stdin=subprocess.DEVNULL,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,start_new_session=True); ' + 'print(p.pid,flush=True)', + ], env=env, stdout=subprocess.PIPE, text=True) + orphan = int(launcher.stdout.readline()) + launcher.wait(timeout=5) + others = [subprocess.Popen([sys.executable, '-c', 'import time;time.sleep(120)'], + env=dict(env, **different)) for different in + [{'IMX_RUN_ID': 'another-run'}, {'IMX_LOG_DIR': str(tmp_path / 'other')}]] + try: + assert _alive(orphan) + assert cleanup_run_processes(run, 'fixture-run') == [] + assert not _alive(orphan) + assert all(p.poll() is None for p in others) + finally: + kill_tree([orphan, *(p.pid for p in others)], term_grace=0.1, kill_grace=0.1) + for p in others: + p.wait(timeout=5) + launcher.stdout.close() From 9f80a574166b0b6d29eccaf1f5bac1dfc04774d7 Mon Sep 17 00:00:00 2001 From: tzhouam Date: Tue, 27 Oct 2026 16:55:36 +0800 Subject: [PATCH 2/2] doc(SPEC): record attempt-scoped cleanup, zombie lock owners and module test budgets Co-authored-by: Cursor --- doc/architecture/SPEC/config.md | 7 ++++++- .../SPEC/engine/steps/rebase_v3.md | 13 ++++++++++++- doc/architecture/SPEC/providers/zcode.md | 6 +++++- doc/architecture/SPEC/rebase_engine.md | 18 +++++++++++++++++- doc/architecture/SPEC/testing.md | 6 +++++- 5 files changed, 45 insertions(+), 5 deletions(-) diff --git a/doc/architecture/SPEC/config.md b/doc/architecture/SPEC/config.md index f74ebbfe3..4e065f488 100644 --- a/doc/architecture/SPEC/config.md +++ b/doc/architecture/SPEC/config.md @@ -1,6 +1,6 @@ # config.py —— 规范 - + `LOC ~658 · 配置 · refactor-status: oversized` @@ -97,6 +97,11 @@ provider 的缺省模型必须在**这一处**解析，否则 `ResolvedTarget.mo 注意：tier 模型名指向 RAW-API 模型，对 harness CLI 无意义，**绝不**转发过去； 为空时回落到 `strict_backend_model`，再回落到 harness 自身默认值。 +`rebase_harness_timeout_sec`（默认 7200，必须 > 0；2026-10-27）是每次 module / +debug harness 会话的墙钟预算，取代 `ModuleRunConfig.harness_timeout_s` 原来的 +硬编码值；zcode 传输同时用它设置工具桥的 MCP `timeoutMs`。模块测试命令自身的 +`TEST_TIMEOUT_SEC` 不在这里——那是 adapter 的 `rebase.testing.module_timeout_sec`。 + ## 自进化接入（2026-10-04） `TRACE_STORE_ROOT` 控制 trace/1 全量采集。`IMPROVE_ENABLED=false` 是总停止开关，`IMPROVE_EVOLVE_ENABLED=false` 控制代码进化；源码、数据和发布 outbox 各有独立目录。周预算默认 20 美元，周度槽位默认周一 05:00 UTC。`PR_CONTEXT_SOURCE=snapshot` 与 `IMPROVE_SHADOW` 由可信实验控制器设置；`KB_DRAFT_MAX_OPERATIONS` 默认 6。 diff --git a/doc/architecture/SPEC/engine/steps/rebase_v3.md b/doc/architecture/SPEC/engine/steps/rebase_v3.md index 37b6fa3ed..04930e7b5 100644 --- a/doc/architecture/SPEC/engine/steps/rebase_v3.md +++ b/doc/architecture/SPEC/engine/steps/rebase_v3.md @@ -1,6 +1,6 @@ # engine/steps/rebase_v3.py —— 规范 - + `LOC ~2204 · step 库（v3 rebase 装配层） · refactor-status: oversized` @@ -100,3 +100,14 @@ harness 会话通过 MCP tool bridge 拿到同一套工具面，plan gate 在派 `build_backends(repo=)` 收的是仓库 **ROOT 路径**（→ `TestRunner(repo_root=)`）， 不是仓库名：传名字会让 run_pytest/run_precommit/reproduce 以 `FileNotFoundError` 静默失效，module 却报告完成。 + +## attempt 级进程回收与测试预算注入（2026-10-27） + +- `_agent_shell_env` 把 adapter 的 `rebase.testing.module_timeout_sec` + （缺省 7200）写成 `TEST_TIMEOUT_SEC`；`_v3_module_rebase` 与 `_run_debug_agent` + 据此填 `ModuleRunConfig.test_timeout_sec`，并用 + `settings.rebase_harness_timeout_sec` 代替硬编码的 harness 会话预算。 +- `_ensure_checkout_locks` 取锁前、锁 finalizer 释放前，以及 `_run_debug_agent` + 的 `finally`，都调用 `rebase_engine.processes.cleanup_run_processes`： + 上一个 attempt 留下的后台 pytest 链不能把 GPU 锁带进下一个 step + （run-20261024 的 `GpuLockTimeout` 阻断即由此而来）。 diff --git a/doc/architecture/SPEC/providers/zcode.md b/doc/architecture/SPEC/providers/zcode.md index 305f1d524..000ba997b 100644 --- a/doc/architecture/SPEC/providers/zcode.md +++ b/doc/architecture/SPEC/providers/zcode.md @@ -1,6 +1,6 @@ # providers/zcode.py —— 规范 - + `harness transport（Z.AI / GLM 订阅） · refactor-status: ok` @@ -39,6 +39,10 @@ argv 单参数上限 128KiB）。 `Read` 读附件时才保留 `Read`，且任何读到临时目录之外的调用都让这次调用失败。 one-shot 的局部配置关闭 memory/use、skills、plugins、subagent 和 MCP，并把 CLI storage 放入会话临时目录；避免保留下来的 Read 接触宿主自动记忆。工具桥会话同样关闭 memory/use。 +- **工具桥的 MCP 超时跟随会话预算。** 会话配置里写 + `mcp.servers..timeoutMs = timeout_s * 1000`；zcode 默认 30s， + 会在桥里仍在跑的测试或嵌套计划评审完成前放弃该工具调用，之后每次桥调用 + 都像挂死。会话 deadline 仍是执行的外层上界（2026-10-27）。 - **失败的运行不能读成空评审。** 非零退出或流里没有 `result` 事件（未超时时），普通调用 和原生事件调用先将已识别失败转换为固定安全消息的 `ZCodeFailure`；未知失败仍抛 `RuntimeError`，保留 zcode 自己的 stderr 诊断。 diff --git a/doc/architecture/SPEC/rebase_engine.md b/doc/architecture/SPEC/rebase_engine.md index e0365eafd..233107d8f 100644 --- a/doc/architecture/SPEC/rebase_engine.md +++ b/doc/architecture/SPEC/rebase_engine.md @@ -1,6 +1,6 @@ # rebase_engine/ —— 规范 - + `LOC ~7500（26 个模块） · repo-rebase-v3 的原生 rebase 引擎 · refactor-status: ok` @@ -31,6 +31,7 @@ | `path_sync.py` | 模块路径图同步 + manifest modules 段重写 + L2 决定应用 | | `phase1_steps.py` | phase-1 组合（归类 + 路径同步），父级报告文件名不变 | | `plan_review.py` | L4 计划评审后端（注入的 LLM client，父级形状的结果） | +| `processes.py` | 回收本 run 的后台测试进程（含 setsid 脱离的链）：按 `IMX_RUN_ID` + `IMX_LOG_DIR` 双标记 + 出生时间识别 | | `prompt_builder.py` | 模块/调试 prompt 渲染 —— 等输入下与父级字节一致（golden 钉住）；`*_live` 变体不受 parity 钉 | | `push_gate.py` | 推送闸裁决：结构性 vs 断言失败的确定性分类（Rev 8 §2.3） | | `push_to_ci.py` | commit+push-to-CI 编排：preflight、WAL 卫生、C4 双闸、单一传输 | @@ -156,3 +157,18 @@ agent 会反复从头探索。flash 级模型在同一 module 上约需 pro 级 现在 infra 记录里会带上已报告的断言 id（复用 `ci_loop.extract_failed_test_ids`）。 实例：2026-09-19 `simple_diffusion_test` 超时前已报 10 个量化断言失败，全部不可见， 直到 Buildkite #3055 重新发现。 + +## attempt 级进程回收与模块测试预算（2026-10-27） + +module agent 曾用 `setsid` 启动 pytest 链并在会话结束后留下它们继续跑： +run-20261024 里这些链在 wave 1 完成后又占着 run 的 GPU 锁约 7 小时， +`tests` step 等锁 3600s 后以 `GpuLockTimeout` 阻断。现在： +- `processes.cleanup_run_processes(run_dir, run_id)` 在每次 module/debug attempt + 结束时（包括正常返回）调用。只回收**同用户**且**同时**携带 `IMX_RUN_ID` + 与 `IMX_LOG_DIR` 两个标记的进程；身份绑定出生时间，PID 复用不会误杀； + 其他 run 的进程不受影响。prompt 不再教 agent 留下脱离链。 +- `ModuleRunConfig.test_timeout_sec` 来自 adapter 的 + `rebase.testing.module_timeout_sec`（vllm-omni 为 21600：offline inference + 整套约 4 小时，旧的 7200s 包装器永远到不了终态），渲染为 prompt 的 + `{MODULE_TEST_TIMEOUT_SEC}` 并写进每次 attempt 末尾的预算说明。 +- `harness_timeout_s` 不再硬编码 7200，取 `settings.rebase_harness_timeout_sec`。 diff --git a/doc/architecture/SPEC/testing.md b/doc/architecture/SPEC/testing.md index e4b52bb2f..fa27d31b0 100644 --- a/doc/architecture/SPEC/testing.md +++ b/doc/architecture/SPEC/testing.md @@ -1,6 +1,6 @@ # testing/ —— 规范 - + `LOC ~1760（6 个文件） · 测试执行底座（rebase agent shell 测试层的 Python 移植） · refactor-status: ok` @@ -47,6 +47,10 @@ test_runner.sh 的 Python 移植。仓库专属数据（watchdog 模式、artifa pid —— 升级 SIGKILL 绝不打中无关进程。 - GPU 锁绝不 unlink 活主的锁：偷锁在 flock side file 的临界区内**重新判定** staleness；release 只删除仍属于自己的锁文件。 +- 锁主存活判定走 `process_tree._alive`：`kill(pid, 0)` 成功但 + `/proc//stat` 为 `Z`/`X`（未被收割的僵尸）算**死**，否则一个已退出 + 但父进程未 wait 的 pytest 包装器会无限期占住互斥锁（2026-10-27， + `test_rebase_process_lifecycle` 用真实僵尸进程钉住）。 - **E2** —— Tier-2 无 reviewer / reviewer 异常 / 无法解析的回复一律 CONTINUE， **绝不 wedge 一次 run**；Tier 1 不经评审即杀。 - **E1（精神）** —— 每次 Tier-2 裁决落 decision log（`record_fn`/`result.decisions`），