rebase: reap attempt-scoped test chains and give module tests a real budget

adapters/vllm_omni/manifest.yaml

@@ -328,6 +328,10 @@ rebase:
       - [/distributed/, scheduler]
       - [/benchmarks/, benchmarks]
   testing:
+    # TEST_TIMEOUT_SEC for the module agents' required pytest commands. The
+    # full offline inference suite takes about four hours on the shared host,
+    # so the previous 7200s default could never produce a terminal result.
+    module_timeout_sec: 21600
     # root-level artifacts tests leak between runs (test_artifact_cleanup.sh
     # TEST_ARTIFACT_GLOBS, verbatim) — gitignored upstream but pile up on disk
     artifact_globs: ["audio_chatcmpl-*.wav", "audio_[0-9]*ch_*.wav",

adapters/vllm_omni/rebase/templates/module_rebase.live.prompt.tmpl

@@ -204,11 +204,18 @@ On its own line (parser-friendly):
 ## Verification (must all pass)
 **Module pytest wrapper:** Every verification line uses `imx-omni-pytest …`. **Do not replace with bare `pytest`.** It applies `test_watchdog.sh`, `TEST_TIMEOUT_SEC`, logs under `\${LOG_DIR}/tests/module_*.log`, and shared GPU lock for **every** pytest line from Phase-2 sessions.
 
-**Long-running shell waits — avoid CLI auth-token expiry:** A single `shellToolCall` blocking idle for >~120 seconds crashes mid-run with `Invalid API key — Please run /login`. The wrapper retries but each retry restarts your session.
-- **Do not** issue `sleep 480 && tail …` to wait for long tests. Use `imx-omni-pytest …` (has its own watchdog + log streaming).
-- For background tasks, prefer **short polled checks** (`for i in 1 2 3 …; do sleep 30; <check>; done`).
-- Cap per-call sleep at 60 s when waiting on a known-running pid; loop with status checks.
-- **Thinking stalls are equally dangerous.** Produce incremental output (file writes, short commands, debug memory entries) to maintain heartbeat visibility. Silent agents are timed out after 7200s with `No code changes from debug agents.`
+**Long-running verification:** The MCP client timeout follows the harness
+session budget; it is not a fixed 30-second limit. Use the module pytest wrapper
+with `TEST_TIMEOUT_SEC={MODULE_TEST_TIMEOUT_SEC}` and allow terminal completion.
+For short observation polls, guard missing paths before invoking commands that
+would otherwise read stdin. Do not duplicate a live test or confuse an
+observation timeout with a test result.
+- All commands, assertions, selected scope and granted GPU geometry remain required.
+- The engine cleans up this attempt's background processes when the attempt ends,
+  including after a normal return. Never return while verification is still live
+  or leave a detached chain for another module to inherit.
+- If background verification is needed, redirect all three standard streams,
+  record the wrapper PID, poll it, and collect its terminal outcome before returning.
 
 Run:
 {IMPORT_CHECK}

doc/architecture/SPEC/config.md

@@ -1,6 +1,6 @@
 # config.py —— 规范
 
-<!-- verified-against: 2026-10-08 -->
+<!-- verified-against: 2026-10-27 -->
 
 `LOC ~658 · 配置 · refactor-status: oversized`
 
@@ -97,6 +97,11 @@ provider 的缺省模型必须在**这一处**解析，否则 `ResolvedTarget.mo
 注意：tier 模型名指向 RAW-API 模型，对 harness CLI 无意义，**绝不**转发过去；
 为空时回落到 `strict_backend_model`，再回落到 harness 自身默认值。
 
+`rebase_harness_timeout_sec`（默认 7200，必须 > 0；2026-10-27）是每次 module /
+debug harness 会话的墙钟预算，取代 `ModuleRunConfig.harness_timeout_s` 原来的
+硬编码值；zcode 传输同时用它设置工具桥的 MCP `timeoutMs`。模块测试命令自身的
+`TEST_TIMEOUT_SEC` 不在这里——那是 adapter 的 `rebase.testing.module_timeout_sec`。
+
 ## 自进化接入（2026-10-04）
 
 `TRACE_STORE_ROOT` 控制 trace/1 全量采集。`IMPROVE_ENABLED=false` 是总停止开关，`IMPROVE_EVOLVE_ENABLED=false` 控制代码进化；源码、数据和发布 outbox 各有独立目录。周预算默认 20 美元，周度槽位默认周一 05:00 UTC。`PR_CONTEXT_SOURCE=snapshot` 与 `IMPROVE_SHADOW` 由可信实验控制器设置；`KB_DRAFT_MAX_OPERATIONS` 默认 6。

doc/architecture/SPEC/engine/steps/rebase_v3.md

@@ -1,6 +1,6 @@
 # engine/steps/rebase_v3.py —— 规范
 
-<!-- verified-against: 2026-09-29 -->
+<!-- verified-against: 2026-10-27 -->
 
 `LOC ~2204 · step 库（v3 rebase 装配层） · refactor-status: oversized`
 
@@ -100,3 +100,14 @@ harness 会话通过 MCP tool bridge 拿到同一套工具面，plan gate 在派
 `build_backends(repo=)` 收的是仓库 **ROOT 路径**（→ `TestRunner(repo_root=)`），
 不是仓库名：传名字会让 run_pytest/run_precommit/reproduce 以
 `FileNotFoundError` 静默失效，module 却报告完成。
+
+## attempt 级进程回收与测试预算注入（2026-10-27）
+
+- `_agent_shell_env` 把 adapter 的 `rebase.testing.module_timeout_sec`
+  （缺省 7200）写成 `TEST_TIMEOUT_SEC`；`_v3_module_rebase` 与 `_run_debug_agent`
+  据此填 `ModuleRunConfig.test_timeout_sec`，并用
+  `settings.rebase_harness_timeout_sec` 代替硬编码的 harness 会话预算。
+- `_ensure_checkout_locks` 取锁前、锁 finalizer 释放前，以及 `_run_debug_agent`
+  的 `finally`，都调用 `rebase_engine.processes.cleanup_run_processes`：
+  上一个 attempt 留下的后台 pytest 链不能把 GPU 锁带进下一个 step
+  （run-20261024 的 `GpuLockTimeout` 阻断即由此而来）。

doc/architecture/SPEC/providers/zcode.md

@@ -1,6 +1,6 @@
 # providers/zcode.py —— 规范
 
-<!-- verified-against: 2026-10-16 -->
+<!-- verified-against: 2026-10-27 -->
 
 `harness transport（Z.AI / GLM 订阅） · refactor-status: ok`
 
@@ -39,6 +39,10 @@ argv 单参数上限 128KiB）。
   `Read` 读附件时才保留 `Read`，且任何读到临时目录之外的调用都让这次调用失败。
   one-shot 的局部配置关闭 memory/use、skills、plugins、subagent 和 MCP，并把 CLI storage
   放入会话临时目录；避免保留下来的 Read 接触宿主自动记忆。工具桥会话同样关闭 memory/use。
+- **工具桥的 MCP 超时跟随会话预算。** 会话配置里写
+  `mcp.servers.<bridge>.timeoutMs = timeout_s * 1000`；zcode 默认 30s，
+  会在桥里仍在跑的测试或嵌套计划评审完成前放弃该工具调用，之后每次桥调用
+  都像挂死。会话 deadline 仍是执行的外层上界（2026-10-27）。
 - **失败的运行不能读成空评审。** 非零退出或流里没有 `result` 事件（未超时时），普通调用
   和原生事件调用先将已识别失败转换为固定安全消息的 `ZCodeFailure`；未知失败仍抛
   `RuntimeError`，保留 zcode 自己的 stderr 诊断。

doc/architecture/SPEC/rebase_engine.md

@@ -1,6 +1,6 @@
 # rebase_engine/ —— 规范
 
-<!-- verified-against: 2026-09-29 -->
+<!-- verified-against: 2026-10-27 -->
 
 `LOC ~7500（26 个模块） · repo-rebase-v3 的原生 rebase 引擎 · refactor-status: ok`
 
@@ -31,6 +31,7 @@
 | `path_sync.py` | 模块路径图同步 + manifest modules 段重写 + L2 决定应用 |
 | `phase1_steps.py` | phase-1 组合（归类 + 路径同步），父级报告文件名不变 |
 | `plan_review.py` | L4 计划评审后端（注入的 LLM client，父级形状的结果） |
+| `processes.py` | 回收本 run 的后台测试进程（含 setsid 脱离的链）：按 `IMX_RUN_ID` + `IMX_LOG_DIR` 双标记 + 出生时间识别 |
 | `prompt_builder.py` | 模块/调试 prompt 渲染 —— 等输入下与父级字节一致（golden 钉住）；`*_live` 变体不受 parity 钉 |
 | `push_gate.py` | 推送闸裁决：结构性 vs 断言失败的确定性分类（Rev 8 §2.3） |
 | `push_to_ci.py` | commit+push-to-CI 编排：preflight、WAL 卫生、C4 双闸、单一传输 |
@@ -156,3 +157,18 @@ agent 会反复从头探索。flash 级模型在同一 module 上约需 pro 级
 现在 infra 记录里会带上已报告的断言 id（复用 `ci_loop.extract_failed_test_ids`）。
 实例：2026-09-19 `simple_diffusion_test` 超时前已报 10 个量化断言失败，全部不可见，
 直到 Buildkite #3055 重新发现。
+
+## attempt 级进程回收与模块测试预算（2026-10-27）
+
+module agent 曾用 `setsid` 启动 pytest 链并在会话结束后留下它们继续跑：
+run-20261024 里这些链在 wave 1 完成后又占着 run 的 GPU 锁约 7 小时，
+`tests` step 等锁 3600s 后以 `GpuLockTimeout` 阻断。现在：
+- `processes.cleanup_run_processes(run_dir, run_id)` 在每次 module/debug attempt
+  结束时（包括正常返回）调用。只回收**同用户**且**同时**携带 `IMX_RUN_ID`
+  与 `IMX_LOG_DIR` 两个标记的进程；身份绑定出生时间，PID 复用不会误杀；
+  其他 run 的进程不受影响。prompt 不再教 agent 留下脱离链。
+- `ModuleRunConfig.test_timeout_sec` 来自 adapter 的
+  `rebase.testing.module_timeout_sec`（vllm-omni 为 21600：offline inference
+  整套约 4 小时，旧的 7200s 包装器永远到不了终态），渲染为 prompt 的
+  `{MODULE_TEST_TIMEOUT_SEC}` 并写进每次 attempt 末尾的预算说明。
+- `harness_timeout_s` 不再硬编码 7200，取 `settings.rebase_harness_timeout_sec`。

doc/architecture/SPEC/testing.md

@@ -1,6 +1,6 @@
 # testing/ —— 规范
 
-<!-- verified-against: 2026-08-25 -->
+<!-- verified-against: 2026-10-27 -->
 
 `LOC ~1760（6 个文件） · 测试执行底座（rebase agent shell 测试层的 Python 移植） · refactor-status: ok`
 
@@ -47,6 +47,10 @@ test_runner.sh 的 Python 移植。仓库专属数据（watchdog 模式、artifa
   pid —— 升级 SIGKILL 绝不打中无关进程。
 - GPU 锁绝不 unlink 活主的锁：偷锁在 flock side file 的临界区内**重新判定**
   staleness；release 只删除仍属于自己的锁文件。
+- 锁主存活判定走 `process_tree._alive`：`kill(pid, 0)` 成功但
+  `/proc/<pid>/stat` 为 `Z`/`X`（未被收割的僵尸）算**死**，否则一个已退出
+  但父进程未 wait 的 pytest 包装器会无限期占住互斥锁（2026-10-27，
+  `test_rebase_process_lifecycle` 用真实僵尸进程钉住）。
 - **E2** —— Tier-2 无 reviewer / reviewer 异常 / 无法解析的回复一律 CONTINUE，
   **绝不 wedge 一次 run**；Tier 1 不经评审即杀。
 - **E1（精神）** —— 每次 Tier-2 裁决落 decision log（`record_fn`/`result.decisions`），

src/infermatrix_copilot/config.py

@@ -7,7 +7,7 @@
 from typing import Annotated, Literal
 from urllib.parse import urlparse
 
-from pydantic import field_validator, model_validator
+from pydantic import Field, field_validator, model_validator
 from pydantic_settings import BaseSettings, NoDecode, SettingsConfigDict
 
 _PACKAGE_ROOT = Path(__file__).resolve().parent
@@ -256,6 +256,7 @@ class Settings(BaseSettings):
     # is meaningless to a harness CLI, so it is never forwarded there; empty
     # falls back to `strict_backend_model`, then the harness's own default.
     rebase_backend_model: str = ""
+    rebase_harness_timeout_sec: int = Field(default=7200, gt=0)
     # v3 remote CI (rebase_engine/ci_loop.run_ci_rounds) — neutral knobs;
     # pipeline identities live in the adapter. Defaults are the parent's.
     rebase_ci_retries: int = 2          # push/rebuild rounds after the first

src/infermatrix_copilot/engine/steps/rebase_v3.py

@@ -185,6 +185,9 @@ def _agent_shell_env(ctx: StepContext, manifest: dict, repo_root: str,
     # with the run identity and merges the runtime noise overlay
     _state = getattr(ctx, "state", None) or {}
     env["IMX_RUN_ID"] = str(_state.get("run_id", "") or ctx.run_dir.name)
+    env["TEST_TIMEOUT_SEC"] = str(
+        ((manifest.get("rebase") or {}).get("testing") or {})
+        .get("module_timeout_sec", 7200))
     _settings = getattr(ctx, "settings", None)
     if _settings is not None:
         from ...memory.paths import KnowledgePaths as _KP2
@@ -689,11 +692,17 @@ def _ensure_checkout_locks(ctx: StepContext, manifest: dict,
         held.append(lock)
     _HELD_LOCKS[key] = held
     from ..lifecycle import register_finalizer
+    from ...rebase_engine.processes import cleanup_run_processes
+
+    cleanup_run_processes(ctx.run_dir, ctx.run_dir.name)
 
     async def _release_locks(_outcome, _key=key) -> None:
-        for h in _HELD_LOCKS.pop(_key, []):
-            h.release()
-        _drop_serial_lock(_key)
+        try:
+            cleanup_run_processes(ctx.run_dir, ctx.run_dir.name)
+        finally:
+            for h in _HELD_LOCKS.pop(_key, []):
+                h.release()
+            _drop_serial_lock(_key)
 
     register_finalizer(ctx.run_dir, _release_locks)
     ctx.trace.record("checkout_locks_acquired",
@@ -1165,6 +1174,9 @@ async def _run_debug_agent(ctx: StepContext, manifest: dict, module: str,
     except Exception as exc:  # noqa: BLE001 - a debug crash is STRUCTURAL
         ctx.trace.record("debug_attempt_error", slug=slug, error=str(exc))
         return f"error:debug agent crashed: {exc}"
+    finally:
+        from ...rebase_engine.processes import cleanup_run_processes
+        cleanup_run_processes(ctx.run_dir, ctx.run_dir.name)
     if result.get("done"):
         return "done"
     # done=False from the loop is ALWAYS machinery/budget (fatal auth,
@@ -1804,6 +1816,8 @@ async def _v3_module_rebase(ctx: StepContext) -> StepResult:
         # harness delegation: "api" (default) keeps the in-process loop
         backend=getattr(ctx.settings, "rebase_backend", "api") or "api",
         backend_model=getattr(ctx.settings, "rebase_backend_model", "") or "",
+        harness_timeout_s=ctx.settings.rebase_harness_timeout_sec,
+        test_timeout_sec=int(paths.env["TEST_TIMEOUT_SEC"]),
         settings=ctx.settings,
         manifest_path=str(adapter_dir / "manifest.yaml"),
         paths_spec={k: val for k, val in _dc_asdict(paths).items()

src/infermatrix_copilot/providers/zcode.py

@@ -200,13 +200,18 @@ def auth_gap(self) -> str | None:
 
     # -- process plumbing ----------------------------------------------------
     @staticmethod
-    def _write_mcp_config(session: Path, spec_path: Path) -> None:
+    def _write_mcp_config(session: Path, spec_path: Path, *,
+                          timeout_s: float = 30.0) -> None:
         package_root = Path(__file__).resolve().parents[2]
         config = session / ".zcode" / "config.json"
         config.parent.mkdir()
         config.write_text(json.dumps({"features": {"memory": False}, "memory": {"use": False},
                                      "mcp": {"servers": {_BRIDGE_SERVER: {
             "type": "stdio",
+            # ZCode otherwise abandons tools after 30s, even while a valid
+            # test or nested plan review is still running in the bridge.
+            # The session deadline remains the outer bound for execution.
+            "timeoutMs": max(1, int(timeout_s * 1000)),
             "command": sys.executable,
             "args": ["-m", "infermatrix_copilot.tool_bridge",
                      "--spec", str(spec_path)],
@@ -608,7 +613,8 @@ def run_session(self, req: AgentSessionRequest) -> AgentOutcome:
         session = Path(tempfile.mkdtemp(prefix="imc-zcode-session-"))
         try:
             if req.bridge_spec_path is not None:
-                self._write_mcp_config(session, req.bridge_spec_path)
+                self._write_mcp_config(session, req.bridge_spec_path,
+                                       timeout_s=req.timeout_s)
             header = ""
             if req.scope.root:
                 header = (f"The repository checkout is at {req.scope.root}. "

src/infermatrix_copilot/rebase_engine/module_rebase.py

@@ -59,6 +59,7 @@ class ModuleRunConfig:
     # the harness HAS one (claude --max-turns); cursor/codex have none, so the
     # real bound there is this timeout plus the prompt's budget discipline.
     harness_timeout_s: float = 7200.0
+    test_timeout_sec: int = 7200
     max_debug_retries: int = 3
     plan_review_max_rounds: int = 2
     model_aliases: Mapping[str, str] | None = None
@@ -180,6 +181,13 @@ async def _harness_attempt(prompt: str, *, module: str, config,
     from ..tool_bridge import write_bridge_spec
 
     run_dir = Path(config.log_dir)
+    prompt += (f"\n\nCurrent operator-selected budgets: this harness session has "
+               f"{config.harness_timeout_s:g} seconds; required module test "
+               f"commands have TEST_TIMEOUT_SEC={config.test_timeout_sec}. "
+               "These replace older attempt budgets. All test commands, "
+               "assertions and model geometry remain required. Background "
+               "tests belong to this attempt and will be stopped when it ends; "
+               "do not leave a detached chain running after returning.")
     transport = transport_for_id(config.settings, config.backend)
     spec_path = write_bridge_spec(
         run_dir=run_dir, step_name=f"rebase.module.{module}", scope=scope,
@@ -266,7 +274,8 @@ async def rebase_module(
         plan_review_max_rounds=config.plan_review_max_rounds,
         broken_imports=broken_imports, module_test_plan=module_test_plan,
         adaptive_guidance=guidance, live=True,
-        baseline_ref=config.baseline_ref)
+        baseline_ref=config.baseline_ref,
+        test_timeout_sec=config.test_timeout_sec)
 
     substate.update({"modules": {module: {"status": "running"}}})
     agent_log = str(Path(config.log_dir) / "agents" / f"module-{module}.log")
@@ -291,6 +300,9 @@ async def _attempt(p: str, *, require_plan_review: bool = True) -> dict:
         except Exception as exc:  # noqa: BLE001 - failure is substate data
             return {"done": False, "text": f"agent loop error: {exc}",
                     "turns": 0, "plan_done": False}
+        finally:
+            from .processes import cleanup_run_processes
+            cleanup_run_processes(Path(config.log_dir), substate.run_id)
 
     # parent parity: an incomplete first run gets up to max_debug_retries
     # follow-up attempts with the debug prompt built from the failure text.

src/infermatrix_copilot/rebase_engine/processes.py

@@ -0,0 +1,40 @@
+"""Reap run-owned background tests, including reparented setsid chains."""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+from ..testing.process_tree import _is_ancestor_of_self, _start_time, kill_tree
+
+
+def cleanup_run_processes(run_dir: Path, run_id: str) -> list[int]:
+    """Only same-user processes carrying BOTH exact run markers are owned.
+
+    Descendant snapshots cannot find a shell that daemonized between samples.
+    The inherited markers survive that reparenting. Record process birth times
+    before killing so a reused PID cannot turn cleanup into a bystander kill.
+    """
+    if not run_id:
+        return []
+    root = Path(run_dir).resolve()
+    identity: dict[int, int] = {}
+    for entry in Path('/proc').iterdir():
+        if not entry.name.isdigit():
+            continue
+        pid = int(entry.name)
+        try:
+            if entry.stat().st_uid != os.getuid():
+                continue
+            born = _start_time(pid)
+            env = dict(item.split(b'=', 1) for item in
+                       (entry / 'environ').read_bytes().split(b'\0') if b'=' in item)
+            if env.get(b'IMX_RUN_ID') != os.fsencode(run_id):
+                continue
+            log_dir = env.get(b'IMX_LOG_DIR')
+            if not log_dir or Path(os.fsdecode(log_dir)).resolve() != root:
+                continue
+            if born is not None and not _is_ancestor_of_self(pid):
+                identity[pid] = born
+        except (OSError, ValueError):
+            continue
+    return kill_tree(sorted(identity), identity=identity) if identity else []

src/infermatrix_copilot/rebase_engine/prompt_builder.py

@@ -213,6 +213,7 @@ def build_module_prompt(
     adaptive_guidance: str = "",
     live: bool = False,
     baseline_ref: str = "origin/main",
+    test_timeout_sec: int = 7200,
     run_git: Callable[[list[str], str], str] | None = None,
 ) -> str:
     """Byte-parity render of the parent's `build_module_prompt`. `script_dir`
@@ -261,6 +262,7 @@ def build_module_prompt(
     vars_map = {
         "MODULE_NAME":       module,
         "MODULE_KEY":        module,
+        "MODULE_TEST_TIMEOUT_SEC": str(test_timeout_sec),
         "VLLM_PATH":         vllm_path,
         "OMNI_PATH":         omni_path,
         "VLLM_COMMIT":       vllm_commit,

src/infermatrix_copilot/testing/gpu_lock.py

@@ -161,13 +161,8 @@ def visible_devices(spec: str) -> list[str]:
 
 
 def _pid_alive(pid: int) -> bool:
-    try:
-        os.kill(pid, 0)
-        return True
-    except ProcessLookupError:
-        return False
-    except PermissionError:  # exists, other user
-        return True
+    from .process_tree import _alive
+    return _alive(pid)
 
 
 # -- nvidia-smi hygiene (injectable for tests) ---------------------------------