Repair MCP retention diagnostic accounting and natural/GC evidence

docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst

@@ -0,0 +1,104 @@
+MCP diagnostic repair: issue131 / PR654
+======================================
+
+Delivery scope
+--------------
+
+The existing diagnostic no longer vetoes calls at an invented 8-GiB host floor,
+2-GiB process RSS ceiling, ten-round maximum or 180-second overall quota.
+Caller-selected finite rounds/cadence and per-call timeout retain a bounded
+workload. Linux availability, swap usage and full PSI are observations on
+ProcessMemoryReceipt, not a second admission policy or automatic stop.
+
+Natural end-of-round samples determine the retention slopes. Python's ordinary
+automatic GC remains enabled; no explicit GC precedes or interrupts these
+samples. Optional final --gc-at-end records separate signed sensitivity deltas.
+RetentionMetric annotations own both calculations, including inherited/new
+metrics. The strict read-only request admission remains unchanged. Declared
+health results are consumed through McpDevToolResult.decoded_payload_as, not
+decoded a second time as mappings.
+
+Source and ownership
+--------------------
+
+Production pin: 283c2378a, following initial checkpoint bcde628d0. Latest main
+49322f10b was normally integrated afterward, then main4d7f18858 (paper/review
+changes only); neither the diagnostic module nor its inherited launch module
+changed. The installed target is honestly pinned to
+283c2378a, not relabelled as later main. Foreign gitlinks/keepers were untouched.
+
+Existing audit Package/Repository AST: 710 MCP/runtime/unit modules, zero parse
+omissions; related declarations, imports, inherited launch and calls recorded.
+Git caller closure names only the existing diagnostic, launch, ownership guard
+and unit family as active consumers. Literal Linux accounting is decoded once
+inside ProcessMemoryReceipt.capture (BOUND-1/BOUND-2). DiagnosticServerSpec keeps
+the original McpDevServerSpec/import-authority lifecycle (IMPL-13); no replacement
+launcher, metric roster, codec, quota or runtime store was added.
+
+Original controls and failures
+------------------------------
+
+ownership02.log: existing focused AST ownership guard PASS.
+controls02.log/time: 13 PASS, 6.76 seconds, peak323940KiB, swaps0. Includes inherited
+metric eligibility, strict unknown/bool rejection, read-only mutation rejection,
+exact-source child launch, separate natural/GC calculations and acceptance of a
+synthetic4-GiB RSS sample with512-MiB available host memory as observation rather
+than a veto. Those last figures are a controlled sample, not actual host load.
+
+controls01 failed before collection because repository pytest addopts requested
+an unavailable tests plugin. The corrected invocation selected the isolated
+installed target and explicit pytest configuration; two cache warnings were
+nonfatal. No production guard or assertion was weakened.
+
+live01 failed before any memory sample: the diagnostic's old health reader tried
+dataclass_from_mapping on an already-typed result. Original client1720561 and
+MCP1720693 exited; JSON/log/time remain unchanged. The owning consumer was fixed
+before the explicitly new live02 attempt. No UNKNOWN input was replayed.
+
+Ordinary installation and real MCP
+----------------------------------
+
+Private receiving02 ordinary wheel/build/install uses the retained interpreter
+and five qualified dependency wheels, no downloads/new environment/shared
+installation changes. Build01 terminal0,139.46seconds, peak443660KiB, swaps0.
+BYTE-QUALIFICATION.json PASS:920 wheel RECORD entries,817 exact tracked source
+files,792 Python files,13 packaged skill files and complete installed RECORD
+accounting. Existing ordinary CLI synchronized only the private skills directory.
+Wheel SHA256:b967b610dfa8c51f2880c4766505fb07f4ed77a393ebc6898e828dfb387b01bb.
+
+live02 used the real installed diagnostic CLI, DiagnosticServerSpec and MCP
+transport: health, first_use and capability search, one warm-up plus three
+repeated rounds at0.2-second cadence, followed by one explicit final GC.
+Original tool handle94550 terminal0;18.95seconds, time-reported peak396100KiB,
+swaps0. All30 typed samples came from MCP1737168 and the exact receiving02
+import root. The original session context closed it; /proc1737168 is absent.
+No diagnostic client remains, and no native/viewer/science process was adopted.
+
+Natural end RSS:396696,397128,397444KiB. Declaration-derived RSS/PSS/private-dirty
+slopes:374KiB/round; process swap slope0. Final explicit GC delta0 for each
+declared metric. Minimum observed host availability9190232KiB; maximum full PSI
+avg10=1.17. These measurements were recorded, not converted into an invented
+pressure cutoff. LIVE-ACCEPTANCE02.json verifies exact natural/GC separation,
+source/PID identity, declared calculations and actual process exit.
+
+Remaining original acceptance
+-----------------------------
+
+This useful tool repair is not a no-leak finding, a causal explanation of the
+stable P0013.78-GB MCP sample, or completion of the original38-minute mixed
+request/revision workload. A new synthetic registration must use the ordinary
+typed registration path with an actually owned execution endpoint and prepared
+native catalog; none was borrowed or launched for this read-only qualification.
+No source revision was submitted. The old UNCERTAIN revision remains immutable
+and must never be replayed. Planck retains #131 follow-through; Dewey owns any
+future actual lane custody. #132/#169 remain separate unfinished obligations.
+
+Original evidence root
+----------------------
+
+/home/ts/wt/openhcs-issue-batch-20260929/engineering-mcp-retention-131-20261005
+
+Keep live01/live02 JSON/log/time, controls01/02, AST/caller closure and both
+receiving source origins/wheel proofs. receiving19 and every scientific frozen
+installation are unchanged. This receipt is source qualification plus the
+specified installed diagnostic path, not biological or whole-session acceptance.

openhcs/mcp/memory_diagnostic.py

@@ -12,6 +12,7 @@
 import asyncio
 import gc
 import json
+import math
 import os
 import sys
 import tempfile
@@ -73,17 +74,6 @@ def require_authority(self, authority: OpenHCSRuntimeImportAuthority) -> None:
             raise RuntimeError("Diagnostic child imported a different OpenHCS source")
 
 
-def require_ram_headroom() -> None:
-    """Refuse further requests under the batch's 8 GiB available-RAM floor."""
-    for line in Path("/proc/meminfo").read_text().splitlines():
-        if line.startswith("MemAvailable:"):
-            available_kib = int(line.split()[1])
-            if available_kib < 8 * 1024**2:
-                raise RuntimeError(f"Available RAM below 8 GiB: {available_kib} KiB")
-            return
-    raise RuntimeError("Linux MemAvailable accounting is unavailable")
-
-
 @dataclass(frozen=True)
 class RetentionMetric:
     """A sample-field declaration owning its projection and slope calculation."""
@@ -100,10 +90,13 @@ def slope(self, samples: Sequence[ProcessMemoryReceipt]) -> float:
             (index - center) * (value - mean) for index, value in enumerate(values)
         ) / sum((index - center) ** 2 for index in range(len(values)))
 
+    def change(self, before: ProcessMemoryReceipt, after: ProcessMemoryReceipt) -> int:
+        return self.project(after) - self.project(before)
+
 
 @dataclass(frozen=True)
 class ProcessMemoryReceipt:
-    """Linux process accounting, not host swap or execution-tree accounting."""
+    """Process accounting plus contemporaneous host telemetry, not a quota."""
 
     pid: int
     monotonic_seconds: float
@@ -124,6 +117,11 @@ class ProcessMemoryReceipt:
     history_snapshots: int
     imported_source_paths: tuple[str, ...]
     source_identity: DiagnosticSourceIdentity
+    host_mem_available_kib: int
+    host_swap_used_kib: int
+    host_full_psi_avg10: float
+    host_full_psi_avg60: float
+    host_full_psi_avg300: float
 
     @classmethod
     def from_payload(cls, payload: Mapping[str, object]) -> ProcessMemoryReceipt:
@@ -142,6 +140,18 @@ def retention_slopes(
             if isinstance(metric, RetentionMetric)
         }
 
+    @classmethod
+    def retention_changes(
+        cls, before: ProcessMemoryReceipt, after: ProcessMemoryReceipt
+    ) -> dict[str, int]:
+        """GC sensitivity uses the same declaration-owned metrics as retention."""
+        return {
+            name: metric.change(before, after)
+            for name, annotation in get_type_hints(cls, include_extras=True).items()
+            for metric in get_args(annotation)[1:]
+            if isinstance(metric, RetentionMetric)
+        }
+
     @classmethod
     def capture(cls, *, collect: bool) -> ProcessMemoryReceipt:
         from arraybridge import MemoryType
@@ -154,6 +164,16 @@ def capture(cls, *, collect: bool) -> ProcessMemoryReceipt:
             name, separator, value = line.partition(":")
             if separator:
                 accounting[name] = int(value.split()[0])
+        host = {
+            name: int(value.split()[0])
+            for line in Path("/proc/meminfo").read_text().splitlines()
+            for name, value in (line.split(":", 1),)
+        }
+        full_pressure = next(
+            line for line in Path("/proc/pressure/memory").read_text().splitlines()
+            if line.startswith("full ")
+        )
+        pressure = dict(part.split("=", 1) for part in full_pressure.split()[1:])
         libraries = tuple(
             sorted(
                 {
@@ -202,6 +222,11 @@ def capture(cls, *, collect: bool) -> ProcessMemoryReceipt:
                 if name in modules
             ),
             source_identity=DiagnosticSourceIdentity.capture(),
+            host_mem_available_kib=host["MemAvailable"],
+            host_swap_used_kib=host["SwapTotal"] - host["SwapFree"],
+            host_full_psi_avg10=float(pressure["avg10"]),
+            host_full_psi_avg60=float(pressure["avg60"]),
+            host_full_psi_avg300=float(pressure["avg300"]),
         )
 
 
@@ -222,20 +247,31 @@ class DiagnosticFailure:
 class MemoryDiagnosticReport:
     rounds: int
     request_sequence: tuple[McpDevToolCall, ...]
+    round_interval_seconds: float = 0.0
+    gc_at_end: bool = False
     events: list[MemorySampleEvent] = field(default_factory=list)
     health: McpServerHealthResult | None = None
     post_warmup_slopes_kib_per_round: dict[str, float] = field(default_factory=dict)
+    post_gc_change_kib: dict[str, int] = field(default_factory=dict)
     failure: DiagnosticFailure | None = None
     server_stderr_tail: str | None = None
     scope: str = (
         "one disposable real MCP process; no execution tree or host tmpfs attribution"
     )
     interpretation: str = (
-        "Slopes describe the recorded bounded request sequence only. Growth is not proof of a leak; "
+        "Slopes describe natural, pre-GC repeated-round samples only; optional final GC change "
+        "is separate sensitivity evidence, not the retention slope. Host telemetry is observational, "
+        "not admission or an automatic stop. Growth is not proof of a leak; "
         "flatness does not exonerate untested catalog, compile, custom-source, execution, JVM or GPU paths. "
         "Compare import/library/cache counts and repeat longer before attributing retention."
     )
 
+    def __post_init__(self) -> None:
+        if self.rounds < 2:
+            raise ValueError("A retention slope requires at least two repeated rounds.")
+        if not math.isfinite(self.round_interval_seconds) or self.round_interval_seconds < 0:
+            raise ValueError("Round interval must be finite and nonnegative.")
+
     def write(self, output: Path) -> None:
         output.write_text(json.dumps(asdict(self), indent=2) + "\n")
 
@@ -253,10 +289,13 @@ class MemoryDiagnosticMcpClient:
     session: McpDevStdioSession
     timeout_seconds: float
 
+    def __post_init__(self) -> None:
+        if not math.isfinite(self.timeout_seconds) or self.timeout_seconds <= 0:
+            raise ValueError("Per-call timeout must be finite and positive.")
+
     async def call(self, name: str, arguments: dict) -> McpDevToolResult:
         from openhcs.mcp.dev_client_core import McpDevToolResult
 
-        require_ram_headroom()
         wire = await self.session.call_tool(
             name, arguments, timeout_seconds=self.timeout_seconds
         )
@@ -272,16 +311,17 @@ async def health(self) -> McpServerHealthResult:
         from openhcs.agent.dto.mcp import McpServerHealthResult
 
         result = await self.call(agent_capabilities.health_check.name, {})
-        return dataclass_from_mapping(McpServerHealthResult, result.payloads[0])
+        health = result.decoded_payload_as(McpServerHealthResult)
+        if health is None:
+            raise RuntimeError("Diagnostic health returned no declared health result")
+        return health
 
     async def sample(self, *, collect: bool) -> ProcessMemoryReceipt:
         result = await self.call(PROBE_TOOL, {"collect": collect})
         receipt = ProcessMemoryReceipt.from_payload(result.payloads[0])
         receipt.source_identity.require_authority(
             OpenHCSRuntimeImportAuthority.current()
         )
-        if receipt.rss_kib > 2 * 1024**2:
-            raise RuntimeError("Diagnostic MCP process exceeded its 2 GiB RSS budget")
         return receipt
 
 
@@ -329,6 +369,8 @@ async def diagnose(
     surface_profile: LocalCapabilitySurfaceProfile,
     scratch_root: Path = SCRATCH_ROOT,
     sequence_path: Path | None = None,
+    round_interval_seconds: float = 0.0,
+    gc_at_end: bool = False,
 ) -> MemoryDiagnosticReport:
     from openhcs.agent.capabilities import agent_capabilities
     from openhcs.mcp.dev_client_core import (
@@ -351,12 +393,14 @@ async def diagnose(
         )
     )
 
-    report = MemoryDiagnosticReport(rounds=rounds, request_sequence=sequence)
+    report = MemoryDiagnosticReport(
+        rounds=rounds, request_sequence=sequence,
+        round_interval_seconds=round_interval_seconds, gc_at_end=gc_at_end,
+    )
     output.parent.mkdir(parents=True, exist_ok=True)
 
     # stderr is bounded to this owned temporary stream; it is never copied into
     # a shared environment or another process's log.
-    require_ram_headroom()
     scratch_root.mkdir(parents=True, exist_ok=True)
     with (
         tempfile.TemporaryDirectory(prefix="mcp-", dir=scratch_root) as scratch,
@@ -369,8 +413,8 @@ async def diagnose(
         )
         try:
             async with McpDevStdioSession(spec, stderr) as session:
-                await session.initialize(timeout_seconds=timeout)
                 client = MemoryDiagnosticMcpClient(session, timeout)
+                await session.initialize(timeout_seconds=client.timeout_seconds)
                 health = await client.health()
                 report.health = health
                 if (
@@ -389,26 +433,33 @@ async def sample(label: str, collect: bool) -> ProcessMemoryReceipt:
                     report.record(label, collect, receipt, output)
                     return receipt
 
-                await sample("baseline", True)
+                await sample("baseline", False)
                 # These are read-only discovery calls. The first round includes
                 # warm-up; subsequent rounds isolate repeated-request retention.
                 for request in sequence:
                     request.require_surface_profile(spec.surface_profile)
                 repeated: list[ProcessMemoryReceipt] = []
                 for round_index in range(rounds + 1):
+                    if round_index:
+                        await asyncio.sleep(report.round_interval_seconds)
                     for request in sequence:
                         await sample(
                             f"round-{round_index}/before/{request.name}", False
                         )
                         await client.call(request.name, request.arguments)
                         await sample(f"round-{round_index}/after/{request.name}", False)
-                        await sample(f"round-{round_index}/gc/{request.name}", True)
-                    end = await sample(f"round-{round_index}/end", True)
+                    end = await sample(f"round-{round_index}/end", False)
                     if round_index:
                         repeated.append(end)
                 report.post_warmup_slopes_kib_per_round = (
                     ProcessMemoryReceipt.retention_slopes(repeated)
                 )
+                report.write(output)
+                if report.gc_at_end:
+                    after_gc = await sample("final/gc", True)
+                    report.post_gc_change_kib = ProcessMemoryReceipt.retention_changes(
+                        repeated[-1], after_gc
+                    )
         except BaseException as error:
             report.failure = DiagnosticFailure(type(error).__name__, str(error))
             stderr.seek(0, 2)
@@ -435,6 +486,8 @@ def main() -> None:
     )
     parser.add_argument("--rounds", type=int, default=4)
     parser.add_argument("--timeout", type=float, default=30)
+    parser.add_argument("--round-interval-seconds", type=float, default=0.0)
+    parser.add_argument("--gc-at-end", action="store_true")
     parser.add_argument("--output", type=Path, default=Path("mcp-memory-receipt.json"))
     parser.add_argument("--scratch-root", type=Path, default=SCRATCH_ROOT)
     parser.add_argument(
@@ -459,19 +512,16 @@ def main() -> None:
     if args.server:
         serve(surface_profile)
         return
-    if not 2 <= args.rounds <= 10 or not 0 < args.timeout <= 60:
-        parser.error("rounds must be 2..10 and timeout must be >0..60 seconds")
     report = asyncio.run(
-        asyncio.wait_for(
-            diagnose(
+        diagnose(
                 args.rounds,
                 args.timeout,
                 args.output,
                 surface_profile=surface_profile,
                 scratch_root=args.scratch_root,
                 sequence_path=args.sequence_json,
-            ),
-            timeout=180,
+                round_interval_seconds=args.round_interval_seconds,
+                gc_at_end=args.gc_at_end,
         )
     )
     print(