From 7e782663c40d787f137281fae8a1612c1b279234 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sun, 4 Oct 2026 14:57:14 -0400 Subject: [PATCH 1/5] Plan remaining MCP, CZI and desktop receiving acceptance (#131, #132, #169) --- .../setup_acceptance_followup_131_132_169.md | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 plans/setup_acceptance_followup_131_132_169.md diff --git a/plans/setup_acceptance_followup_131_132_169.md b/plans/setup_acceptance_followup_131_132_169.md new file mode 100644 index 0000000000..b44a28ebaf --- /dev/null +++ b/plans/setup_acceptance_followup_131_132_169.md @@ -0,0 +1,256 @@ +# Complete the original setup and receiving obligations: #131, #132, #169 + +This draft owns the unfinished acceptance of all three issues. It is not ready +to merge. The linked source fixes are already merged; they do not establish the +remaining long-lived MCP, real Java/CZI, or continuous desktop receiving results. +No new run, environment change, or performance result accompanies this plan. +Keep each issue open until its section below has a determining final receipt. + +## Existing implementation and evidence + +| Issue | Merged implementation | Established evidence | Still required | +| --- | --- | --- | --- | +| #131 | [#208](https://github.com/OpenHCSDev/openhcs/pull/208), [#575](https://github.com/OpenHCSDev/openhcs/pull/575), and the retained-runtime ownership changes on main | A short 51-event read-only installed diagnostic was flat after warmup. Later 30 mixed rounds/five artifact-plan reads and a synthetic 16-field/eight-well native job completed. Default consolidation releases full image/label observations while explicit VALUES and declared PLATE inputs retain their requirements. | The original approximately 38-minute mixed-MCP/custom-revision scope, resource attribution, and an unambiguous revision-transition result. | +| #132 | [#206](https://github.com/OpenHCSDev/openhcs/pull/206) and [PolyStore #16](https://github.com/OpenHCSDev/PolyStore/pull/16) | Selected-source discovery and reader lifetime have source controls; ordinary installed ImageXpress initialization/readback passed. The 128×128 fragmented-label profile preserved parents 7/42/188 and 902 ZIP members with one `find_objects` scan. | Actual installed Java/CZI selection before unrelated container opens; complete ROI identity/geometry/readback and representative export-cost qualification. | +| #169 | [#208](https://github.com/OpenHCSDev/openhcs/pull/208), [ArrayBridge #1](https://github.com/OpenHCSDev/ArrayBridge/pull/1), and the normal desktop restart/history owners on main | An unpublished decorated callable reproduced serialization of `_thread._local`. `ThreadGPUContext` now owns that runtime storage; continuous native declaration/history persistence and undo controls passed. | A fresh installed desktop's continuous capture/restart/restore, complete retained undo/history, and exact original-process disposition. | + +Determining committed receipts, read for this plan: + +- [Memory/session ownership and original dispositions](../docs/validation/memory-session-20260929.md). +- [Runtime-observation receiving and remaining mixed-MCP scope](../docs/validation/runtime-observation-retention-131.rst). +- [Input preparation source/work-count evidence](../docs/validation/input_preparation_parent_20261001.rst). +- [Input preparation ownership correction](../docs/validation/input_preparation_quality_20261001.rst). +- [Installed input-preparation boundary](../docs/validation/input_preparation_installed_20261001.rst). + +The original `issue131-memory/public94-attempt01/PUBLIC-ACCEPTANCE01.rst` and +`QUALIFICATION01.rst` paths under `/home/ts/wt/openhcs-issue-batch-20260929` are +historical references from those receipts. They are not present on this machine; +their original custodian must supply the immutable packet before a follow-up +claims to have inspected it. Do not manufacture or overwrite missing evidence. + +## Receiving boundaries shared by the three issues + +Use the ordinary installed entrypoint and its existing +`OpenHCSRuntimeImportAuthority`, `McpDevServerSpec`, typed `McpDevToolCall` and +`McpDevToolResult`. Record the exact application/dependency commits, actual import +paths, native binary qualification and interpreter/JDK identity. A source-only +pass and an installed/native pass are separate dispositions. + +Reuse an existing clean checkout and existing qualified environments. Retain the +canonical checkout's dirty gitlinks, protected user sessions and scientific +inputs/results. New receiving data is synthetic or explicitly admitted existing +non-biological fixture data, in an owned isolated directory. Use existing path +admission and exact PID-plus-creation-time process ownership; an uncertain tool +outcome is not permission to retry, replace a process or replay science. + +Run receiving lanes serially after their actual resource/desktop owner hands off +the slot. Preserve the original memory, CPU and swap bounds; do not raise a cap +to obtain a pass. Keep startup/import/library preparation separate from repeated +request growth and image/export work. Cleanup affects only identity-proved owned +processes and disposable generated inputs after retaining final evidence. + +## #131: long-lived mixed MCP and source-revision receiving + +The original observation was about 999 MiB RSS plus 461 MiB swap after roughly +38 minutes. That is neither a demonstrated leak nor attribution of host tmpfs +swap to MCP. The later five-GiB native-family OOM remains a different retained +failure: 49/96 fields produced files without terminal native completion. The +synthetic native consolidation acceptance in #575 does not resolve its cause. + +The existing sampler is +`openhcs.mcp.memory_diagnostic.ProcessMemoryReceipt`; its declaration-derived +`RetentionMetric` and `MemoryDiagnosticReport` already own projections/events. +`DiagnosticServerSpec` owns an instrumented real MCP child, and +`MemoryDiagnosticMcpClient` owns typed tool results and samples. Extend these +owners rather than adding a second launcher, observer, memory schema or loop. + +The current CLI has a 180-second overall limit and 2..10 repeat rounds. +`read_request_sequence` admits only declared read-only calls; artifact-plan +inspection and custom registration cannot be smuggled through it. A current +short diagnostic command, after normal installed-source admission, is: + +```sh +"$RECEIVING_PYTHON" -m openhcs.mcp.memory_diagnostic --source-identity +"$RECEIVING_PYTHON" -m openhcs.mcp.memory_diagnostic \ + --rounds 10 --timeout 30 --sequence-json "$READ_ONLY_SEQUENCE" \ + --scratch-root "$RECEIVING_ROOT/mcp-scratch" \ + --output "$RECEIVING_ROOT/read-only-memory.json" +``` + +These commands do not provide the missing long-duration acceptance. Before that +run, make the existing diagnostic's duration/cadence and retained event budget +explicit on its current owner, keeping finite total duration and sample bounds. +Use one continuing child for at least the original 38-minute observation window, +with separate import/warmup, repeated operations and idle-release phases. The +current per-call deadlines stay bounded; elapsed duration is not an increased +tool timeout. Persist typed samples before a failure/ceiling terminates the run. + +For declared mutations, use the existing client's normal public tool-call path +with explicit admission of the owned synthetic source and destinations; retain +the read-only sequence decoder's rejection law. Interleave health, function +catalog, knowledge and artifact-plan reads with registration of a deterministic +synthetic callable A and a distinct admitted revision B. Discover current tool +schemas from the declaration-generated capability catalog. Record registration +receipts and actual callable identities; an unrelated catalog listing cannot +stand in for successful revision admission. The previous revision-two UNCERTAIN +receipt remains unchanged and is not replayed. + +Sample MCP RSS/PSS/private dirty/swap, threads/modules, declared catalog/custom +state and ObjectState snapshots before/after operations. Record natural release +and optional GC observations separately; never restart the measured child or +force collection to present a flat natural trace. Attribute any owned native +worker/viewer/JVM separately through its actual process incarnation. The saved +short #575 native job need not be rerun to pad this MCP acceptance. + +Acceptance requires a complete captured request history and source A→B result, +stable intended cache cardinalities after repeated same-source requests, and +bounded retained ownership after idle/release. Explain observed growth through +its determining cache/import/runtime owner; flat short runs and a slope alone +are not causal evidence. If retained request state grows, fix its existing owner +and repeat the same admitted workload once against that patch. Reject any +resource ceiling or uncertain revision result as unfinished acceptance, retain +the original receipt, and do not merge this follow-up as a no-leak finding. + +## #132: real selected CZI admission and fragmented ROI export + +The original directory contained 16 CZI containers, while one selected axis/two +physical planes were requested. Initialization opened all 16 before compilation; +the successful diagnostic later took about 170.6 seconds. Its round-object step +had 39.425 seconds of execution and 74.394 seconds of finalization. Those clocks +belong to the old source/input and are not a current baseline. + +Existing `NamedSourceBinding`/`SourceSelector` physical filters and +`BioFormatsHandler.initialize_workspace` own admission before discovery. +`BioFormatsJavaAdapter` delegates actual readers to PolyStore's +`BioFormatsJavaContext.open_reader`; that owner controls reader/probe lifetime. +The original `test_selected_container_opens_once_and_projects_both_planes` +uses empty `.czi` files and a controlled decoder. It is useful source admission +evidence, not actual Java/CZI receiving. + +First secure the original custodian's approved non-biological, real CZI fixture +and its channel/series/plane expectations. No genuine CZI fixture is identified +by the committed manifest-style/empty-file controls; that is an explicit input +gate, not permission to substitute NPY metadata for a native container. Prepare +16 independently inventoried allowed paths and declare one physical filename +with the same binding/global selector semantics. Preserve the source bytes. + +Through the ordinary installed MCP admission/compile route, initialize that +workspace and select exactly two declared physical planes. Capture reader-open +events at the existing Java-context boundary without replacing its reader or +discovery semantics. Require the selected container's actual series/channel/ +Z/T identities, calibration and bounded original pixel readbacks; unrelated +containers must not be opened. Compile a tiny source-preserving callable and +execute/read back its declared output. Record actual JVM startup and owned +shutdown separately from selected-container work. If discovery reopens excluded +paths, repair the determining selector/discovery owner, not a filename cache or +an increased preparation timeout. + +ROI extraction remains on PolyStore's `RegionProperties` and +`TwoDimensionalLabeledMaskROIExtractor`; `FijiROIConverter` owns conversion of +parent ROIs to contour members, `NapariShapeMetadata` owns label/area/centroid +projection, and `DiskStorageBackend` owns archive persistence. The original 188 +parents/27,627 members and 10,121 prefilter parents/108,281 members are fragment +counts, not biological cell counts. One parent had 2,422 members. + +Reuse the existing provider-free synthetic frontier first: + +```sh +"$RECEIVING_PYTHON" -m benchmark.fragmented_roi_profile \ + "$RECEIVING_ROOT/fragmented-roi" +``` + +The command already owns a 128×128 fixture with parent IDs 7/42/188, including +900 disconnected fragments of parent 188. It profiles extraction, conversion +and archive writing separately. Its one `find_objects` scan and 902 ZIP members +are historical evidence, not a new result. Extend this same receiving path to +read back through the original archive converter/metadata reader, retaining +parent identities, per-member geometry, calibration and source-plane lineage. +Include holes/concave contours and compare the declared raster/ROI relation; +the current receipt explicitly does not prove polygon-hole subtraction or +raster/viewer equivalence. + +Only after that bounded frontier passes, admit a deterministic synthetic +fragment topology representing the original member-count scale, within the +existing resource budget. Price extraction, conversion, disk publication and +readback on the same admitted source and output contract. Do not repeat the +private 2586×2586 biological diagnostic. Any proposed performance fix must have +a material maximum payoff against the measured finalization gap and preserve +the whole correlated parent/member geometry; saving a small scan is not full +issue completion. Reuse the existing profile and writer, without a parallel +export mode, copied converter or guessed archive-name identity. + +Close #132 only after both the real selected-CZI receiving and the declared +fragmented-ROI identity/fidelity/performance boundary are qualified. Keep +independent Opera issue #172 and biological inference outside that claim. + +## #169: continuous installed desktop history capture and restore + +The original H001 on display `:89`, PID 608239, returned rejected operation +`6633ef33-d5e5-47e8-9e4a-238234830870`: `cannot pickle _thread._local`. Its current +declarations export was not a restorable undo/history capture. The determining +receipt says H001 was subsequently closed by explicit owner disposition, and +there is no authorization to recover its lost in-memory history. Do not treat +the historical PID as live, recreate that biological session, or change H002. + +Use a fresh owned installed desktop and a tiny synthetic plate. Register an +unpublished NumPy-decorated callable, make two discriminating parameter/history +changes, and retain current versus saved declarations, ordered snapshot/branch +identities, selected plate and historical callable results. This is the case +that exercises ArrayBridge's importable `ThreadGPUContext` runtime owner instead +of serializing its unpickleable storage. A live GPU workload is not required to +exercise decorated-callable history persistence. + +`DesktopSessionRestart` delegates capture to `DesktopRestartSession` in +`desktop_update.py`; ObjectState owns durable history, `DesktopHistoryUpgrade` +owns occurrence migration, and `ConsumedDesktopRestartSession` restores the +declarations/history/selection. Use these owners and their ordinary installed +public UI bridge, not a direct dill round-trip as a desktop substitute. + +Use the declaration-generated tools in this order, with the explicitly owned +bridge connection and request schemas discovered from the running UI: + +1. `openhcs_ui_list_actions` and `openhcs_ui_get_code_document`: retain the exact + `main_window` restart action, selection revision, availability and current + document; use `openhcs_ui_list_snapshots`/`openhcs_ui_list_branches` for history. +2. `openhcs_ui_invoke_action`: request `restart_session` exactly once, honoring + its existing confirmation requirement. Capture must finish before the old + window's normal close. Pending work or an incomplete capture must reject. +3. Observe the original operation through `openhcs_ui_get_operation_status` + while its bridge is alive. An accepted receipt precedes process exit; it is + not a durable operation handle in the new process. Rediscover the new bridge + rather than polling the disposed old process or repeating restart. +4. Verify the restored current/saved document and selected plate, complete + branch/snapshot order, historical undo and return to the current head through + `openhcs_ui_restore_snapshot`/`openhcs_ui_time_travel_head`. Execute the tiny + synthetic historical/current callable cases and require their discriminating + results. Preserve genuine missing-occurrence/history rejection controls. +5. Verify exact old-process exit and new-process incarnation. Retain a complete + recovery capture and ordinary closure of the owned replacement only after + receiving has finished. A code export alone never certifies full history. + +Existing source controls to reuse after an actual owner repair are +`test_durable_decorated_history.py`, +`test_pipeline_history_occurrence_restart_regression.py`, +`test_desktop_restart.py` and `test_desktop_history_upgrade.py` under +`tests/unit/pyqt_gui`. Run only affected existing controls; their passing cannot +replace this installed live sequence. No new parallel history serializer, +permissive pickle fallback, history filtering or process-restart policy belongs +in the repair. + +## Final acceptance and merge + +The production consumers of any follow-up change are named above: the actual +MCP diagnostic/public request path, physical Java workspace ingress and ROI +writer/readers, and the installed desktop restart/history flow. This document +adds no runtime behavior or persisted format. Any future format change must +declare its external/internal status and migration under the repository rules. + +Attach each determining packet to this draft with source/import identities, +exact admitted requests, original failures/uncertainties, received pixel/geometry/ +history observables and identity-proved teardown. Retain unresolved obligations +as unresolved; do not interpret one lane's success as all three passing. Sync +current main before concrete implementation and again before qualification. +Mark ready and merge only when all three original scopes have final acceptance, +or split any genuinely independent completion into its own closing-linked PR +while this draft continues to own the remainder. No original issue closes from +this plan-only checkpoint. From bcde628d019a63363b26ebc5cf540939b8740a80 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 5 Oct 2026 17:29:48 -0400 Subject: [PATCH 2/5] Fix MCP diagnostic natural retention and observed resource accounting --- openhcs/mcp/memory_diagnostic.py | 103 +++++++++++++----- .../setup_acceptance_followup_131_132_169.md | 20 ++-- tests/unit/test_mcp_memory_diagnostic.py | 47 ++++++++ 3 files changed, 134 insertions(+), 36 deletions(-) diff --git a/openhcs/mcp/memory_diagnostic.py b/openhcs/mcp/memory_diagnostic.py index d51f3d8ccb..54acac31bb 100644 --- a/openhcs/mcp/memory_diagnostic.py +++ b/openhcs/mcp/memory_diagnostic.py @@ -12,6 +12,7 @@ import asyncio import gc import json +import math import os import sys import tempfile @@ -73,17 +74,6 @@ def require_authority(self, authority: OpenHCSRuntimeImportAuthority) -> None: raise RuntimeError("Diagnostic child imported a different OpenHCS source") -def require_ram_headroom() -> None: - """Refuse further requests under the batch's 8 GiB available-RAM floor.""" - for line in Path("/proc/meminfo").read_text().splitlines(): - if line.startswith("MemAvailable:"): - available_kib = int(line.split()[1]) - if available_kib < 8 * 1024**2: - raise RuntimeError(f"Available RAM below 8 GiB: {available_kib} KiB") - return - raise RuntimeError("Linux MemAvailable accounting is unavailable") - - @dataclass(frozen=True) class RetentionMetric: """A sample-field declaration owning its projection and slope calculation.""" @@ -100,10 +90,13 @@ def slope(self, samples: Sequence[ProcessMemoryReceipt]) -> float: (index - center) * (value - mean) for index, value in enumerate(values) ) / sum((index - center) ** 2 for index in range(len(values))) + def change(self, before: ProcessMemoryReceipt, after: ProcessMemoryReceipt) -> int: + return self.project(after) - self.project(before) + @dataclass(frozen=True) class ProcessMemoryReceipt: - """Linux process accounting, not host swap or execution-tree accounting.""" + """Process accounting plus contemporaneous host telemetry, not a quota.""" pid: int monotonic_seconds: float @@ -124,6 +117,11 @@ class ProcessMemoryReceipt: history_snapshots: int imported_source_paths: tuple[str, ...] source_identity: DiagnosticSourceIdentity + host_mem_available_kib: int + host_swap_used_kib: int + host_full_psi_avg10: float + host_full_psi_avg60: float + host_full_psi_avg300: float @classmethod def from_payload(cls, payload: Mapping[str, object]) -> ProcessMemoryReceipt: @@ -142,6 +140,18 @@ def retention_slopes( if isinstance(metric, RetentionMetric) } + @classmethod + def retention_changes( + cls, before: ProcessMemoryReceipt, after: ProcessMemoryReceipt + ) -> dict[str, int]: + """GC sensitivity uses the same declaration-owned metrics as retention.""" + return { + name: metric.change(before, after) + for name, annotation in get_type_hints(cls, include_extras=True).items() + for metric in get_args(annotation)[1:] + if isinstance(metric, RetentionMetric) + } + @classmethod def capture(cls, *, collect: bool) -> ProcessMemoryReceipt: from arraybridge import MemoryType @@ -154,6 +164,16 @@ def capture(cls, *, collect: bool) -> ProcessMemoryReceipt: name, separator, value = line.partition(":") if separator: accounting[name] = int(value.split()[0]) + host = { + name: int(value.split()[0]) + for line in Path("/proc/meminfo").read_text().splitlines() + for name, value in (line.split(":", 1),) + } + full_pressure = next( + line for line in Path("/proc/pressure/memory").read_text().splitlines() + if line.startswith("full ") + ) + pressure = dict(part.split("=", 1) for part in full_pressure.split()[1:]) libraries = tuple( sorted( { @@ -202,6 +222,11 @@ def capture(cls, *, collect: bool) -> ProcessMemoryReceipt: if name in modules ), source_identity=DiagnosticSourceIdentity.capture(), + host_mem_available_kib=host["MemAvailable"], + host_swap_used_kib=host["SwapTotal"] - host["SwapFree"], + host_full_psi_avg10=float(pressure["avg10"]), + host_full_psi_avg60=float(pressure["avg60"]), + host_full_psi_avg300=float(pressure["avg300"]), ) @@ -222,20 +247,31 @@ class DiagnosticFailure: class MemoryDiagnosticReport: rounds: int request_sequence: tuple[McpDevToolCall, ...] + round_interval_seconds: float = 0.0 + gc_at_end: bool = False events: list[MemorySampleEvent] = field(default_factory=list) health: McpServerHealthResult | None = None post_warmup_slopes_kib_per_round: dict[str, float] = field(default_factory=dict) + post_gc_change_kib: dict[str, int] = field(default_factory=dict) failure: DiagnosticFailure | None = None server_stderr_tail: str | None = None scope: str = ( "one disposable real MCP process; no execution tree or host tmpfs attribution" ) interpretation: str = ( - "Slopes describe the recorded bounded request sequence only. Growth is not proof of a leak; " + "Slopes describe natural, pre-GC repeated-round samples only; optional final GC change " + "is separate sensitivity evidence, not the retention slope. Host telemetry is observational, " + "not admission or an automatic stop. Growth is not proof of a leak; " "flatness does not exonerate untested catalog, compile, custom-source, execution, JVM or GPU paths. " "Compare import/library/cache counts and repeat longer before attributing retention." ) + def __post_init__(self) -> None: + if self.rounds < 2: + raise ValueError("A retention slope requires at least two repeated rounds.") + if not math.isfinite(self.round_interval_seconds) or self.round_interval_seconds < 0: + raise ValueError("Round interval must be finite and nonnegative.") + def write(self, output: Path) -> None: output.write_text(json.dumps(asdict(self), indent=2) + "\n") @@ -253,10 +289,13 @@ class MemoryDiagnosticMcpClient: session: McpDevStdioSession timeout_seconds: float + def __post_init__(self) -> None: + if not math.isfinite(self.timeout_seconds) or self.timeout_seconds <= 0: + raise ValueError("Per-call timeout must be finite and positive.") + async def call(self, name: str, arguments: dict) -> McpDevToolResult: from openhcs.mcp.dev_client_core import McpDevToolResult - require_ram_headroom() wire = await self.session.call_tool( name, arguments, timeout_seconds=self.timeout_seconds ) @@ -280,8 +319,6 @@ async def sample(self, *, collect: bool) -> ProcessMemoryReceipt: receipt.source_identity.require_authority( OpenHCSRuntimeImportAuthority.current() ) - if receipt.rss_kib > 2 * 1024**2: - raise RuntimeError("Diagnostic MCP process exceeded its 2 GiB RSS budget") return receipt @@ -329,6 +366,8 @@ async def diagnose( surface_profile: LocalCapabilitySurfaceProfile, scratch_root: Path = SCRATCH_ROOT, sequence_path: Path | None = None, + round_interval_seconds: float = 0.0, + gc_at_end: bool = False, ) -> MemoryDiagnosticReport: from openhcs.agent.capabilities import agent_capabilities from openhcs.mcp.dev_client_core import ( @@ -351,12 +390,14 @@ async def diagnose( ) ) - report = MemoryDiagnosticReport(rounds=rounds, request_sequence=sequence) + report = MemoryDiagnosticReport( + rounds=rounds, request_sequence=sequence, + round_interval_seconds=round_interval_seconds, gc_at_end=gc_at_end, + ) output.parent.mkdir(parents=True, exist_ok=True) # stderr is bounded to this owned temporary stream; it is never copied into # a shared environment or another process's log. - require_ram_headroom() scratch_root.mkdir(parents=True, exist_ok=True) with ( tempfile.TemporaryDirectory(prefix="mcp-", dir=scratch_root) as scratch, @@ -369,8 +410,8 @@ async def diagnose( ) try: async with McpDevStdioSession(spec, stderr) as session: - await session.initialize(timeout_seconds=timeout) client = MemoryDiagnosticMcpClient(session, timeout) + await session.initialize(timeout_seconds=client.timeout_seconds) health = await client.health() report.health = health if ( @@ -389,26 +430,33 @@ async def sample(label: str, collect: bool) -> ProcessMemoryReceipt: report.record(label, collect, receipt, output) return receipt - await sample("baseline", True) + await sample("baseline", False) # These are read-only discovery calls. The first round includes # warm-up; subsequent rounds isolate repeated-request retention. for request in sequence: request.require_surface_profile(spec.surface_profile) repeated: list[ProcessMemoryReceipt] = [] for round_index in range(rounds + 1): + if round_index: + await asyncio.sleep(report.round_interval_seconds) for request in sequence: await sample( f"round-{round_index}/before/{request.name}", False ) await client.call(request.name, request.arguments) await sample(f"round-{round_index}/after/{request.name}", False) - await sample(f"round-{round_index}/gc/{request.name}", True) - end = await sample(f"round-{round_index}/end", True) + end = await sample(f"round-{round_index}/end", False) if round_index: repeated.append(end) report.post_warmup_slopes_kib_per_round = ( ProcessMemoryReceipt.retention_slopes(repeated) ) + report.write(output) + if report.gc_at_end: + after_gc = await sample("final/gc", True) + report.post_gc_change_kib = ProcessMemoryReceipt.retention_changes( + repeated[-1], after_gc + ) except BaseException as error: report.failure = DiagnosticFailure(type(error).__name__, str(error)) stderr.seek(0, 2) @@ -435,6 +483,8 @@ def main() -> None: ) parser.add_argument("--rounds", type=int, default=4) parser.add_argument("--timeout", type=float, default=30) + parser.add_argument("--round-interval-seconds", type=float, default=0.0) + parser.add_argument("--gc-at-end", action="store_true") parser.add_argument("--output", type=Path, default=Path("mcp-memory-receipt.json")) parser.add_argument("--scratch-root", type=Path, default=SCRATCH_ROOT) parser.add_argument( @@ -459,19 +509,16 @@ def main() -> None: if args.server: serve(surface_profile) return - if not 2 <= args.rounds <= 10 or not 0 < args.timeout <= 60: - parser.error("rounds must be 2..10 and timeout must be >0..60 seconds") report = asyncio.run( - asyncio.wait_for( - diagnose( + diagnose( args.rounds, args.timeout, args.output, surface_profile=surface_profile, scratch_root=args.scratch_root, sequence_path=args.sequence_json, - ), - timeout=180, + round_interval_seconds=args.round_interval_seconds, + gc_at_end=args.gc_at_end, ) ) print( diff --git a/plans/setup_acceptance_followup_131_132_169.md b/plans/setup_acceptance_followup_131_132_169.md index b44a28ebaf..2bde3a3a48 100644 --- a/plans/setup_acceptance_followup_131_132_169.md +++ b/plans/setup_acceptance_followup_131_132_169.md @@ -44,8 +44,8 @@ admission and exact PID-plus-creation-time process ownership; an uncertain tool outcome is not permission to retry, replace a process or replay science. Run receiving lanes serially after their actual resource/desktop owner hands off -the slot. Preserve the original memory, CPU and swap bounds; do not raise a cap -to obtain a pass. Keep startup/import/library preparation separate from repeated +the slot. Observe actual host headroom, pressure and process usage; do not impose +invented memory or swap caps. Keep startup/import/library preparation separate from repeated request growth and image/export work. Cleanup affects only identity-proved owned processes and disposable generated inputs after retaining final evidence. @@ -64,7 +64,9 @@ The existing sampler is `MemoryDiagnosticMcpClient` owns typed tool results and samples. Extend these owners rather than adding a second launcher, observer, memory schema or loop. -The current CLI has a 180-second overall limit and 2..10 repeat rounds. +The diagnostic now records host availability, swap usage and full PSI alongside +process accounting, without an 8-GiB admission floor or a 2-GiB process veto. +Finite caller-selected rounds and cadence replace the 180-second overall quota. `read_request_sequence` admits only declared read-only calls; artifact-plan inspection and custom registration cannot be smuggled through it. A current short diagnostic command, after normal installed-source admission, is: @@ -72,18 +74,18 @@ short diagnostic command, after normal installed-source admission, is: ```sh "$RECEIVING_PYTHON" -m openhcs.mcp.memory_diagnostic --source-identity "$RECEIVING_PYTHON" -m openhcs.mcp.memory_diagnostic \ - --rounds 10 --timeout 30 --sequence-json "$READ_ONLY_SEQUENCE" \ + --rounds 10 --round-interval-seconds 5 --timeout 30 \ + --sequence-json "$READ_ONLY_SEQUENCE" \ --scratch-root "$RECEIVING_ROOT/mcp-scratch" \ --output "$RECEIVING_ROOT/read-only-memory.json" ``` These commands do not provide the missing long-duration acceptance. Before that -run, make the existing diagnostic's duration/cadence and retained event budget -explicit on its current owner, keeping finite total duration and sample bounds. +run, select finite rounds/cadence appropriate to actual resource observations. Use one continuing child for at least the original 38-minute observation window, with separate import/warmup, repeated operations and idle-release phases. The current per-call deadlines stay bounded; elapsed duration is not an increased -tool timeout. Persist typed samples before a failure/ceiling terminates the run. +tool timeout. Persist typed samples before any failed operation terminates the run. For declared mutations, use the existing client's normal public tool-call path with explicit admission of the owned synthetic source and destinations; retain @@ -97,7 +99,9 @@ receipt remains unchanged and is not replayed. Sample MCP RSS/PSS/private dirty/swap, threads/modules, declared catalog/custom state and ObjectState snapshots before/after operations. Record natural release -and optional GC observations separately; never restart the measured child or +and optional final `--gc-at-end` observations separately. Natural end-of-round +samples determine slopes; the final GC delta is separate sensitivity evidence. +Never restart the measured child or force collection to present a flat natural trace. Attribute any owned native worker/viewer/JVM separately through its actual process incarnation. The saved short #575 native job need not be rerun to pad this MCP acceptance. diff --git a/tests/unit/test_mcp_memory_diagnostic.py b/tests/unit/test_mcp_memory_diagnostic.py index 851a746845..dee98ba19a 100644 --- a/tests/unit/test_mcp_memory_diagnostic.py +++ b/tests/unit/test_mcp_memory_diagnostic.py @@ -15,6 +15,8 @@ import openhcs from openhcs.mcp.memory_diagnostic import ( DiagnosticSourceIdentity, + MemoryDiagnosticReport, + MemoryDiagnosticMcpClient, ProcessMemoryReceipt, RetentionMetric, read_request_sequence, @@ -35,6 +37,11 @@ def test_process_receipt_uses_current_process_and_reports_gc_boundary(): assert after.swap_kib >= 0 assert after.threads >= 1 assert after.monotonic_seconds >= before.monotonic_seconds + assert before.host_mem_available_kib > 0 + assert before.host_swap_used_kib >= 0 + assert before.host_full_psi_avg10 >= 0 + assert before.host_full_psi_avg60 >= 0 + assert before.host_full_psi_avg300 >= 0 def test_slope_distinguishes_warmup_from_repeated_request_growth(): @@ -74,6 +81,46 @@ class ExtendedReceipt(ProcessMemoryReceipt): assert "private_clean_kib" not in ProcessMemoryReceipt.retention_slopes( [baseline] * 3 ) + assert ExtendedReceipt.retention_changes(repeated[0], repeated[-1])["private_clean_kib"] == 10 + assert "host_mem_available_kib" not in slopes + + +def test_natural_retention_and_gc_sensitivity_are_separate(): + baseline = ProcessMemoryReceipt.capture(collect=False) + natural = [replace(baseline, rss_kib=value) for value in (100, 130, 160)] + collected = replace(natural[-1], rss_kib=120) + report = MemoryDiagnosticReport(3, (), round_interval_seconds=75, gc_at_end=True) + report.post_warmup_slopes_kib_per_round = ProcessMemoryReceipt.retention_slopes(natural) + report.post_gc_change_kib = ProcessMemoryReceipt.retention_changes(natural[-1], collected) + assert report.post_warmup_slopes_kib_per_round["rss_kib"] == 30 + assert report.post_gc_change_kib["rss_kib"] == -40 + assert MemoryDiagnosticReport(100, ()).rounds == 100 + with pytest.raises(ValueError, match="at least two"): + MemoryDiagnosticReport(1, ()) + for interval in (-1, float("inf"), float("nan")): + with pytest.raises(ValueError, match="finite and nonnegative"): + MemoryDiagnosticReport(2, (), round_interval_seconds=interval) + for timeout in (0, -1, float("inf"), float("nan")): + with pytest.raises(ValueError, match="finite and positive"): + MemoryDiagnosticMcpClient(None, timeout) + + +def test_large_process_low_host_memory_sample_is_observation_not_veto(): + import asyncio + from openhcs.mcp.dev_client_core import McpDevToolResult + + sample = replace(ProcessMemoryReceipt.capture(collect=False), + rss_kib=4 * 1024**2, host_mem_available_kib=512 * 1024) + + class RecordedClient(MemoryDiagnosticMcpClient): + async def call(self, name, arguments): + assert arguments == {"collect": False} + return McpDevToolResult.from_payload(name, { + "content": [{"type": "text", "text": json.dumps(asdict(sample))}], + "isError": False, + }) + + assert asyncio.run(RecordedClient(None, 30).sample(collect=False)) == sample def test_request_sequence_decodes_existing_owner_and_rejects_mutations(tmp_path): From 283c2378a01420e119a6dc4c6a404892907073fa Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 5 Oct 2026 17:34:54 -0400 Subject: [PATCH 3/5] Consume declared MCP health result without a second decode --- openhcs/mcp/memory_diagnostic.py | 5 ++++- plans/setup_acceptance_followup_131_132_169.md | 15 +++++++-------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/openhcs/mcp/memory_diagnostic.py b/openhcs/mcp/memory_diagnostic.py index 54acac31bb..bc15d0317b 100644 --- a/openhcs/mcp/memory_diagnostic.py +++ b/openhcs/mcp/memory_diagnostic.py @@ -311,7 +311,10 @@ async def health(self) -> McpServerHealthResult: from openhcs.agent.dto.mcp import McpServerHealthResult result = await self.call(agent_capabilities.health_check.name, {}) - return dataclass_from_mapping(McpServerHealthResult, result.payloads[0]) + health = result.decoded_payload_as(McpServerHealthResult) + if health is None: + raise RuntimeError("Diagnostic health returned no declared health result") + return health async def sample(self, *, collect: bool) -> ProcessMemoryReceipt: result = await self.call(PROBE_TOOL, {"collect": collect}) diff --git a/plans/setup_acceptance_followup_131_132_169.md b/plans/setup_acceptance_followup_131_132_169.md index 2bde3a3a48..db5953a5f0 100644 --- a/plans/setup_acceptance_followup_131_132_169.md +++ b/plans/setup_acceptance_followup_131_132_169.md @@ -1,9 +1,8 @@ # Complete the original setup and receiving obligations: #131, #132, #169 -This draft owns the unfinished acceptance of all three issues. It is not ready -to merge. The linked source fixes are already merged; they do not establish the +This document retains the unfinished acceptance of all three issues. The +qualified diagnostic repair may ship independently; it does not establish the remaining long-lived MCP, real Java/CZI, or continuous desktop receiving results. -No new run, environment change, or performance result accompanies this plan. Keep each issue open until its section below has a determining final receipt. ## Existing implementation and evidence @@ -246,7 +245,7 @@ in the repair. The production consumers of any follow-up change are named above: the actual MCP diagnostic/public request path, physical Java workspace ingress and ROI writer/readers, and the installed desktop restart/history flow. This document -adds no runtime behavior or persisted format. Any future format change must +records those acceptance boundaries separately from the diagnostic repair. Any future format change must declare its external/internal status and migration under the repository rules. Attach each determining packet to this draft with source/import identities, @@ -254,7 +253,7 @@ exact admitted requests, original failures/uncertainties, received pixel/geometr history observables and identity-proved teardown. Retain unresolved obligations as unresolved; do not interpret one lane's success as all three passing. Sync current main before concrete implementation and again before qualification. -Mark ready and merge only when all three original scopes have final acceptance, -or split any genuinely independent completion into its own closing-linked PR -while this draft continues to own the remainder. No original issue closes from -this plan-only checkpoint. +Merge a qualified useful diagnostic repair without waiting on unrelated CZI or +desktop acceptance. Close an original issue only when its determining scope has +final acceptance; a plan, short diagnostic or another lane's success is not that +evidence. All unresolved obligations remain with their named owners. From e1ba99484cdfde361eea41d65cbe4f367209d6b4 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 5 Oct 2026 17:41:15 -0400 Subject: [PATCH 4/5] Record installed natural retention diagnostic acceptance and preserved failures --- .../ACTUAL-RECEIVING.rst | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst diff --git a/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst b/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst new file mode 100644 index 0000000000..94be54753f --- /dev/null +++ b/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst @@ -0,0 +1,103 @@ +MCP diagnostic repair: issue131 / PR654 +====================================== + +Delivery scope +-------------- + +The existing diagnostic no longer vetoes calls at an invented 8-GiB host floor, +2-GiB process RSS ceiling, ten-round maximum or 180-second overall quota. +Caller-selected finite rounds/cadence and per-call timeout retain a bounded +workload. Linux availability, swap usage and full PSI are observations on +ProcessMemoryReceipt, not a second admission policy or automatic stop. + +Natural end-of-round samples determine the retention slopes. Python's ordinary +automatic GC remains enabled; no explicit GC precedes or interrupts these +samples. Optional final --gc-at-end records separate signed sensitivity deltas. +RetentionMetric annotations own both calculations, including inherited/new +metrics. The strict read-only request admission remains unchanged. Declared +health results are consumed through McpDevToolResult.decoded_payload_as, not +decoded a second time as mappings. + +Source and ownership +-------------------- + +Production pin: 283c2378a, following initial checkpoint bcde628d0. Latest main +49322f10b was normally integrated afterward; neither the diagnostic module nor +its inherited launch module changed. The installed target is honestly pinned to +283c2378a, not relabelled as later main. Foreign gitlinks/keepers were untouched. + +Existing audit Package/Repository AST: 710 MCP/runtime/unit modules, zero parse +omissions; related declarations, imports, inherited launch and calls recorded. +Git caller closure names only the existing diagnostic, launch, ownership guard +and unit family as active consumers. Literal Linux accounting is decoded once +inside ProcessMemoryReceipt.capture (BOUND-1/BOUND-2). DiagnosticServerSpec keeps +the original McpDevServerSpec/import-authority lifecycle (IMPL-13); no replacement +launcher, metric roster, codec, quota or runtime store was added. + +Original controls and failures +------------------------------ + +ownership02.log: existing focused AST ownership guard PASS. +controls02.log/time: 13 PASS, 6.76 seconds, peak323940KiB, swaps0. Includes inherited +metric eligibility, strict unknown/bool rejection, read-only mutation rejection, +exact-source child launch, separate natural/GC calculations and acceptance of a +synthetic4-GiB RSS sample with512-MiB available host memory as observation rather +than a veto. Those last figures are a controlled sample, not actual host load. + +controls01 failed before collection because repository pytest addopts requested +an unavailable tests plugin. The corrected invocation selected the isolated +installed target and explicit pytest configuration; two cache warnings were +nonfatal. No production guard or assertion was weakened. + +live01 failed before any memory sample: the diagnostic's old health reader tried +dataclass_from_mapping on an already-typed result. Original client1720561 and +MCP1720693 exited; JSON/log/time remain unchanged. The owning consumer was fixed +before the explicitly new live02 attempt. No UNKNOWN input was replayed. + +Ordinary installation and real MCP +---------------------------------- + +Private receiving02 ordinary wheel/build/install uses the retained interpreter +and five qualified dependency wheels, no downloads/new environment/shared +installation changes. Build01 terminal0,139.46seconds, peak443660KiB, swaps0. +BYTE-QUALIFICATION.json PASS:920 wheel RECORD entries,817 exact tracked source +files,792 Python files,13 packaged skill files and complete installed RECORD +accounting. Existing ordinary CLI synchronized only the private skills directory. +Wheel SHA256:b967b610dfa8c51f2880c4766505fb07f4ed77a393ebc6898e828dfb387b01bb. + +live02 used the real installed diagnostic CLI, DiagnosticServerSpec and MCP +transport: health, first_use and capability search, one warm-up plus three +repeated rounds at0.2-second cadence, followed by one explicit final GC. +Original tool handle94550 terminal0;18.95seconds, time-reported peak396100KiB, +swaps0. All30 typed samples came from MCP1737168 and the exact receiving02 +import root. The original session context closed it; /proc1737168 is absent. +No diagnostic client remains, and no native/viewer/science process was adopted. + +Natural end RSS:396696,397128,397444KiB. Declaration-derived RSS/PSS/private-dirty +slopes:374KiB/round; process swap slope0. Final explicit GC delta0 for each +declared metric. Minimum observed host availability9190232KiB; maximum full PSI +avg10=1.17. These measurements were recorded, not converted into an invented +pressure cutoff. LIVE-ACCEPTANCE02.json verifies exact natural/GC separation, +source/PID identity, declared calculations and actual process exit. + +Remaining original acceptance +----------------------------- + +This useful tool repair is not a no-leak finding, a causal explanation of the +stable P0013.78-GB MCP sample, or completion of the original38-minute mixed +request/revision workload. A new synthetic registration must use the ordinary +typed registration path with an actually owned execution endpoint and prepared +native catalog; none was borrowed or launched for this read-only qualification. +No source revision was submitted. The old UNCERTAIN revision remains immutable +and must never be replayed. Planck retains #131 follow-through; Dewey owns any +future actual lane custody. #132/#169 remain separate unfinished obligations. + +Original evidence root +---------------------- + +/home/ts/wt/openhcs-issue-batch-20260929/engineering-mcp-retention-131-20261005 + +Keep live01/live02 JSON/log/time, controls01/02, AST/caller closure and both +receiving source origins/wheel proofs. receiving19 and every scientific frozen +installation are unchanged. This receipt is source qualification plus the +specified installed diagnostic path, not biological or whole-session acceptance. From d0e0475c63292474324be56abce7a22ade2ff3f8 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 5 Oct 2026 17:43:16 -0400 Subject: [PATCH 5/5] Record final current-main integration without relabelling installed proof --- .../mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst b/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst index 94be54753f..1e2d97a9e0 100644 --- a/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst +++ b/docs/validation/mcp-natural-retention-131-20261005/ACTUAL-RECEIVING.rst @@ -22,8 +22,9 @@ Source and ownership -------------------- Production pin: 283c2378a, following initial checkpoint bcde628d0. Latest main -49322f10b was normally integrated afterward; neither the diagnostic module nor -its inherited launch module changed. The installed target is honestly pinned to +49322f10b was normally integrated afterward, then main4d7f18858 (paper/review +changes only); neither the diagnostic module nor its inherited launch module +changed. The installed target is honestly pinned to 283c2378a, not relabelled as later main. Foreign gitlinks/keepers were untouched. Existing audit Package/Repository AST: 710 MCP/runtime/unit modules, zero parse