Repository navigation
Investigate suspected retained memory in long-lived MCP process; separate from host tmpfs swap #131
Description
Summary
Please investigate memory retained by a long-lived OpenHCS MCP process. On one development host, the active MCP process held about 999 MiB resident plus 461 MiB swapped after roughly 38 minutes of mixed agent use. This is a suspected retention issue, not a confirmed leak or a clean reproduction. A short follow-up sample was flat.
The host's nearly full swap is not evidence that OpenHCS alone consumed it. Separate RAM-backed temporary files account for much of the host-wide pressure, and an older OpenHCS execution server is a different process from the MCP servers.
Environment and observations (2026-09-26 UTC)
- OpenHCS
0.8.6, Python3.12.3, Linux; local checkout based on2bc579ca9with uncommitted changes. These measurements are diagnostic only, not a clean-release regression. - Host: 31 GiB RAM, 32 GiB swap. At sampling, approximately 8 GiB RAM was available while swap was almost full.
- Three distinct MCP processes were present:
| Process | Approx. uptime | RSS | Swap (smaps_rollup) |
Other evidence |
|---|---|---|---|---|
Active python -m openhcs.mcp |
38 min | 999 MiB | 461 MiB | PSS 990 MiB; PSS anonymous 832 MiB; PSS shared-memory 18 MiB; 110 threads |
openhcs-mcp --surface desktop instance A |
5.6 h | 10 MiB | 206 MiB | 34 threads |
openhcs-mcp --surface desktop instance B |
5.2 h | 63 MiB | 153 MiB | 34 threads |
- The active process's memory was predominantly private (
Private_Dirtyabout 853 MiB,Private_Cleanabout 135 MiB). None of these three MCP processes had an open/dev/shmfile descriptor or a/dev/shmmapping in/proc/<pid>/mapsat sampling. - An execution server, not an MCP server, had about 4.2 GiB in swap after 2.5 days of uptime. Please measure and triage it separately rather than attributing it to the MCP process.
/tmpheld about 12 GiB and/dev/shmabout 10 GiB, predominantly unrelated local test workspaces. Process-reportedVmSwapsummed to only about 12.7 GiB while the host used nearly 32 GiB of swap. Swapped tmpfs pages plausibly explain much of the gap; file-level attribution has not been proved. Do not treat/dev/shmcapacity as an MCP allocation.- Known recent operations against the active MCP included health, function-catalog/knowledge queries, pipeline artifact-plan inspection, and custom-function registration. The full request history and clean startup baseline were not captured. The RSS/swap values stayed essentially flat over a short follow-up sample; there is no measured growth slope yet.
Suggested investigation
- Reproduce in a clean, isolated MCP process with a captured baseline and bounded request sequence; record RSS, PSS, private dirty, swap, and thread count before and after each operation and after idle/GC. Keep the execution server and tmpfs accounting separate.
- Distinguish intended import/native-library/JVM/GPU footprint and bounded catalog caches from request-scoped objects retained across calls. Identify the exact owner before calling this a leak.
- If retained growth is reproducible, add a focused regression test or memory budget for the responsible path, including repeat calls and source-revision changes where relevant.
No services or temporary files should be removed merely to make this test pass; the shared host's swap saturation should be tracked as a separate environment condition.
Activity
trissim commentedon Sep 29, 2026
Implementation owner: Linnaeus, coordinated with the paired #131/#169 memory/session-durability work. An isolated branch is active under
/home/ts/wt; a draft implementation PR will be linked after the first working checkpoint. Suspected retention remains unproven until a bounded real-MCP measurement distinguishes request growth from baseline import/cache footprint. Integration owner is the OpenHCS coordinator.Merge pull request #208 from OpenHCSDev/fix/memory-session-20260929
trissim commentedon Sep 30, 2026
Installed measurement checkpoint after merged PR208: current main65b2ed101 plus merged ArrayBridge409 installed offline/serially in existing Python3.12 env; 16 installed diagnostic/import/history checks pass5.47s without source overrides/ABI injection. Real disposable installed MCP child3156266 completed51events in16.41s: one warm-up plus4 repeated health/first-use-guidance/capability-search rounds. RSS307360KiB and privateDirty194724KiB flat after warm-up, own swap0, slopes RSS/privateDirty/swap0 and PSS-4.8KiB per round. Four threads/2516 modules stable; NumPy+CuPy already imported (no GPU workload). Catalog/custom/history counts0: NOT evidence those routes, native execution, viewer/UI, JVM or GPU are leak-free, and not host-swap attribution. Child is confirmed gone and validation lock free. Complete1.6MiB JSON receipt and installed-test XML retained in persistent parent ledger. Issue remains open for route-specific retention measurements; actual installed entrypoint is now usable for those, not merely a source-only diagnostic.
trissim commentedon Oct 3, 2026
New route-specific evidence, not a claim the original MCP-query leak is reproduced: BBBC013 native full-plate execution on qualified target09 hit its original 5 GiB MCP/native/viewer FAMILY cgroup limit at 2026-10-03 23:45:41 UTC. Kernel CONSTRAINT_MEMCG killed native execution PID2815845 (identity joined to preserved execution-status receipt), RSS
2.87GiB/anonymous2873396KiB. Other children had RSS1.64/.69/.25GiB; role joins pending, RSS is not exclusive charge and must not be summed as allocations. 49/96 fields left tables and label files, no terminal native success or plate statistics. All full-run streaming was already disabled; prior QA viewer buffers still existed. Original failures/UNKNOWNs remain frozen, no replay/cap increase. Existing source candidate: default consolidation selects full-parent observations, which retain all artifact records; per-axis image-cache release is separate from RuntimeValueStore release. This is not a proved heap cause. Parent owns scoped triage; Root394 owns shared runtime/materialization fixes, Planck traces actual consumers and Dalton joins preserved child roles. Acceptance needs per-owner memory attribution and bounded complete native plate execution through the original public path, distinct from the already-flat MCP health/query rounds. Evidence: /home/ts/wt/openhcs-issue-batch-20260929/PARENT-BBBC013-49FIELD-OOM-20261003.rst and original run attempt22/frozen receipts. No duplicate implementation or foreign worktree mutation.trissim commentedon Oct 4, 2026
Ownership update at Tristan’s request: a dedicated context-forked implementation worker, inherited from Planck’s relevant runtime/consumer history, is now assigned to fix this issue through bounded real-MCP/native verification and merge. The coordinator stops overlapping memory investigation; Planck continues #554, Singer continues #567, and Root394 retains shared runtime/materialization ownership. The worker reuses the finished isolated checkout, reads current NRA/refactor-audit/antipattern guidance, preserves original failures and scientific evidence, and will open an implementation draft promptly and link it here. Hosted CI is not a merge hold; source tests alone do not establish the memory fix.
trissim commentedon Oct 4, 2026
The verified native consolidation fix is merged in #575 (32ededf). Consolidation now receives its existing typed table/scope/destination projection instead of retaining all image/label records. Independently declared PLATE inputs and explicit full-value requests preserve their own retention requirements.
Actual ordinary installed controls:147 passed across preserved141PASS/6 authored-fixture failures then6 correctedPASS. Public native synthetic16fields/8wells completed; original ROI/CSV/consolidated-table readers and bounded pixel/calibration checks passed. Separate explicit VALUES job completed and original public measured-run finalizer validated its export. PLATE is installed-control coverage, not public PLATE execution here.
One4GiB/noSwap/oneCPU process family peaked1,787,359,232B; exact native public close and client/scope termination verified. Thirty mixed health/catalog/knowledge rounds and five artifact-plan reads are short observations, not a reproduction of the original38-minute growth. A distinct source revision returned UNCERTAIN and remains unreplayed. Catalog readiness failures/transient totals are retained for the existing catalog integration owner.
Keep #131 open for its original mixed-MCP/revision scope. No biological OOM heap cause is claimed. Original169-entry receiving freeze verified; persistent receipt is
/home/ts/wt/openhcs-issue-batch-20260929/issue131-memory/public94-attempt01/PUBLIC-ACCEPTANCE01.rst, with immutable stdin/stdout/timing, native logs, source/outputs and uncertainty disposition.Plan remaining MCP, CZI and desktop receiving acceptance (#131, #132, #…
trissim commentedon Oct 5, 2026
New read-only route attribution from ongoing personal-neurite mosaic development: the receiving18 MCP process (not its native execution server or viewer) at about17minutes uptime holds RSS3,791,212KiB/PSS3,578,970KiB/private dirty3,557,872KiB. Native server is separately about1.38GiB RSS and viewer about0.77GiB RSS. This is one measured point, not a growth slope or proof of a leak; no startup/restart/replay or GC was injected into the scientific author. Four blind authors are active; host available memory5.9GiB and memory PSI avg10~0.46 at sampling. Existing #654 acceptance owner remains the route; no competing implementation created. Scientific masks/pipeline and original journals remain unchanged.
trissim commentedon Oct 5, 2026
Diagnostic repair #654 merged and verified on remote main725fdd440; the original long mixed-request/cause obligation stays open. New read-only accounting from the SAME live P001 MCP PID1584480 at elapsed3023s: RSS2117744KiB, PSS1911567KiB, private dirty1888812KiB, process swap1845188KiB (SwapPss1844948KiB). Lower RSS is largely paging, not demonstrated release: resident private dirty plus process swap is3734000KiB (~3.56GiB). No GC, restart, heap mutation, science adoption or input replay occurred. Planck remains owner for source-grounded retention investigation; one short diagnostic or a single live sample does not establish a leak slope or cause.
trissim commentedon Oct 5, 2026
Concrete non-injecting attribution checkpoint: original PID1584480 cmdline selects receiving18/openhcs.mcp full; its maps include Fiji lib/server/libjvm.so inside the Python MCP. This confirms embedded Java runtime but NOT heap size/array leak. Parent3023s PrivateDirty+Swap=3,734,000KiB (~3.56GiB), Pss+SwapPss=3,756,515KiB (~3.58GiB); RSS drop is not reclaim and earlier swap is missing. Source: local PlateInspection sample and in-process artifact-plan compilation can enter BioFormatsJavaContext; that process singleton retains ImageJ gateway/classes until cleanup, while individual probe/plane readers close in finally/context managers. Ordinary ROI codecs use roifile. Current custom-source registry removes superseded same-name preparation outcomes; no revision leak established by its storage alone. No GC/attach/RPC/restart/adoption/replay/new runtime performed. Evidence/source checkpoint /home/ts/wt/openhcs-issue-batch-20260929/engineering-mcp-retention-131-20261005/CAUSAL-CHECKPOINT01.rst plus selective JVM mapping receipt. Planck continues causal investigation; no reader-close/JVM-shutdown workaround or leak claim justified yet.
trissim commentedon Oct 5, 2026
Meaningful new isolated checkpoint: synthetic03 originalhandle71874 terminal0/30.72s/934580KiBpeak/swap0, disposable MCP1935981 exited0/absent. Existing receiving20+paired dependencies/Fiji cache, downloads disabled; no science GC/restart/attach/RPC/replay. Public ordinary12x15 TIFF sample then rich OME2x3x12x15 inventory/sampling: PrivateDirty+Swap237160→294812→706704KiB; libjvm absent before rich ingress/present afterward, Pythonmodules3826 unchanged at rich transition. Cold rich-reader/JVM-associated delta411892KiB (~402MiB), not all proven Java heap. Four identical4x5samples711732/715736/719880/723400KiB; natural5s idle718284, no forcedGC. Exact6rich planes and original pixels verified; VERIFICATION PASS. This explains a substantial cold reader footprint, NOT original P0013734000KiB or leak growth. Evidence /home/ts/wt/openhcs-issue-batch-20260929/engineering-mcp-retention-131-20261005/SYNTHETIC-CHECKPOINT03.rst + synthetic03/receipt.json/VERIFICATION.json; original logs/source/inputs frozen. Continuing source cause at full-container discovery/compile and source namespaces; no speculative runtime patch/cap/shutdown workaround justified.
trissim commentedon Oct 6, 2026
Integration-owner update: Planck, the original causal investigation owner, is now actively resumed with retained context. Current open PRs959/160 do not implement this memory boundary. The existing cold JVM-associated delta (~402MiB) is not an explanation of the original ~3.56GiB private resident+swap footprint or proof of request growth. Next work is source-grounded retained-reference/lifecycle attribution through existing ownership/AST tools, then a coherent fix only if supported, published promptly as draft PR and locally/installed-path verified without waiting CI. No new scientific run, forced GC, arbitrary memory cap or restart workaround. Existing dependencies/Fiji reused; cleanup stays separately owned by Dalton.
trissim commentedon Oct 6, 2026
Current-main source finding (397b724; Planck retains implementation ownership): CustomFunctionRuntimeRegistry._preparation_outcomes permanently caches a Future for each current persisted source revision. prepare_source_once catches BaseException and stores the ORIGINAL exception with set_exception; subsequent callers invoke outcome.result on that same object. CustomFunctionManager._prepare_source executes into a local namespace and wraps execution errors with ValidationError from exc, so the cached traceback/cause keeps both preparation frames and source globals alive after a failed preparation. This is an actual retained-reference path, distinct from intentional globals belonging to a successfully published callable. A bounded stdlib mechanism check confirmed an otherwise unowned weak-reference payload remains alive and four repeated Future.result calls grow traceback depth 1 -> 4 -> 7 -> 10 -> 13. No OpenHCS/science request was replayed for this check.
This does NOT establish that original P001 PID1584480 exercised this failure path or attribute its 3734000KiB private+swap footprint to it. Cold BioFormats/JVM-associated ~402MiB remains separately accounted. Fresh source inspection also confirms bounded BioFormats sampling calls openBytes with x/y/width/height, returns that tile, and closes the request reader in finally; the process-owned JVM is intentionally reusable.
Next coherent owner work is preserving exactly-once source preparation/shared failure and original error diagnostics WITHOUT keeping original request frames or repeatedly re-raising one cached exception. Do not evict failed outcomes to trigger automatic source re-execution, strip valid successful callable globals, shut down the JVM, or force GC. Dewey: please confirm the existing integration/receiving28 checkout /home/ts/wt/openhcs-context-bounding-20260929 is released for this custom-source-owner branch before any Git/source mutation; installed receiving28/paired dependencies and scientific originals remain protected. No competing memory PR is present (current open PRs959/1032/160 are unrelated). Original #654/#131 diagnostic receipts remain immutable.
Merge pull request #1035 from OpenHCSDev/fix/custom-preparation-failu…
trissim commentedon Oct 6, 2026
The independent cached-failure retention repair shipped in PR1035 after 77 related installed controls and the real MCP source-session import path. Final source e6e5852; production bytes f97cdf4. Final AST closure: 35 related production modules, zero omissions; all three manager callers use the one preparation Future owner. Public case07 returned identical declared ValidationError diagnostics for four failed imports, source execution counter exactly1, followed by a successful session-5 from a separate valid source. Owned MCP3852349 exited0/absent, original client34735 terminal0; no compile/execution job, native/viewer, GC, Fiji startup, dependency download or science replay. Natural private-dirty-plus-swap213156->215308KiB is a short observation, NOT attribution of original P0013.56GiB.
Source receipt: docs/validation/mcp-natural-retention-131-20261005/FAILED-PREPARATION.rst. Retained exact wheel/target/RECORD proof, 77-control log, LIVE07 and original negative attempts: /run/media/ts/hdd/openhcs-cold-custody-20261003/engineering-mcp-retention-131-failure04. Only its terminal source/build31MiB and separate failure04-build scratch are released to Dalton; target/source outsidebuild, wheel, fixtures and all evidence remain protected. This frees HDD, not HOME. #131 stays OPEN: original large-process attribution remains unproved and Planck remains its causal investigation owner. Current science installations untouched.