Repository navigation
ZMQ execution server is OOM-killed during the frozen 30-case scientific qualification #433
Description
The frozen 30-case native scientific qualification on PR #394 revision 94915db1487c08db3f4053043edf77e5c3fc830c lost its original ZMQ server during cp_tutorial_3d_monolayer. This is a confirmed kernel OOM kill, not a status-timeout hypothesis.
At 2026-10-02 01:28:08 local, the kernel killed endpoint PID 2412356 with 13,360,004 KiB anonymous RSS and 38,079,356 KiB total VM. Earlier successful measured receipts identify the same endpoint PID and startup incarnation. Server logs contain no Python execution failure for the final job. The suite continued on replacement servers and completed 24 PASS / six failures; four other failures are canonical input-context annotation omissions, and one is a threshold entropy mismatch. These must not be conflated with this OOM.
The run used latest shared dependencies at that freeze, one inline worker, retained source-pinned native outputs, VALUES observation export and no memory observer. Server startup/preparation is excluded from pipeline clocks. This is scientific qualification, not a paired runtime experiment. Actual source pixels are 60×256×256 uint16 volumes, and completed receipts report one axis.
Evidence is retained under /home/ts/code/projects/openhcs-benchmark-runs: pr394-94915-all30-native-science-qualification-red-20261002.json, pr394-94915-all30-server-oom-evidence-20261002.log, and shared-runtime-plumbing-current-native-parity-source-20261002.json. Outputs are /var/tmp/openhcs-shared-runtime-plumbing-current-native-parity-20261002/.
Source inventory identifies existing owners to investigate: compile-artifact reuse, runtime observation/store lifetime, inline executor teardown, and memory-backend lifetime. The dominant allocation and retaining references were not captured live; retention is not yet established as the cause. Ordinary compile artifacts are consumed on reuse, and base execution cleanup already drops the orchestrator extra, so simply asserting these are the leak would be wrong.
Acceptance: capture bounded allocation/retaining-owner evidence on representative saved inputs or a controlled repeat sequence; fix the actual nominal owner; preserve observation export, plate publication, checkpoint/debug retention and pixel isolation; demonstrate stable repeated ready-server execution and strict scientific parity. Increasing timeouts or restarting every case is not the production fix. PR #394 owns this draft acceptance gate and will be formally linked.
This concerns the execution server. Existing #131 concerns suspected MCP-process memory retention and explicitly separates execution-server accounting; it is not the same reproduction.
Activity
Record prepared ABI owner gates and unresolved scientific qualificati…
fix(runtime): preserve whole-image domains across main-flow handoff
refactor(runtime): let input data own main-flow image composition
docs(perf): record opaque-domain repair and remaining operand projection
docs: qualify current native frontier and merged readiness repair
Keep publication and output identity on their nominal owners
docs: retain controlled same-server VALUES memory acceptance
trissim commentedon Oct 4, 2026
PR #394 now carries the completed controlled-repeat acceptance for this issue: retained evidence and exact limits.
All 16 public measured jobs completed on one READY server incarnation, with retained VALUES, original source universes and no restart, forced GC or process-cache clear. All 32 strict comparisons to the two native output sets passed. Sampled aggregate peak was 4002 MiB below the 6144 MiB guard; the final seven post-job server measurements plateaued between 2118 and 2210 MiB.
The actual production-query reproduction identifies and removes five process-wide cache families that retained label buffers after store clear and returned stale measurements after replacement. It does not identify the sole cause of the historical 13 GiB OOM. The original dominant allocation was not captured, and finite RSS sampling is not an infinite-run guarantee.
The issue's controlled-repeat acceptance clause is complete and #394 is formally linked to close it on merge. Full 30-case scientific qualification and catalog/scaling acceptance remain explicitly outstanding; they are not represented as passing by this closure.