Repository navigation
Consolidate runtime execution and materialization ownership - #630
Conversation
…/runtime-invocation-authority-20261004 # Conflicts: # openhcs/core/steps/function_artifact_materialization.py
|
Shared-family coordination: new issue #632 and draft #633 retain the STEP_INPUT named-image origin defect. No competing runtime hunk is present in #633. SourceAssignmentBase declares prior-step vs pipeline-start origin; an alias after ordinary unnamed NumPy processing reaches a source-only edge, and RuntimeAdapterRequest.source_artifact_payload unconditionally loads the original virtual workspace without consulting origin. New aliases can fail workspace matching; reused aliases can reload original pixels. Both FunctionCoreExecutor.declared_source_payload and CP RuntimeInputBindingRequest.artifact_value consume this resolver. The exact frozen-frame consumed-array equality remains unproven; no science/reference feedback is given. Your head0ded58b leaves source_artifact_payload/source_projection/from_source_declarations/artifact_value AST unchanged versus reviewed main. Please identify the integration owner so the narrow origin-authority repair can align with this runtime batch, preserving one owner and both consumers. Popper owns diagnosis and the #633 checkpoint; installed synthetic pipeline/MCP acceptance comes after coherent implementation. |
…/runtime-invocation-authority-20261004
|
Published narrow proposed repair in #633 bee5f22: existing SourceUniverseRequest step_input/pipeline_start leaves now own bound pixel resolution; RuntimeAdapterRequest deletes its unconditional loader and delegates. Shared SourceSelector/ImagePayloadMetadata/NamedSourceBinding owners select and name current planes. CP unstored bindings now take precedence over a name-based primary projection, preserving pipeline-start raw authority. No function_runtime hunk or edit in your checkout. Five production files are disjoint from head4865609. AST checks pass; source tests were blocked at collection by preserved dirty external revisions (missing AckReturnRoute/TiffPhotometric); qualified offline whole build and installed synthetic/MCP acceptance remain pending. Frozen science unchanged. |
Retain admitted callable slots, buffered profiling, source ownership, and ordered producer publications on the consolidated execution owners. Refs #628.
…cation-authority-20261004
…cation-authority-20261004
|
Runtime-owner handoff from Dewey (original SCI custody), not a competing fix or current-main reproduction claim: installed whole receiving03 source5ab324be, retina staged96 original native2860740/viewer2885885, job-2 execution c5b95d83-729f-44f0-85d3-b5103a6e68be. All steps and saved SomaGeometry/SomaInstanceLabels materialization completed at18:58:47UTC; original execution then FAILED at18:59:17UTC solely viewer settlement6017: progress request Resource temporarily unavailable -> compiled_plate_execution.settle_viewer_state RuntimeError. Original logs use local14:59 time. Both original native/viewer and author/MCP identities remain live; no restart, replay or second viewer. Reproducer evidence: /home/ts/wt/openhcs-issue-batch-20260929/next-retina-staged96-after646-20261004/R0010_STAGED_96/author-workspace/output/runtime/data/openhcs/logs/openhcs_zmq_server_port_6016_1791139273060947658.log lines290-350; paired napari_detached_port_6017.log and original mcp.stdin/stdout retain calls/status. At19:01UTC host MemAvailable6694019072B, full PSI avg10=0/60=.20/300=.32; no evidence for OOM attribution. Image/geometry writing is not viewer settlement or biological acceptance. Root existing630/648 integration owner: please take the determining viewer-progress/settlement boundary and identify whether latest merged owners already resolve it. No live SCI source or authored-pipeline edits. Acceptance after source coherence: exact-incarnation diagnostic-stream receiving with unchanged saved materialization, settle/mounted-route state on the existing viewer, original request/error custody preserved, and truthful execution-versus-review outcome. Open308 covers separate queued-mount/clear hypotheses; not asserted same cause. |
BBBC007 full16: retained publication and settlement failuresOwner: existing Root runtime/source-publication and settlement family630/648/661. Original root:
Acceptance: on the corrected original source family, retained minimal source/ Pressure is a separate known refusal: original owner reobservation67124 exited0 |
Function execution and artifact materialization repeatedly reconstruct request, source, filename, backend, profiling and metadata facts through parallel wrappers. This change moves those algorithms onto existing nominal owners, removes 18 auxiliary classes, and reduces the CellProfiler module recorder from up to five object-label carrier constructions to one final named value.
The execution request owns loading, running and saving; its genuine post-load scope owns the invocation chain. The core executor owns invocation-selected artifact maps. Output identities validate the saved-path batch directly. The microscope handler supplies parser facts; the existing profile logger owns runtime and step records. Materialization targets own saving, and the existing artifact type family owns record reduction. The existing metadata target family now directly owns publication and final reconciliation. Obsolete facades, duplicate registries, descriptors and forwarding methods are removed; the existing nested metadata target is promoted and renamed, with its three concrete kinds retained.
ObjectLabelSet.from_payload admits already resolved source, parent spacing and variants in one construction. The recorder uses the existing native storage boundary. Direct adapter admission retains its distinct name/alias ordering. Parentless CellProfiler policies still suppress parent spacing. Native contextual object outputs now retain their declared input calibration, fixing four converted-checkpoint cases independently reproduced on main.
Producer selection is keyed by actual producer/requested-output semantics, rather than a temporary plan address. The separate filtered-path cache is removed so live storage aliases are rederived at each epoch. Only missing declared aggregate coordinates are deferred to the existing per-plane filename writer; fixed coordinates, duplicate paths and scalar-image conflicts remain strict.
Validation: the initial 700-pass/13-failure receiving run was followed by fixes and a passing selection of those 13 cases. Subsequent affected materialization/publication selections exposed existing main fixture inconsistencies and the native calibration bug; baseline replays reproduced them. The final bounded selection passes 27 existing controls, including converted checkpoint calibration, parent/no-parent policies, object-label input/domain transitions, same-path scope ownership and live alias mutation. Changed-source hashes remain identical before and after that gate. Original ClassDef/family/caller/effect inventory, import checks and diff checks accompany the changes; these are bounded checks, not a global equivalence proof. No new mirror-test family was added.
Synced with remote main e1400cb, including grouped source-scope projection and merged median/shape changes. The ordinary three-case comparison against that same main is complete: all 12 strict comparisons against both retained native observations per case pass. Source, dependency, environment, input and native-binary guards pass before and after. Both servers exited and the official runtime lease was reacquired and released. The performance gate is negative in this one-observation-per-source comparison. Those measurements remain retained; this PR makes no speedup claim. Startup and registry/kernel warming are excluded from public pipeline timing.
Compile remains 0.315/0.855/0.450 s on the candidate. All candidate execution/total/RSS observations are worse; this single pair does not establish causation or variability. Speckles' increase is mostly the first primary-object step (+301.6 ms), while 3D increases are distributed across several steps. These are full step spans, not isolated numerical kernels. No fresh native clocks, repeats or additional profiles were used for this comparison.
Closes #628
Synced by merging current main dc448b0 into PR HEAD 8a75384. The duplicate aggregate-coordinate fix was resolved on the existing refactored owner; main acquisition-extension and typed source-origin changes are retained. Twelve conflict-related controls pass in 2.79s. This synchronization is not a new scientific or timing qualification. The negative comparison remains tied to its original source refs.
Current receiving disposition: synchronized with main 1804077, including merged PRs #638/#648 (compiled slots, immutable publication, admitted source matching, buffered profiling and transport), #635 (intensity), and #597 (saved compiler resolution). The structural conflict is resolved on the consolidated existing execution-request and loaded-cohort owners; obsolete runtime preparation helpers and the detached returned-output matcher remain deleted. Source discovery, producer ordering and current context-owned workspace projection are retained.
The bounded current-main receiving gate passes 108 existing controls: 45 compiled-slot/profile/cohort/source controls, 48 materialization/source-extension/inheritance controls, and 15 core/output admission controls. The production diff and imports are clean. No new scientific or latency campaign was run for this synchronization. The earlier 12 strict scientific comparisons and negative timing observations remain tied to their original source refs.
The user explicitly requested merging locally qualified semantic simplifications. This change is ready on that basis, with the whole-pipeline performance goal still open and no proven overall speedup attributed to this PR.
Final head fffb50e also receives latest main 4236e8f (#608). That merge changes only the separately qualified morphology kernel and its controls plus documentation; all 108 tested consolidation-owner/control bytes are unchanged from receiving head adb2c46. No duplicate timing or test campaign was run for the disjoint merge.