Preserve current intensity domains across mixed carriers (addresses #599)

docs/validation/mixed-carrier-position-public-receiving06-20261004.rst

@@ -0,0 +1,140 @@
+Combined600/614 ordinary installed and public receiving PASS
+==========================================================
+
+One complete ordinary wheel, source42203213d744b2b0393c9966868e8d02ec45c3d9;
+wheelSHA ebfb65fe1b70741b322d4463d45f2cbd421381c9e603ba3c8240f05a64c1843c.
+Original package02/target is immutable. Whole-byte/RECORD/source/resource proof
+and installed3position controls retain their earlier published qualification.
+The26 installed600 controls remain qualified at identical600 production bytes.
+
+One original public CLI PTY76599, MCP446810, invocation
+acbd7b2fa1de48dc83c52a4aeca5e552, firstepoch1791115322. Health resolved target02.
+One original public native bootstrap447841/create1791115349.67 on6012TCP/7012ACK.
+Original catalog preparation READY sequence988. No viewer, science contact,
+scientific source, private algorithm, installed edits, second client or timeout
+change. The original recorded owner's fresh admission PASS preceded launch.
+
+600 actual path
+---------------
+
+Original two TIFFs were copied byte-exact into the declared writable workspace
+before normal SourceBindings preparation. Public artifact-plan resolved scalar
+Gray/channel1 and RGB Color/channel2 with physical .5/.5micrometer calibration.
+The original importer merged two independent ImageMath declarations into one
+step; three native steps produced four named image artifacts.
+
+Original session1/job1 compilation COMPLETE. Its job3 execution
+07355a20-3504-4c67-aad0-dcb8f868db70 FAILED on automatic final image publication:
+conflicting metadata for persisted Reverse occurrence. Full original traceback,
+negative job and partial outputs remain. This is not a600 normalization success
+receipt and is not claimed fixed by either600 or614; it reaches the shared
+publication family, not a numerical exception or timeout. Historical435 is now
+closed for its accepted cases, so this distinct Reverse variant is NOT described
+as an unresolved435 fix or automatically proven independent of600.
+
+Distinct complete pipeline-artifacts-only02.py keeps identical acquisitions,
+importer, four ImageMath declarations and named-artifact persistence. It changes
+only the supported PathPlanningConfig well_filter=0 (automatic final main-flow
+persistence disabled) and distinct output destination. No metadata guard is
+relaxed. Public plan confirms named persistence enabled and automatic main-flow
+persistence disabled. Session3/job5 compilation COMPLETE; job6 execution
+a80fe53c-3270-4ca2-8b3c-d02da65eda04 COMPLETE.
+
+Public inventory exposed all four exact saved images. Native-resolution public
+samples return180float32 values each: scalar/255, original skimage rgb2gray of
+RGB/255, their average in each operand order. All720 values have ZERO absolute
+error against independently computed expected arrays. Both arithmetic orders
+agree. Public source calibration .5/.5micrometers remains. The existing named
+image contract retains first-operand source identity (Gray for Forward, Color
+for Reverse); no new dual-operand provenance capability is claimed.
+
+614 actual path
+---------------
+
+Original18embedded-header acquisition TIFFs (nine shuffled sites, two channels)
+were copied byte-exact into the same declared workspace. Public SourceBindings
+plan exposes all18 exact source planes, signed coordinates(-2,-1) through(12,9),
+and .25Y/.5Xmicrometer calibration. Original registered acquisition_tile_positions
+and assemble_stack_cpu use the native compiled positions artifact edge.
+Session2/job2 compilation COMPLETE; job4 execution
+b02b921e-5830-4ec5-aa7d-eb22288e1b22 COMPLETE.
+
+Canonical native RuntimeExecutionObservation export validates its own execution
+and artifact expectations. Its two scoped positions records contain all nine
+expected XY coordinates in resolvedsite order, distinct native channel1/2.
+Native log records both registered calls and the original positions artifact
+input join for assembly. Saved JSON positions agree; saved images and checkpoints
+are exposed in the ordinary public inventory.
+
+Both public full14x20uint16 mosaics match ALL560 elements exactly: every4x6 tile
+has value10*channel+site at(row*5,col*7), remaining holes zero. Public calibration
+.25/.5micrometers preserved. Original VirtualWorkspaceSourceProjection decodes
+saved image metadata: exactly nine source contributors/channel, original site
+order1..9, exact signed positions and paths to the prepared source acquisition.
+SpecialArtifact positions stores a list plus scoped identity, not ImageMetadata.
+The image projection is the authoritative owner of saved pixel provenance.
+
+Verification and preserved negatives
+-----------------------------------
+
+verify-received02.py terminal0,3.40s/287932KiB/CPU99%/Swap0. It uses original
+ZMQRuntimeExecutionObservationExport.read/require_valid_observation and original
+VirtualWorkspaceSourceProjection, no alternate runtime codec. Both observations
+PASS; public arrays, source paths and all20 original source-file hashes PASS.
+
+verify-received01.py terminal1 remains: its incorrect assumption that the special
+positions list carries ImagePayloadMetadata was replaced by the real saved-image
+projection checks, not removed provenance assertions. Its original logs/script
+remain. CLI `context` usage error, undeclared get_job_status name and read-only
+plate writable-policy rejection remain in the complete original journal; none
+dispatched an uncertain native mutation. Correct current entrypoints were used.
+
+Closure
+-------
+
+Exactly one public close of the original native handle returned acknowledged,
+process_exited, endpoint_terminated and succeeded alltrue, errors[]. Native447841
+and MCP446810 subsequently absent;6012/7012/6013/7013 listeners empty. No viewer
+was started. Original CLI exit once; actual PTY76599 terminal2, attributable to
+the retained earlier negative commands/jobs, not hidden or rewritten as0.
+No UNKNOWN replay; native numeric OS exit not inferred from ACK.
+
+This qualifies the synthetic source-selection/processing/persistence numerical
+and acquisition-position workflow. It does not prove biological usefulness,
+physicalGPU execution, globalNRA R1 completion or repair of automatic final
+publication. Current main later carries disjoint cancelled-dispatch, operations
+observation and FAST-speckle changes; receiving remains explicitly pinned422.
+Future complete packages must incorporate normal main/dependency floors; this
+receipt is not mislabelled as a live test of those unrelated newer components.
+
+Determining publication attribution
+----------------------------------
+
+Actual failure is original function_outputs.py:942, metadata-writer comparison
+of automatic produced projection with the named artifact at the same pathname.
+That whole file is byte-identical before600 at main634875dd and received422:
+Gitblob82eb17a7e63d54dbe2727dfca236566afb0ec1aa. source_projection.py and
+interop/cellprofiler/runtime/main_flow.py also have no delta across those pins.
+Historical pre600 engineering494 native execution9292bfb8 on2026-10-03 failed
+at the same comparison/message after artifact publication; exact original log
+engineering494/POINTS494_95/author-workspace/output/runtime/scratch/data/openhcs/
+logs/openhcs_zmq_server_port_6014_1791009392666738273.log:270..292 and original
+mcp.stdout:2101/2142. Main's persisted-point-receiver-main-20261003.rst historically
+records Root435 publication as unresolved, distinct from its accepted reopen.
+Current435 comments show the aggregate and Mosaic cases were subsequently
+fixed/accepted in merged394. The old statement is not current owner status.
+This establishes a pre-existing publication family and unchanged writer guard;
+it does NOT prove that this exact Reverse fixture fails on a pre600 runtime.
+No older runtime was launched/replayed and no600-specific interaction is waived.
+
+Durable original evidence
+-------------------------
+
+Original complete journals, typed native VALUES, all tiny source TIFFs, pipeline
+documents, public pixel verification scripts/negative/PASS logs and produced
+metadata are archived byte-exact in the sibling
+mixed-carrier-position-public06-20261004.tar.gz. No original log was rewritten.
+Persistent original receiving root remains
+/home/ts/wt/openhcs-issue-batch-20260929/engineering599/public94-combined02.
+Original VALUES paths are ENGINEERING600614_94/author-workspace/output/
+intensity-values02.json and positions-values01.json under that root.

openhcs/core/aligned_image_payload.py

@@ -396,21 +396,6 @@ def resolve_source_spatial_value(self, value: Any) -> Any:
         return adapter.value_in_payload_domain(reference)
 
 
-def stack_image_payload_context(
-    image_payloads: Sequence[Any],
-    stack: RuntimeArrayData,
-    *,
-    metadata_mode: ImagePayloadMetadataCompositionMode,
-) -> Any:
-    """Attach composed image metadata and masks to a freshly stacked payload."""
-    payloads = tuple(image_payloads)
-    metadata = ImagePayloadMetadata.compose(
-        payloads,
-        mode=metadata_mode,
-    )
-    return metadata.payload_with(stack, _stack_image_payload_mask(payloads, stack))
-
-
 class ImagePayloadStackComposition(ABC):
     """Compose pixels, masks and provenance on one declared leading axis."""
 
@@ -474,13 +459,10 @@ def from_loaded_images(
                     workspace_source_lookups
                 )
             if metadata_mode is ImagePayloadMetadataCompositionMode.STACK:
-                main_data_stack = stack_runtime_slices(
-                    tuple(image_payload_data(payload) for payload in payloads),
-                    execution_plan.input_memory_type,
-                    execution_plan.device_id_for(execution_plan.input_memory_type),
-                )
-                main_data_stack = stack_image_payload_context(
-                    payloads, main_data_stack, metadata_mode=metadata_mode,
+                main_data_stack = stack_image_payloads(
+                    payloads, metadata_mode=metadata_mode,
+                    memory_type=execution_plan.input_memory_type,
+                    device_id=execution_plan.device_id_for(execution_plan.input_memory_type),
                 )
             elif metadata_mode is ImagePayloadMetadataCompositionMode.BUNDLE:
                 main_data_stack = ImagePayloadBundleContext.from_payloads(
@@ -530,12 +512,13 @@ def with_saved_output_context(
     ) -> RuntimeArrayData:
         """Attach saved member context without replacing the independent buffer."""
         data = image_payload_data(stack_payload)
+        current_intensity = image_payload_metadata(stack_payload)
         if (
             len(payloads) == 1
             and single_output_plane_axis is metadata[0].plane_axis
             and np.shape(data) == np.shape(image_payload_data(payloads[0]))
         ):
-            return metadata[0].replace_fields().payload_with(
+            return metadata[0].with_current_intensity_from(current_intensity).payload_with(
                 data, image_payload_mask(stack_payload),
             )
         if np.shape(data)[:1] != (len(payloads),):
@@ -551,16 +534,21 @@ def with_saved_output_context(
             else ImagePayloadMetadataCompositionMode.STACK
         )
         output_metadata = ImagePayloadMetadata.compose(
-            tuple(payloads), mode=mode, source_metadata=tuple(metadata),
+            tuple(payloads), mode=mode, source_metadata=tuple(
+                record.with_current_intensity_from(current_intensity, plane_index=index)
+                for index, record in enumerate(metadata)
+            ),
         )
         return output_metadata.payload_with(
             data, _stack_image_payload_mask(tuple(payloads), data),
         )
 
-    def composition_source_metadata(self) -> tuple[ImagePayloadMetadata, ...]:
+    def composition_source_metadata(
+        self, payloads: Sequence[Any] | None = None,
+    ) -> tuple[ImagePayloadMetadata, ...]:
         return tuple(
             self.composition_payload_metadata(image_payload_metadata(payload))
-            for payload in self.composition_payloads
+            for payload in (self.composition_payloads if payloads is None else payloads)
         )
 
     def composition_payload_metadata(
@@ -569,24 +557,47 @@ def composition_payload_metadata(
         """Preserve each input's declared provenance unless the axis owner projects it."""
         return metadata
 
-    def compose(self) -> Any:
-        payloads = self.composition_payloads
+    @staticmethod
+    def composition_memory_domain(
+        payloads: tuple[RuntimeArrayData, ...], *,
+        memory_type: str | None = None, device_id: int | None = None,
+    ) -> tuple[str, int | None]:
+        """Resolve one composition destination through the original memory owner."""
+        if memory_type is not None:
+            return memory_type, device_id
+        memory_type = detect_memory_type(payloads[0])
+        return memory_type, MemoryType(memory_type).device_id_of(payloads[0])
+
+    def compose(
+        self, *, memory_type: str | None = None, device_id: int | None = None,
+    ) -> Any:
+        # Intensity reconciliation projects pixels to the numerical host domain.
+        # Resolve the destination from the original carrier before that projection.
+        memory_type, device_id = self.composition_memory_domain(
+            tuple(image_payload_data(payload) for payload in self.composition_payloads),
+            memory_type=memory_type, device_id=device_id,
+        )
+        payloads = ImagePayloadMetadata.intensity_coherent_payloads(self.composition_payloads)
         composed = self.compose_unmasked(
-            tuple(image_payload_data(payload) for payload in payloads)
+            tuple(image_payload_data(payload) for payload in payloads),
+            memory_type=memory_type, device_id=device_id,
         )
         metadata = ImagePayloadMetadata.compose(
             payloads,
             mode=self.composition_metadata_mode,
-            source_metadata=self.composition_source_metadata(),
+            source_metadata=self.composition_source_metadata(payloads),
         )
         return metadata.payload_with(composed, self.compose_mask(composed, metadata))
 
     def compose_unmasked(
-        self, payloads: tuple[RuntimeArrayData, ...]
+        self, payloads: tuple[RuntimeArrayData, ...], *,
+        memory_type: str | None = None, device_id: int | None = None,
     ) -> RuntimeArrayData:
-        memory_type = detect_memory_type(payloads[0])
+        memory_type, device_id = self.composition_memory_domain(
+            payloads, memory_type=memory_type, device_id=device_id,
+        )
         return stack_runtime_slices(
-            payloads, memory_type, MemoryType(memory_type).device_id_of(payloads[0])
+            payloads, memory_type, device_id,
         )
 
     def compose_mask(self, composed: Any, metadata: ImagePayloadMetadata) -> Any | None:
@@ -622,10 +633,14 @@ def stack_image_payloads(
     image_payloads: Sequence[Any],
     *,
     metadata_mode: ImagePayloadMetadataCompositionMode,
+    memory_type: str | None = None,
+    device_id: int | None = None,
 ) -> Any:
     """Stack image payloads in their declared memory domain with full context."""
 
-    return ImagePayloadStackContext(image_payloads, metadata_mode).compose()
+    return ImagePayloadStackContext(image_payloads, metadata_mode).compose(
+        memory_type=memory_type, device_id=device_id,
+    )
 
 
 def _stack_image_payload_mask(
@@ -671,7 +686,7 @@ def _stack_image_payload_mask(
     return stack_runtime_slices(
         resolved_masks,
         detect_memory_type(stack),
-        0,
+        MemoryType(detect_memory_type(stack)).device_id_of(stack),
     )
 
 
@@ -854,10 +869,12 @@ def combined_mask(self) -> RuntimeArrayData | None:
     def compose_unmasked(
         self,
         payloads: tuple[RuntimeArrayData, ...],
+        *, memory_type: str | None = None, device_id: int | None = None,
     ) -> RuntimeArrayData:
         """Compose image payload arrays without mask/metadata wrapping."""
-        memory_type = detect_memory_type(payloads[0])
-        device_id = MemoryType(memory_type).device_id_of(payloads[0])
+        memory_type, device_id = self.composition_memory_domain(
+            payloads, memory_type=memory_type, device_id=device_id,
+        )
         channel_axes = tuple(
             metadata.normalized_source_channel_axis(payload)
             for payload, metadata in zip(
@@ -868,7 +885,9 @@ def compose_unmasked(
         )
         declared_channel_count = sum(axis is not None for axis in channel_axes)
         if declared_channel_count in {0, len(payloads)}:
-            return stack_runtime_slices(payloads, memory_type, device_id)
+            return super(ImagePayloadBundleContext, self).compose_unmasked(
+                payloads, memory_type=memory_type, device_id=device_id,
+            )
         return self.compose_mixed_channel_payloads(
             payloads,
             channel_axes=channel_axes,
@@ -1225,9 +1244,9 @@ def copy_projected_output_stack(
             return ImagePayloadBundleContext.from_payloads(
                 payloads, metadata_mode=metadata_mode,
             ).compose()
-        stacked = stack_runtime_slices(data, memory_type, device_id)
-        return stack_image_payload_context(
-            payloads, stacked, metadata_mode=metadata_mode,
+        return stack_image_payloads(
+            payloads, metadata_mode=metadata_mode,
+            memory_type=memory_type, device_id=device_id,
         )
 
     def output_values_for_artifact_specs(

openhcs/core/artifacts.py

@@ -797,8 +797,7 @@ def compose_runtime_values(
             return values[0].data
 
         from openhcs.core.aligned_image_payload import stack_image_payloads
-        from openhcs.core.aligned_image_payload import stack_image_payload_context
-        from openhcs.core.memory import detect_memory_type, stack_runtime_slices
+        from openhcs.core.memory import detect_memory_type
         from openhcs.core.runtime_image_values import (
             ImagePayloadMetadataCompositionMode,
             image_payload_data,
@@ -814,9 +813,8 @@ def compose_runtime_values(
         if producer_group_scope is not None:
             payloads = tuple(value.data for value in values)
             arrays = tuple(image_payload_data(payload) for payload in payloads)
-            return stack_image_payload_context(
-                payloads,
-                stack_runtime_slices(arrays, detect_memory_type(arrays[0]), 0),
+            return stack_image_payloads(
+                payloads, memory_type=detect_memory_type(arrays[0]), device_id=0,
                 metadata_mode=ImagePayloadMetadataCompositionMode.for_plane_axis(
                     RuntimePlaneAxis.RUNTIME_SLICE
                 ),

openhcs/core/image_file_serialization.py

@@ -155,7 +155,8 @@ def project_image_metadata(
             raise ValueError("Saved image metadata requires an actual native dtype.")
         native_scale_governs = self.intensity_scale is not None or not values_preserved
         if not values_preserved:
-            metadata = metadata.without_unit_interval_intensity_scale().replace_fields(
+            metadata = metadata.replace_fields(
+                unit_interval_intensity=None,
                 physical_border_edges_yx=None,
                 mask_defines_border=None,
             )

openhcs/core/projected_image_output.py

@@ -10,7 +10,7 @@
 
 from openhcs.core.aligned_image_payload import (
     AlignedImageStack,
-    stack_image_payload_context,
+    stack_image_payloads,
 )
 from openhcs.core.registry_strategies import NominalTypeKeyedStrategyMixin
 from openhcs.core.runtime_array_values import (
@@ -236,9 +236,8 @@ def contextualize(
                     output_slice,
                 )
             )
-        return stack_image_payload_context(
+        return stack_image_payloads(
             tuple(contextualized_slices),
-            output_data,
             metadata_mode=ImagePayloadMetadataCompositionMode.STACK,
         )