From 45437a8a906b4580c4f273eacb90675f3c802808 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 15:45:03 -0400 Subject: [PATCH 01/22] Repair real BaSiCPy observation and parameter contract (#213) --- docs/source/development/basicpy_readiness.md | 129 ++++++++ .../backends/enhance/basic_processor_jax.py | 312 ++++-------------- .../processing/backends/enhance/flatfield.py | 7 - pyproject.toml | 21 +- requirements-basicpy.txt | 4 + scripts/check_basicpy_source.py | 111 +++++++ 6 files changed, 314 insertions(+), 270 deletions(-) create mode 100644 docs/source/development/basicpy_readiness.md create mode 100644 requirements-basicpy.txt create mode 100644 scripts/check_basicpy_source.py diff --git a/docs/source/development/basicpy_readiness.md b/docs/source/development/basicpy_readiness.md new file mode 100644 index 0000000000..ee192f4e69 --- /dev/null +++ b/docs/source/development/basicpy_readiness.md @@ -0,0 +1,129 @@ +# BaSiCPy readiness checkpoint (#213) + +Status: draft **source integration**, not installed, import- or fit-validated. +Backend/dependency owner: Linnaeus. Parent PR151 owns recipes/policy. Memory/ +session PR208 remains independently pending actual live acceptance. + +## Implementation and dependency source + +`basic_flatfield_correction_jax` delegates to the real JAX BaSiCPy model. There +is one ensemble fit, no copied low-rank algorithm or NumPy/CuPy fallback. Removed +ignored `lambda_sparse`, `lambda_lowrank`, `verbose`, renamed `max_iters` to +the model's `max_iterations`, and deleted the per-volume batch adapter. Repository +call-site search found only its own two duplicated internal callers. No alias +or compatibility reader is retained. Existing user-authored pipelines must use +the actual parameters; exported declarations/saved histories are not modified. + +`FittingMode` comes from BaSiCPy rather than a second OpenHCS enum. Every exposed +knob reaches the model. Corrected output remains floating point; no integer +recast, clipping or unit-interval conversion is performed by this adapter. +`timelapse=False` avoids subtracting the estimated per-observation baseline. + +The paired [BaSiCPy draft](https://github.com/OpenHCSDev/BaSiCPy/pull/1) reuses +Tristan's clean JAX prototype `ae2c647`, rather than PyPI 2.x's PyTorch API. +`requirements-basicpy.txt` pins the reviewed fork commit. As with this project's +existing source-only dependency files, no Git URL enters PyPI metadata. After +review, install that file in the coordinator-selected **isolated** validation +environment; do not run it against the active analysis venv. A published fork +wheel/version and ordinary optional extra can replace the source pin only after +actual import/fit acceptance and release review, not by adding an unavailable +PyPI dependency now. + +The existing gpu/all JAX constraints rejected this modern candidate. They now +delegate exact JAXlib/CUDA-plugin matching to `jax[cuda12-local]>=0.9.2,<0.10`, +preserving the local-CUDA installation route without bundling new CUDA libraries. +JAX owns its dependencies ([0.9.2 metadata](https://pypi.org/pypi/jax/0.9.2/json), +[installation documentation](https://docs.jax.dev/en/latest/installation.html)). +This does **not** claim that unrelated Torch/TensorFlow constraints in all/gpu +support Python3.14, or that this change was installed/validated on CUDA. + +## Observation and artifact boundaries + +BaSiC requires independent, diverse observations of a common illumination field, +same channel/grid/acquisition. It cannot distinguish stationary biological +structure from shading merely because fitting converged. N>=2 is only a sanity +check, not sufficient scientific evidence. + +The existing `PURE_3D`, `required_variable_components(SITE)` and +`allowed_group_by(CHANNEL)` declarations require a real cross-site stack and +reject a Z-only/mixed-channel FunctionStep. Direct calls accept `(N,Y,X)` or +`(N,Z,Y,X)`; they do not split volumes into separate fits. Pipeline acceptance +targets exactly `[SITE]`, with fixed Z/time and channel grouping. Multi-axis +SITE+Z/time grids and time-only pipeline admission are not claimed by this +checkpoint: the existing required-axis declaration checks inclusion, not an +exact alternative independent-axis set. Do not advertise those configurations +as validated. + +Fitted flatfield/darkfield artifacts remain to implement/verify. Existing +`artifact_outputs`, image sidecars, `SourceProjectedImageOutput` and +`ImagePayloadMetadata.collapse_leading_plane_axis()` provide the likely route: +a fitted field is an aggregate of all observations, with contributor provenance, +not one selected source plane. Repeating the field N times or lying about a +plane projection would introduce identity/RAM debt. Core artifact/runtime files +are Zeno-owned; this patch does not edit them or introduce a second artifact +pipeline. Source-only aggregate projection feasibility is being checked before +adding a local behavior-owning output leaf. + +## Actual evidence + +Current metadata (no backend import): active Python3.12.3, JAX/JAXlib0.9.2, +NumPy2.1.3, SciPy1.18.1, scikit-image0.25.2. **BaSiCPy is not installed.** +The independent target `/usr/bin/python3.14` is Python3.14.7. The fork's candidate +matrix is documented in its PR; wheel metadata and AST syntax are not runtime +readiness. Coordinator supplied actual old-wrapper MCP discovery/signature +evidence; no fitting occurred on that running installed server. + +Executed here, source only: + +``` +/usr/bin/python3.14 scripts/check_basicpy_source.py \ + --basicpy-source /home/ts/wt/basicpy-python314-20260929/src/basicpy/basicpy.py +``` + +Five checks pass (0.009s): knobs against actual upstream declaration fields, +enum owner, existing pipeline declarations, no integer cast/hidden dependency +failure, reviewed source pin and JAX-owned dependency matching. Focused Ruff and +diff checks pass. These checks do not import OpenHCS/JAX or execute a model. +Normal merge of freshly fetched `openhcsdev/main` is up to date at `a0263e82a`; +all recorded submodules are initialized in the isolated worktree. + +## Focused antipattern / new-case review + +Scope: changed wrapper, shared flatfield enum, JAX dependency declaration and +paired fork's DCT/Pydantic owners. Focused source/AST/catalog review, not full NRA +scan or global proof. + +- BOUND-7: removed `hasattr(shape/dtype)` probing of a known array argument. + Validate finite observations once, and preserve real model errors. +- MEMB-1/BOUND-2: upstream `FittingMode` owns optimizer choices, not a mirrored + enum. New fitting modes require only BaSiCPy's owner, no consumer switch. +- IMPL-12: delete duplicated batch loops/forwarding. A new N count or volume + shape uses the same model fit, not another branch/registry. +- IDEN-4/TIME-7: replace misleading Z/ignored-knob documentation and copied + optimizer defaults with actual observation semantics and model-owned defaults. +- IMPL-13/TIME-1: paired fork reuses the public JAX DCT owner; copied private + helper is deleted in place. Pydantic models retain durable schema ownership. +- New output case must extend existing image-output/artifact owners and prove + aggregate provenance, not add metric/artifact dictionaries or special-name + dispatch to generic consumers. This part remains under source review. + +## Pending scheduled acceptance + +Parent #151/#212 gets first released validation slot. This worker waits for +explicit next-slot handoff, then uses nonblocking flock/resource guard. No new +environment/GUI/JVM/runtime, install or active baseline changes happened here. + +1. Verify imported paths first; real isolated Python3.14 dependency resolve and + import against the exact paired source, without an old JAX downgrade. +2. Bounded real fit-transform on synthetic independent same-channel shaded + fields. Inspect flatfield/darkfield; report time/RSS/PSS/private/swap/thread + receipt. Compare known shading with moving-object and stationary-pattern + controls to test biological leakage, not just output variance. +3. Compile and execute the same fixture through real OpenHCS/MCP and existing + artifact materialization/inspection. Verify floating saved range and exact + contributor/observation routing, no fake/mock UI or alternate pipeline. +4. Verify the affected installed entrypoint only after coordinator-reviewed + integration/installation. Neither paired PR closes #213 until acceptance. + +No blind inputs/reference answers or saved user declarations are reconstructed +or modified. No new paid provider/service calls. No source or runtime leak claim. diff --git a/openhcs/processing/backends/enhance/basic_processor_jax.py b/openhcs/processing/backends/enhance/basic_processor_jax.py index 02620abe29..5452dedf93 100644 --- a/openhcs/processing/backends/enhance/basic_processor_jax.py +++ b/openhcs/processing/backends/enhance/basic_processor_jax.py @@ -1,268 +1,80 @@ -""" -BaSiC (Background and Shading Correction) Implementation using JAX via BaSiCPy - -This module provides OpenHCS-compatible wrapper functions for BaSiCPy's -JAX-based BaSiC implementation, integrating with OpenHCS memory decorators -and pipeline system. - -Doctrinal Clauses: -- Clause 3 — Declarative Primacy: All functions are pure and stateless -- Clause 65 — Fail Loudly: No silent fallbacks or inferred capabilities -- Clause 88 — No Inferred Capabilities: Explicit JAX dependency via BaSiCPy -- Clause 273 — Memory Backend Restrictions: JAX-only implementation -""" +"""Real BaSiCPy correction of independent, same-channel observations.""" from __future__ import annotations -import logging -from typing import TYPE_CHECKING, Any, Optional, Union +import jax.numpy as jnp +import numpy as np +from basicpy import BaSiC +from basicpy.basicpy import FittingMode -# Import decorator directly from core.memory to avoid circular imports +from openhcs.constants.constants import GroupBy, VariableComponents from openhcs.core.memory import jax as jax_func -from openhcs.processing.backends.enhance.flatfield import BasicFittingMode -from openhcs.utils.import_utils import optional_import_placeholder - -# For type checking only -if TYPE_CHECKING: - import jax.numpy as jnp - -# Import jax.numpy for runtime type hint evaluation -try: - import jax.numpy as jnp -except ImportError: - jnp = None - -# Import BaSiCPy as an optional dependency -basicpy = optional_import_placeholder("basicpy") -if basicpy: - BaSiC = basicpy.BaSiC -else: - BaSiC = None - -logger = logging.getLogger(__name__) - - -def _validate_jax_array(array: Any, name: str = "input") -> None: - """ - Validate that BaSiCPy is available and input is compatible. - - Args: - array: Array to validate - name: Name of the array for error messages - - Raises: - ImportError: If BaSiCPy is not available - ValueError: If the array is not compatible - """ - if not basicpy or BaSiC is None: - raise ImportError( - "BaSiCPy is not available. Please install BaSiCPy for BaSiC correction. " - "Install with: pip install basicpy" - ) +from openhcs.core.pipeline.function_contracts import ( + allowed_group_by, + required_variable_components, +) +from openhcs.processing.backends.lib_registry.unified_registry import ProcessingContract - if not hasattr(array, "shape") or not hasattr(array, "dtype"): - raise ValueError( - f"{name} must be an array-like object with shape and dtype attributes, " - f"got {type(array)}." - ) - -@jax_func +@jax_func(contract=ProcessingContract.PURE_3D) +@allowed_group_by(GroupBy.CHANNEL) +@required_variable_components(VariableComponents.SITE) def basic_flatfield_correction_jax( - image: "jnp.ndarray", - max_iters: int = 50, - lambda_sparse: float = 0.01, - lambda_lowrank: float = 0.1, + image: jnp.ndarray, + max_iterations: int = 50, epsilon: float = 0.1, smoothness_flatfield: float = 1.0, smoothness_darkfield: float = 1.0, sparse_cost_darkfield: float = 0.01, get_darkfield: bool = False, - fitting_mode: BasicFittingMode = BasicFittingMode.LADMAP, - working_size: Optional[Union[int, list]] = 128, - verbose: bool = False, -) -> "jnp.ndarray": - """ - Perform BaSiC-style illumination correction on a 3D image stack using JAX via BaSiCPy. - - This function provides OpenHCS integration for BaSiCPy's sophisticated BaSiC - algorithm implementation, supporting both LADMAP and approximate fitting modes - with automatic parameter tuning capabilities. + fitting_mode: FittingMode = FittingMode.ladmap, + working_size: int | None = 128, +) -> jnp.ndarray: + """Fit one BaSiC model to independent observations and apply its fields. + + The leading N axis contains independent timepoints or mosaic positions, + not channels or the Z planes of a single volume. In a FunctionStep use + group_by=CHANNEL and variable_components=[SITE] to fit across mosaic fields. + Z-only stacks are not independent-observation ensembles and are rejected + by the declared SITE requirement before pipeline execution. + All observations must share a spatial grid and acquisition illumination. + Several diverse observations are needed to separate stationary shading + from biology; two is only a minimum input sanity check, not identifiability. Args: - image: 3D JAX array of shape (Z, Y, X) - max_iters: Maximum number of iterations for optimization - lambda_sparse: Regularization parameter for sparse component (mapped to epsilon) - lambda_lowrank: Regularization parameter for low-rank component (mapped to smoothness) - epsilon: Weight regularization term - smoothness_flatfield: Weight of flatfield term in Lagrangian - smoothness_darkfield: Weight of darkfield term in Lagrangian - sparse_cost_darkfield: Weight of darkfield sparse term in Lagrangian - get_darkfield: Whether to estimate darkfield component - fitting_mode: Fitting mode ('ladmap' or 'approximate') - working_size: Size for running computations (None means no rescaling) - verbose: Whether to print progress information - **kwargs: Additional parameters (ignored for compatibility) + image: Same-channel observations, (N,Y,X) or volumes (N,Z,Y,X). + max_iterations: Maximum iterations per BaSiC optimization. + epsilon: BaSiC weight regularization term. + smoothness_flatfield: Flatfield smoothness weight. + smoothness_darkfield: Darkfield smoothness weight. + sparse_cost_darkfield: Darkfield sparse-cost weight. + get_darkfield: Estimate additive darkfield as well as flatfield. + fitting_mode: BaSiCPy's declared optimization algorithm. + working_size: Spatial working size, or None for no rescaling. Returns: - Corrected 3D JAX array of shape (Z, Y, X) - - Raises: - ImportError: If BaSiCPy is not available - ValueError: If input is not a 3D array - RuntimeError: If BaSiC fitting fails + Floating-point corrected observations, with the input shape. BaSiC's + (image - darkfield) / flatfield retains intensity units, fractions and + negative values; no clipping, normalization, integer recast or temporal + baseline subtraction is applied. Values can exceed the input range. """ - # Validate input and dependencies - _validate_jax_array(image) - - if image.ndim != 3: - raise ValueError(f"Input must be a 3D array, got {image.ndim}D") - - logger.debug(f"BaSiC correction: {image.shape} image, mode={fitting_mode}") - - try: - # Convert JAX array to numpy for BaSiCPy (it handles JAX internally) - import numpy as np - - image_np = np.asarray(image) - - # Create BaSiC instance with parameters - basic = BaSiC( - # Core algorithm parameters - max_iterations=max_iters, - epsilon=epsilon, - smoothness_flatfield=smoothness_flatfield, - smoothness_darkfield=smoothness_darkfield, - sparse_cost_darkfield=sparse_cost_darkfield, - get_darkfield=get_darkfield, - fitting_mode=fitting_mode.value, - working_size=working_size, - # Optimization parameters - optimization_tol=1e-3, - optimization_tol_diff=1e-2, - reweighting_tol=1e-2, - max_reweight_iterations=10, - # Memory and performance - resize_mode="jax", - sort_intensity=False, - ) - - # Fit and transform the image - logger.debug("Starting BaSiC fit and transform") - corrected_np = basic.fit_transform(image_np, timelapse=False) - - # Convert back to JAX array - import jax.numpy as jnp - - corrected = jnp.asarray(corrected_np) - - logger.debug(f"BaSiC correction completed: {corrected.shape}") - return corrected.astype(image.dtype) - - except Exception as e: - logger.error(f"BaSiC correction failed: {e}") - raise RuntimeError(f"BaSiC flat field correction failed: {e}") from e - - -@jax_func -def basic_flatfield_correction_batch_jax( - image_batch: "jnp.ndarray", - *, - batch_dim: int = 0, - max_iters: int = 50, - lambda_sparse: float = 0.01, - lambda_lowrank: float = 0.1, - epsilon: float = 0.1, - smoothness_flatfield: float = 1.0, - smoothness_darkfield: float = 1.0, - sparse_cost_darkfield: float = 0.01, - get_darkfield: bool = False, - fitting_mode: BasicFittingMode = BasicFittingMode.LADMAP, - working_size: Optional[Union[int, list]] = 128, - verbose: bool = False, -) -> "jnp.ndarray": - """ - Apply BaSiC flatfield correction to a batch of 3D image stacks. - - Args: - image_batch: 4D JAX array of shape (B, Z, Y, X) or (Z, B, Y, X) - batch_dim: Dimension along which the batch is organized (0 or 1) - max_iters: Maximum BaSiC optimisation iterations for each stack. - lambda_sparse: Compatibility regularisation value forwarded to each stack; the current per-stack adapter does not apply it. - lambda_lowrank: Compatibility regularisation value forwarded to each stack; the current per-stack adapter does not apply it. - epsilon: Weight regularisation term passed to BaSiC. - smoothness_flatfield: Smoothness weight for the estimated flatfield. - smoothness_darkfield: Smoothness weight for the estimated darkfield. - sparse_cost_darkfield: Sparse-cost weight for the estimated darkfield. - get_darkfield: Estimate a darkfield component in addition to the flatfield. - fitting_mode: BaSiC fitting algorithm used for every stack. - working_size: Spatial working size used by BaSiC, or ``None`` to disable rescaling. - verbose: Compatibility progress flag forwarded to each stack; the current per-stack adapter does not apply it. - - Returns: - Corrected 4D JAX array of the same shape as input - - Raises: - ImportError: If BaSiCPy is not available - ValueError: If input is not a 4D array or batch_dim is invalid - """ - # Validate input - _validate_jax_array(image_batch) - - if image_batch.ndim != 4: - raise ValueError(f"Input must be a 4D array, got {image_batch.ndim}D") - - if batch_dim not in [0, 1]: - raise ValueError(f"batch_dim must be 0 or 1, got {batch_dim}") - - logger.debug(f"BaSiC batch correction: {image_batch.shape}, batch_dim={batch_dim}") - - # Process each 3D stack in the batch - result_list = [] - - if batch_dim == 0: - # Batch is organized as (B, Z, Y, X) - for b in range(image_batch.shape[0]): - corrected = basic_flatfield_correction_jax( - image_batch[b], - max_iters=max_iters, - lambda_sparse=lambda_sparse, - lambda_lowrank=lambda_lowrank, - epsilon=epsilon, - smoothness_flatfield=smoothness_flatfield, - smoothness_darkfield=smoothness_darkfield, - sparse_cost_darkfield=sparse_cost_darkfield, - get_darkfield=get_darkfield, - fitting_mode=fitting_mode, - working_size=working_size, - verbose=verbose, - ) - result_list.append(corrected) - - # Stack along batch dimension - import jax.numpy as jnp - - return jnp.stack(result_list, axis=0) - - # Batch is organized as (Z, B, Y, X) - for b in range(image_batch.shape[1]): - corrected = basic_flatfield_correction_jax( - image_batch[:, b], - max_iters=max_iters, - lambda_sparse=lambda_sparse, - lambda_lowrank=lambda_lowrank, - epsilon=epsilon, - smoothness_flatfield=smoothness_flatfield, - smoothness_darkfield=smoothness_darkfield, - sparse_cost_darkfield=sparse_cost_darkfield, - get_darkfield=get_darkfield, - fitting_mode=fitting_mode, - working_size=working_size, - verbose=verbose, - ) - result_list.append(corrected) - - # Stack along batch dimension - import jax.numpy as jnp - - return jnp.stack(result_list, axis=1) + observations = np.asarray(image) + if observations.ndim not in (3, 4): + raise ValueError("BaSiC requires (N,Y,X) or (N,Z,Y,X) observations.") + if observations.shape[0] < 2: + raise ValueError("BaSiC requires multiple independent observations.") + if not np.isfinite(observations).all(): + raise ValueError("BaSiC observations must contain only finite values.") + + model = BaSiC( + max_iterations=max_iterations, + epsilon=epsilon, + smoothness_flatfield=smoothness_flatfield, + smoothness_darkfield=smoothness_darkfield, + sparse_cost_darkfield=sparse_cost_darkfield, + get_darkfield=get_darkfield, + fitting_mode=fitting_mode, + working_size=working_size, + ) + corrected = model.fit_transform(observations, timelapse=False) + return jnp.asarray(corrected) diff --git a/openhcs/processing/backends/enhance/flatfield.py b/openhcs/processing/backends/enhance/flatfield.py index ea53c8a072..58c541db00 100644 --- a/openhcs/processing/backends/enhance/flatfield.py +++ b/openhcs/processing/backends/enhance/flatfield.py @@ -8,10 +8,3 @@ class FlatfieldCorrectionMode(Enum): DIVIDE = "divide" SUBTRACT = "subtract" - - -class BasicFittingMode(Enum): - """Optimization algorithm exposed by BaSiCPy.""" - - LADMAP = "ladmap" - APPROXIMATE = "approximate" diff --git a/pyproject.toml b/pyproject.toml index f1db0ca6ee..cc082ca344 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -232,13 +232,8 @@ gpu = [ "torch>=2.6.0,<2.8.0", "torchvision>=0.21.0,<0.23.0", - # JAX - "jax>=0.5.3,<0.6.0", - "jaxlib>=0.5.3,<0.6.0", - - # JAX CUDA plugins - "jax-cuda12-pjrt>=0.5.3,<0.6.0", - "jax-cuda12-plugin>=0.5.3,<0.6.0", + # JAX owns matching JAXlib/CUDA plugins; retain local CUDA installation. + "jax[cuda12-local]>=0.9.2,<0.10", # CuPy "cupy-cuda12x>=13.3.0,<14.0.0", @@ -282,10 +277,7 @@ all = [ # GPU "torch>=2.6.0,<2.8.0", "torchvision>=0.21.0,<0.23.0", - "jax>=0.5.3,<0.6.0", - "jaxlib>=0.5.3,<0.6.0", - "jax-cuda12-pjrt>=0.5.3,<0.6.0", - "jax-cuda12-plugin>=0.5.3,<0.6.0", + "jax[cuda12-local]>=0.9.2,<0.10", "cupy-cuda12x>=13.3.0,<14.0.0", "cucim-cu12>=25.6.0,<26.0.0", "tensorflow>=2.19.0,<2.20.0", @@ -367,8 +359,11 @@ openhcs = [ ] # GPU Dependencies Version Notes: -# - JAX 0.4.38: Last version compatible with CuDNN 9.5.x (JAX 0.5.0+ requires CuDNN 9.8+) -# - JAX CUDA plugins must exactly match JAX version for PJRT API compatibility +# - JAX owns exact plugin/JAXlib matching through its cuda12-local extra. +# This uses a locally provisioned CUDA/cuDNN, not bundled CUDA dependencies. +# Review driver/library requirements at https://docs.jax.dev/en/latest/installation.html +# - BaSiCPy JAX fork is source-pinned in requirements-basicpy.txt while its +# paired draft is validated. Direct Git URLs must not enter PyPI metadata. # - PyTorch 2.7.x: Compatible with CUDA 12.6 and CuDNN 9.5.x # - TensorFlow 2.15-2.19: Stable DLPack support (2.12+ required for DLPack) # - CuPy: Use cupy-cuda12x for broad CUDA 12.x compatibility (not cupy-cuda120) diff --git a/requirements-basicpy.txt b/requirements-basicpy.txt new file mode 100644 index 0000000000..a7bdd299d5 --- /dev/null +++ b/requirements-basicpy.txt @@ -0,0 +1,4 @@ +# Reviewed JAX prototype, paired draft https://github.com/OpenHCSDev/BaSiCPy/pull/1 +# Do not replace with PyPI 2.x: it owns a different PyTorch API. +# Source checkpoint; import/fit/compiled MCP acceptance remains pending. +BaSiCPy @ git+https://github.com/OpenHCSDev/BaSiCPy.git@29f0a327d2069324f13d0c6fa0809a5c7566a6d4 diff --git a/scripts/check_basicpy_source.py b/scripts/check_basicpy_source.py new file mode 100644 index 0000000000..9138d053e3 --- /dev/null +++ b/scripts/check_basicpy_source.py @@ -0,0 +1,111 @@ +"""Focused source-only BaSiCPy integration checks, without backend imports.""" + +from __future__ import annotations + +import argparse +import ast +import re +import tomllib +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +WRAPPER = ROOT / "openhcs/processing/backends/enhance/basic_processor_jax.py" + + +class BasicPySourceTests(unittest.TestCase): + upstream_source: Path + + def setUp(self): + self.tree = ast.parse(WRAPPER.read_text()) + self.function = next( + node for node in self.tree.body + if isinstance(node, ast.FunctionDef) + and node.name == "basic_flatfield_correction_jax" + ) + + def test_every_exposed_knob_reaches_real_model_owner(self): + owner = next( + node for node in ast.parse(self.upstream_source.read_text()).body + if isinstance(node, ast.ClassDef) and node.name == "BaSiC" + ) + fields = { + node.target.id for node in owner.body + if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name) + } + call = next( + node for node in ast.walk(self.function) + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) + and node.func.id == "BaSiC" + ) + parameters = {arg.arg for arg in self.function.args.args[1:]} + forwarded = {kw.arg for kw in call.keywords} + self.assertEqual(parameters, forwarded) + self.assertLessEqual(forwarded, fields) + for keyword in call.keywords: + self.assertEqual(ast.unparse(keyword.value), keyword.arg) + + def test_choices_belong_to_upstream_not_a_mirror(self): + imports = { + (node.module, alias.name) + for node in self.tree.body if isinstance(node, ast.ImportFrom) + for alias in node.names + } + self.assertIn(("basicpy.basicpy", "FittingMode"), imports) + flatfield = ast.parse((WRAPPER.parent / "flatfield.py").read_text()) + self.assertFalse(any( + isinstance(node, ast.ClassDef) and node.name == "BasicFittingMode" + for node in flatfield.body + )) + + def test_existing_contracts_own_pipeline_axis_admission(self): + decorators = {ast.unparse(node) for node in self.function.decorator_list} + self.assertIn("jax_func(contract=ProcessingContract.PURE_3D)", decorators) + self.assertIn("allowed_group_by(GroupBy.CHANNEL)", decorators) + self.assertIn("required_variable_components(VariableComponents.SITE)", decorators) + self.assertFalse(any( + isinstance(node, ast.FunctionDef) and "batch" in node.name + for node in self.tree.body + )) + + def test_float_output_and_dependency_failure_are_not_hidden(self): + calls = [node for node in ast.walk(self.function) if isinstance(node, ast.Call)] + self.assertFalse(any( + isinstance(node.func, ast.Attribute) and node.func.attr in ("astype", "clip") + for node in calls + )) + self.assertFalse(any(isinstance(node, ast.Try) for node in ast.walk(self.tree))) + self.assertFalse(any( + isinstance(node.func, ast.Name) and node.func.id == "optional_import_placeholder" + for node in ast.walk(self.tree) if isinstance(node, ast.Call) + )) + transform = next( + node for node in calls + if isinstance(node.func, ast.Attribute) and node.func.attr == "fit_transform" + ) + self.assertEqual(ast.unparse(transform.keywords[0].value), "False") + + def test_reviewed_git_pin_and_jax_owned_version_matching(self): + requirement = next( + line for line in (ROOT / "requirements-basicpy.txt").read_text().splitlines() + if line and not line.startswith("#") + ) + self.assertRegex( + requirement, + re.compile(r"^BaSiCPy @ git\+https://github.com/OpenHCSDev/BaSiCPy.git@[0-9a-f]{40}$"), + ) + project = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"] + for extra in ("gpu", "all"): + jax_requirements = [ + dep for dep in project["optional-dependencies"][extra] + if dep.startswith("jax") + ] + self.assertEqual(jax_requirements, ["jax[cuda12-local]>=0.9.2,<0.10"]) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--basicpy-source", type=Path, required=True) + args = parser.parse_args() + BasicPySourceTests.upstream_source = args.basicpy_source + unittest.main(argv=[__file__], verbosity=2) From a61f8bf8261325394bf8a65160df026d24710811 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 16:23:53 -0400 Subject: [PATCH 02/22] Declare same-fit BaSiC fields and typed native output policy --- docs/source/development/basicpy_readiness.md | 101 ++++++++--- external/arraybridge | 2 +- .../backends/enhance/basic_processor_jax.py | 60 ++++++- .../processing/backends/enhance/flatfield.py | 66 ++++++++ requirements-basicpy.txt | 3 + scripts/check_basicpy_source.py | 159 ++++++++++++++---- tests/integration/test_basicpy_real_fit.py | 79 +++++++++ tests/unit/test_fitted_illumination_fields.py | 129 ++++++++++++++ 8 files changed, 535 insertions(+), 64 deletions(-) create mode 100644 tests/integration/test_basicpy_real_fit.py create mode 100644 tests/unit/test_fitted_illumination_fields.py diff --git a/docs/source/development/basicpy_readiness.md b/docs/source/development/basicpy_readiness.md index ee192f4e69..52fe5900b0 100644 --- a/docs/source/development/basicpy_readiness.md +++ b/docs/source/development/basicpy_readiness.md @@ -2,7 +2,9 @@ Status: draft **source integration**, not installed, import- or fit-validated. Backend/dependency owner: Linnaeus. Parent PR151 owns recipes/policy. Memory/ -session PR208 remains independently pending actual live acceptance. +session PR208 remains independently pending actual live acceptance. Its two +independent authority findings are corrected/published in `d69d65adb`; no memory +MCP measurement or installed-history readiness is inferred from that source fix. ## Implementation and dependency source @@ -19,6 +21,18 @@ knob reaches the model. Corrected output remains floating point; no integer recast, clipping or unit-interval conversion is performed by this adapter. `timelapse=False` avoids subtracting the estimated per-observation baseline. +Source tracing found another range boundary: ArrayBridge's outer decorator +defaults to preserving/rescaling into the input dtype on **direct** calls. Removing +the adapter's `astype` alone was therefore insufficient. The paired +[ArrayBridge PR2](https://github.com/OpenHCSDev/ArrayBridge/pull/2) extends the +existing dtype wrapper's typed `dtype_config_default` keyword and reflected +parameter owner; no BaSiC-specific wrapper, registry or bypass is introduced. +OpenHCS declares `DtypeConfig()` (its existing native-output owner) on this +callable. Explicit caller/step dtype policies can still override that default. +The exact ArrayBridge commit `7f94d27925348f0cf0e4cc200692a4214c3722b2` is recorded +in both the gitlink and source requirements. PR2 is stacked on the separate +retained-history PR1; neither is merged or installed here. + The paired [BaSiCPy draft](https://github.com/OpenHCSDev/BaSiCPy/pull/1) reuses Tristan's clean JAX prototype `ae2c647`, rather than PyPI 2.x's PyTorch API. `requirements-basicpy.txt` pins the reviewed fork commit. As with this project's @@ -48,21 +62,25 @@ The existing `PURE_3D`, `required_variable_components(SITE)` and `allowed_group_by(CHANNEL)` declarations require a real cross-site stack and reject a Z-only/mixed-channel FunctionStep. Direct calls accept `(N,Y,X)` or `(N,Z,Y,X)`; they do not split volumes into separate fits. Pipeline acceptance -targets exactly `[SITE]`, with fixed Z/time and channel grouping. Multi-axis -SITE+Z/time grids and time-only pipeline admission are not claimed by this -checkpoint: the existing required-axis declaration checks inclusion, not an -exact alternative independent-axis set. Do not advertise those configurations -as validated. - -Fitted flatfield/darkfield artifacts remain to implement/verify. Existing -`artifact_outputs`, image sidecars, `SourceProjectedImageOutput` and -`ImagePayloadMetadata.collapse_leading_plane_axis()` provide the likely route: -a fitted field is an aggregate of all observations, with contributor provenance, -not one selected source plane. Repeating the field N times or lying about a -plane projection would introduce identity/RAM debt. Core artifact/runtime files -are Zeno-owned; this patch does not edit them or introduce a second artifact -pipeline. Source-only aggregate projection feasibility is being checked before -adding a local behavior-owning output leaf. +targets exactly `[SITE]`, with fixed Z/time and channel grouping. In addition to +the existing required-axis inclusion check, the fitted-field declaration owns +an exact metadata-backed observation-domain check before fitting: retained +varying components must be SITE only. Mixed SITE+Z/time/channel and Z-only stacks +are rejected, not silently flattened into observations. Raw direct-array callers +declare the leading N themselves; absent source metadata is not invented. +Time-only pipeline admission and numerical volume readiness remain unverified. + +The wrapper now returns corrected main flow plus flatfield/darkfield from the +**same real fit**, through existing `artifact_outputs` image sidecars. Local +`FittedIlluminationFieldOutput(SourceProjectedImageOutput)` owns the aggregate +source-context transformation: complete-stack/count/grid admission, all +contributor provenance via the existing metadata owner's +`collapse_leading_plane_axis()`, and clearing pixel-range metadata that does not +describe fitted parameters. It does not select a false source plane or repeat +the field N times. Existing declaration bindings retain group lineage; artifact +names own persisted filenames, with on-demand viewer inspection. No shared +core artifact/runtime/config file or generic registry was changed. Runtime +materialization and compiled/MCP routing still require real acceptance. ## Actual evidence @@ -80,18 +98,38 @@ Executed here, source only: --basicpy-source /home/ts/wt/basicpy-python314-20260929/src/basicpy/basicpy.py ``` -Five checks pass (0.009s): knobs against actual upstream declaration fields, -enum owner, existing pipeline declarations, no integer cast/hidden dependency -failure, reviewed source pin and JAX-owned dependency matching. Focused Ruff and +Initial five checks passed (0.009s); the current expanded seven source checks +pass in 0.018s under Python3.14.7. The new API guard initially selected a local +variable's name instead of the factory's `decorator` declaration; the selector +was corrected without changing an API or weakening a runtime assertion. Checks +cover aggregate field-owner projection and the paired typed-default API in +addition to knobs/enum/contracts/range/source-pin ownership. Focused Ruff and diff checks pass. These checks do not import OpenHCS/JAX or execute a model. -Normal merge of freshly fetched `openhcsdev/main` is up to date at `a0263e82a`; -all recorded submodules are initialized in the isolated worktree. +Normal merge of fetched `openhcsdev/main283b21275` is `2e4409871`; recorded +submodules are initialized, with only the reviewed paired ArrayBridge gitlink +advanced. Parent #151/#212 is now merged/installed; this worker did not install. + +Provider-free numerical/projection regressions are written but **not run**: + +- Nine fitted-field metadata cases retain all contributor paths/common fixed + components and reject mixed/Z-only axes, one-plane context, wrong grid/count. +- Two real BaSiC tests use only 24 synthetic 32x32 same-channel observations: + direct decorated-call floating range and formula/known-shading field inspection, + plus moving-object versus stationary-pattern biological-leakage controls. + No mocked BaSiC, alternate pipeline or blind input/reference is used. +- Paired ArrayBridge has four real NumPy regressions for native fractions, + negative/out-of-uint16 values, explicit override and unchanged legacy defaults. + +Those runtime assertions are prospective checks, not passing evidence. No new +scientific framework import, fit, MCP server or GUI/JVM startup was performed for +this follow-through under Confucius's shared slot. ## Focused antipattern / new-case review Scope: changed wrapper, shared flatfield enum, JAX dependency declaration and paired fork's DCT/Pydantic owners. Focused source/AST/catalog review, not full NRA -scan or global proof. +scan or global proof. Aggregate outputs and direct-call dtype policy are included +in this follow-through's source review. - BOUND-7: removed `hasattr(shape/dtype)` probing of a known array argument. Validate finite observations once, and preserve real model errors. @@ -103,15 +141,22 @@ scan or global proof. optimizer defaults with actual observation semantics and model-owned defaults. - IMPL-13/TIME-1: paired fork reuses the public JAX DCT owner; copied private helper is deleted in place. Pydantic models retain durable schema ownership. -- New output case must extend existing image-output/artifact owners and prove - aggregate provenance, not add metric/artifact dictionaries or special-name - dispatch to generic consumers. This part remains under source review. +- BOUND-2 / IDEN-4: direct-call dtype semantics come from the existing typed + ArrayBridge/OpenHCS config owner, not the adapter's output dtype assertion. + The paired API changes the callable declaration; generic consumers and + explicit overrides remain on the same wrapper and runner family. +- IDEN-4 / BOUND-2: aggregate field identity belongs to its new existing-base + output leaf and the existing metadata projection owner. New fitted outputs + reuse that leaf plus one artifact declaration; no changes to generic runtime, + artifact catalogs, switches, metric dictionaries or registries are needed. + Added projection tests are the new-case experiment, still pending execution. ## Pending scheduled acceptance -Parent #151/#212 gets first released validation slot. This worker waits for -explicit next-slot handoff, then uses nonblocking flock/resource guard. No new -environment/GUI/JVM/runtime, install or active baseline changes happened here. +Parent #151/#212's finite validation/install completed; Confucius now owns the +heavy live slot and frozen installed harness. This worker remains on source work +until the next-slot handoff, then uses nonblocking flock/resource guard. No new +environment/GUI/JVM/runtime, install or active baseline change happens here. 1. Verify imported paths first; real isolated Python3.14 dependency resolve and import against the exact paired source, without an old JAX downgrade. diff --git a/external/arraybridge b/external/arraybridge index fddd9857bc..7f94d27925 160000 --- a/external/arraybridge +++ b/external/arraybridge @@ -1 +1 @@ -Subproject commit fddd9857bcd21468d20bc8547a769e0c6c9f1906 +Subproject commit 7f94d27925348f0cf0e4cc200692a4214c3722b2 diff --git a/openhcs/processing/backends/enhance/basic_processor_jax.py b/openhcs/processing/backends/enhance/basic_processor_jax.py index 5452dedf93..8816f3b37c 100644 --- a/openhcs/processing/backends/enhance/basic_processor_jax.py +++ b/openhcs/processing/backends/enhance/basic_processor_jax.py @@ -7,18 +7,56 @@ from basicpy import BaSiC from basicpy.basicpy import FittingMode -from openhcs.constants.constants import GroupBy, VariableComponents +from openhcs.constants.constants import GroupBy +from openhcs.core.artifacts import ( + ArtifactSidecarRole, + ArtifactSpec, + ArtifactViewerStreaming, + ImageArtifactType, + MainFlowPlaneProjectionOutputSpec, + MainFlowStackOutputSpec, +) +from openhcs.core.config import DtypeConfig from openhcs.core.memory import jax as jax_func from openhcs.core.pipeline.function_contracts import ( allowed_group_by, + artifact_outputs, required_variable_components, ) +from openhcs.processing.backends.enhance.flatfield import FittedIlluminationFieldOutput from openhcs.processing.backends.lib_registry.unified_registry import ProcessingContract +from openhcs.processing.materialization import ( + ImageFileOptions, + MaterializationSpec, + MaterializedFilenameIdentity, +) + + +def _fitted_field_output(name: str) -> ArtifactSpec: + """Project group lineage; runtime field owns aggregate contributor context.""" + return MainFlowPlaneProjectionOutputSpec.output( + name, + ImageArtifactType, + sidecar_role=ArtifactSidecarRole.QA_CHECKPOINT, + materialization=MaterializationSpec( + ImageFileOptions( + filename_suffix=".tif", + filename_identity=MaterializedFilenameIdentity.ARTIFACT_NAME, + ) + ), + viewer_streaming=ArtifactViewerStreaming.ON_DEMAND, + ) + +CORRECTED_OUTPUT = MainFlowStackOutputSpec.output("basic_corrected", ImageArtifactType) +FLATFIELD_OUTPUT = _fitted_field_output("basic_flatfield") +DARKFIELD_OUTPUT = _fitted_field_output("basic_darkfield") -@jax_func(contract=ProcessingContract.PURE_3D) + +@jax_func(contract=ProcessingContract.PURE_3D, dtype_config_default=DtypeConfig()) @allowed_group_by(GroupBy.CHANNEL) -@required_variable_components(VariableComponents.SITE) +@required_variable_components(FittedIlluminationFieldOutput.observation_axis) +@artifact_outputs(CORRECTED_OUTPUT, FLATFIELD_OUTPUT, DARKFIELD_OUTPUT) def basic_flatfield_correction_jax( image: jnp.ndarray, max_iterations: int = 50, @@ -29,7 +67,7 @@ def basic_flatfield_correction_jax( get_darkfield: bool = False, fitting_mode: FittingMode = FittingMode.ladmap, working_size: int | None = 128, -) -> jnp.ndarray: +) -> tuple[jnp.ndarray, FittedIlluminationFieldOutput, FittedIlluminationFieldOutput]: """Fit one BaSiC model to independent observations and apply its fields. The leading N axis contains independent timepoints or mosaic positions, @@ -53,7 +91,11 @@ def basic_flatfield_correction_jax( working_size: Spatial working size, or None for no rescaling. Returns: - Floating-point corrected observations, with the input shape. BaSiC's + Corrected observations, fitted flatfield and fitted darkfield, all from + this same fit. The first image remains the pipeline's main flow; fields + are persisted image sidecars, available for on-demand inspection. When + get_darkfield=False the darkfield is the model's zero additive field. + Floating-point corrected observations retain the input shape. BaSiC's (image - darkfield) / flatfield retains intensity units, fractions and negative values; no clipping, normalization, integer recast or temporal baseline subtraction is applied. Values can exceed the input range. @@ -65,6 +107,7 @@ def basic_flatfield_correction_jax( raise ValueError("BaSiC requires multiple independent observations.") if not np.isfinite(observations).all(): raise ValueError("BaSiC observations must contain only finite values.") + FittedIlluminationFieldOutput.validate_observation_domain(image) model = BaSiC( max_iterations=max_iterations, @@ -77,4 +120,9 @@ def basic_flatfield_correction_jax( working_size=working_size, ) corrected = model.fit_transform(observations, timelapse=False) - return jnp.asarray(corrected) + observation_count = observations.shape[0] + return ( + jnp.asarray(corrected), + FittedIlluminationFieldOutput(jnp.asarray(model.flatfield), observation_count), + FittedIlluminationFieldOutput(jnp.asarray(model.darkfield), observation_count), + ) diff --git a/openhcs/processing/backends/enhance/flatfield.py b/openhcs/processing/backends/enhance/flatfield.py index 58c541db00..a594e06026 100644 --- a/openhcs/processing/backends/enhance/flatfield.py +++ b/openhcs/processing/backends/enhance/flatfield.py @@ -1,6 +1,14 @@ """Nominal contracts shared by flat-field correction implementations.""" +from dataclasses import dataclass, replace from enum import Enum +from typing import ClassVar + +from openhcs.constants.constants import AllComponents, VariableComponents +from openhcs.core.projected_image_output import SourceProjectedImageOutput +from openhcs.core.runtime_array_values import RuntimeArrayData +from openhcs.core.runtime_image_values import image_payload_metadata +from openhcs.core.runtime_plane_projection import RuntimePlaneAxisValueProjection class FlatfieldCorrectionMode(Enum): @@ -8,3 +16,61 @@ class FlatfieldCorrectionMode(Enum): DIVIDE = "divide" SUBTRACT = "subtract" + + +@dataclass(frozen=True) +class FittedIlluminationFieldOutput(SourceProjectedImageOutput): + """One fitted field, owned by all independent observations, not one plane.""" + + observation_axis: ClassVar[VariableComponents] = VariableComponents.SITE + data: RuntimeArrayData + observation_count: int + + def __post_init__(self) -> None: + if type(self.observation_count) is not int or self.observation_count < 2: + raise ValueError("A fitted field requires multiple observations.") + if self.data.ndim not in (2, 3): + raise ValueError("A fitted field requires a spatial image or volume.") + + @classmethod + def validate_observation_domain(cls, source: RuntimeArrayData) -> None: + """Reject mislabeled metadata-backed ensembles before fitting.""" + metadata = image_payload_metadata(source) + if not metadata.has_values: + # Direct array callers declare N themselves; no source identity to infer. + return + axes = tuple( + AllComponents.from_value(name) + for name in metadata.retained_plane_component_values() + ) + declared_axis = AllComponents.from_value(cls.observation_axis.value) + if axes != (declared_axis,): + raise ValueError( + f"BaSiC requires independent {cls.observation_axis.name} observations " + "with every other source component fixed." + ) + + def with_data(self, data: RuntimeArrayData) -> "FittedIlluminationFieldOutput": + return replace(self, data=data) + + def resolve_source_context( + self, + source: RuntimeArrayData, + projection: RuntimePlaneAxisValueProjection | None, + ) -> RuntimeArrayData: + if projection is None or projection.plane_index is not None: + raise ValueError("A fitted field requires the complete observation stack.") + if projection.axis_size != self.observation_count: + raise ValueError("Fitted field observation count differs from its source.") + if source.shape != (self.observation_count, *self.data.shape): + raise ValueError("A fitted field must retain the source spatial grid.") + self.validate_observation_domain(source) + metadata = image_payload_metadata(source).collapse_leading_plane_axis() + # Model parameters are not unit-interval input pixels. Contributor source + # dtypes/identities remain in provenance, not on the fitted field's range. + metadata = metadata.replace_fields( + intensity_scale=None, + source_dtype=None, + unit_interval_intensity=None, + ) + return metadata.payload_with(self.data, None) diff --git a/requirements-basicpy.txt b/requirements-basicpy.txt index a7bdd299d5..e8ad9b027d 100644 --- a/requirements-basicpy.txt +++ b/requirements-basicpy.txt @@ -2,3 +2,6 @@ # Do not replace with PyPI 2.x: it owns a different PyTorch API. # Source checkpoint; import/fit/compiled MCP acceptance remains pending. BaSiCPy @ git+https://github.com/OpenHCSDev/BaSiCPy.git@29f0a327d2069324f13d0c6fa0809a5c7566a6d4 +# Typed native direct-call default, stacked on reviewed retained-history fix. +# https://github.com/OpenHCSDev/ArrayBridge/pull/2 (depends on PR1) +arraybridge @ git+https://github.com/OpenHCSDev/ArrayBridge.git@7f94d27925348f0cf0e4cc200692a4214c3722b2 diff --git a/scripts/check_basicpy_source.py b/scripts/check_basicpy_source.py index 9138d053e3..2bffb19578 100644 --- a/scripts/check_basicpy_source.py +++ b/scripts/check_basicpy_source.py @@ -19,23 +19,28 @@ class BasicPySourceTests(unittest.TestCase): def setUp(self): self.tree = ast.parse(WRAPPER.read_text()) self.function = next( - node for node in self.tree.body + node + for node in self.tree.body if isinstance(node, ast.FunctionDef) and node.name == "basic_flatfield_correction_jax" ) def test_every_exposed_knob_reaches_real_model_owner(self): owner = next( - node for node in ast.parse(self.upstream_source.read_text()).body + node + for node in ast.parse(self.upstream_source.read_text()).body if isinstance(node, ast.ClassDef) and node.name == "BaSiC" ) fields = { - node.target.id for node in owner.body + node.target.id + for node in owner.body if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name) } call = next( - node for node in ast.walk(self.function) - if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) + node + for node in ast.walk(self.function) + if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) and node.func.id == "BaSiC" ) parameters = {arg.arg for arg in self.function.args.args[1:]} @@ -48,60 +53,156 @@ def test_every_exposed_knob_reaches_real_model_owner(self): def test_choices_belong_to_upstream_not_a_mirror(self): imports = { (node.module, alias.name) - for node in self.tree.body if isinstance(node, ast.ImportFrom) + for node in self.tree.body + if isinstance(node, ast.ImportFrom) for alias in node.names } self.assertIn(("basicpy.basicpy", "FittingMode"), imports) flatfield = ast.parse((WRAPPER.parent / "flatfield.py").read_text()) - self.assertFalse(any( - isinstance(node, ast.ClassDef) and node.name == "BasicFittingMode" - for node in flatfield.body - )) + self.assertFalse( + any( + isinstance(node, ast.ClassDef) and node.name == "BasicFittingMode" + for node in flatfield.body + ) + ) def test_existing_contracts_own_pipeline_axis_admission(self): decorators = {ast.unparse(node) for node in self.function.decorator_list} - self.assertIn("jax_func(contract=ProcessingContract.PURE_3D)", decorators) + self.assertIn( + "jax_func(contract=ProcessingContract.PURE_3D, dtype_config_default=DtypeConfig())", + decorators, + ) self.assertIn("allowed_group_by(GroupBy.CHANNEL)", decorators) - self.assertIn("required_variable_components(VariableComponents.SITE)", decorators) - self.assertFalse(any( - isinstance(node, ast.FunctionDef) and "batch" in node.name - for node in self.tree.body - )) + self.assertIn( + "required_variable_components(FittedIlluminationFieldOutput.observation_axis)", + decorators, + ) + self.assertFalse( + any( + isinstance(node, ast.FunctionDef) and "batch" in node.name + for node in self.tree.body + ) + ) def test_float_output_and_dependency_failure_are_not_hidden(self): calls = [node for node in ast.walk(self.function) if isinstance(node, ast.Call)] - self.assertFalse(any( - isinstance(node.func, ast.Attribute) and node.func.attr in ("astype", "clip") - for node in calls - )) + self.assertFalse( + any( + isinstance(node.func, ast.Attribute) + and node.func.attr in ("astype", "clip") + for node in calls + ) + ) self.assertFalse(any(isinstance(node, ast.Try) for node in ast.walk(self.tree))) - self.assertFalse(any( - isinstance(node.func, ast.Name) and node.func.id == "optional_import_placeholder" - for node in ast.walk(self.tree) if isinstance(node, ast.Call) - )) + self.assertFalse( + any( + isinstance(node.func, ast.Name) + and node.func.id == "optional_import_placeholder" + for node in ast.walk(self.tree) + if isinstance(node, ast.Call) + ) + ) transform = next( - node for node in calls - if isinstance(node.func, ast.Attribute) and node.func.attr == "fit_transform" + node + for node in calls + if isinstance(node.func, ast.Attribute) + and node.func.attr == "fit_transform" ) self.assertEqual(ast.unparse(transform.keywords[0].value), "False") + def test_fields_extend_existing_projection_and_metadata_owners(self): + tree = ast.parse((WRAPPER.parent / "flatfield.py").read_text()) + owner = next( + node + for node in tree.body + if isinstance(node, ast.ClassDef) + and node.name == "FittedIlluminationFieldOutput" + ) + self.assertEqual( + [ast.unparse(base) for base in owner.bases], ["SourceProjectedImageOutput"] + ) + calls = [ + ast.unparse(node.func) + for node in ast.walk(owner) + if isinstance(node, ast.Call) + ] + self.assertIn( + "image_payload_metadata(source).collapse_leading_plane_axis", calls + ) + self.assertIn("metadata.retained_plane_component_values", calls) + self.assertFalse(any(call.endswith((".repeat", ".tile")) for call in calls)) + decorator = next( + node + for node in self.function.decorator_list + if isinstance(node, ast.Call) + and ast.unparse(node.func) == "artifact_outputs" + ) + self.assertEqual( + [ast.unparse(arg) for arg in decorator.args], + ["CORRECTED_OUTPUT", "FLATFIELD_OUTPUT", "DARKFIELD_OUTPUT"], + ) + def test_reviewed_git_pin_and_jax_owned_version_matching(self): requirement = next( - line for line in (ROOT / "requirements-basicpy.txt").read_text().splitlines() + line + for line in (ROOT / "requirements-basicpy.txt").read_text().splitlines() if line and not line.startswith("#") ) self.assertRegex( requirement, - re.compile(r"^BaSiCPy @ git\+https://github.com/OpenHCSDev/BaSiCPy.git@[0-9a-f]{40}$"), + re.compile( + r"^BaSiCPy @ git\+https://github.com/OpenHCSDev/BaSiCPy.git@[0-9a-f]{40}$" + ), ) project = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"] for extra in ("gpu", "all"): jax_requirements = [ - dep for dep in project["optional-dependencies"][extra] + dep + for dep in project["optional-dependencies"][extra] if dep.startswith("jax") ] self.assertEqual(jax_requirements, ["jax[cuda12-local]>=0.9.2,<0.10"]) + def test_native_default_reaches_existing_arraybridge_owner(self): + decorators = ast.parse( + (ROOT / "external/arraybridge/src/arraybridge/decorators.py").read_text() + ) + owner = next( + node + for node in decorators.body + if isinstance(node, ast.FunctionDef) + and node.name == "_create_memory_decorator" + ) + declaration = next( + node + for node in ast.walk(owner) + if isinstance(node, ast.FunctionDef) and node.name == "decorator" + ) + self.assertIn( + "dtype_config_default", + {argument.arg for argument in declaration.args.kwonlyargs}, + ) + wrapper = next( + node + for node in decorators.body + if isinstance(node, ast.FunctionDef) + and node.name == "wrap_dtype_preserving_callable" + ) + self.assertIn( + "dtype_config_default", + {argument.arg for argument in wrapper.args.kwonlyargs}, + ) + requirements = (ROOT / "requirements-basicpy.txt").read_text().splitlines() + self.assertTrue( + any( + re.fullmatch( + r"arraybridge @ git\+https://github.com/OpenHCSDev/ArrayBridge.git@[0-9a-f]{40}", + requirement, + ) + for requirement in requirements + ) + ) + if __name__ == "__main__": parser = argparse.ArgumentParser(description=__doc__) diff --git a/tests/integration/test_basicpy_real_fit.py b/tests/integration/test_basicpy_real_fit.py new file mode 100644 index 0000000000..d25b1033eb --- /dev/null +++ b/tests/integration/test_basicpy_real_fit.py @@ -0,0 +1,79 @@ +"""Real, bounded BaSiC fit/range/field checks; compiled MCP acceptance is separate.""" + +import numpy as np +import pytest + +pytest.importorskip("basicpy", reason="requires reviewed paired BaSiCPy source") + +from openhcs.processing.backends.enhance.basic_processor_jax import ( + basic_flatfield_correction_jax, +) + + +def shaded_observations(*, stationary_biology=False): + """Synthetic acquisition only: no biological data, reference or downloaded image.""" + size = 32 + count = 24 + yy, xx = np.mgrid[-1 : 1 : complex(size), -1 : 1 : complex(size)] + flatfield = 1.2 - 0.3 * (yy**2 + xx**2) + 0.1 * xx + flatfield /= flatfield.mean() + rng = np.random.default_rng(213) + observations = [] + for index in range(count): + background = 3000 + 50 * index + if stationary_biology: + # A stationary, background-scaled biological pattern is fundamentally + # confounded with multiplicative shading, not a convergence failure. + biology = background * np.exp(-((xx - 0.2) ** 2 + (yy + 0.1) ** 2) / 0.09) + else: + cx, cy = rng.uniform(-0.8, 0.8, size=2) + biology = 5000 * np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / 0.008) + frame = (background + biology) * flatfield + observations.append(np.rint(frame).astype(np.uint16)) + return np.stack(observations), flatfield + + +def run_fit(observations): + return basic_flatfield_correction_jax( + observations, + max_iterations=100, + working_size=None, + get_darkfield=False, + ) + + +def test_real_fit_keeps_measurement_range_and_returns_same_fit_fields(): + observations, truth = shaded_observations() + corrected, flatfield, darkfield = run_fit(observations) + corrected = np.asarray(corrected) + flatfield_data = np.asarray(flatfield) + darkfield_data = np.asarray(darkfield) + assert np.issubdtype(corrected.dtype, np.floating) + assert corrected.shape == observations.shape + assert flatfield.shape == darkfield.shape == observations.shape[1:] + assert ( + flatfield.observation_count + == darkfield.observation_count + == observations.shape[0] + ) + assert np.isfinite(corrected).all() + np.testing.assert_allclose(darkfield_data, 0, atol=1e-6) + np.testing.assert_allclose( + corrected, + (observations.astype(np.float32) - darkfield_data) / flatfield_data, + rtol=1e-5, + atol=1e-3, + ) + assert np.sqrt(np.mean((flatfield_data - truth) ** 2)) < 0.15 + assert np.any(corrected != np.floor(corrected)) + + +def test_stationary_biology_negative_control_exposes_flatfield_leakage(): + moving, truth = shaded_observations() + stationary, _ = shaded_observations(stationary_biology=True) + _, moving_field, _ = run_fit(moving) + _, stationary_field, _ = run_fit(stationary) + moving_error = np.sqrt(np.mean((np.asarray(moving_field) - truth) ** 2)) + stationary_error = np.sqrt(np.mean((np.asarray(stationary_field) - truth) ** 2)) + # A successful fit is not evidence that stationary signal is shading-free. + assert stationary_error > moving_error + 0.02 diff --git a/tests/unit/test_fitted_illumination_fields.py b/tests/unit/test_fitted_illumination_fields.py new file mode 100644 index 0000000000..e0498ac977 --- /dev/null +++ b/tests/unit/test_fitted_illumination_fields.py @@ -0,0 +1,129 @@ +"""Real source-owner projection of one field fitted from all observations.""" + +import numpy as np +import pytest + +from openhcs.core.runtime_image_values import ( + ImagePayloadMetadata, + image_payload_metadata, +) +from openhcs.core.runtime_plane_projection import ( + RuntimePlaneAxis, + RuntimePlaneAxisValueProjection, +) +from openhcs.core.source_image_provenance import SourceImageProvenancePlanes +from openhcs.processing.backends.enhance.flatfield import FittedIlluminationFieldOutput + + +def observation_source(components): + count = len(components) + metadata = ImagePayloadMetadata( + plane_axis=RuntimePlaneAxis.RUNTIME_SLICE, + intensity_scale=65535.0, + source_dtype="uint16", + source_image_provenance_planes=SourceImageProvenancePlanes.from_components( + paths=tuple(f"/synthetic/input/observation_{i}.tif" for i in range(count)), + component_metadata=tuple(components), + ), + ) + return metadata.payload_with(np.zeros((count, 8, 9), dtype=np.uint16)) + + +def cross_site_source(): + return observation_source( + [ + {"well": "A01", "site": str(i), "channel": "1", "z_index": "1"} + for i in range(3) + ] + ) + + +def test_field_collapses_pixels_not_contributor_history(): + source = cross_site_source() + field = FittedIlluminationFieldOutput(np.full((8, 9), 1.25, dtype=np.float32), 3) + result = field.resolve_source_context( + source, + RuntimePlaneAxisValueProjection.preserve( + axis=RuntimePlaneAxis.RUNTIME_SLICE, + axis_size=3, + ), + ) + metadata = image_payload_metadata(result) + assert result.shape == (8, 9) + assert metadata.plane_axis is None + assert len(metadata.source_image_paths) == 3 + assert metadata.source_image_provenance_planes.contributor_count == 3 + assert metadata.source_component_metadata == { + "well": "A01", + "channel": "1", + "z_index": "1", + } + assert metadata.intensity_scale is None + assert metadata.source_dtype is None + np.testing.assert_array_equal(np.asarray(result), np.full((8, 9), 1.25)) + + +@pytest.mark.parametrize("changing", ["z_index", "channel", "timepoint"]) +def test_non_observation_axis_variation_is_rejected_before_fit(changing): + components = [ + {"well": "A01", "site": str(i), "channel": "1", "z_index": "1"} + for i in range(3) + ] + for index, component in enumerate(components): + component[changing] = str(index) + with pytest.raises(ValueError, match="every other source component fixed"): + FittedIlluminationFieldOutput.validate_observation_domain( + observation_source(components) + ) + + +def test_z_only_stack_is_not_an_observation_ensemble(): + source = observation_source( + [ + {"well": "A01", "site": "1", "channel": "1", "z_index": str(i)} + for i in range(3) + ] + ) + with pytest.raises(ValueError, match="independent SITE observations"): + FittedIlluminationFieldOutput.validate_observation_domain(source) + + +@pytest.mark.parametrize( + "projection", + [ + None, + RuntimePlaneAxisValueProjection.from_selected_plane( + axis=RuntimePlaneAxis.RUNTIME_SLICE, + plane_index=0, + axis_size=3, + ), + ], +) +def test_field_rejects_one_plane_as_aggregate_source(projection): + field = FittedIlluminationFieldOutput(np.ones((8, 9), dtype=np.float32), 3) + with pytest.raises(ValueError, match="complete observation stack"): + field.resolve_source_context(cross_site_source(), projection) + + +def test_field_keeps_observation_count_through_real_array_conversion(): + field = FittedIlluminationFieldOutput(np.ones((8, 9), dtype=np.float32), 3) + converted = field.with_data(np.asarray(field, dtype=np.float64)) + assert converted.observation_count == 3 + assert converted.dtype == np.float64 + np.testing.assert_array_equal(np.asarray(converted), np.asarray(field)) + + +def test_field_rejects_wrong_grid_or_observation_count(): + source = cross_site_source() + projection = RuntimePlaneAxisValueProjection.preserve( + axis=RuntimePlaneAxis.RUNTIME_SLICE, + axis_size=3, + ) + with pytest.raises(ValueError, match="spatial grid"): + FittedIlluminationFieldOutput(np.zeros((4, 9)), 3).resolve_source_context( + source, projection + ) + with pytest.raises(ValueError, match="observation count"): + FittedIlluminationFieldOutput(np.zeros((8, 9)), 4).resolve_source_context( + source, projection + ) From 432c6e037ca24c8d66cefc0badbef0d7ccfa5df8 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 16:24:44 -0400 Subject: [PATCH 03/22] Record BaSiC follow-through and current-main source evidence --- docs/source/development/basicpy_readiness.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/development/basicpy_readiness.md b/docs/source/development/basicpy_readiness.md index 52fe5900b0..872db7ab21 100644 --- a/docs/source/development/basicpy_readiness.md +++ b/docs/source/development/basicpy_readiness.md @@ -105,9 +105,11 @@ was corrected without changing an API or weakening a runtime assertion. Checks cover aggregate field-owner projection and the paired typed-default API in addition to knobs/enum/contracts/range/source-pin ownership. Focused Ruff and diff checks pass. These checks do not import OpenHCS/JAX or execute a model. -Normal merge of fetched `openhcsdev/main283b21275` is `2e4409871`; recorded -submodules are initialized, with only the reviewed paired ArrayBridge gitlink -advanced. Parent #151/#212 is now merged/installed; this worker did not install. +Normal merge of fetched `openhcsdev/main283b21275` is `2e4409871`. The source +follow-through is `a61f8bf82`, followed by normal merge `3938002b9` of latest +main `de23449a4` (#219, outside this surface). Recorded submodules are initialized, +with only the reviewed paired ArrayBridge gitlink advanced. Parent #151/#212 is +now merged/installed; this worker did not install. Provider-free numerical/projection regressions are written but **not run**: From 08d7d3acc2287619e0278df82e94d021169177c0 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 14:50:02 -0400 Subject: [PATCH 04/22] Record paired BaSiCPy numerical acceptance and remaining ownership gate --- docs/source/development/basicpy_readiness.md | 10 ++- .../checkpoint.rst | 76 +++++++++++++++++++ .../run-source-check.sh | 34 +++++++++ requirements-basicpy.txt | 6 +- 4 files changed, 121 insertions(+), 5 deletions(-) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst create mode 100644 docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh diff --git a/docs/source/development/basicpy_readiness.md b/docs/source/development/basicpy_readiness.md index 872db7ab21..77a323a11f 100644 --- a/docs/source/development/basicpy_readiness.md +++ b/docs/source/development/basicpy_readiness.md @@ -1,7 +1,13 @@ # BaSiCPy readiness checkpoint (#213) -Status: draft **source integration**, not installed, import- or fit-validated. -Backend/dependency owner: Linnaeus. Parent PR151 owns recipes/policy. Memory/ +Current parent checkpoint: real BaSiCPy Python3.14 fits and paired OpenHCS +Python3.12 CPU fits, dtype behavior and field provenance checks passed. See the +canonical [2026-09-30 receipt](../../../validation/basicpy_parent_numeric_20260930/checkpoint.rst). +No install, compiled/MCP execution or biological acceptance is claimed. + +The remaining text records the historical 2026-09-29 source-only checkpoint; +its pending checks and owners must not be read as today's runtime state. +Original backend/dependency owner: Linnaeus. Parent PR151 owns recipes/policy. Memory/ session PR208 remains independently pending actual live acceptance. Its two independent authority findings are corrected/published in `d69d65adb`; no memory MCP measurement or installed-history readiness is inferred from that source fix. diff --git a/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst b/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst new file mode 100644 index 0000000000..0c68e7753e --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst @@ -0,0 +1,76 @@ +BaSiCPy parent numerical integration +==================================== + +Integration owner: parent, 2026-09-30, existing PR217. Original Linnaeus +worktrees are unchanged. Normal current-main merge incorporates +d3a99c0d46dea979bba3f9076da87386e49cbed3. The sole merge conflict was the +ArrayBridge gitlink: paired PR2 was normally merged with its own current main +409b1e0, producing d6d92a716b91f0148dc2c04f9e744bddb7e32ed4. This preserves +both the current main changes and the published typed callable dtype default. +No source file conflict, reset, rebase, force push or installed change. + +Published BaSiCPy fork checkpoint is855af6b10c87e832cb6a6594fb288c28d74f46e5. +Requirements and gitlink pin the exact paired published versions. +The fork's real Python3.14.7 source-import run has6PASS in13.96s pytest, +15.97s process,690508KiB peakRSS, zero swaps: public inverseDCT/SciPy +comparison, 2D/3D fit/correction/profile roundtrip and stationary signal +negative control. Canonical fork receipt: docs/python314-numeric-20260930.rst. + +Paired OpenHCS source checks +--------------------------- + +Original attempt failed collection because this fresh source worktree had +no ``openhcs.core._tabular_native`` build. That failure's original log/XML is +retained, not called a production regression or passing test. Existing declared +``setup.py build_ext --inplace`` built the extensions in3.51s,144852KiB peakRSS, +exit0, no install or download. Re-run through run-source-check.sh: + +15PASS,9.20s pytest,11.93s process,1026844KiB peakRSS, zero swaps, exit0. +No skips or deselections. Two warnings concern disabled async pytest plugin +configuration. Tests cover actual BaSiCPy fit-transform through OpenHCS's +real decorator, floating correction units, same-fit field values, stationary +signal confounding, aggregate source contributors and component metadata, +invalid Z/channel/time variation and partial projections, field conversion, +grid/count rejection, native direct-call dtype defaults, explicit overrides, +unchanged preserve-input behavior and untyped default refusal. + +Actual source paths for OpenHCS, ArrayBridge, BaSiCPy and native2c68 were +asserted. Existing shared Python3.12.3 environment and installed remaining +dependencies were used; CPU-only, one affinity CPU, thread1, isolated scratch, +shared Fiji cache with download explicitly false. No JVM, GUI, blind data, +new environment, package install or provider run. Real app import comes first +so existing OpenHCS metadata configuration owns initialization. + +Source ownership +---------------- + +BOUND-2: model/DCT/dtype/source-projection declarations remain the authorities; +no copied schema or numeric algorithm. BOUND-7: new fork tests access declared +model attributes directly. IMPL-13/TIME-1: the prior private inverse-DCT helper +remains deleted, not revived as a fallback. This checkpoint changes numerical +tests, receipts and exact paired requirements/gitlinks; no extra production +dispatch, alias, store or compatibility reader is introduced. + +This is focused source review and executed numerical behavior, not a complete +NRA/global ownership proof. Original logs/XML/build evidence and the original +ratchet invocation error (absolute --root is disallowed) are preserved in +``/home/ts/wt/openhcs-issue-batch-20260929/basicpy-parent-numeric-20260930``. +The corrected original packaged guard uses repository-relative ``--root openhcs``. + +That guard is NOT passing:5122 compared metrics, two positive metric keys: +TypeIdentity+1 and ForeignAbsenceProbe in flatfield.py+2. Original raw record +is openhcs-ratchet-corrected.json. These are existing PR217 additions, not +hidden by accepting the numerical tests: exact-int validation in fitted-field +construction and caller-side metadata/projection predicates need source-backed +ownership review. Parent owns that follow-through within this PR. No waiver, +copied detector, changed baseline or global proof claim. Current-main source +integration is a visible draft checkpoint, not ready for a main merge. + +Remaining boundaries +-------------------- + +No installed user entrypoint, compiled/MCP execution, GPU, fitted additive +darkfield or biological result is accepted. Retain the paired default policy +and SOURCE/field projection for serialized installed execution plus artifact +readback before declaring application readiness. Socrates owns ACK-routing; +parent retains final runtime integration. Hosted CI is not the waiting gate. diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh b/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh new file mode 100644 index 0000000000..3044087885 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh @@ -0,0 +1,34 @@ +#!/bin/sh +set -eu +export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src +export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 OPENHCS_CPU_ONLY=true +export JAX_PLATFORMS=cpu OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 +export NUMBA_NUM_THREADS=1 XLA_FLAGS=--xla_cpu_multi_thread_eigen=false QT_QPA_PLATFORM=offscreen +export POLYSTORE_IMAGEJ_CACHE_ROOT=/home/ts/.cache/polystore/imagej +export POLYSTORE_IMAGEJ_ALLOW_DOWNLOAD=false +export XDG_CACHE_HOME=/home/ts/.cache/agent-scratch/basicpy-parent-numeric-20260930 +export TMPDIR="$XDG_CACHE_HOME" +cd /home/ts/wt/openhcs-basicpy-parent-integration-20260930 +exec timeout 60s /usr/bin/time -v /home/ts/code/projects/openhcs/.venv/bin/python -B -c ' +import os +from pathlib import Path +os.sched_setaffinity(0, {min(os.sched_getaffinity(0))}) +import openhcs, arraybridge, basicpy, zmqruntime +for module, directory in ( + (openhcs, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/openhcs"), + (arraybridge, "/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src/arraybridge"), + (basicpy, "/home/ts/wt/basicpy-parent-numeric-20260930/src/basicpy"), + (zmqruntime, "/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src/zmqruntime"), +): + assert Path(module.__file__).resolve().is_relative_to(Path(directory)) + print("SOURCE", module.__file__, flush=True) +import pytest +raise SystemExit(pytest.main([ + "--noconftest", "-p", "no:cacheprovider", "-o", "addopts=", + "tests/unit/test_fitted_illumination_fields.py", + "tests/integration/test_basicpy_real_fit.py", + "/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/tests/test_callable_dtype_default.py", + "-q", + "--junitxml=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-parent-numeric-20260930/paired-source-native-built.xml", +])) +' diff --git a/requirements-basicpy.txt b/requirements-basicpy.txt index e8ad9b027d..92609ee815 100644 --- a/requirements-basicpy.txt +++ b/requirements-basicpy.txt @@ -1,7 +1,7 @@ # Reviewed JAX prototype, paired draft https://github.com/OpenHCSDev/BaSiCPy/pull/1 # Do not replace with PyPI 2.x: it owns a different PyTorch API. -# Source checkpoint; import/fit/compiled MCP acceptance remains pending. -BaSiCPy @ git+https://github.com/OpenHCSDev/BaSiCPy.git@29f0a327d2069324f13d0c6fa0809a5c7566a6d4 +# CPU source import/fit validated; installed/compiled MCP acceptance remains pending. +BaSiCPy @ git+https://github.com/OpenHCSDev/BaSiCPy.git@855af6b10c87e832cb6a6594fb288c28d74f46e5 # Typed native direct-call default, stacked on reviewed retained-history fix. # https://github.com/OpenHCSDev/ArrayBridge/pull/2 (depends on PR1) -arraybridge @ git+https://github.com/OpenHCSDev/ArrayBridge.git@7f94d27925348f0cf0e4cc200692a4214c3722b2 +arraybridge @ git+https://github.com/OpenHCSDev/ArrayBridge.git@d6d92a716b91f0148dc2c04f9e744bddb7e32ed4 From e0fbf1b062ed0d73952b1aaae8347c788992d610 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 15:02:03 -0400 Subject: [PATCH 05/22] Keep ensemble and complete-axis admission on existing source owners --- .../run-owner-check.sh | 37 ++++++++++++++++++ openhcs/core/projected_image_output.py | 7 ++-- openhcs/core/runtime_image_values.py | 39 +++++++------------ openhcs/core/runtime_plane_projection.py | 20 ++++++++++ .../processing/backends/enhance/flatfield.py | 24 ++++-------- scripts/check_basicpy_source.py | 17 +++++++- tests/unit/test_fitted_illumination_fields.py | 2 +- tests/unit/test_projected_image_output.py | 13 +++++++ 8 files changed, 112 insertions(+), 47 deletions(-) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh b/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh new file mode 100644 index 0000000000..af845c5581 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh @@ -0,0 +1,37 @@ +#!/bin/sh +set -eu +export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src +export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 OPENHCS_CPU_ONLY=true +export JAX_PLATFORMS=cpu OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 +export NUMBA_NUM_THREADS=1 XLA_FLAGS=--xla_cpu_multi_thread_eigen=false QT_QPA_PLATFORM=offscreen +export POLYSTORE_IMAGEJ_CACHE_ROOT=/home/ts/.cache/polystore/imagej +export POLYSTORE_IMAGEJ_ALLOW_DOWNLOAD=false +export XDG_CACHE_HOME=/home/ts/.cache/agent-scratch/basicpy-owner-closure-20260930 +export TMPDIR="$XDG_CACHE_HOME" +cd /home/ts/wt/openhcs-basicpy-parent-integration-20260930 +exec timeout 60s /usr/bin/time -v /home/ts/code/projects/openhcs/.venv/bin/python -B -c ' +import os +from pathlib import Path +os.sched_setaffinity(0, {min(os.sched_getaffinity(0))}) +import openhcs, arraybridge, basicpy, zmqruntime +for module, directory in ( + (openhcs, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/openhcs"), + (arraybridge, "/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src/arraybridge"), + (basicpy, "/home/ts/wt/basicpy-parent-numeric-20260930/src/basicpy"), + (zmqruntime, "/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src/zmqruntime"), +): + assert Path(module.__file__).resolve().is_relative_to(Path(directory)) + print("SOURCE", module.__file__, flush=True) +import pytest +raise SystemExit(pytest.main([ + "--noconftest", "-p", "no:cacheprovider", "-o", "addopts=", "-q", + "tests/unit/test_projected_image_output.py", + "tests/unit/test_fitted_illumination_fields.py", + "tests/integration/test_basicpy_real_fit.py", + "tests/unit/test_derived_runtime_image_provenance.py", + "tests/unit/test_function_runtime_source_projection.py", + "tests/unit/test_image_output_context_projection_proof.py", + "tests/unit/test_runtime_image_singleton_alignment.py", + "--junitxml=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-owner-closure-20260930/source-tests.xml", +])) +' diff --git a/openhcs/core/projected_image_output.py b/openhcs/core/projected_image_output.py index 87ee5f318f..f9fbc33c54 100644 --- a/openhcs/core/projected_image_output.py +++ b/openhcs/core/projected_image_output.py @@ -52,10 +52,9 @@ def resolve_source_context( source: RuntimeArrayData, projection: RuntimePlaneAxisValueProjection | None, ) -> RuntimeArrayData: - if projection is None or projection.plane_index is not None: - raise ValueError( - "Selecting source planes requires a complete input stack projection." - ) + projection = RuntimePlaneAxisValueProjection.require_complete_projection( + projection, value_name="Selecting source planes" + ) if any(index >= projection.axis_size for index in self.source_indices): raise ValueError("Selected source plane is outside the input stack.") source_metadata = image_payload_metadata(source) diff --git a/openhcs/core/runtime_image_values.py b/openhcs/core/runtime_image_values.py index 222a33f813..43685c27a4 100644 --- a/openhcs/core/runtime_image_values.py +++ b/openhcs/core/runtime_image_values.py @@ -24,7 +24,7 @@ from openhcs.serialization.json import to_jsonable from collections.abc import Mapping -from openhcs.constants.constants import AllComponents +from openhcs.constants.constants import AllComponents, VariableComponents from openhcs.core.alias_property import AliasProperty from openhcs.core.runtime_array_values import ( DataBackedRuntimeArrayPayload, @@ -184,31 +184,22 @@ def retained_plane_component_values( tuple(AllComponents) ) - @classmethod - def from_mapping(cls, values: Mapping[str, object]) -> "ImagePayloadMetadata": - """Restore all declared metadata fields through their canonical codecs.""" - decoded = dict(values) - if "source_provenance" in decoded: - decoded["source_provenance"] = SourceImageProvenance.from_mapping( - decoded["source_provenance"] - ) - return dataclass_from_mapping(cls, decoded) - - def retained_plane_component_values( - self, - ) -> dict[str, tuple[SourceMetadataScalar, ...]]: - """Derive varying source coordinates of the retained nominal plane axis. + def require_independent_observation_axis( + self, axis: VariableComponents, *, value_name: str + ) -> None: + """Admit one source-coordinate ensemble without inventing raw-array identity.""" - Source provenance can also describe contributors after a projection. - Only a retained plane-axis declaration makes those coordinates a pixel - axis; artifact storage/grouping axes do not declare that image domain. - """ - - if self.plane_axis is None: - return {} - return self.source_provenance.varying_plane_component_values( - tuple(AllComponents) + if not self.has_values: + return + retained_axes = tuple( + AllComponents.from_value(name) + for name in self.retained_plane_component_values() ) + if retained_axes != (AllComponents.from_value(axis.value),): + raise ValueError( + f"{value_name} requires independent {axis.name} observations " + "with every other source component fixed." + ) @classmethod def for_array( diff --git a/openhcs/core/runtime_plane_projection.py b/openhcs/core/runtime_plane_projection.py index 24602e1994..79cec16cf2 100644 --- a/openhcs/core/runtime_plane_projection.py +++ b/openhcs/core/runtime_plane_projection.py @@ -348,6 +348,26 @@ def require_plane_index(self) -> int: ) return self.plane_index + def require_complete_axis(self, *, value_name: str) -> Self: + """Require that this declaration preserves every input plane.""" + + if self.plane_index is not None: + raise ValueError(f"{value_name} requires a complete input stack projection.") + return self + + @classmethod + def require_complete_projection( + cls, + projection: RuntimePlaneAxisValueProjection | None, + *, + value_name: str, + ) -> RuntimePlaneAxisValueProjection: + """Admit an optional invocation projection as a complete input axis.""" + + if projection is None: + raise ValueError(f"{value_name} requires a complete input stack projection.") + return projection.require_complete_axis(value_name=value_name) + def dense_shape_carries_axis(self, shape: Sequence[int]) -> bool: """Return whether a dense shape carries this declared leading axis.""" diff --git a/openhcs/processing/backends/enhance/flatfield.py b/openhcs/processing/backends/enhance/flatfield.py index a594e06026..49e616c055 100644 --- a/openhcs/processing/backends/enhance/flatfield.py +++ b/openhcs/processing/backends/enhance/flatfield.py @@ -4,7 +4,7 @@ from enum import Enum from typing import ClassVar -from openhcs.constants.constants import AllComponents, VariableComponents +from openhcs.constants.constants import VariableComponents from openhcs.core.projected_image_output import SourceProjectedImageOutput from openhcs.core.runtime_array_values import RuntimeArrayData from openhcs.core.runtime_image_values import image_payload_metadata @@ -27,7 +27,7 @@ class FittedIlluminationFieldOutput(SourceProjectedImageOutput): observation_count: int def __post_init__(self) -> None: - if type(self.observation_count) is not int or self.observation_count < 2: + if self.observation_count < 2: raise ValueError("A fitted field requires multiple observations.") if self.data.ndim not in (2, 3): raise ValueError("A fitted field requires a spatial image or volume.") @@ -35,20 +35,9 @@ def __post_init__(self) -> None: @classmethod def validate_observation_domain(cls, source: RuntimeArrayData) -> None: """Reject mislabeled metadata-backed ensembles before fitting.""" - metadata = image_payload_metadata(source) - if not metadata.has_values: - # Direct array callers declare N themselves; no source identity to infer. - return - axes = tuple( - AllComponents.from_value(name) - for name in metadata.retained_plane_component_values() + image_payload_metadata(source).require_independent_observation_axis( + cls.observation_axis, value_name="BaSiC" ) - declared_axis = AllComponents.from_value(cls.observation_axis.value) - if axes != (declared_axis,): - raise ValueError( - f"BaSiC requires independent {cls.observation_axis.name} observations " - "with every other source component fixed." - ) def with_data(self, data: RuntimeArrayData) -> "FittedIlluminationFieldOutput": return replace(self, data=data) @@ -58,8 +47,9 @@ def resolve_source_context( source: RuntimeArrayData, projection: RuntimePlaneAxisValueProjection | None, ) -> RuntimeArrayData: - if projection is None or projection.plane_index is not None: - raise ValueError("A fitted field requires the complete observation stack.") + projection = RuntimePlaneAxisValueProjection.require_complete_projection( + projection, value_name="A fitted field" + ) if projection.axis_size != self.observation_count: raise ValueError("Fitted field observation count differs from its source.") if source.shape != (self.observation_count, *self.data.shape): diff --git a/scripts/check_basicpy_source.py b/scripts/check_basicpy_source.py index 2bffb19578..f308b067c5 100644 --- a/scripts/check_basicpy_source.py +++ b/scripts/check_basicpy_source.py @@ -129,7 +129,22 @@ def test_fields_extend_existing_projection_and_metadata_owners(self): self.assertIn( "image_payload_metadata(source).collapse_leading_plane_axis", calls ) - self.assertIn("metadata.retained_plane_component_values", calls) + self.assertIn("image_payload_metadata(source).require_independent_observation_axis", calls) + metadata_tree = ast.parse((ROOT / "openhcs/core/runtime_image_values.py").read_text()) + metadata_owner = next( + node for node in metadata_tree.body + if isinstance(node, ast.ClassDef) and node.name == "ImagePayloadMetadata" + ) + metadata_calls = { + ast.unparse(node.func) for node in ast.walk(metadata_owner) + if isinstance(node, ast.Call) + } + declared_method_names = [ + node.name for node in metadata_owner.body + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) + ] + self.assertEqual(len(declared_method_names), len(set(declared_method_names))) + self.assertIn("self.retained_plane_component_values", metadata_calls) self.assertFalse(any(call.endswith((".repeat", ".tile")) for call in calls)) decorator = next( node diff --git a/tests/unit/test_fitted_illumination_fields.py b/tests/unit/test_fitted_illumination_fields.py index e0498ac977..592187ba13 100644 --- a/tests/unit/test_fitted_illumination_fields.py +++ b/tests/unit/test_fitted_illumination_fields.py @@ -101,7 +101,7 @@ def test_z_only_stack_is_not_an_observation_ensemble(): ) def test_field_rejects_one_plane_as_aggregate_source(projection): field = FittedIlluminationFieldOutput(np.ones((8, 9), dtype=np.float32), 3) - with pytest.raises(ValueError, match="complete observation stack"): + with pytest.raises(ValueError, match="complete input stack projection"): field.resolve_source_context(cross_site_source(), projection) diff --git a/tests/unit/test_projected_image_output.py b/tests/unit/test_projected_image_output.py index 91192c4fa8..54255dcd96 100644 --- a/tests/unit/test_projected_image_output.py +++ b/tests/unit/test_projected_image_output.py @@ -10,6 +10,19 @@ ) +@pytest.mark.parametrize("axis", tuple(RuntimePlaneAxis)) +def test_complete_projection_admission_is_owned_by_each_runtime_axis(axis): + projection = RuntimePlaneAxisValueProjection.preserve(axis=axis, axis_size=3) + assert RuntimePlaneAxisValueProjection.require_complete_projection( + projection, value_name="Test image" + ) is projection + for invalid in (None, projection.selected_plane(1)): + with pytest.raises(ValueError, match="complete input stack projection"): + RuntimePlaneAxisValueProjection.require_complete_projection( + invalid, value_name="Test image" + ) + + @pytest.mark.parametrize("indices", [(), (0, 0), (-1,), (True,), (0.5,)]) def test_selected_output_rejects_invalid_indices(indices): with pytest.raises(ValueError, match="Selected source planes"): From f3412fe61ed2a3cb04b82949be590875a3612d05 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 15:11:10 -0400 Subject: [PATCH 06/22] Verify projection owner closure and real four-dimensional BaSiC correction --- .../checkpoint.rst | 4 + .../owner-closure.rst | 83 +++++++++++++++++++ .../run-owner-check.sh | 8 +- .../run-source-check.sh | 4 +- tests/integration/test_basicpy_real_fit.py | 13 ++- 5 files changed, 102 insertions(+), 10 deletions(-) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst diff --git a/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst b/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst index 0c68e7753e..bddff5121d 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/checkpoint.rst @@ -1,6 +1,10 @@ BaSiCPy parent numerical integration ==================================== +Latest source ownership closure: owner-closure.rst. The prior failed guard +below is preserved historical evidence; its findings are now corrected and +the unmodified guard passes. Installed/MCP acceptance is still separate. + Integration owner: parent, 2026-09-30, existing PR217. Original Linnaeus worktrees are unchanged. Normal current-main merge incorporates d3a99c0d46dea979bba3f9076da87386e49cbed3. The sole merge conflict was the diff --git a/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst b/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst new file mode 100644 index 0000000000..9b19a31dbb --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst @@ -0,0 +1,83 @@ +Fitted-field ownership closure +============================== + +Parent owns this PR217 continuation from08d7d3acc2287619e0278df82e94d021169177c0. +Current main remainsd3a99c0d46dea979bba3f9076da87386e49cbed3. Paginated open +PR256 file inventory and PR206/207 inventories show no overlapping claims on +the three core projection/metadata files. Socrates' native ACK work is untouched. + +The prior guard failure was reproduced and retained, not waived. The source +correction is e0fbf1b06, with final volume-coverage tests and source recipe fixes +included in this checkpoint. Full package census:354488code lines, no parser +warnings. This is lightweight source census and tracing, not a complete NRA +semantic/global proof. Exact patches, not claimed native DSL replay. + +Ownership and migration +----------------------- + +* BOUND-1: observation_count is internally generated from the real model's + NumPy observation shape and carried by an integer-typed runtime declaration. + Remove duplicate exact-type revalidation; keep the genuine count>=2 invariant. + No external MCP input, saved declaration or wire-format decode is weakened. +* BOUND-2/IDEN-3: ImagePayloadMetadata now owns admission of its independent + observation coordinate. It uses original retained provenance and preserves + the raw-array/no-source-identity case. Fitted fields delegate; they no longer + probe or reconstruct the metadata owner's absent/present state themselves. +* BOUND-2/IDEN-3: RuntimePlaneAxisValueProjection owns complete-axis admission + through its existing type, beside its selected-plane and shape contracts. + Optional absence is rejected at that boundary; the instance owns its plane + selection state. Both selected-plane outputs and fitted fields delegate. + No family roster, new dispatcher, wrapper type or metadata mirror is added. +* Remove the two identical overwritten from_mapping and retained_plane_component_values + declarations from ImagePayloadMetadata. Effective bodies remain unchanged; + source-contract checking now rejects duplicate methods in that declaration. + +A new complete-stack consumer calls the existing owner instead of restating +its projection state. Both registered runtime-axis enum members exercise the +same declaration in the new-case test. No MRO change, external format or +persisted-state migration; optional input remains explicitly rejected where a +complete source proof is required. Domain error text carries caller context. + +Executed acceptance +------------------- + +152PASS,15.89s pytest/18.09s process,1133744KiB peakRSS, zero swaps, no skips +or deselections, one CPU/thread, existing Python3.12.3. Four actual source +imports asserted, including the newly initialized own ArrayBridge submodule +at the same published d6d92a7 pin. Real 2D and4D(N,Z,Y,X) OpenHCS BaSiC +fits retain native corrected values and same-fit fields. Covers prior provenance, +source-projection, context proof, singleton alignment and invalid selection/ +independent-axis/grid/count behaviors. Seven source contracts pass in0.032s. +Two warnings are disabled async pytest plugin configuration, not skipped tests. + +First owner-check launch stopped before pytest because the asserted physical +ArrayBridge path still named the separate integration worktree after the own +submodule was initialized. Preserve that original failure; correct the recipe +to the actual pinned checkout, not the production behavior. The initial corrected +suite had151PASS; adding the adapter's documented4D volume case produced152PASS. +All original logs/XML and resources remain under +``/home/ts/wt/openhcs-issue-batch-20260929/basicpy-owner-closure-20260930``. +No install, new environment, download, JVM, GUI or biological input. + +Unmodified original packaged agent-comms ratchet against main:5143metrics, +no positive deltas, exit0. ImagePayloadMetadata class-excess -9, projected-image +foreign absence probe -1, string subscripts -2. The comparison from the prior +PR checkpoint independently reports exact-type checks -1, foreign absence +probes -3, string subscripts -2, and no added dispatch/type switches, codec +subclasses, attribute-name access or long Boolean-chain terms. Original raw +before/after records and complete census are retained, no baseline manipulation. + +Paired library checkpoints use the same original packaged guard: +ArrayBridge38metrics and BaSiCPy43metrics, neither adds a positive metric. +Actual paired dtype tests and CPU BaSiC numerics were executed in the prior +checkpoint. Merge/remote verification is recorded separately from installation. + +Remaining acceptance +-------------------- + +This closes the reported source ownership guard failure, not the full expanded +ZIP goal or installed application acceptance. Parent still owns serialized +compiled/MCP discovery/execution and persisted corrected/flatfield/darkfield +artifact readback. Frozen biology runs, actual installed harness and managed +skill remain unchanged. Hosted CI is not the gate; real application verification +is still required before readiness or activation is claimed. diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh b/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh index af845c5581..1dccbcfad0 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh +++ b/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh @@ -1,6 +1,6 @@ #!/bin/sh set -eu -export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src +export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 OPENHCS_CPU_ONLY=true export JAX_PLATFORMS=cpu OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 export NUMBA_NUM_THREADS=1 XLA_FLAGS=--xla_cpu_multi_thread_eigen=false QT_QPA_PLATFORM=offscreen @@ -16,11 +16,11 @@ os.sched_setaffinity(0, {min(os.sched_getaffinity(0))}) import openhcs, arraybridge, basicpy, zmqruntime for module, directory in ( (openhcs, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/openhcs"), - (arraybridge, "/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src/arraybridge"), + (arraybridge, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/external/arraybridge/src/arraybridge"), (basicpy, "/home/ts/wt/basicpy-parent-numeric-20260930/src/basicpy"), (zmqruntime, "/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src/zmqruntime"), ): - assert Path(module.__file__).resolve().is_relative_to(Path(directory)) + assert Path(module.__file__).resolve().is_relative_to(Path(directory)), (module.__file__, directory) print("SOURCE", module.__file__, flush=True) import pytest raise SystemExit(pytest.main([ @@ -32,6 +32,6 @@ raise SystemExit(pytest.main([ "tests/unit/test_function_runtime_source_projection.py", "tests/unit/test_image_output_context_projection_proof.py", "tests/unit/test_runtime_image_singleton_alignment.py", - "--junitxml=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-owner-closure-20260930/source-tests.xml", + "--junitxml=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-owner-closure-20260930/source-tests-volume.xml", ])) ' diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh b/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh index 3044087885..0a39bc8977 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh +++ b/docs/validation/basicpy_parent_numeric_20260930/run-source-check.sh @@ -1,6 +1,6 @@ #!/bin/sh set -eu -export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src +export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 OPENHCS_CPU_ONLY=true export JAX_PLATFORMS=cpu OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 export NUMBA_NUM_THREADS=1 XLA_FLAGS=--xla_cpu_multi_thread_eigen=false QT_QPA_PLATFORM=offscreen @@ -16,7 +16,7 @@ os.sched_setaffinity(0, {min(os.sched_getaffinity(0))}) import openhcs, arraybridge, basicpy, zmqruntime for module, directory in ( (openhcs, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/openhcs"), - (arraybridge, "/home/ts/wt/arraybridge-basicpy-parent-integration-20260930/src/arraybridge"), + (arraybridge, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/external/arraybridge/src/arraybridge"), (basicpy, "/home/ts/wt/basicpy-parent-numeric-20260930/src/basicpy"), (zmqruntime, "/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src/zmqruntime"), ): diff --git a/tests/integration/test_basicpy_real_fit.py b/tests/integration/test_basicpy_real_fit.py index d25b1033eb..2df4a9f267 100644 --- a/tests/integration/test_basicpy_real_fit.py +++ b/tests/integration/test_basicpy_real_fit.py @@ -10,7 +10,7 @@ ) -def shaded_observations(*, stationary_biology=False): +def shaded_observations(*, stationary_biology=False, volume=False): """Synthetic acquisition only: no biological data, reference or downloaded image.""" size = 32 count = 24 @@ -30,7 +30,11 @@ def shaded_observations(*, stationary_biology=False): biology = 5000 * np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / 0.008) frame = (background + biology) * flatfield observations.append(np.rint(frame).astype(np.uint16)) - return np.stack(observations), flatfield + observations = np.stack(observations) + if volume: + observations = np.stack((observations, observations), axis=1) + flatfield = np.stack((flatfield, flatfield)) + return observations, flatfield def run_fit(observations): @@ -42,8 +46,9 @@ def run_fit(observations): ) -def test_real_fit_keeps_measurement_range_and_returns_same_fit_fields(): - observations, truth = shaded_observations() +@pytest.mark.parametrize("volume", [False, True]) +def test_real_fit_keeps_measurement_range_and_returns_same_fit_fields(volume): + observations, truth = shaded_observations(volume=volume) corrected, flatfield, darkfield = run_fit(observations) corrected = np.asarray(corrected) flatfield_data = np.asarray(flatfield) From a68ca90c5d804027819355a64078819c846bc9a7 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 15:16:32 -0400 Subject: [PATCH 07/22] Record independently verified paired library main merges --- .../basicpy_parent_numeric_20260930/owner-closure.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst b/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst index 9b19a31dbb..d61def6308 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst @@ -72,6 +72,15 @@ ArrayBridge38metrics and BaSiCPy43metrics, neither adds a positive metric. Actual paired dtype tests and CPU BaSiC numerics were executed in the prior checkpoint. Merge/remote verification is recorded separately from installation. +GitHub plus independent git ls-remote verification: +BaSiCPyPR1 merged mainc1d7c3dc6a57a26acc1179de3f2e14b1c5faa0ec at19:07:50UTC. +ArrayBridgePR2 merged into its old stacked base branch, not main; parent +verified that branch821696e7 is tree-identical to tested d6d92a7, includes +current main409, and contains only the two intended main-delta files. +Main-targeted PR3 then merged main ea3f2a4cc91c4810d12343f58f85c1195e1a41e6 +at19:13:30UTC. No optional hosted-CI wait, bypass, force push or install. +OpenHCS retains the same exact tested ancestor pins, not a dependency rollback. + Remaining acceptance -------------------- From 34cf63b5d54ca14a48c626037036379157df8ac2 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 17:25:45 -0400 Subject: [PATCH 08/22] Verify BaSiCPy owner controls on current merged main --- .../owner-closure.rst | 24 +++++++++++++++++++ .../run-owner-check.sh | 6 ++--- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst b/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst index d61def6308..fef702e2ac 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/owner-closure.rst @@ -90,3 +90,27 @@ compiled/MCP discovery/execution and persisted corrected/flatfield/darkfield artifact readback. Frozen biology runs, actual installed harness and managed skill remain unchanged. Hosted CI is not the gate; real application verification is still required before readiness or activation is claimed. + +Current-main integration (2026-09-30) +------------------------------------ + +Parent normally merged main d1c6ab72ca252ea93656eeaaf84d34b81abd8317 into this +isolated PR tree at e43420c69. No rebase, reset or global package change. Paired +ArrayBridge/BaSiCPy revisions stay the same; own native9e11f6f, PolyStoredc34173, +metaclass448cdf0 and pyqt3437d1c source checkouts reuse existing Git objects. +The owner-check recipe now relies on app-owned source dependency projection; +it no longer injects the older unrelated native checkout onto PYTHONPATH. + +Current-main controls:152passed16.83s, process18.95s, peakRSS1174892KiB, zero +swaps. Original packaged ratchet5155metrics with zero positive deltas; metadata +class-excess-9, foreign-absence-1 and string-subscripts-2 remain. Seven source +contracts pass using the actual upstream basicpy.py file. Two mistaken +directory-valued source-contract invocations failed with IsADirectoryError; +their original logs remain separate, no production/backend invocation occurred. +All reports persist in basicpy-owner-closure-20260930; no older report replaced. + +Real fit-field MCP/persisted readback remains pending one named runtime dependency: +Russell's fresh context-isolated trial must release validation.lock first. This +PR source tree is independent of that frozen source and skill; do not mutate +or replay the scientific trial. This checkpoint is source-qualified numeric +acceptance, not installed/live/biological readiness or a reason to wait for CI. diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh b/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh index 1dccbcfad0..0325194e30 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh +++ b/docs/validation/basicpy_parent_numeric_20260930/run-owner-check.sh @@ -1,6 +1,6 @@ #!/bin/sh set -eu -export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/basicpy-parent-numeric-20260930/src:/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src +export PYTHONPATH=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/basicpy-parent-numeric-20260930/src export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 OPENHCS_CPU_ONLY=true export JAX_PLATFORMS=cpu OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 export NUMBA_NUM_THREADS=1 XLA_FLAGS=--xla_cpu_multi_thread_eigen=false QT_QPA_PLATFORM=offscreen @@ -18,7 +18,7 @@ for module, directory in ( (openhcs, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/openhcs"), (arraybridge, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/external/arraybridge/src/arraybridge"), (basicpy, "/home/ts/wt/basicpy-parent-numeric-20260930/src/basicpy"), - (zmqruntime, "/home/ts/wt/openhcs-viewer-live-20260930/external/zmqruntime/src/zmqruntime"), + (zmqruntime, "/home/ts/wt/openhcs-basicpy-parent-integration-20260930/external/zmqruntime/src/zmqruntime"), ): assert Path(module.__file__).resolve().is_relative_to(Path(directory)), (module.__file__, directory) print("SOURCE", module.__file__, flush=True) @@ -32,6 +32,6 @@ raise SystemExit(pytest.main([ "tests/unit/test_function_runtime_source_projection.py", "tests/unit/test_image_output_context_projection_proof.py", "tests/unit/test_runtime_image_singleton_alignment.py", - "--junitxml=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-owner-closure-20260930/source-tests-volume.xml", + "--junitxml=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-owner-closure-20260930/source-tests-merged-main.xml", ])) ' From 51db1d03856bd218c4d926ecd5a9e6545c765d75 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 17:56:08 -0400 Subject: [PATCH 09/22] Declare reviewed BaSiCPy fork as ordinary dependency and share live fit fixture --- docs/source/development/basicpy_readiness.md | 20 +++-- .../project-dependency.rst | 49 +++++++++++++ pyproject.toml | 7 +- requirements-basicpy.txt | 7 -- scripts/check_basicpy_source.py | 56 ++++++++------ .../basicpy_observation_fixture.py | 29 ++++++++ .../create_basicpy_acquisition_fixture.py | 73 +++++++++++++++++++ tests/integration/test_basicpy_real_fit.py | 30 +------- .../unit/test_basicpy_acquisition_fixture.py | 43 +++++++++++ 9 files changed, 247 insertions(+), 67 deletions(-) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst delete mode 100644 requirements-basicpy.txt create mode 100644 tests/diagnostics/basicpy_observation_fixture.py create mode 100644 tests/diagnostics/create_basicpy_acquisition_fixture.py create mode 100644 tests/unit/test_basicpy_acquisition_fixture.py diff --git a/docs/source/development/basicpy_readiness.md b/docs/source/development/basicpy_readiness.md index 77a323a11f..2595d5a04d 100644 --- a/docs/source/development/basicpy_readiness.md +++ b/docs/source/development/basicpy_readiness.md @@ -5,6 +5,15 @@ Python3.12 CPU fits, dtype behavior and field provenance checks passed. See the canonical [2026-09-30 receipt](../../../validation/basicpy_parent_numeric_20260930/checkpoint.rst). No install, compiled/MCP execution or biological acceptance is claimed. +The current packaging change declares `openhcs-basicpy>=1.3.0,<1.4` in the +ordinary project dependencies. It supplies the reviewed JAX fork under its +existing `basicpy` Python API; upstream PyPI `basicpy` 2.x is not substituted. +The separate source requirements file is removed. The paired fork release must +be published and resolved before this dependency change can be merged or called +installed. Linux, Windows and Apple Silicon receive it automatically; Intel +macOS is excluded because modern JAX no longer supplies that platform's wheels. +This does not assert that illumination correction is available on Intel macOS. + The remaining text records the historical 2026-09-29 source-only checkpoint; its pending checks and owners must not be read as today's runtime state. Original backend/dependency owner: Linnaeus. Parent PR151 owns recipes/policy. Memory/ @@ -41,13 +50,10 @@ retained-history PR1; neither is merged or installed here. The paired [BaSiCPy draft](https://github.com/OpenHCSDev/BaSiCPy/pull/1) reuses Tristan's clean JAX prototype `ae2c647`, rather than PyPI 2.x's PyTorch API. -`requirements-basicpy.txt` pins the reviewed fork commit. As with this project's -existing source-only dependency files, no Git URL enters PyPI metadata. After -review, install that file in the coordinator-selected **isolated** validation -environment; do not run it against the active analysis venv. A published fork -wheel/version and ordinary optional extra can replace the source pin only after -actual import/fit acceptance and release review, not by adding an unavailable -PyPI dependency now. +The historical checkpoint used `requirements-basicpy.txt` to pin the reviewed +fork commit; that file has now been removed in favor of the ordinary dependency +described above. No Git URL enters PyPI metadata. The historical source-only +installation route is not an instruction to modify the active analysis venv. The existing gpu/all JAX constraints rejected this modern candidate. They now delegate exact JAXlib/CUDA-plugin matching to `jax[cuda12-local]>=0.9.2,<0.10`, diff --git a/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst b/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst new file mode 100644 index 0000000000..5531584d54 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst @@ -0,0 +1,49 @@ +Ordinary BaSiCPy dependency checkpoint +===================================== + +Integration owner: parent; paired OpenHCS PR217 and BaSiCPy PR2. +Publication explicitly authorized by Tristan on 2026-09-30. + +The project metadata owns ``openhcs-basicpy>=1.3.0,<1.4``. It supplies the +reviewed JAX fork, retaining the upstream ``basicpy`` Python API and CLI. +No copied algorithm, API alias, extra installation registry or source-path +requirement is introduced. ``requirements-basicpy.txt`` is removed (recoverable +in Git); ArrayBridge already has its ordinary project dependency. Linux, +Windows and Apple Silicon receive the fork automatically. Intel macOS retains +core installation without modern JAX, whose CPU wheels exclude that platform. + +Do not merge an unavailable dependency. The fork's merged packaging checkpoint +is ``8ca3be674d2e11602c152e5080ee4cdf94a780a3``; release automation is triggered +by ``v1.3.0``. Registry availability and actual resolution must be verified +separately; neither a tag nor a wheel build is publication. + +Actual validation +----------------- + +Eight source contracts passed in 0.038s, including the ordinary dependency +specifier, platform markers, upstream parameter ownership and dtype declaration. +The fork wheel was installed into an owned target, without dependencies, +downloads or shared environment mutation. Both Python 3.12 and 3.14 import +that exact installed target as ``basicpy``, with metadata identifying +``openhcs-basicpy`` 1.3.0. Six real numerical/profile tests passed on each: +13.24s and 12.18s, respectively. + +Four paired OpenHCS tests passed against that installed fork wheel in 10.36s: +the real decorated 2D/volume fits, stationary-biology negative control and +persisted acquisition fixture. The OpenHCS wheel itself builds successfully +and its METADATA declares the ordinary fork requirement, with no Git URLs. + +The shared deterministic numerical control is now written as 24 independent +ImageXpress SITE files through the existing typed filename and image-format +owners. A real acquisition parser proves the calibration, SITE/channel/Z +values and exact input pixels. Existing fit assertions use the same numerical +generator, rather than a second scientific fixture recipe. Overwrite refusal +preserves every existing input file. These are synthetic controls, not blind +benchmark inputs or answers. + +This receipt is not installed user-entrypoint, compiled/MCP, CUDA or biological +acceptance. The frozen blind run and shared installed runtime are unchanged. +Durable wheel/XML receipts are under +``/home/ts/wt/openhcs-issue-batch-20260929/basicpy-project-dependency-20260930``. +Owned disposable targets and bounded generated test files are under +``/home/ts/.cache/agent-scratch/basicpy-package-20260930``. diff --git a/pyproject.toml b/pyproject.toml index 9da4796cd8..ec3b17da33 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -52,6 +52,9 @@ dependencies = [ "scikit-learn>=1.7.1", "scipy>=1.12.0", "PyWavelets>=1.6.0", + # Reviewed JAX BaSiCPy fork, not upstream basicpy 2.x's PyTorch API. + # Modern JAX has no Intel macOS wheels; preserve that desktop's core install. + "openhcs-basicpy>=1.3.0,<1.4; platform_system != 'Darwin' or platform_machine != 'x86_64'", "pandas>=2.3.1", "openpyxl>=3.1.0", "xlsxwriter>=3.2.0", @@ -362,8 +365,8 @@ openhcs = [ # - JAX owns exact plugin/JAXlib matching through its cuda12-local extra. # This uses a locally provisioned CUDA/cuDNN, not bundled CUDA dependencies. # Review driver/library requirements at https://docs.jax.dev/en/latest/installation.html -# - BaSiCPy JAX fork is source-pinned in requirements-basicpy.txt while its -# paired draft is validated. Direct Git URLs must not enter PyPI metadata. +# - openhcs-basicpy supplies the reviewed JAX fork as an ordinary dependency. +# Direct Git URLs must not enter PyPI metadata. # - PyTorch 2.7.x: Compatible with CUDA 12.6 and CuDNN 9.5.x # - TensorFlow 2.15-2.19: Stable DLPack support (2.12+ required for DLPack) # - CuPy: Use cupy-cuda12x for broad CUDA 12.x compatibility (not cupy-cuda120) diff --git a/requirements-basicpy.txt b/requirements-basicpy.txt deleted file mode 100644 index 92609ee815..0000000000 --- a/requirements-basicpy.txt +++ /dev/null @@ -1,7 +0,0 @@ -# Reviewed JAX prototype, paired draft https://github.com/OpenHCSDev/BaSiCPy/pull/1 -# Do not replace with PyPI 2.x: it owns a different PyTorch API. -# CPU source import/fit validated; installed/compiled MCP acceptance remains pending. -BaSiCPy @ git+https://github.com/OpenHCSDev/BaSiCPy.git@855af6b10c87e832cb6a6594fb288c28d74f46e5 -# Typed native direct-call default, stacked on reviewed retained-history fix. -# https://github.com/OpenHCSDev/ArrayBridge/pull/2 (depends on PR1) -arraybridge @ git+https://github.com/OpenHCSDev/ArrayBridge.git@d6d92a716b91f0148dc2c04f9e744bddb7e32ed4 diff --git a/scripts/check_basicpy_source.py b/scripts/check_basicpy_source.py index f308b067c5..77eef9631c 100644 --- a/scripts/check_basicpy_source.py +++ b/scripts/check_basicpy_source.py @@ -4,11 +4,12 @@ import argparse import ast -import re import tomllib import unittest from pathlib import Path +from packaging.requirements import Requirement + ROOT = Path(__file__).resolve().parents[1] WRAPPER = ROOT / "openhcs/processing/backends/enhance/basic_processor_jax.py" @@ -25,6 +26,28 @@ def setUp(self): and node.name == "basic_flatfield_correction_jax" ) + def test_reviewed_fork_is_an_ordinary_project_dependency(self): + project = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"] + requirements = tuple(Requirement(value) for value in project["dependencies"]) + [fork] = [value for value in requirements if value.name == "openhcs-basicpy"] + self.assertIsNone(fork.url) + self.assertIn("1.3.0", fork.specifier) + self.assertNotIn("2.0.0", fork.specifier) + self.assertIsNotNone(fork.marker) + for system, machine, admitted in ( + ("Linux", "x86_64", True), + ("Windows", "AMD64", True), + ("Darwin", "arm64", True), + ("Darwin", "x86_64", False), + ): + with self.subTest(system=system, machine=machine): + self.assertEqual( + fork.marker.evaluate( + {"platform_system": system, "platform_machine": machine} + ), + admitted, + ) + def test_every_exposed_knob_reaches_real_model_owner(self): owner = next( node @@ -157,18 +180,7 @@ def test_fields_extend_existing_projection_and_metadata_owners(self): ["CORRECTED_OUTPUT", "FLATFIELD_OUTPUT", "DARKFIELD_OUTPUT"], ) - def test_reviewed_git_pin_and_jax_owned_version_matching(self): - requirement = next( - line - for line in (ROOT / "requirements-basicpy.txt").read_text().splitlines() - if line and not line.startswith("#") - ) - self.assertRegex( - requirement, - re.compile( - r"^BaSiCPy @ git\+https://github.com/OpenHCSDev/BaSiCPy.git@[0-9a-f]{40}$" - ), - ) + def test_jax_owns_plugin_version_matching(self): project = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"] for extra in ("gpu", "all"): jax_requirements = [ @@ -207,16 +219,14 @@ def test_native_default_reaches_existing_arraybridge_owner(self): "dtype_config_default", {argument.arg for argument in wrapper.args.kwonlyargs}, ) - requirements = (ROOT / "requirements-basicpy.txt").read_text().splitlines() - self.assertTrue( - any( - re.fullmatch( - r"arraybridge @ git\+https://github.com/OpenHCSDev/ArrayBridge.git@[0-9a-f]{40}", - requirement, - ) - for requirement in requirements - ) - ) + project = tomllib.loads((ROOT / "pyproject.toml").read_text())["project"] + [arraybridge] = [ + Requirement(value) + for value in project["dependencies"] + if Requirement(value).name == "arraybridge" + ] + self.assertIsNone(arraybridge.url) + self.assertIn("0.3.4", arraybridge.specifier) if __name__ == "__main__": diff --git a/tests/diagnostics/basicpy_observation_fixture.py b/tests/diagnostics/basicpy_observation_fixture.py new file mode 100644 index 0000000000..adcadb261e --- /dev/null +++ b/tests/diagnostics/basicpy_observation_fixture.py @@ -0,0 +1,29 @@ +"""Shared numerical acquisition controls; never blind microscopy inputs.""" + +import numpy as np + + +def shaded_observations(*, stationary_biology=False, volume=False): + """Known shading with either moving objects or a confounded stationary pattern.""" + size = 32 + count = 24 + yy, xx = np.mgrid[-1 : 1 : complex(size), -1 : 1 : complex(size)] + flatfield = 1.2 - 0.3 * (yy**2 + xx**2) + 0.1 * xx + flatfield /= flatfield.mean() + rng = np.random.default_rng(213) + observations = [] + for index in range(count): + background = 3000 + 50 * index + if stationary_biology: + # Successful convergence cannot separate stationary biology from shading. + biology = background * np.exp(-((xx - 0.2) ** 2 + (yy + 0.1) ** 2) / 0.09) + else: + cx, cy = rng.uniform(-0.8, 0.8, size=2) + biology = 5000 * np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / 0.008) + frame = (background + biology) * flatfield + observations.append(np.rint(frame).astype(np.uint16)) + observations = np.stack(observations) + if volume: + observations = np.stack((observations, observations), axis=1) + flatfield = np.stack((flatfield, flatfield)) + return observations, flatfield diff --git a/tests/diagnostics/create_basicpy_acquisition_fixture.py b/tests/diagnostics/create_basicpy_acquisition_fixture.py new file mode 100644 index 0000000000..b9dea03eb8 --- /dev/null +++ b/tests/diagnostics/create_basicpy_acquisition_fixture.py @@ -0,0 +1,73 @@ +"""Persist the existing bounded numerical control as an ImageXpress acquisition.""" + +import argparse +import hashlib +import json +from pathlib import Path + +import numpy as np + +from openhcs.constants import AllComponents +from openhcs.core.components.parser_metaprogramming import FilenameParseResult +from openhcs.core.image_file_serialization import ImageFileFormat +from openhcs.microscopes.imagexpress import ImageXpressFilenameParser + +# Initialize OpenHCS's existing metadata configuration before PolyStore. +# isort: split +from polystore.disk import DiskStorageBackend +from polystore.filemanager import FileManager + +from tests.diagnostics.basicpy_observation_fixture import shaded_observations + + +def create_acquisition(destination: Path) -> dict: + """Never replace a previous fixture or its original receipt.""" + destination.mkdir(parents=True, exist_ok=False) + raw = destination / "TimePoint_1" + raw.mkdir() + observations, truth = shaded_observations() + (destination / "plate.HTD").write_text( + '"XSites", 6\n"YSites", 4\n"PixelSizeUM", 0.65\n' + ) + parser = ImageXpressFilenameParser(FileManager({"disk": DiskStorageBackend()})) + paths = [] + for site, pixels in enumerate(observations, start=1): + components = FilenameParseResult( + ( + (AllComponents.WELL, "A01"), + (AllComponents.SITE, site), + (AllComponents.CHANNEL, 1), + (AllComponents.Z_INDEX, 1), + (AllComponents.TIMEPOINT, 1), + ), + extension=".tif", + ) + path = raw / parser.construct_filename(components) + ImageFileFormat.require_path(path).write(path, pixels) + paths.append( + {"path": str(path.relative_to(destination)), + "sha256": hashlib.sha256(path.read_bytes()).hexdigest()} + ) + np.save(destination / "synthetic-flatfield-truth.npy", truth) + return { + "purpose": "Synthetic fit/publication control; not a blind biological assay", + "plate_path": str(destination), + "shape_nyx": list(observations.shape), + "dtype": str(observations.dtype), + "independent_component": AllComponents.SITE.value, + "source_xy_spacing_um": 0.65, + "files": paths, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("destination", type=Path) + args = parser.parse_args() + if args.destination.exists(): + parser.error("Refusing to overwrite an existing fixture.") + print(json.dumps(create_acquisition(args.destination), indent=2)) + + +if __name__ == "__main__": + main() diff --git a/tests/integration/test_basicpy_real_fit.py b/tests/integration/test_basicpy_real_fit.py index 2df4a9f267..e133b0afb1 100644 --- a/tests/integration/test_basicpy_real_fit.py +++ b/tests/integration/test_basicpy_real_fit.py @@ -3,38 +3,12 @@ import numpy as np import pytest -pytest.importorskip("basicpy", reason="requires reviewed paired BaSiCPy source") +pytest.importorskip("basicpy", reason="requires the openhcs-basicpy distribution") from openhcs.processing.backends.enhance.basic_processor_jax import ( basic_flatfield_correction_jax, ) - - -def shaded_observations(*, stationary_biology=False, volume=False): - """Synthetic acquisition only: no biological data, reference or downloaded image.""" - size = 32 - count = 24 - yy, xx = np.mgrid[-1 : 1 : complex(size), -1 : 1 : complex(size)] - flatfield = 1.2 - 0.3 * (yy**2 + xx**2) + 0.1 * xx - flatfield /= flatfield.mean() - rng = np.random.default_rng(213) - observations = [] - for index in range(count): - background = 3000 + 50 * index - if stationary_biology: - # A stationary, background-scaled biological pattern is fundamentally - # confounded with multiplicative shading, not a convergence failure. - biology = background * np.exp(-((xx - 0.2) ** 2 + (yy + 0.1) ** 2) / 0.09) - else: - cx, cy = rng.uniform(-0.8, 0.8, size=2) - biology = 5000 * np.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / 0.008) - frame = (background + biology) * flatfield - observations.append(np.rint(frame).astype(np.uint16)) - observations = np.stack(observations) - if volume: - observations = np.stack((observations, observations), axis=1) - flatfield = np.stack((flatfield, flatfield)) - return observations, flatfield +from tests.diagnostics.basicpy_observation_fixture import shaded_observations def run_fit(observations): diff --git a/tests/unit/test_basicpy_acquisition_fixture.py b/tests/unit/test_basicpy_acquisition_fixture.py new file mode 100644 index 0000000000..04cea48e5f --- /dev/null +++ b/tests/unit/test_basicpy_acquisition_fixture.py @@ -0,0 +1,43 @@ +"""The live fixture uses the numerical control and the real acquisition parser.""" + +import numpy as np +import pytest + +from openhcs.constants import AllComponents +from openhcs.core.image_file_serialization import ImageFileFormat +from openhcs.microscopes.imagexpress import ImageXpressHandler + +# Initialize OpenHCS's existing metadata configuration before PolyStore. +# isort: split +from polystore.disk import DiskStorageBackend +from polystore.filemanager import FileManager + +from tests.diagnostics.basicpy_observation_fixture import shaded_observations +from tests.diagnostics.create_basicpy_acquisition_fixture import create_acquisition + + +def test_saved_numerical_control_has_real_independent_site_calibration(tmp_path): + plate = tmp_path / "basic-control" + receipt = create_acquisition(plate) + handler = ImageXpressHandler(FileManager({"disk": DiskStorageBackend()})) + observations, truth = shaded_observations() + assert receipt["shape_nyx"] == [24, 32, 32] + assert handler.metadata_handler.get_pixel_size(plate) == 0.65 + files = sorted((plate / "TimePoint_1").glob("*.tif")) + assert len(files) == observations.shape[0] + sites = [] + for path, expected in zip(files, observations, strict=True): + parsed = handler.parser.parse_filename(path.name) + assert parsed is not None + sites.append(parsed.required_value(AllComponents.SITE)) + assert parsed.required_value(AllComponents.CHANNEL) == 1 + assert parsed.required_value(AllComponents.Z_INDEX) == 1 + np.testing.assert_array_equal(ImageFileFormat.require_path(path).read(path), expected) + assert sites == list(range(1, 25)) + np.testing.assert_array_equal( + np.load(plate / "synthetic-flatfield-truth.npy"), truth + ) + original = [path.read_bytes() for path in files] + with pytest.raises(FileExistsError): + create_acquisition(plate) + assert [path.read_bytes() for path in files] == original From 9c28c6a865bd862e5baf4e8ab15f15a85481ef9c Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 18:03:22 -0400 Subject: [PATCH 10/22] Require corrected fork release and record CLI publication blocker --- docs/source/development/basicpy_readiness.md | 2 +- .../project-dependency.rst | 9 ++++++--- pyproject.toml | 2 +- scripts/check_basicpy_source.py | 3 ++- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/source/development/basicpy_readiness.md b/docs/source/development/basicpy_readiness.md index 2595d5a04d..7b4e183b04 100644 --- a/docs/source/development/basicpy_readiness.md +++ b/docs/source/development/basicpy_readiness.md @@ -5,7 +5,7 @@ Python3.12 CPU fits, dtype behavior and field provenance checks passed. See the canonical [2026-09-30 receipt](../../../validation/basicpy_parent_numeric_20260930/checkpoint.rst). No install, compiled/MCP execution or biological acceptance is claimed. -The current packaging change declares `openhcs-basicpy>=1.3.0,<1.4` in the +The current packaging change declares `openhcs-basicpy>=1.3.1,<1.4` in the ordinary project dependencies. It supplies the reviewed JAX fork under its existing `basicpy` Python API; upstream PyPI `basicpy` 2.x is not substituted. The separate source requirements file is removed. The paired fork release must diff --git a/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst b/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst index 5531584d54..716c2a8ebd 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/project-dependency.rst @@ -4,8 +4,8 @@ Ordinary BaSiCPy dependency checkpoint Integration owner: parent; paired OpenHCS PR217 and BaSiCPy PR2. Publication explicitly authorized by Tristan on 2026-09-30. -The project metadata owns ``openhcs-basicpy>=1.3.0,<1.4``. It supplies the -reviewed JAX fork, retaining the upstream ``basicpy`` Python API and CLI. +The project metadata owns ``openhcs-basicpy>=1.3.1,<1.4``. It supplies the +reviewed JAX fork, retaining the upstream ``basicpy`` Python API. No copied algorithm, API alias, extra installation registry or source-path requirement is introduced. ``requirements-basicpy.txt`` is removed (recoverable in Git); ArrayBridge already has its ordinary project dependency. Linux, @@ -14,7 +14,10 @@ core installation without modern JAX, whose CPU wheels exclude that platform. Do not merge an unavailable dependency. The fork's merged packaging checkpoint is ``8ca3be674d2e11602c152e5080ee4cdf94a780a3``; release automation is triggered -by ``v1.3.0``. Registry availability and actual resolution must be verified +by ``v1.3.0``. That queued job was cancelled before any steps ran after an +installed-entrypoint check found a legacy TODO-only CLI. Issue313 and the +corrective fork checkpoint remove the dead CLI in place and advance to1.3.1; +the original tag/history are preserved. Registry availability and actual resolution must be verified separately; neither a tag nor a wheel build is publication. Actual validation diff --git a/pyproject.toml b/pyproject.toml index ec3b17da33..b868c9c29a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -54,7 +54,7 @@ dependencies = [ "PyWavelets>=1.6.0", # Reviewed JAX BaSiCPy fork, not upstream basicpy 2.x's PyTorch API. # Modern JAX has no Intel macOS wheels; preserve that desktop's core install. - "openhcs-basicpy>=1.3.0,<1.4; platform_system != 'Darwin' or platform_machine != 'x86_64'", + "openhcs-basicpy>=1.3.1,<1.4; platform_system != 'Darwin' or platform_machine != 'x86_64'", "pandas>=2.3.1", "openpyxl>=3.1.0", "xlsxwriter>=3.2.0", diff --git a/scripts/check_basicpy_source.py b/scripts/check_basicpy_source.py index 77eef9631c..24634cdce9 100644 --- a/scripts/check_basicpy_source.py +++ b/scripts/check_basicpy_source.py @@ -31,7 +31,8 @@ def test_reviewed_fork_is_an_ordinary_project_dependency(self): requirements = tuple(Requirement(value) for value in project["dependencies"]) [fork] = [value for value in requirements if value.name == "openhcs-basicpy"] self.assertIsNone(fork.url) - self.assertIn("1.3.0", fork.specifier) + self.assertIn("1.3.1", fork.specifier) + self.assertNotIn("1.3.0", fork.specifier) self.assertNotIn("2.0.0", fork.specifier) self.assertIsNotNone(fork.marker) for system, machine, admitted in ( From a3d0ad1c0c5db66e96d01771a3b8b44cf53c23c7 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 19:07:14 -0400 Subject: [PATCH 11/22] Declare real BaSiCPy NumPy boundary for CPU workflows --- .../cpu-installed-checkpoint.rst | 67 +++++++++++++++++++ .../pipeline-live-wheel.py | 53 +++++++++++++++ .../run-live-wheel.sh | 25 +++++++ .../backends/enhance/basic_processor_jax.py | 18 ++--- openhcs/utils/environment.py | 3 + tests/integration/test_basicpy_real_fit.py | 19 ++++++ tests/unit/test_process_environment.py | 8 +++ 7 files changed, 185 insertions(+), 8 deletions(-) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/cpu-installed-checkpoint.rst create mode 100644 docs/validation/basicpy_parent_numeric_20260930/pipeline-live-wheel.py create mode 100644 docs/validation/basicpy_parent_numeric_20260930/run-live-wheel.sh diff --git a/docs/validation/basicpy_parent_numeric_20260930/cpu-installed-checkpoint.rst b/docs/validation/basicpy_parent_numeric_20260930/cpu-installed-checkpoint.rst new file mode 100644 index 0000000000..88eb43cfb0 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/cpu-installed-checkpoint.rst @@ -0,0 +1,67 @@ +Real solver CPU transport and installed candidate +================================================ + +Parent owns issue320 and PR217. Current main2b7969f700 was normally merged +into this isolated tree at4640bbaf4; all exact recorded submodules are present. +No shared checkout, shared environment or frozen biological trial is changed. + +The first live attempt exposed two actual boundaries. McpDevClient deliberately +projects a minimal child environment, so a temporary PYTHONPATH wheel target +does not establish installed dependency availability. The public source plan +reported ModuleNotFoundError: basicpy. Separately the CPU-only function catalog +admits NumPy array transport; the real adapter's JAX-array declaration was +absent from discovery. These are retained failures, not successful MCP proof. + +BaSiCPy's public fit_transform accepts and returns NumPy arrays. Its internal +JAX solver does not make its public boundary GPU-only. The adapter now declares +that existing NumPy transport through the original memory decorator; it still +calls the real BaSiC fit_transform and returns the same-fit fields with unchanged +typed provenance and observation counts. No approximate solver, callable alias, +registry exception, GPU-family roster or new wrapper is introduced. Original +import path and analytical output units/shapes remain; array storage changes +from JAX to the dependency's actual NumPy API and is declared explicitly. + +IDEN-1/BOUND-2: array transport and solver implementation were conflated. The +existing callable/memory declaration owns transport, while BaSiCPy owns its +algorithm. OpenHCSProcessEnvironment owns CPU-only policy and projects +JAX_PLATFORMS=cpu before dependency import. GPU-import suppression without +CPU-only mode does not force that platform. A new NumPy-facing solver needs +only its memory declaration, not catalog/compiler/device exceptions. + +Twenty focused source tests pass in8.83s: actual2D/volume fits preserve float +values and same-fit fields, stationary biology remains a negative control, +the current declaration is CPU-catalog eligible, and process-mode projection +does not weaken ordinary GPU suppression. The initial test launch imported +PolyStore before source-owned initialization and failed collection; the +corrected launch imports OpenHCS first. Both XML receipts remain separate. +This is not full NRA/R1 or installed/live biological acceptance. + +The genuine candidate environment is persistent: +/home/ts/wt/basicpy-live-candidate-20260930/.venv, about51MiB before later +cache growth. It contains actual wheels for OpenHCS0.8.7, openhcs-basicpy1.3.1, +ArrayBridge0.3.4 and all seven recorded companion libraries, including native +0.3.0 and PolyStore0.3.0. A single validation-only site hook reuses heavy +third-party dependencies from the existing environment; none were copied or +redownloaded. All OpenHCS/fork/companion import origins must be local to this +candidate. The shared installation is explicitly untouched. + +The first installed attempt inherited the old native0.2.24 and rejected owned +startup before spawn because EndpointStartupUncertainError was unavailable. +No returned child handle or listener existed. The stopped attempt is retained; +the paired installation supplies the exact already-tested native wheel rather +than weakening startup or changing the application dependency floor. + +Fresh paired MCP reports an actual site-packages source, healthy0.8.7 and no +stale files. Owned native PID13045/create_time1790809475.24 on5996 is preparing +through its original handle. The accepted public artifact plan covers24 SITE +observations of one channel, corrected float images and persistent same-fit +flatfield/darkfield outputs. Compile, execution, persisted readback, viewer +field inspection and exact owned cleanup remain in progress; they are not +claimed by this checkpoint. Original request/receipts remain under +/home/ts/wt/openhcs-issue-batch-20260929/basicpy-live-wheel-20260930. + +The public fork release job36783511705 remains queued without a runner. PR217 +must not merge an unavailable ordinary dependency, and no release is claimed. +All original failure receipts and model controls remain separate from the new +candidate. This fixture is synthetic; no held-out or reference-answer input +is used, and no biological recipe or accuracy improvement is claimed. diff --git a/docs/validation/basicpy_parent_numeric_20260930/pipeline-live-wheel.py b/docs/validation/basicpy_parent_numeric_20260930/pipeline-live-wheel.py new file mode 100644 index 0000000000..aef9eb3e42 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/pipeline-live-wheel.py @@ -0,0 +1,53 @@ +"""Controlled 24-site same-channel fit, not a blind biological recipe.""" + +from pathlib import Path + +from openhcs.constants import GroupBy, Microscope, VariableComponents +from openhcs.core.config import ( + LazyNapariStreamingConfig, + LazyPathPlanningConfig, + LazyProcessingConfig, + LazyStepMaterializationConfig, + LazyVFSConfig, + MaterializationBackend, + PipelineConfig, +) +from openhcs.core.steps.function_step import FunctionStep +from openhcs.processing.backends.enhance.basic_processor_jax import ( + basic_flatfield_correction_jax, +) +from zmqruntime.config import TransportMode + +pipeline_config = PipelineConfig( + num_workers=1, + use_threading=True, + microscope=Microscope.IMAGEXPRESS, + path_planning_config=LazyPathPlanningConfig( + global_output_folder=Path( + "/home/ts/wt/openhcs-issue-batch-20260929/" + "basicpy-live-wheel-20260930/outputs" + ), + ), + vfs_config=LazyVFSConfig(materialization_backend=MaterializationBackend.DISK), +) +pipeline_steps = [ + FunctionStep( + func=( + basic_flatfield_correction_jax, + {"max_iterations": 100, "working_size": None, "get_darkfield": False}, + ), + name="BaSiCFit", + processing_config=LazyProcessingConfig( + variable_components=[VariableComponents.SITE], + group_by=GroupBy.CHANNEL, + ), + step_materialization_config=LazyStepMaterializationConfig(enabled=True), + napari_streaming_config=LazyNapariStreamingConfig( + enabled=True, + port=5596, + host="127.0.0.1", + transport_mode=TransportMode.TCP, + persistent=True, + ), + ), +] diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-live-wheel.sh b/docs/validation/basicpy_parent_numeric_20260930/run-live-wheel.sh new file mode 100644 index 0000000000..4072195641 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/run-live-wheel.sh @@ -0,0 +1,25 @@ +#!/bin/sh +# Three actual installed candidate wheels; no shared installation. +set -eu +unset PYTHONPATH +export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 +export OPENHCS_CPU_ONLY=true OPENHCS_SUBPROCESS_NO_GPU=1 POLYSTORE_SUBPROCESS_NO_GPU=1 +export OPENHCS_USE_THREADING=true OPENHCS_HEADLESS=true CUDA_VISIBLE_DEVICES= +export JAX_PLATFORMS=cpu XLA_FLAGS=--xla_cpu_multi_thread_eigen=false +export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMBA_NUM_THREADS=1 +export QT_QPA_PLATFORM=xcb DISPLAY=:91 LIBGL_ALWAYS_SOFTWARE=1 +export POLYSTORE_IMAGEJ_CACHE_ROOT=/home/ts/.cache/polystore/imagej +export POLYSTORE_IMAGEJ_ALLOW_DOWNLOAD=false +export XDG_CACHE_HOME=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/cache +export XDG_CONFIG_HOME=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/config +export XDG_DATA_HOME=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/data +export XDG_STATE_HOME=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/state +export XDG_RUNTIME_DIR=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/runtime +export TMPDIR=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930 +export NUMBA_CACHE_DIR=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/numba +export MPLCONFIGDIR=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/matplotlib +export OPENHCS_UI_CONFIG_CACHE_FILE=/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930/ui_config.config +export OPENHCS_AGENT_READ_ROOTS=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/openhcs-issue-batch-20260929/basicpy-live-wheel-20260930:/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930:/home/ts/.cache/agent-scratch/basicpy-package-20260930/installed131:/home/ts/.openhcs/tcp/5996.startup.lock:/home/ts/.openhcs/tcp/6996.startup.lock +export OPENHCS_AGENT_WRITE_ROOTS=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-live-wheel-20260930:/home/ts/.cache/agent-scratch/basicpy-live-wheel-20260930:/home/ts/.openhcs/tcp/5996.startup.lock:/home/ts/.openhcs/tcp/6996.startup.lock +cd /home/ts/wt/basicpy-live-candidate-20260930 +exec /home/ts/wt/basicpy-live-candidate-20260930/.venv/bin/python -B "$@" diff --git a/openhcs/processing/backends/enhance/basic_processor_jax.py b/openhcs/processing/backends/enhance/basic_processor_jax.py index 8816f3b37c..12b0a5087f 100644 --- a/openhcs/processing/backends/enhance/basic_processor_jax.py +++ b/openhcs/processing/backends/enhance/basic_processor_jax.py @@ -2,7 +2,6 @@ from __future__ import annotations -import jax.numpy as jnp import numpy as np from basicpy import BaSiC from basicpy.basicpy import FittingMode @@ -17,7 +16,7 @@ MainFlowStackOutputSpec, ) from openhcs.core.config import DtypeConfig -from openhcs.core.memory import jax as jax_func +from openhcs.core.memory import numpy as numpy_func from openhcs.core.pipeline.function_contracts import ( allowed_group_by, artifact_outputs, @@ -53,12 +52,12 @@ def _fitted_field_output(name: str) -> ArtifactSpec: DARKFIELD_OUTPUT = _fitted_field_output("basic_darkfield") -@jax_func(contract=ProcessingContract.PURE_3D, dtype_config_default=DtypeConfig()) +@numpy_func(contract=ProcessingContract.PURE_3D, dtype_config_default=DtypeConfig()) @allowed_group_by(GroupBy.CHANNEL) @required_variable_components(FittedIlluminationFieldOutput.observation_axis) @artifact_outputs(CORRECTED_OUTPUT, FLATFIELD_OUTPUT, DARKFIELD_OUTPUT) def basic_flatfield_correction_jax( - image: jnp.ndarray, + image: np.ndarray, max_iterations: int = 50, epsilon: float = 0.1, smoothness_flatfield: float = 1.0, @@ -67,7 +66,7 @@ def basic_flatfield_correction_jax( get_darkfield: bool = False, fitting_mode: FittingMode = FittingMode.ladmap, working_size: int | None = 128, -) -> tuple[jnp.ndarray, FittedIlluminationFieldOutput, FittedIlluminationFieldOutput]: +) -> tuple[np.ndarray, FittedIlluminationFieldOutput, FittedIlluminationFieldOutput]: """Fit one BaSiC model to independent observations and apply its fields. The leading N axis contains independent timepoints or mosaic positions, @@ -75,6 +74,9 @@ def basic_flatfield_correction_jax( group_by=CHANNEL and variable_components=[SITE] to fit across mosaic fields. Z-only stacks are not independent-observation ensembles and are rejected by the declared SITE requirement before pipeline execution. + BaSiCPy's public fit/transform boundary uses NumPy arrays; its internal JAX + solver does not imply JAX-array transport or require a GPU. CPU-only process + policy selects the solver's CPU platform before importing the dependency. All observations must share a spatial grid and acquisition illumination. Several diverse observations are needed to separate stationary shading from biology; two is only a minimum input sanity check, not identifiability. @@ -122,7 +124,7 @@ def basic_flatfield_correction_jax( corrected = model.fit_transform(observations, timelapse=False) observation_count = observations.shape[0] return ( - jnp.asarray(corrected), - FittedIlluminationFieldOutput(jnp.asarray(model.flatfield), observation_count), - FittedIlluminationFieldOutput(jnp.asarray(model.darkfield), observation_count), + np.asarray(corrected), + FittedIlluminationFieldOutput(np.asarray(model.flatfield), observation_count), + FittedIlluminationFieldOutput(np.asarray(model.darkfield), observation_count), ) diff --git a/openhcs/utils/environment.py b/openhcs/utils/environment.py index 38ca51f16c..935b70c4b3 100644 --- a/openhcs/utils/environment.py +++ b/openhcs/utils/environment.py @@ -26,6 +26,7 @@ class OpenHCSProcessEnvironment: numba_sys_monitoring_key = "NUMBA_ENABLE_SYS_MONITORING" subprocess_no_gpu_key = "OPENHCS_SUBPROCESS_NO_GPU" polystore_subprocess_no_gpu_key = "POLYSTORE_SUBPROCESS_NO_GPU" + jax_platforms_key = "JAX_PLATFORMS" use_threading_key = "OPENHCS_USE_THREADING" @staticmethod @@ -119,6 +120,8 @@ def project_dependency_gpu_import_policy( if cls.gpu_imports_disabled(values): values[cls.subprocess_no_gpu_key] = "1" values[cls.polystore_subprocess_no_gpu_key] = "1" + if cls.cpu_only_mode(values): + values[cls.jax_platforms_key] = "cpu" @classmethod def worker_profile_directory( diff --git a/tests/integration/test_basicpy_real_fit.py b/tests/integration/test_basicpy_real_fit.py index e133b0afb1..8c271094f9 100644 --- a/tests/integration/test_basicpy_real_fit.py +++ b/tests/integration/test_basicpy_real_fit.py @@ -11,6 +11,25 @@ from tests.diagnostics.basicpy_observation_fixture import shaded_observations +def test_real_algorithm_uses_numpy_transport_and_is_cpu_catalog_eligible(monkeypatch): + from openhcs.constants import MemoryType + from openhcs.core.callable_contract import CallableContract + from openhcs.processing.backends.lib_registry.openhcs_registry import ( + _catalog_memory_types, + _module_declares_allowed_memory_type, + ) + + monkeypatch.setenv("OPENHCS_CPU_ONLY", "true") + contract = CallableContract.from_callable(basic_flatfield_correction_jax) + assert contract.declared_memory_types == frozenset({MemoryType.NUMPY}) + assert _catalog_memory_types(basic_flatfield_correction_jax) == frozenset( + {MemoryType.NUMPY} + ) + assert _module_declares_allowed_memory_type( + basic_flatfield_correction_jax.__module__, frozenset({MemoryType.NUMPY.value}) + ) + + def run_fit(observations): return basic_flatfield_correction_jax( observations, diff --git a/tests/unit/test_process_environment.py b/tests/unit/test_process_environment.py index 77f0c8b27a..02c2aa5cc5 100644 --- a/tests/unit/test_process_environment.py +++ b/tests/unit/test_process_environment.py @@ -81,6 +81,7 @@ def test_cpu_only_mode_projects_gpu_import_policy_to_dependencies() -> None: OpenHCSProcessEnvironment.cpu_only_key: "true", OpenHCSProcessEnvironment.subprocess_no_gpu_key: "1", OpenHCSProcessEnvironment.polystore_subprocess_no_gpu_key: "1", + OpenHCSProcessEnvironment.jax_platforms_key: "cpu", } assert OpenHCSProcessEnvironment.gpu_imports_disabled(environment) is True @@ -93,3 +94,10 @@ def test_subprocess_gpu_suppression_projects_without_enabling_cpu_only() -> None assert OpenHCSProcessEnvironment.cpu_only_mode(environment) is False assert environment[OpenHCSProcessEnvironment.subprocess_no_gpu_key] == "1" assert environment[OpenHCSProcessEnvironment.polystore_subprocess_no_gpu_key] == "1" + assert OpenHCSProcessEnvironment.jax_platforms_key not in environment + + +def test_cpu_only_mode_selects_cpu_even_if_jax_requested_an_accelerator() -> None: + environment = {OpenHCSProcessEnvironment.jax_platforms_key: "cuda"} + OpenHCSProcessEnvironment.enable_cpu_only_mode(environment) + assert environment[OpenHCSProcessEnvironment.jax_platforms_key] == "cpu" From 25e695c9f8b69ae97c29b73ef11e5c824b54ab75 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 19:18:28 -0400 Subject: [PATCH 12/22] Retain installed BaSiCPy numeric and metadata failure witness --- .../check-persisted-live.py | 53 +++++++++++++++++++ .../installed-failure-receipt.rst | 51 ++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/check-persisted-live.py create mode 100644 docs/validation/basicpy_parent_numeric_20260930/installed-failure-receipt.rst diff --git a/docs/validation/basicpy_parent_numeric_20260930/check-persisted-live.py b/docs/validation/basicpy_parent_numeric_20260930/check-persisted-live.py new file mode 100644 index 0000000000..0b55cc7f9e --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/check-persisted-live.py @@ -0,0 +1,53 @@ +"""Read-only numeric witness for the retained synthetic failed live run. + +This checks pixels and MCP transport, not successful metadata publication or +biological validity. It neither fits a new model nor modifies scientific data. +""" + +import json +import sys +from pathlib import Path + +import numpy as np +import tifffile + + +def main(root: Path) -> None: + output = root / "outputs" / "synthetic_openhcs" + flat = tifffile.imread(output / "checkpoints_results" / "A01_w1_basic_flatfield_step0.tif") + dark = tifffile.imread(output / "checkpoints_results" / "A01_w1_basic_darkfield_step0.tif") + assert flat.shape == dark.shape == (32, 32) + assert flat.dtype == dark.dtype == np.dtype("float32") + assert np.isfinite(flat).all() and (flat > 0).all() + assert np.count_nonzero(dark) == 0 + corrected_files = sorted((output / "images").glob("*.tif")) + assert len(corrected_files) == 24 + max_error = 0.0 + fractional_count = 0 + for index, path in enumerate(corrected_files, start=1): + source_name = f"A01_s{index:03d}_w1_z001_t001.tif" + raw = tifffile.imread(root / "synthetic" / "TimePoint_1" / source_name) + corrected = tifffile.imread(path) + checkpoint = tifffile.imread(output / "checkpoints" / path.name) + assert corrected.dtype == np.dtype("float32") + assert corrected.shape == raw.shape == (32, 32) + np.testing.assert_array_equal(corrected, checkpoint) + expected = (raw.astype(np.float32) - dark) / flat + np.testing.assert_allclose(corrected, expected, rtol=2e-6, atol=2e-3) + max_error = max(max_error, float(np.max(np.abs(corrected - expected)))) + fractional_count += int(np.count_nonzero(corrected != np.floor(corrected))) + assert fractional_count > 0 + receipts = root / "mcp-installed-paired" + for index, expected in ((20, tifffile.imread(corrected_files[0])), (21, dark), (22, flat)): + receipt = json.loads((receipts / f"receipt-{index:03d}.json").read_text()) + assert receipt["returncode"] == 0 + payload = receipt["payload"]["results"][0]["payloads"][0] + assert payload["observed"] and payload["sample_omitted_count"] == 0 + record = payload["records"][0] + assert record["components"]["site"] == 1 + np.testing.assert_array_equal(np.asarray(record["array_values"], dtype=np.float32), expected) + print(json.dumps({"corrected_images": 24, "shape": [32, 32], "dtype": "float32", "max_same_fit_formula_error": max_error, "fractional_pixels": fractional_count, "mcp_exact_full_plane_matches": 3, "flatfield_range": [float(flat.min()), float(flat.max())], "darkfield_nonzero": 0, "pipeline_status": "FAILED_METADATA_PUBLICATION", "biological_acceptance": "NOT_ASSESSED"})) + + +if __name__ == "__main__": + main(Path(sys.argv[1])) diff --git a/docs/validation/basicpy_parent_numeric_20260930/installed-failure-receipt.rst b/docs/validation/basicpy_parent_numeric_20260930/installed-failure-receipt.rst new file mode 100644 index 0000000000..4738f20b6a --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/installed-failure-receipt.rst @@ -0,0 +1,51 @@ +Installed solver execution: numeric success, publication failure +=============================================================== + +Parent PR217 candidate a3d0ad1c0 includes normally merged main2b7969f700. +The paired installed environment is +``/home/ts/wt/basicpy-live-candidate-20260930/.venv``. Shared installation, +desktop :0, protected H002 viewer and frozen blind trials remain unchanged. +This is a synthetic control, not biological acceptance or a promoted recipe. + +The retained receipt root is +``/home/ts/wt/openhcs-issue-batch-20260929/basicpy-live-wheel-20260930``. +Requests and receipts below are under ``mcp-installed-paired``. + +* Healthy installed MCP reports its site-packages source and no stale files. +* Actual CPU discovery exposes the real BaSiCPy adapter, not the approximation. +* Accepted complete lazy pipeline assembles24 independent SITE observations of + one channel. Compile job-1 succeeds in1.31s. +* Execution job-2, f76830cc-57cc-4854-ab0f-cd18f84c09da, was submitted once. + Its fitting axis completes, but final status is FAILED in11.34s:24 ordinary + checkpoint images lack typed produced addresses. Receipt014 and native log + retain the original failure. Neither checkpoint removal nor guard weakening + is an acceptance substitute. Avicenna owns the coherent issue315/PR316 repair + and has this new multisite, multi-output witness. +* All24 persisted corrected images are32x32 float32 and exactly equal to their + named checkpoints and ``(raw - same_fit_darkfield) / same_fit_flatfield``. + Maximum formula error is0;24566 pixels retain fractions. Same-fit fields are + float32, flatfield range0.5081148744–1.2236384153, darkfield zero as declared. + Reproduce read-only with ``check-persisted-live.py RECEIPT_ROOT``. This proves + numeric persistence, not successful final metadata reconciliation. +* Actual managed Napari reports3 mounted routes,24 corrected planes and two + fitted fields, with.65 micrometer XY scaling. MCP exact full32x32 samples of + corrected site1, darkfield and flatfield match persisted pixels exactly + (receipts020–022). No completeness or biological inference comes from counts. +* The first probe omitted declared TCP transport and timed out (receipt015). + Read-only reconciliation with transport_mode=tcp returns promptly (017). + No restart, execution replay or increased observation timeout was used. +* Exact viewer20904/create_time1790809725.82 and runtime13045/create_time + 1790809475.24 were closed through MCP with process_exited=true (023–024). + MCP12670 exited normally. Independent ps and ss confirm all three gone and + ports5596/5996/6996 absent. Parent validation lock was released. + +Native logs, startup journals and viewer settings were archived and listed +successfully as ``runtime-scratch-logs-verified.tgz`` before removing the owned +37MiB disposable scratch directory. Failed outputs, requests, receipts, wheels, +candidate environment and reproducible source remain on persistent storage. + +The fork's actual PyPI release job36783511705 remains queued without a runner; +no publication or ordinary dependency availability is claimed. PR217 stays +draft until publication and coherent installed end-to-end acceptance. Global +user installation is not modified. Full NRA/R1 and biological QA are not proved +by focused tests, the original structural ratchet or this numeric witness. From a4c092fc1460b5070623a0eacfcd52b0180c1b76 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 19:51:48 -0400 Subject: [PATCH 13/22] Verify installed BaSiC fit and complete typed checkpoint publication --- .../check-publication-fixed.py | 79 +++++++++++++++++++ .../pipeline-publication-fixed.py | 53 +++++++++++++ .../publication-fixed-installed.rst | 76 ++++++++++++++++++ .../run-publication-fixed.sh | 25 ++++++ 4 files changed, 233 insertions(+) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py create mode 100644 docs/validation/basicpy_parent_numeric_20260930/pipeline-publication-fixed.py create mode 100644 docs/validation/basicpy_parent_numeric_20260930/publication-fixed-installed.rst create mode 100644 docs/validation/basicpy_parent_numeric_20260930/run-publication-fixed.sh diff --git a/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py b/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py new file mode 100644 index 0000000000..1274bfe136 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py @@ -0,0 +1,79 @@ +"""Read-only installed synthetic fit and complete typed publication witness.""" + +import hashlib +import json +import sys +from pathlib import Path + +import numpy as np +import tifffile + +from openhcs.core.runtime_image_values import ImagePayloadMetadata +from openhcs.core.source_metadata import SourceVoxelSpacing +from openhcs.core.source_workspace_projection import VirtualWorkspaceSourceProjection + + +def main(root: Path) -> None: + output = root / "outputs" / "synthetic_openhcs" + receipts = root / "mcp-installed" + status = json.loads((receipts / "receipt-015.json").read_text()) + execution = status["payload"]["results"][0]["payloads"][0] + assert status["returncode"] == 0 and execution["status"] == "complete" + assert execution["response"]["execution"]["error"] is None + fixture = json.loads((root / "fixture.json").read_text()) + for entry in fixture["files"]: + assert hashlib.sha256((root / "synthetic" / entry["path"]).read_bytes()).hexdigest() == entry["sha256"] + metadata = json.loads((output / "openhcs_metadata.json").read_text()) + reopened = VirtualWorkspaceSourceProjection.from_openhcs_metadata(output, metadata) + inventory = {} + for name, expected_count in (("images", 24), ("checkpoints", 48), ("checkpoints_results", 2)): + entry = metadata["subdirectories"][name] + paths = {str(path.relative_to(output)) for path in (output / name).glob("*.tif")} + assert len(paths) == expected_count + assert set(entry["image_files"]) == paths + assert {item["virtual_path"] for item in entry["source_projection"]} == paths + assert paths.issubset(reopened.source_projections_by_virtual_path) + for item in entry["source_projection"]: + retained = ImagePayloadMetadata.from_mapping(item["image_metadata"]) + assert retained.source_voxel_spacing == SourceVoxelSpacing((0.65, 0.65)) + assert retained.source_spatial_domain.source_shape_yx == (32, 32) + if name == "checkpoints_results": + assert retained.source_image_provenance_planes.contributor_count == 24 + assert retained.plane_axis is None + inventory[name] = expected_count + flat = tifffile.imread(output / "checkpoints_results" / "A01_w1_basic_flatfield_step0.tif") + dark = tifffile.imread(output / "checkpoints_results" / "A01_w1_basic_darkfield_step0.tif") + assert flat.shape == dark.shape == (32, 32) + assert flat.dtype == dark.dtype == np.dtype("float32") + assert np.isfinite(flat).all() and (flat > 0).all() + assert np.count_nonzero(dark) == 0 + corrected_files = sorted((output / "images").glob("*.tif")) + max_error = 0.0 + fractional_count = 0 + for index, path in enumerate(corrected_files, start=1): + raw = tifffile.imread(root / "synthetic" / "TimePoint_1" / f"A01_s{index:03d}_w1_z001_t001.tif") + corrected = tifffile.imread(path) + assert corrected.dtype == np.dtype("float32") + assert corrected.shape == raw.shape == (32, 32) + np.testing.assert_array_equal(corrected, tifffile.imread(output / "checkpoints" / path.name)) + expected = (raw.astype(np.float32) - dark) / flat + np.testing.assert_allclose(corrected, expected, rtol=2e-6, atol=2e-3) + max_error = max(max_error, float(np.max(np.abs(corrected - expected)))) + fractional_count += int(np.count_nonzero(corrected != np.floor(corrected))) + assert fractional_count > 0 + for index, expected in ((18, tifffile.imread(corrected_files[0])), (19, dark), (20, flat)): + receipt = json.loads((receipts / f"receipt-{index:03d}.json").read_text()) + assert receipt["returncode"] == 0 + payload = receipt["payload"]["results"][0]["payloads"][0] + assert payload["observed"] and payload["sample_omitted_count"] == 0 + record = payload["records"][0] + assert record["components"]["site"] == 1 + np.testing.assert_array_equal(np.asarray(record["array_values"], dtype=np.float32), expected) + print(json.dumps({"pipeline_status": "COMPLETE", "typed_file_inventory": inventory, + "max_same_fit_formula_error": max_error, "fractional_pixels": fractional_count, + "mcp_exact_full_plane_matches": 3, "source_pixels_unchanged": True, + "biological_acceptance": "NOT_ASSESSED"})) + + +if __name__ == "__main__": + main(Path(sys.argv[1])) diff --git a/docs/validation/basicpy_parent_numeric_20260930/pipeline-publication-fixed.py b/docs/validation/basicpy_parent_numeric_20260930/pipeline-publication-fixed.py new file mode 100644 index 0000000000..6af69ac835 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/pipeline-publication-fixed.py @@ -0,0 +1,53 @@ +"""Same 24-site synthetic fit; only the output root differs from predecessor.""" + +from pathlib import Path + +from openhcs.constants import GroupBy, Microscope, VariableComponents +from openhcs.core.config import ( + LazyNapariStreamingConfig, + LazyPathPlanningConfig, + LazyProcessingConfig, + LazyStepMaterializationConfig, + LazyVFSConfig, + MaterializationBackend, + PipelineConfig, +) +from openhcs.core.steps.function_step import FunctionStep +from openhcs.processing.backends.enhance.basic_processor_jax import ( + basic_flatfield_correction_jax, +) +from zmqruntime.config import TransportMode + +pipeline_config = PipelineConfig( + num_workers=1, + use_threading=True, + microscope=Microscope.IMAGEXPRESS, + path_planning_config=LazyPathPlanningConfig( + global_output_folder=Path( + "/home/ts/wt/openhcs-issue-batch-20260929/" + "basicpy-publication-fixed-20260930/outputs" + ), + ), + vfs_config=LazyVFSConfig(materialization_backend=MaterializationBackend.DISK), +) +pipeline_steps = [ + FunctionStep( + func=( + basic_flatfield_correction_jax, + {"max_iterations": 100, "working_size": None, "get_darkfield": False}, + ), + name="BaSiCFit", + processing_config=LazyProcessingConfig( + variable_components=[VariableComponents.SITE], + group_by=GroupBy.CHANNEL, + ), + step_materialization_config=LazyStepMaterializationConfig(enabled=True), + napari_streaming_config=LazyNapariStreamingConfig( + enabled=True, + port=5596, + host="127.0.0.1", + transport_mode=TransportMode.TCP, + persistent=True, + ), + ), +] diff --git a/docs/validation/basicpy_parent_numeric_20260930/publication-fixed-installed.rst b/docs/validation/basicpy_parent_numeric_20260930/publication-fixed-installed.rst new file mode 100644 index 0000000000..13ced7e71e --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/publication-fixed-installed.rst @@ -0,0 +1,76 @@ +Installed BaSiC publication checkpoint +===================================== + +Parent integration owner; source candidate2b85aeb26a9292116dc84d5e4cb27df57d3169ed. +Normally merged current main447c926346 and artifact owner PR316416de2. +The authoritative output-storage target now publishes its actual persisted +runtime images. No checkpoint removal, metadata guard weakening, approximate +solver, copied registry or shared installation change is included. + +Sixteen integrated controls passed in18.44s, including actual numerical fitting, +acquisition parsing and named/ordinary output publication. Peak source process +RSS1303392KiB. An earlier invocation guessed a nonexistent test filename and +exited4 without running tests; that failure is not included in the pass count. +The unchanged packaged structural ratchet compared5197 actual metrics with +ForeignAbsenceProbe +1 in function_outputs.py. This is NOT a guard pass or full +NRA/R1 proof. Parent owns final integration; the consolidation requested of +PR316's declaration owner is documented in issuecomment5921773256. Do not +waive or change the baseline to hide the positive delta. + +Persistent receipt root: +``/home/ts/wt/openhcs-issue-batch-20260929/basicpy-publication-fixed-20260930``. +All local wheels and requests/receipts remain there. The isolated installed +candidate is ``/home/ts/wt/basicpy-live-candidate-20260930/.venv`` and reuses +existing heavy third-party packages without downloading or duplicating Fiji. +Its OpenHCS wheel declares openhcs-basicpy>=1.3.1,<1.4 as an ordinary dependency, +preserving the explicit unsupported Intel-macOS marker, with no direct Git URL. +ArrayBridge uses the exact recorded d6d92a7 source. Shared main/env are untouched. + +``pipeline-publication-fixed.py`` preserves the predecessor's real solver, +24 SITE observation domain, one channel, fitting settings and all checkpoints. +Only the output root differs. The first attempt to plan the read-only predecessor +fixture was rejected by path policy before dispatch. The existing deterministic +fixture writer then produced an independent identical synthetic acquisition in +the new writable ledger. It was not permission to modify the old inputs. +Receipt006 retains rejection;008 retains the accepted artifact plan. No blind +input, held-out answer or frozen biological trial was opened or changed. + +Installed MCP81957 reports current site-packages source, packaged resources +ready, no stale files. Exact owned execution child82703/create1790811951.09 +started once and was observed through its original handle to readiness. +Preparation returned READY; no observation deadline was raised or startup +replayed. Compile2538e165-1b94-4712-a2db-682f26555c86 succeeded in1.42s. +Execution0173d6d6-94d8-4ac4-9b59-1cd5a3b86b34 completed in12.16s with no error, +including final strict metadata reconciliation. Receipt015 is the terminal +success; the original f76830cc failed predecessor remains independently retained. + +``check-publication-fixed.py RECEIPT_ROOT`` verifies read-only: + +* All24 primary images,48 checkpoint images and two same-fit field images have + complete matching typed metadata inventories and reopen through the existing + virtual workspace source projection. No cross-target projection is substituted. +* All images/fields retain32x32 spatial domains and.65 micrometer XY calibration; + both aggregate fields retain24 source contributors and no plane axis. +* All24 corrected float32 planes match their checkpoints and + (raw - darkfield) / flatfield with maximum error0.0.24566 fractional pixels + survive, and every synthetic input SHA256 still matches its original receipt. +* Native MCP full32x32 corrected, darkfield and flatfield samples match disk + exactly, with no omitted elements. Actual Napari reports three managed routes, + 26 items and.65 XY scaling. This is transport/numeric evidence, not visual + segmentation, biological accuracy or a promoted illumination recipe. + +Viewer88539/create1790812104.98 and native82703 were closed through MCP with +request_attempted/acknowledged/process_exited all true (021/022). MCP81957 exited +normally, and independent ps/ss confirmed those exact processes and ports +5596/5996/6996 absent. Desktop :0, H002 and the parent isolated X/VNC remain. +Logs/startup/settings were archived and listed as runtime-logs-and-settings.tgz +before removing the owned21MiB scratch and20MiB generated build directory. +All source, outputs, failed requests, receipts, candidate and wheels are retained. + +Fork release36783511705 remains queued with no assigned runner or executed +steps. PyPI's exact1.3.1 JSON endpoint still returns404. No duplicate publisher, +manual token extraction, publication claim or unavailable-dependency main merge +is justified. The ordinary dependency source is visible in PR217; successful +registry publication/resolution and the remaining owner guard are separate +requirements before final merge/activation. The expanded persistent goal remains +active; this checkpoint does not establish full ZIP-refactor completion. diff --git a/docs/validation/basicpy_parent_numeric_20260930/run-publication-fixed.sh b/docs/validation/basicpy_parent_numeric_20260930/run-publication-fixed.sh new file mode 100644 index 0000000000..f23e572900 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/run-publication-fixed.sh @@ -0,0 +1,25 @@ +#!/bin/sh +# Fresh installed publication control; the failed predecessor is preserved. +set -eu +unset PYTHONPATH +export PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 +export OPENHCS_CPU_ONLY=true OPENHCS_SUBPROCESS_NO_GPU=1 POLYSTORE_SUBPROCESS_NO_GPU=1 +export OPENHCS_USE_THREADING=true OPENHCS_HEADLESS=true CUDA_VISIBLE_DEVICES= +export JAX_PLATFORMS=cpu XLA_FLAGS=--xla_cpu_multi_thread_eigen=false +export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMBA_NUM_THREADS=1 +export QT_QPA_PLATFORM=xcb DISPLAY=:91 LIBGL_ALWAYS_SOFTWARE=1 +export POLYSTORE_IMAGEJ_CACHE_ROOT=/home/ts/.cache/polystore/imagej +export POLYSTORE_IMAGEJ_ALLOW_DOWNLOAD=false +export XDG_CACHE_HOME=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/cache +export XDG_CONFIG_HOME=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/config +export XDG_DATA_HOME=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/data +export XDG_STATE_HOME=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/state +export XDG_RUNTIME_DIR=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/runtime +export TMPDIR=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930 +export NUMBA_CACHE_DIR=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/numba +export MPLCONFIGDIR=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/matplotlib +export OPENHCS_UI_CONFIG_CACHE_FILE=/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930/ui_config.config +export OPENHCS_AGENT_READ_ROOTS=/home/ts/wt/openhcs-basicpy-parent-integration-20260930:/home/ts/wt/openhcs-issue-batch-20260929/basicpy-live-wheel-20260930:/home/ts/wt/openhcs-issue-batch-20260929/basicpy-publication-fixed-20260930:/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930:/home/ts/.openhcs/tcp/5996.startup.lock:/home/ts/.openhcs/tcp/6996.startup.lock +export OPENHCS_AGENT_WRITE_ROOTS=/home/ts/wt/openhcs-issue-batch-20260929/basicpy-publication-fixed-20260930:/home/ts/.cache/agent-scratch/basicpy-publication-fixed-20260930:/home/ts/.openhcs/tcp/5996.startup.lock:/home/ts/.openhcs/tcp/6996.startup.lock +cd /home/ts/wt/basicpy-live-candidate-20260930 +exec /home/ts/wt/basicpy-live-candidate-20260930/.venv/bin/python -B "$@" From 71b699cd6b34c8809f248ad72e0d8bffe3b99340 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 20:18:09 -0400 Subject: [PATCH 14/22] Record installed typed-field fit and require released dtype-default API --- .../check-publication-fixed.py | 24 +++++-- .../target-owner-installed.rst | 65 +++++++++++++++++++ pyproject.toml | 2 +- 3 files changed, 83 insertions(+), 8 deletions(-) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst diff --git a/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py b/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py index 1274bfe136..78216d99b9 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py +++ b/docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py @@ -1,8 +1,8 @@ """Read-only installed synthetic fit and complete typed publication witness.""" +import argparse import hashlib import json -import sys from pathlib import Path import numpy as np @@ -13,10 +13,11 @@ from openhcs.core.source_workspace_projection import VirtualWorkspaceSourceProjection -def main(root: Path) -> None: - output = root / "outputs" / "synthetic_openhcs" - receipts = root / "mcp-installed" - status = json.loads((receipts / "receipt-015.json").read_text()) +def main(root: Path, output_directory: str, receipt_directory: str, + status_receipt: int, first_sample_receipt: int) -> None: + output = root / output_directory / "synthetic_openhcs" + receipts = root / receipt_directory + status = json.loads((receipts / f"receipt-{status_receipt:03d}.json").read_text()) execution = status["payload"]["results"][0]["payloads"][0] assert status["returncode"] == 0 and execution["status"] == "complete" assert execution["response"]["execution"]["error"] is None @@ -61,7 +62,8 @@ def main(root: Path) -> None: max_error = max(max_error, float(np.max(np.abs(corrected - expected)))) fractional_count += int(np.count_nonzero(corrected != np.floor(corrected))) assert fractional_count > 0 - for index, expected in ((18, tifffile.imread(corrected_files[0])), (19, dark), (20, flat)): + for index, expected in enumerate((tifffile.imread(corrected_files[0]), dark, flat), + start=first_sample_receipt): receipt = json.loads((receipts / f"receipt-{index:03d}.json").read_text()) assert receipt["returncode"] == 0 payload = receipt["payload"]["results"][0]["payloads"][0] @@ -76,4 +78,12 @@ def main(root: Path) -> None: if __name__ == "__main__": - main(Path(sys.argv[1])) + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("root", type=Path) + parser.add_argument("--output-directory", default="outputs") + parser.add_argument("--receipt-directory", default="mcp-installed") + parser.add_argument("--status-receipt", type=int, default=15) + parser.add_argument("--first-sample-receipt", type=int, default=18) + args = parser.parse_args() + main(args.root, args.output_directory, args.receipt_directory, + args.status_receipt, args.first_sample_receipt) diff --git a/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst b/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst new file mode 100644 index 0000000000..05647c2480 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst @@ -0,0 +1,65 @@ +Installed fit after target-owner consolidation +============================================= + +Parent integration owner. Installed OpenHCS source c57eaf094fd761755997661b4254076e4efeb017 +normally combines PR217 with PR316's tested target-owner consolidation. +PR316 is merged at 631831bbc1b211a2d1d3da04836dc28cfd4079e7. Its final documentation +was subsequently merged into this branch without changing the fitted source. +The existing materialization declaration owns persistent-backend admission; +consumers do not duplicate its storage flags. The unchanged original structural +ratchets report zero positive deltas. The original R1 comparison timed out at +its unchanged 160-second budget and is not represented as a pass. + +Persistent receipt root:: + + /home/ts/wt/openhcs-issue-batch-20260929/basicpy-publication-fixed-20260930 + +The fresh installed MCP journey uses ``mcp-target-owner`` and +``outputs-target-owner``. Previous outputs and failed attempts are unchanged. +The same 24-SITE synthetic acquisition, real BaSiC solver, iteration settings +and full checkpoint declarations were used. Only the output directory changed. +This is a technical control, not biological or blind-analysis acceptance. + +Compile 4a03a247-b49a-4a7d-994a-b9308badefa2 completed in 1.36 seconds. +Execution c0e62c70-408a-497e-af08-ec0e2af097e5 completed in 12.39 seconds, +including strict final metadata reconciliation. Receipt012 records terminal +success. No execution was replayed or observation timeout increased. + +The read-only checker was run as:: + + sh docs/validation/basicpy_parent_numeric_20260930/run-publication-fixed.sh \ + docs/validation/basicpy_parent_numeric_20260930/check-publication-fixed.py \ + RECEIPT_ROOT --output-directory outputs-target-owner \ + --receipt-directory mcp-target-owner --status-receipt 12 \ + --first-sample-receipt 15 + +It verified all 24 output images, 48 checkpoint images and two same-fit field +images against complete typed inventories and the existing workspace projection. +The correction formula's maximum error is 0.0; 24,566 fractional pixels survive. +All input hashes remain unchanged. Three full-plane native viewer samples match +disk exactly. Both aggregate fields retain 24 contributors, their 32x32 domain +and 0.65-micrometer XY calibration, without a plane axis. + +Receipts018/019 confirm acknowledged MCP close and process exit for the owned +viewer 115466/create1790813119.32 and native worker 109040/create1790812855.10. +MCP107852 subsequently exited normally. Independent process/socket checks found +these processes and ports5596/5996/6996 absent. Logs/settings were archived to +``runtime-target-owner-logs.tgz`` before owned disposable scratch/build cleanup. +All source, inputs, outputs, receipts and installed candidate wheels remain. + +Registry installation is a separate unfinished acceptance boundary. A fresh +34-kilobyte PyPI ArrayBridge0.3.4 wheel rejects ``dtype_config_default`` during +the installed BaSiC adapter's import. The exception and wheel remain in +``artifact-parent-20261001/registry-arraybridge-probe.log`` and +``artifact-parent-20261001/registry-wheels`` under the parent issue-batch ledger. +The successful local candidate contains the reviewed newer ArrayBridge source, +not that published wheel. Accordingly the normal dependency floor is raised +to ArrayBridge0.3.5; no old-API fallback, direct-Git dependency or manual source +pin is added. A companion release must be published before ordinary dependency +resolution is claimed. + +The approved openhcs-basicpy1.3.1 publication job36783511705 remains queued +without a runner or steps. Its exact PyPI endpoint returned HTTP404 on the +2026-10-01 checkpoint. Do not merge an unavailable ordinary dependency into +main or claim publication because an upload job was submitted. PR217 remains +visible as a draft while these public dependency releases are outstanding. diff --git a/pyproject.toml b/pyproject.toml index b868c9c29a..79363717ac 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -104,7 +104,7 @@ dependencies = [ "objectstate>=1.1.8,<2", "python-introspect>=0.1.14,<0.2", "metaclass-registry>=0.2.1,<0.3", - "arraybridge>=0.3.4,<0.4", + "arraybridge>=0.3.5,<0.4", "polystore>=0.3.0,<0.4", "pyqt-reactive>=0.3.25,<0.4", ] From 8e7fa95b607b8a31e45b0256858ae3d095ba57c9 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 20:53:07 -0400 Subject: [PATCH 15/22] Record paired-field main integration and frozen blind candidate boundary --- .../target-owner-installed.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst b/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst index 05647c2480..4212e10089 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/target-owner-installed.rst @@ -58,6 +58,16 @@ to ArrayBridge0.3.5; no old-API fallback, direct-Git dependency or manual source pin is added. A companion release must be published before ordinary dependency resolution is claimed. +After PR317 merged, this dependency branch normally integrates +main74eac059f67389ca071055c93fcbdd556ba79f1a at771dbdd45. Sixteen combined +real-BaSiC numerical and source-image-set identity controls pass in7.12 seconds, +including the stationary-signal negative control. This is source integration +evidence, not a new installed acceptance claim for the combined dependency +branch. The existing isolated candidate is frozen for the independent-author +H003f trial until its terminal freeze and exact process cleanup; no package, +instruction or shared-install mutation is permitted there during that trial. +The companion ArrayBridge release is visible in OpenHCSDev/ArrayBridge PR4. + The approved openhcs-basicpy1.3.1 publication job36783511705 remains queued without a runner or steps. Its exact PyPI endpoint returned HTTP404 on the 2026-10-01 checkpoint. Do not merge an unavailable ordinary dependency into From b519f0a16d262742e0827a8772919f25ae353f43 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 21:12:58 -0400 Subject: [PATCH 16/22] Record BaSiCPy current-main source and actual import boundary --- .../main-integration-20261002.rst | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst new file mode 100644 index 0000000000..7927c9a511 --- /dev/null +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -0,0 +1,41 @@ +BaSiCPy current main integration +============================== + +Parent integration owner. Ordinary merge of main25d56ae3fb9b80acda80f3cf4e1c8667939147eb +into retained PR217 produces ce68038a0ece28cd2fadace8f18cc61f9395d3d2. +No foreign worktree or installed application was edited. Existing gitlink +worktrees remain at their previous source identities; index pins are explicit. + +Three real conflicts were resolved at their original owners: + +* Projection selection keeps main418's separate selection capability, original + ordered-index hook and singleton collapse. Complete-axis validation uses + the existing projection owner's method added by this PR, not a copied check. +* ImagePayloadMetadata keeps main's single decoder/retained-plane methods and + the PR's independent-observation admission. No deleted duplicate reader is + restored. PR394 continues to own its unmerged materialization changes. +* Metadata retains metaclass-registry>=0.2.2 and sets ArrayBridge>=0.3.6 after + the failed0.3.5 publisher and reviewed corrected release. Gitlink1e53d03d + selects exact corrected ArrayBridge main. No existing tag was replaced. + +32 source controls pass,21.06s pytest/25.3s supervisor/1023.7MiB combined peak, +oneCPU,1536MiB kernel ceiling/no swap/60seconds. This includes the real BaSiC +2-D/volume fits, stationary-biology negative control, aggregate contributor +and grid identity, selection and independent cooperative declaration controls. +Original basicpy-current-main-source.log and command.json are retained at +parent issue-batch/s1-installed-20261001. Two original pytest warnings remain. + +Important import boundary: OpenHCS source bootstrap correctly selected this +checkout's existing vendored ArrayBridge d6d92a716 and metaclass448cdf07, +not the wheel paths supplied later in sys.path. Actual import paths are in the +receipt. Thus this source check is not a claim of public dependency selection +or a corrected036 installed context. Do not bypass the bootstrap's stale-source +guard. Current wheel/fresh installation acceptance must use the original +packaged path, where vendored source directories are absent. + +BaSiCPy1.3.1 and metaclass0.2.2 are verified available on PyPI; ArrayBridge's +new original publisher36949376897 is queued at this checkpoint. PR217 remains +draft until actual artifact/API and normal published dependency resolution. +Previous field/MCP24SITE and Python3.14 controls remain pinned to their original +sources, not projected through this main integration. Original failures and +UNKNOWN requests remain. No science success or reserve release is claimed. From b874c1b40d148a3111f1495320138039586c0ac4 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 21:16:01 -0400 Subject: [PATCH 17/22] Keep observation admission with fitted fields instead of generic metadata --- .../main-integration-20261002.rst | 13 +++++++++++++ openhcs/core/runtime_image_values.py | 19 +------------------ .../processing/backends/enhance/flatfield.py | 15 ++++++++++++--- 3 files changed, 26 insertions(+), 21 deletions(-) diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst index 7927c9a511..46876a74c6 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -39,3 +39,16 @@ draft until actual artifact/API and normal published dependency resolution. Previous field/MCP24SITE and Python3.14 controls remain pinned to their original sources, not projected through this main integration. Original failures and UNKNOWN requests remain. No science success or reserve release is claimed. + +Current-main R0 rejected the inherited domain admission method on the generic +ImagePayloadMetadata: GodClassExcess grows17, original exit1,24.83s/79.44MiB. +The original complete changed-product scan and all limits are retained. Fix +ownership, not the measure: independent-observation admission belongs to the +existing FittedIlluminationFieldOutput declaring observation_axis. Move that +algorithm into its existing validate_observation_domain, deleting the generic +metadata extension and forwarding call. It still queries the original metadata +owner's retained component values; no facts are mirrored or new source registry +introduced. IMPL-4/AGENT-4: complete the real field owner, not the generic god +class. This removal touches only PR217's own metadata addition and leaves +current main/PR394's metadata implementation unchanged. Controls must retain +the same negative-axis and contributor behavior. diff --git a/openhcs/core/runtime_image_values.py b/openhcs/core/runtime_image_values.py index cbe5ba6709..4186586d5f 100644 --- a/openhcs/core/runtime_image_values.py +++ b/openhcs/core/runtime_image_values.py @@ -24,7 +24,7 @@ from openhcs.serialization.json import to_jsonable from collections.abc import Mapping -from openhcs.constants.constants import AllComponents, VariableComponents +from openhcs.constants.constants import AllComponents from openhcs.core.alias_property import AliasProperty from openhcs.core.runtime_array_values import ( DataBackedRuntimeArrayPayload, @@ -196,23 +196,6 @@ def retained_plane_component_values( tuple(AllComponents) ) - def require_independent_observation_axis( - self, axis: VariableComponents, *, value_name: str - ) -> None: - """Admit one source-coordinate ensemble without inventing raw-array identity.""" - - if not self.has_values: - return - retained_axes = tuple( - AllComponents.from_value(name) - for name in self.retained_plane_component_values() - ) - if retained_axes != (AllComponents.from_value(axis.value),): - raise ValueError( - f"{value_name} requires independent {axis.name} observations " - "with every other source component fixed." - ) - @classmethod def for_array( cls, diff --git a/openhcs/processing/backends/enhance/flatfield.py b/openhcs/processing/backends/enhance/flatfield.py index 49e616c055..55a25e2bae 100644 --- a/openhcs/processing/backends/enhance/flatfield.py +++ b/openhcs/processing/backends/enhance/flatfield.py @@ -4,7 +4,7 @@ from enum import Enum from typing import ClassVar -from openhcs.constants.constants import VariableComponents +from openhcs.constants.constants import AllComponents, VariableComponents from openhcs.core.projected_image_output import SourceProjectedImageOutput from openhcs.core.runtime_array_values import RuntimeArrayData from openhcs.core.runtime_image_values import image_payload_metadata @@ -35,9 +35,18 @@ def __post_init__(self) -> None: @classmethod def validate_observation_domain(cls, source: RuntimeArrayData) -> None: """Reject mislabeled metadata-backed ensembles before fitting.""" - image_payload_metadata(source).require_independent_observation_axis( - cls.observation_axis, value_name="BaSiC" + metadata = image_payload_metadata(source) + if not metadata.has_values: + return + retained_axes = tuple( + AllComponents.from_value(name) + for name in metadata.retained_plane_component_values() ) + if retained_axes != (AllComponents.from_value(cls.observation_axis.value),): + raise ValueError( + f"BaSiC requires independent {cls.observation_axis.name} observations " + "with every other source component fixed." + ) def with_data(self, data: RuntimeArrayData) -> "FittedIlluminationFieldOutput": return replace(self, data=data) From 67584e43266a4641a0d1e56fc3946683c5463320 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 21:21:29 -0400 Subject: [PATCH 18/22] Apply fitted observation restriction to declared acquisition domains --- .../main-integration-20261002.rst | 18 ++++++++++++++++++ .../processing/backends/enhance/flatfield.py | 8 +++++--- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst index 46876a74c6..bfce983d2d 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -52,3 +52,21 @@ introduced. IMPL-4/AGENT-4: complete the real field owner, not the generic god class. This removal touches only PR217's own metadata addition and leaves current main/PR394's metadata implementation unchanged. Controls must retain the same negative-axis and contributor behavior. + +The next R0 retained one foreign negative-absence probe in the field's early +return, exit1,20.5s/83.11MiB. The actual rule is a restriction on annotated +sources, not missing-state recovery. State it positively: when the existing +metadata owner reports a domain present, require exactly the declared SITE +variation. Plain NumPy remains the original supported unannotated direct-call +contract; retained-plane derivation remains inside its original metadata owner. +No attribute-default reader, reinterpretation of None, exceptions, measure +suppression or extra registry is introduced. Independent direct-array and +wrong annotated-axis controls retain this exact distinction. + +The first mixed fixture helper failed the real stale-external import guard +because it preloaded a foreign metaclass checkout before this populated source +tree. It remains failed; no guard was bypassed. Original source bootstrap next +passes40 controls and fails both real publication journeys because the existing +PolyStore gitlink worktree is older and lacks metadata_config. That actual +source-dependency mismatch is not an application-regression or green journey. +Use the reviewed package/paired source identities for whole-path acceptance. diff --git a/openhcs/processing/backends/enhance/flatfield.py b/openhcs/processing/backends/enhance/flatfield.py index 55a25e2bae..6e0669022e 100644 --- a/openhcs/processing/backends/enhance/flatfield.py +++ b/openhcs/processing/backends/enhance/flatfield.py @@ -36,13 +36,15 @@ def __post_init__(self) -> None: def validate_observation_domain(cls, source: RuntimeArrayData) -> None: """Reject mislabeled metadata-backed ensembles before fitting.""" metadata = image_payload_metadata(source) - if not metadata.has_values: - return retained_axes = tuple( AllComponents.from_value(name) for name in metadata.retained_plane_component_values() ) - if retained_axes != (AllComponents.from_value(cls.observation_axis.value),): + # Plain NumPy observations have no claimed acquisition domain. Restrict + # annotated sources using the metadata owner's declared presence state. + if metadata.has_values and retained_axes != ( + AllComponents.from_value(cls.observation_axis.value), + ): raise ValueError( f"BaSiC requires independent {cls.observation_axis.name} observations " "with every other source component fixed." From 2ee0feb8cd4dfea6a8aad80cf4671aac13c42333 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 21:29:36 -0400 Subject: [PATCH 19/22] Record zero-debt paired BaSiCPy owner and persistence qualification --- .../main-integration-20261002.rst | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst index bfce983d2d..55700117b9 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -70,3 +70,34 @@ passes40 controls and fails both real publication journeys because the existing PolyStore gitlink worktree is older and lacks metadata_config. That actual source-dependency mismatch is not an application-regression or green journey. Use the reviewed package/paired source identities for whole-path acceptance. + +Final paired source checkpoint67584e432 +------------------------------------- + +The original full changed-product R0 now reports zero positive deltas and one +removed projection absence probe: basicpy-domain-rule-r0, exit0, +20.5s/82.26MiB. No limit, detector, exclusion, measure or threshold changed. +Generic runtime_image_values.py is byte-identical to current main. + +The existing owned dependency clones were clean, not symlinks, and their +git-common directories belong to this worktree. Normal submodule initialization +and update aligns PolyStore84f322e, ArrayBridge1e53d03d, metaclass393a7e0, +pyqt-reactivead494877 and ZMQRuntime668edafc with the recorded index. The first +no-fetch attempt did not have all objects/initializations and remains retained; +the ordinary update fetched only the missing source history, with no new +environment, duplicated installation or shared-package mutation. + +All42 unchanged field/family/selection/materialization/checkpoint persistence +and reopen controls pass using that real source bootstrap and actual paired +dependencies: basicpy-domain-rule-paired-source, exit0,24.1s/466.57MiB, +kernel768MiB/no swap/oneCPU/60seconds. Two pytest configuration warnings remain. +Earlier real numerical32-control fit receipt remains separately pinned; its +solver/correction implementation is unchanged by this observation-owner repair. + +The original no-isolation offline package build at67584e432 completes in +9.9s/230.18MiB. The working wheel remains at parent ledger +basicpy-current-main-20261002/wheels-domain-owner, with no source external +directories. Five affected production files match the source exactly. Public +dependency requirements are present in wheel metadata. This is a prepared +candidate, not a private install, registry selection, native viewer or biological +acceptance. Keep PR217 visible and retain all original failed evidence. From c5fe91dee54f45d324ae815b33ebc12bf5c244b1 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 21:44:43 -0400 Subject: [PATCH 20/22] Record public dependency acceptance and remaining release chain --- .../main-integration-20261002.rst | 39 +++++++++++++++++-- 1 file changed, 36 insertions(+), 3 deletions(-) diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst index 55700117b9..22818307e4 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -33,9 +33,11 @@ or a corrected036 installed context. Do not bypass the bootstrap's stale-source guard. Current wheel/fresh installation acceptance must use the original packaged path, where vendored source directories are absent. -BaSiCPy1.3.1 and metaclass0.2.2 are verified available on PyPI; ArrayBridge's -new original publisher36949376897 is queued at this checkpoint. PR217 remains -draft until actual artifact/API and normal published dependency resolution. +BaSiCPy1.3.1, metaclass0.2.2 and ArrayBridge0.3.6 are verified available on +PyPI. ArrayBridge's original publisher36949376897 succeeded01:33:35Z. +PR217 remains draft until normal published dependency resolution; that +original attempt now identifies the additional PolyStore/ZMQ/Qt release +chain described below rather than a pending ArrayBridge publisher. Previous field/MCP24SITE and Python3.14 controls remain pinned to their original sources, not projected through this main integration. Original failures and UNKNOWN requests remain. No science success or reserve release is claimed. @@ -101,3 +103,34 @@ directories. Five affected production files match the source exactly. Public dependency requirements are present in wheel metadata. This is a prepared candidate, not a private install, registry selection, native viewer or biological acceptance. Keep PR217 visible and retain all original failed evidence. + +Actual public dependency checkpoint +---------------------------------- + +The public ArrayBridge0.3.6 wheel AND sdist contain all17 reviewed source +files byte-identical to main/tag1e53d03d. Original metadata/index probe and +materialize_release_files were used once; the hosted wheel SHA256 is +f7fa23f5dcf1925742592109b396fe17662fb56f15e1755a304a45c90e79a249. +Its sdist SHA256 is +b3b8a5821597a3341739657298ce94928cd1f4297a083fec3b28ceb406e09afb. +11 unchanged controls pass importing the actual public wheel in a fresh +process,0.66s/57.57MiB,512MiB kernel/no swap/oneCPU. This is actual +dependency artifact/API acceptance, not an installed OpenHCS journey. + +The original ordinary uv installer dry-run of the prepared wheel against +PyPI fails because only PolyStore<=0.2.19 is public and existing OpenHCS main +requires>=0.3.0,<0.4. Exact probes also show ZMQRuntime0.3.0 and +pyqt-reactive0.3.25 unavailable. PolyStore main84f322e requires ZMQ0.3.0, +so publication order follows that dependency. Parent owns PolyStore release; +Schrodinger owns ZMQ/Qt release coordination. No downgrade, local-only +substitute, repeated download, private environment mutation or optional CI +wait. Original failed normal-resolution log/command remains retained. + +PolyStore source54 controls pass at the unchanged reviewed main84f322e +using current source ZMQ and metaclass plus the actual public ArrayBridge +wheel,4.83s/447.61MiB,768MiB kernel/no swap/oneCPU. Payload fidelity, +namespace declaration ownership, ACK routes, cropped ROI topology/archive +reopen, memory backend and lossless PNG controls remain unchanged. +The earlier42 actual OpenHCS persistence/reopen controls also used84f322e. +These are source release qualification, not public PolyStore availability +or a fresh installed native user journey. That release chain remains active. From 33778332f38a06f946bee3032f4c3422cc74c2db Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 22:52:55 -0400 Subject: [PATCH 21/22] Record current merged MCP and viewer qualification --- .../main-integration-20261002.rst | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst index 22818307e4..739dd9b5e1 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -134,3 +134,36 @@ reopen, memory backend and lossless PNG controls remain unchanged. The earlier42 actual OpenHCS persistence/reopen controls also used84f322e. These are source release qualification, not public PolyStore availability or a fresh installed native user journey. That release chain remains active. + +Current merged MCP/viewer checkpoint +----------------------------------- + +Ordinary merge8c0db128d600b3895f0f30af143f21e159828673 integrates remote +mainfaf8e1f263244bf4d3dbb1e16833c416bbc811bf. This retains the other machine's +compilation/startup placement from PR420 and includes merged PR422 progress +acknowledgements and PR413 original typed native viewer records. No conflict, +shared-checkout edit or installed-package mutation occurred. + +The same42 field/family/selection/materialization/persistence/reopen controls +pass through the actual source bootstrap,13.65s/424.37MiB combined RSS under +the unchanged768MiB/60s supervisor. Original receipt is +s1-installed-20261001/basicpy-faf8-current-main-source.command.json and log in +the parent issue-batch ledger. The two existing pytest configuration warnings +remain. Original changed-product R0 against exact faf8e1f/8c0db128 passes, +13.0s/83.79MiB, no positive deltas; its original receipt is +basicpy-faf8-current-main-r0.command.json and log. No global R1 claim follows. + +Cleanup exposed four Git object alternates in this worktree's own submodule +metadata pointing to a retired worktree. The original stable main submodule +object stores contain every pinned commit. Repointing only those four alternates +restores exact identities; PolyStore release84f322e remains clean and original +git fsck --connectivity-only --no-dangling passes. No source or commit was lost, +no reference/tag rewritten, and no further cleanup is authorized or performed. + +Original ZMQRuntime publisher36952353836 is terminal failure at candidate tests +(exit143); no artifact publication occurred. Schrodinger owns its repair and +the Qt release. Public ArrayBridge0.3.6, metaclass0.2.2 and BaSiCPy1.3.1 remain +available. Parent owns PolyStore publication after public ZMQ exists, then +ordinary dependency resolution and the closed private installed/native journey. +The old configured MCP health check is healthy0.8.6 from installed-main-20260929, +not this checkpoint. Scientific candidate5 remains rejected; reserve sealed. From d908c27ff6a3a68124c20e8acb5bd426e07de938 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Thu, 1 Oct 2026 22:57:24 -0400 Subject: [PATCH 22/22] Ship qualified dependency integration separately from release availability --- .../main-integration-20261002.rst | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst index 739dd9b5e1..18ec59c4a1 100644 --- a/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst +++ b/docs/validation/basicpy_parent_numeric_20260930/main-integration-20261002.rst @@ -1,6 +1,13 @@ BaSiCPy current main integration ============================== +Current delivery decision: merge the qualified source checkpoint without +optional hosted CI waiting. BaSiCPy1.3.1 and ArrayBridge0.3.6, the new registry +requirements of this PR, are public. The still-unpublished PolyStore/ZMQ/Qt +minimums are already present on main and do not originate in this PR. Their +publication and normal installed/live acceptance remain parent-owned next +actions, not an assertion that this source merge makes the install ready. + Parent integration owner. Ordinary merge of main25d56ae3fb9b80acda80f3cf4e1c8667939147eb into retained PR217 produces ce68038a0ece28cd2fadace8f18cc61f9395d3d2. No foreign worktree or installed application was edited. Existing gitlink @@ -35,9 +42,10 @@ packaged path, where vendored source directories are absent. BaSiCPy1.3.1, metaclass0.2.2 and ArrayBridge0.3.6 are verified available on PyPI. ArrayBridge's original publisher36949376897 succeeded01:33:35Z. -PR217 remains draft until normal published dependency resolution; that -original attempt now identifies the additional PolyStore/ZMQ/Qt release -chain described below rather than a pending ArrayBridge publisher. +The earlier draft checkpoint waited for normal published dependency resolution; +that attempt identifies the additional PolyStore/ZMQ/Qt release chain described +below rather than a pending ArrayBridge publisher. The current delivery decision +above ships source independently while retaining that installed boundary. Previous field/MCP24SITE and Python3.14 controls remain pinned to their original sources, not projected through this main integration. Original failures and UNKNOWN requests remain. No science success or reserve release is claimed. @@ -167,3 +175,10 @@ available. Parent owns PolyStore publication after public ZMQ exists, then ordinary dependency resolution and the closed private installed/native journey. The old configured MCP health check is healthy0.8.6 from installed-main-20260929, not this checkpoint. Scientific candidate5 remains rejected; reserve sealed. + +Prepared wheel33778332f builds offline in7.41s/227.71MiB, passes Twine and +has SHA2562990c91e8b8665cea047653a181e9c367f9fd5e866c0794f1f9757fc8b6359ef. +It includes current merged MCP/viewer production code plus this BaSiCPy +integration. Location: parent issue-batch/basicpy-current-main-20261002/ +wheels-faf8-current-main/openhcs-0.8.7-cp311-abi3-linux_x86_64.whl. +No scientific/native or installed-package mutation occurred during this build.