From 8c886e79267f72ad88bda4143845ed1721325dd8 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 14:57:25 -0400 Subject: [PATCH 01/12] Reuse owned region properties for 2D ROI contours (#132) --- ...i_region_properties_checkpoint_20260929.md | 85 +++++++++++ src/polystore/roi.py | 25 ++-- tests/test_roi_region_properties.py | 139 ++++++++++++++++++ 3 files changed, 233 insertions(+), 16 deletions(-) create mode 100644 docs/roi_region_properties_checkpoint_20260929.md create mode 100644 tests/test_roi_region_properties.py diff --git a/docs/roi_region_properties_checkpoint_20260929.md b/docs/roi_region_properties_checkpoint_20260929.md new file mode 100644 index 0000000..2ea4e86 --- /dev/null +++ b/docs/roi_region_properties_checkpoint_20260929.md @@ -0,0 +1,85 @@ +# ROI region-properties checkpoint + +Owner: Lovelace. Integration: OpenHCS coordinator. +Authorized paired work for OpenHCSDev/openhcs #132 and draft PR #206. +Branch: `fix/fragmented-roi-properties-20260929`. +Isolated checkout: `/home/ts/wt/openhcs-input-preparation-20260929/external/PolyStore`. +Fetched/merged base: `origin/main` at `f94bbbe8631a4c78f7ceaec72fb99671b58c9a18`. +Only open PR at orientation: Dependabot #10; no overlapping feature PR. + +## Owned change and deletion + +`TwoDimensionalLabeledMaskROIExtractor.extract` stays the only 2D behavior +owner in the existing `LabeledMaskROIExtractor` registered family. The installed +scikit-image `RegionProperties.image` already caches the exact binary bbox crop; +area/perimeter evaluation uses it. `regionprops` itself calls `find_objects`. +The extractor previously called `find_objects` again, re-indexed its slices, +re-cropped the source, recomputed equality and cast that crop to uint8. + +Remove that scan, import, indexed-slice guard, second crop/equality construction +and cast. Pad the existing unfilled `region.image`, and translate contours using +the already-owned bbox. Keep area thresholds, labels, order, contour level, +origin, metadata, full-canvas MaskShape representation and archive codecs. +No alternate extractor, cache, format, registry or source-pruning change. + +## Pattern review and new cases + +- BOUND-2: consume public `RegionProperties.bbox`/`image`, not a separately + reconstructed representation of facts that object already models. The 2D + method no longer imports or calls `find_objects`. +- IMPL-12/13: delete the repeated scan/crop procedure in place; do not copy it + into an OpenHCS extractor or benchmark cache. Numeric dimensional acceptance + and contour options remain existing geometric contracts, not string dispatch. +- IDEN-2/6: parents retain original sparse integer labels (7, 42, 188). + Disconnected components and hole rings are shapes/members, not new identities. +- MEMB-1/2: the existing extractor and shape/converter registries remain the only + family authorities. A new parent value or topology needs no roster/dispatch edit. + +New-case witnesses in `tests/test_roi_region_properties.py`: a ring surrounding +islands with another parent label; objects touching all four image borders; +one-pixel necks, diagonal contact and disconnected border pixels. Zero/nonzero +spatial origins exercise bbox translation. Tests compare every contour vertex +and ordering with independent full-canvas scikit-image tracing, inspect parent +area/centroid/bbox, save/reopen the actual ImageJ ZIP, and compare every member's +coordinates and parent metadata. A work-count guard expects one whole-mask bbox +scan; min-area filtering is checked for contour and mask paths. + +The existing full-canvas MaskShape path reconstructs exact labels, including +holes/topology. OpenHCS adds an actual raster-only TIFF materialization witness +with holes/borders and a guard against invoking its unselected ROI writer. + +## Fidelity boundary + +The polygon contract represents independent contours. It does not declare a +hole-subtraction relationship. Archive reopen preserves rings/parent metadata, +but that alone does not prove a viewer or third-party consumer rasterizes holes +correctly. MaskShape is explicitly unsupported by the ImageJ ROI codec. No +ZIP-to-label-raster algorithm or equivalence claim is invented here. Exact mask +and label-TIFF preservation are separate evidence. No biological data/output, +held-out files, JVM, GUI, installed package or shared source is modified. + +## Validation and resource receipt + +Implementation and tests authored; behavioral validation pending Euler's shared +slot. Nonblocking validation-lock probe returned 75 (busy), so no tests/JVM/GUI +started and no polling/sleeping. Resource guard: 13.2 GiB available RAM, +historical swap warning 11.5 GiB. Maintain at least 8 GiB available. + +Light evidence: new test passes Ruff; source/test AST and `git diff --check` +are the scoped guards. No full NRA scan or broad proof. The before-change bounded +128x128 OpenHCS profile observed two bbox scans, 3 parents and 902 members; +single-fixture timings remain in the parent receipt, not a speedup claim. + +When the slot is released, run the new/existing ROI tests and OpenHCS fragmented +materialization tests with `/home/ts/code/projects/openhcs/.venv/bin/python`, +CPU-only mode and verified worktree/submodule source paths. Disable automatic +coverage output for this focused run. Re-run the bounded phase profile; do not +start a large CZI, GUI or JVM journey. + +Scratch owner: Lovelace. Purpose: tiny pytest archives and profile output. +Planned path: `/home/ts/.cache/agent-scratch/openhcs-issue-input-20260929`. +The previous 8.1 MiB fixtures were removed; no new generated output yet. +Receipts, code and PR body files remain on persistent worktree storage. + +Parent recorded gitlink remains at the base until a published paired commit is +ready for coordinator review. No merge/install performed. diff --git a/src/polystore/roi.py b/src/polystore/roi.py index c855e5a..94179c2 100644 --- a/src/polystore/roi.py +++ b/src/polystore/roi.py @@ -253,14 +253,12 @@ def accepts(self, labeled_mask: np.ndarray) -> bool: def extract(self, request: LabeledMaskROIExtractionRequest) -> List[ROI]: from skimage import measure from skimage.measure import regionprops - from scipy.ndimage import find_objects labeled_mask = request.labeled_mask if not np.issubdtype(labeled_mask.dtype, np.integer): labeled_mask = labeled_mask.astype(np.int32) regions = regionprops(labeled_mask) - slices = find_objects(labeled_mask) origin_y, origin_x = request.spatial_origin_yx or (0, 0) rois = [] @@ -293,20 +291,15 @@ def extract(self, request: LabeledMaskROIExtractionRequest) -> List[ROI]: shapes = [] if request.extract_contours: - label_idx = region.label - 1 - if label_idx < len(slices) and slices[label_idx] is not None: - slice_y, slice_x = slices[label_idx] - cropped_mask = labeled_mask[slice_y, slice_x] - binary_mask = (cropped_mask == region.label).astype(np.uint8) - padded_mask = np.pad(binary_mask, pad_width=1, mode="constant", constant_values=0) - contours = measure.find_contours(padded_mask, level=0.5) - offset_y = slice_y.start - offset_x = slice_x.start - padding_offset = np.array([offset_y + origin_y, offset_x + origin_x]) - 1 - for contour in contours: - if len(contour) >= 3: - contour_full = contour + padding_offset - shapes.append(PolygonShape(coordinates=contour_full)) + # regionprops already scanned the labels and cached this crop + # while computing area/perimeter. Preserve its unfilled image, + # including disconnected pieces and holes, for contour tracing. + padded_mask = np.pad(region.image, pad_width=1, mode="constant") + contours = measure.find_contours(padded_mask, level=0.5) + padding_offset = np.array([min_y + origin_y, min_x + origin_x]) - 1 + for contour in contours: + if len(contour) >= 3: + shapes.append(PolygonShape(coordinates=contour + padding_offset)) else: binary_mask = labeled_mask == region.label shapes.append(MaskShape(mask=binary_mask, bbox=metadata["bbox"])) diff --git a/tests/test_roi_region_properties.py b/tests/test_roi_region_properties.py new file mode 100644 index 0000000..ca98642 --- /dev/null +++ b/tests/test_roi_region_properties.py @@ -0,0 +1,139 @@ +"""Reuse region properties without changing parent-label or contour geometry.""" + +from collections import Counter + +import numpy as np +import pytest +from scipy import ndimage +from skimage import measure + +from polystore.disk import DiskStorageBackend +from polystore.roi import MaskShape, extract_rois_from_labeled_mask, load_rois_from_zip + + +def hole_labels(): + labels = np.zeros((32, 32), dtype=np.int32) + labels[3:29, 3:29] = 7 + labels[8:24, 8:24] = 0 + labels[12:16, 12:16] = 42 + labels[19:21, 19:21] = 42 + return labels + + +def border_labels(): + labels = np.zeros((32, 32), dtype=np.int32) + labels[:4, :6] = 7 + labels[:4, -6:] = 7 + labels[-4:, :6] = 7 + labels[-4:, -6:] = 7 + labels[:12, 14:18] = 42 + labels[14:18, 10:22] = 42 + return labels + + +def topology_labels(): + labels = np.zeros((32, 32), dtype=np.int32) + labels[2:8, 2:8] = 7 + labels[2:8, 12:18] = 7 + labels[4, 8:12] = 7 # One-pixel neck between two lobes. + labels[4:6, 4:6] = 0 + labels[np.arange(20, 26), np.arange(20, 26)] = 188 # Diagonal contact. + labels[31, 31] = 188 # Disconnected border pixel, same parent. + return labels + + +@pytest.fixture( + params=(hole_labels, border_labels, topology_labels), ids=("holes", "borders", "topology") +) +def labels(request): + return request.param() + + +@pytest.mark.parametrize("origin", ((0, 0), (10, 20))) +def test_region_crop_contours_equal_full_canvas_and_archive_reopen(labels, origin, tmp_path): + original = labels.copy() + parents = extract_rois_from_labeled_mask( + labels, min_area=1, spatial_origin_yx=origin, source_spatial_shape_yx=(64, 64) + ) + regions = measure.regionprops(labels) + assert [roi.metadata["label"] for roi in parents] == [region.label for region in regions] + expected_member_labels = [] + expected_contours = [] + for roi, region in zip(parents, regions, strict=True): + # Independent full-canvas tracing tests the crop/offset optimization, + # retaining the existing external marching-squares topology contract. + contours = measure.find_contours(np.pad(labels == region.label, 1), level=0.5) + contours = [contour + np.asarray(origin) - 1 for contour in contours] + assert len(roi.shapes) == len(contours) + assert roi.metadata["area"] == np.count_nonzero(labels == region.label) + assert roi.metadata["centroid"] == tuple(np.asarray(region.centroid) + origin) + min_y, min_x, max_y, max_x = region.bbox + assert roi.metadata["bbox"] == ( + min_y + origin[0], + min_x + origin[1], + max_y + origin[0], + max_x + origin[1], + ) + assert roi.metadata["source_spatial_shape_yx"] == (64, 64) + for shape, contour in zip(roi.shapes, contours, strict=True): + np.testing.assert_array_equal(shape.coordinates, contour) + np.testing.assert_array_equal(contour[0], contour[-1]) + expected_contours.extend(contours) + expected_member_labels.extend([region.label] * len(contours)) + archive = tmp_path / "topology.roi.zip" + DiskStorageBackend().save(parents, archive) + restored = load_rois_from_zip(archive) + assert [roi.metadata["label"] for roi in restored] == expected_member_labels + assert Counter(roi.metadata["label"] for roi in restored) == Counter(expected_member_labels) + metadata_by_parent = {roi.metadata["label"]: roi.metadata for roi in parents} + for roi, contour in zip(restored, expected_contours, strict=True): + assert roi.metadata == metadata_by_parent[roi.metadata["label"]] + np.testing.assert_array_equal(roi.shapes[0].coordinates, contour) + # Contours and metadata preserve rings, but the polygon contract does not + # declare hole subtraction. This is not a ZIP-to-label-raster proof. + np.testing.assert_array_equal(labels, original) + + +def test_hole_rings_keep_opposite_orientation(): + (parent, _islands) = extract_rois_from_labeled_mask(hole_labels(), min_area=1) + assert parent.metadata["label"] == 7 + assert len(parent.shapes) == 2 + signed_areas = [] + for shape in parent.shapes: + y, x = shape.coordinates.T + signed_areas.append(np.sum(x[:-1] * y[1:] - x[1:] * y[:-1]) / 2) + assert signed_areas[0] * signed_areas[1] < 0 + assert abs(signed_areas[0]) > abs(signed_areas[1]) + + +def test_existing_mask_representation_preserves_exact_topology(labels): + parents = extract_rois_from_labeled_mask(labels, min_area=1, extract_contours=False) + reconstructed = np.zeros_like(labels) + for roi in parents: + (shape,) = roi.shapes + assert isinstance(shape, MaskShape) + np.testing.assert_array_equal(shape.mask, labels == roi.metadata["label"]) + reconstructed[shape.mask] = roi.metadata["label"] + np.testing.assert_array_equal(reconstructed, labels) + + +@pytest.mark.parametrize("extract_contours", (True, False)) +def test_region_properties_scan_labels_once_and_honor_min_area( + labels, monkeypatch, extract_contours +): + scans = [] + original_find_objects = ndimage.find_objects + + def counted_scan(mask, *args, **kwargs): + scans.append(mask.shape) + return original_find_objects(mask, *args, **kwargs) + + monkeypatch.setattr(ndimage, "find_objects", counted_scan) + rois = extract_rois_from_labeled_mask(labels, min_area=10, extract_contours=extract_contours) + assert scans == [labels.shape] + expected = [ + label + for label in np.unique(labels) + if label != 0 and np.count_nonzero(labels == label) >= 10 + ] + assert [roi.metadata["label"] for roi in rois] == expected From 1d018d9e83229254760b73347cb1ea113e55b7be Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 15:56:42 -0400 Subject: [PATCH 02/12] Retain finite paired ROI fidelity results and runtime release --- ...i_region_properties_checkpoint_20260929.md | 56 +++++++++++++++---- 1 file changed, 45 insertions(+), 11 deletions(-) diff --git a/docs/roi_region_properties_checkpoint_20260929.md b/docs/roi_region_properties_checkpoint_20260929.md index 2ea4e86..dd87560 100644 --- a/docs/roi_region_properties_checkpoint_20260929.md +++ b/docs/roi_region_properties_checkpoint_20260929.md @@ -60,25 +60,59 @@ held-out files, JVM, GUI, installed package or shared source is modified. ## Validation and resource receipt -Implementation and tests authored; behavioral validation pending Euler's shared -slot. Nonblocking validation-lock probe returned 75 (busy), so no tests/JVM/GUI -started and no polling/sleeping. Resource guard: 13.2 GiB available RAM, -historical swap warning 11.5 GiB. Maintain at least 8 GiB available. +Earlier nonblocking validation-lock probe returned 75 (busy); no tests started +then. The coordinator handed off a finite slot on 2026-09-29, after installing +the independent #151/#212 harness checkpoint. Tested source: OpenHCS +`006361ff28528c0159bed0999149c71b8befda94` plus PolyStore +`8c886e79267f72ad88bda4143845ed1721325dd8`, with all nine package import locations +verified under this isolated worktree and explicit submodule PYTHONPATH. + +Retained first attempt: 29 passed / 138 setup errors in 8.49s because the cleaned +scratch parent did not exist. Create only the owned scratch parent and retain the +same assertions. The combined retry produced 141 passed / 26 failed in 37.12s. +All sixteen new ROI fidelity/work-count cases and the existing ROI/fragmented +materialization cases passed, including hole/border/topology contour and archive +reopen fidelity, exact masks/TIFFs, original parent identities and one bbox scan. +Four actual fresh owned MCP generate/inspect/sample journeys passed as well. + +The 26 source-workspace failures looked for `polystore_metadata.json` where the +application wrote `openhcs_metadata.json`. Source witness: +`polystore.metadata_writer.MetadataConfig` reads the generic environment/default +at module initialization; `openhcs/__init__.py:20` sets the application filename. +Combined dependency/application collection can initialize PolyStore first. The +retry runner had not imported OpenHCS before starting pytest, unlike the original +path-verification runner. This is a source-supported import-order explanation, +not an executed preservation proof. Lovelace owns the next separate-context +application regression check; no metadata API/source rewrite is authorized here. + +The profile did not execute because the test subprocess exited nonzero. No +after-change timing/speedup claim. Run it and the application regressions in their +proper separately bootstrapped contexts at the next handed-off slot; do not race +the newly authorized fresh author. + +The finite commands are terminal and released flock. A tagged-environment +process audit found zero owned handles, including pytest/MCP children, after +each terminal command. No restart on observation timeout, no JVM or GUI. Entry +RAM guard: 13.4 GiB available, historical swap warning 11.9 GiB; 8 GiB minimum +was checked before starting the run. No continuously sampled peak-RAM claim. +No source merge, installation or frozen-harness mutation in this validation. Light evidence: new test passes Ruff; source/test AST and `git diff --check` are the scoped guards. No full NRA scan or broad proof. The before-change bounded 128x128 OpenHCS profile observed two bbox scans, 3 parents and 902 members; single-fixture timings remain in the parent receipt, not a speedup claim. -When the slot is released, run the new/existing ROI tests and OpenHCS fragmented -materialization tests with `/home/ts/code/projects/openhcs/.venv/bin/python`, -CPU-only mode and verified worktree/submodule source paths. Disable automatic -coverage output for this focused run. Re-run the bounded phase profile; do not -start a large CZI, GUI or JVM journey. +Executed command used `/home/ts/code/projects/openhcs/.venv/bin/python -m pytest +-q --tb=short --import-mode=importlib -o addopts=` with the new/existing ROI, +fragmented materialization, preparation selection, Bio-Formats, source workspace, +plane-store, zero-geometry and real MCP journey files. CPU-only mode; exact source +paths; automatic coverage disabled. The tests' assertions were not weakened. Scratch owner: Lovelace. Purpose: tiny pytest archives and profile output. -Planned path: `/home/ts/.cache/agent-scratch/openhcs-issue-input-20260929`. -The previous 8.1 MiB fixtures were removed; no new generated output yet. +Path: `/home/ts/.cache/agent-scratch/openhcs-issue-input-20260929`. +The previous 8.1 MiB fixtures were removed. The finite check regenerated 4.2 MiB +of disposable synthetic fixtures; that directory was also removed after retaining +this receipt. No source/saved session/private biological data was removed. Receipts, code and PR body files remain on persistent worktree storage. Parent recorded gitlink remains at the base until a published paired commit is From b981523a0f5b897f04ab4e290753e296bb1192a1 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 15:57:48 -0400 Subject: [PATCH 03/12] Specify declaration-owned application bootstrap for next check --- docs/roi_region_properties_checkpoint_20260929.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/roi_region_properties_checkpoint_20260929.md b/docs/roi_region_properties_checkpoint_20260929.md index dd87560..67af740 100644 --- a/docs/roi_region_properties_checkpoint_20260929.md +++ b/docs/roi_region_properties_checkpoint_20260929.md @@ -84,6 +84,10 @@ retry runner had not imported OpenHCS before starting pytest, unlike the origina path-verification runner. This is a source-supported import-order explanation, not an executed preservation proof. Lovelace owns the next separate-context application regression check; no metadata API/source rewrite is authorized here. +Next driver uses separate dependency/application processes and imports the +existing `openhcs` entrypoint before application pytest collection, so its +declared initialization owns the filename. It will not monkeypatch metadata, +copy companion files, add a filename fallback or change a failed assertion. The profile did not execute because the test subprocess exited nonzero. No after-change timing/speedup claim. Run it and the application regressions in their From 0aa057bc3b4241d77f5f3905e10e31efe2ee87e4 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 16:10:11 -0400 Subject: [PATCH 04/12] Own metadata-free Bio-Formats probes in Java context --- src/polystore/bioformats_java.py | 20 ++++++- tests/test_bioformats_java.py | 96 ++++++++++++++++++++++++++++++++ 2 files changed, 114 insertions(+), 2 deletions(-) diff --git a/src/polystore/bioformats_java.py b/src/polystore/bioformats_java.py index a93530d..c91a0f4 100644 --- a/src/polystore/bioformats_java.py +++ b/src/polystore/bioformats_java.py @@ -2,7 +2,8 @@ from __future__ import annotations -from collections.abc import Callable +from collections.abc import Callable, Iterator +from contextlib import contextmanager from dataclasses import dataclass from pathlib import Path from threading import Lock @@ -151,10 +152,25 @@ def open_reader(self, source_path: str | Path) -> BioFormatsOpenedReader: def declares_path(self, source_path: str | Path) -> bool: """Return whether Bio-Formats positively identifies this source path.""" + with self._probe_reader() as reader: + return bool(reader.isThisType(str(source_path))) + + def is_single_file(self, source_path: str | Path) -> bool: + """Ask the decoder whether this source is independent of companion files. + + The format probe does not call setId or initialize OME metadata. Callers + own selection policy; this context owns Java initialization and lifetime. + """ + with self._probe_reader() as reader: + return bool(reader.isSingleFile(str(source_path))) + + @contextmanager + def _probe_reader(self) -> Iterator[Any]: + """Own a metadata-free reader for the duration of one format probe.""" self.ensure_initialized() reader = self.ImageReader() try: - return bool(reader.isThisType(str(source_path))) + yield reader finally: reader.close() diff --git a/tests/test_bioformats_java.py b/tests/test_bioformats_java.py index 2e33bbd..20f5ac5 100644 --- a/tests/test_bioformats_java.py +++ b/tests/test_bioformats_java.py @@ -1,6 +1,8 @@ from __future__ import annotations +import ast from dataclasses import dataclass +from pathlib import Path import pytest @@ -125,3 +127,97 @@ def test_test_runtime_cleanup_stops_process_context(monkeypatch) -> None: assert runtime.shutdown_gateways == [gateway] assert scyjava.shutdown_count == 1 assert BioFormatsJavaContext._instance is None + + +class _ProbeReader: + def __init__(self, result): + self.result = result + self.paths = [] + self.close_count = 0 + + def _evaluate(self, path): + self.paths.append(path) + if isinstance(self.result, Exception): + raise self.result + return self.result + + isThisType = _evaluate + isSingleFile = _evaluate + + def setId(self, path): + raise AssertionError("A format probe must not open container metadata") + + def close(self): + self.close_count += 1 + + +class _ProbeScyJava(_ScyJava): + def __init__(self, result): + super().__init__() + self.result = result + self.readers = [] + + def _reader(self): + reader = _ProbeReader(self.result) + self.readers.append(reader) + return reader + + def jimport(self, name): + declaration = super().jimport(name) + return self._reader if name == "loci.formats.ImageReader" else declaration + + +@pytest.mark.parametrize("operation", ("declares_path", "is_single_file")) +@pytest.mark.parametrize("result", (True, False)) +def test_format_probe_owns_initialization_and_reader_lifetime(monkeypatch, operation, result): + runtime = _Runtime() + scyjava = _ProbeScyJava(result) + monkeypatch.setattr(bioformats_java, "FIJI_IMAGEJ_RUNTIME", runtime) + context = BioFormatsJavaContext("imagej", scyjava) + probe = getattr(context, operation) + + assert probe(Path("first.czi")) is result + assert probe("second.czi") is result + + assert len(runtime.gateways) == 1 + assert len(scyjava.imports) == 3 + assert [reader.paths for reader in scyjava.readers] == [["first.czi"], ["second.czi"]] + assert [reader.close_count for reader in scyjava.readers] == [1, 1] + context.dispose() + assert runtime.gateways[0].dispose_count == 1 + + +@pytest.mark.parametrize("operation", ("declares_path", "is_single_file")) +def test_format_probe_closes_failed_reader_and_can_retry(monkeypatch, operation): + runtime = _Runtime() + scyjava = _ProbeScyJava(RuntimeError("decoder probe failed")) + monkeypatch.setattr(bioformats_java, "FIJI_IMAGEJ_RUNTIME", runtime) + context = BioFormatsJavaContext("imagej", scyjava) + probe = getattr(context, operation) + + with pytest.raises(RuntimeError, match="decoder probe failed"): + probe("bad.czi") + assert scyjava.readers[0].close_count == 1 + scyjava.result = True + assert probe("good.czi") + assert scyjava.readers[1].close_count == 1 + assert len(runtime.gateways) == 1 + context.dispose() + + +def test_format_probe_operations_share_the_context_reader_lifetime(): + tree = ast.parse(Path(bioformats_java.__file__).read_text()) + context = next( + node + for node in tree.body + if isinstance(node, ast.ClassDef) and node.name == "BioFormatsJavaContext" + ) + for operation in ("declares_path", "is_single_file"): + method = next( + node + for node in context.body + if isinstance(node, ast.FunctionDef) and node.name == operation + ) + attributes = {node.attr for node in ast.walk(method) if isinstance(node, ast.Attribute)} + assert "_probe_reader" in attributes + assert not attributes & {"ImageReader", "ensure_initialized", "close", "setId"} From a14a4fc2e83a9acdd7c5f78a60fe248b9c6d61dd Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 16:14:10 -0400 Subject: [PATCH 05/12] Retain Java owner review and lightweight contract evidence --- ...i_region_properties_checkpoint_20260929.md | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/roi_region_properties_checkpoint_20260929.md b/docs/roi_region_properties_checkpoint_20260929.md index 67af740..0b0f4cb 100644 --- a/docs/roi_region_properties_checkpoint_20260929.md +++ b/docs/roi_region_properties_checkpoint_20260929.md @@ -7,6 +7,55 @@ Isolated checkout: `/home/ts/wt/openhcs-input-preparation-20260929/external/Poly Fetched/merged base: `origin/main` at `f94bbbe8631a4c78f7ceaec72fb99671b58c9a18`. Only open PR at orientation: Dependabot #10; no overlapping feature PR. +## Independent PR206 review: dependency-owned Java probe + +Latest owner explicitly extended this existing paired PR to correct the Java +lifecycle finding in +https://github.com/OpenHCSDev/openhcs/pull/206#issuecomment-5897315214. +No overlapping feature PR found: only this draft16 and Dependabot10 are open. +Source commit `0aa057bc3b4241d77f5f3905e10e31efe2ee87e4`, paired OpenHCS +`32a2b27e22220b28f4a75db1977973aa61384a10`. + +IMPL-13 / BOUND-2: existing `BioFormatsJavaContext.is_single_file` at +`src/polystore/bioformats_java.py:158` now owns the decoder's capability. Existing +`declares_path:153` and this operation share `_probe_reader:168`, which initializes, +constructs and unconditionally closes a metadata-free Java reader. Neither probe +calls setId/OME metadata initialization. Delete the OpenHCS helper that imitated +this lifecycle; its adapter now invokes this public context operation. OpenHCS +still owns path-selection/compound-entrypoint policy. No alternate context, +extractor, cache, registry or filename/companion roster. ROI source remains the +already-reviewed in-place RegionProperties implementation unchanged. + +Light check: **28 passed in 0.50s**, peak Python RSS **72.8 MiB**. Eleven child +cases (four existing context checks, six new true/false/failure/retry cases, one +AST ownership guard) plus seventeen parent binding-path behavior/guard cases. +Probe tests use controlled external responses but actual context initialization +and lifetime; zero JVM/ImageJ/MCP/GUI processes. A new probe reuses one lifetime +instead of reconstructing construction/cleanup in consumers. AST guard prevents +the two public operations from bypassing `_probe_reader`; parent guard prevents +its adapter from constructing Java readers or reauthoring binding policy. +New tests and touched child source pass Ruff; both diffs pass whitespace checks. +Current archive SKILL plus complete implementation/boundaries/membership chapters +read; scope source/AST/contract evidence, not full NRA/native proof. + +Recorded interpreter/explicit worktree and all required submodule src paths. +Actual imports verified before testing. Bootstrap OpenHCS first, disable plugin +autoload, `pytest.main` with `-q --tb=short --noconftest --import-mode=importlib +-p no:cacheprovider -o addopts=` on the child Java context and parent path-admission +files; shell bound45s. Two unused asyncio config warnings with plugins disabled. +Initial26-case check passed before adding ownership guards (0.23s/67.1 MiB). +Guard reports11.4 GiB available/historical swap11.9 GiB, explicit8 GiB admission; +final11.2 GiB available. No continuous peak-RAM claim. Tiny owned92 KiB scratch +at the recorded path removed after results retained. No validation-lock takeover, +frozen installation/source/skill change, merge/install or blind-data inspection. + +Original **141 passed / 26 failed** remains unresolved by these narrow passes; +the profile **did not run**. Separate-context preservation/affected source-control +rerun, bounded ROI after-profile and real Java/installed acceptance remain gated +on the next finite runtime handoff. Parent recorded gitlink stays `f94bbbe`; +integration now requires this paired public capability too. Coordinator owns +validated adoption/installation, not this worker. No closure/readiness claim. + ## Owned change and deletion `TwoDimensionalLabeledMaskROIExtractor.extract` stays the only 2D behavior From 5e656f8c20b0949139fe1273c31390931ea7ef6d Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 17:15:17 -0400 Subject: [PATCH 06/12] Project bundle-only Fiji cache root through canonical distribution --- src/polystore/imagej_distribution.py | 28 ++++++++- tests/imagej_cache_process_fixture.py | 72 ++++++++++++++++++++++ tests/test_imagej_cache_environment.py | 85 ++++++++++++++++++++++++++ 3 files changed, 183 insertions(+), 2 deletions(-) create mode 100644 tests/imagej_cache_process_fixture.py create mode 100644 tests/test_imagej_cache_environment.py diff --git a/src/polystore/imagej_distribution.py b/src/polystore/imagej_distribution.py index d9bfbe5..2fade34 100644 --- a/src/polystore/imagej_distribution.py +++ b/src/polystore/imagej_distribution.py @@ -14,12 +14,12 @@ import urllib.request import zipfile from abc import ABC, abstractmethod -from collections.abc import Iterator +from collections.abc import Iterator, Mapping from contextlib import contextmanager from dataclasses import dataclass from enum import Enum from pathlib import Path, PurePosixPath -from typing import Any +from typing import Any, ClassVar from platformdirs import user_cache_path @@ -354,6 +354,8 @@ def apply( class FijiArchiveDistribution(ImageJDistributionABC): """Checksummed Fiji application bundle from one immutable official archive.""" + cache_root_environment_key: ClassVar[str] = "POLYSTORE_IMAGEJ_CACHE_ROOT" + release_id: str imagej_version: str archive_base_url: str @@ -362,6 +364,27 @@ class FijiArchiveDistribution(ImageJDistributionABC): lock_timeout_seconds: float = 1800.0 download_policy: ImageJArchiveDownloadPolicy = ImageJArchiveDownloadPolicy() + @classmethod + def cache_root_from_environment( + cls, environment: Mapping[str, str] | None = None + ) -> Path | None: + """Decode the bundle-only root at process configuration, not each use. + + This does not redirect XDG caches, logs, preferences or other runtime + artifacts. Unset retains platformdirs policy; invalid explicit settings + fail instead of silently creating another per-run bundle. + """ + values = os.environ if environment is None else environment + declared = values.get(cls.cache_root_environment_key) + if declared is None: + return None + root = Path(declared).expanduser() + if not declared.strip() or not root.is_absolute(): + raise ImageJDistributionUnavailableError( + f"{cls.cache_root_environment_key} must be an absolute bundle-cache path." + ) + return root.resolve(strict=False) + @property def label(self) -> str: """Return the declared Fiji version and immutable archive revision.""" @@ -494,6 +517,7 @@ def require_compatible_gateway(self, gateway: Any) -> None: release_id="20260718-0417", imagej_version="2.18.0", archive_base_url=("https://downloads.imagej.net/fiji/archive/latest/20260718-0417"), + cache_root=FijiArchiveDistribution.cache_root_from_environment(), runtime_overlays=( ImageJRuntimeOverlay( label="PyImageJ 1.7.0 Fiji bridge overlay", diff --git a/tests/imagej_cache_process_fixture.py b/tests/imagej_cache_process_fixture.py new file mode 100644 index 0000000..d7dce1d --- /dev/null +++ b/tests/imagej_cache_process_fixture.py @@ -0,0 +1,72 @@ +"""Tiny verified artifact response for fresh-process bundle-cache tests; no JVM.""" + +from __future__ import annotations + +import argparse +import json +import os +from dataclasses import dataclass, field, replace +from pathlib import Path +from unittest.mock import patch + +import polystore.imagej_distribution as distribution_module +from polystore.imagej_distribution import ( + FIJI_IMAGEJ_DISTRIBUTION, + FijiBundleAsset, + ImageJArchiveDownloadPolicy, + ImageJRuntimeArchive, +) +from polystore.imagej_runtime import FIJI_IMAGEJ_RUNTIME + + +@dataclass(frozen=True) +class FixtureAsset: + source: Path + sha256: str + name: str = "LOCAL_FIXTURE" + + def archive(self, *, archive_base_url, label): + return ImageJRuntimeArchive(label, self.source.as_uri(), self.sha256) + + +@dataclass(frozen=True) +class CountedDownloadPolicy(ImageJArchiveDownloadPolicy): + downloaded: list[str] = field(default_factory=list) + + def download(self, archive, *, target_directory): + self.downloaded.append(archive.sha256) + return super().download(archive, target_directory=target_directory) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("archive", type=Path) + parser.add_argument("sha256") + args = parser.parse_args() + assert FIJI_IMAGEJ_RUNTIME.distribution is FIJI_IMAGEJ_DISTRIBUTION + policy = CountedDownloadPolicy(retry_delays_seconds=()) + # Only the external artifact/host response is controlled. Canonical root, + # checksum verification, cache locking/staging/discovery remain production. + distribution = replace( + FIJI_IMAGEJ_RUNTIME.distribution, runtime_overlays=(), download_policy=policy + ) + with patch.object( + FijiBundleAsset, "for_current_host", return_value=FixtureAsset(args.archive, args.sha256) + ): + launch = distribution.materialize() + print( + json.dumps( + { + "cache_root": str(FIJI_IMAGEJ_DISTRIBUTION.cache_root), + "imagej_directory": str(launch.imagej_directory), + "java_home": str(launch.java_home), + "downloads": len(policy.downloaded), + "xdg_cache_home": os.environ["XDG_CACHE_HOME"], + "source": str(Path(distribution_module.__file__).resolve()), + } + ) + ) + + +if __name__ == "__main__": + main() diff --git a/tests/test_imagej_cache_environment.py b/tests/test_imagej_cache_environment.py new file mode 100644 index 0000000..9658b18 --- /dev/null +++ b/tests/test_imagej_cache_environment.py @@ -0,0 +1,85 @@ +"""Bundle-only configuration survives independent process cache policies.""" + +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +import zipfile +from pathlib import Path + +import pytest + +from polystore.imagej_distribution import ( + FijiArchiveDistribution, + ImageJDistributionUnavailableError, +) + + +def test_unset_bundle_root_retains_distribution_default(): + assert FijiArchiveDistribution.cache_root_from_environment({}) is None + + +@pytest.mark.parametrize("value", ("", " ", "relative/cache")) +def test_invalid_explicit_bundle_root_fails_without_a_fallback(value): + with pytest.raises(ImageJDistributionUnavailableError, match="absolute bundle-cache"): + FijiArchiveDistribution.cache_root_from_environment( + {FijiArchiveDistribution.cache_root_environment_key: value} + ) + + +def test_bundle_root_decode_is_independent_of_xdg_cache(tmp_path): + root = tmp_path / "bundles" + for run in ("first", "second"): + assert ( + FijiArchiveDistribution.cache_root_from_environment( + { + FijiArchiveDistribution.cache_root_environment_key: str(root), + "XDG_CACHE_HOME": str(tmp_path / run), + } + ) + == root + ) + + +def test_fresh_processes_share_one_verified_bundle_not_their_xdg_caches(tmp_path): + archive = tmp_path / "tiny-fiji.zip" + with zipfile.ZipFile(archive, "w") as fixture: + fixture.writestr("Fiji/jars/core.jar", b"verified fixture") + fixture.writestr("Fiji/plugins/plugin.jar", b"fixture") + fixture.writestr("Fiji/java/linux64/jdk/bin/java", b"no JVM execution") + digest = hashlib.sha256(archive.read_bytes()).hexdigest() + root = tmp_path / "shared-bundles" + helper = Path(__file__).with_name("imagej_cache_process_fixture.py") + results = [] + for run in ("first", "second"): + isolated = tmp_path / run + isolated.mkdir() + (isolated / "run.log").write_text(run) + environment = dict(os.environ) + environment[FijiArchiveDistribution.cache_root_environment_key] = str(root) + environment["XDG_CACHE_HOME"] = str(isolated) + child = subprocess.run( + [sys.executable, str(helper), str(archive), digest], + env=environment, + capture_output=True, + text=True, + check=True, + timeout=10, + ) + result = json.loads(child.stdout) + assert result["cache_root"] == str(root) + assert result["xdg_cache_home"] == str(isolated) + assert result["source"] == str( + Path(__file__).resolve().parents[1] / "src/polystore/imagej_distribution.py" + ) + assert (isolated / "run.log").read_text() == run + assert not (isolated / "polystore/imagej").exists() + results.append(result) + assert [result["downloads"] for result in results] == [1, 0] + assert results[0]["imagej_directory"] == results[1]["imagej_directory"] + assert results[0]["java_home"] == results[1]["java_home"] + assert len(tuple(root.glob("fiji-*"))) == 1 + assert not tuple(root.glob(".imagej-download.*")) From 9e205fee234de361d9169a0fd1b36dc982ef0bee Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Tue, 29 Sep 2026 17:20:50 -0400 Subject: [PATCH 07/12] Retain bundle cache projection and cold progress checkpoint --- ...i_region_properties_checkpoint_20260929.md | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/docs/roi_region_properties_checkpoint_20260929.md b/docs/roi_region_properties_checkpoint_20260929.md index 0b0f4cb..6dc0998 100644 --- a/docs/roi_region_properties_checkpoint_20260929.md +++ b/docs/roi_region_properties_checkpoint_20260929.md @@ -7,6 +7,62 @@ Isolated checkout: `/home/ts/wt/openhcs-input-preparation-20260929/external/Poly Fetched/merged base: `origin/main` at `f94bbbe8631a4c78f7ceaec72fb99671b58c9a18`. Only open PR at orientation: Dependabot #10; no overlapping feature PR. +## Bundle-only cache entrypoint follow-through: OpenHCS224 + +Issue https://github.com/OpenHCSDev/openhcs/issues/224, Lovelace implementation, +existing paired drafts16/206. Parent owns disk recovery/integration. Reported +fresh H001/H002 inspection duplicates roughly859 MiB bundles under per-run XDG +cache and blocks cold MCP before disk-guard shutdown. Preserve uncertain queued +reads; no worker replay/restart, biological receipts, coaching or frozen changes. + +Source `5e656f8c20b0949139fe1273c31390931ea7ef6d`, paired application +`101a6a4b0ec80a8ce970478399fb9527ef2f5b51`. Constructor cache_root had no canonical +application projection. Existing FijiArchiveDistribution now declares +**POLYSTORE_IMAGEJ_CACHE_ROOT** at imagej_distribution.py:357, decodes it through +cache_root_from_environment:368 and projects it into existing cache_root on the +canonical constructor at:520. Set an absolute reviewed root before import/start; +unset keeps platformdirs, invalid explicit values fail. No XDG/full-cache +redirect, caller symlink/monkeypatch, root relocation, second distribution/runtime +or environment/default registry. Immutable release/digest/overlays, verification, +locking/staging and runtime policy unchanged. OpenHCS child projection derives +this key and its inspection declaration consumes existing worker/progress with +truthful optional effects. ROI and Java probe implementations remain unchanged. + +**28 passes in5.07s / peak driver RSS267.4 MiB**:17 existing distribution cases, +six new cache-root cases and five application projection/progress cases. Child +tests test_imagej_cache_environment.py plus imagej_cache_process_fixture.py use +two fresh Python imports, real canonical root/runtime, tiny ZIP SHA verification +and production cache lock/staging/discovery. Only host/artifact response is +controlled; no overlays/JVM. Separate run logs/XDG roots survive; one bundle path, +verified download counts[1,0], no temporary residue. Actual in-process FastMCP +binding keeps discovery responsive during controlled held worker I/O; existing +helper emits progress and preserves identical terminal errors. Not a real +859 MiB/cold Java/continuous installed protocol journey or timing claim. + +Retain first10 passes/1 failure4.91s/280.1 MiB: registry correctly rejected +missing mutating metadata; correct declaration, not validator/tests. Two unused +asyncio-option warnings with disabled plugins. New tests/helper/touched source +pass Ruff; six files parse, both diffs pass whitespace. Parent environment file +has preexisting S110/BLE001 unrelated to this change. Recorded interpreter, +explicit assigned src paths, imports verified, OpenHCS before pytest, +plugins/conftests disabled,45s shell bound. Guard10.6 GiB available/swap11.9 GiB +historical warning, final10.2 GiB, explicit8 GiB admission. Not continuous fleet +peak-RAM evidence. Owned316 KiB disposable fixtures removed by normal non-force +removal after rejected force-style command; exact validated path only. + +BOUND-2/6: existing field/distribution is sole authority. MEMB-2/3: process +projection derives one owner key, no copied default/registry. IMPL-13: retain +existing runtime/lifetime/materialization mechanisms, not another store. Changed +roots need configuration, no caller edits/copies. Current NRA/archive catalog +read; focused source/AST/contract scope, no full scan/native proof. + +Original141-pass/26-failure receipt and **not-run** after-change ROI profile remain +open. Real cold stdio/progress-token/queued reads/Java/installed acceptance await +finite handoff. Recorded gitlink staysf94bbbe until validated paired adoption; +frozen installation lacks this new projection. No MCP process/listen, GUI/JVM, +remote bundle download, environment creation, shared lock takeover, merge/install +or skill/source changes outside isolated owner checkout. No closure/readiness. + ## Independent PR206 review: dependency-owned Java probe Latest owner explicitly extended this existing paired PR to correct the Java From 4939979978439a6f38145788df9aac3bce65bc70 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 21:14:00 -0400 Subject: [PATCH 08/12] Record paired current-main preservation and namespace defect evidence --- .../input_preparation_parent_20261001.rst | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 docs/source/input_preparation_parent_20261001.rst diff --git a/docs/source/input_preparation_parent_20261001.rst b/docs/source/input_preparation_parent_20261001.rst new file mode 100644 index 0000000..25a16e1 --- /dev/null +++ b/docs/source/input_preparation_parent_20261001.rst @@ -0,0 +1,32 @@ +Parent paired source checkpoint +=============================== + +Existing PR16 is normally integrated with the OpenHCS-recorded PolyStore +dc3417392cef19083a907c8c9b88a15466f81df4 at +cb12a2acd91061d60cb6ec65981a9f5c01e18963. Parent is the integration owner; +the original worker worktree remains unchanged. Existing ROI/context owners and +external formats are retained without a parallel implementation. + +Forty-eight standalone dependency tests pass in0.82s, including existing/new ROI +geometry/work-count and Java-context lifetime controls. The source-qualified +process checks its generic metadata namespace before collection. Separate real +application-entrypoint bootstrap passes163 source workspace/selection/materialization +and generator controls in19.12s. These use independent processes and real typed +owners, no filename patch/fallback. The original141-pass/26-failure combined run +is not overwritten or relabeled. + +Intentional dependency-first application embedding still reproduces the metadata +namespace defect:1 failure in3.34s. Parent owns the application/dependency boundary +follow-up; no cached singleton mutation or second file reader is introduced here. +The initial control's missing scratch-parent setup error is retained separately. + +Existing bounded128x128 profile now observes one bbox scan rather than the older +two, with parents7/42/188 and902 fragments/native ZIP members unchanged. This is +a bounded work-count witness, not a general latency or biological claim. Polygon +ring preservation does not prove hole subtraction/raster-viewer equivalence. + +Full source commands/raw XML/logs/profile are preserved in +/home/ts/wt/openhcs-issue-batch-20260929/input-parent-20261001 and the paired +OpenHCS docs/validation/input_preparation_parent_20261001.rst. Actual installed +MCP/Java acceptance and the complete ZIP/NRA review remain open. No package, +Fiji download, JVM, scientific worker or frozen installation is modified. From 48621c1e049c570eed83be13db943931d6b1501a Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 21:57:10 -0400 Subject: [PATCH 09/12] Carry metadata namespace owners through workspace reopen and worker handoff --- docs/source/metadata_namespace_327.rst | 89 +++++++++++++++ src/polystore/metadata_writer.py | 11 +- src/polystore/virtual_workspace.py | 27 ++++- tests/metadata_namespace_worker.py | 35 ++++++ tests/test_metadata_namespace.py | 125 +++++++++++++++++++++ tests/test_metadata_namespace_ownership.py | 25 +++++ 6 files changed, 305 insertions(+), 7 deletions(-) create mode 100644 docs/source/metadata_namespace_327.rst create mode 100644 tests/metadata_namespace_worker.py create mode 100644 tests/test_metadata_namespace.py create mode 100644 tests/test_metadata_namespace_ownership.py diff --git a/docs/source/metadata_namespace_327.rst b/docs/source/metadata_namespace_327.rst new file mode 100644 index 0000000..d42997f --- /dev/null +++ b/docs/source/metadata_namespace_327.rst @@ -0,0 +1,89 @@ +Issue327: workspace-owned metadata namespace +=========================================== + +SOURCE-ONLY implementation owner; parent remains integration owner. +Audited dependency: PR16 at4939979978439a6f38145788df9aac3bce65bc70. +Paired application: PR206 ata3179b214b888308133e55682e964f77c2781879. +This is a dependency-stack extension, not another input-preparation implementation. + +Boundary, witnesses and ownership +-------------------------------- + +Before: metadata_writer.py:54 declares frozen MetadataConfig, but +virtual_workspace.py:242 resolves metadata through the module-global helper. +Its connection parameters at218-238 carry only plate_root. OpenHCS's richer +OpenHCSMetadataConfig declares its own filename, paths and lock ownership in +openhcs/core/virtual_workspace_metadata.py:39-59, while openhcs/__init__.py:20 +projects the application filename into the dependency's environment. A dependency +import before that projection leaves its decoded config unchanged. + +BOUND-2: the consumer bypassed the richer application's existing config owner. +IMPL-13: application path/lock behavior was repeated rather than inherited. +TIME-7: bootstrap supplied another component's ambient default. No application +namespace is added to PolyStore; the application declares that specialization. + +After: MetadataConfig owns metadata_path and managed_paths. VirtualWorkspaceBackend +retains the exact MetadataConfig instance, resolves through its method, and carries +the owner through the existing PicklableBackend connection-parameter contract. +FileManager's existing registry and native reconstruction remain unchanged. +Its frozen generic default remains the same object used by get_metadata_path. +Explicit custom configurations are not overwritten by another importing consumer. +OpenHCSMetadataConfig inherits shared behavior and declares only its application +filename. Parent must bind this owner at microscope backend registration. + +Durable state and new-case experiment +------------------------------------ + +Metadata remains durable JSON at its declared path, with the existing structured +SourcePixelRef mapping. No new store, copy, rename, alternate reader or converter. +The runtime FileManager handoff carries the config object along with plate_root; +old runtime connection dictionaries are not accepted through a fallback. + +Before, embedding a new namespace required coordinating an environment projection, +dependency import timing and application producer declaration. After, one frozen +MetadataConfig declaration/value is passed to the workspace constructor. The +new-case test changes filename and SUBDIRECTORIES_KEY to collections and loads +exact pixels without changing workspace/registry/consumer code. The native test +uses actual FileManager pickle reconstruction in a fresh Python process, not a +mock registry or serialization replacement. + +Source evidence and guards +-------------------------- + +Seven focused checks pass in4.45s, peak combined process RSS298.66 MiB, one CPU. +They cover generic/custom namespace, explicit environment config with no mutation +of the generic snapshot, real persisted reopen, connection replacement and cache +refresh, absence of alternate-filename readers, source axes, exact pixels, +sample provenance and native handoff. Site-specific AST guard requires the path +call to use self.metadata_config and the handoff methods to carry that owner. +No guard exceptions or assertion weakening. + +Lightweight full-package AST census at pinned HEAD completes in1.06s with12,209 +code lines; it is not a global semantic audit. The original application failure +was separately reproduced unchanged in4.35s at275.77 MiB RSS. Original XML/logs +and the earlier native-extension collection error are retained in the paired +application receipt's source archive. Own native extensions were built normally +from source, not copied from the installation. + +This is a hand-authored semantic patch based on the fully read NRA/refactor-audit +rules, pattern catalog and inspected owners. No NRA transaction, completed R1 +raw-findings scan, native-equivalence certificate or global architecture approval +is claimed. Uncompleted heavy scans do not block these bounded source checks; +their missing proof remains missing, not waived. + +Resource and acceptance boundary +------------------------------- + +Own persistent source worktree: /home/ts/wt/polystore-metadata-namespace-327-20261001. +Disposable output owner: this agent; path +/home/ts/.cache/agent-scratch/metadata-namespace-327, maximum256 MiB. +Every source shard is bounded60s and512 MiB combined RSS. Resource guard reports +the owner's warning-only20 GiB disk threshold; actual8.4-8.5 GiB free and more +than15 GiB RAM available provide room for these explicit bounds. +No installed packages, parent/foreign sources, UI/MCP, scientific datasets, +science-runtime locks, Fiji/environments/interpreter downloads are changed. + +Working source contract is ready for parent review. Complete microscope startup, +application-first/dependency-first entrypoints, installed native/MCP and affected +installed acceptance require parent binding and paired integration. Neither this +receipt nor passing source checks closes issue327 or certifies live readiness. diff --git a/src/polystore/metadata_writer.py b/src/polystore/metadata_writer.py index 8f8db7d..a238ba8 100644 --- a/src/polystore/metadata_writer.py +++ b/src/polystore/metadata_writer.py @@ -60,6 +60,15 @@ class MetadataConfig: AVAILABLE_BACKENDS_KEY: str = "available_backends" DEFAULT_TIMEOUT: float = LOCK_CONFIG.DEFAULT_TIMEOUT + def metadata_path(self, plate_root: Union[str, Path]) -> Path: + """Resolve this namespace's metadata, independent of other consumers.""" + return Path(plate_root) / self.METADATA_FILENAME + + def managed_paths(self, plate_root: Union[str, Path]) -> tuple[Path, Path]: + """Metadata and lock owned by this namespace's transaction.""" + path = self.metadata_path(plate_root) + return path, LOCK_CONFIG.lock_path(path) + METADATA_CONFIG = MetadataConfig() @@ -151,4 +160,4 @@ def get_metadata_path(plate_root: Union[str, Path]) -> Path: Returns: Path to the metadata file """ - return Path(plate_root) / METADATA_CONFIG.METADATA_FILENAME + return METADATA_CONFIG.metadata_path(plate_root) diff --git a/src/polystore/virtual_workspace.py b/src/polystore/virtual_workspace.py index ff0dc4e..4a68ef2 100644 --- a/src/polystore/virtual_workspace.py +++ b/src/polystore/virtual_workspace.py @@ -22,7 +22,7 @@ ) from .constants import Backend from .exceptions import StorageResolutionError -from .metadata_writer import get_metadata_path +from .metadata_writer import METADATA_CONFIG, MetadataConfig if TYPE_CHECKING: import numpy as np @@ -204,8 +204,14 @@ class VirtualWorkspaceBackend(ReadOnlyBackend, PicklableBackend): _backend_type = Backend.VIRTUAL_WORKSPACE.value - def __init__(self, plate_root: Path): + def __init__( + self, + plate_root: Path, + *, + metadata_config: MetadataConfig = METADATA_CONFIG, + ): self.plate_root = Path(plate_root) + self.metadata_config = metadata_config self._registry: Mapping[str, BackendBase] | None = None self._mapping_cache: Optional[Dict[str, SourcePixelRef]] = None self._cache_mtime: Optional[float] = None @@ -222,15 +228,22 @@ def from_connection_params( ) -> "VirtualWorkspaceBackend": if not params: raise ValueError("VirtualWorkspaceBackend requires plate_root.") - return cls(plate_root=Path(params["plate_root"])) + return cls( + plate_root=Path(params["plate_root"]), + metadata_config=params["metadata_config"], + ) def get_connection_params(self) -> Optional[Dict[str, Any]]: - return {"plate_root": str(self.plate_root)} + return { + "plate_root": str(self.plate_root), + "metadata_config": self.metadata_config, + } def set_connection_params(self, params: Optional[Dict[str, Any]]) -> None: if not params: raise ValueError("VirtualWorkspaceBackend requires plate_root.") self.plate_root = Path(params["plate_root"]) + self.metadata_config = params["metadata_config"] self._mapping_cache = None self._cache_mtime = None self._load_mapping() @@ -241,7 +254,7 @@ def _normalize_relative_path(path_str: str) -> str: return "" if normalized == "." else normalized def _load_mapping(self) -> Dict[str, SourcePixelRef]: - metadata_path = get_metadata_path(self.plate_root) + metadata_path = self.metadata_config.metadata_path(self.plate_root) if not metadata_path.exists(): raise FileNotFoundError(f"Metadata not found: {metadata_path}") current_mtime = metadata_path.stat().st_mtime @@ -250,7 +263,9 @@ def _load_mapping(self) -> Dict[str, SourcePixelRef]: with metadata_path.open("r", encoding="utf-8") as stream: metadata = json.load(stream) combined_mapping: Dict[str, SourcePixelRef] = {} - for subdirectory in metadata.get("subdirectories", {}).values(): + for subdirectory in metadata.get( + self.metadata_config.SUBDIRECTORIES_KEY, {} + ).values(): for virtual_path, source_ref in subdirectory.get( "workspace_mapping", {}, diff --git a/tests/metadata_namespace_worker.py b/tests/metadata_namespace_worker.py new file mode 100644 index 0000000..efcf1c1 --- /dev/null +++ b/tests/metadata_namespace_worker.py @@ -0,0 +1,35 @@ +"""Fresh native Python recipient of a real FileManager handoff.""" + +import pickle +import sys +from pathlib import Path + +import numpy as np + +import polystore +from polystore.base import ImageSamplingRequest + + +def main() -> None: + assert Path(polystore.__file__).resolve().parents[2] == Path(__file__).resolve().parents[1] + with Path(sys.argv[1]).open("rb") as stream: + manager, root, config, expected = pickle.load(stream) + workspace = manager.registry["virtual_workspace"] + assert workspace.metadata_config == config + assert workspace._registry is manager.registry + assert workspace._resolve_ref(root / "virtual.npy").source_axis_indices == (1,) + np.testing.assert_array_equal( + manager.load(root / "virtual.npy", backend="virtual_workspace"), expected + ) + sample = manager.sample( + root / "virtual.npy", + backend="virtual_workspace", + request=ImageSamplingRequest(origin_yx=(1, 1), shape_yx=(2, 3)), + ) + np.testing.assert_array_equal(sample.data, expected[1:3, 1:4]) + assert sample.source_shape == expected.shape + print("native handoff: exact namespace, registry, pixels, axes and sample provenance") + + +if __name__ == "__main__": + main() diff --git a/tests/test_metadata_namespace.py b/tests/test_metadata_namespace.py new file mode 100644 index 0000000..df4e98b --- /dev/null +++ b/tests/test_metadata_namespace.py @@ -0,0 +1,125 @@ +"""Namespace-scoped durable workspace reopen through the existing owners.""" + +import json +import os +import pickle +import subprocess +import sys +from pathlib import Path + +import numpy as np +import pytest + +from polystore.disk import DiskBackend +from polystore.filemanager import FileManager +from polystore.metadata_writer import METADATA_CONFIG, MetadataConfig, get_metadata_path +from polystore.virtual_workspace import SourcePixelRef, VirtualWorkspaceBackend + + +def persist_workspace(root: Path, config: MetadataConfig, index: int = 1) -> np.ndarray: + source = np.arange(2 * 4 * 5, dtype=np.uint16).reshape(2, 4, 5) + np.save(root / "source.npy", source) + ref = SourcePixelRef("disk", "source.npy", (index,)) + config.metadata_path(root).write_text( + json.dumps( + { + config.SUBDIRECTORIES_KEY: { + ".": { + "workspace_mapping": {"virtual.npy": ref.to_workspace_mapping()}, + } + }, + } + ) + ) + return source[index] + + +@pytest.mark.parametrize("filename", ["polystore_metadata.json", "custom-metadata.json"]) +def test_namespace_reopen_connection_restore_and_native_handoff(tmp_path, filename): + config = MetadataConfig(METADATA_FILENAME=filename) + expected = persist_workspace(tmp_path, config) + backend = VirtualWorkspaceBackend(tmp_path, metadata_config=config) + assert backend.metadata_config is config + params = backend.get_connection_params() + assert params["metadata_config"] is config + restored = VirtualWorkspaceBackend.from_connection_params(params) + assert restored.metadata_config is config + manager = FileManager({"disk": DiskBackend(), "virtual_workspace": restored}) + np.testing.assert_array_equal( + manager.load(tmp_path / "virtual.npy", backend="virtual_workspace"), + expected, + ) + handoff = tmp_path / "native-handoff.pickle" + with handoff.open("wb") as stream: + pickle.dump((manager, tmp_path, config, expected), stream) + result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("metadata_namespace_worker.py")), + str(handoff), + ], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + assert result.returncode == 0, result.stdout + result.stderr + assert "exact namespace" in result.stdout + assert sorted(p.name for p in tmp_path.glob("*metadata.json")) == [filename] + + +def test_new_namespace_declaration_needs_no_workspace_consumer_edits(tmp_path): + config = MetadataConfig(METADATA_FILENAME="embedded.json", SUBDIRECTORIES_KEY="collections") + expected = persist_workspace(tmp_path, config) + backend = VirtualWorkspaceBackend(tmp_path, metadata_config=config) + manager = FileManager({"disk": DiskBackend(), "virtual_workspace": backend}) + np.testing.assert_array_equal( + manager.load(tmp_path / "virtual.npy", backend="virtual_workspace"), expected + ) + assert config.managed_paths(tmp_path)[0] == tmp_path / "embedded.json" + + +def test_explicit_config_decodes_once_without_mutating_generic_snapshot(tmp_path, monkeypatch): + snapshot = METADATA_CONFIG + old_filename = snapshot.METADATA_FILENAME + monkeypatch.setenv("POLYSTORE_METADATA_FILENAME", "embedding.json") + config = MetadataConfig() + persist_workspace(tmp_path, config) + backend = VirtualWorkspaceBackend(tmp_path, metadata_config=config) + monkeypatch.setenv("POLYSTORE_METADATA_FILENAME", "later.json") + assert backend.metadata_config == config + assert backend._load_mapping() + assert METADATA_CONFIG is snapshot + assert snapshot.METADATA_FILENAME == old_filename + assert get_metadata_path(tmp_path) == snapshot.metadata_path(tmp_path) + persist_workspace(tmp_path, snapshot) + assert VirtualWorkspaceBackend(tmp_path).metadata_config is snapshot + + +def test_namespace_isolation_has_no_alternate_filename_reader(tmp_path): + persist_workspace(tmp_path, MetadataConfig(METADATA_FILENAME="foreign.json")) + with pytest.raises(FileNotFoundError, match="missing.json"): + VirtualWorkspaceBackend( + tmp_path, metadata_config=MetadataConfig(METADATA_FILENAME="missing.json") + ) + + +def test_connection_retarget_invalidates_mapping_and_namespace(tmp_path): + first = MetadataConfig(METADATA_FILENAME="first.json") + second = MetadataConfig(METADATA_FILENAME="second.json") + persist_workspace(tmp_path, first, index=0) + expected = persist_workspace(tmp_path, second) + backend = VirtualWorkspaceBackend(tmp_path, metadata_config=first) + backend.set_connection_params( + VirtualWorkspaceBackend(tmp_path, metadata_config=second).get_connection_params() + ) + manager = FileManager({"disk": DiskBackend(), "virtual_workspace": backend}) + np.testing.assert_array_equal( + manager.load(tmp_path / "virtual.npy", backend="virtual_workspace"), expected + ) + persist_workspace(tmp_path, second, index=0) + path = second.metadata_path(tmp_path) + os.utime(path, (path.stat().st_atime, path.stat().st_mtime + 1)) + np.testing.assert_array_equal( + manager.load(tmp_path / "virtual.npy", backend="virtual_workspace"), expected - 20 + ) diff --git a/tests/test_metadata_namespace_ownership.py b/tests/test_metadata_namespace_ownership.py new file mode 100644 index 0000000..534e3bd --- /dev/null +++ b/tests/test_metadata_namespace_ownership.py @@ -0,0 +1,25 @@ +"""Site-specific guards against the replaced ambient namespace mechanism.""" + +import ast +import inspect + +from polystore.virtual_workspace import VirtualWorkspaceBackend + + +def test_workspace_retains_its_namespace_owner_not_a_filename_projection(): + tree = ast.parse(inspect.getsource(VirtualWorkspaceBackend)) + calls = [node for node in ast.walk(tree) if isinstance(node, ast.Call)] + assert not any( + isinstance(call.func, ast.Name) and call.func.id == "get_metadata_path" for call in calls + ) + namespace_path_calls = [ + call + for call in calls + if isinstance(call.func, ast.Attribute) and call.func.attr == "metadata_path" + ] + assert len(namespace_path_calls) == 1 + assert ast.unparse(namespace_path_calls[0].func.value) == "self.metadata_config" + methods = {node.name: node for node in tree.body[0].body if isinstance(node, ast.FunctionDef)} + assert "self.metadata_config" in ast.unparse(methods["get_connection_params"]) + for method in ("from_connection_params", "set_connection_params"): + assert "params['metadata_config']" in ast.unparse(methods[method]) From ee1e5438d64cb7d37e2641c2d9b3ee8870663adf Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 22:05:51 -0400 Subject: [PATCH 10/12] Decode native workspace parameters through the existing constructor contract --- docs/source/metadata_namespace_327.rst | 5 +++-- src/polystore/virtual_workspace.py | 22 ++++++++++++---------- tests/test_metadata_namespace.py | 12 +++++++++++- tests/test_metadata_namespace_ownership.py | 13 ++++++++++++- 4 files changed, 38 insertions(+), 14 deletions(-) diff --git a/docs/source/metadata_namespace_327.rst b/docs/source/metadata_namespace_327.rst index d42997f..390c104 100644 --- a/docs/source/metadata_namespace_327.rst +++ b/docs/source/metadata_namespace_327.rst @@ -36,8 +36,9 @@ Durable state and new-case experiment Metadata remains durable JSON at its declared path, with the existing structured SourcePixelRef mapping. No new store, copy, rename, alternate reader or converter. -The runtime FileManager handoff carries the config object along with plate_root; -old runtime connection dictionaries are not accepted through a fallback. +The runtime FileManager handoff carries the config object along with plate_root. +Constructor keyword binding decodes the native dictionary into the owning +backend; no key-rewriting, version conversion or alternate metadata reader. Before, embedding a new namespace required coordinating an environment projection, dependency import timing and application producer declaration. After, one frozen diff --git a/src/polystore/virtual_workspace.py b/src/polystore/virtual_workspace.py index 4a68ef2..d3fb4ce 100644 --- a/src/polystore/virtual_workspace.py +++ b/src/polystore/virtual_workspace.py @@ -210,9 +210,18 @@ def __init__( *, metadata_config: MetadataConfig = METADATA_CONFIG, ): + self._registry: Mapping[str, BackendBase] | None = None + self._open_workspace(plate_root=plate_root, metadata_config=metadata_config) + + def _open_workspace( + self, + *, + plate_root: Path, + metadata_config: MetadataConfig, + ) -> None: + """Open one declared workspace, replacing its namespace-bound cache.""" self.plate_root = Path(plate_root) self.metadata_config = metadata_config - self._registry: Mapping[str, BackendBase] | None = None self._mapping_cache: Optional[Dict[str, SourcePixelRef]] = None self._cache_mtime: Optional[float] = None self._load_mapping() @@ -228,10 +237,7 @@ def from_connection_params( ) -> "VirtualWorkspaceBackend": if not params: raise ValueError("VirtualWorkspaceBackend requires plate_root.") - return cls( - plate_root=Path(params["plate_root"]), - metadata_config=params["metadata_config"], - ) + return cls(**params) def get_connection_params(self) -> Optional[Dict[str, Any]]: return { @@ -242,11 +248,7 @@ def get_connection_params(self) -> Optional[Dict[str, Any]]: def set_connection_params(self, params: Optional[Dict[str, Any]]) -> None: if not params: raise ValueError("VirtualWorkspaceBackend requires plate_root.") - self.plate_root = Path(params["plate_root"]) - self.metadata_config = params["metadata_config"] - self._mapping_cache = None - self._cache_mtime = None - self._load_mapping() + self._open_workspace(**params) @staticmethod def _normalize_relative_path(path_str: str) -> str: diff --git a/tests/test_metadata_namespace.py b/tests/test_metadata_namespace.py index df4e98b..e89e661 100644 --- a/tests/test_metadata_namespace.py +++ b/tests/test_metadata_namespace.py @@ -110,10 +110,11 @@ def test_connection_retarget_invalidates_mapping_and_namespace(tmp_path): persist_workspace(tmp_path, first, index=0) expected = persist_workspace(tmp_path, second) backend = VirtualWorkspaceBackend(tmp_path, metadata_config=first) + manager = FileManager({"disk": DiskBackend(), "virtual_workspace": backend}) backend.set_connection_params( VirtualWorkspaceBackend(tmp_path, metadata_config=second).get_connection_params() ) - manager = FileManager({"disk": DiskBackend(), "virtual_workspace": backend}) + assert backend._registry is manager.registry np.testing.assert_array_equal( manager.load(tmp_path / "virtual.npy", backend="virtual_workspace"), expected ) @@ -123,3 +124,12 @@ def test_connection_retarget_invalidates_mapping_and_namespace(tmp_path): np.testing.assert_array_equal( manager.load(tmp_path / "virtual.npy", backend="virtual_workspace"), expected - 20 ) + + +def test_native_parameter_boundary_uses_constructor_binding(tmp_path): + config = MetadataConfig(METADATA_FILENAME="explicit.json") + persist_workspace(tmp_path, config) + params = VirtualWorkspaceBackend(tmp_path, metadata_config=config).get_connection_params() + params["unknown"] = "not a workspace field" + with pytest.raises(TypeError, match="unknown"): + VirtualWorkspaceBackend.from_connection_params(params) diff --git a/tests/test_metadata_namespace_ownership.py b/tests/test_metadata_namespace_ownership.py index 534e3bd..4e8c0bd 100644 --- a/tests/test_metadata_namespace_ownership.py +++ b/tests/test_metadata_namespace_ownership.py @@ -22,4 +22,15 @@ def test_workspace_retains_its_namespace_owner_not_a_filename_projection(): methods = {node.name: node for node in tree.body[0].body if isinstance(node, ast.FunctionDef)} assert "self.metadata_config" in ast.unparse(methods["get_connection_params"]) for method in ("from_connection_params", "set_connection_params"): - assert "params['metadata_config']" in ast.unparse(methods[method]) + calls = [node for node in ast.walk(methods[method]) if isinstance(node, ast.Call)] + assert any( + keyword.arg is None and ast.unparse(keyword.value) == "params" + for call in calls + for keyword in call.keywords + ) + assert not any( + isinstance(node, ast.Subscript) + and isinstance(node.value, ast.Name) + and node.value.id == 'params' + for node in ast.walk(methods[method]) + ) From 91fd7e854ca760a16e7472d8d558b4012e89b6ba Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 22:08:20 -0400 Subject: [PATCH 11/12] Record constructor-bound source checks and unchanged structural guard --- docs/source/metadata_namespace_327.rst | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/source/metadata_namespace_327.rst b/docs/source/metadata_namespace_327.rst index 390c104..9197bc6 100644 --- a/docs/source/metadata_namespace_327.rst +++ b/docs/source/metadata_namespace_327.rst @@ -51,14 +51,32 @@ mock registry or serialization replacement. Source evidence and guards -------------------------- -Seven focused checks pass in4.45s, peak combined process RSS298.66 MiB, one CPU. +Final dependency sourceee1e5438d64cb7d37e2641c2d9b3ee8870663adf passes eleven +checks in4.42s, peak combined process RSS294.78 MiB, one CPU. This includes +three unchanged existing FileManager workspace/pickle preservation controls. They cover generic/custom namespace, explicit environment config with no mutation of the generic snapshot, real persisted reopen, connection replacement and cache refresh, absence of alternate-filename readers, source axes, exact pixels, sample provenance and native handoff. Site-specific AST guard requires the path -call to use self.metadata_config and the handoff methods to carry that owner. +call to use self.metadata_config and the handoff methods to use constructor +keyword binding rather than raw params subscripts. Constructor binding rejects +unknown fields. Reconnection preserves the existing execution-local registry. No guard exceptions or assertion weakening. +The unchanged original structural ratchet at agent-comms3b03785f45df2ef5dc62ba6aed99294192ecbb01 +passes against PR16 for the final committed source: StringSubscript delta-2, +all other deltas zero. The first draft48621c1 had delta+2 and failed; this is +retained as failed evidence, not waived. The fix removed duplicated dictionary +decoding through actual constructor/open-workspace contracts, not constant +substitutions or detector changes. Original tool SHA256 is +e323c94d49c2b72d9524a5169f123e64b4a6e46a41035ca9fb4497e49b6ca562. +The tool's package import fails under Python3.12 with NameError: InputDocument; +original CI uses3.14, and the existing system Python3.14.7 runs the unchanged +tool from a pinned own local Git worktree. No interpreter/package download or +tool source change. Product tests and native handoffs remain on Python3.12. +An initial site guard counted typing annotations as raw subscripts; its failure +is retained and the corrected guard targets params accesses specifically. + Lightweight full-package AST census at pinned HEAD completes in1.06s with12,209 code lines; it is not a global semantic audit. The original application failure was separately reproduced unchanged in4.35s at275.77 MiB RSS. Original XML/logs From 2b178e49854c663cbbcc09c7fb78cd9a48a27fc9 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Wed, 30 Sep 2026 23:15:52 -0400 Subject: [PATCH 12/12] Record paired installed namespace and input preparation checkpoint --- .../input_preparation_installed_20261001.rst | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 docs/source/input_preparation_installed_20261001.rst diff --git a/docs/source/input_preparation_installed_20261001.rst b/docs/source/input_preparation_installed_20261001.rst new file mode 100644 index 0000000..6cefac4 --- /dev/null +++ b/docs/source/input_preparation_installed_20261001.rst @@ -0,0 +1,31 @@ +Paired installed namespace checkpoint +===================================== + +PR16 parent production91fd7e854 includes namespace PR20 through normal +fast-forward. MetadataConfig owns path and managed-path operations; +VirtualWorkspaceBackend retains the exact declaration in its existing native +FileManager connection state. The application owns its filename through a +MetadataConfig subclass rather than changing the dependency singleton or globals. +BOUND-2/IMPL-12: one shared owner, no alternate metadata reader or mirror. + +Application PR206 production5ce9e3087 binds that application declaration at the +microscope registration boundary. Current application29 checks pass in both +import orders; five namespace checks pass with independent custom filenames. +Worker child11 controls and unchanged child structural guard remain evidence +at this identical production source. Earlier failures are not erased or counted +as current passes. No full-context NRA/native-proof claim. + +The paired PolyStore0.3.0 wheel is installed with OpenHCS0.8.7 in an isolated +candidate, no index/dependency/Fiji/environment downloads or global modification. +Actual installed MCP reopening and bounded samples succeed with the dependency +configured generic-private.json independently of OpenHCS metadata. Real raw +ImageXpress workspace initialization through source artifact-plan inspection +then persisted virtual sampling succeeds. Empty processing steps test only +initialization/compile, not analysis or a ZMQ job. Actual Java/CZI/native execution +and raw singleton Opera grid initialization remain explicit follow-ups; the +Opera structured failure is preserved and assigned under OpenHCS issue172. + +Canonical application receipt: docs/validation/input_preparation_installed_20261001.rst. +Exact wheel and original MCP/build/source evidence are retained in the durable +OpenHCS parent namespace-installed-20261001 ledger. This is local paired-wheel +acceptance, not public registry availability or an installed global release.