From 3e1e336089ccf2d3d3e20a849a885193daff1bba Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sat, 3 Oct 2026 00:33:54 -0400 Subject: [PATCH 1/6] Derive native catalog readiness from current public projections --- .../catalog-current-readiness-20261003.rst | 83 +++++++++++++++++++ .../services/function_catalog_service.py | 42 +++++++++- .../runtime/function_catalog_preparation.py | 32 +++++-- 3 files changed, 148 insertions(+), 9 deletions(-) create mode 100644 docs/validation/catalog-current-readiness-20261003.rst diff --git a/docs/validation/catalog-current-readiness-20261003.rst b/docs/validation/catalog-current-readiness-20261003.rst new file mode 100644 index 0000000000..61413481d4 --- /dev/null +++ b/docs/validation/catalog-current-readiness-20261003.rst @@ -0,0 +1,83 @@ +Current native function-catalog readiness +======================================== + +Owner and reproducer +-------------------- + +FunctionCatalogService owns the existing metadata-derived public projections; +FunctionCatalogPreparation owns their worker, future, cancellation and progress. +The native READ/SEARCH/DETAIL/REFERENCE template already calls this owner. + +The retained H001_REPEAT02 journey demonstrated successful custom registration, +an old READY preparation receipt, and subsequent catalog READ timeout at 4975ms. +An intervening describe succeeded; the preceding search is UNKNOWN, not replayed. +No inner per-callable cost was measured. This is a stale readiness correction, +not a claim that one particular reflection operation has been timed or optimized. + +The old successful future was reused forever, while metadata invalidation could +make the next public request rebuild a complete catalog projection in REP. Only +the compact view was prepared before serving; default full-view search could +also cause that work synchronously. + +Change +------ + +* Derive current readiness from the existing projection keys and metadata, + RegistryService.cached_metadata_snapshot, and the original persisted-source + revision owners. Do not maintain another generation, catalog or revision cache. +* Prepare both existing public views before advertising READY. Refresh only + these projections after invalidation, using the original preparation worker; + do not repeat native kernel startup for an ordinary source change. +* Coalesce pending work; preserve failed/cancelled futures without automatic + retry. A completed historical future is not current READY after invalidation. +* Apply the existing OperationCancellation to per-callable projection work. +* Leave registration admission, receipt and original invalidation untouched. + The next public catalog request uses the same existing preparation gate. + +Production scope is only agent/services/function_catalog_service.py and +runtime/function_catalog_preparation.py. RegistryService, CallableProjection, +custom-source decoding, compiler, native/MCP protocol and deadlines are excluded. +Root394 was notified before production edits in comments 5965485052/5965541595; +its current 4eaacf17 head does not change these files. + +Source and custody +------------------ + +Parent explicitly released the former programme source claim on the existing +/home/ts/wt/openhcs-input-preparation-20260929 checkout, originally e0dd14ed. +The exact full path matches Dalton's direct ownership response. Parent reports +privileged whole-borrower count 0 and access omissions 0. No source reuse of the +borrowed Singer505 or live436 checkouts was made. The branch was switched +nonrecursively from current main87d9a99; foreign submodule directories and .git +files were not updated/reset/cleaned. Their changed status against current-main +gitlinks is retained and excluded from this change. No environment was created. + +Full current NRA/refactor-audit skills, required pattern catalog and applicable +antipatterns were read. The existing selected-family AST receipt covers 28 +production modules and 8 installed dependency modules, 5235 source sites, with +zero parse errors. Dynamic behavior is not proven by that receipt. The standing +OPENHCS-HISTORY.md path is absent at main87d9a99 and the canonical checkout; no +replacement reminder/history store was invented. + +Acceptance and remaining boundary +--------------------------------- + +After the coherent source change, bounded controls must cover: + +* cold preparation and ordinary startup; warm repeated compact and full views; +* one-worker coalescing after add, same-id update and delete, without kernel rewarm; +* body-only persisted-source changes despite unchanged catalog membership ids; +* historical READY rejection and honest pending progress before refresh completes; +* projection failure, cancellation, retained terminal futures and incarnation mismatch; +* original read/search/detail/reference strategy consumers and admission behavior. + +Source controls do not establish installed/native acceptance. A later explicitly +released engineering slot must exercise the ordinary installed public registration, +search, describe and reference journey with unchanged budgets, plus responsiveness +during preparation. No currently live scientist, native, UI or UNKNOWN attempt +will be contacted/replayed. Inner reflection cost remains unmeasured. A registry +inventory lock held by a separate concurrent registry worker is an excluded Root +seam if encountered; it is not silently fixed with another lock or timeout here. + +Initial source checkpoint: verification pending; no build, installation or native +launch performed. Saved diagnosis and all scientific records remain immutable. diff --git a/openhcs/agent/services/function_catalog_service.py b/openhcs/agent/services/function_catalog_service.py index fa6b314ca5..9077c27cd4 100644 --- a/openhcs/agent/services/function_catalog_service.py +++ b/openhcs/agent/services/function_catalog_service.py @@ -6,6 +6,7 @@ import re from abc import ABC, abstractmethod from collections.abc import Callable +from concurrent.futures import CancelledError from dataclasses import dataclass from enum import Enum from pathlib import Path @@ -636,7 +637,42 @@ def prepare( status_callback=status_callback, cancellation=cancellation, ) - super().prepare(status_callback=status_callback, cancellation=cancellation) + self.prepare_projections(status_callback=status_callback, cancellation=cancellation) + + def projections_current(self) -> bool: + """Derive readiness from the original registry and cached public views.""" + from openhcs.processing.custom_functions.runtime_registry import ( + CustomFunctionRuntimeRegistry, + ) + + if self._projection_metadata is None or any( + (signature_view, SummaryView[signature_view.name]) not in self._projections + for signature_view in SignatureView + ): + return False + if self._projection_metadata != RegistryService.cached_metadata_snapshot(): + return False + manager = custom_function_manager.CustomFunctionManager(create_storage=False) + return manager.source_revision() == CustomFunctionRuntimeRegistry.source_revision() + + def prepare_projections( + self, + *, + status_callback: Callable[[str], None] | None = None, + cancellation: OperationCancellation | None = None, + ) -> None: + """Prepare both public views, without repeating native kernel warmup.""" + while True: + for signature_view in SignatureView: + if cancellation is not None and cancellation.requested(): + raise CancelledError + self.catalog( + compact_signatures=signature_view.compact, + status_callback=status_callback, + cancellation=cancellation, + ) + if self.projections_current(): + return def __init__(self, path_policy: AgentPathPolicy | None = None) -> None: self._path_policy = path_policy or AgentPathPolicy.from_environment() @@ -796,6 +832,7 @@ def _matching_entries( signature_view=signature_view, summary_view=summary_view, status_callback=status_callback, + cancellation=cancellation, ) candidates = [] for projection in projections: @@ -830,6 +867,7 @@ def _search_projections( signature_view: SignatureView, summary_view: SummaryView, status_callback: Callable[[str], None] | None, + cancellation: OperationCancellation | None = None, ) -> tuple[CatalogSearchProjection, ...]: """Project a registry revision once and reuse it across text queries.""" @@ -844,6 +882,8 @@ def _search_projections( status_callback("Projecting function metadata for the execution endpoint") projections = [] for function_id, metadata in sorted(metadata_by_id.items()): + if cancellation is not None and cancellation.requested(): + raise CancelledError contract = CallableContract.from_callable(metadata.func) projections.append( CatalogSearchProjection( diff --git a/openhcs/runtime/function_catalog_preparation.py b/openhcs/runtime/function_catalog_preparation.py index 37bcd9338a..34e0ba0ada 100644 --- a/openhcs/runtime/function_catalog_preparation.py +++ b/openhcs/runtime/function_catalog_preparation.py @@ -8,6 +8,7 @@ from collections.abc import Callable from concurrent.futures import CancelledError, Future from concurrent.futures import TimeoutError as FutureTimeoutError +from dataclasses import replace from typing import TYPE_CHECKING from zmqruntime import OperationCancellation @@ -20,7 +21,7 @@ FunctionCatalogPreparationState, ) from openhcs.agent.services.function_catalog_service import ( - FunctionCatalogServiceABC, + FunctionCatalogService, ) @@ -48,7 +49,7 @@ def cancel_preparation(_signal_number, _frame) -> None: def __init__( self, - function_catalog: "FunctionCatalogServiceABC", + function_catalog: "FunctionCatalogService", ) -> None: self._lock = threading.RLock() self._future: Future[None] | None = None @@ -63,17 +64,25 @@ def __init__( ) def ensure_started(self) -> Future[None]: - """Return the one preparation future, starting it when first requested.""" + """Coalesce preparation of the current projections, not historical READY.""" with self._lock: + prepare = self._function_catalog.prepare if self._future is not None: - return self._future + if ( + not self._future.done() + or self._future.cancelled() + or self._future.exception() is not None + or self._function_catalog.projections_current() + ): + return self._future + prepare = self._function_catalog.prepare_projections future = self._new_preparation_future() if future.cancelled(): return future thread = threading.Thread( target=self._prepare, - args=(future, self._function_catalog.prepare), + args=(future, prepare), name="openhcs-function-catalog-preparation", daemon=True, ) @@ -112,8 +121,7 @@ def _prepare_current_process(self, *, status_callback, cancellation) -> None: self.prepare_persistent_catalog(status_callback=status_callback) if cancellation.requested(): raise CancelledError - self._function_catalog.catalog( - compact_signatures=True, + self._function_catalog.prepare_projections( status_callback=status_callback, cancellation=cancellation, ) @@ -204,7 +212,15 @@ def observe( ), ) else: - outcome = Outcome.READY + if self._function_catalog.projections_current(): + outcome = Outcome.READY + else: + outcome = Outcome.NOT_STARTED + snapshot = replace( + snapshot, + phase=EndpointStartupPhase.PREPARING_CAPABILITIES, + message="Current function catalog projections require preparation", + ) return FunctionCatalogPreparationState( schema_version=SCHEMA_VERSION, handle=handle, From e3195c80704c385ad7c59d79544f46910c1d0dfd Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sat, 3 Oct 2026 00:41:29 -0400 Subject: [PATCH 2/6] Record bounded catalog controls and protected dependency boundary --- .../catalog-current-readiness-20261003.rst | 39 ++- .../services/function_catalog_service.py | 8 +- tests/unit/test_catalog_current_readiness.py | 222 ++++++++++++++++++ 3 files changed, 265 insertions(+), 4 deletions(-) create mode 100644 tests/unit/test_catalog_current_readiness.py diff --git a/docs/validation/catalog-current-readiness-20261003.rst b/docs/validation/catalog-current-readiness-20261003.rst index 61413481d4..10fec212f7 100644 --- a/docs/validation/catalog-current-readiness-20261003.rst +++ b/docs/validation/catalog-current-readiness-20261003.rst @@ -79,5 +79,40 @@ will be contacted/replayed. Inner reflection cost remains unmeasured. A registry inventory lock held by a separate concurrent registry worker is an excluded Root seam if encountered; it is not silently fixed with another lock or timeout here. -Initial source checkpoint: verification pending; no build, installation or native -launch performed. Saved diagnosis and all scientific records remain immutable. +Source checkpoint and actual controls +------------------------------------ + +Draft PR511 was published at 3e1e336089ccf2d3d3e20a849a885193daff1bba before +longer verification. The postchange selected-family AST receipt has 28 production +modules, 5262 sites, 8 dependency modules and zero parse errors. Runtime claims +remain separate from this evidence. ``git diff --check`` passed. + +All four bounded collection attempts are retained in the original engineering +receipt root as catalog511-source01..04.log/time; each used CPU1/512MiB/Swap0/60s. +They executed no scientific functions, native servers, sockets or viewers: + +1. systemd working directory omitted: pytest rejected relative confcutdir. +2. corrected directory: dependency import preceded source bootstrap; corrected + the test import ordering through the original source owner. +3. collection then reached missing compiled openhcs.core._tabular_native. +4. reused the accepted505 real extension directly, without copying/building it; + its C++ source matches this checkout, SHA256 of the ABI3 extension is + 7de8f671e79263518e56219b30085b2c39c9518db63739298c4ffc9265196aae. + Collection then failed because the protected old external/zmqruntime checkout + lacks ViewerReuseAdmissionABC required by current-main viewer_protocol. + +No assertions ran: this is a precise dependency-collection blocker, not a passing +regression suite. Attempt04 peaked at 179752KiB RSS (unit peak123.3MiB), zero swap, +3.015s. The last run reused an explicit actual compiled dependency for source +controls only; it is not ordinary installed/public/native acceptance. No substitute +registry/factory/backend, native extension stub, path fallback or protocol was +introduced into production. All foreign external trees remain unchanged. They +will not be updated, patched or bypassed to hide this source-import boundary. + +The authored controls cover warm full/compact public projection, current-source +revision, add/update/delete/body-only invalidation, refresh coalescing, retained +failure/cancellation, incarnation rejection and original main-thread startup. +Run them using current qualified dependencies after an explicit receiving release; +then perform the separately authorized ordinary installed native public journey. +No build, installation or native launch was performed. Saved diagnosis and all +scientific records remain immutable. diff --git a/openhcs/agent/services/function_catalog_service.py b/openhcs/agent/services/function_catalog_service.py index 9077c27cd4..5d382e864e 100644 --- a/openhcs/agent/services/function_catalog_service.py +++ b/openhcs/agent/services/function_catalog_service.py @@ -637,7 +637,9 @@ def prepare( status_callback=status_callback, cancellation=cancellation, ) - self.prepare_projections(status_callback=status_callback, cancellation=cancellation) + self.prepare_projections( + status_callback=status_callback, cancellation=cancellation + ) def projections_current(self) -> bool: """Derive readiness from the original registry and cached public views.""" @@ -653,7 +655,9 @@ def projections_current(self) -> bool: if self._projection_metadata != RegistryService.cached_metadata_snapshot(): return False manager = custom_function_manager.CustomFunctionManager(create_storage=False) - return manager.source_revision() == CustomFunctionRuntimeRegistry.source_revision() + return ( + manager.source_revision() == CustomFunctionRuntimeRegistry.source_revision() + ) def prepare_projections( self, diff --git a/tests/unit/test_catalog_current_readiness.py b/tests/unit/test_catalog_current_readiness.py new file mode 100644 index 0000000000..d0f2c63b8d --- /dev/null +++ b/tests/unit/test_catalog_current_readiness.py @@ -0,0 +1,222 @@ +"""Bounded source controls; no native, socket, viewer or kernel preparation.""" + +from concurrent.futures import CancelledError +from dataclasses import replace +from threading import Event +from types import SimpleNamespace + +import pytest + +# Activate the source owner's declared externals before dependency imports. +import openhcs # noqa: F401 +from zmqruntime import OperationCancellation +from zmqruntime.messages import ProcessIdentity + +from openhcs.agent.dto.execution_connection import ExecutionConnectionSpec +from openhcs.agent.dto.functions import ( + FunctionCatalogPreparationHandle, + FunctionCatalogPreparationOutcome as Outcome, +) +from openhcs.agent.services.function_catalog_service import FunctionCatalogService +from openhcs.processing.backends.lib_registry.registry_service import RegistryService +from openhcs.processing.backends.lib_registry.unified_registry import ( + FunctionMetadata, + ProcessingContract, +) +from openhcs.processing.custom_functions.manager import CustomFunctionManager +from openhcs.processing.custom_functions.runtime_registry import CustomFunctionRuntimeRegistry +from openhcs.runtime.function_catalog_preparation import FunctionCatalogPreparation + + +def declaration(image, gain: float = 1.0): + """A nonexecuted public-contract probe.""" + raise AssertionError("source controls must never execute a processing function") + + +@pytest.fixture +def catalog(monkeypatch, tmp_path): + monkeypatch.setenv("XDG_DATA_HOME", str(tmp_path / "data")) + manager = CustomFunctionManager() + metadata = FunctionMetadata( + name="declaration", func=declaration, + contract=ProcessingContract.FLEXIBLE, + registry=SimpleNamespace(library_name="probe"), + module=__name__, doc=declaration.__doc__, + ) + monkeypatch.setattr(RegistryService, "_metadata_cache", {"probe:declaration": metadata}) + monkeypatch.setattr(CustomFunctionRuntimeRegistry, "_source_revision", manager.source_revision()) + + def projected_metadata(self, **kwargs): + # Model the original registry/source owners publishing their revision. + CustomFunctionRuntimeRegistry._source_revision = manager.source_revision() + return RegistryService._metadata_cache + + monkeypatch.setattr(FunctionCatalogService, "_all_metadata", projected_metadata) + kernel_calls = [] + monkeypatch.setattr( + RegistryService, "prepare_persistent_catalog", + lambda **kwargs: kernel_calls.append("kernel"), + ) + return FunctionCatalogService(), manager, kernel_calls + + +def handle(): + # Nominal identity only: these controls never connect to this port. + return FunctionCatalogPreparationHandle( + ExecutionConnectionSpec(port=22319), ProcessIdentity.current() + ) + + +def test_both_public_views_are_warm(catalog, monkeypatch): + service, _, _ = catalog + calls = [] + original = service._entry + + def entry(*args, **kwargs): + calls.append(args[2:4]) + return original(*args, **kwargs) + + monkeypatch.setattr(service, "_entry", entry) + assert not service.projections_current() + service.prepare_projections() + assert service.projections_current() + assert len(calls) == 2 + for compact in (False, True, False, True): + page = service.catalog(compact_signatures=compact) + assert service.search(query="declaration", compact_signatures=compact).items == page.items + service.require_revision(page.revision) + assert len(calls) == 2 + assert service.get("probe:declaration").entry.name == "declaration" + assert service.reference("probe:declaration") is not None + + +@pytest.mark.parametrize("mutation", ("add", "update", "delete", "body_only")) +def test_stale_success_refreshes_without_kernel_rewarm(catalog, mutation): + service, manager, kernels = catalog + preparation = FunctionCatalogPreparation(service) + first = preparation.ensure_started() + first.result(timeout=2) + assert preparation.observe(handle()).outcome is Outcome.READY + old = RegistryService._metadata_cache + if mutation == "body_only": + manager.storage_dir.joinpath("source_probe.py").write_text("# source revision changed\n") + elif mutation == "delete": + RegistryService._metadata_cache = {} + elif mutation == "update": + RegistryService._metadata_cache = { + "probe:declaration": replace(old["probe:declaration"], doc="Updated declaration") + } + else: + RegistryService._metadata_cache = { + **old, "probe:second": replace(old["probe:declaration"], name="second") + } + state = preparation.observe(handle()) + assert state.outcome is Outcome.NOT_STARTED + assert "require preparation" in state.progress.message + refreshed = preparation.ensure_started() + assert refreshed is not first + refreshed.result(timeout=2) + assert preparation.ensure_started() is refreshed + assert preparation.observe(handle()).outcome is Outcome.READY + assert kernels == ["kernel"] + preparation.cancel_and_join() + + +def test_current_refresh_coalesces_and_reports_pending(catalog, monkeypatch): + service, _, _ = catalog + preparation = FunctionCatalogPreparation(service) + preparation.ensure_started().result(timeout=2) + RegistryService._metadata_cache = dict(RegistryService._metadata_cache) + RegistryService._metadata_cache["probe:second"] = replace( + RegistryService._metadata_cache["probe:declaration"], name="second" + ) + started, release = Event(), Event() + original = service._entry + + def delayed_entry(*args, **kwargs): + started.set() + assert release.wait(timeout=2) + return original(*args, **kwargs) + + monkeypatch.setattr(service, "_entry", delayed_entry) + future = preparation.ensure_started() + try: + assert started.wait(timeout=2) + assert preparation.ensure_started() is future + assert preparation.observe(handle()).outcome is Outcome.PENDING + finally: + release.set() + future.result(timeout=2) + preparation.cancel_and_join() + + +def test_failure_is_terminal_not_replayed(catalog, monkeypatch): + service, _, kernels = catalog + + def fail(*args, **kwargs): + raise ValueError("projection failure witness") + + monkeypatch.setattr(service, "_entry", fail) + preparation = FunctionCatalogPreparation(service) + future = preparation.ensure_started() + with pytest.raises(ValueError, match="failure witness"): + future.result(timeout=2) + assert preparation.observe(handle()).outcome is Outcome.FAILED + assert preparation.ensure_started() is future + assert kernels == ["kernel"] + preparation.cancel_and_join() + + +def test_projection_cancellation_does_not_publish_partial_view(catalog, monkeypatch): + service, _, _ = catalog + cancellation = OperationCancellation() + original = service._entry + + def cancelling_entry(*args, **kwargs): + cancellation.cancel() + return original(*args, **kwargs) + + monkeypatch.setattr(service, "_entry", cancelling_entry) + with pytest.raises(CancelledError): + service.prepare_projections(cancellation=cancellation) + assert not service.projections_current() + + +def test_cancelled_owner_remains_terminal(catalog): + service, _, kernels = catalog + preparation = FunctionCatalogPreparation(service) + preparation.cancel_and_join() + future = preparation.ensure_started() + assert future.cancelled() + assert preparation.observe(handle()).outcome is Outcome.CANCELLED + assert preparation.ensure_started() is future + assert not kernels + + +def test_wrong_incarnation_is_rejected_before_preparation(catalog): + service, _, kernels = catalog + preparation = FunctionCatalogPreparation(service) + identity = ProcessIdentity.current() + wrong = FunctionCatalogPreparationHandle( + ExecutionConnectionSpec(port=22319), + replace(identity, create_time=identity.create_time + 1), + ) + with pytest.raises(RuntimeError): + preparation.observe(wrong) + assert not kernels + + +def test_startup_uses_original_main_thread_warmup(catalog, monkeypatch): + service, _, child_kernels = catalog + main_kernels = [] + monkeypatch.setattr( + FunctionCatalogPreparation, "prepare_persistent_catalog", + lambda **kwargs: main_kernels.append("main"), + ) + preparation = FunctionCatalogPreparation(service) + preparation.prepare_before_serving() + assert preparation.observe(handle()).outcome is Outcome.READY + assert service.projections_current() + assert main_kernels == ["main"] + assert not child_kernels + preparation.cancel_and_join() From 593eecfcb4c5cd0950a1fa153ccca9b4165c7dce Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sat, 3 Oct 2026 01:13:55 -0400 Subject: [PATCH 3/6] Keep catalog readiness on its ABC and reject obsolete projection commits --- .../catalog-current-readiness-20261003.rst | 70 +++++- .../endpoint_function_catalog_service.py | 31 ++- .../services/function_catalog_service.py | 88 ++++--- .../runtime/function_catalog_preparation.py | 6 +- tests/unit/test_catalog_current_readiness.py | 217 +++++++++++++++++- ...st_execution_server_startup_preparation.py | 6 + tests/unit/test_function_catalog_zmq.py | 16 +- 7 files changed, 379 insertions(+), 55 deletions(-) diff --git a/docs/validation/catalog-current-readiness-20261003.rst b/docs/validation/catalog-current-readiness-20261003.rst index 10fec212f7..bbad52d2ee 100644 --- a/docs/validation/catalog-current-readiness-20261003.rst +++ b/docs/validation/catalog-current-readiness-20261003.rst @@ -31,14 +31,26 @@ Change * Coalesce pending work; preserve failed/cancelled futures without automatic retry. A completed historical future is not current READY after invalidation. * Apply the existing OperationCancellation to per-callable projection work. + Check cancellation after the last entry and before successful publication. +* Keep FunctionCatalogPreparation on the original FunctionCatalogServiceABC + boundary. That ABC owns the shared two-view refresh recipe; local and endpoint + implementations own their respective current-readiness decisions. The endpoint + implementation asks the original bound native preparation-status owner. +* Move the endpoint implementation's existing state lock to the ABC owner; + reuse it for short cache adoption/publication sections in both implementations. + Reflection and source-revision polling remain outside it. An old view cannot + enter a cache that another caller has already switched to new metadata. * Leave registration admission, receipt and original invalidation untouched. The next public catalog request uses the same existing preparation gate. -Production scope is only agent/services/function_catalog_service.py and +Production scope is agent/services/function_catalog_service.py, +agent/services/endpoint_function_catalog_service.py and runtime/function_catalog_preparation.py. RegistryService, CallableProjection, custom-source decoding, compiler, native/MCP protocol and deadlines are excluded. Root394 was notified before production edits in comments 5965485052/5965541595; -its current 4eaacf17 head does not change these files. +the additional ABC/endpoint hunk was declared in comment 5965707429. Its checked +8e06a382 head does not change these files. No competing catalog authority, +generation store, protocol, timeout or fallback was introduced. Source and custody ------------------ @@ -46,7 +58,10 @@ Source and custody Parent explicitly released the former programme source claim on the existing /home/ts/wt/openhcs-input-preparation-20260929 checkout, originally e0dd14ed. The exact full path matches Dalton's direct ownership response. Parent reports -privileged whole-borrower count 0 and access omissions 0. No source reuse of the +privileged whole-borrower count 0 and access omissions 0. The complete Dalton +receipt was personally read at neurite-development-skill383-20261001/output/ +resource-owner-20261002/COPERNICUS-CATALOG-SOURCE-REUSE-INPUT-PREPARATION.rst +under the existing issue-batch root. No source reuse of the borrowed Singer505 or live436 checkouts was made. The branch was switched nonrecursively from current main87d9a99; foreign submodule directories and .git files were not updated/reset/cleaned. Their changed status against current-main @@ -101,18 +116,55 @@ They executed no scientific functions, native servers, sockets or viewers: Collection then failed because the protected old external/zmqruntime checkout lacks ViewerReuseAdmissionABC required by current-main viewer_protocol. -No assertions ran: this is a precise dependency-collection blocker, not a passing +No assertions ran in those four attempts: this was a precise dependency-collection blocker, not a passing regression suite. Attempt04 peaked at 179752KiB RSS (unit peak123.3MiB), zero swap, 3.015s. The last run reused an explicit actual compiled dependency for source controls only; it is not ordinary installed/public/native acceptance. No substitute registry/factory/backend, native extension stub, path fallback or protocol was introduced into production. All foreign external trees remain unchanged. They -will not be updated, patched or bypassed to hide this source-import boundary. +were not updated or patched. Parent then explicitly authorized binding existing +qualified dependency origins read-only for source controls; no installed file or +environment was changed. The authored controls cover warm full/compact public projection, current-source revision, add/update/delete/body-only invalidation, refresh coalescing, retained failure/cancellation, incarnation rejection and original main-thread startup. -Run them using current qualified dependencies after an explicit receiving release; -then perform the separately authorized ordinary installed native public journey. -No build, installation or native launch was performed. Saved diagnosis and all -scientific records remain immutable. +Receiving controls and nominal family correction +----------------------------------------------- + +The existing shared interpreter is +/home/ts/wt/openhcs-paired-raw-installed-parent-20261001/.venv/bin/python. +The explicitly asserted dependency origins are that environment's +lib/python3.12/site-packages for zmqruntime, PolyStore, arraybridge, +metaclass-registry and pyqt-reactive, and +/home/ts/wt/basicpy-live-candidate-20260930/.venv/lib/python3.12/site-packages +for python-introspect (the shared interpreter's original declared dependency). +The real accepted505 ABI3 extension remains at the existing engineering receipt +root's scratch/installed505/openhcs/core/_tabular_native.abi3.so. It was not copied, +built or substituted. Source module paths and dependency origins/hashes are +asserted and recorded by catalog511-shared-dependency-controls.py on each run. +Determining zmqruntime, PolyStore and python-introspect files were compared against +the current recorded Git objects through the canonical repository; this is not +a claim of whole-package identity or installed/native acceptance. + +All receiving attempts remain in the same receipt root: + +* source05: 18 passed, one test-fixture binding failure. +* source06: 71 passed, six fixture failures (router-only fixtures did not declare + current catalog readiness; one cleanup cancelled work before expecting success). +* source07: 77 passed after coherent fixture corrections. Real invalidation, + cancellation and failed-future negatives were retained. +* source08: 78 passed, including the overlapping-reader obsolete-cache-commit + control, midpoint metadata/source invalidation, all nominal ABC implementations + and the final-entry cancellation checks. Test time 9.57s, wall12.94s, + cgroup peak214.2MiB, Swap0, CPU1/512MiB/60s, exit0. Two pytest configuration + warnings result from deliberately disabled plugin autoload, not failing tests. + +The production ABC owns the only shared refresh recipe and family state-lock +declaration. Both implementations retain their original catalog projections; +all preparation/strategy consumers take the existing nominal boundary. No +reflection or source-poll performance claim follows from these controls. +No build, installation, scientific execution, server or viewer launch was +performed. Saved diagnosis, UNKNOWN requests and scientific records remain +immutable. Actual ordinary installed public/native acceptance still awaits an +explicitly released engineering slot; :95 is not yet released. diff --git a/openhcs/agent/services/endpoint_function_catalog_service.py b/openhcs/agent/services/endpoint_function_catalog_service.py index ec0092a0bb..4055c4bc4c 100644 --- a/openhcs/agent/services/endpoint_function_catalog_service.py +++ b/openhcs/agent/services/endpoint_function_catalog_service.py @@ -216,6 +216,7 @@ def __init__( client_factory: FunctionCatalogClientFactory | None = None, path_policy: AgentPathPolicy | None = None, ) -> None: + super().__init__() self._config_provider = config_provider self._path_policy = path_policy or AgentPathPolicy.from_environment() self._client_factory = client_factory or self._new_client @@ -223,7 +224,6 @@ def __init__( self._endpoint_state: FunctionCatalogEndpointState | None = None self._preparation: FunctionCatalogPreparation | None = None self._closed = False - self._state_lock = threading.RLock() def _new_client(self, config: OpenHCSZMQConfig) -> ZMQExecutionClient: from openhcs.runtime.zmq_execution_client import ZMQExecutionClient @@ -236,6 +236,35 @@ def projection(self) -> FunctionCatalogProjection | None: state = self._endpoint_state return state if isinstance(state, FunctionCatalogProjection) else None + def projections_current(self) -> bool: + """Ask the bound native owner, not a historical local page, for readiness.""" + endpoint = self._config_provider() + with self._state_lock: + projection = self.projection + session = self._client_session + if ( + self._closed + or projection is None + or projection.endpoint != endpoint + or session is None + or session.endpoint != endpoint + ): + return False + connected = session.client.connected_endpoint + if connected is None or connected.process_identity is None: + return False + handle = FunctionCatalogPreparationHandle( + ExecutionConnectionSpec( + host=endpoint.client_host, + port=endpoint.default_port, + transport_mode=endpoint.transport_mode, + persistent=endpoint.persistent, + ), + connected.process_identity, + ) + state = self.catalog_preparation_status(handle) + return state.outcome.ready + def prepare( self, *, diff --git a/openhcs/agent/services/function_catalog_service.py b/openhcs/agent/services/function_catalog_service.py index 5d382e864e..f71566984b 100644 --- a/openhcs/agent/services/function_catalog_service.py +++ b/openhcs/agent/services/function_catalog_service.py @@ -4,6 +4,7 @@ import inspect import re +import threading from abc import ABC, abstractmethod from collections.abc import Callable from concurrent.futures import CancelledError @@ -551,6 +552,9 @@ def parameter_description( class FunctionCatalogServiceABC(ABC): """Callable-catalog authority consumed by agent authoring services.""" + def __init__(self) -> None: + self._state_lock = threading.RLock() + def prepare( self, *, @@ -559,12 +563,35 @@ def prepare( ) -> None: """Prepare the authoritative catalog through this service's transport.""" - self.catalog( - compact_signatures=True, - status_callback=status_callback, - cancellation=cancellation, + self.prepare_projections( + status_callback=status_callback, cancellation=cancellation ) + @abstractmethod + def projections_current(self) -> bool: + """Whether the owned public projections match their current authority.""" + + def prepare_projections( + self, + *, + status_callback: Callable[[str], None] | None = None, + cancellation: OperationCancellation | None = None, + ) -> None: + """Prepare both public views against the implementation's authority.""" + while True: + for signature_view in SignatureView: + if cancellation is not None and cancellation.requested(): + raise CancelledError + self.catalog( + compact_signatures=signature_view.compact, + status_callback=status_callback, + cancellation=cancellation, + ) + if cancellation is not None and cancellation.requested(): + raise CancelledError + if self.projections_current(): + return + @abstractmethod def register_custom_function( self, @@ -647,38 +674,22 @@ def projections_current(self) -> bool: CustomFunctionRuntimeRegistry, ) - if self._projection_metadata is None or any( - (signature_view, SummaryView[signature_view.name]) not in self._projections - for signature_view in SignatureView - ): - return False - if self._projection_metadata != RegistryService.cached_metadata_snapshot(): + with self._state_lock: + metadata = self._projection_metadata + if metadata is None or any( + (signature_view, SummaryView[signature_view.name]) not in self._projections + for signature_view in SignatureView + ): + return False + if metadata != RegistryService.cached_metadata_snapshot(): return False manager = custom_function_manager.CustomFunctionManager(create_storage=False) return ( manager.source_revision() == CustomFunctionRuntimeRegistry.source_revision() ) - def prepare_projections( - self, - *, - status_callback: Callable[[str], None] | None = None, - cancellation: OperationCancellation | None = None, - ) -> None: - """Prepare both public views, without repeating native kernel warmup.""" - while True: - for signature_view in SignatureView: - if cancellation is not None and cancellation.requested(): - raise CancelledError - self.catalog( - compact_signatures=signature_view.compact, - status_callback=status_callback, - cancellation=cancellation, - ) - if self.projections_current(): - return - def __init__(self, path_policy: AgentPathPolicy | None = None) -> None: + super().__init__() self._path_policy = path_policy or AgentPathPolicy.from_environment() self._projection_metadata: dict[str, FunctionMetadata] | None = None self._projections: dict[ @@ -875,13 +886,14 @@ def _search_projections( ) -> tuple[CatalogSearchProjection, ...]: """Project a registry revision once and reuse it across text queries.""" - if metadata_by_id is not self._projection_metadata: - self._projection_metadata = metadata_by_id - self._projections.clear() cache_key = (signature_view, summary_view) - cached = self._projections.get(cache_key) - if cached is not None: - return cached + with self._state_lock: + if metadata_by_id is not self._projection_metadata: + self._projection_metadata = metadata_by_id + self._projections.clear() + cached = self._projections.get(cache_key) + if cached is not None: + return cached if status_callback is not None: status_callback("Projecting function metadata for the execution endpoint") projections = [] @@ -905,8 +917,12 @@ def _search_projections( ), ) ) + if cancellation is not None and cancellation.requested(): + raise CancelledError projected = tuple(projections) - self._projections[cache_key] = projected + with self._state_lock: + if metadata_by_id is self._projection_metadata: + self._projections[cache_key] = projected return projected def get( diff --git a/openhcs/runtime/function_catalog_preparation.py b/openhcs/runtime/function_catalog_preparation.py index 34e0ba0ada..613726ffb7 100644 --- a/openhcs/runtime/function_catalog_preparation.py +++ b/openhcs/runtime/function_catalog_preparation.py @@ -21,7 +21,7 @@ FunctionCatalogPreparationState, ) from openhcs.agent.services.function_catalog_service import ( - FunctionCatalogService, + FunctionCatalogServiceABC, ) @@ -49,7 +49,7 @@ def cancel_preparation(_signal_number, _frame) -> None: def __init__( self, - function_catalog: "FunctionCatalogService", + function_catalog: "FunctionCatalogServiceABC", ) -> None: self._lock = threading.RLock() self._future: Future[None] | None = None @@ -269,6 +269,8 @@ def report(message: str) -> None: status_callback=report, cancellation=self._cancellation, ) + if self._cancellation.requested(): + raise CancelledError except CancelledError: self._set_message( "Function catalog preparation cancelled", diff --git a/tests/unit/test_catalog_current_readiness.py b/tests/unit/test_catalog_current_readiness.py index d0f2c63b8d..789d23dd06 100644 --- a/tests/unit/test_catalog_current_readiness.py +++ b/tests/unit/test_catalog_current_readiness.py @@ -180,6 +180,105 @@ def cancelling_entry(*args, **kwargs): with pytest.raises(CancelledError): service.prepare_projections(cancellation=cancellation) assert not service.projections_current() + assert not service._projections + + +@pytest.mark.parametrize("source_change", (False, True)) +def test_midprojection_change_rebuilds_both_views_before_ready( + catalog, monkeypatch, source_change +): + service, manager, _ = catalog + old = RegistryService._metadata_cache["probe:declaration"] + changed = Event() + calls = [] + original = service._entry + + def entry(*args, **kwargs): + calls.append((args[1], args[2])) + result = original(*args, **kwargs) + if not changed.is_set(): + assert not preparation._future.done() + assert preparation.observe(handle()).outcome is Outcome.PENDING + RegistryService._metadata_cache = { + "probe:declaration": replace(old, doc="Current declaration prose") + } + if source_change: + manager.storage_dir.joinpath("midway.py").write_text("# current source\n") + changed.set() + return result + + monkeypatch.setattr(service, "_entry", entry) + preparation = FunctionCatalogPreparation(service) + preparation.ensure_started().result(timeout=2) + assert preparation.observe(handle()).outcome is Outcome.READY + assert service.projections_current() + from openhcs.agent.services.function_catalog_service import SignatureView + + current = RegistryService._metadata_cache["probe:declaration"] + assert (current, SignatureView.FULL) in calls + assert (current, SignatureView.COMPACT) in calls + assert all( + projection.metadata is current + for view in service._projections.values() + for projection in view + ) + preparation.cancel_and_join() + + +def test_cancel_during_last_entry_cancels_future_without_ready(catalog, monkeypatch): + service, _, _ = catalog + from openhcs.agent.services.function_catalog_service import SignatureView + + original = service._entry + + def cancelling_entry(*args, **kwargs): + result = original(*args, **kwargs) + if args[2] is SignatureView.COMPACT: + preparation._cancellation.cancel() + return result + + monkeypatch.setattr(service, "_entry", cancelling_entry) + preparation = FunctionCatalogPreparation(service) + future = preparation.ensure_started() + with pytest.raises(CancelledError): + future.result(timeout=2) + assert preparation.observe(handle()).outcome is Outcome.CANCELLED + assert preparation.ensure_started() is future + assert not service.projections_current() + preparation.cancel_and_join() + + +def test_overlapping_projection_cannot_commit_old_view_into_new_metadata(catalog, monkeypatch): + service, _, _ = catalog + from openhcs.agent.services.function_catalog_service import SignatureView + + changed = Event() + original = service._entry + old = RegistryService._metadata_cache["probe:declaration"] + current = replace(old, doc="Current overlapping projection") + + def overlapping_entry(*args, **kwargs): + result = original(*args, **kwargs) + if not changed.is_set(): + changed.set() + RegistryService._metadata_cache = {"probe:declaration": current} + service.catalog(compact_signatures=True) + assert not preparation._future.done() + assert preparation.observe(handle()).outcome is Outcome.PENDING + return result + + monkeypatch.setattr(service, "_entry", overlapping_entry) + preparation = FunctionCatalogPreparation(service) + preparation.ensure_started().result(timeout=2) + assert preparation.observe(handle()).outcome is Outcome.READY + assert service.projections_current() + assert all( + projection.metadata is current + for view in service._projections.values() + for projection in view + ) + assert {key[0] for key in service._projections} == set(SignatureView) + preparation.cancel_and_join() def test_cancelled_owner_remains_terminal(catalog): @@ -211,7 +310,7 @@ def test_startup_uses_original_main_thread_warmup(catalog, monkeypatch): main_kernels = [] monkeypatch.setattr( FunctionCatalogPreparation, "prepare_persistent_catalog", - lambda **kwargs: main_kernels.append("main"), + staticmethod(lambda **kwargs: main_kernels.append("main")), ) preparation = FunctionCatalogPreparation(service) preparation.prepare_before_serving() @@ -220,3 +319,119 @@ def test_startup_uses_original_main_thread_warmup(catalog, monkeypatch): assert main_kernels == ["main"] assert not child_kernels preparation.cancel_and_join() + + +def test_original_abc_owns_refresh_and_both_implementations_are_concrete(): + from openhcs.agent.services.function_catalog_service import FunctionCatalogServiceABC + from openhcs.agent.services.endpoint_function_catalog_service import ZMQFunctionCatalogService + + assert "projections_current" in FunctionCatalogServiceABC.__abstractmethods__ + assert FunctionCatalogService.prepare_projections is FunctionCatalogServiceABC.prepare_projections + assert ZMQFunctionCatalogService.prepare_projections is FunctionCatalogServiceABC.prepare_projections + assert not FunctionCatalogService.__abstractmethods__ + assert not ZMQFunctionCatalogService.__abstractmethods__ + + +@pytest.mark.parametrize("native_outcome", (Outcome.READY, Outcome.PENDING, Outcome.FAILED)) +def test_endpoint_current_readiness_uses_bound_native_status(native_outcome): + from openhcs.agent.dto.common import SCHEMA_VERSION + from openhcs.agent.dto.functions import FunctionCatalogPreparationState, FunctionCatalogEntry, catalog_page + from openhcs.agent.services.endpoint_function_catalog_service import ( + FunctionCatalogClientSession, FunctionCatalogProjection, ZMQFunctionCatalogService, + ) + from openhcs.runtime.zmq_config import OPENHCS_ZMQ_CONFIG + from zmqruntime.startup import EndpointStartupPhase, EndpointStartupStatus + + endpoint = replace(OPENHCS_ZMQ_CONFIG, default_port=22319) + seen = [] + + def status(request): + seen.append(request) + return FunctionCatalogPreparationState( + schema_version=SCHEMA_VERSION, handle=request.handle, + outcome=native_outcome, + progress=EndpointStartupStatus( + sequence=1, phase=EndpointStartupPhase.PREPARING_CAPABILITIES, + message="Native readiness witness", timestamp=0, + ), + ) + + client = SimpleNamespace( + connected_endpoint=SimpleNamespace(process_identity=ProcessIdentity.current()), + function_catalog_preparation=status, + ) + service = ZMQFunctionCatalogService(lambda: endpoint, client_factory=lambda _: client) + assert not service.projections_current() + assert not seen + entry = FunctionCatalogEntry( + function_id="probe:declaration", import_path=__name__ + ".declaration", + name="declaration", module=__name__, library="probe", + signature="declaration(image)", summary="Witness", backend_tags=(), + ) + page = catalog_page( + items=(entry,), catalog_items=(entry,), total=1, limit=1, query=None, library=None, + ) + service._client_session = FunctionCatalogClientSession(endpoint, client) + service._endpoint_state = FunctionCatalogProjection.from_page( + endpoint, page, compact_signatures=True, + ) + assert service.projections_current() is (native_outcome is Outcome.READY) + assert len(seen) == 1 + assert seen[0].handle.server_identity == ProcessIdentity.current() + + +@pytest.mark.parametrize("request_kind", ("read", "search", "detail", "reference")) +def test_original_control_family_stays_pending_during_refresh( + catalog, monkeypatch, request_kind +): + from openhcs.agent.dto.functions import ( + FunctionCatalogControlPayload, FunctionCatalogControlRequest, + FunctionCatalogPreparationControlResponse, FunctionDetailControlRequest, + FunctionReferenceControlRequest, FunctionSearchRequest, + ) + from openhcs.runtime.zmq_control import ZMQControlMessageRouter, ZMQControlRequestContext + + service, _, _ = catalog + preparation = FunctionCatalogPreparation(service) + preparation.ensure_started().result(timeout=2) + revision = service.catalog().revision + RegistryService._metadata_cache = { + "probe:declaration": replace( + RegistryService._metadata_cache["probe:declaration"], doc="Changed" + ) + } + entered, release = Event(), Event() + original = service._entry + + def delayed(*args, **kwargs): + entered.set() + assert release.wait(timeout=2) + return original(*args, **kwargs) + + monkeypatch.setattr(service, "_entry", delayed) + future = preparation.ensure_started() + requests = { + "read": FunctionCatalogControlRequest(), + "search": FunctionSearchRequest(query="declaration"), + "detail": FunctionDetailControlRequest( + function_id="probe:declaration", catalog_revision=revision + ), + "reference": FunctionReferenceControlRequest( + function_id="probe:declaration", catalog_revision=revision + ), + } + try: + assert entered.wait(timeout=2) + response = ZMQControlMessageRouter.handle( + FunctionCatalogControlPayload.from_request(requests[request_kind]).to_dict(), + ZMQControlRequestContext( + compiled_artifacts={}, function_catalog=service, + function_catalog_preparation=preparation, + ), + ) + pending = FunctionCatalogPreparationControlResponse.from_control_response(response) + assert pending.status is not None + assert not future.done() + finally: + release.set() + preparation.cancel_and_join() diff --git a/tests/unit/test_execution_server_startup_preparation.py b/tests/unit/test_execution_server_startup_preparation.py index 3180e125d6..6011773fdb 100644 --- a/tests/unit/test_execution_server_startup_preparation.py +++ b/tests/unit/test_execution_server_startup_preparation.py @@ -27,6 +27,12 @@ def prepare(self, **kwargs): def catalog(self, **kwargs): self.events.append("catalog") + def prepare_projections(self, **kwargs): + self.catalog(**kwargs) + + def projections_current(self): + return "catalog" in self.events + def test_direct_server_start_warms_main_thread_before_bind_and_reuses_future( monkeypatch, diff --git a/tests/unit/test_function_catalog_zmq.py b/tests/unit/test_function_catalog_zmq.py index 6fec2f9f13..cf740e73f6 100644 --- a/tests/unit/test_function_catalog_zmq.py +++ b/tests/unit/test_function_catalog_zmq.py @@ -284,6 +284,10 @@ def _context( if preparation_future is None: ready.set_result(None) catalog = FunctionCatalogService() + # Router-only fixture: the catalog is explicitly current as well as the + # injected future being complete. Actual invalidation/readiness is covered + # by test_catalog_current_readiness, not by these mocked payload handlers. + catalog.projections_current = lambda: True preparation = FunctionCatalogPreparation(catalog) preparation._future = ready return ZMQControlRequestContext( @@ -840,7 +844,7 @@ def reject_preparation(): def test_native_preparation_start_status_cancel_are_responsive_and_incarnation_owned(): entered = threading.Event() - class ControlledCatalog: + class ControlledCatalog(FunctionCatalogService): def prepare(self, *, status_callback, cancellation): status_callback("Controlled preparation remains pending") entered.set() @@ -1022,7 +1026,7 @@ def test_function_catalog_preparation_cancellation_reaches_catalog_owner() -> No started = threading.Event() - class CancellableCatalog: + class CancellableCatalog(FunctionCatalogService): def prepare( self, *, @@ -1086,13 +1090,12 @@ def prepare(*, status_callback, cancellation): started.set() assert release.wait(timeout=5) - def catalog(self, *, compact_signatures, status_callback, cancellation): - assert compact_signatures is True + def projections(self, *, status_callback, cancellation): events.append("catalog") return _catalog() monkeypatch.setattr(RegistryService, "prepare_persistent_catalog", prepare) - monkeypatch.setattr(FunctionCatalogService, "catalog", catalog) + monkeypatch.setattr(FunctionCatalogService, "prepare_projections", projections) preparation = FunctionCatalogPreparation(FunctionCatalogService()) future = preparation.ensure_started() try: @@ -1100,10 +1103,11 @@ def catalog(self, *, compact_signatures, status_callback, cancellation): assert preparation.ensure_started() is future assert not future.done() assert preparation.snapshot().message == "Preparing declared kernels" + release.set() + assert future.result(timeout=2) is None finally: release.set() preparation.cancel_and_join() - assert future.result() is None assert events == ["kernels", "catalog"] From 5dc21d5720baca32f8170f89b2359c8b8408b4ba Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sat, 3 Oct 2026 01:19:03 -0400 Subject: [PATCH 4/6] Use nominal startup fixture and retain current-main dependency boundary --- .../catalog-current-readiness-20261003.rst | 34 +++++++++++++++++++ ...st_execution_server_startup_preparation.py | 15 ++++---- 2 files changed, 42 insertions(+), 7 deletions(-) diff --git a/docs/validation/catalog-current-readiness-20261003.rst b/docs/validation/catalog-current-readiness-20261003.rst index bbad52d2ee..ffec222a72 100644 --- a/docs/validation/catalog-current-readiness-20261003.rst +++ b/docs/validation/catalog-current-readiness-20261003.rst @@ -168,3 +168,37 @@ No build, installation, scientific execution, server or viewer launch was performed. Saved diagnosis, UNKNOWN requests and scientific records remain immutable. Actual ordinary installed public/native acceptance still awaits an explicitly released engineering slot; :95 is not yet released. + +Current-main receiving boundary +------------------------------- + +Working checkpoint 593eecfcb was normally merged with current main236644e43 at +3cccbb0f1, including guide512/d11032292 and disjoint registered-source warmup514. +No dependency working tree was changed by this nonrecursive merge. The main +change records ObjectState9c31e4b07b5537c0ed2307c5a63034c708b86bbb and +python-introspect6f5ac0b79d6cef163df6a8389e72a889a91ed548. + +Merged-source attempt09 is retained: 77 passed, one failed in 8.87s, +unit12.161s/peak203.3MiB/Swap0/exit1. The original real-registry startup control +reaches main514's new LazyDataclassFactory.registered_type_pairs call, absent in +the previously qualified dependency. This is an exact receiving-dependency +boundary, not passing current-main acceptance; the negative was not weakened. +The two current dependency Git objects are not present in the canonical local +dependency stores, whose working heads differ and include foreign dirty edits. +They were neither imported nor fetched/updated. Parent was asked for existing +qualified origins for those exact main514 dependencies. No fallback, stub, +monkeypatch or installation substitutes for this missing behavior. + +The complete selected family includes production implementations and test +subclasses/fakes/consumers. The existing startup PreparedCatalog fake now inherits +the nominal service and shared ABC recipe rather than copying a one-view recipe; +startup expectations observe both views. The first expanded AST attempt03 +also matched a historical pytest .txt transcript: its non-Python parse error is +retained, not erased. Subsequent source enumeration explicitly selects .py files; +the transcript is evidence, not a source module. + +Owned receiving-control scratch is recorded under the original receipt root's +scratch/catalog511-controls06, 07, 08 and 09 (pytest/data/cache), with the earlier +attempt scratch retained alongside it. No uncertain request or science evidence +is disposable. All bounded source-control processes have known terminal exits; +there is no retained engineering native/server/viewer handle. diff --git a/tests/unit/test_execution_server_startup_preparation.py b/tests/unit/test_execution_server_startup_preparation.py index 6011773fdb..80a65a2b92 100644 --- a/tests/unit/test_execution_server_startup_preparation.py +++ b/tests/unit/test_execution_server_startup_preparation.py @@ -10,6 +10,7 @@ from openhcs.agent.dto.execution_connection import ExecutionConnectionSpec from openhcs.agent.dto.functions import FunctionCatalogPreparationOutcome +from openhcs.agent.services.function_catalog_service import FunctionCatalogService from openhcs.core.function_contract_metadata import FunctionContractAttribute from openhcs.core.processing_preparation import PreparationCacheBatch, PreparationOperation from openhcs.processing.backends.lib_registry.registry_service import RegistryService @@ -17,8 +18,9 @@ from openhcs.runtime.zmq_execution_server import ZMQExecutionServer -class PreparedCatalog: +class PreparedCatalog(FunctionCatalogService): def __init__(self, events): + super().__init__() self.events = events def prepare(self, **kwargs): @@ -27,9 +29,6 @@ def prepare(self, **kwargs): def catalog(self, **kwargs): self.events.append("catalog") - def prepare_projections(self, **kwargs): - self.catalog(**kwargs) - def projections_current(self): return "catalog" in self.events @@ -57,12 +56,12 @@ def warm(*, status_callback): future = preparation.ensure_started() assert future.done() and future.exception() is None assert preparation._thread is None - assert events == ["warm", "catalog", "bind"] + assert events == ["warm", "catalog", "catalog", "bind"] state = preparation.start(ExecutionConnectionSpec(port=22319)) assert state.outcome is FunctionCatalogPreparationOutcome.READY assert preparation.ensure_started() is future server.prepare_runtime_capabilities() - assert events == ["warm", "catalog", "bind"] + assert events == ["warm", "catalog", "catalog", "bind"] def test_startup_warm_failure_cannot_bind_or_report_catalogue_ready(monkeypatch): @@ -179,7 +178,9 @@ def parent_hook(): else: server.start() server.prepare_runtime_capabilities() - assert events == ["cache work", "unselected parent hook", "catalog", "bind"] + assert events == [ + "cache work", "unselected parent hook", "catalog", "catalog", "bind" + ] assert preparation._future.done() and preparation._future.exception() is None finally: PreparationOperation.reset() From cb263946e92929c2476f1125517964713ac931ae Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sat, 3 Oct 2026 01:23:45 -0400 Subject: [PATCH 5/6] Record complete catalog family and main514 receiving dependency custody --- .../catalog-current-readiness-20261003.rst | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/docs/validation/catalog-current-readiness-20261003.rst b/docs/validation/catalog-current-readiness-20261003.rst index ffec222a72..5def7e7c83 100644 --- a/docs/validation/catalog-current-readiness-20261003.rst +++ b/docs/validation/catalog-current-readiness-20261003.rst @@ -197,8 +197,31 @@ also matched a historical pytest .txt transcript: its non-Python parse error is retained, not erased. Subsequent source enumeration explicitly selects .py files; the transcript is evidence, not a source module. +Expanded AST attempt04 also remains retained: a Git pathspec union accidentally +selected an unpackaged benchmark script, causing a missing installed-comparison +file error. Corrected attempt05 restricts the additional family to Python files +under tests: 28 production modules/5291 sites, 27 test modules/3561 sites and +8 original receiving-dependency modules, zero parse errors. Source identity is +5dc21d5720baca32f8170f89b2359c8b8408b4ba; receipt SHA256 +f63738f8a40989b50f133b4db8a220f6da54f088c5c9d72f6434b7f642fe5645. +The selected production family has exactly one self._state_lock declaration and +one prepare_projections recipe, both on FunctionCatalogServiceABC. This is +focused static source evidence, not global/dynamic or installed proof. + +Two targeted startup-fixture checks were already submitted before the parent's +instruction to resolve main514 origins before more checks; attempt10 completed +2 passed/1.80s, unit5.103s/peak139.8MiB/Swap0/exit0. It verifies the final nominal +fixture consumes the shared two-view recipe; it does not supersede attempt09's +failed real-registry startup control. No more tests/builds were submitted after +that instruction. Main514/issue513 and the determining upstream PR bodies were +read completely: their named owner is Tristan/trissim, with recorded21-pass/ +5-skip integration controls and R0/R1 acceptance, but no published exact qualified +dependency-origin receipt path. Parent was asked for the original tested artifact +or source snapshot through its ordinary dependency mechanism. Current foreign +dependency worktrees and their missing-object boundary were left unchanged. + Owned receiving-control scratch is recorded under the original receipt root's -scratch/catalog511-controls06, 07, 08 and 09 (pytest/data/cache), with the earlier +scratch/catalog511-controls06, 07, 08, 09 and 10 (pytest/data/cache), with the earlier attempt scratch retained alongside it. No uncertain request or science evidence is disposable. All bounded source-control processes have known terminal exits; there is no retained engineering native/server/viewer handle. From b1d0b6c3e52842604a83bd3fda85c415e7fa8c5d Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Sat, 3 Oct 2026 03:42:34 -0400 Subject: [PATCH 6/6] Scope development conclusions and verify custom API declarations --- .../analysis-claim-scope-guide-20261003.rst | 92 +++++++++++++++++++ .../references/analysis-strategy.md | 26 ++++++ .../references/custom-function-authoring.md | 12 +++ .../use-openhcs/references/viewer-qa.md | 7 +- 4 files changed, 135 insertions(+), 2 deletions(-) create mode 100644 docs/validation/analysis-claim-scope-guide-20261003.rst diff --git a/docs/validation/analysis-claim-scope-guide-20261003.rst b/docs/validation/analysis-claim-scope-guide-20261003.rst new file mode 100644 index 0000000000..0620a65f2a --- /dev/null +++ b/docs/validation/analysis-claim-scope-guide-20261003.rst @@ -0,0 +1,92 @@ +Claim-scoped development guidance: source-only checkpoint +======================================================== + +Scope and original owners +------------------------- + +This is a generic how-to correction, not a biological acceptance, a new +evaluation rule, an assay recipe, or a hot update to an active scientist. +Previously frozen outcomes and the qualified catalog511-package01 target remain +unchanged. No benchmark answers, object counts, tuned parameters or trial +transcripts are included in the guide changes. + +Current main inspected: f7de9efad393bce83a3756a271f0cd4383783ea9. PR472 current +head 6aad9f1818f464ac6b8eea24bbd0a1569f559131 owns historical pinned394/404 wheel +receiving, not these guides. Singer confirmed no analysis/custom guide claim; +Planck confirmed no current guide claim and no borrower of this checkout. +Their current522/494 source families do not edit the three changed references. +Current open PR heads and the nearest related owners' claims were checked; +no independent implementation or guide owner was inferred from an old PR title. + +The original ``mcp_knowledge_base_manifest.json`` projects these existing sources: + +* ``openhcs_autonomous_analysis_strategy``: packaged ``analysis-strategy.md``; +* ``openhcs_viewer_qa``: packaged ``viewer-qa.md``; +* ``openhcs_custom_function_workflow``: packaged ``custom-function-authoring.md``. + +``ImageAnalysisWorkflowAuthoringContext`` and +``CustomFunctionAuthoringContext`` already route to those document identities. +The plugin's original ``AgentSkillBundle`` includes the same reference files; +no manifest entry, generated duplicate, skill mirror or new guide is required. + +The canonical reStructuredText entry/reference owners are +``docs/source/guide_for_biologists/domain_expert_onboarding.rst``, +``docs/source/user_guide/custom_functions.rst``, and +``docs/source/development/callable_artifact_authoring.rst``. They retain their +existing scope and are not copied or rewritten here. Live QA gate declarations +remain owned by ``ImageAnalysisQaPolicy``; no runtime policy/codec/catalogue or +enum definition changes. + +Gap and correction +------------------ + +Existing guidance already requires earliest-stage diagnosis, foreground before +unclumping, ambiguity triage, one semantic change, regression witnesses and +continued development within the authorised resource budget. Those procedures +are not duplicated. The missing instruction was how to scope conclusions when +only some objects or relationships remain unresolved. + +The strategy now preserves independently supported claims, records unresolved +cases and exclusions, permits justified bounds/sensitivity analysis, and limits +whole-population claims when the unresolved cases affect them. It neither +certifies completeness from sampled positives nor treats partial support as a +reason to stop repairing a clear failure. Viewer QA links to that single owner +and ties candidate rejection to the requested claim's acceptance criteria. + +Custom authoring now requires evidence for actual imports, decorators, helper +types and enum members in the installed version before source submission. +Existing public architecture inspection accepts curated symbol IDs, not arbitrary +import paths. ``InternalApiSymbol`` returns signature/doc/source-location facts, +not complete enum members or source text. The guide does not promise that this +route alone proves a member, invent a reflection endpoint, or copy an enum list. +An unavailable declaration is an exact authoring-contract gap, not evidence that +unrelated scientific outputs are invalid. + +Acceptance boundary and preservation +----------------------------------- + +Complete use-openhcs and Diataxis skills were read. The strategy, viewer QA, +custom authoring and analysis-learning references and relevant original context, +knowledge manifest, skill bundle and architecture projection relationships were +read. Diataxis kept the change an action-oriented how-to, with facts linked to +their existing reference owners rather than duplicated API catalogues. + +Normal current-main integration preceded the three reference edits. It preserves +merged520 resource guidance and merged503 source; all foreign submodule worktrees +and gitlinks are untouched. The three catalog511 production files remain +byte-identical to qualified cb263946e92929c2476f1125517964713ac931ae. + +Source review and ``git diff --check`` pass. Original knowledge routes still name +the same canonical paths and relative links resolve to existing references. +No imports, tests, builds, installs, native/MCP/client/provider launches or +scientific contact were used. This is source availability/reasoning evidence, +not installed retrieval acceptance or an autonomous biological improvement. +Future package/retrieval and fresh blind performance evaluation remain separate; +the previously qualified target still contains the previous guide bytes. + +Read-only lookup negatives were retained in the tool history: an initial +``gh`` request inferred the wrong repository and returned no PR472; explicit +``-R OpenHCSDev/openhcs`` resolved the actual PR and current open heads. Guessed +``skill_manifest.py``/``architecture_service.py`` names were absent; actual +``skill_bundle.py``/``architecture_projection_service.py`` owners were read. +These were source lookup failures, not scientific attempts or runtime operations. diff --git a/packaging/codex/openhcs/skills/use-openhcs/references/analysis-strategy.md b/packaging/codex/openhcs/skills/use-openhcs/references/analysis-strategy.md index a54dcc2f8b..bd1478ee15 100644 --- a/packaging/codex/openhcs/skills/use-openhcs/references/analysis-strategy.md +++ b/packaging/codex/openhcs/skills/use-openhcs/references/analysis-strategy.md @@ -78,6 +78,32 @@ missing biological information, a missing tool contract, new authority or unsafe resource use, report the exact boundary and retain the best candidate with its known failures. Do not force unsupported structures into a mask to finish. +## Scope conclusions to the evidence + +Judge each requested claim at its declared object, relationship and spatial +scope. Keep supported findings, clear failures and ambiguous cases distinct, +with their witness and persisted identities. Uncertain body boundaries or path +ownership do not automatically invalidate independently supported centres or +path geometry; those findings do not establish complete bodies or correct +body-to-path associations either. Technical completion remains separate from +these biological judgements. + +Report reviewed coverage, inclusion rules, exclusions with their denominator, +and unresolved cases alongside the supported result. When ambiguity affects a +total, retain a justified lower/upper bound or sensitivity analysis if the evidence permits; +do not invent a confidence interval or silently drop uncertain objects. A few +accepted witnesses do not establish whole-field completeness. Withhold a +whole-population claim when unresolved cases invalidate it, not every unrelated +finding merely because one claim remains uncertain. + +Partial support is not a stopping rule for a clear failure. Follow the existing +[earliest-stage diagnosis](segmentation-diagnostics.md), make one discriminating +repair while the authorised budget permits, and revisit distributed regression +controls. Improved downstream paths cannot repair an unchanged failed body +stage; diagnose that stage rather than repeatedly tuning faint-signal thresholds. +Preserve frozen evaluations unchanged; this reporting rule applies to newly +authorised development, not retroactive acceptance of an earlier attempt. + ## Development corrections and autonomous evaluation Declare the run's purpose before starting. Development aims to reach an diff --git a/packaging/codex/openhcs/skills/use-openhcs/references/custom-function-authoring.md b/packaging/codex/openhcs/skills/use-openhcs/references/custom-function-authoring.md index e1549e2965..4fc8c68394 100644 --- a/packaging/codex/openhcs/skills/use-openhcs/references/custom-function-authoring.md +++ b/packaging/codex/openhcs/skills/use-openhcs/references/custom-function-authoring.md @@ -16,6 +16,18 @@ backend, outputs and empty-input behaviour before writing source. For centre detection, specify what defines a centre, coordinate order/origin, label identity and whether a count covers one plane or the whole volume. +Before writing source, verify the intended imports, decorators, helper types +and enum members against the actual public declarations for the installed +version. Use the exposed reflected schemas and source-backed knowledge; inspect +curated architecture symbols only when that capability is exposed. A symbol's +signature or source location alone does not prove an enum member or helper API. +Retain the declaration/source identity and retrieve untruncated relevant content. +If the required declaration is not available through the authorised routes, +use a verified contract-compatible alternative if available; otherwise record +that exact contract gap. Do not invent a member or submit source as a discovery +probe. Keep definitions with their existing owners, not a copied enum catalogue +in the callable or this guide. + Implement only that operation. Keep channel selection, filename interpretation, grouping and output destinations in the pipeline declarations. Do not read scientific files or close over live GUI/viewer state inside the callable. diff --git a/packaging/codex/openhcs/skills/use-openhcs/references/viewer-qa.md b/packaging/codex/openhcs/skills/use-openhcs/references/viewer-qa.md index 76650efe27..79aceaaa84 100644 --- a/packaging/codex/openhcs/skills/use-openhcs/references/viewer-qa.md +++ b/packaging/codex/openhcs/skills/use-openhcs/references/viewer-qa.md @@ -198,10 +198,13 @@ inspect body-channel support and growth beyond each object's own primary seed; matching counts or retained seed IDs do not prove cell bodies. For neurites, inspect faint supported soma-to-process continuity, endpoints, crossings, branches and background bridges. Reject the candidate for clear supported -misses, erased paths or induced background bridges/artifacts; saturation of +misses, erased paths or induced background bridges/artifacts that violate the +requested claim's acceptance criteria; saturation of bright somas alone is not a diagnostic or segmentation rejection gate. Triage ambiguous debris separately so it does not prevent review of clear supported -misses. +misses. Apply [claim-scoped conclusions](analysis-strategy.md#scope-conclusions-to-the-evidence) +to retain supported findings and identify which objects or relationships remain +uncertain, rather than converting every result into a blanket abstention. Use [segmentation diagnostics](segmentation-diagnostics.md) for the earliest failed stage and [preprocessing](image-preprocessing.md) for its nuisance model.