From 5d0c22329fd5f6062303fe9dc87d619c889799e5 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 08:35:34 -0400 Subject: [PATCH 1/5] R1d receipt: cold catalog preparation, measured on main Co-Authored-By: Claude Opus 5.5 --- .../R1d-catalog-prepare-cold-start.md | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md diff --git a/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md b/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md new file mode 100644 index 000000000..f78e346f8 --- /dev/null +++ b/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md @@ -0,0 +1,75 @@ +# R1d: cold start of function-catalog preparation, with compilation kept in preparation + +Branch `refactor/R1d-catalog-prepare-cold-start`. Replaces the closed PR #1222 (R1c). + +## Owner decision + +Every Numba kernel is compiled during catalog preparation +(`RegistryService.prepare_in_current_process` and its forked cache children), never at +first use. R1c deferred compilation and its own measurements showed first pipeline runs +18 to 33 s slower; that is rejected. This surface makes cold preparation faster while it +still compiles the whole catalog before readiness. + +## What is wrong (measured on main `ac19abb49`, Linux, Ryzen 5 5500, 6 cores / 12 threads) + +Probe scripts are scratch (`.claude/prof/`, not committed). Cold means an empty +`NUMBA_CACHE_DIR`, an empty catalog cache (`XDG_DATA_HOME`) and `OPENHCS_CPU_ONLY=1` +(the mode in which cache children fork). + +| Phase | Wall | +| --- | --- | +| import of the registry service | 0.9 s | +| discovery | 18.6 s | +| kernel preparation in cache children | 23.1 s | +| kernel preparation left to the server process, serial | 9.5 s | +| signatures, configuration declarations | 0.8 s | +| total | 52.9 s | + +1. **Discovery, 18.6 s.** Wall-clock attribution (no profiler): + - `SkimageRegistry` 10.4 s, of which 10.2 s is `classify_function_behavior`: 164 + functions are each run on a 3x20x20 stack and a 20x20 plane to learn whether they + accept a stack, a plane, and the rank they return. Three calls are 9.6 s of it: + `rolling_ball` 5.1 s, `denoise_nl_means` 2.3 s, `hough_ellipse` 2.3 s. The other + 161 take 0.5 s together. + - `OpenHCSRegistry` 7.3 s, of which 4.5 s is `_openhcs_requirements` re-parsing the + installed metadata: 63 138 `packaging.Requirement` parses over 619 calls. +2. **Cache children, 23.1 s.** 31 children, one operation each, at most 12 at a time; + mean concurrency 9.0 of 12; 198 CPU-seconds, 195 of them Numba compilation. The + longest child is 19.8 s (`ShapeMeasurementBackendStrategy`), so the window is within + 3 s of that floor. 1091 compilations of 561 distinct kernel signatures: 56.5 s is + the same kernel compiled by more than one child at once (9 children each compile + `DenseArrayObjectLabelStorageStrategy.prepare_coordinates`, about 30 s of it). +3. **Serial tail, 9.5 s.** After the children the server compiles 86 more kernels on one + CPU: 75 in 26 callable hooks declared with `@processing_prepare` (8.1 s) and 11 in two + registry families that never declared persistent kernel work + (`EdgeEnhancementStrategy`, `ImagePayloadUint8Strategy`; 0.8 s). A cache child only + ever ran registry-family operations; a callable hook had no way into one. +4. **CI.** No workflow persists `NUMBA_CACHE_DIR`; every job compiles from nothing. + +## Target + +- Installed metadata parsed once per process. +- The behaviour probe stays the single source of each skimage contract (no table of + answers), on the smallest arrays that give the same answers. +- Every preparation operation that compiles kernels runs in a cache child first; the + server process loads what they cached. +- CI restores the Numba cache between runs. + +## Guards + +Written below under "Done" as they land. + +## Done when + +Cold preparation is measurably faster per phase, a second start compiles nothing, first +and second `run-well-throughput` runs are not slower than main, parity is 30/30. + +## Progress notes (for a resumed run) + +- [x] baseline measured +- [ ] requirement parse once +- [ ] probe size +- [ ] hooks in cache children +- [ ] families that compile but never declared it +- [ ] CI cache +- [ ] gates From 735c9c45d8857b54ff016663caee07080ee04ba9 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 08:46:13 -0400 Subject: [PATCH 2/5] WIP R1d: parse requirements once, smaller behaviour probe, hooks and two families in cache children Co-Authored-By: Claude Opus 5.5 --- openhcs/core/image_file_serialization.py | 3 +- openhcs/core/processing_preparation.py | 81 ++++++++++++++----- .../processing/backends/cellprofiler/edge.py | 4 +- .../backends/lib_registry/openhcs_registry.py | 20 ++++- .../backends/lib_registry/unified_registry.py | 10 ++- 5 files changed, 91 insertions(+), 27 deletions(-) diff --git a/openhcs/core/image_file_serialization.py b/openhcs/core/image_file_serialization.py index 3e97cf52c..1b7d4186f 100644 --- a/openhcs/core/image_file_serialization.py +++ b/openhcs/core/image_file_serialization.py @@ -21,6 +21,7 @@ TiffResultFile, ) from openhcs.core.callable_contract import CompilerPreparedAutoRegisterFamily +from openhcs.core.processing_preparation import PersistentNumbaKernelPreparation from openhcs.core.image_quantization_numba import quantize_image_uint8 from metaclass_registry.strategies import NominalTypeStrategyFamilyMixin from openhcs.core.runtime_image_values import ( @@ -680,7 +681,7 @@ def preserves_pixel_values(self, source_dtype: Any) -> bool: class ImagePayloadUint8Strategy( NominalTypeStrategyFamilyMixin, - CompilerPreparedAutoRegisterFamily, + PersistentNumbaKernelPreparation, metaclass=AutoRegisterMeta, ): """Nominal family for dtype-specific uint8 image conversion.""" diff --git a/openhcs/core/processing_preparation.py b/openhcs/core/processing_preparation.py index 6c093f9d2..ed8f8f646 100644 --- a/openhcs/core/processing_preparation.py +++ b/openhcs/core/processing_preparation.py @@ -11,7 +11,7 @@ from collections.abc import Callable, Hashable, Iterable, Iterator from contextlib import ExitStack from dataclasses import dataclass, field -from itertools import islice +from itertools import chain, islice from multiprocessing.connection import Connection, wait from multiprocessing.process import BaseProcess from threading import Lock @@ -166,6 +166,10 @@ class DeclaredHookPreparation(PreparationOperation): def execute(self) -> None: self.hook() + def can_prepare_in_child(self) -> bool: + """A hook's kernels reach the server through the persistent Numba cache.""" + return PersistentNumbaKernelPreparation.can_prepare_in_child() + @dataclass(frozen=True, slots=True) class ModuleHookPreparation(DeclaredHookPreparation): @@ -256,6 +260,14 @@ def cache_sources(self) -> tuple[PreparationOperation, ...]: return () return (ModuleRegistryPreparation(module_name),) + def hooks(self) -> tuple[DeclaredHookPreparation, ...]: + """The declared hooks, which load the kernels the registries cached.""" + return tuple( + operation + for operation in self.operations() + if isinstance(operation, DeclaredHookPreparation) + ) + def _execute_cache_preparation( operations: tuple[PreparationOperation, ...], result_connection: Connection @@ -352,20 +364,34 @@ def _schedulable_cpu_count() -> int: @dataclass(frozen=True, slots=True) class PreparationCacheBatch: - """Schedule derived cache operations without interpreting backend families.""" + """Schedule derived cache operations without interpreting backend families. + + Stages run in order and each finishes before the next starts, so a later + stage loads from the persistent cache what an earlier one compiled. + """ preparations: tuple[PreparationOperation, ...] + hooks: tuple[PreparationOperation, ...] = () @classmethod def from_callables( cls, callables: Iterable[Callable[..., object]] ) -> PreparationCacheBatch: + """Registry kernels first, then the hooks that call them.""" + preparations = tuple( + CallablePreparation.from_callable(func) for func in callables + ) sources = { operation.identity: operation - for func in callables - for operation in CallablePreparation.from_callable(func).cache_sources() + for preparation in preparations + for operation in preparation.cache_sources() + } + hooks = { + operation.hook: operation + for preparation in preparations + for operation in preparation.hooks() } - return cls(tuple(sources.values())) + return cls(tuple(sources.values()), tuple(hooks.values())) def populate_child_caches( self, @@ -373,29 +399,48 @@ def populate_child_caches( max_workers: int = 1, status_callback: Callable[[str], None] | None = None, ) -> None: - """Isolate admitted cache work, never parent process-local readiness.""" + """Isolate declared cache work, never parent process-local readiness.""" if max_workers < 1: raise ValueError("Preparation worker budget must be positive.") if "fork" not in multiprocessing.get_all_start_methods(): return worker_capacity = min(max_workers, _schedulable_cpu_count()) - operations = { - operation.identity: operation - for preparation in self.preparations - for operation in preparation.cache_operations() - } - children = tuple( + context = multiprocessing.get_context("fork") + registries = tuple( operation - for operation in operations.values() + for operation in { + operation.identity: operation + for preparation in self.preparations + for operation in preparation.cache_operations() + }.values() if operation.can_prepare_in_child() ) - if not children: - return - context = multiprocessing.get_context("fork") + hooks = tuple( + operation for operation in self.hooks if operation.can_prepare_in_child() + ) + # A registry operation compiles for seconds, so each gets a child. A + # hook mostly loads what the registries cached, and a child costs about + # a second before it does anything: hooks share one child per CPU. + for tasks in ( + tuple((operation,) for operation in registries), + tuple(hooks[index::worker_capacity] for index in range(worker_capacity)), + ): + self._run_children( + context, tuple(filter(None, tasks)), worker_capacity, status_callback + ) + + @staticmethod + def _run_children( + context, + tasks: tuple[tuple[PreparationOperation, ...], ...], + worker_capacity: int, + status_callback: Callable[[str], None] | None, + ) -> None: + """Run each task in its own child, or all in one child on a single CPU.""" pending = ( - iter((children,)) + iter(filter(None, (tuple(chain.from_iterable(tasks)),))) if worker_capacity == 1 - else iter((operation,) for operation in children) + else iter(tasks) ) with ExitStack() as resources: workers: list[PreparationCacheWorker] = [] diff --git a/openhcs/processing/backends/cellprofiler/edge.py b/openhcs/processing/backends/cellprofiler/edge.py index 22f279d83..fc9705fc1 100644 --- a/openhcs/processing/backends/cellprofiler/edge.py +++ b/openhcs/processing/backends/cellprofiler/edge.py @@ -40,7 +40,7 @@ class EnhanceEdgesModule(CellProfilerModule): from abc import ABC, abstractmethod -from openhcs.core.callable_contract import CompilerPreparedAutoRegisterFamily +from openhcs.core.processing_preparation import PersistentNumbaKernelPreparation from dataclasses import dataclass, replace from enum import Enum from typing import ClassVar @@ -182,7 +182,7 @@ def fallback_strategy_key(self) -> EdgeEnhancementStrategyKey: class EdgeEnhancementStrategy( - CompilerPreparedAutoRegisterFamily, ABC, metaclass=AutoRegisterMeta + PersistentNumbaKernelPreparation, ABC, metaclass=AutoRegisterMeta ): """Nominal dispatch point for one backend/method/direction edge algorithm.""" diff --git a/openhcs/processing/backends/lib_registry/openhcs_registry.py b/openhcs/processing/backends/lib_registry/openhcs_registry.py index 74317254d..1d0edbfc5 100644 --- a/openhcs/processing/backends/lib_registry/openhcs_registry.py +++ b/openhcs/processing/backends/lib_registry/openhcs_registry.py @@ -145,7 +145,7 @@ def eligible(module_name: str) -> bool: source = spec.loader.get_source(module_name) if source is None: return True - module_ast = ast.parse(source, filename=spec.origin or module_name) + module_ast = _parsed_source(source, spec.origin or module_name) selection = _CatalogDeclarationSourceSelection( missing, attributes, normalize ) @@ -296,6 +296,13 @@ def _catalog_memory_types(func: Callable) -> frozenset[MemoryType] | None: return declared +@lru_cache(maxsize=1024) +def _parsed_source(source: str, origin: str) -> ast.Module: + """Parse one module source once for every question discovery asks of it.""" + + return ast.parse(source, filename=origin) + + def _module_source(module_name: str) -> tuple[str, str] | None: """Return loader-decoded module source and origin, or None without source.""" @@ -324,8 +331,13 @@ def _module_declares_allowed_memory_type( _REQUIRED_DISTRIBUTIONS_NAME = "REQUIRED_DISTRIBUTIONS" +@lru_cache(maxsize=1) def _openhcs_requirements() -> tuple[Requirement, ...]: - """Return the installed OpenHCS distribution's declared requirements.""" + """Return the installed OpenHCS distribution's declared requirements. + + Installed metadata is fixed for the life of the process; parsing it once + serves every module the catalog asks about. + """ return tuple( Requirement(value) for value in importlib.metadata.requires("openhcs") or () @@ -368,7 +380,7 @@ def _distribution_excluded_by_package_metadata(name: str) -> bool: def _source_required_distributions(source: str, origin: str) -> tuple[str, ...]: """Read a module's literal ``REQUIRED_DISTRIBUTIONS`` declaration.""" - for statement in ast.parse(source, filename=origin).body: + for statement in _parsed_source(source, origin).body: if isinstance(statement, ast.Assign): targets = statement.targets elif isinstance(statement, ast.AnnAssign): @@ -456,7 +468,7 @@ def _source_declares_allowed_memory_type( """Inspect loader-decoded source; content identity invalidates edited modules.""" try: - module_ast = ast.parse(source, filename=origin) + module_ast = _parsed_source(source, origin) except SyntaxError: return True import_bindings = _module_import_bindings(module_ast) diff --git a/openhcs/processing/backends/lib_registry/unified_registry.py b/openhcs/processing/backends/lib_registry/unified_registry.py index 0e5efdf37..5a49fd60e 100644 --- a/openhcs/processing/backends/lib_registry/unified_registry.py +++ b/openhcs/processing/backends/lib_registry/unified_registry.py @@ -935,12 +935,18 @@ def probe_spatial_rank() -> int: return AxisFamily.active().payload_spatial_rank + # A probe call costs what the library's default parameters make of the + # array (skimage's rolling_ball builds a radius-100 ball over every pixel), + # so the arrays are as small as still classifies every function the same. + PROBE_STACK_DEPTH = 3 + PROBE_PLANE_EDGE = 8 + def create_test_arrays(self) -> Tuple[Any, Any]: """Return (stack probe, plane probe) at the active family's spatial rank.""" - plane_shape = (20,) * self.probe_spatial_rank() + plane_shape = (self.PROBE_PLANE_EDGE,) * self.probe_spatial_rank() dtype = self._get_float_dtype() return ( - self._create_array((3, *plane_shape), dtype), + self._create_array((self.PROBE_STACK_DEPTH, *plane_shape), dtype), self._create_array(plane_shape, dtype), ) From e6b6d305e96a1921e9965d6dee94d9cbeb8d67d5 Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 08:54:11 -0400 Subject: [PATCH 3/5] WIP R1d: cache children fork wherever Numba persists; hooks grouped one child per CPU; guards Co-Authored-By: Claude Opus 5.5 --- openhcs/core/processing_preparation.py | 47 +++++++++++++------ ...ellprofiler_callable_kernel_preparation.py | 43 ++++++++++++++++- tests/unit/test_fork_compiler_preparation.py | 3 +- tests/unit/test_library_registry_discovery.py | 21 +++++++++ 4 files changed, 97 insertions(+), 17 deletions(-) diff --git a/openhcs/core/processing_preparation.py b/openhcs/core/processing_preparation.py index ed8f8f646..eb92e465b 100644 --- a/openhcs/core/processing_preparation.py +++ b/openhcs/core/processing_preparation.py @@ -37,13 +37,16 @@ def requires_persistent_kernel_cache(cls) -> bool: @classmethod def can_prepare_in_child(cls) -> bool: - if not OpenHCSProcessEnvironment.cpu_only_mode(): - return False - if not cls.requires_persistent_kernel_cache(): - return False - from numba import config as numba_config + """A forked child compiles for the server wherever Numba persists. - return bool(numba_config.CACHE_DIR) + Numba always has a cache for a ``cache=True`` kernel: ``NUMBA_CACHE_DIR``, + else ``__pycache__`` beside the source, else its per-user directory. It + refuses to define the kernel otherwise. + """ + return ( + OpenHCSProcessEnvironment.cpu_only_mode() + and cls.requires_persistent_kernel_cache() + ) class PreparationOperation(ABC): @@ -166,6 +169,13 @@ class DeclaredHookPreparation(PreparationOperation): def execute(self) -> None: self.hook() + def cache_operations(self) -> tuple[PreparationOperation, ...]: + """A hook bound to a declared operation is that operation's cache work.""" + operation = getattr(self.hook, "__self__", None) + if isinstance(operation, PreparationOperation): + return operation.cache_operations() + return (self,) + def can_prepare_in_child(self) -> bool: """A hook's kernels reach the server through the persistent Numba cache.""" return PersistentNumbaKernelPreparation.can_prepare_in_child() @@ -406,17 +416,24 @@ def populate_child_caches( return worker_capacity = min(max_workers, _schedulable_cpu_count()) context = multiprocessing.get_context("fork") - registries = tuple( + registry_operations = { + operation.identity: operation + for preparation in self.preparations + for operation in preparation.cache_operations() + } + hook_operations = dict.fromkeys( operation - for operation in { - operation.identity: operation - for preparation in self.preparations - for operation in preparation.cache_operations() - }.values() - if operation.can_prepare_in_child() + for hook in self.hooks + for operation in hook.cache_operations() + if operation.identity not in registry_operations ) - hooks = tuple( - operation for operation in self.hooks if operation.can_prepare_in_child() + registries, hooks = ( + tuple( + operation + for operation in operations + if operation.can_prepare_in_child() + ) + for operations in (registry_operations.values(), hook_operations) ) # A registry operation compiles for seconds, so each gets a child. A # hook mostly loads what the registries cached, and a child costs about diff --git a/tests/unit/test_cellprofiler_callable_kernel_preparation.py b/tests/unit/test_cellprofiler_callable_kernel_preparation.py index bd8df949e..2460fe474 100644 --- a/tests/unit/test_cellprofiler_callable_kernel_preparation.py +++ b/tests/unit/test_cellprofiler_callable_kernel_preparation.py @@ -137,8 +137,9 @@ def test_declared_cache_admission_requires_cpu_and_explicit_cache( index.touch() assert GridKernelPreparation.can_prepare_in_child() index.unlink() + # Without a configured directory Numba caches beside the source. monkeypatch.setattr(numba_config, "CACHE_DIR", "") - assert not GridKernelPreparation.can_prepare_in_child() + assert GridKernelPreparation.can_prepare_in_child() monkeypatch.setattr(numba_config, "CACHE_DIR", str(tmp_path)) monkeypatch.setenv("OPENHCS_CPU_ONLY", "false") assert not GridKernelPreparation.can_prepare_in_child() @@ -367,3 +368,43 @@ def execute(self): (output / 'second').write_text(str(os.getpid())) assert all( int((tmp_path / name).read_text()) in owned for name in ("first", "second") ) + + +@pytest.mark.skipif( + "fork" not in multiprocessing.get_all_start_methods(), + reason="cache children fork", +) +def test_declared_hooks_run_in_cache_children_before_the_server_runs_them( + monkeypatch, kernel_module, tmp_path +): + """A hook's kernels are compiled by a child; the server only loads them.""" + monkeypatch.setenv("OPENHCS_CPU_ONLY", "true") + monkeypatch.setattr(numba_config, "CACHE_DIR", str(tmp_path / "cache")) + + def record(name): + def hook(): + with (tmp_path / name).open("a") as stream: + stream.write(f"{os.getpid()}\n") + + return hook + + kernel_module.__openhcs_prepare__ = record("module") + shared = record("shared") + callables = ( + declare_callable(kernel_module, shared), + declare_callable(kernel_module, shared), + declare_callable(kernel_module, record("own")), + ) + PreparationCacheBatch.from_callables(callables).populate_child_caches( + max_workers=2 + ) + assert not PreparationOperation._completed + for name in ("module", "shared", "own"): + pids = (tmp_path / name).read_text().splitlines() + assert len(pids) == 1 and int(pids[0]) != os.getpid() + + monkeypatch.delenv("OPENHCS_CPU_ONLY") + PreparationCacheBatch.from_callables(callables).populate_child_caches( + max_workers=2 + ) + assert len((tmp_path / "own").read_text().splitlines()) == 1 diff --git a/tests/unit/test_fork_compiler_preparation.py b/tests/unit/test_fork_compiler_preparation.py index f641c558c..5fae9bd8e 100644 --- a/tests/unit/test_fork_compiler_preparation.py +++ b/tests/unit/test_fork_compiler_preparation.py @@ -99,8 +99,9 @@ def test_backend_child_preparation_requires_explicit_cpu_cache(monkeypatch, tmp_ index.touch() assert ObjectIntensityBackendStrategy.can_prepare_in_child() index.unlink() + # Without a configured directory Numba caches beside the source. monkeypatch.setattr(numba_config, "CACHE_DIR", "") - assert not ObjectIntensityBackendStrategy.can_prepare_in_child() + assert ObjectIntensityBackendStrategy.can_prepare_in_child() monkeypatch.setattr(numba_config, "CACHE_DIR", str(tmp_path)) monkeypatch.setenv("OPENHCS_CPU_ONLY", "false") assert not ObjectIntensityBackendStrategy.can_prepare_in_child() diff --git a/tests/unit/test_library_registry_discovery.py b/tests/unit/test_library_registry_discovery.py index 492a5ae1f..6eefb58c9 100644 --- a/tests/unit/test_library_registry_discovery.py +++ b/tests/unit/test_library_registry_discovery.py @@ -999,3 +999,24 @@ def pump_server(): # directly without paying that work during a pipeline or catalog request. assert result["catalog_request_seconds"] < 5.0 assert {"openhcs", "skimage"}.issubset(result["library_roots"]) + + +def test_installed_requirements_are_parsed_once_per_process(monkeypatch): + """Every module's distribution question reads one parse of the metadata.""" + from openhcs.processing.backends.lib_registry import openhcs_registry + + reads = [] + real_requires = openhcs_registry.importlib.metadata.requires + + def counted(name): + reads.append(name) + return real_requires(name) + + openhcs_registry._openhcs_requirements.cache_clear() + monkeypatch.setattr(openhcs_registry.importlib.metadata, "requires", counted) + try: + for _ in range(3): + openhcs_registry._catalog_policy_context() + assert reads == ["openhcs"] + finally: + openhcs_registry._openhcs_requirements.cache_clear() From 0e13c9646d3ab03bb711fece98f120a18036008d Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 09:05:29 -0400 Subject: [PATCH 4/5] R1d receipt: what was done, measurements, what was not done Co-Authored-By: Claude Opus 5.5 --- .../R1d-catalog-prepare-cold-start.md | 198 +++++++++++++----- 1 file changed, 141 insertions(+), 57 deletions(-) diff --git a/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md b/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md index f78e346f8..b244b2b0e 100644 --- a/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md +++ b/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md @@ -8,68 +8,152 @@ Every Numba kernel is compiled during catalog preparation (`RegistryService.prepare_in_current_process` and its forked cache children), never at first use. R1c deferred compilation and its own measurements showed first pipeline runs 18 to 33 s slower; that is rejected. This surface makes cold preparation faster while it -still compiles the whole catalog before readiness. +still compiles the whole catalog before readiness. Nothing here changes when a kernel is +compiled; the cache-worker machinery stays. ## What is wrong (measured on main `ac19abb49`, Linux, Ryzen 5 5500, 6 cores / 12 threads) Probe scripts are scratch (`.claude/prof/`, not committed). Cold means an empty -`NUMBA_CACHE_DIR`, an empty catalog cache (`XDG_DATA_HOME`) and `OPENHCS_CPU_ONLY=1` -(the mode in which cache children fork). - -| Phase | Wall | -| --- | --- | -| import of the registry service | 0.9 s | -| discovery | 18.6 s | -| kernel preparation in cache children | 23.1 s | -| kernel preparation left to the server process, serial | 9.5 s | -| signatures, configuration declarations | 0.8 s | -| total | 52.9 s | - -1. **Discovery, 18.6 s.** Wall-clock attribution (no profiler): - - `SkimageRegistry` 10.4 s, of which 10.2 s is `classify_function_behavior`: 164 - functions are each run on a 3x20x20 stack and a 20x20 plane to learn whether they - accept a stack, a plane, and the rank they return. Three calls are 9.6 s of it: - `rolling_ball` 5.1 s, `denoise_nl_means` 2.3 s, `hough_ellipse` 2.3 s. The other - 161 take 0.5 s together. - - `OpenHCSRegistry` 7.3 s, of which 4.5 s is `_openhcs_requirements` re-parsing the - installed metadata: 63 138 `packaging.Requirement` parses over 619 calls. -2. **Cache children, 23.1 s.** 31 children, one operation each, at most 12 at a time; - mean concurrency 9.0 of 12; 198 CPU-seconds, 195 of them Numba compilation. The - longest child is 19.8 s (`ShapeMeasurementBackendStrategy`), so the window is within - 3 s of that floor. 1091 compilations of 561 distinct kernel signatures: 56.5 s is - the same kernel compiled by more than one child at once (9 children each compile - `DenseArrayObjectLabelStorageStrategy.prepare_coordinates`, about 30 s of it). -3. **Serial tail, 9.5 s.** After the children the server compiles 86 more kernels on one - CPU: 75 in 26 callable hooks declared with `@processing_prepare` (8.1 s) and 11 in two - registry families that never declared persistent kernel work - (`EdgeEnhancementStrategy`, `ImagePayloadUint8Strategy`; 0.8 s). A cache child only - ever ran registry-family operations; a callable hook had no way into one. -4. **CI.** No workflow persists `NUMBA_CACHE_DIR`; every job compiles from nothing. - -## Target - -- Installed metadata parsed once per process. -- The behaviour probe stays the single source of each skimage contract (no table of - answers), on the smallest arrays that give the same answers. -- Every preparation operation that compiles kernels runs in a cache child first; the - server process loads what they cached. -- CI restores the Numba cache between runs. - -## Guards - -Written below under "Done" as they land. +`NUMBA_CACHE_DIR`, an empty catalog cache (`XDG_DATA_HOME`) and `OPENHCS_CPU_ONLY=1`. -## Done when +1. **No cache child forks unless `NUMBA_CACHE_DIR` is set.** + `PersistentNumbaKernelPreparation.can_prepare_in_child` required a non-empty + `numba.config.CACHE_DIR`. Only the desktop launcher scripts and the benchmark + harnesses set that variable; no CI workflow does, nor does a plain server start. In + that case the server compiled all 665 kernel signatures on one CPU: 94.5 s here, 0 + children (the 222 s on `macos-15-intel` is this path). Numba was caching all along: + it wrote 614 cache files into `__pycache__` beside the sources during that run. +2. **Discovery, 18.6 s.** Wall-clock attribution, no profiler: + - `SkimageRegistry` 10.4 s, of which 10.2 s is `classify_function_behavior`. skimage + declares no dimensionality, so each of 164 functions is run on a 3x20x20 stack and + a 20x20 plane to learn whether it accepts a stack, a plane, and the rank it + returns. Three calls are 9.6 s: `rolling_ball` 5.1 s, `denoise_nl_means` 2.3 s, + `hough_ellipse` 2.3 s. The other 161 take 0.5 s together. The cost comes from + default parameters applied per pixel (a radius-100 ball, 7x7x7 patches in a + 21-wide search, all pixel pairs), so it scales with the probe array. + - `OpenHCSRegistry` 7.3 s: `_openhcs_requirements` re-parsed the installed metadata + on each of 619 calls (63 138 `packaging.Requirement` parses), and each module + source was parsed to an AST about four times (529 parses for 141 modules). +3. **Cache children (with the directory set), 23.1 s.** 31 children, one registry + operation each, at most 12 at a time, mean concurrency 9.0 of 12; 198 CPU-seconds, + 195 of them Numba compilation. The longest child is 19.8 s, so the window is within + 3 s of its floor: the batch partitioning already saturates the machine. +4. **Serial tail, 9.5 s.** After the children the server compiled 86 more kernel + signatures on one CPU: 75 inside 26 callable hooks declared with + `@processing_prepare` (8.1 s) and 11 in two registry families that compile Numba + kernels but never declared it (`EdgeEnhancementStrategy`, + `ImagePayloadUint8Strategy`; 0.8 s). A cache child only ever ran registry-family + operations; a callable hook had no way into one. + +## Done + +- `can_prepare_in_child` no longer requires `NUMBA_CACHE_DIR`. Numba always persists a + `cache=True` kernel (the configured directory, else `__pycache__` beside the source, + else its per-user directory) and refuses to define the kernel otherwise, so the + conjunct restated a fact Numba owns, wrongly. +- `_openhcs_requirements` parses the installed metadata once per process (ported from + R1c; the only thing taken from that branch). `_parsed_source` parses each module + source once for the three questions discovery asks of it. +- The behaviour probe runs on a 3x8x8 stack and an 8x8 plane + (`RuntimeTestingRegistryBase.PROBE_STACK_DEPTH`, `PROBE_PLANE_EDGE`). The probe stays + the one source of each contract; no table of answers was added. Checked: all 164 + skimage functions classify identically at plane edges 20, 16, 12, 10, 8, 6 and 4 + (50 rejected, 74 flexible, 32 pure 2D, 2 pure 3D, 6 volumetric-to-slice), and with + different random arrays. A stack depth of 2 does change six `max_tree` functions, so + the depth stays 3. Editing `unified_registry.py` invalidates existing catalog caches + through the source-mtime signature. +- `PreparationCacheBatch` runs declared hooks in cache children after the registry + children finish, so a hook loads what the registries cached and compiles only what is + its own. Hooks share one child per CPU: a child costs about a second of CPU before it + does anything, and one child per hook (26 children) measured 14 child-seconds more for + no shorter a window. A hook bound to a declared operation + (`ObjectSizeShapeKernelPreparation().execute`) is that operation and is not run twice. +- `EdgeEnhancementStrategy` and `ImagePayloadUint8Strategy` inherit + `PersistentNumbaKernelPreparation`, so their kernels are compiled by children. +- The server process now compiles 0 kernels itself on a cold start (was 86, or 665 + without the directory). A second start compiles 0 in the server and 0 in all 45 + children. + +## Measurements + +Cold preparation, 12 threads, `NUMBA_CACHE_DIR` set: + +| Phase | main | this branch | +| --- | --- | --- | +| import of the registry service | 0.9 s | 0.6 s | +| discovery | 18.6 s | 7.8 s | +| kernel cache children | 23.1 s (31 children) | 27.7 s (33 registry + 12 hook children) | +| kernels compiled by the server, serial | 9.5 s (86 compilations) | 1.3 s (0 compilations) | +| signatures, configuration declarations | 0.8 s | 0.5 s | +| total | 52.9 s | 37.9 s | + +| Other cold cases | main | this branch | +| --- | --- | --- | +| `NUMBA_CACHE_DIR` unset (CI, plain server start) | 0 children; 94.5 s serial compile, 103.6 s total measured with this branch's discovery fixes and the old rule (R1c measured 123.8 s on main) | 37.6 s, 45 children | +| restricted to 4 physical cores (`taskset -c 0-3`) | 60.4 s | 49.5 s | +| second start, warm caches | 9.8 s | 7.7 s | + +`run-well-throughput`, ExampleHuman, `official30_portable_axis1.json`, wall time of the +whole command (server start, preparation, run, shutdown), caches emptied before the +first run: -Cold preparation is measurably faster per phase, a second start compiles nothing, first -and second `run-well-throughput` runs are not slower than main, parity is 30/30. +| Workers / wells | main first | main second | branch first | branch second | +| --- | --- | --- | --- | --- | +| 1 / 1 | 64.7 s | 22.1 s | 46.6 s | 19.0 s | +| 4 / 4 | 69.4 s | 22.4 s | 50.7 s | 20.0 s | -## Progress notes (for a resumed run) +Pipeline `total_seconds` inside those runs is unchanged (1.14 to 1.22 s with 1 worker, +1.52 to 1.66 s with 4): nothing is compiled during execution on either tree. + +## Guards and tests + +- `test_declared_hooks_run_in_cache_children_before_the_server_runs_them`: module and + callable hooks each run once in a child, a hook shared by two callables once, and + none without CPU-only mode. +- `test_new_declarations_populate_children_then_share_parent_hook_readiness` (existing): + a hook bound to a registered operation is not run a second time. +- `test_installed_requirements_are_parsed_once_per_process`. +- Two existing assertions changed with the rule they stated: with `CACHE_DIR` empty a + persistent family now can prepare in a child. + +## Behaviour changed (certificates) + +- The `NUMBA_CACHE_DIR` requirement for cache children: **derived**. Numba's cache + locator owns whether a kernel persists, and it always does for a kernel that could be + defined. What tests do not exercise: a source tree and home directory that are both + read-only make Numba raise when the kernel module is imported, before any child. +- Probe array size 20 to 8: **tests-only**. Checked on the installed scikit-image only; + CuPy and pyclesperanto (the other two runtime-tested registries) are not installed + here, so their classification at edge 8 is unverified. +- Hooks now also run in children. In fixture-capture mode + (`OPENHCS_CAPTURE_CELLPROFILER_FIXTURES_DIR`, a developer option) a hook that calls a + capturing function writes its warm-up arrays from the child as well as from the + server. `CellProfilerCallableKernelPreparation` keeps its own capture check. + +## Not done, and why + +- **Persisting the Numba cache in CI.** Numba validates each cache index against the + source file's `(mtime, size)` and absolute path (`numba/core/caching.py`). The jobs + that start a server install freshly built wheels, so every restored entry would be + stale. Making it work needs the installed sources' mtimes normalised in each job and + an exact key over all kernel sources (a prefix fallback would run stale machine code + after a cross-file kernel edit, which Numba does not track). That key misses on every + PR that touches `openhcs/**/*.py`. The CI gain in this PR comes from the first item + under "Done": those jobs never set the variable, so they compiled serially. +- **Compiling shared kernels once.** 56.5 of 195 compile seconds are the same kernel + compiled by several children at once (26 s is + `DenseArrayObjectLabelStorageStrategy.prepare_coordinates` in 10 children). Removing + it needs a declared order between operations; it moves that work in front of the same + children, so the longest chain is not shorter and the expected gain is under 2 s. +- **Children outside CPU-only mode.** Without `OPENHCS_CPU_ONLY` no child forks and the + server compiles serially (the 94.5 s path above). The rule predates this surface, + presumably because forking after CUDA initialisation is unsafe; not changed, and not + measured separately. +- **Discovery remainder (7.8 s)**: about 4 s importing 165 backend modules (under the + profiler) and 3 s of probes, of which `rolling_ball` is still 1.4 s. `matplotlib.font_manager` + was not attributable in wall time on Linux; macOS was not measured. +- Intel macOS was not re-measured; no debug workflow was added. + +## Done when -- [x] baseline measured -- [ ] requirement parse once -- [ ] probe size -- [ ] hooks in cache children -- [ ] families that compile but never declared it -- [ ] CI cache -- [ ] gates +Gates in the PR description. From 454e771d99031f972f138a0def4bdf5eb2c9dd9c Mon Sep 17 00:00:00 2001 From: Tristan Simas Date: Mon, 12 Oct 2026 10:37:12 -0400 Subject: [PATCH 5/5] R1d receipt: probe size verified on CuPy, cuCIM and pyclesperanto Co-Authored-By: Claude Opus 5.5 --- .../R1d-catalog-prepare-cold-start.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md b/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md index b244b2b0e..53004092b 100644 --- a/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md +++ b/docs/refactor/consolidation-20261009/R1d-catalog-prepare-cold-start.md @@ -157,3 +157,18 @@ Pipeline `total_seconds` inside those runs is unchanged (1.14 to 1.22 s with 1 w ## Done when Gates in the PR description. + +## Probe size on the GPU libraries (checked on the desktop, read-only, 2026-10-12) + +Run against the desktop's installed libraries (cupy-cuda12x 13.3.0, cucim-cu12 25.6.0, +pyclesperanto 0.19.0, RTX 2070 SUPER) with the registries of its checkout (`97bc871cb`), +classifying every candidate function at several probe sizes and comparing with 3x20x20: + +| Registry | Functions | Differ at 3x8x8 | Differ at 3x6x6 | Differ at 2x8x8 | +| --- | --- | --- | --- | --- | +| pyclesperanto | 273 (168 flexible, 24 volumetric-to-slice, 2 pure 3D, 79 rejected) | 0 | 0 | 0 | +| CuPy / cuCIM | 133 (70 flexible, 25 pure 2D, 5 volumetric-to-slice, 1 pure 3D, 32 rejected) | 0 | 0 | 0 | + +The smaller probe did not make pyclesperanto discovery faster in that run (5.2 s at edge +20, 27.7 s at edge 8, run in that order in one process); not investigated, the sizes ran +once each.