diff --git a/docs/plans/domain_extraction/ui_streaming_inventory.md b/docs/plans/domain_extraction/ui_streaming_inventory.md index b4340324ac..01be5a8b51 100644 --- a/docs/plans/domain_extraction/ui_streaming_inventory.md +++ b/docs/plans/domain_extraction/ui_streaming_inventory.md @@ -243,7 +243,7 @@ make the generic factory know OpenHCS config classes. `UIConfig` and `PyQtGuiRuntimeContext` are typed process-level policies (`openhcs/pyqt_gui/config.py:351-419`). Cache load/save and direct endpoint -projection already use typed `ConfigCache`/`ConfigDocumentAuthority` +projection already use typed `ConfigCache`/`ConfigDocumentCodec` (`:439-494`). `OpenHCSPyQtApp.setup_application()` registers the global/UI ObjectState roots, sets the global configuration context, and installs the OpenHCS provider set (`openhcs/pyqt_gui/app.py:178-247`). These are valuable diff --git a/docs/refactor/consolidation-20261009/E2-restated-declarations.md b/docs/refactor/consolidation-20261009/E2-restated-declarations.md new file mode 100644 index 0000000000..7ab02cd279 --- /dev/null +++ b/docs/refactor/consolidation-20261009/E2-restated-declarations.md @@ -0,0 +1,99 @@ +# E2: Classes that restate an existing declaration + +**Head audited:** `openhcs` `main` at `ae6ddace5` (#1210, K1 merged); `pycodify` `main` at `e3e3d99` (0.2.0). **Rules:** [00-RULES.md](00-RULES.md). **Step 4 sweep.** +**Shared abstractions** ([02-SHARED-ABSTRACTIONS.md](02-SHARED-ABSTRACTIONS.md)). *Builds:* pycodify `SourceDocument`. *Uses:* pycodify `Assignment` / `generate_python_source` (L6), `MeasurementDialect.row_identity_contract`. + +## What is wrong + +**A second class restates a fact that a declaration next to it already holds, and the two are kept equal by convention.** Agreement between two copies does not say which one is the source ([P10] Proposition 2.2); a copy is one implementation or two, never a saving ([P1] Theorem 4.7). + +1. **Document field enums: a grid of document kind × "Field".** `ConfigDocumentField` (`core/config_document.py:15`, 1 member), `PipelineDocumentField` (`core/pipeline_document_fields.py:6`, 2 members), `FunctionStepDocumentField` (`core/function_step_document.py:13`, 1 member). Each spells the module-level assignments of a generated `.py` file. `PipelineDocument` and `FunctionStepDocument` are dataclasses whose fields have exactly those names; the config document has no dataclass at all. Each of the three codecs hand-writes the same three steps (read a namespace by name, `exec` source, build `Assignment`s). A fourth document kind needs a fourth enum and a fourth copy: one class per cell of the grid ([P1] Theorem 4.5). pycodify has `Assignment` and `generate_python_source` but nothing that maps a dataclass to its assignments; `objectstate.codegen` has formatters only. +2. **One-member enums** (13 in `openhcs/`): each holds one fact. Verdicts in the table below. +3. **Agent config labels.** `ConfigFieldRequirementLabel` and `ConfigFieldResolutionLabel` (`agent/services/llm_context_service.py:615, 626`) turn `ConfigFieldSchema.required` and `.lazy` (two booleans) into two enums and back into text at one call site. +4. **Scan** for the same two shapes (115 enums with one or two members, 23 glued class names). The 15 most suspicious outside the brief are in the second table. + +## Target + +- **pycodify `SourceDocument`** (`pycodify/documents.py`, library PR, no version bump). A dataclass that inherits it is saved as one Python module: each constructor field (`init=True`) is one module-level assignment, in declaration order. It supplies `assignment_names()`, `from_namespace()` (a field with a default may be absent; a required one raises `ValueError`), `from_source()`, `body()` and `to_source(header=, clean_mode=)`. Identity is nominal ([P1] Proposition 3.1): a document is a document because it inherits `SourceDocument`, not because it has a field called `step`. +- **OpenHCS documents.** `PipelineStepsDocument(pipeline_steps)` (the steps-only file), `PipelineDocument(PipelineStepsDocument)` adding `pipeline_config` (default `PipelineConfig()`) and the non-constructor `source_text` (was `original_source`, rule 1b), `FunctionStepDocument(step)`, new `ConfigDocument(config)`. The codecs keep only OpenHCS's type checks and step normalization and call the inherited methods. `ConfigDocumentAuthority` is renamed `ConfigDocumentCodec` (rule 1b). +- **Deleted:** `ConfigDocumentField`, `PipelineDocumentField` and its module `core/pipeline_document_fields.py`, `FunctionStepDocumentField`, `McpReconnectOwner`, `RuntimeBatchCallableMetadataField` and `RUNTIME_BATCH_EXECUTORS_ATTR`, `CellProfilerSpreadsheetRowField`, `ConfigFieldRequirementLabel`, `ConfigFieldResolutionLabel`. +- The refactor merges no two cases that need different answers ([P10] Theorem 5.3): every deleted enum had one member, or (pipeline, labels) members in one-to-one correspondence with the declaration that replaces them. + +### Candidates and verdicts (brief) + +| Candidate | Fact it holds | Verdict | +|---|---|---| +| `ConfigDocumentField`, `PipelineDocumentField`, `FunctionStepDocumentField` | assignment names of a saved `.py` | **derived** from the document dataclass fields (`SourceDocument`); deleted | +| `McpReconnectOwner` | "the MCP client reopens the stdio session", set iff `restart_required` | **derived**: `reconnect_owner`, `reconnect_required`, `retry_after_reconnect` become properties of the recovery record computed from `restart_required`; enum deleted, wire value `"mcp_client"` unchanged | +| `McpWireContentType` | MCP content block `type: "text"` | **kept**, external spelling: Model Context Protocol tool-result content blocks (the dev client speaks raw JSON-RPC and does not import the SDK types) | +| `DesktopInstallerSchemaVersion` | `"openhcs.installer.v4"` | **kept**, external spelling: the installer contract read by `packaging/installers/macos/install-openhcs.sh` and `windows/Install-OpenHCS.ps1` | +| `RuntimeBatchCallableMetadataField` | one private attribute name on a callable | **deleted**: one module constant; the alias `RUNTIME_BATCH_EXECUTORS_ATTR` was unused | +| `SaveImagesImageKind` | CellProfiler SaveImages "type of image to save" | **kept**, external spelling: `.cppipe` SaveImages setting (`Image`); the other CellProfiler values are unsupported, so the parser must reject them | +| `CellProfilerSpreadsheetRowField` | row key `image_number` | **derived** from `CellProfilerMeasurementDialect.row_identity_contract.sample_number_field`; deleted | +| `_ClassificationResultFieldRole`, `_AlignShiftFieldRole` | which result field carries a routing role | **kept**: an `Annotated` marker is the nominal way to find a field without its name string; the vehicle should be K3's row-role family (reassigned to K3 with `MeasurementRowAxisField`) | +| `AlignModule.AdditionalMode` | CellProfiler Align "Similarly" | **kept**, external spelling: `.cppipe` Align setting | +| `PlateInspectionWorkflowScope` | `workflow_scope: "diagnostic"` | **kept**, external spelling: MCP tool-result schema of plate inspection | +| `ConfigFieldSchema` | one reflected config field | **kept**: MCP result DTO, built only by `agent/services/config_service.py` from `dataclasses.fields` and the step signature; not hand-maintained | +| `ConfigFieldRequirementLabel`, `ConfigFieldResolutionLabel` | text for two booleans | **deleted**: the context section formats the booleans | + +### Scan (15 most suspicious outside the brief) + +| Candidate | Verdict | +|---|---| +| `PipelineDocumentNormalizationStrategy`, `StepDocumentNormalizationStrategy`, `ConfigDocumentNormalizationStrategy` (`pyqt_gui/services/code_document_normalization.py`) | kept: members of a registered strategy family keyed by declaration type, whose other members (plate manager, custom function, function pattern) are not documents; the three collapse to one when the codecs fold into the documents (see Outcome) | +| `FunctionStepCodeDocumentDriver` (`pyqt_gui/services/function_step_code_document.py`) | glued; used by the dual editor (U-surface agents hold those files): later pass | +| `SyntheticPlateFormat` (`core/synthetic_plate_generation.py`) | restates the microscope handler family: G4 | +| `FormattingMeasurementFeatureTemplate` (1 member, `interop/cellprofiler/runtime/measurement_rows.py`) | K3 | +| `McpBootstrapFailurePhase`, `PlateWorkspacePreparationOperation` | kept, MCP result spellings | +| `DesktopUpdateProgressMode` | kept, Tk progress-bar mode spellings | +| `WormControlPointAxis` | kept, CellProfiler UntangleWorms feature spelling | +| `IntensityDistributionSettingSchema` | kept, `.cppipe` revision layouts | +| `StartupWindowPresentation`, `ViewerControlPingMode`, `ViewerShapePayloadProjection`, `PlateViewerTab` | enums carrying behaviour through `__new__`: rule 1a families; later pass (U3 / G6) | +| `ObjectLabelRepresentation` | restates the `ObjectLabelVariant` family beside it: later pass (G3) | +| `SeededSegmentationRule` | prose in enum values: later pass | +| `ConfigCacheSpec` | a record, not a grid cell: fine | + +Not examined, for a later pass: the remaining two-member enums outside `processing/backends/` (`CapabilityTransport`, `PlateFileKind`, `WriteMode`, `FijiPayloadKind`, `ViewerDescriptorField`, `ViewerPersistenceMode`, the K3 and K5 enums E1 reassigned) and the glued names `ObjectLabelVariantData`, `LiveMeasurementTableEntry`, `ViewerComponentValueDomainEntry`, `VirtualWorkspaceSourceProjectionCacheEntry`, `Napari{Image,Points,Shapes}LayerDisplayHandler`, `FijiImagePayloadHandler`, `WindowSpec`, `FunctionStepSpec`. + +## Persisted state + +Saved `.py` pipelines, steps and configs are durable. The assignment names (`config`, `pipeline_config`, `pipeline_steps`, `step`) are now the dataclass field names and do not change; the round-trip test proves each kind loads what it wrote and writes exactly those names. + +## Guards + +`tests/unit/test_e2_restated_declaration_guards.py`: +- none of the deleted names appears in `openhcs/`; +- `openhcs/core/pipeline_document_fields.py` does not exist; +- no module under `openhcs/` outside the document modules builds a `pycodify.Assignment` with a string literal naming a document field; +- every `SourceDocument` subclass in `openhcs/` is a dataclass. + +## Tests + +- `tests/unit/test_source_documents.py`: one family test over every `SourceDocument` subclass: render, load, compare, and assert the assigned names are the constructor fields (`config`; `pipeline_config`, `pipeline_steps`; `pipeline_steps`; `step`). +- pycodify: `tests/test_documents.py` (names, defaults, missing required field, `init=False` skipped, round trip). +- The three per-enum value assertions in the document tests are deleted. + +## New-case experiments + +A fourth saved document (say a plate layout). Today: a field enum, a codec with its own namespace reader, `exec` wrapper and `Assignment` list. After: one dataclass inheriting `SourceDocument`. + +## Done when + +The deleted names are gone with no alias; the guards, the round-trip test, the document, codegen and MCP contract tests pass; `guardrails rc=0`; `smoke rc=0`; parity 30/30 (spreadsheet export is touched). + +## Outcome and corrections + +Done: pycodify `SourceDocument` (library PR OpenHCSDev/pycodify#5, no version bump; the submodule points at its branch head), the four documents and three codecs, the one-member enums, the labels, the guards. + +Corrections to the plan above: +- **`pipeline_steps_from_namespace` is kept.** It has no caller in `openhcs/`, but `tests/unit/test_cellprofiler_public_boundary_contracts.py` runs it in a fresh interpreter as the public reader of a generated steps file. It now reads through `PipelineStepsDocument`. +- **Assignment order in a complete pipeline file changes** to `pipeline_steps`, then `pipeline_config`: `PipelineDocument` inherits the steps field from `PipelineStepsDocument`, and dataclass fields are ordered base first. Names and values are unchanged, and files written in either order load. Keeping the old order needs a config-only base class that exists only to order two fields; that is a helper added to fit one chain ([P1] Theorem 4.6), so it was not added. +- **Error text changes** for a missing assignment: `PipelineDocument source must define 'pipeline_steps'.` (was `Pipeline document must define ...`), likewise for the other kinds. A wrong member type in a pipeline reads `Pipeline member N must be FunctionStep` from the one shared check. +- **`PipelineDocument.original_source` is `source_text`** (rule 1b) and is not a constructor field, so it is not an assignment; `read_from(source_text)` attaches it. +- **Certificates.** *derived*: the three field enums (`SourceDocument.assignment_names`), `McpReconnectOwner` and the three stored reconnect fields (properties of `restart_required`), `CellProfilerSpreadsheetRowField` (`row_identity_contract.sample_number_field`, asserted equal to `"image_number"` by the spreadsheet tests). *unused*: `RUNTIME_BATCH_EXECUTORS_ATTR`, `RuntimeBatchCallableMetadataField.owner_label` as a method (inlined at its two sites). *derived*: the two label enums (text of two booleans, same output except that `lazy` is joined by the formatter). No tests-only deletion. + +Not done, with owner: +- The codecs still exist: they hold OpenHCS's type checks and step normalization and forward the rest to the document. Folding validation into the documents' `__post_init__` deletes the three codecs and collapses three normalization strategies into one; it touches 60 files (43 of them tests) and belongs to a surface that owns `pipeline_editor.py` and the code-document services. +- `FunctionStepTransportAuthority` keeps its name (rule 1b asks for a rename on touch): 12 product files and 27 test files, several held by open PRs. Proposed for the same later pass. + +Test evidence is in the PR description. diff --git a/external/pycodify b/external/pycodify index e3e3d993e9..4495224c06 160000 --- a/external/pycodify +++ b/external/pycodify @@ -1 +1 @@ -Subproject commit e3e3d993e9727cacd8aaf6fd7f720b764f40e5e8 +Subproject commit 4495224c06fdd80fd410c28acb287027101f9fb2 diff --git a/openhcs/agent/services/config_service.py b/openhcs/agent/services/config_service.py index aa464cbaab..5e441c6185 100644 --- a/openhcs/agent/services/config_service.py +++ b/openhcs/agent/services/config_service.py @@ -60,7 +60,7 @@ PipelineConfig, StreamingConfig, ) -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.steps.abstract import AbstractStep from openhcs.core.steps.function_step import FunctionStep from openhcs.core.axes import AxisFamily, GroupingDeclaration @@ -313,7 +313,7 @@ def render_source( return RenderedSource( schema_version=SCHEMA_VERSION, title=f"{type(instance).__name__} source", - source=ConfigDocumentAuthority.render( + source=ConfigDocumentCodec.render( instance, expected_config_type=type(instance), clean_mode=clean, diff --git a/openhcs/agent/services/llm_context_service.py b/openhcs/agent/services/llm_context_service.py index af3d2864b7..16f537bf66 100644 --- a/openhcs/agent/services/llm_context_service.py +++ b/openhcs/agent/services/llm_context_service.py @@ -4,7 +4,6 @@ from abc import ABC, abstractmethod from dataclasses import replace -from enum import Enum from typing import ClassVar from metaclass_registry import AutoRegisterMeta @@ -384,11 +383,8 @@ def _schema_lines(cls, schema: ConfigSchema, max_config_fields: int) -> list[str visible_fields = tuple(field for field in schema.fields if not field.ui_hidden) lines = [f"\n## {schema.config_type}"] for field in visible_fields[:max_config_fields]: - requirement = ConfigFieldRequirementLabel.from_required(field.required) - resolution = ConfigFieldResolutionLabel.from_lazy(field.lazy) - lines.append( - f"- {field.path}: {field.type_repr} ({requirement.value}{resolution.value})" - ) + traits = ("required" if field.required else "optional", *(("lazy",) if field.lazy else ())) + lines.append(f"- {field.path}: {field.type_repr} ({', '.join(traits)})") if len(visible_fields) > max_config_fields: remaining = len(visible_fields) - max_config_fields lines.append(f"- ... {remaining} more fields") @@ -612,28 +608,6 @@ class ObjectStateEditingStepsSection( - List recovery state with {agent_capabilities.ui_list_snapshots.name} and {agent_capabilities.ui_list_branches.name}. Use {agent_capabilities.ui_restore_snapshot.name} for an approved exact rollback, {agent_capabilities.ui_time_travel_head.name} to return to the current branch head, and {agent_capabilities.ui_switch_branch.name} only for an intentional branch change. Re-read the affected field and state surface after every recovery mutation.""" -class ConfigFieldRequirementLabel(Enum): - REQUIRED = "required" - OPTIONAL = "optional" - - @classmethod - def from_required(cls, required: bool) -> "ConfigFieldRequirementLabel": - if required: - return cls.REQUIRED - return cls.OPTIONAL - - -class ConfigFieldResolutionLabel(Enum): - LAZY = ", lazy" - EAGER = "" - - @classmethod - def from_lazy(cls, lazy: bool) -> "ConfigFieldResolutionLabel": - if lazy: - return cls.LAZY - return cls.EAGER - - class AgentAuthoringContextService: """Build bounded, registry-grounded context for agents authoring OpenHCS.""" diff --git a/openhcs/core/config_cache.py b/openhcs/core/config_cache.py index 2c6cdfb23c..bb67420ebb 100644 --- a/openhcs/core/config_cache.py +++ b/openhcs/core/config_cache.py @@ -15,7 +15,7 @@ ) from openhcs.core.config import GlobalPipelineConfig -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec logger = logging.getLogger(__name__) @@ -61,7 +61,7 @@ def load_config_sync(spec: ConfigCacheSpec[ConfigT]) -> ConfigT | None: try: if not spec.cache_file.exists(): return None - cached_config = ConfigDocumentAuthority.from_source( + cached_config = ConfigDocumentCodec.from_source( spec.cache_file.read_text(encoding="utf-8"), expected_config_type=spec.config_type, ) @@ -91,7 +91,7 @@ def save_config_sync(config: ConfigT, spec: ConfigCacheSpec[ConfigT]) -> bool: ) try: spec.cache_file.parent.mkdir(parents=True, exist_ok=True) - payload = ConfigDocumentAuthority.render( + payload = ConfigDocumentCodec.render( config, expected_config_type=spec.config_type, ).encode("utf-8") diff --git a/openhcs/core/config_document.py b/openhcs/core/config_document.py index 9b174d3d8f..30a0f9d6f8 100644 --- a/openhcs/core/config_document.py +++ b/openhcs/core/config_document.py @@ -1,24 +1,29 @@ -"""Canonical Python document contract for one OpenHCS configuration object.""" +"""Python document for one OpenHCS configuration object.""" from __future__ import annotations import ast from collections.abc import Mapping -from dataclasses import MISSING, Field, fields -from enum import Enum +from dataclasses import MISSING, Field, dataclass, fields from typing import Any, TypeVar, cast, get_type_hints +from pycodify import SourceDocument + ConfigT = TypeVar("ConfigT") FieldT = TypeVar("FieldT") -class ConfigDocumentField(str, Enum): - """Exact public assignment in a configuration Python document.""" +@dataclass(frozen=True, slots=True) +class ConfigDocument(SourceDocument): + """One configuration object, saved as a Python file that assigns it.""" + + config: object - CONFIG = "config" +(_CONFIG_NAME,) = ConfigDocument.assignment_names() -class ConfigDocumentAuthority: + +class ConfigDocumentCodec: """Validate, parse, and render the canonical configuration document shape.""" HEADER = "# OpenHCS configuration" @@ -32,11 +37,7 @@ def from_namespace( ) -> ConfigT: """Read the exact config assignment from an executed namespace.""" - config_field = ConfigDocumentField.CONFIG.value - if config_field not in namespace: - raise ValueError(f"Config document must define {config_field!r}.") - - config = namespace[config_field] + config = ConfigDocument.from_namespace(namespace).config cls._require_config_type(config, expected_config_type) return cast(ConfigT, config) @@ -49,13 +50,9 @@ def from_source( ) -> ConfigT: """Execute Python source and read its canonical config assignment.""" - namespace: dict[str, object] = {} - code = compile(source, "", "exec") - exec(code, namespace) # noqa: S102 - executable Python is the document format - return cls.from_namespace( - namespace, - expected_config_type=expected_config_type, - ) + config = ConfigDocument.from_source(source).config + cls._require_config_type(config, expected_config_type) + return cast(ConfigT, config) @classmethod def project_dataclass_field_from_source( @@ -87,7 +84,7 @@ def project_dataclass_field_from_source( dataclass_field = matching_fields[0] module = ast.parse(source, filename="") - config_field = ConfigDocumentField.CONFIG.value + config_field = _CONFIG_NAME assignments = tuple( statement for statement in module.body @@ -162,13 +159,10 @@ def render( ) -> str: """Render a validated config object as reviewable Python source.""" - from pycodify import Assignment, generate_python_source - import openhcs.serialization.pycodify_formatters # noqa: F401 cls._require_config_type(config, expected_config_type) - return generate_python_source( - Assignment(ConfigDocumentField.CONFIG.value, config), + return ConfigDocument(config=config).to_source( header=cls.HEADER, clean_mode=clean_mode, ) @@ -179,9 +173,8 @@ def _require_config_type( expected_config_type: type[ConfigT], ) -> None: if not isinstance(config, expected_config_type): - config_field = ConfigDocumentField.CONFIG.value raise TypeError( - f"Config document variable {config_field!r} must be " + f"Config document variable {_CONFIG_NAME!r} must be " f"{expected_config_type.__name__}, got {type(config).__name__}." ) diff --git a/openhcs/core/function_step_document.py b/openhcs/core/function_step_document.py index cd4133928c..1211c23753 100644 --- a/openhcs/core/function_step_document.py +++ b/openhcs/core/function_step_document.py @@ -4,20 +4,15 @@ from collections.abc import Mapping from dataclasses import dataclass -from enum import Enum + +from pycodify import SourceDocument from openhcs.core.function_step_transport import FunctionStepTransportAuthority from openhcs.core.steps.function_step import FunctionStep -class FunctionStepDocumentField(str, Enum): - """Exact public assignment in a FunctionStep Python document.""" - - STEP = "step" - - @dataclass(frozen=True, slots=True) -class FunctionStepDocument: +class FunctionStepDocument(SourceDocument): """Normalized public declaration for one FunctionStep.""" step: FunctionStep @@ -42,21 +37,15 @@ def from_namespace( cls, namespace: Mapping[str, object], ) -> FunctionStepDocument: - """Read the exact public step assignment from an executed namespace.""" + """Read the public step assignment from an executed namespace.""" - field_name = FunctionStepDocumentField.STEP.value - if field_name not in namespace: - raise ValueError(f"FunctionStep document must define {field_name!r}.") - return cls.from_value(namespace[field_name]) + return cls.from_value(FunctionStepDocument.from_namespace(namespace).step) @classmethod def from_source(cls, source: str) -> FunctionStepDocument: """Execute Python source and read its FunctionStep assignment.""" - namespace: dict[str, object] = {} - code = compile(source, "", "exec") - exec(code, namespace) - return cls.from_namespace(namespace) + return cls.from_value(FunctionStepDocument.from_source(source).step) @classmethod def render( @@ -68,11 +57,8 @@ def render( """Render a validated FunctionStep document as reviewable Python source.""" import openhcs.serialization.pycodify_formatters # noqa: F401 - from pycodify import Assignment, generate_python_source - normalized = cls.from_value(document.step) - return generate_python_source( - Assignment(FunctionStepDocumentField.STEP.value, normalized.step), + return cls.from_value(document.step).to_source( header=cls.HEADER, clean_mode=clean_mode, ) diff --git a/openhcs/core/function_step_transport.py b/openhcs/core/function_step_transport.py index 66a732d89f..a8485ab8dd 100644 --- a/openhcs/core/function_step_transport.py +++ b/openhcs/core/function_step_transport.py @@ -2,7 +2,7 @@ from __future__ import annotations -from dataclasses import replace +from dataclasses import dataclass, replace from collections.abc import Mapping, Sequence from typing import Any @@ -18,11 +18,18 @@ FunctionReference, FunctionReferenceTransportAuthority, ) -from openhcs.core.pipeline_document_fields import PipelineDocumentField from openhcs.core.steps.function_step import FunctionStep +from pycodify import SourceDocument +@dataclass(frozen=True, slots=True, kw_only=True) +class PipelineStepsDocument(SourceDocument): + """The steps of one pipeline, saved as a Python file that assigns them.""" + + pipeline_steps: list[FunctionStep] + + class FunctionStepTransportAuthority: """Canonicalize FunctionStep callables before process or ZMQ transport.""" @@ -35,13 +42,12 @@ def source_from_pipeline( ) -> str: """Normalize and pycodify one public FunctionStep list.""" import openhcs.serialization.pycodify_formatters # noqa: F401 - from pycodify import Assignment, generate_python_source steps = list(pipeline_steps) cls._require_function_steps(steps) - normalized = cls.normalize_pipeline(steps) - return generate_python_source( - Assignment(PipelineDocumentField.PIPELINE_STEPS.value, normalized), + return PipelineStepsDocument( + pipeline_steps=cls.normalize_pipeline(steps) + ).to_source( header="# Edit this pipeline and save to apply changes", clean_mode=clean_mode, ) @@ -52,13 +58,16 @@ def pipeline_steps_from_namespace( namespace: Mapping[str, object], ) -> list[FunctionStep]: """Require and normalize the public FunctionStep export.""" - field_name = PipelineDocumentField.PIPELINE_STEPS.value - if field_name not in namespace: - raise ValueError(f"Pipeline source must define {field_name!r}.") - pipeline_steps = namespace[field_name] + return cls.normalized_steps( + PipelineStepsDocument.from_namespace(namespace).pipeline_steps + ) + + @classmethod + def normalized_steps(cls, pipeline_steps: object) -> list[FunctionStep]: + """Require a list of FunctionStep and normalize each member.""" if type(pipeline_steps) is not list: raise TypeError( - f"{field_name} must be a list[FunctionStep], " + "pipeline_steps must be a list[FunctionStep], " f"got {type(pipeline_steps).__name__}." ) cls._require_function_steps(pipeline_steps) diff --git a/openhcs/core/pipeline_document.py b/openhcs/core/pipeline_document.py index bae002c41f..8abde39d74 100644 --- a/openhcs/core/pipeline_document.py +++ b/openhcs/core/pipeline_document.py @@ -3,21 +3,32 @@ from __future__ import annotations from collections.abc import Mapping, Sequence -from dataclasses import dataclass, replace +from dataclasses import dataclass, field, replace from openhcs.core.config import PipelineConfig -from openhcs.core.pipeline_document_fields import PipelineDocumentField -from openhcs.core.function_step_transport import FunctionStepTransportAuthority +from openhcs.core.function_step_transport import ( + FunctionStepTransportAuthority, + PipelineStepsDocument, +) from openhcs.core.steps.function_step import FunctionStep -@dataclass(frozen=True, slots=True) -class PipelineDocument: - """Normalized public configuration and steps for one pipeline.""" +@dataclass(frozen=True, slots=True, kw_only=True) +class PipelineDocument(PipelineStepsDocument): + """Normalized public configuration and steps for one pipeline. - pipeline_config: PipelineConfig - pipeline_steps: list[FunctionStep] - original_source: str | None = None + A file that assigns only the steps is a pipeline with the default config. + """ + + pipeline_config: PipelineConfig = field(default_factory=PipelineConfig) + source_text: str | None = field(default=None, init=False) + + def read_from(self, source_text: str | None) -> PipelineDocument: + """Return this document carrying the source text it was read from.""" + + document = replace(self) + object.__setattr__(document, "source_text", source_text) + return document class PipelineDocumentCodec: @@ -34,14 +45,16 @@ def from_values( ) -> PipelineDocument: """Build a document from typed values and normalize its steps.""" - cls._require_pipeline_config(pipeline_config) - steps = list(pipeline_steps) - cls._require_pipeline_steps(steps) - normalized_steps = FunctionStepTransportAuthority.normalize_pipeline(steps) - cls._require_pipeline_steps(normalized_steps) + if not isinstance(pipeline_config, PipelineConfig): + raise TypeError( + "pipeline_config must be a PipelineConfig, " + f"got {type(pipeline_config).__name__}." + ) return PipelineDocument( pipeline_config=pipeline_config, - pipeline_steps=normalized_steps, + pipeline_steps=FunctionStepTransportAuthority.normalized_steps( + list(pipeline_steps) + ), ) @classmethod @@ -51,41 +64,32 @@ def from_namespace( ) -> PipelineDocument: """Read the public assignments from an executed namespace.""" - config_field = PipelineDocumentField.PIPELINE_CONFIG.value - steps_field = PipelineDocumentField.PIPELINE_STEPS.value - if steps_field not in namespace: - raise ValueError(f"Pipeline document must define {steps_field!r}.") - - pipeline_config = ( - namespace[config_field] if config_field in namespace else PipelineConfig() - ) - pipeline_steps = namespace[steps_field] - cls._require_pipeline_config(pipeline_config) - if type(pipeline_steps) is not list: - raise TypeError( - f"{steps_field} must be a list[FunctionStep], " - f"got {type(pipeline_steps).__name__}." - ) - return cls.from_values( - pipeline_config=pipeline_config, - pipeline_steps=pipeline_steps, - ) + return cls._validated(PipelineDocument.from_namespace(namespace)) @classmethod def from_source(cls, source: str) -> PipelineDocument: """Execute Python source and read its pipeline assignments.""" - namespace: dict[str, object] = {} - code = compile(source, "", "exec") - exec(code, namespace) - return replace(cls.from_namespace(namespace), original_source=source) + return cls._validated(PipelineDocument.from_source(source)).read_from(source) + + @classmethod + def _validated(cls, document: PipelineDocument) -> PipelineDocument: + if type(document.pipeline_steps) is not list: + raise TypeError( + "pipeline_steps must be a list[FunctionStep], " + f"got {type(document.pipeline_steps).__name__}." + ) + return cls.from_values( + pipeline_config=document.pipeline_config, + pipeline_steps=document.pipeline_steps, + ) @classmethod def execution_source(cls, document: PipelineDocument) -> str: - """Return the source it was parsed from, otherwise freshly rendered source.""" + """Return the source it was read from, otherwise freshly rendered source.""" - if document.original_source is not None: - return document.original_source + if document.source_text is not None: + return document.source_text return cls.render(document) @classmethod @@ -98,43 +102,8 @@ def render( """Render a validated pipeline document as reviewable Python source.""" import openhcs.serialization.pycodify_formatters # noqa: F401 - from pycodify import Assignment, BlankLine, CodeBlock, generate_python_source - normalized = cls.from_values( + return cls.from_values( pipeline_config=document.pipeline_config, pipeline_steps=document.pipeline_steps, - ) - body = CodeBlock.from_items( - ( - Assignment( - PipelineDocumentField.PIPELINE_CONFIG.value, - normalized.pipeline_config, - ), - BlankLine(), - Assignment( - PipelineDocumentField.PIPELINE_STEPS.value, - normalized.pipeline_steps, - ), - ) - ) - return generate_python_source( - body, - header=cls.HEADER, - clean_mode=clean_mode, - ) - - @staticmethod - def _require_pipeline_config(value: object) -> None: - if not isinstance(value, PipelineConfig): - raise TypeError( - f"pipeline_config must be a PipelineConfig, got {type(value).__name__}." - ) - - @staticmethod - def _require_pipeline_steps(values: Sequence[object]) -> None: - for index, step in enumerate(values): - if not isinstance(step, FunctionStep): - raise TypeError( - f"pipeline_steps member {index} must be FunctionStep, " - f"got {type(step).__name__}." - ) + ).to_source(header=cls.HEADER, clean_mode=clean_mode) diff --git a/openhcs/core/pipeline_document_fields.py b/openhcs/core/pipeline_document_fields.py deleted file mode 100644 index 1bfa8d22f7..0000000000 --- a/openhcs/core/pipeline_document_fields.py +++ /dev/null @@ -1,10 +0,0 @@ -"""Public field names shared by complete and step-fragment pipeline documents.""" - -from enum import Enum - - -class PipelineDocumentField(str, Enum): - """Exact public assignments owned by the pipeline document contract.""" - - PIPELINE_CONFIG = "pipeline_config" - PIPELINE_STEPS = "pipeline_steps" diff --git a/openhcs/core/runtime_batch_contracts.py b/openhcs/core/runtime_batch_contracts.py index f076a71478..b36e0ba646 100644 --- a/openhcs/core/runtime_batch_contracts.py +++ b/openhcs/core/runtime_batch_contracts.py @@ -171,17 +171,7 @@ def from_contract( ) -class RuntimeBatchCallableMetadataField(str, Enum): - """Callable metadata fields owned by runtime batch contracts.""" - - EXECUTORS = "__openhcs_runtime_batch_executors__" - - def owner_label(self, func: Callable) -> str: - """Return a compact field label for validation errors.""" - return f"{func}.{self.value}" - - -RUNTIME_BATCH_EXECUTORS_ATTR = RuntimeBatchCallableMetadataField.EXECUTORS.value +_BATCH_EXECUTORS_ATTRIBUTE = "__openhcs_runtime_batch_executors__" @dataclass(frozen=True, slots=True) @@ -362,20 +352,19 @@ def runtime_batch_executors_from_callable( func: Callable, ) -> Mapping[RuntimeBatchExecutionDomain, Callable]: """Return declared batch executors keyed by runtime batch domain.""" - executors_field = RuntimeBatchCallableMetadataField.EXECUTORS try: namespace = vars(func) except TypeError: declared = {} else: - if executors_field.value in namespace: - declared = namespace[executors_field.value] + if _BATCH_EXECUTORS_ATTRIBUTE in namespace: + declared = namespace[_BATCH_EXECUTORS_ATTRIBUTE] else: declared = {} if declared is None: declared = {} if not isinstance(declared, Mapping): - raise TypeError(f"{executors_field.owner_label(func)} must be a mapping.") + raise TypeError(f"{func}.{_BATCH_EXECUTORS_ATTRIBUTE} must be a mapping.") batch_executors: dict[RuntimeBatchExecutionDomain, Callable] = {} for raw_domain, executor in declared.items(): domain = ( @@ -385,7 +374,7 @@ def runtime_batch_executors_from_callable( ) if not callable(executor): raise TypeError( - f"{executors_field.owner_label(func)}[{domain.value!r}] must " + f"{func}.{_BATCH_EXECUTORS_ATTRIBUTE}[{domain.value!r}] must " f"be callable, got {type(executor).__name__}." ) batch_executors[domain] = executor @@ -417,9 +406,7 @@ def decorator(func: F) -> F: raise TypeError( f"{func!r} cannot carry runtime batch executor metadata." ) from exc - namespace[RuntimeBatchCallableMetadataField.EXECUTORS.value] = MappingProxyType( - batch_executors - ) + namespace[_BATCH_EXECUTORS_ATTRIBUTE] = MappingProxyType(batch_executors) return func return decorator diff --git a/openhcs/mcp/lifecycle.py b/openhcs/mcp/lifecycle.py index c8025a71c9..4a3dcb7ceb 100644 --- a/openhcs/mcp/lifecycle.py +++ b/openhcs/mcp/lifecycle.py @@ -13,7 +13,6 @@ from collections.abc import Mapping, Sequence from dataclasses import dataclass import errno -from enum import Enum from hashlib import sha256 import json import os @@ -100,12 +99,6 @@ class SourceAndInstallationChanged(McpProcessRecoveryReason): name = "source_and_installation_changed" -class McpReconnectOwner(str, Enum): - """Protocol participant responsible for opening a new stdio session.""" - - CLIENT = "mcp_client" - - @dataclass(frozen=True, slots=True) class McpInstallationPointerSnapshot: """Identity of the installer-owned current-environment pointer.""" @@ -180,12 +173,24 @@ class McpProcessRecoveryStatus: restart_required: bool restart_command: tuple[str, ...] restart_command_is_stable: bool - reconnect_required: bool - reconnect_owner: McpReconnectOwner | None - retry_after_reconnect: bool automatic_recovery_on_reconnect: bool hint: str | None + @property + def reconnect_required(self) -> bool: + """A restarted stdio server is a new session, so the client reconnects.""" + return self.restart_required + + @property + def reconnect_owner(self) -> str | None: + """Who opens the new stdio session: in MCP, always the client.""" + return "mcp_client" if self.restart_required else None + + @property + def retry_after_reconnect(self) -> bool: + """The refused call is safe to repeat on the new session.""" + return self.restart_required + def _validated_absolute_path(raw_path: str, *, variable_name: str) -> Path: if not raw_path or "\x00" in raw_path: @@ -342,9 +347,6 @@ def recovery_status( restart_required=restart_required, restart_command=restart_command if restart_required else (), restart_command_is_stable=command_is_stable, - reconnect_required=restart_required, - reconnect_owner=(McpReconnectOwner.CLIENT if restart_required else None), - retry_after_reconnect=restart_required, automatic_recovery_on_reconnect=automatic_recovery, hint=hint, ) diff --git a/openhcs/mcp/server.py b/openhcs/mcp/server.py index e03f3cd404..8602cc6d53 100644 --- a/openhcs/mcp/server.py +++ b/openhcs/mcp/server.py @@ -494,9 +494,7 @@ def _mcp_server_health() -> McpServerHealthResult: restart_command=recovery.restart_command, restart_command_is_stable=recovery.restart_command_is_stable, reconnect_required=recovery.reconnect_required, - reconnect_owner=( - None if recovery.reconnect_owner is None else recovery.reconnect_owner.value - ), + reconnect_owner=recovery.reconnect_owner, retry_after_reconnect=recovery.retry_after_reconnect, automatic_recovery_on_reconnect=(recovery.automatic_recovery_on_reconnect), restart_hint=recovery.hint, @@ -927,11 +925,7 @@ def _mcp_server_stale_error(tool_name: str) -> JsonValue: restart_command=recovery.restart_command, restart_command_is_stable=recovery.restart_command_is_stable, reconnect_required=recovery.reconnect_required, - reconnect_owner=( - None - if recovery.reconnect_owner is None - else recovery.reconnect_owner.value - ), + reconnect_owner=recovery.reconnect_owner, retry_after_reconnect=recovery.retry_after_reconnect, automatic_recovery_on_reconnect=(recovery.automatic_recovery_on_reconnect), restart_hint=recovery.hint or "", diff --git a/openhcs/processing/backends/cellprofiler/spreadsheet_export.py b/openhcs/processing/backends/cellprofiler/spreadsheet_export.py index 2fcfce116c..7d6d460956 100644 --- a/openhcs/processing/backends/cellprofiler/spreadsheet_export.py +++ b/openhcs/processing/backends/cellprofiler/spreadsheet_export.py @@ -161,12 +161,6 @@ def from_cellprofiler(cls, value: str) -> "SpreadsheetNanRepresentation": raise ValueError(f"Unsupported spreadsheet NaN representation {value!r}.") -class CellProfilerSpreadsheetRowField(str, Enum): - """CellProfiler-owned spreadsheet row fields.""" - - IMAGE_NUMBER = "image_number" - - @dataclass(frozen=True, slots=True) class SpreadsheetColumnSelection: """One CellProfiler subject/measurement column selection.""" @@ -309,7 +303,7 @@ def combined_rows( ) -> ColumnarRows: if len(selected_tables) == 1: return selected_tables[0][1] - image_field = CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value + image_field = CellProfilerMeasurementDialect.row_identity_contract.sample_number_field grouped: list[tuple[str, ColumnarRows, OrderedDict[object, list[int]]]] = [] image_counts: OrderedDict[object, int] = OrderedDict() for subject, rows in selected_tables: @@ -443,7 +437,7 @@ def csv_schema( if len(active_subjects) > 1: bindings = [] subjects = sorted(active_subjects, key=len, reverse=True) - image_field = CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value + image_field = CellProfilerMeasurementDialect.row_identity_contract.sample_number_field for name in columns: if name == image_field: bindings.append(("Image", name)) @@ -1062,7 +1056,7 @@ def _relationship_rows( def _cellprofiler_rows(rows: ColumnarRows) -> ColumnarRows: """Project declared image coordinates once while retaining sparse columns.""" slice_field = MeasurementRowAxisField.SLICE_INDEX.value - image_field = CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value + image_field = CellProfilerMeasurementDialect.row_identity_contract.sample_number_field if slice_field not in rows.columns: return rows source_values = rows.column_values(slice_field) @@ -1132,7 +1126,7 @@ def _selected_table_columns( return tables axis_fields = ( *MeasurementRowAxisField.field_names(), - CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value, + CellProfilerMeasurementDialect.row_identity_contract.sample_number_field, ) selected = OrderedDict() for subject, rows in tables.items(): @@ -1171,7 +1165,7 @@ def _with_requested_aggregates( ) if not (mean or median or standard_deviation or reference_means): return tables - image_field = CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value + image_field = CellProfilerMeasurementDialect.row_identity_contract.sample_number_field image = tables.get("Image") image_numbers = ( () @@ -1277,7 +1271,7 @@ def _numeric_features( axis_fields = frozenset( ( *MeasurementRowAxisField.field_names(), - CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value, + CellProfilerMeasurementDialect.row_identity_contract.sample_number_field, ) ) features = [] @@ -1326,7 +1320,7 @@ def _with_image_columns_on_objects( return tables # Image metadata is small; retain its existing naming policy, without rebuilding # complete object row dictionaries just to project the selected columns. - image_field = CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value + image_field = CellProfilerMeasurementDialect.row_identity_contract.sample_number_field image_columns_by_number = { row[image_field]: _image_columns_for_objects( row, @@ -1432,7 +1426,7 @@ def _rows_by_resolved_path( ) if not tokens: return ((path_template, rows),) - image_field = CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value + image_field = CellProfilerMeasurementDialect.row_identity_contract.sample_number_field image_row_mappings = tuple(image_rows.iter_row_mappings()) image_rows_by_number = { row[image_field]: row for row in image_row_mappings if image_field in row @@ -1510,7 +1504,7 @@ class SpreadsheetPartitionInvocation: kwargs: dict[str, object] image_numbers: CellProfilerImageSetNumbering partition_fields: tuple[str, ...] = ( - CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value, + CellProfilerMeasurementDialect.row_identity_contract.sample_number_field, ) csv_fields: tuple[FieldSpec, ...] = () @@ -1643,7 +1637,7 @@ def _partitioned_spreadsheet_export( relationship_domains = set() relationship_fields = ( *ObjectRelationshipDeclaration.exported_field_names, - CellProfilerSpreadsheetRowField.IMAGE_NUMBER.value, + CellProfilerMeasurementDialect.row_identity_contract.sample_number_field, ) if relationship_selections and relationships_partitionable: relationship_kwargs = { diff --git a/openhcs/pyqt_gui/config.py b/openhcs/pyqt_gui/config.py index dd2d42aee8..9a686dfe13 100644 --- a/openhcs/pyqt_gui/config.py +++ b/openhcs/pyqt_gui/config.py @@ -380,13 +380,13 @@ def load_cached_ui_config_sync() -> UIConfig: def load_cached_ui_execution_endpoint_sync() -> OpenHCSZMQConfig: """Project the execution endpoint directly from the canonical UI document.""" - from openhcs.core.config_document import ConfigDocumentAuthority + from openhcs.core.config_document import ConfigDocumentCodec cache_file = ui_config_cache_spec().cache_file if not cache_file.exists(): return OpenHCSZMQConfig() try: - return ConfigDocumentAuthority.project_dataclass_field_from_source( + return ConfigDocumentCodec.project_dataclass_field_from_source( cache_file.read_text(encoding="utf-8"), expected_config_type=UIConfig, expected_field_type=OpenHCSZMQConfig, diff --git a/openhcs/pyqt_gui/services/code_document_normalization.py b/openhcs/pyqt_gui/services/code_document_normalization.py index 9764ef99ce..f8b8cef570 100644 --- a/openhcs/pyqt_gui/services/code_document_normalization.py +++ b/openhcs/pyqt_gui/services/code_document_normalization.py @@ -10,7 +10,7 @@ from formbridge.services.function_pattern_code_document import FunctionPatternValue from python_introspect import AnnotatedDataclassValidationMixin -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.function_step_document import ( FunctionStepDocument, FunctionStepDocumentCodec, @@ -102,11 +102,11 @@ def normalize_source( declaration_type: type[DeclarationT], clean_mode: bool, ) -> str: - config = ConfigDocumentAuthority.from_source( + config = ConfigDocumentCodec.from_source( source, expected_config_type=declaration_type, ) - return ConfigDocumentAuthority.render( + return ConfigDocumentCodec.render( config, expected_config_type=declaration_type, clean_mode=clean_mode, diff --git a/openhcs/pyqt_gui/windows/config_window.py b/openhcs/pyqt_gui/windows/config_window.py index 7218f0de85..d5a93cb58f 100644 --- a/openhcs/pyqt_gui/windows/config_window.py +++ b/openhcs/pyqt_gui/windows/config_window.py @@ -6,22 +6,22 @@ from pyqt_reactive.windows.config_window import ConfigDocumentFormat, ConfigWindow -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from formbridge.bridge.identities import UiStableWindowIdentityDeclaration class OpenHCSConfigDocument(ConfigDocumentFormat): - """The OpenHCS configuration document (``ConfigDocumentAuthority``).""" + """The OpenHCS configuration document (``ConfigDocumentCodec``).""" def render(self, config: object, *, expected_config_type: type, clean_mode: bool) -> str: - return ConfigDocumentAuthority.render( + return ConfigDocumentCodec.render( config, expected_config_type=expected_config_type, clean_mode=clean_mode, ) def from_source(self, source: str, *, expected_config_type: type) -> object: - return ConfigDocumentAuthority.from_source( + return ConfigDocumentCodec.from_source( source, expected_config_type=expected_config_type, ) diff --git a/openhcs/runtime/zmq_execution_client.py b/openhcs/runtime/zmq_execution_client.py index 14efb43965..ed2ab830ba 100644 --- a/openhcs/runtime/zmq_execution_client.py +++ b/openhcs/runtime/zmq_execution_client.py @@ -57,7 +57,7 @@ from openhcs.agent.exceptions import AgentFacingErrorMixin from openhcs.core.artifact_inspection import CompiledArtifactInspection from openhcs.core.config import GlobalPipelineConfig, PipelineConfig -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.debug import DebugExecutionConfig from openhcs.core.execution_state import ExecutionOutputSummary from openhcs.core.pipeline_document import ( @@ -257,9 +257,8 @@ def __init__( pipeline_config=pipeline_document.pipeline_config, pipeline_steps=pipeline_document.pipeline_steps, ) - normalized_document = replace( - normalized_document, - original_source=pipeline_document.original_source, + normalized_document = normalized_document.read_from( + pipeline_document.source_text ) self._set_parts( identity=ZMQExecutionIdentity( @@ -503,7 +502,7 @@ def phase_scope(phase: ZMQPipelineRunPhase) -> AbstractContextManager[None]: def _pycodify_config_source( config: GlobalPipelineConfig | OpenHCSZMQConfig, ) -> str: - return ConfigDocumentAuthority.render( + return ConfigDocumentCodec.render( config, expected_config_type=type(config), ) diff --git a/openhcs/runtime/zmq_execution_server.py b/openhcs/runtime/zmq_execution_server.py index 49cd9c8607..b7c2bd4727 100644 --- a/openhcs/runtime/zmq_execution_server.py +++ b/openhcs/runtime/zmq_execution_server.py @@ -26,7 +26,7 @@ from openhcs.core.config import GlobalPipelineConfig, PipelineConfig from objectstate.object_state import ObjectState from objectstate.object_state_registry import ObjectStateRegistry -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.orchestrator.cancellation import ExecutionCancelledError from openhcs.core.orchestrator.worker_execution import PreparedForkWorkerLaneRunner from openhcs.core.pipeline_document import PipelineDocumentCodec @@ -491,7 +491,7 @@ def _resolve_request_config( pipeline_config: PipelineConfig, ) -> OpenHCSExecutionConfigBundle: if request_payload.config_code is not None: - global_config = ConfigDocumentAuthority.from_source( + global_config = ConfigDocumentCodec.from_source( request_payload.config_code, expected_config_type=GlobalPipelineConfig, ) diff --git a/pyproject.toml b/pyproject.toml index bdaf693030..c7ffa0c268 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -100,7 +100,7 @@ dependencies = [ # External dependencies (PyPI versions for production builds) "zmqruntime>=0.6.2,<0.7", - "pycodify>=0.2.0,<0.3", + "pycodify>=0.2.1,<0.3", "objectstate>=2.1.1,<3", "python-introspect>=0.2.8,<0.3", "metaclass-registry>=0.3.0,<0.4", diff --git a/tests/integration/test_benchmark_openhcs_adapter_cppipe.py b/tests/integration/test_benchmark_openhcs_adapter_cppipe.py index 42621085c3..de09888fc7 100644 --- a/tests/integration/test_benchmark_openhcs_adapter_cppipe.py +++ b/tests/integration/test_benchmark_openhcs_adapter_cppipe.py @@ -19,7 +19,7 @@ from benchmark.pipelines.registry import NUCLEI_SEGMENTATION from benchmark.runner import run_benchmark from openhcs.core.config import GlobalPipelineConfig, PathPlanningConfig -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.demo.synthetic_data import ( SyntheticMicroscopyGenerator, ) @@ -95,7 +95,7 @@ def test_openhcs_adapter_retains_declared_path_policy_and_explicit_exports( assert result.success is True assert result.provenance["csv_output_count"] > 0 assert result.provenance["image_output_count"] == 0 - submitted_config = ConfigDocumentAuthority.from_source( + submitted_config = ConfigDocumentCodec.from_source( (output_dir / "submitted_global_config.py").read_text(encoding="utf-8"), expected_config_type=GlobalPipelineConfig, ) diff --git a/tests/unit/agent/test_agent_services.py b/tests/unit/agent/test_agent_services.py index 032bf1c3ef..5179542cc8 100644 --- a/tests/unit/agent/test_agent_services.py +++ b/tests/unit/agent/test_agent_services.py @@ -96,7 +96,7 @@ PipelineConfig, LazySourceBindingsConfig, ) -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.pipeline.function_contracts import artifact_inputs, artifact_outputs from openhcs.core.pipeline_document import PipelineDocumentCodec @@ -2588,7 +2588,7 @@ def test_config_service_validates_and_renders_config_source(): assert result.valid is True assert "num_workers=2" in rendered.source assert ( - ConfigDocumentAuthority.from_source( + ConfigDocumentCodec.from_source( rendered.source, expected_config_type=type(config), ) diff --git a/tests/unit/agent/test_mcp_affine_real_compile.py b/tests/unit/agent/test_mcp_affine_real_compile.py index 2456133df9..2d986a0e4b 100644 --- a/tests/unit/agent/test_mcp_affine_real_compile.py +++ b/tests/unit/agent/test_mcp_affine_real_compile.py @@ -83,7 +83,7 @@ async def exercise(): assert result["steps"][0]["step_name"] == "DeclaredNlm" assert result["source_workspace"]["file_count"] == 1 assert len(calls) == 1 and progress_threads - assert calls[0].pipeline_document.original_source == source + assert calls[0].pipeline_document.source_text == source assert result["progress_event_count"] == 1 stages = ( "Resolving pipeline source document", diff --git a/tests/unit/agent/test_mcp_lifecycle.py b/tests/unit/agent/test_mcp_lifecycle.py index 74d240a597..b0198c98b6 100644 --- a/tests/unit/agent/test_mcp_lifecycle.py +++ b/tests/unit/agent/test_mcp_lifecycle.py @@ -60,7 +60,7 @@ def test_source_change_requires_client_owned_reconnect(): ) assert changed.restart_command_is_stable is False assert changed.reconnect_required is True - assert changed.reconnect_owner.value == "mcp_client" + assert changed.reconnect_owner == "mcp_client" assert changed.retry_after_reconnect is True assert changed.automatic_recovery_on_reconnect is False assert "new MCP initialize handshake" in changed.hint diff --git a/tests/unit/agent/test_owned_runtime_bootstrap.py b/tests/unit/agent/test_owned_runtime_bootstrap.py index 6b9be0297e..bdee8cdb75 100644 --- a/tests/unit/agent/test_owned_runtime_bootstrap.py +++ b/tests/unit/agent/test_owned_runtime_bootstrap.py @@ -127,7 +127,7 @@ def test_admitted_plan_owns_materialization_and_child_configuration( ): import subprocess import sys - from openhcs.core.config_document import ConfigDocumentAuthority + from openhcs.core.config_document import ConfigDocumentCodec from openhcs.runtime import zmq_execution_client as module from openhcs.runtime.import_authority import OpenHCSRuntimeImportAuthority @@ -172,7 +172,7 @@ def test_admitted_plan_owns_materialization_and_child_configuration( "openhcs.runtime.zmq_execution_server_launcher" ) ) - child_config = ConfigDocumentAuthority.from_source( + child_config = ConfigDocumentCodec.from_source( command[command.index("--config-source") + 1], expected_config_type=OpenHCSZMQConfig, ) diff --git a/tests/unit/pyqt_gui/test_config_window_state_resolution.py b/tests/unit/pyqt_gui/test_config_window_state_resolution.py index 342decb5ad..9301cdb789 100644 --- a/tests/unit/pyqt_gui/test_config_window_state_resolution.py +++ b/tests/unit/pyqt_gui/test_config_window_state_resolution.py @@ -30,7 +30,7 @@ GlobalPipelineConfig, PipelineConfig, ) -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.pyqt_gui.config import UIConfig, get_default_ui_config from openhcs.pyqt_gui.services.ui_window_ids import OpenHCSUiWindowId from openhcs.pyqt_gui.services.window_handlers import ( @@ -195,11 +195,11 @@ def test_ui_config_pycodify_round_trip_preserves_exact_zmq_type() -> None: ports_per_server_type=4, ), ) - source = ConfigDocumentAuthority.render( + source = ConfigDocumentCodec.render( config, expected_config_type=UIConfig, ) - restored = ConfigDocumentAuthority.from_source( + restored = ConfigDocumentCodec.from_source( source, expected_config_type=UIConfig, ) @@ -292,7 +292,7 @@ def test_config_window_page_count_and_visual_scope_own_presentation(qapp) -> Non } code_document = plate_window.window_code_document_driver().read_document() assert ( - ConfigDocumentAuthority.from_source( + ConfigDocumentCodec.from_source( code_document.source, expected_config_type=PipelineConfig, ) @@ -544,7 +544,7 @@ def test_pipeline_config_code_document_emits_only_authored_override(qapp) -> Non "partition_filter_config.partition_filter", } assert source == authored_source - restored = ConfigDocumentAuthority.from_source( + restored = ConfigDocumentCodec.from_source( source, expected_config_type=PipelineConfig, ) diff --git a/tests/unit/pyqt_gui/test_ui_agent_bridge.py b/tests/unit/pyqt_gui/test_ui_agent_bridge.py index a930690b19..62b875e237 100644 --- a/tests/unit/pyqt_gui/test_ui_agent_bridge.py +++ b/tests/unit/pyqt_gui/test_ui_agent_bridge.py @@ -120,7 +120,7 @@ LazyPathPlanningConfig, PipelineConfig, ) -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.debug import ( DebugCursor, DebugProgressContext, @@ -306,7 +306,7 @@ def replacement_function(image, threshold: int = 2): def test_config_code_document_honors_clean_flag() -> None: driver = ConfigCodeDocumentDriver( - document_format=ConfigDocumentAuthority, + document_format=ConfigDocumentCodec, title="View/Edit GlobalPipelineConfig", config_type=GlobalPipelineConfig, current_config=lambda: GlobalPipelineConfig(num_workers=2), @@ -325,7 +325,7 @@ def test_pycodified_config_document_delegates_to_config_authority() -> None: replacement = GlobalPipelineConfig(num_workers=3) applied: list[GlobalPipelineConfig] = [] driver = ConfigCodeDocumentDriver( - document_format=ConfigDocumentAuthority, + document_format=ConfigDocumentCodec, title="View/Edit GlobalPipelineConfig", config_type=GlobalPipelineConfig, current_config=lambda: original, @@ -333,7 +333,7 @@ def test_pycodified_config_document_delegates_to_config_authority() -> None: ) document = driver.read_document() - replacement_source = ConfigDocumentAuthority.render( + replacement_source = ConfigDocumentCodec.render( replacement, expected_config_type=GlobalPipelineConfig, ) @@ -341,7 +341,7 @@ def test_pycodified_config_document_delegates_to_config_authority() -> None: driver.apply_source(replacement_source) assert ( - ConfigDocumentAuthority.from_source( + ConfigDocumentCodec.from_source( document.source, expected_config_type=GlobalPipelineConfig, ) @@ -353,7 +353,7 @@ def test_pycodified_config_document_delegates_to_config_authority() -> None: def test_pycodified_config_document_authorizes_before_apply() -> None: applied: list[GlobalPipelineConfig] = [] driver = ConfigCodeDocumentDriver( - document_format=ConfigDocumentAuthority, + document_format=ConfigDocumentCodec, title="View/Edit GlobalPipelineConfig", config_type=GlobalPipelineConfig, current_config=GlobalPipelineConfig, @@ -370,7 +370,7 @@ def test_pycodified_config_document_authorizes_before_apply() -> None: def test_pyqt_codegen_provider_delegates_config_documents() -> None: config = GlobalPipelineConfig(num_workers=4) - source = ConfigDocumentAuthority.render( + source = ConfigDocumentCodec.render( config, expected_config_type=GlobalPipelineConfig, ) @@ -382,7 +382,7 @@ def test_pyqt_codegen_provider_delegates_config_documents() -> None: ) assert ( - ConfigDocumentAuthority.from_source( + ConfigDocumentCodec.from_source( normalized_source, expected_config_type=GlobalPipelineConfig, ) diff --git a/tests/unit/test_cellprofiler_public_boundary_contracts.py b/tests/unit/test_cellprofiler_public_boundary_contracts.py index 73c03c7160..0a47e923ec 100644 --- a/tests/unit/test_cellprofiler_public_boundary_contracts.py +++ b/tests/unit/test_cellprofiler_public_boundary_contracts.py @@ -8,7 +8,6 @@ from pathlib import Path from openhcs.core.function_step_transport import FunctionStepTransportAuthority -from openhcs.core.pipeline_document_fields import PipelineDocumentField from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends import cellprofiler as cellprofiler_backend from openhcs.processing.backends.cellprofiler.illumination import ( @@ -43,12 +42,6 @@ def test_raw_callable_signatures_expose_behavior_not_canonical_identity() -> Non assert "output_name" in calculate_math_parameters -def test_step_fragment_transport_uses_pipeline_document_field_authority() -> None: - assert callable(FunctionStepTransportAuthority.source_from_pipeline) - assert callable(FunctionStepTransportAuthority.pipeline_steps_from_namespace) - assert PipelineDocumentField.PIPELINE_STEPS.value == "pipeline_steps" - - def test_pycodified_public_cp_pipeline_reconstructs_in_fresh_process( tmp_path: Path, ) -> None: diff --git a/tests/unit/test_config_document.py b/tests/unit/test_config_document.py index 157caefc62..73cc20d22a 100644 --- a/tests/unit/test_config_document.py +++ b/tests/unit/test_config_document.py @@ -13,10 +13,7 @@ TiffCompression, TiffConfig, ) -from openhcs.core.config_document import ( - ConfigDocumentAuthority, - ConfigDocumentField, -) +from openhcs.core.config_document import ConfigDocumentCodec @dataclass(frozen=True) @@ -24,10 +21,6 @@ class _ProjectionRoot: target: timedelta = field(default_factory=lambda: timedelta(seconds=1)) -def test_config_document_field_owns_exact_public_name() -> None: - assert tuple(field.value for field in ConfigDocumentField) == ("config",) - - @pytest.mark.parametrize( "config", ( @@ -44,11 +37,11 @@ def test_config_document_field_owns_exact_public_name() -> None: def test_config_document_round_trip(config: object) -> None: config_type = type(config) - source = ConfigDocumentAuthority.render( + source = ConfigDocumentCodec.render( config, expected_config_type=config_type, ) - restored = ConfigDocumentAuthority.from_source( + restored = ConfigDocumentCodec.from_source( source, expected_config_type=config_type, ) @@ -60,7 +53,7 @@ def test_config_document_round_trip(config: object) -> None: def test_config_document_from_namespace_requires_exact_field() -> None: with pytest.raises(ValueError, match="'config'"): - ConfigDocumentAuthority.from_namespace( + ConfigDocumentCodec.from_namespace( {"pipeline_config": PipelineConfig()}, expected_config_type=PipelineConfig, ) @@ -68,20 +61,20 @@ def test_config_document_from_namespace_requires_exact_field() -> None: def test_config_document_rejects_wrong_expected_type() -> None: with pytest.raises(TypeError, match="PipelineConfig"): - ConfigDocumentAuthority.from_namespace( + ConfigDocumentCodec.from_namespace( {"config": GlobalPipelineConfig()}, expected_config_type=PipelineConfig, ) with pytest.raises(TypeError, match="PipelineConfig"): - ConfigDocumentAuthority.render( + ConfigDocumentCodec.render( GlobalPipelineConfig(), expected_config_type=PipelineConfig, ) def test_config_document_projects_one_explicit_dataclass_field() -> None: - projected = ConfigDocumentAuthority.project_dataclass_field_from_source( + projected = ConfigDocumentCodec.project_dataclass_field_from_source( "from datetime import timedelta\nconfig = Root(target=timedelta(seconds=2))", expected_config_type=_ProjectionRoot, expected_field_type=timedelta, @@ -91,7 +84,7 @@ def test_config_document_projects_one_explicit_dataclass_field() -> None: def test_config_document_projection_uses_the_declared_field_default() -> None: - projected = ConfigDocumentAuthority.project_dataclass_field_from_source( + projected = ConfigDocumentCodec.project_dataclass_field_from_source( "config = Root()", expected_config_type=_ProjectionRoot, expected_field_type=timedelta, @@ -102,7 +95,7 @@ def test_config_document_projection_uses_the_declared_field_default() -> None: def test_config_document_projection_rejects_the_wrong_field_type() -> None: with pytest.raises(TypeError, match="target.*timedelta.*list"): - ConfigDocumentAuthority.project_dataclass_field_from_source( + ConfigDocumentCodec.project_dataclass_field_from_source( 'config = Root(target=["one"])', expected_config_type=_ProjectionRoot, expected_field_type=timedelta, diff --git a/tests/unit/test_dataset_source_family.py b/tests/unit/test_dataset_source_family.py index 145735000b..70a3d65dfb 100644 --- a/tests/unit/test_dataset_source_family.py +++ b/tests/unit/test_dataset_source_family.py @@ -14,7 +14,7 @@ from openhcs.agent.services.artifact_plan_inspection_service import ArtifactPlanInspectionService from openhcs.agent.services.plate_inspection_service import PlateInspectionService from openhcs.core.config import LazyPartitionFilterConfig, PipelineConfig -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.pipeline_document import PipelineDocumentCodec from openhcs.core.steps.function_step import FunctionStep from openhcs.core.dataset_sources.source import ( @@ -125,7 +125,7 @@ def test_config_schema_patch_and_source_share_opera_handler_identity() -> None: assert config.dataset_source is OperaPhenixHandler assert "dataset_source=OperaPhenixHandler" in rendered.source assert ( - ConfigDocumentAuthority.from_source( + ConfigDocumentCodec.from_source( rendered.source, expected_config_type=PipelineConfig, ).dataset_source diff --git a/tests/unit/test_e2_restated_declaration_guards.py b/tests/unit/test_e2_restated_declaration_guards.py new file mode 100644 index 0000000000..64f8d40c58 --- /dev/null +++ b/tests/unit/test_e2_restated_declaration_guards.py @@ -0,0 +1,126 @@ +"""E2 guards: the deleted restatements stay deleted.""" + +from __future__ import annotations + +import ast +import re +import sys +from dataclasses import is_dataclass +from pathlib import Path + +from pycodify import SourceDocument + +import openhcs.core.config_document +import openhcs.core.function_step_document +import openhcs.core.pipeline_document # noqa: F401 + +REPO = Path(__file__).resolve().parents[2] +PRODUCT = REPO / "openhcs" + +DELETED_NAMES = ( + "ConfigDocumentField", + "PipelineDocumentField", + "FunctionStepDocumentField", + "ConfigDocumentAuthority", + "McpReconnectOwner", + "RuntimeBatchCallableMetadataField", + "RUNTIME_BATCH_EXECUTORS_ATTR", + "CellProfilerSpreadsheetRowField", + "ConfigFieldRequirementLabel", + "ConfigFieldResolutionLabel", +) + + +def _product_sources() -> list[tuple[Path, str]]: + return [(path, path.read_text(encoding="utf-8")) for path in sorted(PRODUCT.rglob("*.py"))] + + +def _document_types(root: type = SourceDocument) -> list[type]: + found: list[type] = [] + for subclass in root.__subclasses__(): + # dataclass(slots=True) leaves the class it replaced in __subclasses__. + module = sys.modules.get(subclass.__module__) + if subclass.__module__.startswith("openhcs.") and ( + getattr(module, subclass.__name__, None) is subclass + ): + found.append(subclass) + found.extend(_document_types(subclass)) + return found + + +def test_deleted_names_do_not_return() -> None: + pattern = re.compile(r"\b(" + "|".join(DELETED_NAMES) + r")\b") + offenders = [ + f"{path.relative_to(REPO)}: {match.group(1)}" + for path, text in _product_sources() + for match in pattern.finditer(text) + ] + assert offenders == [] + assert not (PRODUCT / "core" / "pipeline_document_fields.py").exists() + + +def test_document_field_names_are_spelled_only_by_their_dataclass() -> None: + """No product module passes a document field name to ``Assignment`` as a literal.""" + + names = {name for kind in _document_types() for name in kind.assignment_names()} + assert names == {"config", "pipeline_config", "pipeline_steps", "step"} + offenders = [] + for path, text in _product_sources(): + if "Assignment" not in text: + continue + for node in ast.walk(ast.parse(text)): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) + and node.func.id == "Assignment" + and node.args + and isinstance(node.args[0], ast.Constant) + and node.args[0].value in names + ): + offenders.append(f"{path.relative_to(REPO)}:{node.lineno}") + assert offenders == [] + + +def test_every_source_document_is_a_dataclass() -> None: + kinds = _document_types() + assert kinds + assert [kind.__name__ for kind in kinds if not is_dataclass(kind)] == [] + + +def test_no_one_member_enum_was_added() -> None: + """One-member enums hold one fact. The listed ones spell an external contract.""" + + external_spellings = { + "McpWireContentType": "MCP tool-result content block type", + "DesktopInstallerSchemaVersion": "installer contract read by packaging/installers scripts", + "SaveImagesImageKind": "CellProfiler SaveImages setting", + "AdditionalMode": "CellProfiler Align setting", + "PlateInspectionWorkflowScope": "MCP plate-inspection result schema", + "FormattingMeasurementFeatureTemplate": "CellProfiler feature-name template (K3)", + "_ClassificationResultFieldRole": "Annotated field marker (K3 row-role family)", + "_AlignShiftFieldRole": "Annotated field marker (K3 row-role family)", + } + enum_bases = {"Enum", "IntEnum", "StrEnum", "Flag", "IntFlag"} + found = set() + for _path, text in _product_sources(): + if "Enum" not in text and "Flag" not in text: + continue + for node in ast.walk(ast.parse(text)): + if not isinstance(node, ast.ClassDef): + continue + base_names = {ast.unparse(base).rsplit(".", 1)[-1] for base in node.bases} + if not base_names & enum_bases: + continue + members = [ + statement + for statement in node.body + if isinstance(statement, ast.Assign) + and all( + isinstance(target, ast.Name) and not target.id.startswith("_") + for target in statement.targets + ) + ] + if len(members) == 1: + found.add(node.name) + assert found - set(external_spellings) == set() + assert set(external_spellings) - found == set(), "delete the entry with its enum" diff --git a/tests/unit/test_function_step_document.py b/tests/unit/test_function_step_document.py index b822b262e5..34a9b2f999 100644 --- a/tests/unit/test_function_step_document.py +++ b/tests/unit/test_function_step_document.py @@ -7,7 +7,6 @@ from openhcs.core.function_step_document import ( FunctionStepDocument, FunctionStepDocumentCodec, - FunctionStepDocumentField, ) from openhcs.core.function_step_transport import FunctionStepTransportAuthority from openhcs.core.steps.function_step import FunctionStep @@ -20,10 +19,6 @@ def _step() -> FunctionStep: return FunctionStep(func=cellprofiler_backend.crop, name="Crop") -def test_function_step_document_field_owns_exact_public_name() -> None: - assert tuple(field.value for field in FunctionStepDocumentField) == ("step",) - - def test_default_function_patterns_are_not_shared_between_steps() -> None: first = FunctionStep() second = FunctionStep() diff --git a/tests/unit/test_openhcs_adapter.py b/tests/unit/test_openhcs_adapter.py index 766cc3ac75..02b2bbf225 100644 --- a/tests/unit/test_openhcs_adapter.py +++ b/tests/unit/test_openhcs_adapter.py @@ -42,9 +42,9 @@ PartitionFilterConfig, ) from openhcs.domains.microscopy.config import AnalysisConsolidationConfig -from openhcs.core.config_document import ConfigDocumentAuthority +from openhcs.core.config_document import ConfigDocumentCodec from openhcs.core.pipeline_document import PipelineDocumentCodec -from openhcs.core.pipeline_document_fields import PipelineDocumentField +from openhcs.core.function_step_transport import PipelineStepsDocument from openhcs.core.runtime_execution_validation import ( RuntimeArtifactAxisExpectation, RuntimeArtifactExecutionExpectation, @@ -388,7 +388,7 @@ def poll_status(self, execution_id): == FakeZMQExecutionClient.submitted[1].pipeline_code ) assert all( - submission.global_config_code == ConfigDocumentAuthority.render( + submission.global_config_code == ConfigDocumentCodec.render( global_config, expected_config_type=GlobalPipelineConfig ) for submission in FakeZMQExecutionClient.submitted @@ -624,5 +624,5 @@ def test_benchmark_submission_matches_pyqt_submission_payload() -> None: for target in node.targets if isinstance(target, ast.Name) } - assert PipelineDocumentField.PIPELINE_STEPS.value in assigned_names + assert set(PipelineStepsDocument.assignment_names()) <= assigned_names assert "__openhcs_step_invocation_contracts" not in assigned_names diff --git a/tests/unit/test_pipeline_document.py b/tests/unit/test_pipeline_document.py index b49962d9f3..e6c8a185ff 100644 --- a/tests/unit/test_pipeline_document.py +++ b/tests/unit/test_pipeline_document.py @@ -9,7 +9,6 @@ from openhcs.core.pipeline_document import ( PipelineDocument, PipelineDocumentCodec, - PipelineDocumentField, ) from openhcs.core.steps.function_step import FunctionStep from openhcs.processing.backends import cellprofiler as cellprofiler_backend @@ -20,13 +19,6 @@ def _step() -> FunctionStep: return FunctionStep(func=cellprofiler_backend.crop, name="Crop") -def test_pipeline_document_fields_own_exact_public_names() -> None: - assert tuple(field.value for field in PipelineDocumentField) == ( - "pipeline_config", - "pipeline_steps", - ) - - def test_pipeline_document_round_trip_renders_config_and_normalized_steps() -> None: document = PipelineDocumentCodec.from_values( pipeline_config=PipelineConfig(), @@ -45,7 +37,7 @@ def test_pipeline_document_round_trip_renders_config_and_normalized_steps() -> N cellprofiler_backend.crop ) assert PipelineDocumentCodec.render(restored) == source - assert restored.original_source == source + assert restored.source_text == source assert PipelineDocumentCodec.execution_source(restored) == source @@ -87,7 +79,7 @@ def test_execution_source_preserves_reviewed_source_without_parallel_state() -> document = PipelineDocumentCodec.from_source(reviewed_source) - assert document.original_source == reviewed_source + assert document.source_text == reviewed_source assert PipelineDocumentCodec.execution_source(document) == reviewed_source diff --git a/tests/unit/test_source_documents.py b/tests/unit/test_source_documents.py new file mode 100644 index 0000000000..ab25246f5c --- /dev/null +++ b/tests/unit/test_source_documents.py @@ -0,0 +1,117 @@ +"""Every saved Python document assigns its dataclass fields and loads them back.""" + +from __future__ import annotations + +import ast +import sys + +import pytest +from pycodify import SourceDocument + +import openhcs.serialization.pycodify_formatters # noqa: F401 +from openhcs.core.config import GlobalPipelineConfig, PipelineConfig +from openhcs.core.config_document import ConfigDocument +from openhcs.core.function_step_document import FunctionStepDocumentCodec +from openhcs.core.function_step_transport import ( + FunctionStepTransportAuthority, + PipelineStepsDocument, +) +from openhcs.core.pipeline_document import PipelineDocument, PipelineDocumentCodec +from openhcs.core.steps.function_step import FunctionStep +from openhcs.processing.backends import cellprofiler as cellprofiler_backend + + +def _steps() -> list[FunctionStep]: + return FunctionStepTransportAuthority.normalize_pipeline( + [FunctionStep(func=cellprofiler_backend.crop, name="Crop")] + ) + + +def _document_types(root: type = SourceDocument) -> list[type[SourceDocument]]: + found: list[type[SourceDocument]] = [] + for subclass in root.__subclasses__(): + # dataclass(slots=True) leaves the class it replaced in __subclasses__. + module = sys.modules.get(subclass.__module__) + if subclass.__module__.startswith("openhcs.") and ( + getattr(module, subclass.__name__, None) is subclass + ): + found.append(subclass) + found.extend(_document_types(subclass)) + return found + + +# The names inside saved files. They are durable: a user's saved pipeline, +# step or config must keep loading, so a field rename has to fail here. +SAVED_NAMES = { + "ConfigDocument": ("config",), + "PipelineStepsDocument": ("pipeline_steps",), + "PipelineDocument": ("pipeline_steps", "pipeline_config"), + "FunctionStepDocument": ("step",), +} + + +def _sample(document_type: type[SourceDocument]) -> SourceDocument: + samples = { + "ConfigDocument": lambda: ConfigDocument(config=GlobalPipelineConfig(num_workers=3)), + "PipelineStepsDocument": lambda: PipelineStepsDocument(pipeline_steps=_steps()), + "PipelineDocument": lambda: PipelineDocumentCodec.from_values( + pipeline_config=PipelineConfig(), pipeline_steps=_steps() + ), + "FunctionStepDocument": lambda: FunctionStepDocumentCodec.from_value(_steps()[0]), + } + return samples[document_type.__name__]() + + +def _assigned_names(source: str) -> tuple[str, ...]: + return tuple( + target.id + for statement in ast.parse(source).body + if isinstance(statement, ast.Assign) + for target in statement.targets + if isinstance(target, ast.Name) + ) + + +def test_every_document_kind_is_covered() -> None: + assert {kind.__name__ for kind in _document_types()} == set(SAVED_NAMES) + + +@pytest.mark.parametrize("document_type", _document_types(), ids=lambda kind: kind.__name__) +@pytest.mark.parametrize("clean_mode", (True, False)) +def test_document_round_trips_under_its_field_names( + document_type: type[SourceDocument], clean_mode: bool +) -> None: + document = _sample(document_type) + + source = document.to_source(clean_mode=clean_mode) + restored = document_type.from_source(source) + + assert document_type.assignment_names() == SAVED_NAMES[document_type.__name__] + assert _assigned_names(source) == SAVED_NAMES[document_type.__name__] + assert type(restored) is document_type + assert restored.to_source(clean_mode=clean_mode) == source + + +def test_config_document_restores_an_equal_config() -> None: + document = _sample(ConfigDocument) + + assert ConfigDocument.from_source(document.to_source(clean_mode=True)) == document + + +def test_steps_only_file_is_a_pipeline_with_the_default_config() -> None: + steps_source = _sample(PipelineStepsDocument).to_source(clean_mode=True) + + pipeline = PipelineDocumentCodec.from_source(steps_source) + + assert isinstance(pipeline.pipeline_config, PipelineConfig) + assert len(pipeline.pipeline_steps) == 1 + assert pipeline.source_text == steps_source + + +def test_pipeline_file_gives_its_steps_to_the_steps_only_reader() -> None: + namespace: dict[str, object] = {} + exec(PipelineDocumentCodec.render(_sample(PipelineDocument)), namespace) # noqa: S102 + + (step,) = FunctionStepTransportAuthority.pipeline_steps_from_namespace(namespace) + + assert step.name == "Crop"