L6: code generation owned by pycodify and objectstate.codegen

docs/refactor/consolidation-20261009/L6-codegen.md

@@ -0,0 +1,77 @@
+# L6: pycodify and ObjectState own Python source generation
+
+**Head audited:** `openhcs` `main` at `7f41c419c` (#1197); `pycodify` `main` at `108a8ed` (0.1.3 plus the unreleased nominal-subtype fix); `ObjectState` `main` at `19aafdc` (2.0.0). **Rules:** [00-RULES.md](00-RULES.md). **Step 4.**
+**Shared abstractions** ([02-SHARED-ABSTRACTIONS.md](02-SHARED-ABSTRACTIONS.md)). *Builds:* none. *Uses:* none.
+
+Pipelines are saved as `.py` files generated by pycodify (pickle `.pipeline` files are deprecated), so code generation is the persistence format: a formatter bug is a data-loss bug.
+
+## What is wrong
+
+**pycodify writes values it cannot read back, and OpenHCS fixes this with private copies of generic mechanisms instead of fixing the library.**
+
+pycodify `formatters.py` (verified with a script at this head):
+- `EnumFormatter` (`:77-87`) imports `cls.__name__`: a member of `Outer.Mode` renders `from m import Mode` / `Mode.B`, an `ImportError` on load.
+- `_format_dataclass_expression` (`:285-287`) and `CallableFormatter` (`:129-130`) make the same `__name__` mistake for nested dataclasses and callables.
+- Classes have no formatter: `CallableFormatter` excludes types, so a class value falls to `FallbackFormatter` and renders as `<class 'm.X'>`, a `SyntaxError`. `FallbackFormatter` (`:352-359`) emits `repr` for any value, so every unknown type is silent garbage.
+- Non-clean mode replaces a `None` field whose annotation is a dataclass with a placeholder instance of all-`None` fields (`:310-314`, `_create_placeholder_dataclass_instance`): `Optional[Inner] = None` saves as `Inner(x=None)`, a different value.
+- `init=False` fields are emitted as constructor keywords (`:300`): a `TypeError` on load.
+- ObjectState's lazy semantics are duck-typed into the generic formatter: `hasattr(instance, "_resolve_field_value")`, `hasattr(cls, "_is_lazy_dataclass")` (`:251, 258, 262, 303`).
+
+OpenHCS copies (`openhcs/serialization/`, 747 lines; `openhcs/core/python_source_literal.py`, 25):
+- `EnumMemberFormatter` is the nested-enum fix, written in OpenHCS (`pycodify_formatters.py:131-149`); `AxisDeclarationFormatter` (`:69-82`) is the same qualified-name rule for axis classes; `OpenHCSImportableTypeFormatter` (`:54-66`) supplies the missing class formatter through a `FunctionReference`; `NameMappingLookup` (`:25-34`) restates `Mapping.get`. The qualified-name rule is written three times.
+- `PythonSourceLiteral` (`core/python_source_literal.py`) and its formatter (`pycodify_formatters.py:85-97`): a generic "value owns its source expression" hook, with three methods (`source_literal`, `source_literal_imports`, `source_literal_with_names`) for one fact.
+- `source_path_factoring.py` (195 lines) and `OpenHCSPathFormatter` (`:100-128`): document-level path factoring, generic, plugged in through pycodify's type-keyed `FormatContext.extensions` bag, whose only user it is. `OpenHCSPythonSourceDocument` restates `generate_python_source`; six production callers construct it.
+- The LazyDataclass formatter (`pycodify_formatters.py:313-481`, 170 lines: `LazyDataclassFieldEmissionState`, `LazyDataclassSerializedField`, `LazyDataclassSerializationPlan`, `LazyDataclassFormatEligibility`, `LazyDataclassFormatter`, and `FunctionStepCleanModeFieldPolicy`): ObjectState's raw-value and explicitly-set-field semantics, read from `__dict__["_explicitly_set_fields"]` outside ObjectState.
+- `OpenHCSCallableFormatter.format` (`:46-51`) catches `RuntimeError` and emits `repr(function)`: an unreferenceable callable saves as `<function f at 0x…>`.
+
+ObjectState: field-name-gated debug logging names OpenHCS's old `well_filter` field (and `num_workers`, `sub_dir`, `output_dir_suffix`) in `dual_axis_resolver.py`, `object_state.py`, `object_state_registry.py`, `context_manager.py` (G7 leftover); docstrings use `WellFilterConfig` examples.
+
+## Target
+
+pycodify 0.2.0:
+- `imported_name(module, qualname, context) -> SourceFragment`: the one qualified-name rule (import the root of `__qualname__`, spell the rest as attributes, honour name mappings, no import for builtins, `TypeError` for `<locals>`). Enum members, classes (new `TypeFormatter`), callables and dataclass constructors all use it.
+- `PythonSourceLiteral(ABC)` with one method, `source_fragment(context) -> SourceFragment`, and its formatter.
+- `paths.py`: a `PathRendering` family on `FormatContext.paths` (plain, collecting, factored). `generate_python_source` always factors absolute paths (discovery pass, plan, render). `FormatContext.extensions`/`extension()` are deleted.
+- Deleted: `FallbackFormatter` (unformattable values raise `TypeError`), the placeholder-instance substitution, the lazy `hasattr` probes; `init=False` fields are skipped. `frozenset`, `bytes` and `complex` get exact formatters.
+
+ObjectState 2.1.0 (adds a module and a `pycodify` dependency; removes only private debug branches):
+- `objectstate.codegen`: `LazyResolutionFormatter` for every `LazyResolutionDataclass` (raw values; clean mode writes fields whose raw value differs from the field default), refined by `LazyDataclassFormatter` (constructor-set fields), and `field_needs_source(current, default, context)` for owners such as `FunctionStep` that format lazy-config fields.
+- The field-name-gated debug branches are deleted; docstring examples renamed to `partition_filter`.
+
+OpenHCS:
+- Deleted: `openhcs/core/python_source_literal.py`, `openhcs/serialization/source_path_factoring.py`, `NameMappingLookup`, `EnumMemberFormatter`, `AxisDeclarationFormatter`, `OpenHCSImportableTypeFormatter`, `PythonSourceLiteralFormatter`, `OpenHCSPathFormatter`, the six LazyDataclass classes and `FunctionStepCleanModeFieldPolicy`, the `repr` catch in `OpenHCSCallableFormatter`.
+- `FunctionReference` subclasses pycodify's `PythonSourceLiteral`; the six document callers call `generate_python_source`.
+
+**Found by the round-trip test.** Base configs that inherit through `None` (`StepPartitionFilterConfig`, `PathPlanningConfig`, the streaming configs and 5 others) are `LazyResolutionDataclass` but not `LazyDataclass`; pycodify's generic dataclass formatter wrote their *resolved* values in full mode, so a saved `GlobalPipelineConfig` stopped inheriting (for example `step_partition_filter_config.partition_filter_mode` was baked to `IncludePartitions`). Fixed by `LazyResolutionFormatter`.
+
+## Persisted state
+
+Saved `.py` pipelines and configs are durable. Output that loaded before still loads: the qualified-name, class and path spellings are unchanged for every value that round-tripped; values that rendered unloadable source now render loadable source or raise at save time. No cutover tool.
+
+## Guards
+
+`tests/unit/test_l6_codegen_guards.py`:
+- `openhcs/core/python_source_literal.py` and `openhcs/serialization/source_path_factoring.py` do not exist.
+- No `SourceFormatter` subclass under `openhcs/` handles `Enum`, `Path`, `type`, `PythonSourceLiteral` or `LazyDataclass` (AST: no `isinstance(value, Enum|Path|LazyDataclass|PythonSourceLiteral)` and no `inspect.isclass` in a `can_format`).
+- No `__qualname__.partition` and no `_explicitly_set_fields` under `openhcs/`.
+- No string `well_filter` in `external/ObjectState/src`.
+
+## Tests
+
+- OpenHCS: one family-level round-trip test (`tests/unit/test_source_round_trip.py`): every registered config dataclass (`LazyDataclassFactory.registered_type_pairs()`, both lazy and base), each with every field set to a non-default value derived from its annotation (nested enums, partition-filter mode family members, axis classes, paths), plus every function-pattern shape the step editor produces (callable, `(callable, kwargs)`, chain, dict by group-by value, dict of chains) inside `FunctionStep`s inside a pipeline document, in clean and full mode, re-imported and compared for equality. The narrow formatter tests it covers are deleted; tests of deleted modules move to the libraries.
+- pycodify: the generic tests (qualified names, classes, path factoring, `PythonSourceLiteral`, unformattable values raise).
+- ObjectState: lazy formatter tests (clean omission, explicit empty configs, full mode keeps `None`, round trip).
+
+## New-case experiments
+
+- A new nested enum or axis class in a config: today it needs an OpenHCS formatter at a priority above pycodify's broken one. After: nothing.
+- A new value type that owns its source spelling: today it subclasses an OpenHCS ABC with three methods. After: one `source_fragment` method on pycodify's ABC.
+- A second application persisting ObjectState configs as source: today it must copy 170 lines out of OpenHCS. After: `import objectstate.codegen`.
+
+## Release order
+
+pycodify 0.2.0, then ObjectState 2.1.0 (depends on `pycodify>=0.2.0,<0.3`), then the OpenHCS repin (`pycodify>=0.2.0,<0.3`, `objectstate>=2.1,<3`).
+
+## Done when
+
+The OpenHCS copies are gone, the guards and the round-trip test pass, the pycodify and ObjectState PRs are open with their tests green, and the OpenHCS submodules point at those branch heads.

openhcs/core/config_document.py

@@ -162,19 +162,16 @@ def render(
     ) -> str:
         """Render a validated config object as reviewable Python source."""
 
-        from pycodify import Assignment
+        from pycodify import Assignment, generate_python_source
 
         import openhcs.serialization.pycodify_formatters  # noqa: F401
-        from openhcs.serialization.source_path_factoring import (
-            OpenHCSPythonSourceDocument,
-        )
 
         cls._require_config_type(config, expected_config_type)
-        return OpenHCSPythonSourceDocument(
+        return generate_python_source(
             Assignment(ConfigDocumentField.CONFIG.value, config),
             header=cls.HEADER,
             clean_mode=clean_mode,
-        ).render()
+        )
 
     @staticmethod
     def _require_config_type(

openhcs/core/function_reference.py

@@ -2,17 +2,17 @@
 
 from __future__ import annotations
 
-import builtins
 import dataclasses
 import importlib
 import inspect
 import logging
 from abc import ABC, abstractmethod
-from collections.abc import Callable, Mapping, Sequence
+from collections.abc import Callable, Sequence
 from dataclasses import dataclass
 from types import ModuleType
 from typing import TYPE_CHECKING, ClassVar
 
+from pycodify import FormatContext, PythonSourceLiteral, SourceFragment, imported_name
 from python_introspect import Enableable
 
 from openhcs.core.callable_contract import (
@@ -21,7 +21,6 @@
     FunctionStepExecutionScope,
 )
 from openhcs.core.function_contract_metadata import FunctionContractAttribute
-from openhcs.core.python_source_literal import PythonSourceLiteral
 
 if TYPE_CHECKING:
     from openhcs.core.steps.abstract import AbstractStep
@@ -75,24 +74,10 @@ def source_import_identity(self) -> CallableImportIdentity:
     def source_expression(self, imported_name: str) -> str:
         """Render through the source authority, including pycodify name aliases."""
 
-    def source_literal(self) -> str:
-        return self.source_expression(self.source_import_identity.function_name)
-
-    def source_literal_with_names(
-        self, name_mappings: Mapping[tuple[str, str], str],
-    ) -> str:
-        identity = self.source_import_identity
-        imported_name = name_mappings.get(
-            (identity.module_name, identity.function_name), identity.function_name,
-        )
-        return self.source_expression(imported_name)
-
-    def source_literal_imports(self) -> frozenset[tuple[str, str]]:
-        imports = super().source_literal_imports()
+    def source_fragment(self, context: FormatContext) -> SourceFragment:
         identity = self.source_import_identity
-        if identity.module_name == builtins.__name__:
-            return imports
-        return imports | frozenset({(identity.module_name, identity.function_name)})
+        name = imported_name(identity.module_name, identity.function_name, context)
+        return SourceFragment(self.source_expression(name.code), name.imports)
 
     @abstractmethod
     def resolve(self) -> Callable:

openhcs/core/function_step_document.py

@@ -68,17 +68,14 @@ def render(
         """Render a validated FunctionStep document as reviewable Python source."""
 
         import openhcs.serialization.pycodify_formatters  # noqa: F401
-        from pycodify import Assignment
-        from openhcs.serialization.source_path_factoring import (
-            OpenHCSPythonSourceDocument,
-        )
+        from pycodify import Assignment, generate_python_source
 
         normalized = cls.from_value(document.step)
-        return OpenHCSPythonSourceDocument(
+        return generate_python_source(
             Assignment(FunctionStepDocumentField.STEP.value, normalized.step),
             header=cls.HEADER,
             clean_mode=clean_mode,
-        ).render()
+        )
 
     @staticmethod
     def _require_step(value: object) -> None:

openhcs/core/function_step_transport.py

@@ -36,19 +36,16 @@ def source_from_pipeline(
     ) -> str:
         """Normalize and pycodify one public FunctionStep list."""
         import openhcs.serialization.pycodify_formatters  # noqa: F401
-        from pycodify import Assignment
-        from openhcs.serialization.source_path_factoring import (
-            OpenHCSPythonSourceDocument,
-        )
+        from pycodify import Assignment, generate_python_source
 
         steps = list(pipeline_steps)
         cls._require_function_steps(steps)
         normalized = cls.normalize_pipeline(steps)
-        return OpenHCSPythonSourceDocument(
+        return generate_python_source(
             Assignment(PipelineDocumentField.PIPELINE_STEPS.value, normalized),
             header="# Edit this pipeline and save to apply changes",
             clean_mode=clean_mode,
-        ).render()
+        )
 
     @classmethod
     def pipeline_steps_from_namespace(

openhcs/core/pipeline_document.py

@@ -98,10 +98,7 @@ def render(
         """Render a validated pipeline document as reviewable Python source."""
 
         import openhcs.serialization.pycodify_formatters  # noqa: F401
-        from pycodify import Assignment, BlankLine, CodeBlock
-        from openhcs.serialization.source_path_factoring import (
-            OpenHCSPythonSourceDocument,
-        )
+        from pycodify import Assignment, BlankLine, CodeBlock, generate_python_source
 
         normalized = cls.from_values(
             pipeline_config=document.pipeline_config,
@@ -120,11 +117,11 @@ def render(
                 ),
             )
         )
-        return OpenHCSPythonSourceDocument(
+        return generate_python_source(
             body,
             header=cls.HEADER,
             clean_mode=clean_mode,
-        ).render()
+        )
 
     @staticmethod
     def _require_pipeline_config(value: object) -> None:

openhcs/pyqt_gui/services/reactor_providers.py

@@ -69,17 +69,13 @@ def render_assignment(
         header: str,
         clean_mode: bool,
     ) -> str:
-        from pycodify import Assignment
+        from pycodify import Assignment, generate_python_source
 
-        from openhcs.serialization.source_path_factoring import (
-            OpenHCSPythonSourceDocument,
-        )
-
-        return OpenHCSPythonSourceDocument(
+        return generate_python_source(
             Assignment(assignment_name, value),
             header=header,
             clean_mode=clean_mode,
-        ).render()
+        )
 
     def normalize_source(
         self,