Re/03 core schema

docs/SCHEMA_HISTORY.md

@@ -1,10 +1,113 @@
 # Schema Evolution History
 
+PowderLine recipes are validated against a **core schema** (the shared top level
+and structural models, owned by PowderLine) plus one **engine schema** per
+gateway (`gsasii`, `topas`, `easydiffraction`), each in its engine's native
+conventions. Every schema has its own version. A recipe states both
+(`core_schema_version`, `engine_schema_version`), and each PowderLine release
+**declares** which versions it accepts. Compatibility is never inferred from the
+version numbers (`powderline.support_matrix()` lists the declarations).
+
+> **Refactor in progress (v0.2.0).** The schemas below are being introduced on
+> the `refactor/multi-engine` branch. Until the 0.26.0 schema is removed (re/07),
+> `powderline.run()` / `validate()` still validate recipes against the
+> **unified 0.26.0 schema** documented in the archive at the bottom of this page.
+
+---
+
+## Core schema
+
+### 1.0.0 — introduced in PowderLine 0.2.0 (in development)
+
+Defined in `src/powderline/schema_core.py`. Not used by any gateway yet: the
+engine schemas adopt it in re/04–06.
+
+- **Recipe frame** (`CoreRecipe`): `schema_name` (`"<engine>.<workflow>"`),
+  `core_schema_version`, `engine_schema_version`, `metadata`, `payload` (typed
+  by the engine schema). Unknown keys are errors at every level.
+- **`metadata`**: free-form and never interpreted; must be JSON-serializable and
+  at most **1 MiB** as UTF-8 JSON. The cap is the constant
+  `schema_core.METADATA_MAX_BYTES`; change it there and record the change here.
+- **Parameters**: `[value, refine_flag]` (no bounds) or
+  `[value, refine_flag, min, max]` (bounds the engine honors). A bounded list
+  where bounds aren't supported is an error. `refine_flag` is a JSON boolean.
+  `value` may be `null` (= start from the phase structure's value) only for
+  quantities that mirror a structure value.
+- **Data and ranges**: `xrd_data` (2θ in degrees, weights 1/σ², validated as in
+  0.26.0); `fit_range` `[min, max]`, with `max > min`, inside the data's 2θ range.
+- **Background**: Chebyshev (`num_coefficients`, `coefficients`, `refine_flag`).
+  Coefficient semantics are documented per engine.
+- **Units** are fixed per field in the schema, never written in a recipe.
+- **Numbers** are JSON numbers: a quoted number (`"0.25"`) or a boolean is an
+  error, never converted. Integer fields (`Multiplicity`, `num_coefficients`)
+  take a whole number (`4` or `4.0`), not `4.5`.
+- **Accepted core versions** (this release): `==1.0.0`.
+
+- **Phase structure** (`PhaseStructure`): `phase_name`, `space_group`,
+  `unit_cell`, `atoms` (keyed by label). Structural interpretation is checked
+  once, in core, so every engine gets the same structure. All structural
+  problems are reported together, each at its own field (`unit_cell`,
+  `atoms.<label>`, `atoms.<label>.Multiplicity`, `atoms.<label>.Uaniso`); they
+  are checked once the individual fields are valid:
+  - **Space group**: a Hermann–Mauguin symbol. Two-origin groups need an explicit
+    `:1`/`:2` and rhombohedral groups `:H`/`:R` (e.g. `"F d -3 m:2"`,
+    `"R -3 m:H"`). Individual engines may accept fewer settings.
+  - **Unit cell**: `a`, `b`, `c` (Å, > 0), `alpha`, `beta`, `gamma` (degrees,
+    0–180). No `volume` (every engine derives it). The cell must fit the
+    space group **exactly** (to floating-point precision): e.g. cubic
+    `a = b = c` and all angles 90°. A mismatch is an error naming each
+    parameter and its symmetric value. (GSAS-II alone would silently apply the
+    symmetry only when the cell is refined.)
+  - **Elements**: a bare element symbol spelled exactly (`"Fe"`, not `"FE"`).
+    Charged scattering types (`"Fe3+"`) are not supported yet and are rejected,
+    never reduced to the neutral atom.
+  - **Occupancy**: 0 to 1 inclusive, even where an engine would accept more.
+  - **Special positions**: an atom whose symmetry images lie within 5e-6
+    (fractional) of itself is on a special position, so write special
+    coordinates with **at least 6 decimals** (e.g. `0.333333`). An atom 5e-6 to
+    2e-3 from a special position is an **error** (`0.33`, `0.3333`, `0.33333`):
+    state it to ≥ 6 decimals or move it off. Accepted special positions are
+    replaced by their **exact** coordinates (fixed coordinates become the exact
+    fraction; coupled ones satisfy their relation, by the smallest change). Each
+    adjustment is reported as a structured warning, and these exact values are
+    what every engine receives.
+  - **Multiplicity**: optional; when stated it must equal the multiplicity
+    derived from the space group.
+  - **ADPs**: `ADP` is `"Uiso"` (with `Uiso`, Ų) or `"Uaniso"` (with all of
+    `U11 U22 U33 U12 U13 U23`, Ų). Anisotropic ADPs must respect the site
+    symmetry: within 1e-6 Ų they are set to the symmetric values and reported;
+    beyond that it is an error.
+
+## gsasii engine schema
+
+### 1.0.0 — introduced in PowderLine 0.2.0
+
+Entries added by re/04 (`gsasii.rietveld`, `gsasii.spf`).
+
+## topas engine schema
+
+### 1.0.0 — introduced in PowderLine 0.2.0
+
+Entries added by re/05 (`topas.rietveld`, `topas.spf`).
+
+## easydiffraction engine schema
+
+### 1.0.0 — introduced in PowderLine 0.2.0
+
+Entries added by re/06 (`easydiffraction.rietveld`).
+
+---
+
+## Archive: unified-schema era (0.21–0.26)
+
+The single schema shared by all engines up to PowderLine 0.1.1, kept verbatim
+(headings moved one level down).
+
 This document provides a concise summary of PowderLine's schema evolution for context. For detailed commit history, see the repository's git log. Current schema: **0.26.0**
 
 ---
 
-## Schema 0.26.0: Per-Parameter Refinement Flags Honored (Current)
+### Schema 0.26.0: Per-Parameter Refinement Flags Honored (Current)
 
 **Status**: Active (Q3 2026)
 
@@ -64,7 +167,7 @@ file reader/parser that builds the block.
 
 ---
 
-## Schema 0.25.4: Consistent Failure Metadata
+### Schema 0.25.4: Consistent Failure Metadata
 
 **Status**: Superseded by 0.26.0
 
@@ -90,7 +193,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.25.3: Refinement Failure Detection and Windows Support
+### Schema 0.25.3: Refinement Failure Detection and Windows Support
 
 **Status**: Superseded by 0.25.4
 
@@ -110,7 +213,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.25.2: Programmatic Python API
+### Schema 0.25.2: Programmatic Python API
 
 **Status**: Superseded by 0.25.3
 
@@ -147,7 +250,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.25.1: Refined Parameters Export + Extended SPF Returns
+### Schema 0.25.1: Refined Parameters Export + Extended SPF Returns
 
 **Status**: Superseded by 0.25.2
 
@@ -175,7 +278,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.25: Simplification
+### Schema 0.25: Simplification
 
 **Status**: Superseded by 0.25.1
 
@@ -202,7 +305,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.24: Multi-Strategy Refinement System
+### Schema 0.24: Multi-Strategy Refinement System
 
 **Status**: Deprecated (replaced by schema 0.25)
 
@@ -222,7 +325,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.23: Initial Multi-Strategy Exploration
+### Schema 0.23: Initial Multi-Strategy Exploration
 
 **Status**: Deprecated
 
@@ -235,7 +338,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.22: ADP Field Requirements
+### Schema 0.22: ADP Field Requirements
 
 **Status**: Deprecated
 
@@ -249,7 +352,7 @@ No other recipe changes required.
 
 ---
 
-## Schema 0.21 and Earlier
+### Schema 0.21 and Earlier
 
 **Status**: Historical
 
@@ -262,7 +365,7 @@ Early development versions focused on basic Rietveld refinement capabilities:
 
 ---
 
-## Origin (Schema 0.21–0.24)
+### Origin (Schema 0.21–0.24)
 
 PowderLine began as a minimal proof-of-concept to confirm that GSAS-II could be driven
 from a JSON file via the G2scripts API. The initial scope established the directory
@@ -276,7 +379,7 @@ configuration system, programmatic Python API) were subsequently delivered in sc
 
 ---
 
-## Schema Design Philosophy
+### Schema Design Philosophy
 
 **Example-Driven Development**: Features are added when real refinement examples need them, not speculatively. Schema evolution is guided by committed examples in `examples/` directory.
 
@@ -288,7 +391,7 @@ configuration system, programmatic Python API) were subsequently delivered in sc
 
 ---
 
-## Migration Philosophy
+### Migration Philosophy
 
 - **Explicit schema versions**: Every recipe declares its `"schema_version"`. The code
   accepts one current version — **0.26.0** — and recipes written for older versions must

src/powderline/__init__.py

@@ -14,6 +14,9 @@
 # requiring GSAS-II. The remaining GSAS-II-backed helpers are imported eagerly
 # when GSAS-II is available and skipped otherwise (the TOPAS path stays usable).
 from powderline.engine import run
+# Declared schema-version support per gateway (A65); outside __all__ until the
+# public-API rework.
+from powderline.compat import support_matrix
 
 try:  # GSAS-II-backed helpers + client (require the GSAS-II package)
     from powderline.kicker import (

src/powderline/compat.py

@@ -0,0 +1,146 @@
+"""Schema-version compatibility: declared support sets, never version arithmetic (A40, A65).
+
+Declarations:
+
+- core (:mod:`powderline.schema_core`): ``CORE_SCHEMA_VERSION`` and
+  ``SUPPORTED_CORE_SCHEMAS`` (a PEP 440 specifier);
+- each gateway, as keys of its ``capabilities()`` dict (re/04-06):
+
+  - ``engine_schema_version`` -- the version new recipes are written against;
+  - ``supported_engine_schemas`` -- specifier of accepted engine schema versions;
+  - ``requires_core_schema`` -- specifier of the core versions it needs, or a
+    mapping ``{engine-schema specifier: core specifier}`` when different engine
+    schema versions need different core versions.
+
+A gateway that does not declare these yet is reported as undeclared by
+:func:`support_matrix` and rejected by :func:`check_engine_schema_version`.
+All checks raise ``ValueError`` so engine schema models can call them from a
+pydantic validator and have them reported with the other validation errors.
+
+Engine-free (enforced by an import-block test).
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Mapping, Optional, Union
+
+from packaging.specifiers import InvalidSpecifier, SpecifierSet
+
+from powderline.schema_core import (
+    CORE_SCHEMA_VERSION,
+    SUPPORTED_CORE_SCHEMAS,
+    check_core_schema_version,
+    parse_version,
+)
+
+DECLARATION_KEYS = ("engine_schema_version", "supported_engine_schemas", "requires_core_schema")
+
+RequiresCore = Union[str, Mapping[str, str]]
+
+
+def _powderline_version() -> str:
+    from powderline import __version__
+
+    return __version__
+
+
+def _specifier(what: str, spec: str) -> SpecifierSet:
+    try:
+        return SpecifierSet(spec)
+    except InvalidSpecifier:
+        raise ValueError(f"{what} {spec!r} is not a valid PEP 440 specifier") from None
+
+
+def declarations(capabilities: Mapping[str, Any]) -> Optional[Dict[str, Any]]:
+    """The schema declarations in a gateway's ``capabilities()``, or ``None`` if absent.
+
+    Raises ``ValueError`` when only some of the keys are declared.
+    """
+    present = [k for k in DECLARATION_KEYS if k in capabilities]
+    if not present:
+        return None
+    if len(present) != len(DECLARATION_KEYS):
+        missing = sorted(set(DECLARATION_KEYS) - set(present))
+        raise ValueError(f"gateway {capabilities.get('name')!r} declares {present} but not {missing}")
+    return {k: capabilities[k] for k in DECLARATION_KEYS}
+
+
+def required_core_schemas(requires: RequiresCore, engine_schema_version: str) -> str:
+    """The core specifier that ``engine_schema_version`` requires.
+
+    ``requires`` is a single specifier, or a mapping whose first key (in
+    declaration order) containing ``engine_schema_version`` applies.
+    """
+    if isinstance(requires, str):
+        return requires
+    if not isinstance(requires, Mapping):
+        raise ValueError(
+            "requires_core_schema must be a specifier string or a mapping "
+            f"{{engine-schema specifier: core specifier}}, got {type(requires).__name__}"
+        )
+    version = parse_version("engine_schema_version", engine_schema_version)
+    for engine_spec, core_spec in requires.items():
+        if version in _specifier("requires_core_schema key", engine_spec):
+            return core_spec
+    raise ValueError(
+        f"requires_core_schema has no entry for engine schema {engine_schema_version} "
+        f"(keys: {', '.join(requires)})"
+    )
+
+
+def check_engine_schema_version(core_schema_version: str, engine_schema_version: str, *,
+                                gateway: str, capabilities: Mapping[str, Any]) -> None:
+    """Raise ``ValueError`` unless the recipe's version pair is accepted.
+
+    Checks, in order: the core version against :data:`SUPPORTED_CORE_SCHEMAS`; the
+    engine version against the gateway's ``supported_engine_schemas``; the core
+    version against what that engine version ``requires_core_schema``.
+    """
+    check_core_schema_version(core_schema_version)
+    decl = declarations(capabilities)
+    if decl is None:
+        raise ValueError(f"gateway {gateway!r} declares no supported engine schema versions")
+    pair = (f"core_schema_version {core_schema_version!r}, "
+            f"engine_schema_version {engine_schema_version!r}")
+    engine_v = parse_version("engine_schema_version", engine_schema_version)
+    supported = decl["supported_engine_schemas"]
+    if engine_v not in _specifier("supported_engine_schemas", supported):
+        raise ValueError(
+            f"{pair}: the {gateway!r} gateway of PowderLine {_powderline_version()} accepts "
+            f"engine schema {supported} (current: {decl['engine_schema_version']})"
+        )
+    needs = required_core_schemas(decl["requires_core_schema"], engine_schema_version)
+    if parse_version("core_schema_version", core_schema_version) not in _specifier(
+            "requires_core_schema", needs):
+        raise ValueError(
+            f"{pair}: {gateway!r} engine schema {engine_schema_version} requires core "
+            f"schema {needs} (PowderLine {_powderline_version()})"
+        )
+
+
+def support_matrix() -> List[Dict[str, Any]]:
+    """One row per registered gateway: its declarations plus core's (A40).
+
+    Loads each gateway's engine-free factory through the registry; imports no
+    engine. Undeclared gateways get ``"declared": False``. A gateway whose
+    declarations are malformed (only some keys present) raises ``ValueError``
+    naming it, rather than being listed as undeclared. Consumed by the docs
+    generation in re/10.
+    """
+    from powderline import registry
+
+    rows = []
+    for name in registry.available():
+        caps = registry.get(name).capabilities()
+        decl = declarations(caps)
+        rows.append({
+            "gateway": name,
+            "powderline_version": _powderline_version(),
+            "core_schema_version": CORE_SCHEMA_VERSION,
+            "supported_core_schemas": SUPPORTED_CORE_SCHEMAS,
+            "workflows": tuple(caps.get("workflows", ())),
+            "engine_version_spec": caps.get("engine_version_spec"),
+            "declared": decl is not None,
+            **(decl or {k: None for k in DECLARATION_KEYS}),
+        })
+    return rows

src/powderline/easydiff/builder.py

@@ -22,7 +22,7 @@
 )
 from powderline.easydiff.errors import EasyDiffractionTranslationError
 from powderline.easydiff.policy import check_unsupported, param_flag, param_value
-from powderline.topas.errors import TopasTranslationError
+from powderline.exceptions import SymmetryError
 from powderline.symmetry import cell_constraints
 
 
@@ -359,7 +359,7 @@ def build_project(recipe: dict, workdir) -> BuildResult:
         # Get symmetry rules
         try:
             rules = cell_constraints(sg_name)
-        except TopasTranslationError as e:
+        except SymmetryError as e:
             raise EasyDiffractionTranslationError(str(e)) from e
 
         length_map = {