Re/05 topas schema

docs/SCHEMA_HISTORY.md

@@ -42,7 +42,7 @@ engine schemas adopt it in re/04–06.
   `value` is always a number and `refine_flag` a JSON boolean; neither is ever
   `null`.
 - **Data and ranges**: `xrd_data` (2θ in degrees, weights 1/σ², validated as in
-  0.26.0); `fit_range` `[min, max]`, with `max > min`, inside the data's 2θ range.
+  0.26.0); `fit_range` `[min, max]`, with `max > min`, inside the data's 2θ range, holding at least one point with a positive weight (re/05).
 - **Background**: Chebyshev (`num_coefficients`, `coefficients`, `refine_flag`).
   Coefficient semantics are documented per engine.
 - **Units** are fixed per field in the schema, never written in a recipe.
@@ -238,6 +238,12 @@ core `==1.0.0`. A 0.26.0 `GSASII_*` recipe is converted with
   refined; U..Z may not). GSAS-II silently ignores a refine flag on the side
   not in use, so such a flag is an error.
 - **Simulation** (`refinement_cycles` 1): every refine flag must be false.
+- **Fit window** (re/05): GSAS-II gets the first and last weighted data
+  points inside `fit_range` as its limits (an open end, or no `fit_range`: the
+  data's first or last weighted point), so it fits exactly min ≤ 2θ ≤ max, the
+  points every engine fits, and its Chebyshev basis spans the same points (with
+  a limit between data points GSAS-II would also fit the next point above the
+  maximum; with no limit it would start at a zero-weight edge point).
 - **At run time**, after phase setup: an atom GSAS-II reads with the wrong
   multiplicity (some 2-fold sites in R32, R-3m, R-3c), or whose site it cannot
   name while a coordinate or Uaniso is refined, is an error naming an
@@ -254,9 +260,75 @@ core `==1.0.0`. A 0.26.0 `GSASII_*` recipe is converted with
 
 ## topas engine schema
 
-### 1.0.0 — introduced in PowderLine 0.2.0
+### 1.0.0 — introduced in PowderLine 0.2.0 (in development)
 
-Entries added by re/05 (`topas.rietveld`, `topas.spf`).
+Defined in `src/powderline/gateways/topas/schema.py` (engine-free), on core
+1.0.0. Accepted versions are declared: engine schema `==1.0.0`, requiring core
+`==1.0.0`. Native recipes state TOPAS's own keywords, macro parameters and
+units: the numbers in the recipe are the numbers in the INP. There is no
+converter from 0.26.0 recipes (the 0.26.0 `GSASII_*` → TOPAS path is
+unchanged).
+
+- **Workflows**: `topas.rietveld` (`phases`) and `topas.spf` (`single_peaks`).
+- **TOPAS version**: required `engine_version`, the TOPAS major version the
+  recipe is written for; 1.0.0 supports `"6"` only (declared,
+  `supported_engine_versions == "==6"`). A run needs the installed version
+  declared (`topas_version` or `.powderline_config.yaml` `topas.version`) and
+  equal to it; `tc.exe` carries no version PowderLine could read.
+- **No defaults**: a block left out means the feature is not modelled; every
+  value PowderLine would otherwise choose is a required field.
+- **Parameters**: `[value, refine_flag, min, max]`. A stated bound replaces
+  TOPAS's default limit on that side; `null` keeps TOPAS's default (keyword
+  limits of the TOPAS Technical Reference, TOPAS.INC macro limits). A refined
+  start outside a default limit on a `null` side is an error (TOPAS would move
+  it silently).
+- **Instrument**: `radiation` (one `lam` line: `ymin_on_ymax`, `la`, `lo` in Å,
+  `lh` required, `lg` only with `lh`; `la`, `lo`, `lh`, `lg` > 0, since a line
+  with no area, wavelength or width does not exist); `geometry` (`Rp`, `Rs` in mm, required
+  exactly where an axial model reads them); `corrections` keyed by macro
+  (`Zero_Error`, `LP_Factor` or `LP_Factor_Synchrotron`, `Simple_Axial_Model`
+  or `Full_Axial_Model`, `capillary` with a `parallel` or `divergent` beam and
+  a diameter > 0);
+  `broadening` `{peak_type, parameters}` with `TCHZ_Peak_Type`, `PV_Peak_Type`
+  or `PVII_Peak_Type` (TOPAS's TCHZ U..Z are not GSAS-II's; a **fixed**
+  peak-type term is not checked, so fixed terms that make a width negative in
+  the window, e.g. TCHZ X < 0 with Y = 0, validate, while TOPAS 6 stops on a
+  negative FWHM: not yet verified for the peak types, documented only). Geometry
+  corrections act on every peak, background and SPF peaks included;
+  Lorentz-polarisation sits in each phase (Bragg peaks only); the peak type in
+  each phase and each SPF peak.
+- **Background**: `chebyshev` (TOPAS's basis over the fitted points),
+  `One_on_X`, and `peaks` (`xo`, `I`, a `pv`/`spv`/`spvii` shape).
+- **Phase block**: core's `space_group`, `unit_cell`, `atoms`, plus `scale`
+  (> 0) and `peak_broadening` (`size_broadening` `CS_L`/`CS_G` in nm,
+  `strain_broadening` `Strain_L`/`Strain_G`; these are not interpretable
+  sizes or strains: LVol-IB and e0 are reported as TOPAS computes them). `Uiso`
+  is written as `beq = 8π²·Uiso` (with a warning), `Uaniso` as `u11..u23`.
+  Symmetry ties become one TOPAS parameter per tie group with exact equations.
+- **Space groups**: the canonical name in TOPAS spelling; 541 settings
+  accepted, 23 refused (20 TOPAS cannot name, 3 it reads as another setting).
+- **Single peak fitting**: each peak is the instrument peak type convolved
+  with its own `gauss_fwhm` / `lor_fwhm` (≥ 0).
+- **Refinement controls** (required): `iters` (0 = simulation, every refine
+  flag false), `chi2_convergence_criteria`, `x_calculation_step`.
+- **Fit window**: TOPAS gets the first and last weighted data points inside
+  `fit_range` as its limits (open ends: the data's first or last weighted
+  point), so it fits exactly min ≤ 2θ ≤ max (the same points every engine
+  fits); a window with no weighted point is an error. Numbers are written to
+  the INP to 10 significant digits (`%.10g`), as in the `.xye`.
+- **Results**: `rwp`, `r_exp`, `gof`, `chi2_red` from PowderLine's uniform fit
+  statistics over the stated window (equal to TOPAS's own); TOPAS's values,
+  `parameters_requested` and `parameters_varied` under `engine_details`. The
+  background column comes from a second, background-only TOPAS run at the
+  refined values (TOPAS 6 cannot output the background of the fit). Warnings:
+  `topas_adp_converted`, `topas_parameter_at_limit` (naming the stated bound
+  or TOPAS's default), `topas_parameters_not_varied`,
+  `topas_phase_contributes_nothing` (informative: a phase absent from the
+  sample legitimately contributes nothing), `topas_background_not_calculated`.
+  Success is judged from the output files the INP asks for, removed before
+  the run (TOPAS 6's exit code says nothing; timestamps are not used); any
+  non-finite number in them (`nan`, `1.#QNAN`, `-nan(ind)`, …) is a divergence
+  error, while an undetermined ESD is left empty.
 
 ## easydiffraction engine schema
 

pixi.toml

@@ -8,7 +8,7 @@ preview = ["pixi-build"]
 
 [tasks]
 kicker = "PYTHONPATH=$PWD/src:$PYTHONPATH python src/powderline/kicker.py"
-topas-kicker = "PYTHONPATH=$PWD/src:$PYTHONPATH python -m powderline.topas.kicker"
+topas-kicker = "PYTHONPATH=$PWD/src:$PYTHONPATH python -m powderline.gateways.topas.kicker"
 # Cross-platform (task-level env avoids Unix-only shell syntax; see
 # docs/cross-platform-guide.md Issue B for the remaining tasks)
 mp-simulate = { cmd = "python -m powderline.mp_simulate", env = { PYTHONPATH = "$PIXI_PROJECT_ROOT/src" } }

pyproject.toml

@@ -61,7 +61,7 @@ dependencies = [
 # registry finds no gateways.
 [project.entry-points."powderline.gateways"]
 gsasii = "powderline.gateways.gsasii.gateway:get_gateway"
-topas = "powderline.topas.gateway:get_gateway"
+topas = "powderline.gateways.topas.gateway:get_gateway"
 easydiffraction = "powderline.easydiff.gateway:get_gateway"
 
 [project.urls]

src/powderline/__init__.py

@@ -6,7 +6,7 @@
 # preserving the original behaviour exactly (including `powderline.kicker` being
 # registered as a submodule attribute). When GSAS-II is absent -- on Windows, or
 # on a TOPAS-only dev box -- these imports are skipped so that the GSAS-II-free
-# TOPAS path (`powderline.topas`) still imports cleanly. `powderline.schema` is
+# TOPAS path (`powderline.gateways.topas`) still imports cleanly. `powderline.schema` is
 # GSAS-II-free and always loaded, so `RecipeModel` is available either way.
 
 # `run` is the GSAS-II-free engine dispatcher (engine="gsasii"|"topas"); it loads

src/powderline/_code_hash.json

@@ -2,7 +2,7 @@
   "gsasii_gateway_hashes": {
     "_legacy_executor.py": "b541c8646e4a67ad809696793ea09c27",
     "constraints.py": "dc7a28029d0a071cf35dd352e76eaee9",
-    "executors.py": "0761a600e29e5f2d314feffc167509ab",
+    "executors.py": "81ebfd54e379b01c863a2ddff0a01662",
     "extractors.py": "30cc981566bc069f2ba309cfc1f6242c",
     "helpers.py": "5829fa4cb24cc41a69f27271061926f0",
     "project.py": "90c618bf9b4433f6b4b876a4266a1781",

src/powderline/exceptions.py

@@ -70,9 +70,10 @@ def __init__(self, *, gateway: str, dependency: str, env: str,
 class EngineVersionError(PowderLineError):
     """The installed engine version is outside the gateway's supported range."""
 
-    def __init__(self, *, gateway: str, engine: str, installed: str, supported: str) -> None:
+    def __init__(self, *, gateway: str, engine: str, installed: str, supported: str,
+                 message: Optional[str] = None) -> None:
         super().__init__(
-            f"the {gateway!r} gateway supports {engine} {supported}, but {installed} "
+            message or f"the {gateway!r} gateway supports {engine} {supported}, but {installed} "
             "is installed; install a supported version"
         )
         self.gateway = gateway

src/powderline/gateways/gsasii/executors.py

@@ -29,6 +29,7 @@
 from powderline.gateways.gsasii.fit_report import fit_report
 from powderline.gateways.gsasii.sites import check_sites
 from powderline.exceptions import EngineExecutionError
+from powderline.schema_core import fit_limits_on_data
 
 from powderline.gateways.gsasii.helpers import (
     DEFAULT_HIST_SCALE_REFINE_FLAG,
@@ -328,7 +329,7 @@ def run_refinement(recipe: GsasiiRietveldRecipe | GsasiiSpfRecipe, output_dir: P
         set_hist_scale(proj, hist, hist_scale_val=DEFAULT_HIST_SCALE_VAL, hist_scale_refine_flag=DEFAULT_HIST_SCALE_REFINE_FLAG, print_info=verbose)
 
         # 5. Set fit range
-        fit_range = (recipe.payload.fit_range.min, recipe.payload.fit_range.max) if recipe.payload.fit_range else (None, None)
+        fit_range = fit_limits_on_data(recipe.payload.fit_range, recipe.payload.xrd_data)  # on data points (A149, R17)
         if fit_range != (None, None):
             set_fit_range_hist(hist, fit_range, print_info=verbose)
 

src/powderline/gateways/gsasii/fit_report.py

@@ -34,46 +34,10 @@
 import re
 
 import numpy as np
-from pydantic import BaseModel
 
 from powderline.exceptions import EngineExecutionError, StructuredWarning
 from powderline.fitstats import compute_fit_statistics
-from powderline.schema_core import ChebyshevBackground, Phase, RefinableParameter
-from powderline.symmetry import cell_tie_groups, coupling_groups
-
-
-def parameters_requested(recipe) -> int:
-    """Independently refined parameters the recipe asks for: one per symmetry tie group (A94), per flag otherwise."""
-    count = 0
-
-    def walk(obj) -> None:
-        nonlocal count
-        if isinstance(obj, RefinableParameter):
-            count += obj.refine_flag
-        elif isinstance(obj, ChebyshevBackground):
-            count += obj.num_coefficients if obj.refine_flag else 0
-        elif isinstance(obj, dict):
-            for v in obj.values():
-                walk(v)
-        elif isinstance(obj, list):
-            for v in obj:
-                walk(v)
-        elif isinstance(obj, BaseModel):
-            for name in type(obj).model_fields:
-                walk(getattr(obj, name))
-            if isinstance(obj, Phase):  # a refined tie group is one engine parameter
-                tied = [(obj.unit_cell, g.members) for g in cell_tie_groups(obj.space_group).groups]
-                for atom in obj.atoms.values():
-                    ties = coupling_groups(obj.space_group, (atom.x.value, atom.y.value, atom.z.value))
-                    tied += [(atom, g.members) for g in ties.xyz.groups]
-                    if atom.Uaniso is not None:
-                        tied += [(atom.Uaniso, g.members) for g in ties.uij.groups]
-                for holder, members in tied:
-                    refined = sum(getattr(holder, m).refine_flag for m in members)
-                    count -= max(refined - 1, 0)
-
-    walk(recipe.payload)
-    return count
+from powderline.schema_core import parameters_requested  # noqa: F401  (moved to core, re/05 D12; re-exported)
 
 
 def _dropped(msg: str) -> int | None:

src/powderline/topas/__init__.py → src/powderline/gateways/topas/__init__.py

@@ -5,7 +5,7 @@
 pair. It imports **zero** GSAS-II code: TOPAS INP generation and the round-trip
 must run on Windows, where GSAS-II does not install cleanly.
 
-Public entry point: :func:`powderline.topas.writer.write_topas_inp`.
+Public entry point: :func:`powderline.gateways.topas.writer.write_topas_inp`.
 
 Citation notation used throughout this subpackage: ``findings §X`` / ``plan §Y``
 / ``D-numbers`` cite the internal TOPAS design record (empirical findings and

src/powderline/topas/errors.py → src/powderline/gateways/topas/errors.py

@@ -1,7 +1,7 @@
 """Shared exception type for the TOPAS translation path.
 
 Kept in its own module so ``writer`` and the TOPAS engine can share it without
-a circular import. Re-exported from ``powderline.topas`` for convenience. Core
+a circular import. Re-exported from ``powderline.gateways.topas`` for convenience. Core
 :class:`~powderline.exceptions.SymmetryError` is converted to it by the writer
 (A61).
 """

src/powderline/gateways/topas/gateway.py

@@ -0,0 +1,124 @@
+"""Engine-free factory for the TOPAS gateway (entry point ``powderline.gateways:topas``).
+
+- **Schema layer** — ``validate`` / ``capabilities``: a ``topas.*`` recipe
+  validates against the native schemas (:mod:`.schema`, re/05); a 0.26.0
+  ``GSASII_*`` recipe builds the ``RecipeModel`` and renders the INP in memory,
+  exactly the check ``validate_only`` runs today (raises
+  ``TopasTranslationError`` on untranslatable input), until re/07.
+- **Runtime layer** — ``run``: a native recipe runs through
+  :func:`.native_run.run_native`; a 0.26.0 recipe through
+  ``powderline.gateways.topas.engine.run_topas_recipe`` (unchanged, A146).
+
+TOPAS has no Python engine dependency, so ``EngineNotAvailableError`` never
+applies: without ``tc.exe`` a run is generate-only. TOPAS also has no
+queryable runtime version (``tc.exe`` carries no version metadata and prints
+its banner only to its console window; devkit ``re05-topas6-compat.md`` C29),
+so version checks use the **declared** version (``topas_version`` argument,
+else ``.powderline_config.yaml`` ``topas.version``):
+
+- native recipes (A144, A145): a run that will execute TOPAS needs a declared
+  version, equal to the recipe's ``engine_version`` and within
+  ``supported_engine_versions``; a missing declaration is an error;
+- 0.26.0 recipes (A53, A146): ``ENGINE_VERSION_SPEC``, skipped when none is
+  configured.
+"""
+
+from __future__ import annotations
+
+from packaging.version import Version
+
+from powderline.exceptions import EngineVersionError
+from powderline.gateways.topas import schema
+from powderline.registry import Gateway, check_engine_version
+from powderline.schema import RecipeModel
+
+NAME = "topas"
+#: TOPAS versions of the 0.26.0 translation path (A41/A44; unchanged until re/07, A146).
+#: Native ``topas.*`` recipes declare theirs in ``schema.SUPPORTED_ENGINE_VERSIONS`` (A144).
+ENGINE_VERSION_SPEC = ">=7,<8"
+#: Workflows served: the native schemas, and the 0.26.0 names until re/07 removes them.
+WORKFLOWS = (*schema.SCHEMA_NAMES, "GSASII_Rietveld", "GSASII_SPF")
+
+
+def capabilities() -> dict:
+    return {"name": NAME, "workflows": WORKFLOWS, "engine_version_spec": ENGINE_VERSION_SPEC,
+            **schema.DECLARATIONS}
+
+
+def validate(recipe, *, verbose: bool = False):
+    """Validate a recipe (dict or model); return the validated model (A50, A59).
+
+    A ``topas.*`` recipe gives its native model (``TopasRietveldRecipe`` /
+    ``TopasSpfRecipe``); a 0.26.0 one the legacy ``RecipeModel``. ``verbose`` is
+    part of the uniform gateway contract; this gateway prints nothing extra.
+    """
+    if schema.is_native_recipe(recipe):
+        return schema.validate_recipe(recipe)
+    from powderline.gateways.topas.writer import render_topas
+
+    model = recipe if isinstance(recipe, RecipeModel) else RecipeModel.model_validate(recipe)
+    render_topas(model.model_dump(), "topas")  # validates translatability; may raise
+    return model
+
+
+def configured_engine_version(topas_version=None):
+    """The configured TOPAS version, or ``None`` when none is configured."""
+    if topas_version is not None:
+        return topas_version
+    from powderline.gateways.topas.runner import _read_topas_config
+
+    return _read_topas_config().get("version")
+
+
+def check_native_engine_version(model, topas_version=None, *, allow_unsupported: bool = False) -> None:
+    """A145: the declared installed TOPAS version must exist and equal the recipe's ``engine_version``."""
+    if allow_unsupported:
+        return
+    declared = configured_engine_version(topas_version)
+    supported = schema.SUPPORTED_ENGINE_VERSIONS
+    if declared is None:
+        raise EngineVersionError(
+            gateway=NAME, engine="TOPAS", installed="undeclared", supported=supported,
+            message=(f"the TOPAS version to run is not declared: PowderLine cannot read it from tc.exe, so set "
+                     f"topas_version (run argument) or topas.version in .powderline_config.yaml; this recipe is "
+                     f"written for TOPAS {model.engine_version}"))
+    check_engine_version(gateway=NAME, engine="TOPAS", installed=declared, spec=supported)
+    if Version(str(declared)) != Version(model.engine_version):
+        raise EngineVersionError(
+            gateway=NAME, engine="TOPAS", installed=str(declared), supported=f"=={model.engine_version}",
+            message=(f"the recipe is written for TOPAS {model.engine_version} (engine_version), but TOPAS "
+                     f"{declared} is declared as installed"))
+
+
+def run(recipe, output_dir, *, verbose: bool = False, validate_only: bool = False,
+        topas_dir=None, topas_version=None,
+        allow_unsupported_engine_version: bool = False) -> dict:
+    """Run a recipe; see :func:`.native_run.run_native` (native) or ``run_topas_recipe`` (0.26.0).
+
+    ``allow_unsupported_engine_version`` is the test-only bypass of the engine
+    version check (A53); the check is skipped for ``validate_only``.
+    """
+    if schema.is_native_recipe(recipe):
+        from powderline.gateways.topas import native_run
+
+        model = schema.validate_recipe(recipe)  # the model validated from the user's recipe (A115)
+        if validate_only:
+            return native_run.run_native_validate(model)
+        return native_run.run_native(
+            model, output_dir, topas_dir=topas_dir, topas_version=topas_version,
+            engine_version_check=lambda _tc: check_native_engine_version(
+                model, topas_version, allow_unsupported=allow_unsupported_engine_version))
+    if not validate_only:
+        version = configured_engine_version(topas_version)
+        if version is not None:
+            check_engine_version(gateway=NAME, engine="TOPAS", installed=version,
+                                 spec=ENGINE_VERSION_SPEC,
+                                 allow_unsupported=allow_unsupported_engine_version)
+    from powderline.gateways.topas.engine import run_topas_recipe
+
+    return run_topas_recipe(recipe, output_dir, verbose=verbose, validate_only=validate_only,
+                            topas_dir=topas_dir, topas_version=topas_version)
+
+
+def get_gateway() -> Gateway:
+    return Gateway(name=NAME, capabilities=capabilities, validate=validate, run=run)

src/powderline/topas/kicker.py → src/powderline/gateways/topas/kicker.py

@@ -12,7 +12,7 @@
     pixi run topas-kicker recipe.json --validate-only                 # translate + report, no files
     pixi run topas-kicker recipe.json --parse-results out_results.csv # round-trip an existing result
 
-Entry point: ``python -m powderline.topas.kicker`` (D9). Not part of the 6-name
+Entry point: ``python -m powderline.gateways.topas.kicker`` (D9). Not part of the 6-name
 public API and no ``run()`` dispatch yet -- both follow once the demo lands.
 """