Skip to content

Add trade studies of constellations (tradespace) and sampling sufficiency analysis - #158

Merged
ptgrogan merged 57 commits into
mainfrom
tradespace
Nov 15, 2026
Merged

ptgrogan merged 57 commits into
mainfrom
tradespace

Conversation

@ptgrogan

Copy link
Copy Markdown
Contributor

Summary

This PR adds two things:

  • tatc.tradespace (prototype): trade studies of constellation designs. It covers declaring a design space, evaluating designs over uncertain replicates, judging and searching them, and modeling the deployment, failures, and life-cycle cost of each design.
  • tatc.analysis.sufficiency: estimates whether a constellation's observations sample a field well enough to estimate its mean over a period. It has been validated against IMERG precipitation.

Everything is additive. Existing modules keep their API and gain only new exports. The new optional dependencies (joblib, matplotlib, Optuna, pymoo) are in a tradespace extra (pip install -e ".[tradespace]").

What's added

Sampling sufficiency (analysis.sufficiency, generation)

  • estimate_sampling_error: the error in a period mean estimated from each target's samples, for a field with an exponential correlation in time. It also gives the effective number of independent samples.
  • sample_field and compare_sampled_means: check those estimates against a gridded field such as a nature run.
  • Other statistics: compute_revisit_statistics and compute_local_time_coverage.
  • generation.generate_points_rotated_lattice: a Fibonacci lattice in a random orientation. This stratified sample lowers the variance between replicates of a study.
  • Validation: checked against IMERG precipitation in ValidateSamplingErrorIMERG, with limitations stated in the validation summary.

Trade space (tradespace)

  • Design spaces (DesignSpace):
    • variables (Real, Integer, Categorical), Derived values, and Uncertain parameters drawn per replicate;
    • hierarchical Choice and Repeat;
    • full enumeration, define-by-run suggestions for Optuna, and a unit-hypercube encoding for pymoo.
  • Studies (Study, ObservationStudy):
    • each design is evaluated over replicates with common random numbers (draws are keyed by design-space path), in parallel;
    • SatelliteCache computes each satellite once across designs;
    • ResultStore (SQLite) keys results by a fingerprint of the study's definition, so a study that is interrupted or extended resumes.
  • Judging designs:
    • Criteria (objectives, Requirements, statistics over replicates);
    • Pareto fronts with tolerances, hypervolume, and pareto_probability;
    • refine (adaptive replication), confirm (fresh replicates against the winner's curse), and screen (multi-fidelity);
    • sensitivity, attribute_variance, Utility, and exceedance.
  • Scenarios: designs are compared across scenarios rather than averaged over them (Study(scenarios=...), robustness). Across lets searches find designs robust to scenarios.
  • Searches: optimize_optuna and optimize_pymoo evaluate each design only once. optimize_optuna can stop at a budget of designs (max_designs); optimize_pymoo stops on pymoo's termination criterion.
  • Missions (MissionPlan):
    • DeploymentSchedule (development, production rate, launch cadence and capacity);
    • failures of satellites and launches, with a RestorationPolicy;
    • staged deployments (Stage, Decision);
    • CostModel life-cycle costs: learning curves, production blocks, present value, peak annual cost, and cost risk structured by component, after Maier, Yeakel & Wendoloski (2023).
  • Constellation building blocks: parts of design spaces for Walker constellations and sun-synchronous planes. build_constellation builds them with instruments or with entries of the instrument catalog.
  • Plots: plot_designs, plot_progress, plot_means, and plot_exceedance.

Examples

  • New: SearchTradeSpace, RankUncertainDesigns, CompareScenarios, TradeCostAndUtility, and PlanMissions. They share a design space and study defined in docs/examples/trade_space_example.py.
  • Changed: RunOSSEMonsoon now declares its constellation trade as a design space.

Notes for reviewers

  • Size:

    Part Lines added
    tradespace source ~14k
    Its tests ~8.5k
    Sufficiency (source and tests) ~1k
    Notebooks with outputs, docs, and API reference the rest
  • Where to start: the SearchTradeSpace notebook walks through the main path. A suggested order for the source:

    1. space.py
    2. study.py
    3. criteria.py and pareto.py
    4. mission.py, deployment.py, failures.py, and cost.py
    5. observations.py
  • Layering: tests/tradespace/test_layering.py checks that the general modules (spaces, studies, criteria) don't import the mission-specific ones.

  • Prototype: tradespace is marked as a prototype in the changelog, so its API may still change.

  • Branch history: the branch merged instrument-catalog (Add a validated instrument catalog #157) along the way and is up to date with main, so this diff contains only the trade-space and sufficiency work.

Testing

  • CI: the unit-test workflow now installs .[preprocess,tradespace], so it also runs the searches (Optuna, pymoo), parallel studies (joblib), and plots (matplotlib). pymoo has wheels for Python 3.10 through 3.14.
  • Locally (Python 3.10, Windows), on the branch merged with main:
    • the full test suite passes, including the Optuna (5.0) and pymoo (0.6.2) tests;
    • the docs build without warnings (notebooks are stored with their outputs and aren't executed in the build);
    • black passes, and pylint passes the 9.0 threshold that CI uses.
  • Without the extra: the new tests also pass with only .[preprocess] installed. Tests that need Optuna, pymoo, joblib, matplotlib, or xarray skip when the package is missing.

Add tatc.tradespace: hierarchical design spaces (Real, Integer,
Categorical, Derived, Uncertain, Choice, Repeat) with enumeration,
Latin hypercube sampling, and unit-hypercube encoding; trade studies
evaluated in replicates with common random numbers, in parallel worker
processes, with caches of analyses by satellite; Pareto fronts, ranks,
tolerances, exact hypervolume, and the bootstrap probability of
efficiency; screening with studies of increasing fidelity; and searches
with Optuna and pymoo that use the violations of constraints.

Port the RunOSSEMonsoon constellation trade to a design space (cached,
parallel, with Pareto-efficient designs) and add the SearchTradeSpace
example. Add the optional dependencies `tradespace` (joblib, Optuna,
pymoo).
…stellation building blocks

Add statistics of metrics over replicates (Mean, Quantile, Minimum,
Maximum, StandardDeviation, Probability) and requirements on them as
output constraints, with aggregate, and use them in pareto_probability,
screen, and the Optuna and pymoo searches. Add refine, which adds
replicates only to designs whose Pareto efficiency is uncertain, with
tolerances of differences that do not matter, and evaluate given
replicates with Study.run.

Add building blocks of constellations: walker_space and sso_planes_space
fragments, build_constellation with instruments or instrument catalog
entries, and count_satellites and count_planes. Use them in the
SearchTradeSpace example, with sections on refinement and reliable
sampling.
…tivity

Add tatc.analysis.sufficiency: estimate_sampling_error (exact error of the
mean of a field with an exponential correlation in time, with effective
samples), sample_field and compare_sampled_means (sampling errors against
a nature run), compute_revisit_statistics (revisit quantiles and longest
gaps), and compute_local_time_coverage.

Add to tatc.tradespace: ResultStore, a persistent SQLite store of the
evaluations of a study, so that interrupted or extended studies evaluate
only what is new; Uncertain.seed and remove_failures for failures of
satellites; tolerances as percentages; and sensitivity and main_effects.
Return an empty frame from a SatelliteCache without satellites, and skip
the executor when every evaluation is stored.

Extend the SearchTradeSpace example with the share of points accurately
sampled, sensitivity, and robustness to failures.
Document the Variable base class of design spaces, render the sampling
error formula as a literal block, add a blank line before the columns of
compute_latencies, link the license and citation files by their GitHub
URLs in the README, and refer to the Looking Up section of the
CollectPlatformTrack example without an anchor that the documentation
lacks.
…eplicates, minimum replicates of refinement, and adjusted sensitivity
…ches, record evaluation errors, enumerate unordered items once, and sweep large Pareto fronts
…ns of studies

- Designs are infeasible only if their functions raise Infeasible or fail
  the validation of TAT-C's schemas; other errors are raised (or recorded
  as failed with errors="record") and are not stored.
- Criteria (objectives, statistics, requirements, tolerances, and by)
  replace the keyword arguments of every function that judges designs;
  summarize wraps aggregate, and statistics name the columns they need.
- Fingerprints hash the definitions of studies, with the source of the
  TAT-C modules they use and data frames by content; stores keep the
  definitions and explain changes, records are JSON, and workers store
  each evaluation as it completes. Satellite caches keep each definition
  apart and warn once.
- refine stops refining designs whose objectives are known within a
  precision of their tolerances, within a budget of evaluations, and
  pareto_probability gives standard errors; searches return a Search to
  refine and confirm.
- Replicates may draw named parameters as in other replicates (shift) or
  be evaluated in scenarios that set values of the design space, each
  kept apart in stores.
- pymoo searches eliminate the proposals of designs proposed or evaluated
  before.
…ios and by utility

- attribute_variance attributes the variance of metrics between the
  replicates of each design to its uncertain parameters (first and total
  shares, by drawing each again with the others held, with bootstrap
  standard errors); Attribution.summary pools designs, and differences
  attributes the variance of differences between pairs of designs.
- robustness gives the Pareto trace of each design across scenarios, and
  its regret in each scenario and at most.
- Utility combines single-attribute utilities of metrics (LinearUtility,
  ExponentialUtility, PiecewiseUtility) by weights, additively or in the
  multiplicative form, as a metric of any statistic (its mean is the
  expected utility).
- Layouts of constellations (Layout and Plane, by layout_of: Walker,
  sun-synchronous planes, planes, and several layouts) replace the
  dispatch on the kind of a layout; launches and production move to a
  module of events, so that modules import one way.
- A Stage of a deployment is produced and launched only if its rule
  decides to, at times during the mission, from what it knows then (a
  Decision) and the values of the design; deployed stages are replaced and
  restored as other planes, and others cost nothing. Exact expectations
  of missions need deployments that are not staged.
- The metrics of phases, missions, and their cache are given the design
  if they take it (keyed by scenario).
- Phases are documented as quasi-static (one window represents the
  mission), a design life as not retiring a satellite, and a planned
  replacement whose launch fails as not flown again.
… spaces

- ObservationStudy runs a standard pipeline: the targets of each replicate
  (Targets, fixed or generated from a seed of the design), the
  observations of each satellite over a period, cached on disk
  (Observations, whose start may be a value of the design, such as an
  epoch drawn for each replicate), a MissionPlan of deployment, costs, and
  stages, Failures, or the phases of deployment, and metrics that may
  take the design.
- plot_designs, plot_progress, and plot_means plot designs, the progress
  of searches, and means with their uncertainty (matplotlib, in the
  tradespace extra); hypervolume_progress gives the progress of a search.
- A test keeps the general modules of trade studies from importing those
  of constellations and missions, and the modules free of cycles.
…d example

SearchTradeSpace runs on observation studies, and adds sections that
compare designs across scenarios (fields of different decorrelation
times), find what spreads the replicates (and their differences), trade
utility against cost for two stakeholders, stage the deployment of a
second plane, and check the one window of quasi-static phases against
epochs drawn over a year. Its refinement stops at a precision, and its
pymoo search proposes no design twice.
The schedule and cost model of a MissionPlan may be functions of the
design, launches limited in mass take the masses of the cost model's
satellites, and a launch mass capacity may be a function of each plane
(also for the bookings of restorations).
Criteria across the scenarios (Across) combine each design's statistics
by their worst or weighted mean over the scenarios, in aggregate,
pareto_probability (a joint bootstrap of the replicates), and the
searches; requirements judge the combined value. The example compares
the fronts of the worst and the mean, and searches by the mean.
…cision

Decision.capability gives the study's metrics of the satellites operating
at a decision, from the Capability that the evaluations of an
ObservationStudy give each deployment (Deployment.capability), with the
cache of its phases (which now also caches the phases of studies without
failures). The example adds a staging that measures the first plane.
The functions that judge designs (pareto_front, pareto_ranks, hypervolume,
sensitivity, main_effects) raise if given results with several rows of a
design, and aggregate reads a mapping as objectives, as everywhere else,
with named statistics in its own argument. The steps after an evaluation
(refine, confirm, screen, attribute_variance) take the study, its criteria
or metrics, and then results or designs (as designs, keys, or rows of
a summary, which DesignSpace.designs_of rebuilds), and their outcomes
give results, a summary, and designs. DesignSpace.extend,
DesignSpace.design with overrides, and Study.replace make variants of
a space, design, and study, which share a store, which warns only of
the same study run with other code. Columns follow one convention
(se, mission, histories, pareto_trace, max_regret), an ObservationStudy
reports when its plan deploys, and the package exports the names of
a study (others from their modules). The API reference is split by step
and the example into five notebooks sharing trade_space_example.py.
Growth of each type of satellite; costs of satellites as sums of components
whose growth by maturity is drawn from scores of the components and a common
score (risk_space), so that designs sharing a component share its growth and
develop it once; the probability that a metric of each design exceeds that of
each other in the same replicate (exceedance, plot_exceedance); and a section
of TradeCostAndUtility comparing uncertain costs (after Maier, Yeakel, and
Wendoloski, 2023).
…arning

A block size and a share of development repeated for each block after the
first (dated just before its first unit is produced), with the learning
curve restarting each block, and the blocks of each type in cost estimates
(after Maier, Yeakel, and Wendoloski, 2023).
Main names the configurations of ATMS, VIIRS and OMPS by their platform,
so the catalog payload in build_constellation's docstring and test is
now atms:swath_noaa_20.
…blib

CI installs the tradespace extra, so the tests of searches with Optuna and
pymoo, of parallel studies, and of plots run there. Without joblib, the
tests of parallel studies skip, as those of Optuna and pymoo do without
them.
pandas 3 (on Python 3.11 and later) gives read-only views of a frame's
values, so attribute_variance's differences failed to assign NaN to them.
…s whose costs are those incurred by the end of the mission
…, each adding one idea, in docs/examples/tradespace
…olor plots of designs by at most three groups of planes
…onstellation fragments (planes and per_plane), and add fragments of streets-of-coverage constellations and mutual orbiting groups
…ons, fingerprint geometries by content, count availability and annual costs correctly, read design values by path, add the efficient designs to summaries of criteria, give an observation study the mission of a design, describe layouts, and share the logic of deployments, phases, and bootstraps (new modules simulation, dominance, and hashing)
…vations of the same study with other code (keeping those of studies that differ in constants or analyses), key designs given by value as their levels, fingerprint the classes and objects of a study's own modules by their attributes and code, summarize criteria across scenarios, name statistics by convention, color designs by values, plot shares of variance, apply a style to the plots of trade studies, refer to a design's restoration policy, give the histories of a design's failures, and seed searches with Optuna by default
…udy's functions by content, evaluate designs of other spaces in the study's space, check constraints in scenarios, judge designs by their feasible replicates (with the share feasible), cache the phases of missions by replicate, keep the launch interval in replacements, replace only the occupied positions, price launches by the masses of the deployment, fall back from exact expectations, chain screenings, give each step the designs it selects, configure policies and spares in one place (MissionPlan.policy), give the mission of a design (mission_of), make options keyword-only, and split criteria, space, and observations (new modules statistics, summary, walks, alerts, plans, and references)
… orientation (count, rotate, and seed) in place of generate_points_rotated_lattice, whose rotated lattices now also sample masks that cross the antimeridian
…ingerprint__ (designs, and caches of satellites by their function and namespace), reject decisions that name no child, record errors of realizing designs, plan each deployment with its restoration policy (plan(policy=), Deployment.policy), launch ground spares at the next slot, give an infinite deployed time to deployments that outlast the mission, judge resamples of pareto_probability by their drawn replicates, charge refine's budget for every scenario, give the spread of statistics across scenarios, reject statistics named as columns of the results (P(<requirement>) histories), keep the caller's pymoo algorithm, compute the metrics of every satellite once, speed up exact expectations, generate targets only to observe them, keep contexts in caches of satellites, export configure, value_of, Statistic, and the classes of steps, make options keyword-only, and move shared helpers into Criteria and new modules tables, expectations, and memory
…es aggregate (spread=, criteria as Criteria or directions, metrics and statistics in statistics=, the standard errors of summarize in pareto_probability), studies describe their designs (describe=, a description column), enumerate gives the extensions of designs (DesignSpace.enumerate(designs)), Failures take statistics over histories (Quantile, Probability, StandardDeviation) and require a seed, requirements on phases are the study's with or without failures, every field of a MissionPlan is a value or a reference (also stages) with the design's policy by default, masses come from the plan (no SatelliteCost.mass), Triangular takes its mode by keyword, the optimizers' options are keyword-only, plot_designs rings the front by default and annotates with annotate, the outcomes of steps share their first fields (Screening.summaries one frame), StandardError and design_columns are gone, the internals of criteria, design spaces, stores, missions, and observations are private (DesignSpace no longer encodes designs), simulate_histories folds into simulate_many, phases are evaluated one way, fingerprint rules are a table, and spreads of infinite values are missing without warnings
…r attributes, count only the resamples in which a design is judged in the standard errors of pareto_probability (and refine designs whose standard error is unknown), keep infinite means under control variates, take statistics over histories of equal weights as over replicates, reject two requirements on the phases of one metric and design values policy that configure no policy, key the cache of phases by the satellites of each content, judge rows without a value of by as a group of their own, give the summaries of searches, screenings, and confirmations the spreads of means, take a fixed seed of targets, a single constraint of layout fragments, and references in per-type masses, list the values of designs for completion (with suggestions of misspelled names), give the direction error for misspelled directions in summarize, require a front to annotate designs, and enumerate at most 4,096 sets for exact expectations
@ptgrogan
ptgrogan merged commit 4349ed3 into main Nov 15, 2026
9 checks passed
@ptgrogan
ptgrogan deleted the tradespace branch November 15, 2026 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant