Repository navigation
Add trade studies of constellations (tradespace) and sampling sufficiency analysis - #158
Merged
Merged
Conversation
Add tatc.tradespace: hierarchical design spaces (Real, Integer, Categorical, Derived, Uncertain, Choice, Repeat) with enumeration, Latin hypercube sampling, and unit-hypercube encoding; trade studies evaluated in replicates with common random numbers, in parallel worker processes, with caches of analyses by satellite; Pareto fronts, ranks, tolerances, exact hypervolume, and the bootstrap probability of efficiency; screening with studies of increasing fidelity; and searches with Optuna and pymoo that use the violations of constraints. Port the RunOSSEMonsoon constellation trade to a design space (cached, parallel, with Pareto-efficient designs) and add the SearchTradeSpace example. Add the optional dependencies `tradespace` (joblib, Optuna, pymoo).
…stellation building blocks Add statistics of metrics over replicates (Mean, Quantile, Minimum, Maximum, StandardDeviation, Probability) and requirements on them as output constraints, with aggregate, and use them in pareto_probability, screen, and the Optuna and pymoo searches. Add refine, which adds replicates only to designs whose Pareto efficiency is uncertain, with tolerances of differences that do not matter, and evaluate given replicates with Study.run. Add building blocks of constellations: walker_space and sso_planes_space fragments, build_constellation with instruments or instrument catalog entries, and count_satellites and count_planes. Use them in the SearchTradeSpace example, with sections on refinement and reliable sampling.
…tivity Add tatc.analysis.sufficiency: estimate_sampling_error (exact error of the mean of a field with an exponential correlation in time, with effective samples), sample_field and compare_sampled_means (sampling errors against a nature run), compute_revisit_statistics (revisit quantiles and longest gaps), and compute_local_time_coverage. Add to tatc.tradespace: ResultStore, a persistent SQLite store of the evaluations of a study, so that interrupted or extended studies evaluate only what is new; Uncertain.seed and remove_failures for failures of satellites; tolerances as percentages; and sensitivity and main_effects. Return an empty frame from a SatelliteCache without satellites, and skip the executor when every evaluation is stored. Extend the SearchTradeSpace example with the share of points accurately sampled, sensitivity, and robustness to failures.
Document the Variable base class of design spaces, render the sampling error formula as a literal block, add a blank line before the columns of compute_latencies, link the license and citation files by their GitHub URLs in the README, and refer to the Looking Up section of the CollectPlatformTrack example without an anchor that the documentation lacks.
…eplicates, minimum replicates of refinement, and adjusted sensitivity
…ches, record evaluation errors, enumerate unordered items once, and sweep large Pareto fronts
…aclasses without mutable globals
… times of each type
…ies that restore them
… sampling of failures
…s in the trade space example
…ns of studies - Designs are infeasible only if their functions raise Infeasible or fail the validation of TAT-C's schemas; other errors are raised (or recorded as failed with errors="record") and are not stored. - Criteria (objectives, statistics, requirements, tolerances, and by) replace the keyword arguments of every function that judges designs; summarize wraps aggregate, and statistics name the columns they need. - Fingerprints hash the definitions of studies, with the source of the TAT-C modules they use and data frames by content; stores keep the definitions and explain changes, records are JSON, and workers store each evaluation as it completes. Satellite caches keep each definition apart and warn once. - refine stops refining designs whose objectives are known within a precision of their tolerances, within a budget of evaluations, and pareto_probability gives standard errors; searches return a Search to refine and confirm. - Replicates may draw named parameters as in other replicates (shift) or be evaluated in scenarios that set values of the design space, each kept apart in stores. - pymoo searches eliminate the proposals of designs proposed or evaluated before.
…ios and by utility - attribute_variance attributes the variance of metrics between the replicates of each design to its uncertain parameters (first and total shares, by drawing each again with the others held, with bootstrap standard errors); Attribution.summary pools designs, and differences attributes the variance of differences between pairs of designs. - robustness gives the Pareto trace of each design across scenarios, and its regret in each scenario and at most. - Utility combines single-attribute utilities of metrics (LinearUtility, ExponentialUtility, PiecewiseUtility) by weights, additively or in the multiplicative form, as a metric of any statistic (its mean is the expected utility).
- Layouts of constellations (Layout and Plane, by layout_of: Walker, sun-synchronous planes, planes, and several layouts) replace the dispatch on the kind of a layout; launches and production move to a module of events, so that modules import one way. - A Stage of a deployment is produced and launched only if its rule decides to, at times during the mission, from what it knows then (a Decision) and the values of the design; deployed stages are replaced and restored as other planes, and others cost nothing. Exact expectations of missions need deployments that are not staged. - The metrics of phases, missions, and their cache are given the design if they take it (keyed by scenario). - Phases are documented as quasi-static (one window represents the mission), a design life as not retiring a satellite, and a planned replacement whose launch fails as not flown again.
… spaces - ObservationStudy runs a standard pipeline: the targets of each replicate (Targets, fixed or generated from a seed of the design), the observations of each satellite over a period, cached on disk (Observations, whose start may be a value of the design, such as an epoch drawn for each replicate), a MissionPlan of deployment, costs, and stages, Failures, or the phases of deployment, and metrics that may take the design. - plot_designs, plot_progress, and plot_means plot designs, the progress of searches, and means with their uncertainty (matplotlib, in the tradespace extra); hypervolume_progress gives the progress of a search. - A test keeps the general modules of trade studies from importing those of constellations and missions, and the modules free of cycles.
…d example SearchTradeSpace runs on observation studies, and adds sections that compare designs across scenarios (fields of different decorrelation times), find what spreads the replicates (and their differences), trade utility against cost for two stakeholders, stage the deployment of a second plane, and check the one window of quasi-static phases against epochs drawn over a year. Its refinement stops at a precision, and its pymoo search proposes no design twice.
The schedule and cost model of a MissionPlan may be functions of the design, launches limited in mass take the masses of the cost model's satellites, and a launch mass capacity may be a function of each plane (also for the bookings of restorations).
Criteria across the scenarios (Across) combine each design's statistics by their worst or weighted mean over the scenarios, in aggregate, pareto_probability (a joint bootstrap of the replicates), and the searches; requirements judge the combined value. The example compares the fronts of the worst and the mean, and searches by the mean.
…cision Decision.capability gives the study's metrics of the satellites operating at a decision, from the Capability that the evaluations of an ObservationStudy give each deployment (Deployment.capability), with the cache of its phases (which now also caches the phases of studies without failures). The example adds a staging that measures the first plane.
The functions that judge designs (pareto_front, pareto_ranks, hypervolume, sensitivity, main_effects) raise if given results with several rows of a design, and aggregate reads a mapping as objectives, as everywhere else, with named statistics in its own argument. The steps after an evaluation (refine, confirm, screen, attribute_variance) take the study, its criteria or metrics, and then results or designs (as designs, keys, or rows of a summary, which DesignSpace.designs_of rebuilds), and their outcomes give results, a summary, and designs. DesignSpace.extend, DesignSpace.design with overrides, and Study.replace make variants of a space, design, and study, which share a store, which warns only of the same study run with other code. Columns follow one convention (se, mission, histories, pareto_trace, max_regret), an ObservationStudy reports when its plan deploys, and the package exports the names of a study (others from their modules). The API reference is split by step and the example into five notebooks sharing trade_space_example.py.
Growth of each type of satellite; costs of satellites as sums of components whose growth by maturity is drawn from scores of the components and a common score (risk_space), so that designs sharing a component share its growth and develop it once; the probability that a metric of each design exceeds that of each other in the same replicate (exceedance, plot_exceedance); and a section of TradeCostAndUtility comparing uncertain costs (after Maier, Yeakel, and Wendoloski, 2023).
…arning A block size and a share of development repeated for each block after the first (dated just before its first unit is produced), with the learning curve restarting each block, and the blocks of each type in cost estimates (after Maier, Yeakel, and Wendoloski, 2023).
Main names the configurations of ATMS, VIIRS and OMPS by their platform, so the catalog payload in build_constellation's docstring and test is now atms:swath_noaa_20.
…blib CI installs the tradespace extra, so the tests of searches with Optuna and pymoo, of parallel studies, and of plots run there. Without joblib, the tests of parallel studies skip, as those of Optuna and pymoo do without them.
pandas 3 (on Python 3.11 and later) gives read-only views of a frame's values, so attribute_variance's differences failed to assign NaN to them.
… (2023) in a validation notebook
…s whose costs are those incurred by the end of the mission
…iciency methods with a synthetic nature run
…, each adding one idea, in docs/examples/tradespace
…oaded nature run in its data directory
…olor plots of designs by at most three groups of planes
…onstellation fragments (planes and per_plane), and add fragments of streets-of-coverage constellations and mutual orbiting groups
…ons, fingerprint geometries by content, count availability and annual costs correctly, read design values by path, add the efficient designs to summaries of criteria, give an observation study the mission of a design, describe layouts, and share the logic of deployments, phases, and bootstraps (new modules simulation, dominance, and hashing)
…vations of the same study with other code (keeping those of studies that differ in constants or analyses), key designs given by value as their levels, fingerprint the classes and objects of a study's own modules by their attributes and code, summarize criteria across scenarios, name statistics by convention, color designs by values, plot shares of variance, apply a style to the plots of trade studies, refer to a design's restoration policy, give the histories of a design's failures, and seed searches with Optuna by default
…udy's functions by content, evaluate designs of other spaces in the study's space, check constraints in scenarios, judge designs by their feasible replicates (with the share feasible), cache the phases of missions by replicate, keep the launch interval in replacements, replace only the occupied positions, price launches by the masses of the deployment, fall back from exact expectations, chain screenings, give each step the designs it selects, configure policies and spares in one place (MissionPlan.policy), give the mission of a design (mission_of), make options keyword-only, and split criteria, space, and observations (new modules statistics, summary, walks, alerts, plans, and references)
… orientation (count, rotate, and seed) in place of generate_points_rotated_lattice, whose rotated lattices now also sample masks that cross the antimeridian
…ingerprint__ (designs, and caches of satellites by their function and namespace), reject decisions that name no child, record errors of realizing designs, plan each deployment with its restoration policy (plan(policy=), Deployment.policy), launch ground spares at the next slot, give an infinite deployed time to deployments that outlast the mission, judge resamples of pareto_probability by their drawn replicates, charge refine's budget for every scenario, give the spread of statistics across scenarios, reject statistics named as columns of the results (P(<requirement>) histories), keep the caller's pymoo algorithm, compute the metrics of every satellite once, speed up exact expectations, generate targets only to observe them, keep contexts in caches of satellites, export configure, value_of, Statistic, and the classes of steps, make options keyword-only, and move shared helpers into Criteria and new modules tables, expectations, and memory
…es aggregate (spread=, criteria as Criteria or directions, metrics and statistics in statistics=, the standard errors of summarize in pareto_probability), studies describe their designs (describe=, a description column), enumerate gives the extensions of designs (DesignSpace.enumerate(designs)), Failures take statistics over histories (Quantile, Probability, StandardDeviation) and require a seed, requirements on phases are the study's with or without failures, every field of a MissionPlan is a value or a reference (also stages) with the design's policy by default, masses come from the plan (no SatelliteCost.mass), Triangular takes its mode by keyword, the optimizers' options are keyword-only, plot_designs rings the front by default and annotates with annotate, the outcomes of steps share their first fields (Screening.summaries one frame), StandardError and design_columns are gone, the internals of criteria, design spaces, stores, missions, and observations are private (DesignSpace no longer encodes designs), simulate_histories folds into simulate_many, phases are evaluated one way, fingerprint rules are a table, and spreads of infinite values are missing without warnings
…r attributes, count only the resamples in which a design is judged in the standard errors of pareto_probability (and refine designs whose standard error is unknown), keep infinite means under control variates, take statistics over histories of equal weights as over replicates, reject two requirements on the phases of one metric and design values policy that configure no policy, key the cache of phases by the satellites of each content, judge rows without a value of by as a group of their own, give the summaries of searches, screenings, and confirmations the spreads of means, take a fixed seed of targets, a single constraint of layout fragments, and references in per-type masses, list the values of designs for completion (with suggestions of misspelled names), give the direction error for misspelled directions in summarize, require a front to annotate designs, and enumerate at most 4,096 sets for exact expectations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds two things:
tatc.tradespace(prototype): trade studies of constellation designs. It covers declaring a design space, evaluating designs over uncertain replicates, judging and searching them, and modeling the deployment, failures, and life-cycle cost of each design.tatc.analysis.sufficiency: estimates whether a constellation's observations sample a field well enough to estimate its mean over a period. It has been validated against IMERG precipitation.Everything is additive. Existing modules keep their API and gain only new exports. The new optional dependencies (joblib, matplotlib, Optuna, pymoo) are in a
tradespaceextra (pip install -e ".[tradespace]").What's added
Sampling sufficiency (
analysis.sufficiency,generation)estimate_sampling_error: the error in a period mean estimated from each target's samples, for a field with an exponential correlation in time. It also gives the effective number of independent samples.sample_fieldandcompare_sampled_means: check those estimates against a gridded field such as a nature run.compute_revisit_statisticsandcompute_local_time_coverage.generation.generate_points_rotated_lattice: a Fibonacci lattice in a random orientation. This stratified sample lowers the variance between replicates of a study.ValidateSamplingErrorIMERG, with limitations stated in the validation summary.Trade space (
tradespace)DesignSpace):Real,Integer,Categorical),Derivedvalues, andUncertainparameters drawn per replicate;ChoiceandRepeat;Study,ObservationStudy):SatelliteCachecomputes each satellite once across designs;ResultStore(SQLite) keys results by a fingerprint of the study's definition, so a study that is interrupted or extended resumes.Criteria(objectives,Requirements, statistics over replicates);hypervolume, andpareto_probability;refine(adaptive replication),confirm(fresh replicates against the winner's curse), andscreen(multi-fidelity);sensitivity,attribute_variance,Utility, andexceedance.Study(scenarios=...),robustness).Acrosslets searches find designs robust to scenarios.optimize_optunaandoptimize_pymooevaluate each design only once.optimize_optunacan stop at a budget of designs (max_designs);optimize_pymoostops on pymoo's termination criterion.MissionPlan):DeploymentSchedule(development, production rate, launch cadence and capacity);RestorationPolicy;Stage,Decision);CostModellife-cycle costs: learning curves, production blocks, present value, peak annual cost, and cost risk structured by component, after Maier, Yeakel & Wendoloski (2023).build_constellationbuilds them with instruments or with entries of the instrument catalog.plot_designs,plot_progress,plot_means, andplot_exceedance.Examples
SearchTradeSpace,RankUncertainDesigns,CompareScenarios,TradeCostAndUtility, andPlanMissions. They share a design space and study defined indocs/examples/trade_space_example.py.RunOSSEMonsoonnow declares its constellation trade as a design space.Notes for reviewers
Size:
tradespacesourceWhere to start: the
SearchTradeSpacenotebook walks through the main path. A suggested order for the source:space.pystudy.pycriteria.pyandpareto.pymission.py,deployment.py,failures.py, andcost.pyobservations.pyLayering:
tests/tradespace/test_layering.pychecks that the general modules (spaces, studies, criteria) don't import the mission-specific ones.Prototype:
tradespaceis marked as a prototype in the changelog, so its API may still change.Branch history: the branch merged
instrument-catalog(Add a validated instrument catalog #157) along the way and is up to date withmain, so this diff contains only the trade-space and sufficiency work.Testing
.[preprocess,tradespace], so it also runs the searches (Optuna, pymoo), parallel studies (joblib), and plots (matplotlib). pymoo has wheels for Python 3.10 through 3.14.main:blackpasses, andpylintpasses the 9.0 threshold that CI uses..[preprocess]installed. Tests that need Optuna, pymoo, joblib, matplotlib, or xarray skip when the package is missing.