diff --git a/workshops/idetc26/.gitignore b/workshops/idetc26/.gitignore index 48854efa..a08a5c46 100644 --- a/workshops/idetc26/.gitignore +++ b/workshops/idetc26/.gitignore @@ -4,3 +4,6 @@ cache/ # Scratch dir a notebook run creates next to itself; never shipped. notebooks/idetc26_work/ + +# Local planning notes are not part of the workshop deliverable. +PLAN.md diff --git a/workshops/idetc26/README.md b/workshops/idetc26/README.md index 70761a2c..f427aaff 100644 --- a/workshops/idetc26/README.md +++ b/workshops/idetc26/README.md @@ -1,25 +1,26 @@ # IDETC'26 — Find the Best Model -A hands-on workshop about what you would have to measure before you were -entitled to say one generative model is better than another. +A hands-on workshop about the evidence needed to compare generative models for +engineering design. -Teams get a bank of models for one engineering design problem — a lookup table, -a supervised deconvolutional network, GANs, VQGANs, a diffusion model, an -autoencoder, all named for what they are — and every cheap instrument the benchmark has. Then they find -that the columns disagree, that they disagree differently depending on the space -they measure in, and that the ranking moves when only the random seed changes. +Teams work with ten candidate models for one topology-optimization problem: two +baselines, five trained generative models, and three constructed candidates. The +models are named, evaluated under the same conditions, and compared using cost, +distribution, constraint, and physics-based metrics. The resulting rankings +depend on both the selected metric and the representation space. -**The answer is meant to be unknown.** Not withheld, not rigged — genuinely -undetermined by the metrics the field reports. The bank is chosen by searching -the published checkpoint pool for the subset whose metrics disagree most, and -disagreement is measured rather than hoped for. +There is no model that is best independently of the intended use. Participants +must decide which evidence matters for their application and state what remains +uncertain. -**Status: harness working end to end; bank selection in progress.** See -[Open questions](#open-questions). +**Status:** the Beams2D participant notebook, candidate bank, controls, cached +designs, and published physics results are implemented and runnable in Colab. +The main operational dependency is a cold-runtime download of approximately +510 MB after package installation. --- -## The argument +## Workshop motivation > Habibi et al., **[When Is it Actually Worth Learning Inverse Design?](https://asmedigitalcollection.asme.org/mechanicaldesign/article/148/6/061704/1229197/When-is-it-Actually-Worth-Learning-Inverse-Design)** > *J. Mech. Des.* 148(6):061704 (2026), first presented at IDETC-CIE 2023. @@ -30,153 +31,136 @@ deconvolutional networks when training data is limited**, once the cost of generating that data is counted. It is an IDEAL Lab paper, on this task, first given at this conference. -So the session is the natural sequel: *you* try to answer that question for a -bank of real models, using the metrics you can afford, and discover that you -cannot. Both sides of that paper's comparison are in the bank as **competitive -entries**, not as jokes: `knn_retrieval` at k=1 is the retrieval method, and -`deconv_regression` is the deconvolutional network it was measured against. +The workshop revisits that question with a broader model bank and a larger set +of evaluation criteria. Both sides of the paper's comparison are included as +full candidates: `knn_retrieval` at k=1 and `deconv_regression`. -On `beams2d` the kNN baseline currently beats the conditional GAN on condition -adherence, at 1/1600th of VQGAN's sampling cost, and the two are -within 5% of each other on MMD. A leaderboard topped by retrieval is a result, -not a broken leaderboard. +On `beams2d`, the kNN baseline outperforms the conditional GAN on condition +adherence at 1/1600th of VQGAN's sampling cost, while kNN and VQGAN are within +5% of each other on MMD. This is why the baselines are ranked alongside the +trained models. ## Run it +[Open the participant notebook in Google Colab](https://colab.research.google.com/github/IDEALLab/EngiOpt/blob/feat/idetc26-workshop/workshops/idetc26/notebooks/01_find_the_best_model.ipynb) + +The GitHub copy is read-only. Participants should select **File > Save a copy in +Drive** before editing. + ```bash jupyter lab workshops/idetc26/notebooks/01_find_the_best_model.ipynb ``` Colab installs EngiOpt from the branch named by the single `BRANCH` constant in -`tools/build_notebook.py`. A GPU speeds up sampling from the trained models; it -does **not** speed up the physics, which is CPU-bound topology optimization. - -## The notebook runs in one direction - -Ten suspects are in the room, each claiming to be a good generative model for -the same problem, each handed the same 50 design briefs. One of them is the -best. Six sections, and each is about the same objects as the one before it, so -nothing needs re-introducing: - -**1 · the data** → **2 · the suspects**, looked at, and a ranking written down -before any number → **3 · the questions**, one family at a time, several of -which contradict each other → **4 · the same questions in a fitted space**, -which is the pixel columns again with a projection in front of them → -**5 · the board**, the participant's own columns against the published simulator -run → **6 · free work**, with `case.help()` as the reference. The accusation at -the end is spoken aloud, not recorded by code. - -That is a deliberate reversal of the earlier "toolbox with no order" build. A -catalogue of independent tools is a reference manual, and nobody learns a -subject from a reference manual in ninety minutes. The order is what gives a -participant a reason to run the next cell, and it gives the disagreement between -metrics somewhere to land. - -**The prose is short on purpose.** Sentence fragments where a fragment says it, -full sentences only where a method is being explained, and no justification of -the API's design — that is what this README is for. The audience is researchers; -a cell that argues with them about why it exists is a cell they skim. +`tools/build_notebook.py` and EngiBench from `ENGIBENCH_REF`. The setup writes a +versioned marker under `/content` after verifying the installation. Repeated +executions skip installation and do not restart the runtime. + +The notebook requests a T4. A GPU speeds up sampling after a cache miss; it does +**not** speed up the physics, which is CPU-bound topology optimization. A cold +runtime still downloads the EngiBench dataset and latent encoder from Hugging +Face, so network access is required during setup. + +## Notebook structure + +Ten suspects are evaluated on the same 50 design conditions. The notebook has +six sections: + +**1 · the data** → **2 · the suspects** → **3 · the metric families** → +**4 · pixel, PCA, and learned latent spaces** → **5 · a selected metric board** → +**6 · the participant's accusation**. The final recommendation is discussed in +the room rather than recorded by the notebook. + +The sequence moves from visual inspection to numerical evaluation and then to +the participant's own comparison. The notebook keeps API explanation brief; +the detailed command reference is available through `case.help()`. **There are two commands.** `case.evaluate` for numbers, `case.show` for pictures: ```python +from engiopt.workshops.idetc26 import Case + case = Case.open("beams2d") case.help() # the cheat sheet: five calls, then every knob case.models() # who is in the line-up case.metrics() # what may be asked, by line of questioning -case.evaluate("diversity") # one whole line of questioning +case.evaluate("diversity") # all diversity metrics case.evaluate("mmd", models="knn") # one question, one suspect -case.evaluate(["cost", "similarity"], controls=True) +answers = case.evaluate(["mmd", "novelty_ratio", "pixel_vendi"], controls=True) case.show("diffusion") # its designs case.show("knn", "diffusion") # two suspects on the same brief case.show(answers) # a table of answers, drawn as ranks -case.show(answers, physics) # two boards joined and drawn as one +case.show(answers, case.physics()) # selected metrics with published physics case.show("knn", how="nearest_training") # each design beside its closest training design case.show("vqgan", "test", how="space_map") # where its designs sit in a fitted space ``` -**Neither command has a blanket form.** `case.evaluate()` and `case.show()` both -raise, and the omission is the pedagogy: a board of every cheap column against -every suspect is read rather than argued with, and a contact sheet of all ten -line-ups is skimmed. Which three columns a person chose is the session, so -naming the question is the work. Both errors say which call to make instead — a -bare `TypeError` in front of ninety people is a room full of raised hands. - -**`case.help()` is the cheat sheet**, run as the sixth cell and pointed at again -in the free-work section. It opens with the five calls that matter — `models`, -`metrics`, `explain`, `evaluate`, `show` — and everything verbose sits below -that: the views, the latent map, the controls, every knob with its default. The -notebook documents none of it. Markdown argument tables went stale every time an -argument moved and made each section twice as long as the idea in it; a printed -sheet beside the code cannot. - -The previous build had `compute`, `evaluate`, `board`, `score`, `run_physics`, -`seed_stability`, `reference_row`, `rank` and `winners` — nine ways to get a -number, two of which had names that meant the same thing in English. They are -one function now, with the differences expressed as arguments. A participant who -has to work out which of `compute` and `evaluate` they wanted is spending their -attention on the API rather than on the argument. - -The interface is **simple by default and complete underneath**: `evaluate("mmd")` -is one word, and `evaluate("mmd", sigma=0.5, n_samples=10, random_conditions=True)` -argues with the kernel bandwidth and the comparison set for anyone who wants to. - -**Suspects are named for what they are**, from the first cell — `knn_retrieval`, -`diffusion_2d_cond`, `cgan_cnn_2d` — and reachable by any unambiguous fragment -(`"diffusion"`, `"knn"`). The anonymized `Model A`/`Model B` framing is gone: -"why is the lookup table beating the diffusion model" is the question worth -having, and it cannot be asked of `Model C`. - -**No two suspects differ only by training seed.** A seed is not a model, and a -line-up containing seed-pairs invites a ranking that separates them — which is a -ranking of the random number generator. - -**And there is no `seeds=` argument**, deliberately. The spec freezes the 50 -briefs (`EvalSpec.condition_seed`), so the only thing a seed could vary in -`evaluate` is the *sampling noise* on a fixed checkpoint — a far weaker question -than it appears, offered under a name that invited it to be mistaken for the -strong one. The question with teeth needs checkpoints at several **training** -seeds. The Hub has them (seeds 1–10 for each family's replicated configuration); -pulling them is the natural next version of the session. - -### Six lines of questioning, and a seventh that is not about the models - -Nobody arrives knowing what `pca_coverage` is. They can hold six questions in -their head, so every column the benchmark computes is filed under one, and -naming a family is as real a call as naming a metric: - -| | asks | satisfied perfectly by | +**Neither command has a blanket form.** Calling `case.evaluate()` or +`case.show()` without specifying what to evaluate or display raises an error. +This requires participants to choose the metrics and models relevant to their +argument instead of generating a complete board by default. The errors include +an example of a valid call. + +**`case.help()` is the command reference.** It starts with the five calls used in +the notebook — `models`, `metrics`, `explain`, `evaluate`, and `show` — followed +by the available views, controls, and optional arguments. Keeping this reference +in the API prevents notebook instructions from drifting when an argument +changes. + +An earlier API exposed separate methods for computing, scoring, ranking, and +running physics. The current interface consolidates numerical evaluation under +`case.evaluate()` and expresses the differences as arguments. + +The short form, `evaluate("mmd")`, uses the frozen evaluation specification. +Optional arguments such as `sigma`, `n_samples`, and `random_conditions` allow +participants to test how evaluation choices affect the result. + +**Suspects are named by model family** from the first cell — `knn_retrieval`, +`diffusion_2d_cond`, and `cgan_cnn_2d`, for example — and can be addressed by an +unambiguous fragment such as `"diffusion"` or `"knn"`. Naming the models allows +participants to connect metric behavior with architecture and training method. + +**No two suspects differ only by training seed.** The bank compares model +families and configurations rather than treating repeated training runs as +separate candidates. + +There is no `seeds=` argument in the participant API. The specification freezes +the 50 design conditions, and changing the sampling seed on one checkpoint does +not measure variation across training runs. A training-seed study requires +separate checkpoints; those are outside the current participant workflow. + +### Metric groups + +Metrics are grouped by the question they answer. Participants can evaluate a +whole group or select individual metrics: + +| Group | Question | Limitation or failure mode | |---|---|---| -| `cost` | what did it take to put this model in the room? | — | -| `similarity` | do its designs look like the real ones? | handing back the training set | -| `memorization` | is it inventing, or copying out of the case files? | random noise | -| `diversity` | has it more than one answer, or one story it repeats? | corruption | -| `obedience` | did it answer the question it was actually asked? | material that carries no load | -| `performance` | are the designs actually any good? | *(needs the simulator)* | - -The seventh, `latent_space`, holds `lv_dual_gap`, `lv_active_dims` and -`pca_dims` — diagnostics of the fitted space the `lv_` columns are measured in, -never ranked, because they are not properties of a model. It was called -`instrument` and the accessor beside it is `case.latent_space()`; both are -`latent_space` now, since "instrument" told a participant nothing about which -part of the suite it belonged to. `case.latent_space()` names the autoencoder, -its measured active width, and the PCA width matched to it. - -The right-hand column is the point: every family has a way of being satisfied by -a model that is obviously bad, which is why no single one settles the argument. - -**These are not the registry's own families.** The registry files `novelty` -under `distribution`, beside `mmd` — but `mmd` is *minimized* by copying the -training set and `novelty` is the column that catches you doing it. Filing them -together is precisely what lets a board look coherent while containing its own -refutation, so `memorization` is split out. Likewise the registry files every -`lv_*` column under `latent`, describing what it depends on rather than what it -asks; here `lv_mmd` joins `mmd`, and the **space** it measures in becomes its own -column of the catalogue: +| `cost` | What did training and generation require? | Does not measure design quality | +| `similarity` | Do generated designs resemble held-out designs? | Returning training designs can score well | +| `memorization` | Are generated designs distinct from the training set? | Random noise can appear novel | +| `diversity` | How much does the generated set vary? | Corruption can increase diversity scores | +| `obedience` | Do designs satisfy the requested conditions? | Meeting a volume budget does not imply structural quality | +| `performance` | How do designs perform under optimization? | Requires the simulator | + +The seventh group, `latent_space`, contains `lv_dual_gap`, `lv_active_dims`, and +`pca_dims`. These values describe the fitted representation rather than a +candidate model, so they are not ranked. `case.latent_space()` identifies the +autoencoder, its active width, and the matched PCA width. + +Each metric group has a known failure mode, so no group is sufficient on its +own. + +These workshop groups differ from the metric registry's implementation +families. The registry places `novelty` and `mmd` under `distribution`, but they +answer different questions: copying the training set can improve MMD, while +novelty is intended to detect that behavior. The workshop therefore separates +memorization. Similarly, each `lv_*` metric is grouped by the question it +answers, while its learned-latent dependency appears in the `space` column: | the question | pixels | PCA subspace | learned latent | |---|---|---|---| @@ -186,56 +170,51 @@ column of the catalogue: | how many distinct designs? | `pixel_vendi`, `dpp_geometric` | `pca_vendi` | `lv_vendi` | | did it answer the brief? | `pixel_paired_distance` | — | `lv_paired_distance` | -Those rows disagree, and the disagreement is a fact about the spaces rather than -about the suspects. - -Nobody writes plotting code. `case.show` dispatches on what it is handed — a -suspect name, two names, a board, a board and two of its columns — with the -specialist views behind `how=`: `designs`, `compare`, `conditions`, -`nearest_training`, `space_map`. - -`space_map` is the only one that draws a *distribution* rather than designs: a -suspect's 50 designs as 50 points in the pinned latent space or the matched PCA -subspace, over the training set as a backdrop graded by how good each design is. -It is the shape that `mmd` reduces to one number, and collapse, near-miss and -missing coverage are all legible in it and none of them are legible in the -value. Drawing the same model in `space="lv"` and `space="pca"` is the cheapest -demonstration that a distribution metric is a claim about a space. - -`case.evaluate` is `python -m engiopt.evaluate` in a notebook: same evaluator, -same frozen spec, same argument names, and it prints the CLI line that produces -the same numbers, so the skill transfers out of the notebook. Metrics are never -computed on a private code path. - -The physics is **not a separate reveal** — it is the seventh line of questioning, -asked the same way as the other six. Ask it and the estimated time is printed -before the run starts, so a run that is longer than somebody wanted gets -interrupted like any other cell. The published board (`case.physics()`) is there -because nobody can run the simulator on 50 designs during a session. - -## Nothing waits on sampling - -Drawing the spec's 50 conditions from every suspect — one of them a diffusion model -on a CPU runtime — used to be the first thing a room did, in silence, twice if -anyone restarted the kernel. Those designs are a fixture: identical for every -team, every time. So they are precomputed and shipped inside the package -(`engiopt/workshops/idetc26/cache/`), and looking at a model costs nothing. +Differences between these columns can come from the representation space rather +than the candidate model. + +`case.show()` accepts a model name, two model names, a metric board, or selected +columns from a board. The `how=` argument provides the `designs`, `compare`, +`conditions`, `nearest_training`, and `space_map` views. + +`space_map` displays a candidate's 50 designs in the pinned latent space or the +matched PCA subspace, with the training set as a reference. It exposes collapse, +poor coverage, and isolated samples that are difficult to interpret from a +single MMD value. Comparing `space="lv"` with `space="pca"` shows how the chosen +representation affects a distribution metric. + +`case.evaluate()` uses the same evaluator, frozen specification, and argument +names as `python -m engiopt.evaluate`. It prints the equivalent CLI command so +results can be reproduced outside the notebook. + +Physics-based metrics use the same evaluation interface as the cheap metrics. +The published board, available through `case.physics()`, avoids running the +simulator over all 50 designs during the session. A live request prints its +estimated cost before simulation begins. + +## Cached designs and network requirements + +Candidate designs for the specification's 50 conditions are precomputed and +shipped under `engiopt/workshops/idetc26/cache/`. The normal participant path +therefore does not sample checkpoints or download model weights. ```bash # rebuild after the bank changes -- a cluster/workstation job, not a laptop one python workshops/idetc26/tools/build_design_cache.py --problem-id beams2d --seeds 1 2 3 ``` -Costs that are worth feeling are still paid in full: `case.evaluate(fresh=True)` -resamples from the checkpoints and times it, and the simulator costs what it -costs. A replayed `gen_seconds` says which machine measured it, since wall-clock -only compares models timed on the same one. +`case.evaluate(fresh=True)` explicitly resamples checkpoints and records the +runtime. Replayed `gen_seconds` values retain the machine on which they were +measured because wall-clock comparisons are only meaningful on comparable +hardware. -A cached bank member also does not download its checkpoint at `Case.open`, -so opening the workshop no longer pulls a 1.8 GB VQGAN before showing anybody -anything. +A cached candidate does not download its checkpoint at `Case.open()`, avoiding a +1.8 GB VQGAN download in the participant path. A cold runtime still downloads +the EngiBench dataset and latent encoder from Hugging Face. The measured transfer +was approximately 510 MB after package installation, so the workshop requires +reliable network access during setup. -## What is in the box +## Repository structure ``` workshops/idetc26/ @@ -244,53 +223,46 @@ workshops/idetc26/ tools/build_design_cache.py pre-sample the bank so the session never waits tools/search_pool.py score the pool, then search it for a bank tools/condition_probe.py does a model respond to its condition at all? - tools/build_sealed_board.py compute and seal the physics board tools/verify_notebook.py execute every cell, fail on the first error -engiopt/workshops/idetc26/ bank, challenge, config, views, sealing - problems/beams2d.json the parameterization unit - sealed/ the preregistered physics board - cache/ pre-sampled designs, shipped +engiopt/workshops/idetc26/ participant API, bank, config, and views + problems/beams2d.json per-problem configuration + cache/ packaged, pre-sampled designs engiopt/baselines/ dataset-fitted models (see below) engiopt/generators/knn_retrieval/ kNN as a first-class generator package engiopt/generators/linear_regression/ ridge regression, likewise tests/test_idetc26.py harness verification ``` -The directory is `tools/`, not `build/`, because the root `.gitignore` ignores -`build/` and silently dropped every script in it from a commit. +Colab installs the `engiopt` package rather than the repository's `workshops/` +tree. Problem configurations and design caches therefore live inside the package +and are included through `[tool.setuptools.package-data]`. A source checkout is +also supported; the package location is checked first. -**The data lives inside the package, not beside the notebooks.** Colab runs -`pip install git+...`, which ships `engiopt` and nothing else — no `workshops/` -tree. Problem configs, the sealed board and the design cache were unreachable -from a Colab runtime while they sat at the repository root, and are now package -data (`[tool.setuptools.package-data]` carries `*.json`, `sealed/*.enc` and -`cache/**/*.npz`). A source checkout still works: both locations are searched, -package first. +## The model bank and controls -## The bank, and what is not in it +The Beams2D bank contains ten ranked candidates. Seven are pretrained entries: +`knn_retrieval`, `deconv_regression`, `vqgan`, `diffusion_2d_cond`, +`cgan_cnn_2d`, `gan_cnn_2d`, and `constrained_plvae_2d`. The other three are +constructed candidates designed to expose weaknesses in individual evaluation +criteria. -Bank members are **plausibly-valid models somebody might actually use**: trained -checkpoints from the Euler sweep (`cgan_cnn_2d`, `gan_cnn_2d`, `vqgan`, -`diffusion_2d_cond`, `constrained_plvae_2d`) plus the two published baselines. -Selection is by measured disagreement, not by hand. +The participant-facing API describes what every model does but does not disclose +which candidates were constructed for the workshop. The facilitator must make +that disclosure after participants present their recommendations. The notebook +does not contain a passphrase or an automated reveal. -`engiopt/baselines/references.py` holds four deliberately-flawed models — -`collapsed`, `volume_only`, `noise_doped`, `checkerboard`. Every one carries -`bank_eligible = False` and `ModelBank` refuses to rank them. They are -**calibration standards**: what does a diversity metric read on one repeated -design, or on real designs plus noise at known severity? That belongs on a -reference row beneath a board, the way a scale bar belongs on a micrograph. -Putting them in the bank would be a trick; putting them beside it is the -measurement. +The bank also provides three controls: `collapsed`, `noise_doped`, and +`volume_only`. Controls provide reference values for interpreting metrics and +are never included in candidate rankings. -## The baselines are published like any other model +## Published baselines `knn_retrieval` and `deconv_regression` are full generator packages with training scripts that write checkpoints through `save_checkpoint_package`. They -are not notebook-only conveniences. (`linear_regression` is one too and remains -in the repo; it was dropped from the bank in favour of the deconvolutional -network, which is the comparator the paper actually used.) +are not notebook-specific implementations. `linear_regression` remains available +in the repository but is not part of the workshop bank; the deconvolutional +network is used because it is the comparator from the cited paper. | | `knn_retrieval` | `deconv_regression` | `diffusion_2d_cond` | `vqgan` | |---|---|---|---|---| @@ -298,13 +270,15 @@ network, which is the comparator the paper actually used.) | parameters | its training set | 3.7 M | 17.5 M | 102 M | | 50 samples | 0.054 s | one forward pass | 452 s | 85 s | -Every one reports a real parameter count, so a kNN's data cost and a network's -weight cost land on the same axis. The checkpoint sizes side by side are a -better answer to *what is a model?* than any definition. +Each package reports its parameter count, allowing the stored training data used +by kNN and the learned weights used by neural models to be compared on the same +cost axis. ## Measured costs -From actual runs on a laptop CPU, not estimates: +Development measurements on a laptop CPU gave the following ranges. The full +board used for these measurements contained eight candidates; the current bank +contains ten. | | cost | |---|---| @@ -312,83 +286,80 @@ From actual runs on a laptop CPU, not estimates: | the physics board, per model × 50 samples | ~90–125 s | | a whole 8-model physics board | **~15 min** | -The project's working assumption was ~16 GPU-hours. That came from SLURM -*requested* walltime and had never been measured; the real figure is two orders -of magnitude smaller. `beams2d`'s optimizer is CPU-bound, so **a GPU does not -help the physics at all.** The practical consequence is that teams could compute -part of the expensive board live, which is a different session from the one -where it has to be sealed. +`beams2d` optimization is CPU-bound, so a GPU does not reduce the physics cost. +Participants can evaluate a small subset live, while the complete physics board +is read from published results. -## Findings that changed the plan +## Known model behavior -**The diffusion model's conditioning is wired backwards.** `tools/condition_probe.py` -sweeps the requested volume fraction at fixed sampling noise and regresses -realized on requested: +`tools/condition_probe.py` sweeps the requested volume fraction at fixed sampling +noise and regresses the realized fraction on the request. The current diffusion +checkpoint has a negative conditioning response: | model | slope | r² | reading | |---|---|---|---| | `diffusion_2d_cond` | **−0.220** | 0.504 | ignores, and *anti-correlates with*, the request | | `vqgan` | 0.902 | 0.993 | follows the request | -Ask for more material, get less. VQGAN goes through identical plumbing at slope -0.90, so this is not a loading bug. The post-NeurIPS diffusion fixes addressed -design-range normalization and the noise schedule; none touched conditioning. -Hypothesis worth testing: conditions reach the UNet as a single length-4 -cross-attention token, **unnormalized**, with `volfrac` ∈ [0.15, 0.45] beside -`rmin` ∈ [1, 3]. Normalizing before `encoder_hid_proj` is the first thing to try, -and it needs a retrain. - -**The checkpoints are the current Euler runs.** Repos created 2026-08-06, still -being written. The new packages carry `design_min`/`design_max`; the legacy -NeurIPS checkpoints have no such keys, which is how you can tell them apart. -Euler's `ops/euler-sweeps` contains every diffusion and evaluation fix commit. - -**Two bugs of my own, both silent.** The three LVAE adapters dropped `problem` -when building, so all 18 PLVAE packages were unloadable. And the pool scorer -appended per-row frames with per-row columns to one CSV, shifting metric columns -for any family whose config keys differed from the first row's — which read as -NaN for exactly `vqgan` and `diffusion_2d_cond`, got them dropped by `dropna`, -and let the selector report six-way disagreement over a board containing only -GANs. Both fixed; the corrupted board was discarded rather than patched. +VQGAN follows the requested volume fraction through the same evaluation path, +which argues against a loading or measurement error. A current hypothesis is +that the UNet receives an unnormalized length-four conditioning token in which +`volfrac` ranges from 0.15 to 0.45 while `rmin` ranges from 1 to 3. Testing that +hypothesis requires retraining the diffusion model. + +Current checkpoint packages include `design_min` and `design_max`. Legacy +NeurIPS checkpoints do not, which provides a direct way to distinguish the two +formats. ## Adding a problem -Write `problems/.json`. Nothing else changes: +Add `engiopt/workshops/idetc26/problems/.json` with the frozen +EngiBench specification, display name, metric list, model bank, and controls: ```json { - "problem_id": "photonics2d", - "spec": "photonics2d/v1", - "opening_metrics": ["mmd", "dpp", "viol"], - "withheld_metrics": ["novelty", "cond_err", "pixel_vendi", "pca_mmd", "gen_seconds"], - "expensive_metrics": ["iog", "cog", "fog"], - "bank": [{"kind": "baseline", "algo": "knn_retrieval", "seed": 1}] + "problem_id": "example2d", + "spec": "example2d/v1", + "display_name": "Example2D", + "metrics": ["mmd", "pixel_vendi", "viol", "iog"], + "bank": [ + { + "kind": "pretrained", + "algo": "knn_retrieval", + "seed": 1, + "config_fingerprint": "..." + } + ], + "controls": [ + {"kind": "reference", "algo": "collapsed"}, + {"kind": "reference", "algo": "noise_doped"} + ] } ``` -`unavailable_metrics()` is **derived from the frozen spec**, not declared — so on -photonics2d, whose spec sets `volume_condition` to null, `viol` and the other -condition columns report themselves unavailable with a reason. A team -discovering mid-session that the column their neighbours are arguing about does -not exist for their problem is content, not a bug. - -## Open questions - -- **Bank not yet selected.** The pool rescore is running after the CSV fix; - `tools/search_pool.py select` then ranks candidate banks by distinct rank-1 - winners, mean Kendall τ, and rank spread. The sealed board must be rebuilt - afterwards — the committed one is for a superseded bank. -- **The slow families are under-sampled.** `vqgan` and `diffusion_2d_cond` are - capped at 4 packages each because sampling costs 85–540 s per package, against - 46 apiece for the GANs. A bank chosen from that pool over-represents GANs - unless selection is stratified, which it now is — but the underlying coverage - is still uneven and `select` reports `families` per candidate so it cannot be - missed. -- **kNN and linear regression are not on the Hub yet**, and not in the Euler - manifest. Both train in seconds on CPU and need no GPU arm. -- **Flow matching**: `codex/flow-matching-model` has a complete `core.py`; the - adapter is ~80 lines but there are no checkpoints, so it needs a training run. -- **No test covers `engiopt/baselines/`** since the old adversarial tests were - removed with the models they asserted. -- **`tests/test_idetc26.py` takes 49 minutes** because opening a challenge - downloads checkpoints. It needs the `network` marker the repo already declares. +`unavailable_metrics()` derives unsupported metrics from the frozen spec. For +example, `photonics2d` has no volume-fraction budget, so `viol` and related +condition metrics report themselves unavailable with a reason. + +Every candidate must have a resolvable checkpoint package or construction. +Before using a new problem in a workshop, build its design cache, publish or +verify its physics results, and execute the notebook against the new +configuration. + +## Editing and verification + +The notebook is generated from `tools/build_notebook.py`. Edit the builder and +commit the source and generated notebook together: + +```bash +python workshops/idetc26/tools/build_notebook.py +``` + +After changing the notebook or participant API, run: + +```bash +pytest tests/test_idetc26.py -m "not network" +python workshops/idetc26/tools/verify_notebook.py +``` + +The notebook verification executes every code cell and requires network access. diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index e2598a6d..62b89a7f 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -2,14 +2,14 @@ "cells": [ { "cell_type": "markdown", - "id": "323eb03c", + "id": "6f0b624a", "metadata": {}, "source": [ "# The generative model murder mystery\n", "\n", "**IDETC-CIE 2026 · EngiBench hands-on workshop**\n", "\n", - "*Ten generative models were trained on the same topology optimization problem. You must put your detective hat on and find the best performing model. That is our killer.*" + "*Ten generative models were trained on the same topology optimization problem. The best-performing model is hidden among the suspects. Your task is to identify it from the evidence.*" ] }, { @@ -32,34 +32,33 @@ { "cell_type": "code", "execution_count": null, - "id": "cd557831", + "id": "9d0491a1", "metadata": {}, "outputs": [], "source": [ "import sys\n", + "from pathlib import Path\n", "\n", "if \"google.colab\" in sys.modules:\n", - " %pip install -q \"git+https://github.com/IDEALLab/EngiOpt.git@feat/idetc26-workshop\"\n", - " # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls the\n", - " # requirement satisfied and leaves photonics2d at v0. --force-reinstall is\n", - " # what actually swaps the code; --no-deps because main declares none that\n", - " # 0.2.0 did not.\n", - " %pip install -q --force-reinstall --no-deps \"engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@main\"\n", + " setup_marker = Path(\"/content/.idetc26_setup_v1\")\n", + " if setup_marker.exists():\n", + " print(\"Setup already complete.\")\n", + " else:\n", + " %pip install -q \"git+https://github.com/IDEALLab/EngiOpt.git@feat/idetc26-workshop\"\n", + " # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls\n", + " # the requirement satisfied and leaves photonics2d at v0. Force the\n", + " # GitHub version into the current environment before EngiBench is first\n", + " # imported by the next cell.\n", + " %pip install -q --force-reinstall --no-deps \"engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@main\"\n", "\n", - " import sysconfig\n", - " from pathlib import Path\n", - " installed = Path(sysconfig.get_paths()[\"purelib\"]) / \"engibench/problems/photonics2d/v1.py\"\n", - " print(f\"photonics2d v1 present: {installed.exists()}\")\n", + " import sysconfig\n", "\n", - " # Restart automatically. The install replaces files on disk, but a runtime\n", - " # that already imported the old engibench keeps serving it from sys.modules\n", - " # -- which looks like the install silently failing, on a runtime where it\n", - " # plainly succeeded. Asking people to restart by hand is the step that gets\n", - " # skipped. Colab reconnects on its own; just run the next cell.\n", - " print(\"Restarting the runtime. When it reconnects, carry on from the next cell.\")\n", - " import IPython\n", + " installed = Path(sysconfig.get_paths()[\"purelib\"]) / \"engibench/problems/photonics2d/v1.py\"\n", + " if not installed.exists():\n", + " raise RuntimeError(\"EngiBench installation failed: photonics2d v1 is missing.\")\n", "\n", - " IPython.Application.instance().kernel.do_shutdown(restart=True)" + " setup_marker.write_text(\"EngiOpt feat/idetc26-workshop; EngiBench main\\n\")\n", + " print(\"Setup complete. Continue with the next cell.\")" ] }, { @@ -76,10 +75,10 @@ }, { "cell_type": "markdown", - "id": "9d0adfad", + "id": "d00b179e", "metadata": {}, "source": [ - "The cheat sheet — refer to this anytime you want to call a function" + "Use this cheat sheet whenever you need the API." ] }, { @@ -147,10 +146,10 @@ }, { "cell_type": "markdown", - "id": "ba048273", + "id": "2488a059", "metadata": {}, "source": [ - "Each model has a case file explaining who they are. Pull it with `case.explain()`" + "Use `case.explain()` to open a model's case file." ] }, { @@ -165,12 +164,12 @@ }, { "cell_type": "markdown", - "id": "da0aeb69", + "id": "bc7cfef5", "metadata": {}, "source": [ "### The evidence (visualizing our suspects)\n", "\n", - "Show generated outputs from a given model" + "Show outputs generated by one model" ] }, { @@ -229,13 +228,13 @@ }, { "cell_type": "markdown", - "id": "18a906e9", + "id": "345a310e", "metadata": {}, "source": [ "---\n", "# 3 · The interrogation/questions\n", "\n", - "We can look at many metrics to assist in our final verdict. Metrics provide insight into how well the model performs in various categories: cost, similarity, novelty, diversity, feasibility, and performance" + "Each metric examines a different aspect of the model: cost, similarity, novelty, diversity, feasibility, or performance." ] }, { @@ -250,12 +249,12 @@ }, { "cell_type": "markdown", - "id": "cadf7d8e", + "id": "37ceb325", "metadata": {}, "source": [ "## Cost — how expensive is the model?\n", "\n", - "If the training/inferencing the model is nearly as expensive as the traditional optimizer, is it worth it?\n", + "If training or running the model costs nearly as much as the traditional optimizer, does the model offer a practical advantage? Is it worth it?\n", "\n", "Cost metrics: `train_minutes`, `gen_seconds`, `params`" ] @@ -272,12 +271,12 @@ }, { "cell_type": "markdown", - "id": "2a3d009f", + "id": "c0a3f758", "metadata": {}, "source": [ "## Similarity — does it look like the real thing?\n", "\n", - "How close do generated designs resemble traditional optimized designs?\n", + "How close do generated designs resemble traditionally optimized designs?\n", "\n", "Similarity metrics: `mmd`, `pixel_paired_distance`" ] @@ -294,15 +293,14 @@ }, { "cell_type": "markdown", - "id": "e5c60626", + "id": "92d18a24", "metadata": {}, "source": [ "## Novelty — Are the designs new or memorized?\n", "\n", - "`novelty_ratio` is a scaled distance from each generated design to the nearest design\n", - "it could have copied. Around 1 is as novel as a real held-out design, near 0 is\n", - "memorized, far above 1 looks like nothing in the data — which random pixels also\n", - "achieve." + "Values around 1 are comparable to held-out designs, while values near 0 suggest\n", + "memorization. Values far above 1 may indicate out-of-distribution outputs;\n", + "random noise can also appear highly novel." ] }, { @@ -335,18 +333,19 @@ }, { "cell_type": "markdown", - "id": "5633a5af", + "id": "f4065cac", "metadata": {}, "source": [ "## Diversity — How different are our generated designs from each other?\n", "\n", - "Diversity scores are hard to interpret without a relative scale. We can add some control data augmentations for reference with `controls=True`\n", + "Diversity scores need reference points. Use `controls=True` to include three known controls.\n", + "\n", "\n", "| control | what it means |\n", "|---|---|\n", "| `collapsed` | an averaged design repeated 50 times |\n", "| `noise_doped` | real optima plus Gaussian noise |\n", - "| `volume_only` | random blob that hits volume budget exactly|\n", + "| `volume_only` | random design satisfying only the volume constraint|\n", "\n", "Diversity metrics: `vendi`, `dpp`" ] @@ -373,15 +372,15 @@ }, { "cell_type": "markdown", - "id": "0408ecae", + "id": "adca601b", "metadata": {}, "source": [ "## Obedience — did it meet the constraints?\n", "\n", "The design is not valid if it doesn't meet its budget\n", "\n", - "`cond_err` — Average distance from specified conditions (volume)\n", - "`viol` — a proportion of designs that missed by more than a tolerance" + "`cond_err` — mean error in the specified conditions (e.g., volume fraction)\n", + "`viol` — fraction of designs exceeding the permitted tolerance" ] }, { @@ -406,12 +405,15 @@ }, { "cell_type": "markdown", - "id": "52719265", + "id": "e381da38", "metadata": {}, "source": [ "## Performance — is the design actually any good?\n", "\n", - "Is the design close to optimum on generation (iog), does it require little effort to reach optimum on warmstart (cog), or does it converge to better/worse design when warmstarting (fog)?" + "Is the design close to optimum on generation (iog), does it require little effort to reach optimum on warmstart (cog), or does it converge to better/worse design when warmstarting (fog)?\n", + "`iog` measures the initial optimality gap before re-optimization. `cog` measures\n", + "the cumulative gap during warm-start optimization. `fog` measures the final gap\n", + "after optimization." ] }, { @@ -426,13 +428,13 @@ }, { "cell_type": "markdown", - "id": "7b5f69af", + "id": "c371a6be", "metadata": {}, "source": [ "---\n", "# 4 · The same questions, somewhere other than pixels\n", "\n", - "Every similarity/distance so far compared designs in pixel space. We can also project to featural spaces. Here we do so with PCA or through a learned autoencoder\n", + "Every similarity/distance used so far compared designs in pixel space. We can also project to featural spaces. Here we compute the corresponding metrics in a PCA subspace or through a learned autoencoder latent space\n", "\n", "| the question | pixels | PCA subspace | learned latent |\n", "|---|---|---|---|\n", @@ -455,10 +457,12 @@ }, { "cell_type": "markdown", - "id": "fb09fd9c", + "id": "2945afa8", "metadata": {}, "source": [ - "PCA and LV project to the same dimensionality for this notebook (determined through least volume). LV refers to the encoded latent space from `constrained_plvae_2d`" + "For this comparison, PCA is matched to the active dimensionality of the\n", + "least-volume autoencoder. The `lv_...` metrics use the encoded latent space of\n", + "`constrained_plvae_2d`." ] }, { @@ -491,10 +495,10 @@ }, { "cell_type": "markdown", - "id": "f3eaa313", + "id": "8be4570b", "metadata": {}, "source": [ - "Same designs, linear space. Nothing about the model changed; the picture can." + "Same designs projected in linear PCA space. Nothing about the model changed; differences between plots come from the representation." ] }, { @@ -529,17 +533,17 @@ }, { "cell_type": "markdown", - "id": "24b0dd33", + "id": "dff6aab5", "metadata": {}, "source": [ "---\n", "# 6 · Your accusation\n", "\n", - "Out loud, to the room:\n", + "Choose the model you would use and be ready to explain your choice to the room:\n", "\n", "1. **Who did it.** The model you think is the best.\n", "2. **On what evidence.** Why do you think so?\n", - "3. **What you could not rule out.** What might have the decision easier?\n", + "3. **What you could not rule out.** What additional evidence would make your decision easier?\n", "\n", "The cell below is yours to build with.\n", "\n", diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index 73d8a37e..3dcc7215 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -100,7 +100,7 @@ def code(text: str) -> dict: **IDETC-CIE 2026 · EngiBench hands-on workshop** -*Ten generative models were trained on the same topology optimization problem. You must put your detective hat on and find the best performing model. That is our killer.* +*Ten generative models were trained on the same topology optimization problem. The best-performing model is hidden among the suspects. Your task is to identify it from the evidence.* """ ), md("""**Before you edit anything:** File → Save a copy in Drive. Opens read-only from GitHub."""), @@ -114,29 +114,28 @@ def code(text: str) -> dict: code( f""" import sys +from pathlib import Path if "google.colab" in sys.modules: - %pip install -q "git+https://github.com/IDEALLab/EngiOpt.git@{BRANCH}" - # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls the - # requirement satisfied and leaves photonics2d at v0. --force-reinstall is - # what actually swaps the code; --no-deps because main declares none that - # 0.2.0 did not. - %pip install -q --force-reinstall --no-deps "engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@{ENGIBENCH_REF}" - - import sysconfig - from pathlib import Path - installed = Path(sysconfig.get_paths()["purelib"]) / "engibench/problems/photonics2d/v1.py" - print(f"photonics2d v1 present: {{installed.exists()}}") - - # Restart automatically. The install replaces files on disk, but a runtime - # that already imported the old engibench keeps serving it from sys.modules - # -- which looks like the install silently failing, on a runtime where it - # plainly succeeded. Asking people to restart by hand is the step that gets - # skipped. Colab reconnects on its own; just run the next cell. - print("Restarting the runtime. When it reconnects, carry on from the next cell.") - import IPython - - IPython.Application.instance().kernel.do_shutdown(restart=True) + setup_marker = Path("/content/.idetc26_setup_v1") + if setup_marker.exists(): + print("Setup already complete.") + else: + %pip install -q "git+https://github.com/IDEALLab/EngiOpt.git@{BRANCH}" + # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls + # the requirement satisfied and leaves photonics2d at v0. Force the + # GitHub version into the current environment before EngiBench is first + # imported by the next cell. + %pip install -q --force-reinstall --no-deps "engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@{ENGIBENCH_REF}" + + import sysconfig + + installed = Path(sysconfig.get_paths()["purelib"]) / "engibench/problems/photonics2d/v1.py" + if not installed.exists(): + raise RuntimeError("EngiBench installation failed: photonics2d v1 is missing.") + + setup_marker.write_text("EngiOpt {BRANCH}; EngiBench {ENGIBENCH_REF}\\n") + print("Setup complete. Continue with the next cell.") """ ), code( @@ -146,7 +145,7 @@ def code(text: str) -> dict: case = Case.open("beams2d") # <- the problem you are working on """ ), - md("""The cheat sheet — refer to this anytime you want to call a function"""), + md("""Use this cheat sheet whenever you need the API."""), code("""case.help()"""), # ---- 1 · the scene of the crime ---- md( @@ -170,14 +169,14 @@ def code(text: str) -> dict: """ ), code("""case.models()"""), - md("""Each model has a case file explaining who they are. Pull it with `case.explain()`"""), + md("""Use `case.explain()` to open a model's case file."""), code("""case.explain("knn_retrieval")"""), # ---- the evidence ---- md( """ ### The evidence (visualizing our suspects) -Show generated outputs from a given model +Show outputs generated by one model """ ), code("""case.show("diffusion", n=12)"""), @@ -203,7 +202,7 @@ def code(text: str) -> dict: --- # 3 · The interrogation/questions -We can look at many metrics to assist in our final verdict. Metrics provide insight into how well the model performs in various categories: cost, similarity, novelty, diversity, feasibility, and performance +Each metric examines a different aspect of the model: cost, similarity, novelty, diversity, feasibility, or performance. """ ), code("""case.metrics()"""), @@ -212,7 +211,7 @@ def code(text: str) -> dict: """ ## Cost — how expensive is the model? -If the training/inferencing the model is nearly as expensive as the traditional optimizer, is it worth it? +If training or running the model costs nearly as much as the traditional optimizer, does the model offer a practical advantage? Is it worth it? Cost metrics: `train_minutes`, `gen_seconds`, `params` """ @@ -223,7 +222,7 @@ def code(text: str) -> dict: """ ## Similarity — does it look like the real thing? -How close do generated designs resemble traditional optimized designs? +How close do generated designs resemble traditionally optimized designs? Similarity metrics: `mmd`, `pixel_paired_distance` """ @@ -234,10 +233,9 @@ def code(text: str) -> dict: """ ## Novelty — Are the designs new or memorized? -`novelty_ratio` is a scaled distance from each generated design to the nearest design -it could have copied. Around 1 is as novel as a real held-out design, near 0 is -memorized, far above 1 looks like nothing in the data — which random pixels also -achieve. +Values around 1 are comparable to held-out designs, while values near 0 suggest +memorization. Values far above 1 may indicate out-of-distribution outputs; +random noise can also appear highly novel. """ ), code("""case.evaluate("novelty_ratio").round(3)"""), @@ -248,13 +246,14 @@ def code(text: str) -> dict: """ ## Diversity — How different are our generated designs from each other? -Diversity scores are hard to interpret without a relative scale. We can add some control data augmentations for reference with `controls=True` +Diversity scores need reference points. Use `controls=True` to include three known controls. + | control | what it means | |---|---| | `collapsed` | an averaged design repeated 50 times | | `noise_doped` | real optima plus Gaussian noise | -| `volume_only` | random blob that hits volume budget exactly| +| `volume_only` | random design satisfying only the volume constraint| Diversity metrics: `vendi`, `dpp` """ @@ -268,8 +267,8 @@ def code(text: str) -> dict: The design is not valid if it doesn't meet its budget -`cond_err` — Average distance from specified conditions (volume) -`viol` — a proportion of designs that missed by more than a tolerance +`cond_err` — mean error in the specified conditions (e.g., volume fraction) +`viol` — fraction of designs exceeding the permitted tolerance """ ), code("""case.evaluate(["cond_err", "viol"], controls=True).round(4)"""), @@ -280,6 +279,9 @@ def code(text: str) -> dict: ## Performance — is the design actually any good? Is the design close to optimum on generation (iog), does it require little effort to reach optimum on warmstart (cog), or does it converge to better/worse design when warmstarting (fog)? +`iog` measures the initial optimality gap before re-optimization. `cog` measures +the cumulative gap during warm-start optimization. `fog` measures the final gap +after optimization. """ ), code("""case.evaluate("cog", models=["knn_retrieval", "cgan_cnn_2d"]).round(3)"""), @@ -289,7 +291,7 @@ def code(text: str) -> dict: --- # 4 · The same questions, somewhere other than pixels -Every similarity/distance so far compared designs in pixel space. We can also project to featural spaces. Here we do so with PCA or through a learned autoencoder +Every similarity/distance used so far compared designs in pixel space. We can also project to featural spaces. Here we compute the corresponding metrics in a PCA subspace or through a learned autoencoder latent space | the question | pixels | PCA subspace | learned latent | |---|---|---|---| @@ -302,13 +304,15 @@ def code(text: str) -> dict: ), code("""case.show(case.evaluate(["mmd", "pca_mmd", "lv_mmd"]))"""), md( - """PCA and LV project to the same dimensionality for this notebook (determined through least volume). LV refers to the encoded latent space from `constrained_plvae_2d`""" + """For this comparison, PCA is matched to the active dimensionality of the +least-volume autoencoder. The `lv_...` metrics use the encoded latent space of +`constrained_plvae_2d`.""" ), code("""case.latent_space() # which autoencoder, how wide, and the PCA width matched to it"""), # ---- looking at the space ---- md("""### Looking at the space"""), code("""case.show("knn_retrieval", "test", how="space_map")"""), - md("""Same designs, linear space. Nothing about the model changed; the picture can."""), + md("""Same designs projected in linear PCA space. Nothing about the model changed; differences between plots come from the representation."""), code("""case.show("knn_retrieval", "test", how="space_map", space="pca")"""), # ---- 5 · visualizing a board ---- md( @@ -330,11 +334,11 @@ def code(text: str) -> dict: --- # 6 · Your accusation -Out loud, to the room: +Choose the model you would use and be ready to explain your choice to the room: 1. **Who did it.** The model you think is the best. 2. **On what evidence.** Why do you think so? -3. **What you could not rule out.** What might have the decision easier? +3. **What you could not rule out.** What additional evidence would make your decision easier? The cell below is yours to build with.