From 405ab28bc0be7273c715d3982a5b1bb427976009 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 17:25:41 +0200 Subject: [PATCH 1/9] docs(workshop): refine IDETC notebook prose --- .../notebooks/01_find_the_best_model.ipynb | 77 ++++++++++--------- workshops/idetc26/tools/build_notebook.py | 45 ++++++----- 2 files changed, 66 insertions(+), 56 deletions(-) diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index e2598a6d..094257d0 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -2,14 +2,14 @@ "cells": [ { "cell_type": "markdown", - "id": "323eb03c", + "id": "6f0b624a", "metadata": {}, "source": [ "# The generative model murder mystery\n", "\n", "**IDETC-CIE 2026 · EngiBench hands-on workshop**\n", "\n", - "*Ten generative models were trained on the same topology optimization problem. You must put your detective hat on and find the best performing model. That is our killer.*" + "*Ten generative models were trained on the same topology optimization problem. The best-performing model is hidden among the suspects. Your task is to identify it from the evidence.*" ] }, { @@ -76,10 +76,10 @@ }, { "cell_type": "markdown", - "id": "9d0adfad", + "id": "d00b179e", "metadata": {}, "source": [ - "The cheat sheet — refer to this anytime you want to call a function" + "Use this cheat sheet whenever you need the API." ] }, { @@ -147,10 +147,10 @@ }, { "cell_type": "markdown", - "id": "ba048273", + "id": "2488a059", "metadata": {}, "source": [ - "Each model has a case file explaining who they are. Pull it with `case.explain()`" + "Use `case.explain()` to open a model's case file." ] }, { @@ -165,12 +165,12 @@ }, { "cell_type": "markdown", - "id": "da0aeb69", + "id": "bc7cfef5", "metadata": {}, "source": [ "### The evidence (visualizing our suspects)\n", "\n", - "Show generated outputs from a given model" + "Show outputs generated by one model" ] }, { @@ -229,13 +229,13 @@ }, { "cell_type": "markdown", - "id": "18a906e9", + "id": "345a310e", "metadata": {}, "source": [ "---\n", "# 3 · The interrogation/questions\n", "\n", - "We can look at many metrics to assist in our final verdict. Metrics provide insight into how well the model performs in various categories: cost, similarity, novelty, diversity, feasibility, and performance" + "Each metric examines a different aspect of the model: cost, similarity, novelty, diversity, feasibility, or performance." ] }, { @@ -250,12 +250,12 @@ }, { "cell_type": "markdown", - "id": "cadf7d8e", + "id": "37ceb325", "metadata": {}, "source": [ "## Cost — how expensive is the model?\n", "\n", - "If the training/inferencing the model is nearly as expensive as the traditional optimizer, is it worth it?\n", + "If training or running the model costs nearly as much as the traditional optimizer, does the model offer a practical advantage? Is it worth it?\n", "\n", "Cost metrics: `train_minutes`, `gen_seconds`, `params`" ] @@ -272,12 +272,12 @@ }, { "cell_type": "markdown", - "id": "2a3d009f", + "id": "c0a3f758", "metadata": {}, "source": [ "## Similarity — does it look like the real thing?\n", "\n", - "How close do generated designs resemble traditional optimized designs?\n", + "How close do generated designs resemble traditionally optimized designs?\n", "\n", "Similarity metrics: `mmd`, `pixel_paired_distance`" ] @@ -294,15 +294,14 @@ }, { "cell_type": "markdown", - "id": "e5c60626", + "id": "92d18a24", "metadata": {}, "source": [ "## Novelty — Are the designs new or memorized?\n", "\n", - "`novelty_ratio` is a scaled distance from each generated design to the nearest design\n", - "it could have copied. Around 1 is as novel as a real held-out design, near 0 is\n", - "memorized, far above 1 looks like nothing in the data — which random pixels also\n", - "achieve." + "Values around 1 are comparable to held-out designs, while values near 0 suggest\n", + "memorization. Values far above 1 may indicate out-of-distribution outputs;\n", + "random noise can also appear highly novel." ] }, { @@ -335,18 +334,19 @@ }, { "cell_type": "markdown", - "id": "5633a5af", + "id": "f4065cac", "metadata": {}, "source": [ "## Diversity — How different are our generated designs from each other?\n", "\n", - "Diversity scores are hard to interpret without a relative scale. We can add some control data augmentations for reference with `controls=True`\n", + "Diversity scores need reference points. Use `controls=True` to include three known controls.\n", + "\n", "\n", "| control | what it means |\n", "|---|---|\n", "| `collapsed` | an averaged design repeated 50 times |\n", "| `noise_doped` | real optima plus Gaussian noise |\n", - "| `volume_only` | random blob that hits volume budget exactly|\n", + "| `volume_only` | random design satisfying only the volume constraint|\n", "\n", "Diversity metrics: `vendi`, `dpp`" ] @@ -373,15 +373,15 @@ }, { "cell_type": "markdown", - "id": "0408ecae", + "id": "adca601b", "metadata": {}, "source": [ "## Obedience — did it meet the constraints?\n", "\n", "The design is not valid if it doesn't meet its budget\n", "\n", - "`cond_err` — Average distance from specified conditions (volume)\n", - "`viol` — a proportion of designs that missed by more than a tolerance" + "`cond_err` — mean error in the specified conditions (e.g., volume fraction)\n", + "`viol` — fraction of designs exceeding the permitted tolerance" ] }, { @@ -406,12 +406,15 @@ }, { "cell_type": "markdown", - "id": "52719265", + "id": "e381da38", "metadata": {}, "source": [ "## Performance — is the design actually any good?\n", "\n", - "Is the design close to optimum on generation (iog), does it require little effort to reach optimum on warmstart (cog), or does it converge to better/worse design when warmstarting (fog)?" + "Is the design close to optimum on generation (iog), does it require little effort to reach optimum on warmstart (cog), or does it converge to better/worse design when warmstarting (fog)?\n", + "`iog` measures the initial optimality gap before re-optimization. `cog` measures\n", + "the cumulative gap during warm-start optimization. `fog` measures the final gap\n", + "after optimization." ] }, { @@ -426,13 +429,13 @@ }, { "cell_type": "markdown", - "id": "7b5f69af", + "id": "c371a6be", "metadata": {}, "source": [ "---\n", "# 4 · The same questions, somewhere other than pixels\n", "\n", - "Every similarity/distance so far compared designs in pixel space. We can also project to featural spaces. Here we do so with PCA or through a learned autoencoder\n", + "Every similarity/distance used so far compared designs in pixel space. We can also project to featural spaces. Here we compute the corresponding metrics in a PCA subspace or through a learned autoencoder latent space\n", "\n", "| the question | pixels | PCA subspace | learned latent |\n", "|---|---|---|---|\n", @@ -455,10 +458,12 @@ }, { "cell_type": "markdown", - "id": "fb09fd9c", + "id": "2945afa8", "metadata": {}, "source": [ - "PCA and LV project to the same dimensionality for this notebook (determined through least volume). LV refers to the encoded latent space from `constrained_plvae_2d`" + "For this comparison, PCA is matched to the active dimensionality of the\n", + "least-volume autoencoder. The `lv_...` metrics use the encoded latent space of\n", + "`constrained_plvae_2d`." ] }, { @@ -491,10 +496,10 @@ }, { "cell_type": "markdown", - "id": "f3eaa313", + "id": "8be4570b", "metadata": {}, "source": [ - "Same designs, linear space. Nothing about the model changed; the picture can." + "Same designs projected in linear PCA space. Nothing about the model changed; differences between plots come from the representation." ] }, { @@ -529,7 +534,7 @@ }, { "cell_type": "markdown", - "id": "24b0dd33", + "id": "238b05f9", "metadata": {}, "source": [ "---\n", @@ -539,9 +544,9 @@ "\n", "1. **Who did it.** The model you think is the best.\n", "2. **On what evidence.** Why do you think so?\n", - "3. **What you could not rule out.** What might have the decision easier?\n", + "3. **What you could not rule out.** What additional evidence would make the decision easier?\n", "\n", - "The cell below is yours to build with.\n", + "The cell below is yours to build with. Get to work.\n", "\n", "Remember `case.help()`" ] diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index 73d8a37e..85718f20 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -100,7 +100,7 @@ def code(text: str) -> dict: **IDETC-CIE 2026 · EngiBench hands-on workshop** -*Ten generative models were trained on the same topology optimization problem. You must put your detective hat on and find the best performing model. That is our killer.* +*Ten generative models were trained on the same topology optimization problem. The best-performing model is hidden among the suspects. Your task is to identify it from the evidence.* """ ), md("""**Before you edit anything:** File → Save a copy in Drive. Opens read-only from GitHub."""), @@ -146,7 +146,7 @@ def code(text: str) -> dict: case = Case.open("beams2d") # <- the problem you are working on """ ), - md("""The cheat sheet — refer to this anytime you want to call a function"""), + md("""Use this cheat sheet whenever you need the API."""), code("""case.help()"""), # ---- 1 · the scene of the crime ---- md( @@ -170,14 +170,14 @@ def code(text: str) -> dict: """ ), code("""case.models()"""), - md("""Each model has a case file explaining who they are. Pull it with `case.explain()`"""), + md("""Use `case.explain()` to open a model's case file."""), code("""case.explain("knn_retrieval")"""), # ---- the evidence ---- md( """ ### The evidence (visualizing our suspects) -Show generated outputs from a given model +Show outputs generated by one model """ ), code("""case.show("diffusion", n=12)"""), @@ -203,7 +203,7 @@ def code(text: str) -> dict: --- # 3 · The interrogation/questions -We can look at many metrics to assist in our final verdict. Metrics provide insight into how well the model performs in various categories: cost, similarity, novelty, diversity, feasibility, and performance +Each metric examines a different aspect of the model: cost, similarity, novelty, diversity, feasibility, or performance. """ ), code("""case.metrics()"""), @@ -212,7 +212,7 @@ def code(text: str) -> dict: """ ## Cost — how expensive is the model? -If the training/inferencing the model is nearly as expensive as the traditional optimizer, is it worth it? +If training or running the model costs nearly as much as the traditional optimizer, does the model offer a practical advantage? Is it worth it? Cost metrics: `train_minutes`, `gen_seconds`, `params` """ @@ -223,7 +223,7 @@ def code(text: str) -> dict: """ ## Similarity — does it look like the real thing? -How close do generated designs resemble traditional optimized designs? +How close do generated designs resemble traditionally optimized designs? Similarity metrics: `mmd`, `pixel_paired_distance` """ @@ -234,10 +234,9 @@ def code(text: str) -> dict: """ ## Novelty — Are the designs new or memorized? -`novelty_ratio` is a scaled distance from each generated design to the nearest design -it could have copied. Around 1 is as novel as a real held-out design, near 0 is -memorized, far above 1 looks like nothing in the data — which random pixels also -achieve. +Values around 1 are comparable to held-out designs, while values near 0 suggest +memorization. Values far above 1 may indicate out-of-distribution outputs; +random noise can also appear highly novel. """ ), code("""case.evaluate("novelty_ratio").round(3)"""), @@ -248,13 +247,14 @@ def code(text: str) -> dict: """ ## Diversity — How different are our generated designs from each other? -Diversity scores are hard to interpret without a relative scale. We can add some control data augmentations for reference with `controls=True` +Diversity scores need reference points. Use `controls=True` to include three known controls. + | control | what it means | |---|---| | `collapsed` | an averaged design repeated 50 times | | `noise_doped` | real optima plus Gaussian noise | -| `volume_only` | random blob that hits volume budget exactly| +| `volume_only` | random design satisfying only the volume constraint| Diversity metrics: `vendi`, `dpp` """ @@ -268,8 +268,8 @@ def code(text: str) -> dict: The design is not valid if it doesn't meet its budget -`cond_err` — Average distance from specified conditions (volume) -`viol` — a proportion of designs that missed by more than a tolerance +`cond_err` — mean error in the specified conditions (e.g., volume fraction) +`viol` — fraction of designs exceeding the permitted tolerance """ ), code("""case.evaluate(["cond_err", "viol"], controls=True).round(4)"""), @@ -280,6 +280,9 @@ def code(text: str) -> dict: ## Performance — is the design actually any good? Is the design close to optimum on generation (iog), does it require little effort to reach optimum on warmstart (cog), or does it converge to better/worse design when warmstarting (fog)? +`iog` measures the initial optimality gap before re-optimization. `cog` measures +the cumulative gap during warm-start optimization. `fog` measures the final gap +after optimization. """ ), code("""case.evaluate("cog", models=["knn_retrieval", "cgan_cnn_2d"]).round(3)"""), @@ -289,7 +292,7 @@ def code(text: str) -> dict: --- # 4 · The same questions, somewhere other than pixels -Every similarity/distance so far compared designs in pixel space. We can also project to featural spaces. Here we do so with PCA or through a learned autoencoder +Every similarity/distance used so far compared designs in pixel space. We can also project to featural spaces. Here we compute the corresponding metrics in a PCA subspace or through a learned autoencoder latent space | the question | pixels | PCA subspace | learned latent | |---|---|---|---| @@ -302,13 +305,15 @@ def code(text: str) -> dict: ), code("""case.show(case.evaluate(["mmd", "pca_mmd", "lv_mmd"]))"""), md( - """PCA and LV project to the same dimensionality for this notebook (determined through least volume). LV refers to the encoded latent space from `constrained_plvae_2d`""" + """For this comparison, PCA is matched to the active dimensionality of the +least-volume autoencoder. The `lv_...` metrics use the encoded latent space of +`constrained_plvae_2d`.""" ), code("""case.latent_space() # which autoencoder, how wide, and the PCA width matched to it"""), # ---- looking at the space ---- md("""### Looking at the space"""), code("""case.show("knn_retrieval", "test", how="space_map")"""), - md("""Same designs, linear space. Nothing about the model changed; the picture can."""), + md("""Same designs projected in linear PCA space. Nothing about the model changed; differences between plots come from the representation."""), code("""case.show("knn_retrieval", "test", how="space_map", space="pca")"""), # ---- 5 · visualizing a board ---- md( @@ -334,9 +339,9 @@ def code(text: str) -> dict: 1. **Who did it.** The model you think is the best. 2. **On what evidence.** Why do you think so? -3. **What you could not rule out.** What might have the decision easier? +3. **What you could not rule out.** What additional evidence would make the decision easier? -The cell below is yours to build with. +The cell below is yours to build with. Get to work. Remember `case.help()` """ From 3a934b6b7f0a8a1e6739846d418eba5476aecf2b Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 17:39:34 +0200 Subject: [PATCH 2/9] fix(workshop): make Colab setup idempotent --- .../notebooks/01_find_the_best_model.ipynb | 43 +++++++++-------- workshops/idetc26/tools/build_notebook.py | 47 +++++++++---------- 2 files changed, 44 insertions(+), 46 deletions(-) diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index 094257d0..8d7b1c9a 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -32,34 +32,33 @@ { "cell_type": "code", "execution_count": null, - "id": "cd557831", + "id": "53267677", "metadata": {}, "outputs": [], "source": [ "import sys\n", + "from pathlib import Path\n", "\n", "if \"google.colab\" in sys.modules:\n", - " %pip install -q \"git+https://github.com/IDEALLab/EngiOpt.git@feat/idetc26-workshop\"\n", - " # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls the\n", - " # requirement satisfied and leaves photonics2d at v0. --force-reinstall is\n", - " # what actually swaps the code; --no-deps because main declares none that\n", - " # 0.2.0 did not.\n", - " %pip install -q --force-reinstall --no-deps \"engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@main\"\n", + " setup_marker = Path(\"/content/.idetc26_setup_v1\")\n", + " if setup_marker.exists():\n", + " print(\"Setup already complete.\")\n", + " else:\n", + " %pip install -q \"git+https://github.com/IDEALLab/EngiOpt.git@codex/idetc26-workshop-participant-rework\"\n", + " # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls\n", + " # the requirement satisfied and leaves photonics2d at v0. Force the\n", + " # GitHub version into the current environment before EngiBench is first\n", + " # imported by the next cell.\n", + " %pip install -q --force-reinstall --no-deps \"engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@main\"\n", "\n", - " import sysconfig\n", - " from pathlib import Path\n", - " installed = Path(sysconfig.get_paths()[\"purelib\"]) / \"engibench/problems/photonics2d/v1.py\"\n", - " print(f\"photonics2d v1 present: {installed.exists()}\")\n", + " import sysconfig\n", "\n", - " # Restart automatically. The install replaces files on disk, but a runtime\n", - " # that already imported the old engibench keeps serving it from sys.modules\n", - " # -- which looks like the install silently failing, on a runtime where it\n", - " # plainly succeeded. Asking people to restart by hand is the step that gets\n", - " # skipped. Colab reconnects on its own; just run the next cell.\n", - " print(\"Restarting the runtime. When it reconnects, carry on from the next cell.\")\n", - " import IPython\n", + " installed = Path(sysconfig.get_paths()[\"purelib\"]) / \"engibench/problems/photonics2d/v1.py\"\n", + " if not installed.exists():\n", + " raise RuntimeError(\"EngiBench installation failed: photonics2d v1 is missing.\")\n", "\n", - " IPython.Application.instance().kernel.do_shutdown(restart=True)" + " setup_marker.write_text(\"EngiOpt codex/idetc26-workshop-participant-rework; EngiBench main\\n\")\n", + " print(\"Setup complete. Continue with the next cell.\")" ] }, { @@ -534,17 +533,17 @@ }, { "cell_type": "markdown", - "id": "238b05f9", + "id": "f23ba99f", "metadata": {}, "source": [ "---\n", "# 6 · Your accusation\n", "\n", - "Out loud, to the room:\n", + "Choose the model you would use and be ready to explain your choice to the room:\n", "\n", "1. **Who did it.** The model you think is the best.\n", "2. **On what evidence.** Why do you think so?\n", - "3. **What you could not rule out.** What additional evidence would make the decision easier?\n", + "3. **What you could not rule out.** What additional evidence would make your decision easier?\n", "\n", "The cell below is yours to build with. Get to work.\n", "\n", diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index 85718f20..76d8c3bf 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -38,7 +38,7 @@ NOTEBOOK_DIR = Path(__file__).resolve().parents[1] / "notebooks" -BRANCH = "feat/idetc26-workshop" +BRANCH = "codex/idetc26-workshop-participant-rework" """The EngiOpt branch Colab installs from.""" ENGIBENCH_REF = "main" @@ -114,29 +114,28 @@ def code(text: str) -> dict: code( f""" import sys +from pathlib import Path if "google.colab" in sys.modules: - %pip install -q "git+https://github.com/IDEALLab/EngiOpt.git@{BRANCH}" - # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls the - # requirement satisfied and leaves photonics2d at v0. --force-reinstall is - # what actually swaps the code; --no-deps because main declares none that - # 0.2.0 did not. - %pip install -q --force-reinstall --no-deps "engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@{ENGIBENCH_REF}" - - import sysconfig - from pathlib import Path - installed = Path(sysconfig.get_paths()["purelib"]) / "engibench/problems/photonics2d/v1.py" - print(f"photonics2d v1 present: {{installed.exists()}}") - - # Restart automatically. The install replaces files on disk, but a runtime - # that already imported the old engibench keeps serving it from sys.modules - # -- which looks like the install silently failing, on a runtime where it - # plainly succeeded. Asking people to restart by hand is the step that gets - # skipped. Colab reconnects on its own; just run the next cell. - print("Restarting the runtime. When it reconnects, carry on from the next cell.") - import IPython - - IPython.Application.instance().kernel.do_shutdown(restart=True) + setup_marker = Path("/content/.idetc26_setup_v1") + if setup_marker.exists(): + print("Setup already complete.") + else: + %pip install -q "git+https://github.com/IDEALLab/EngiOpt.git@{BRANCH}" + # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls + # the requirement satisfied and leaves photonics2d at v0. Force the + # GitHub version into the current environment before EngiBench is first + # imported by the next cell. + %pip install -q --force-reinstall --no-deps "engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@{ENGIBENCH_REF}" + + import sysconfig + + installed = Path(sysconfig.get_paths()["purelib"]) / "engibench/problems/photonics2d/v1.py" + if not installed.exists(): + raise RuntimeError("EngiBench installation failed: photonics2d v1 is missing.") + + setup_marker.write_text("EngiOpt {BRANCH}; EngiBench {ENGIBENCH_REF}\\n") + print("Setup complete. Continue with the next cell.") """ ), code( @@ -335,11 +334,11 @@ def code(text: str) -> dict: --- # 6 · Your accusation -Out loud, to the room: +Choose the model you would use and be ready to explain your choice to the room: 1. **Who did it.** The model you think is the best. 2. **On what evidence.** Why do you think so? -3. **What you could not rule out.** What additional evidence would make the decision easier? +3. **What you could not rule out.** What additional evidence would make your decision easier? The cell below is yours to build with. Get to work. From 6dd51c6f216a6325d8c41cd1e526de6b5542ace9 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 17:46:46 +0200 Subject: [PATCH 3/9] docs(workshop): align IDETC README with implementation --- workshops/idetc26/README.md | 506 ++++++++++++------------------------ 1 file changed, 163 insertions(+), 343 deletions(-) diff --git a/workshops/idetc26/README.md b/workshops/idetc26/README.md index 70761a2c..be605c4a 100644 --- a/workshops/idetc26/README.md +++ b/workshops/idetc26/README.md @@ -1,394 +1,214 @@ -# IDETC'26 — Find the Best Model +# IDETC-CIE 2026: Find the Best Model -A hands-on workshop about what you would have to measure before you were -entitled to say one generative model is better than another. +This workshop asks participants to compare generative models for an engineering +design problem. The models are evaluated under the same conditions, but the +metrics do not necessarily agree. Participants inspect the designs, select the +evidence they consider relevant, and explain which model they would use. -Teams get a bank of models for one engineering design problem — a lookup table, -a supervised deconvolutional network, GANs, VQGANs, a diffusion model, an -autoencoder, all named for what they are — and every cheap instrument the benchmark has. Then they find -that the columns disagree, that they disagree differently depending on the space -they measure in, and that the ranking moves when only the random seed changes. +## Status -**The answer is meant to be unknown.** Not withheld, not rigged — genuinely -undetermined by the metrics the field reports. The bank is chosen by searching -the published checkpoint pool for the subset whose metrics disagree most, and -disagreement is measured rather than hoped for. +The participant notebook, ten-model Beams2D bank, controls, cached designs, and +published physics results are implemented. The notebook runs in Google Colab and +uses one participant path from setup to the final discussion. -**Status: harness working end to end; bank selection in progress.** See -[Open questions](#open-questions). +The main operational dependency is network access. A cold Colab runtime installs +EngiOpt and EngiBench and downloads the EngiBench dataset and latent encoder. +Testing measured approximately 510 MB of downloads after package installation. ---- +## Participant notebook -## The argument +[Open the workshop in Google Colab](https://colab.research.google.com/github/IDEALLab/EngiOpt/blob/codex/idetc26-workshop-participant-rework/workshops/idetc26/notebooks/01_find_the_best_model.ipynb) -> Habibi et al., **[When Is it Actually Worth Learning Inverse Design?](https://asmedigitalcollection.asme.org/mechanicaldesign/article/148/6/061704/1229197/When-is-it-Actually-Worth-Learning-Inverse-Design)** -> *J. Mech. Des.* 148(6):061704 (2026), first presented at IDETC-CIE 2023. +The notebook opens read-only from GitHub. Participants should select +**File > Save a copy in Drive** before making changes. -That paper compared inverse-design model families on a topology-optimization -warm-start task and found that **k-nearest neighbours and random forests beat -deconvolutional networks when training data is limited**, once the cost of -generating that data is counted. It is an IDEAL Lab paper, on this task, first -given at this conference. - -So the session is the natural sequel: *you* try to answer that question for a -bank of real models, using the metrics you can afford, and discover that you -cannot. Both sides of that paper's comparison are in the bank as **competitive -entries**, not as jokes: `knn_retrieval` at k=1 is the retrieval method, and -`deconv_regression` is the deconvolutional network it was measured against. - -On `beams2d` the kNN baseline currently beats the conditional GAN on condition -adherence, at 1/1600th of VQGAN's sampling cost, and the two are -within 5% of each other on MMD. A leaderboard topped by retrieval is a result, -not a broken leaderboard. - -## Run it +To run it from a local checkout: ```bash jupyter lab workshops/idetc26/notebooks/01_find_the_best_model.ipynb ``` -Colab installs EngiOpt from the branch named by the single `BRANCH` constant in -`tools/build_notebook.py`. A GPU speeds up sampling from the trained models; it -does **not** speed up the physics, which is CPU-bound topology optimization. +### Colab setup -## The notebook runs in one direction +The setup cell installs: -Ten suspects are in the room, each claiming to be a good generative model for -the same problem, each handed the same 50 design briefs. One of them is the -best. Six sections, and each is about the same objects as the one before it, so -nothing needs re-introducing: +- EngiOpt from the branch defined by `BRANCH` in `tools/build_notebook.py`. +- EngiBench from the Git reference defined by `ENGIBENCH_REF`. -**1 · the data** → **2 · the suspects**, looked at, and a ranking written down -before any number → **3 · the questions**, one family at a time, several of -which contradict each other → **4 · the same questions in a fitted space**, -which is the pixel columns again with a projection in front of them → -**5 · the board**, the participant's own columns against the published simulator -run → **6 · free work**, with `case.help()` as the reference. The accusation at -the end is spoken aloud, not recorded by code. +The installation is guarded by `/content/.idetc26_setup_v1`. The first run +installs and verifies the required EngiBench problem version. Repeated executions +skip installation, so **Run all** does not trigger a restart loop. A factory reset +clears both the installed packages and the marker. -That is a deliberate reversal of the earlier "toolbox with no order" build. A -catalogue of independent tools is a reference manual, and nobody learns a -subject from a reference manual in ninety minutes. The order is what gives a -participant a reason to run the next cell, and it gives the disagreement between -metrics somewhere to land. +The notebook requests a T4 runtime. The normal participant path reads cached +designs and published physics, but a GPU prevents a long wait if a design cache +miss requires sampling from a checkpoint. -**The prose is short on purpose.** Sentence fragments where a fragment says it, -full sentences only where a method is being explained, and no justification of -the API's design — that is what this README is for. The audience is researchers; -a cell that argues with them about why it exists is a cell they skim. +## Workshop sequence -**There are two commands.** `case.evaluate` for numbers, `case.show` for -pictures: +The notebook contains six sections: -```python -case = Case.open("beams2d") +1. **The dataset**: inspect the training and held-out test designs as the problem + conditions change. +2. **The suspects**: inspect the ten candidate models and compare generated + designs under the same conditions. +3. **The interrogation**: evaluate cost, similarity, novelty, diversity, + constraint satisfaction, and optimization performance. +4. **Representation spaces**: compare distance-based measurements in pixels, a + matched PCA subspace, and a learned latent space. +5. **The board**: assemble a ranking from selected metrics. +6. **The accusation**: choose a model, explain the evidence, and identify what + else should be checked before using it in practice. -case.help() # the cheat sheet: five calls, then every knob -case.models() # who is in the line-up -case.metrics() # what may be asked, by line of questioning +The murder-mystery terminology is the workshop theme. The model names and metric +values are not anonymized. -case.evaluate("diversity") # one whole line of questioning -case.evaluate("mmd", models="knn") # one question, one suspect -case.evaluate(["cost", "similarity"], controls=True) +## Model bank -case.show("diffusion") # its designs -case.show("knn", "diffusion") # two suspects on the same brief -case.show(answers) # a table of answers, drawn as ranks -case.show(answers, physics) # two boards joined and drawn as one -case.show("knn", how="nearest_training") # each design beside its closest training design -case.show("vqgan", "test", how="space_map") # where its designs sit in a fitted space -``` +The Beams2D bank contains ten candidates evaluated on the same frozen EngiBench +specification: -**Neither command has a blanket form.** `case.evaluate()` and `case.show()` both -raise, and the omission is the pedagogy: a board of every cheap column against -every suspect is read rather than argued with, and a contact sheet of all ten -line-ups is skimmed. Which three columns a person chose is the session, so -naming the question is the work. Both errors say which call to make instead — a -bare `TypeError` in front of ninety people is a room full of raised hands. - -**`case.help()` is the cheat sheet**, run as the sixth cell and pointed at again -in the free-work section. It opens with the five calls that matter — `models`, -`metrics`, `explain`, `evaluate`, `show` — and everything verbose sits below -that: the views, the latent map, the controls, every knob with its default. The -notebook documents none of it. Markdown argument tables went stale every time an -argument moved and made each section twice as long as the idea in it; a printed -sheet beside the code cannot. - -The previous build had `compute`, `evaluate`, `board`, `score`, `run_physics`, -`seed_stability`, `reference_row`, `rank` and `winners` — nine ways to get a -number, two of which had names that meant the same thing in English. They are -one function now, with the differences expressed as arguments. A participant who -has to work out which of `compute` and `evaluate` they wanted is spending their -attention on the API rather than on the argument. - -The interface is **simple by default and complete underneath**: `evaluate("mmd")` -is one word, and `evaluate("mmd", sigma=0.5, n_samples=10, random_conditions=True)` -argues with the kernel bandwidth and the comparison set for anyone who wants to. - -**Suspects are named for what they are**, from the first cell — `knn_retrieval`, -`diffusion_2d_cond`, `cgan_cnn_2d` — and reachable by any unambiguous fragment -(`"diffusion"`, `"knn"`). The anonymized `Model A`/`Model B` framing is gone: -"why is the lookup table beating the diffusion model" is the question worth -having, and it cannot be asked of `Model C`. - -**No two suspects differ only by training seed.** A seed is not a model, and a -line-up containing seed-pairs invites a ranking that separates them — which is a -ranking of the random number generator. - -**And there is no `seeds=` argument**, deliberately. The spec freezes the 50 -briefs (`EvalSpec.condition_seed`), so the only thing a seed could vary in -`evaluate` is the *sampling noise* on a fixed checkpoint — a far weaker question -than it appears, offered under a name that invited it to be mistaken for the -strong one. The question with teeth needs checkpoints at several **training** -seeds. The Hub has them (seeds 1–10 for each family's replicated configuration); -pulling them is the natural next version of the session. - -### Six lines of questioning, and a seventh that is not about the models - -Nobody arrives knowing what `pca_coverage` is. They can hold six questions in -their head, so every column the benchmark computes is filed under one, and -naming a family is as real a call as naming a metric: - -| | asks | satisfied perfectly by | -|---|---|---| -| `cost` | what did it take to put this model in the room? | — | -| `similarity` | do its designs look like the real ones? | handing back the training set | -| `memorization` | is it inventing, or copying out of the case files? | random noise | -| `diversity` | has it more than one answer, or one story it repeats? | corruption | -| `obedience` | did it answer the question it was actually asked? | material that carries no load | -| `performance` | are the designs actually any good? | *(needs the simulator)* | - -The seventh, `latent_space`, holds `lv_dual_gap`, `lv_active_dims` and -`pca_dims` — diagnostics of the fitted space the `lv_` columns are measured in, -never ranked, because they are not properties of a model. It was called -`instrument` and the accessor beside it is `case.latent_space()`; both are -`latent_space` now, since "instrument" told a participant nothing about which -part of the suite it belonged to. `case.latent_space()` names the autoencoder, -its measured active width, and the PCA width matched to it. - -The right-hand column is the point: every family has a way of being satisfied by -a model that is obviously bad, which is why no single one settles the argument. - -**These are not the registry's own families.** The registry files `novelty` -under `distribution`, beside `mmd` — but `mmd` is *minimized* by copying the -training set and `novelty` is the column that catches you doing it. Filing them -together is precisely what lets a board look coherent while containing its own -refutation, so `memorization` is split out. Likewise the registry files every -`lv_*` column under `latent`, describing what it depends on rather than what it -asks; here `lv_mmd` joins `mmd`, and the **space** it measures in becomes its own -column of the catalogue: - -| the question | pixels | PCA subspace | learned latent | -|---|---|---|---| -| does this look like the real data? | `mmd` | `pca_mmd` | `lv_mmd` | -| is it copying the training set? | `novelty_ratio` | — | `lv_novelty` | -| did it cover the real modes? | — | `pca_coverage` | `lv_coverage` | -| how many distinct designs? | `pixel_vendi`, `dpp_geometric` | `pca_vendi` | `lv_vendi` | -| did it answer the brief? | `pixel_paired_distance` | — | `lv_paired_distance` | - -Those rows disagree, and the disagreement is a fact about the spaces rather than -about the suspects. - -Nobody writes plotting code. `case.show` dispatches on what it is handed — a -suspect name, two names, a board, a board and two of its columns — with the -specialist views behind `how=`: `designs`, `compare`, `conditions`, -`nearest_training`, `space_map`. - -`space_map` is the only one that draws a *distribution* rather than designs: a -suspect's 50 designs as 50 points in the pinned latent space or the matched PCA -subspace, over the training set as a backdrop graded by how good each design is. -It is the shape that `mmd` reduces to one number, and collapse, near-miss and -missing coverage are all legible in it and none of them are legible in the -value. Drawing the same model in `space="lv"` and `space="pca"` is the cheapest -demonstration that a distribution metric is a claim about a space. - -`case.evaluate` is `python -m engiopt.evaluate` in a notebook: same evaluator, -same frozen spec, same argument names, and it prints the CLI line that produces -the same numbers, so the skill transfers out of the notebook. Metrics are never -computed on a private code path. - -The physics is **not a separate reveal** — it is the seventh line of questioning, -asked the same way as the other six. Ask it and the estimated time is printed -before the run starts, so a run that is longer than somebody wanted gets -interrupted like any other cell. The published board (`case.physics()`) is there -because nobody can run the simulator on 50 designs during a session. - -## Nothing waits on sampling - -Drawing the spec's 50 conditions from every suspect — one of them a diffusion model -on a CPU runtime — used to be the first thing a room did, in silence, twice if -anyone restarted the kernel. Those designs are a fixture: identical for every -team, every time. So they are precomputed and shipped inside the package -(`engiopt/workshops/idetc26/cache/`), and looking at a model costs nothing. +- k-nearest-neighbour retrieval +- supervised deconvolutional regression +- VQGAN +- conditional diffusion +- conditional GAN +- unconditional GAN +- constrained least-volume autoencoder +- three constructed candidates designed to expose weaknesses in individual + evaluation criteria -```bash -# rebuild after the bank changes -- a cluster/workstation job, not a laptop one -python workshops/idetc26/tools/build_design_cache.py --problem-id beams2d --seeds 1 2 3 -``` +The constructed candidates participate in the ranking. Their construction is +not printed by the participant-facing API. The facilitator must disclose them +after participants present their recommendations; there is no passphrase or +automated reveal in the notebook. -Costs that are worth feeling are still paid in full: `case.evaluate(fresh=True)` -resamples from the checkpoints and times it, and the simulator costs what it -costs. A replayed `gen_seconds` says which machine measured it, since wall-clock -only compares models timed on the same one. +Three controls provide reference values and are never ranked with the candidate +models: -A cached bank member also does not download its checkpoint at `Case.open`, -so opening the workshop no longer pulls a 1.8 GB VQGAN before showing anybody -anything. +- `collapsed`: one averaged design repeated for every condition +- `noise_doped`: held-out optimized designs with added Gaussian noise +- `volume_only`: random designs satisfying only the volume constraint -## What is in the box +## Evaluation interface -``` -workshops/idetc26/ - notebooks/01_find_the_best_model.ipynb the session (a build product) - tools/build_notebook.py notebook SOURCE -- edit here, not the .ipynb - tools/build_design_cache.py pre-sample the bank so the session never waits - tools/search_pool.py score the pool, then search it for a bank - tools/condition_probe.py does a model respond to its condition at all? - tools/build_sealed_board.py compute and seal the physics board - tools/verify_notebook.py execute every cell, fail on the first error - -engiopt/workshops/idetc26/ bank, challenge, config, views, sealing - problems/beams2d.json the parameterization unit - sealed/ the preregistered physics board - cache/ pre-sampled designs, shipped -engiopt/baselines/ dataset-fitted models (see below) -engiopt/generators/knn_retrieval/ kNN as a first-class generator package -engiopt/generators/linear_regression/ ridge regression, likewise -tests/test_idetc26.py harness verification +Participants work with one `Case` object: + +```python +from engiopt.workshops.idetc26 import Case + +case = Case.open("beams2d") +case.models() +case.metrics() +case.explain("diffusion") +case.show("diffusion") +case.evaluate("mmd") ``` -The directory is `tools/`, not `build/`, because the root `.gitignore` ignores -`build/` and silently dropped every script in it from a commit. +`case.evaluate()` returns numerical results. `case.show()` handles design +galleries, comparisons, condition checks, nearest-training examples, +representation-space maps, and metric boards. `case.help()` prints the full +participant reference. -**The data lives inside the package, not beside the notebooks.** Colab runs -`pip install git+...`, which ships `engiopt` and nothing else — no `workshops/` -tree. Problem configs, the sealed board and the design cache were unreachable -from a Colab runtime while they sat at the repository root, and are now package -data (`[tool.setuptools.package-data]` carries `*.json`, `sealed/*.enc` and -`cache/**/*.npz`). A source checkout still works: both locations are searched, -package first. +The workshop groups metrics by the question they answer: -## The bank, and what is not in it +| Group | Question | +|---|---| +| `cost` | What did training and generation require? | +| `similarity` | How closely does the generated set match held-out designs? | +| `memorization` | Are generated designs distinct from the training set? | +| `diversity` | How much does the generated set vary? | +| `obedience` | Do designs satisfy the requested conditions? | +| `performance` | How do designs perform under physics-based optimization? | +| `latent_space` | Is the fitted representation suitable for measurement? | -Bank members are **plausibly-valid models somebody might actually use**: trained -checkpoints from the Euler sweep (`cgan_cnn_2d`, `gan_cnn_2d`, `vqgan`, -`diffusion_2d_cond`, `constrained_plvae_2d`) plus the two published baselines. -Selection is by measured disagreement, not by hand. +The `latent_space` group contains diagnostics for the representation and is not +used to rank models. Expensive performance results are read from published +physics records when available. Passing `fresh=True` explicitly resamples a +model or recomputes results on the current machine. -`engiopt/baselines/references.py` holds four deliberately-flawed models — -`collapsed`, `volume_only`, `noise_doped`, `checkerboard`. Every one carries -`bank_eligible = False` and `ModelBank` refuses to rank them. They are -**calibration standards**: what does a diversity metric read on one repeated -design, or on real designs plus noise at known severity? That belongs on a -reference row beneath a board, the way a scale bar belongs on a micrograph. -Putting them in the bank would be a trick; putting them beside it is the -measurement. +## Data and runtime behavior -## The baselines are published like any other model +Pre-sampled candidate designs are packaged under +`engiopt/workshops/idetc26/cache/`. The normal participant path therefore does +not download candidate checkpoints or run model sampling. -`knn_retrieval` and `deconv_regression` are full generator packages with -training scripts that write checkpoints through `save_checkpoint_package`. They -are not notebook-only conveniences. (`linear_regression` is one too and remains -in the repo; it was dropped from the bank in favour of the deconvolutional -network, which is the comparator the paper actually used.) +`Case.open()` still resolves the frozen EngiBench dataset and the latent-space +encoder. These assets are downloaded from Hugging Face on a cold runtime and +then cached by Colab. Internet access is therefore required during setup. -| | `knn_retrieval` | `deconv_regression` | `diffusion_2d_cond` | `vqgan` | -|---|---|---|---|---| -| checkpoint | 38.9 MB | **14.8 MB** | ~70 MB | ~1.8 GB | -| parameters | its training set | 3.7 M | 17.5 M | 102 M | -| 50 samples | 0.054 s | one forward pass | 452 s | 85 s | +Published physics values are also read from the Hub. The notebook evaluates only +a small subset live; a complete simulator sweep is not expected during the +session. -Every one reports a real parameter count, so a kNN's data cost and a network's -weight cost land on the same axis. The checkpoint sizes side by side are a -better answer to *what is a model?* than any definition. +## Repository layout -## Measured costs +```text +workshops/idetc26/ + notebooks/01_find_the_best_model.ipynb generated participant notebook + tools/build_notebook.py canonical notebook source + tools/build_design_cache.py rebuild pre-sampled designs + tools/search_pool.py inspect and select candidate banks + tools/verify_notebook.py execute the notebook end to end + +engiopt/workshops/idetc26/ + case.py participant API + bank.py candidate and control loading + config.py per-problem configuration + designs.py cached design access + families.py participant metric groups + views.py notebook visualizations + problems/ workshop problem configurations + cache/ packaged generated designs +``` -From actual runs on a laptop CPU, not estimates: +## Editing the notebook -| | cost | -|---|---| -| the cheap board, 8 models × 50 samples | seconds | -| the physics board, per model × 50 samples | ~90–125 s | -| a whole 8-model physics board | **~15 min** | - -The project's working assumption was ~16 GPU-hours. That came from SLURM -*requested* walltime and had never been measured; the real figure is two orders -of magnitude smaller. `beams2d`'s optimizer is CPU-bound, so **a GPU does not -help the physics at all.** The practical consequence is that teams could compute -part of the expensive board live, which is a different session from the one -where it has to be sealed. - -## Findings that changed the plan - -**The diffusion model's conditioning is wired backwards.** `tools/condition_probe.py` -sweeps the requested volume fraction at fixed sampling noise and regresses -realized on requested: - -| model | slope | r² | reading | -|---|---|---|---| -| `diffusion_2d_cond` | **−0.220** | 0.504 | ignores, and *anti-correlates with*, the request | -| `vqgan` | 0.902 | 0.993 | follows the request | - -Ask for more material, get less. VQGAN goes through identical plumbing at slope -0.90, so this is not a loading bug. The post-NeurIPS diffusion fixes addressed -design-range normalization and the noise schedule; none touched conditioning. -Hypothesis worth testing: conditions reach the UNet as a single length-4 -cross-attention token, **unnormalized**, with `volfrac` ∈ [0.15, 0.45] beside -`rmin` ∈ [1, 3]. Normalizing before `encoder_hid_proj` is the first thing to try, -and it needs a retrain. - -**The checkpoints are the current Euler runs.** Repos created 2026-08-06, still -being written. The new packages carry `design_min`/`design_max`; the legacy -NeurIPS checkpoints have no such keys, which is how you can tell them apart. -Euler's `ops/euler-sweeps` contains every diffusion and evaluation fix commit. - -**Two bugs of my own, both silent.** The three LVAE adapters dropped `problem` -when building, so all 18 PLVAE packages were unloadable. And the pool scorer -appended per-row frames with per-row columns to one CSV, shifting metric columns -for any family whose config keys differed from the first row's — which read as -NaN for exactly `vqgan` and `diffusion_2d_cond`, got them dropped by `dropna`, -and let the selector report six-way disagreement over a board containing only -GANs. Both fixed; the corrupted board was discarded rather than patched. +The `.ipynb` file is generated. Edit `tools/build_notebook.py`, then rebuild it: + +```bash +python workshops/idetc26/tools/build_notebook.py +``` + +The builder assigns stable cell identifiers and produces deterministic JSON. +Commit the source and generated notebook together. + +After changing the notebook or participant API, run: + +```bash +pytest tests/test_idetc26.py -m "not network" +python workshops/idetc26/tools/verify_notebook.py +``` + +The second command executes the complete notebook and requires network access. ## Adding a problem -Write `problems/.json`. Nothing else changes: +Add `engiopt/workshops/idetc26/problems/.json` with the frozen +EngiBench spec, display name, metrics, candidate bank, and controls: ```json { - "problem_id": "photonics2d", - "spec": "photonics2d/v1", - "opening_metrics": ["mmd", "dpp", "viol"], - "withheld_metrics": ["novelty", "cond_err", "pixel_vendi", "pca_mmd", "gen_seconds"], - "expensive_metrics": ["iog", "cog", "fog"], - "bank": [{"kind": "baseline", "algo": "knn_retrieval", "seed": 1}] + "problem_id": "example2d", + "spec": "example2d/v1", + "display_name": "Example2D", + "metrics": ["mmd", "pixel_vendi", "viol", "iog"], + "bank": [ + { + "kind": "pretrained", + "algo": "knn_retrieval", + "seed": 1, + "config_fingerprint": "..." + } + ], + "controls": [ + {"kind": "reference", "algo": "collapsed"}, + {"kind": "reference", "algo": "noise_doped"} + ] } ``` -`unavailable_metrics()` is **derived from the frozen spec**, not declared — so on -photonics2d, whose spec sets `volume_condition` to null, `viol` and the other -condition columns report themselves unavailable with a reason. A team -discovering mid-session that the column their neighbours are arguing about does -not exist for their problem is content, not a bug. - -## Open questions - -- **Bank not yet selected.** The pool rescore is running after the CSV fix; - `tools/search_pool.py select` then ranks candidate banks by distinct rank-1 - winners, mean Kendall τ, and rank spread. The sealed board must be rebuilt - afterwards — the committed one is for a superseded bank. -- **The slow families are under-sampled.** `vqgan` and `diffusion_2d_cond` are - capped at 4 packages each because sampling costs 85–540 s per package, against - 46 apiece for the GANs. A bank chosen from that pool over-represents GANs - unless selection is stratified, which it now is — but the underlying coverage - is still uneven and `select` reports `families` per candidate so it cannot be - missed. -- **kNN and linear regression are not on the Hub yet**, and not in the Euler - manifest. Both train in seconds on CPU and need no GPU arm. -- **Flow matching**: `codex/flow-matching-model` has a complete `core.py`; the - adapter is ~80 lines but there are no checkpoints, so it needs a training run. -- **No test covers `engiopt/baselines/`** since the old adversarial tests were - removed with the models they asserted. -- **`tests/test_idetc26.py` takes 49 minutes** because opening a challenge - downloads checkpoints. It needs the `network` marker the repo already declares. +Every configured metric must be registered in EngiOpt and supported by the +frozen spec. Every candidate must have a resolvable package or construction. +Before using the problem in a workshop, build its design cache, publish or +verify its physics results, and run the notebook verification against it. From 674718581eca0fc82c057df703e1e3d76f8493db Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 17:54:02 +0200 Subject: [PATCH 4/9] docs(workshop): restore rationale and update IDETC guide --- workshops/idetc26/README.md | 431 ++++++++++++++++++++++++------------ 1 file changed, 291 insertions(+), 140 deletions(-) diff --git a/workshops/idetc26/README.md b/workshops/idetc26/README.md index be605c4a..a5324594 100644 --- a/workshops/idetc26/README.md +++ b/workshops/idetc26/README.md @@ -1,191 +1,320 @@ -# IDETC-CIE 2026: Find the Best Model +# IDETC'26 — Find the Best Model -This workshop asks participants to compare generative models for an engineering -design problem. The models are evaluated under the same conditions, but the -metrics do not necessarily agree. Participants inspect the designs, select the -evidence they consider relevant, and explain which model they would use. +A hands-on workshop about the evidence needed to compare generative models for +engineering design. -## Status +Teams work with ten candidate models for one topology-optimization problem: two +baselines, five trained generative models, and three constructed candidates. The +models are named, evaluated under the same conditions, and compared using cost, +distribution, constraint, and physics-based metrics. The resulting rankings +depend on both the selected metric and the representation space. -The participant notebook, ten-model Beams2D bank, controls, cached designs, and -published physics results are implemented. The notebook runs in Google Colab and -uses one participant path from setup to the final discussion. +There is no model that is best independently of the intended use. Participants +must decide which evidence matters for their application and state what remains +uncertain. -The main operational dependency is network access. A cold Colab runtime installs -EngiOpt and EngiBench and downloads the EngiBench dataset and latent encoder. -Testing measured approximately 510 MB of downloads after package installation. +**Status:** the Beams2D participant notebook, candidate bank, controls, cached +designs, and published physics results are implemented and runnable in Colab. +The main operational dependency is a cold-runtime download of approximately +510 MB after package installation. -## Participant notebook +--- -[Open the workshop in Google Colab](https://colab.research.google.com/github/IDEALLab/EngiOpt/blob/codex/idetc26-workshop-participant-rework/workshops/idetc26/notebooks/01_find_the_best_model.ipynb) +## Workshop motivation -The notebook opens read-only from GitHub. Participants should select -**File > Save a copy in Drive** before making changes. +> Habibi et al., **[When Is it Actually Worth Learning Inverse Design?](https://asmedigitalcollection.asme.org/mechanicaldesign/article/148/6/061704/1229197/When-is-it-Actually-Worth-Learning-Inverse-Design)** +> *J. Mech. Des.* 148(6):061704 (2026), first presented at IDETC-CIE 2023. -To run it from a local checkout: +That paper compared inverse-design model families on a topology-optimization +warm-start task and found that **k-nearest neighbours and random forests beat +deconvolutional networks when training data is limited**, once the cost of +generating that data is counted. It is an IDEAL Lab paper, on this task, first +given at this conference. + +The workshop revisits that question with a broader model bank and a larger set +of evaluation criteria. Both sides of the paper's comparison are included as +full candidates: `knn_retrieval` at k=1 and `deconv_regression`. + +On `beams2d`, the kNN baseline outperforms the conditional GAN on condition +adherence at 1/1600th of VQGAN's sampling cost, while kNN and VQGAN are within +5% of each other on MMD. This is why the baselines are ranked alongside the +trained models. + +## Run it + +[Open the participant notebook in Google Colab](https://colab.research.google.com/github/IDEALLab/EngiOpt/blob/codex/idetc26-workshop-participant-rework/workshops/idetc26/notebooks/01_find_the_best_model.ipynb) + +The GitHub copy is read-only. Participants should select **File > Save a copy in +Drive** before editing. ```bash jupyter lab workshops/idetc26/notebooks/01_find_the_best_model.ipynb ``` -### Colab setup +Colab installs EngiOpt from the branch named by the single `BRANCH` constant in +`tools/build_notebook.py` and EngiBench from `ENGIBENCH_REF`. The setup writes a +versioned marker under `/content` after verifying the installation. Repeated +executions skip installation and do not restart the runtime. -The setup cell installs: +The notebook requests a T4. A GPU speeds up sampling after a cache miss; it does +**not** speed up the physics, which is CPU-bound topology optimization. A cold +runtime still downloads the EngiBench dataset and latent encoder from Hugging +Face, so network access is required during setup. -- EngiOpt from the branch defined by `BRANCH` in `tools/build_notebook.py`. -- EngiBench from the Git reference defined by `ENGIBENCH_REF`. +## Notebook structure -The installation is guarded by `/content/.idetc26_setup_v1`. The first run -installs and verifies the required EngiBench problem version. Repeated executions -skip installation, so **Run all** does not trigger a restart loop. A factory reset -clears both the installed packages and the marker. +Ten suspects are evaluated on the same 50 design conditions. The notebook has +six sections: -The notebook requests a T4 runtime. The normal participant path reads cached -designs and published physics, but a GPU prevents a long wait if a design cache -miss requires sampling from a checkpoint. +**1 · the data** → **2 · the suspects** → **3 · the metric families** → +**4 · pixel, PCA, and learned latent spaces** → **5 · a selected metric board** → +**6 · the participant's accusation**. The final recommendation is discussed in +the room rather than recorded by the notebook. -## Workshop sequence +The sequence moves from visual inspection to numerical evaluation and then to +the participant's own comparison. The notebook keeps API explanation brief; +the detailed command reference is available through `case.help()`. -The notebook contains six sections: +**There are two commands.** `case.evaluate` for numbers, `case.show` for +pictures: -1. **The dataset**: inspect the training and held-out test designs as the problem - conditions change. -2. **The suspects**: inspect the ten candidate models and compare generated - designs under the same conditions. -3. **The interrogation**: evaluate cost, similarity, novelty, diversity, - constraint satisfaction, and optimization performance. -4. **Representation spaces**: compare distance-based measurements in pixels, a - matched PCA subspace, and a learned latent space. -5. **The board**: assemble a ranking from selected metrics. -6. **The accusation**: choose a model, explain the evidence, and identify what - else should be checked before using it in practice. +```python +from engiopt.workshops.idetc26 import Case -The murder-mystery terminology is the workshop theme. The model names and metric -values are not anonymized. +case = Case.open("beams2d") -## Model bank +case.help() # the cheat sheet: five calls, then every knob +case.models() # who is in the line-up +case.metrics() # what may be asked, by line of questioning -The Beams2D bank contains ten candidates evaluated on the same frozen EngiBench -specification: +case.evaluate("diversity") # all diversity metrics +case.evaluate("mmd", models="knn") # one question, one suspect +answers = case.evaluate(["mmd", "novelty_ratio", "pixel_vendi"], controls=True) -- k-nearest-neighbour retrieval -- supervised deconvolutional regression -- VQGAN -- conditional diffusion -- conditional GAN -- unconditional GAN -- constrained least-volume autoencoder -- three constructed candidates designed to expose weaknesses in individual - evaluation criteria +case.show("diffusion") # its designs +case.show("knn", "diffusion") # two suspects on the same brief +case.show(answers) # a table of answers, drawn as ranks +case.show(answers, case.physics()) # selected metrics with published physics +case.show("knn", how="nearest_training") # each design beside its closest training design +case.show("vqgan", "test", how="space_map") # where its designs sit in a fitted space +``` -The constructed candidates participate in the ranking. Their construction is -not printed by the participant-facing API. The facilitator must disclose them -after participants present their recommendations; there is no passphrase or -automated reveal in the notebook. +**Neither command has a blanket form.** Calling `case.evaluate()` or +`case.show()` without specifying what to evaluate or display raises an error. +This requires participants to choose the metrics and models relevant to their +argument instead of generating a complete board by default. The errors include +an example of a valid call. + +**`case.help()` is the command reference.** It starts with the five calls used in +the notebook — `models`, `metrics`, `explain`, `evaluate`, and `show` — followed +by the available views, controls, and optional arguments. Keeping this reference +in the API prevents notebook instructions from drifting when an argument +changes. + +An earlier API exposed separate methods for computing, scoring, ranking, and +running physics. The current interface consolidates numerical evaluation under +`case.evaluate()` and expresses the differences as arguments. + +The short form, `evaluate("mmd")`, uses the frozen evaluation specification. +Optional arguments such as `sigma`, `n_samples`, and `random_conditions` allow +participants to test how evaluation choices affect the result. + +**Suspects are named by model family** from the first cell — `knn_retrieval`, +`diffusion_2d_cond`, and `cgan_cnn_2d`, for example — and can be addressed by an +unambiguous fragment such as `"diffusion"` or `"knn"`. Naming the models allows +participants to connect metric behavior with architecture and training method. + +**No two suspects differ only by training seed.** The bank compares model +families and configurations rather than treating repeated training runs as +separate candidates. + +There is no `seeds=` argument in the participant API. The specification freezes +the 50 design conditions, and changing the sampling seed on one checkpoint does +not measure variation across training runs. A training-seed study requires +separate checkpoints; those are outside the current participant workflow. + +### Metric groups + +Metrics are grouped by the question they answer. Participants can evaluate a +whole group or select individual metrics: + +| Group | Question | Limitation or failure mode | +|---|---|---| +| `cost` | What did training and generation require? | Does not measure design quality | +| `similarity` | Do generated designs resemble held-out designs? | Returning training designs can score well | +| `memorization` | Are generated designs distinct from the training set? | Random noise can appear novel | +| `diversity` | How much does the generated set vary? | Corruption can increase diversity scores | +| `obedience` | Do designs satisfy the requested conditions? | Meeting a volume budget does not imply structural quality | +| `performance` | How do designs perform under optimization? | Requires the simulator | + +The seventh group, `latent_space`, contains `lv_dual_gap`, `lv_active_dims`, and +`pca_dims`. These values describe the fitted representation rather than a +candidate model, so they are not ranked. `case.latent_space()` identifies the +autoencoder, its active width, and the matched PCA width. + +Each metric group has a known failure mode, so no group is sufficient on its +own. + +These workshop groups differ from the metric registry's implementation +families. The registry places `novelty` and `mmd` under `distribution`, but they +answer different questions: copying the training set can improve MMD, while +novelty is intended to detect that behavior. The workshop therefore separates +memorization. Similarly, each `lv_*` metric is grouped by the question it +answers, while its learned-latent dependency appears in the `space` column: + +| the question | pixels | PCA subspace | learned latent | +|---|---|---|---| +| does this look like the real data? | `mmd` | `pca_mmd` | `lv_mmd` | +| is it copying the training set? | `novelty_ratio` | — | `lv_novelty` | +| did it cover the real modes? | — | `pca_coverage` | `lv_coverage` | +| how many distinct designs? | `pixel_vendi`, `dpp_geometric` | `pca_vendi` | `lv_vendi` | +| did it answer the brief? | `pixel_paired_distance` | — | `lv_paired_distance` | + +Differences between these columns can come from the representation space rather +than the candidate model. + +`case.show()` accepts a model name, two model names, a metric board, or selected +columns from a board. The `how=` argument provides the `designs`, `compare`, +`conditions`, `nearest_training`, and `space_map` views. + +`space_map` displays a candidate's 50 designs in the pinned latent space or the +matched PCA subspace, with the training set as a reference. It exposes collapse, +poor coverage, and isolated samples that are difficult to interpret from a +single MMD value. Comparing `space="lv"` with `space="pca"` shows how the chosen +representation affects a distribution metric. + +`case.evaluate()` uses the same evaluator, frozen specification, and argument +names as `python -m engiopt.evaluate`. It prints the equivalent CLI command so +results can be reproduced outside the notebook. + +Physics-based metrics use the same evaluation interface as the cheap metrics. +The published board, available through `case.physics()`, avoids running the +simulator over all 50 designs during the session. A live request prints its +estimated cost before simulation begins. + +## Cached designs and network requirements + +Candidate designs for the specification's 50 conditions are precomputed and +shipped under `engiopt/workshops/idetc26/cache/`. The normal participant path +therefore does not sample checkpoints or download model weights. -Three controls provide reference values and are never ranked with the candidate -models: +```bash +# rebuild after the bank changes -- a cluster/workstation job, not a laptop one +python workshops/idetc26/tools/build_design_cache.py --problem-id beams2d --seeds 1 2 3 +``` -- `collapsed`: one averaged design repeated for every condition -- `noise_doped`: held-out optimized designs with added Gaussian noise -- `volume_only`: random designs satisfying only the volume constraint +`case.evaluate(fresh=True)` explicitly resamples checkpoints and records the +runtime. Replayed `gen_seconds` values retain the machine on which they were +measured because wall-clock comparisons are only meaningful on comparable +hardware. -## Evaluation interface +A cached candidate does not download its checkpoint at `Case.open()`, avoiding a +1.8 GB VQGAN download in the participant path. A cold runtime still downloads +the EngiBench dataset and latent encoder from Hugging Face. The measured transfer +was approximately 510 MB after package installation, so the workshop requires +reliable network access during setup. -Participants work with one `Case` object: +## Repository structure -```python -from engiopt.workshops.idetc26 import Case - -case = Case.open("beams2d") -case.models() -case.metrics() -case.explain("diffusion") -case.show("diffusion") -case.evaluate("mmd") +``` +workshops/idetc26/ + notebooks/01_find_the_best_model.ipynb the session (a build product) + tools/build_notebook.py notebook SOURCE -- edit here, not the .ipynb + tools/build_design_cache.py pre-sample the bank so the session never waits + tools/search_pool.py score the pool, then search it for a bank + tools/condition_probe.py does a model respond to its condition at all? + tools/verify_notebook.py execute every cell, fail on the first error + +engiopt/workshops/idetc26/ participant API, bank, config, and views + problems/beams2d.json per-problem configuration + cache/ packaged, pre-sampled designs +engiopt/baselines/ dataset-fitted models (see below) +engiopt/generators/knn_retrieval/ kNN as a first-class generator package +engiopt/generators/linear_regression/ ridge regression, likewise +tests/test_idetc26.py harness verification ``` -`case.evaluate()` returns numerical results. `case.show()` handles design -galleries, comparisons, condition checks, nearest-training examples, -representation-space maps, and metric boards. `case.help()` prints the full -participant reference. +Colab installs the `engiopt` package rather than the repository's `workshops/` +tree. Problem configurations and design caches therefore live inside the package +and are included through `[tool.setuptools.package-data]`. A source checkout is +also supported; the package location is checked first. -The workshop groups metrics by the question they answer: +## The model bank and controls -| Group | Question | -|---|---| -| `cost` | What did training and generation require? | -| `similarity` | How closely does the generated set match held-out designs? | -| `memorization` | Are generated designs distinct from the training set? | -| `diversity` | How much does the generated set vary? | -| `obedience` | Do designs satisfy the requested conditions? | -| `performance` | How do designs perform under physics-based optimization? | -| `latent_space` | Is the fitted representation suitable for measurement? | +The Beams2D bank contains ten ranked candidates. Seven are pretrained entries: +`knn_retrieval`, `deconv_regression`, `vqgan`, `diffusion_2d_cond`, +`cgan_cnn_2d`, `gan_cnn_2d`, and `constrained_plvae_2d`. The other three are +constructed candidates designed to expose weaknesses in individual evaluation +criteria. -The `latent_space` group contains diagnostics for the representation and is not -used to rank models. Expensive performance results are read from published -physics records when available. Passing `fresh=True` explicitly resamples a -model or recomputes results on the current machine. +The participant-facing API describes what every model does but does not disclose +which candidates were constructed for the workshop. The facilitator must make +that disclosure after participants present their recommendations. The notebook +does not contain a passphrase or an automated reveal. -## Data and runtime behavior +The bank also provides three controls: `collapsed`, `noise_doped`, and +`volume_only`. Controls provide reference values for interpreting metrics and +are never included in candidate rankings. -Pre-sampled candidate designs are packaged under -`engiopt/workshops/idetc26/cache/`. The normal participant path therefore does -not download candidate checkpoints or run model sampling. +## Published baselines -`Case.open()` still resolves the frozen EngiBench dataset and the latent-space -encoder. These assets are downloaded from Hugging Face on a cold runtime and -then cached by Colab. Internet access is therefore required during setup. +`knn_retrieval` and `deconv_regression` are full generator packages with +training scripts that write checkpoints through `save_checkpoint_package`. They +are not notebook-specific implementations. `linear_regression` remains available +in the repository but is not part of the workshop bank; the deconvolutional +network is used because it is the comparator from the cited paper. -Published physics values are also read from the Hub. The notebook evaluates only -a small subset live; a complete simulator sweep is not expected during the -session. +| | `knn_retrieval` | `deconv_regression` | `diffusion_2d_cond` | `vqgan` | +|---|---|---|---|---| +| checkpoint | 38.9 MB | **14.8 MB** | ~70 MB | ~1.8 GB | +| parameters | its training set | 3.7 M | 17.5 M | 102 M | +| 50 samples | 0.054 s | one forward pass | 452 s | 85 s | -## Repository layout +Each package reports its parameter count, allowing the stored training data used +by kNN and the learned weights used by neural models to be compared on the same +cost axis. -```text -workshops/idetc26/ - notebooks/01_find_the_best_model.ipynb generated participant notebook - tools/build_notebook.py canonical notebook source - tools/build_design_cache.py rebuild pre-sampled designs - tools/search_pool.py inspect and select candidate banks - tools/verify_notebook.py execute the notebook end to end - -engiopt/workshops/idetc26/ - case.py participant API - bank.py candidate and control loading - config.py per-problem configuration - designs.py cached design access - families.py participant metric groups - views.py notebook visualizations - problems/ workshop problem configurations - cache/ packaged generated designs -``` +## Measured costs -## Editing the notebook +Development measurements on a laptop CPU gave the following ranges. The full +board used for these measurements contained eight candidates; the current bank +contains ten. -The `.ipynb` file is generated. Edit `tools/build_notebook.py`, then rebuild it: +| | cost | +|---|---| +| the cheap board, 8 models × 50 samples | seconds | +| the physics board, per model × 50 samples | ~90–125 s | +| a whole 8-model physics board | **~15 min** | -```bash -python workshops/idetc26/tools/build_notebook.py -``` +`beams2d` optimization is CPU-bound, so a GPU does not reduce the physics cost. +Participants can evaluate a small subset live, while the complete physics board +is read from published results. -The builder assigns stable cell identifiers and produces deterministic JSON. -Commit the source and generated notebook together. +## Known model behavior -After changing the notebook or participant API, run: +`tools/condition_probe.py` sweeps the requested volume fraction at fixed sampling +noise and regresses the realized fraction on the request. The current diffusion +checkpoint has a negative conditioning response: -```bash -pytest tests/test_idetc26.py -m "not network" -python workshops/idetc26/tools/verify_notebook.py -``` +| model | slope | r² | reading | +|---|---|---|---| +| `diffusion_2d_cond` | **−0.220** | 0.504 | ignores, and *anti-correlates with*, the request | +| `vqgan` | 0.902 | 0.993 | follows the request | -The second command executes the complete notebook and requires network access. +VQGAN follows the requested volume fraction through the same evaluation path, +which argues against a loading or measurement error. A current hypothesis is +that the UNet receives an unnormalized length-four conditioning token in which +`volfrac` ranges from 0.15 to 0.45 while `rmin` ranges from 1 to 3. Testing that +hypothesis requires retraining the diffusion model. + +Current checkpoint packages include `design_min` and `design_max`. Legacy +NeurIPS checkpoints do not, which provides a direct way to distinguish the two +formats. ## Adding a problem Add `engiopt/workshops/idetc26/problems/.json` with the frozen -EngiBench spec, display name, metrics, candidate bank, and controls: +EngiBench specification, display name, metric list, model bank, and controls: ```json { @@ -208,7 +337,29 @@ EngiBench spec, display name, metrics, candidate bank, and controls: } ``` -Every configured metric must be registered in EngiOpt and supported by the -frozen spec. Every candidate must have a resolvable package or construction. -Before using the problem in a workshop, build its design cache, publish or -verify its physics results, and run the notebook verification against it. +`unavailable_metrics()` derives unsupported metrics from the frozen spec. For +example, `photonics2d` has no volume-fraction budget, so `viol` and related +condition metrics report themselves unavailable with a reason. + +Every candidate must have a resolvable checkpoint package or construction. +Before using a new problem in a workshop, build its design cache, publish or +verify its physics results, and execute the notebook against the new +configuration. + +## Editing and verification + +The notebook is generated from `tools/build_notebook.py`. Edit the builder and +commit the source and generated notebook together: + +```bash +python workshops/idetc26/tools/build_notebook.py +``` + +After changing the notebook or participant API, run: + +```bash +pytest tests/test_idetc26.py -m "not network" +python workshops/idetc26/tools/verify_notebook.py +``` + +The notebook verification executes every code cell and requires network access. From 968db9c5b8a21c5b8c2ba54133b860d650a046b2 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 17:58:04 +0200 Subject: [PATCH 5/9] chore(workshop): remove obsolete IDETC plan --- workshops/idetc26/.gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/workshops/idetc26/.gitignore b/workshops/idetc26/.gitignore index 48854efa..a08a5c46 100644 --- a/workshops/idetc26/.gitignore +++ b/workshops/idetc26/.gitignore @@ -4,3 +4,6 @@ cache/ # Scratch dir a notebook run creates next to itself; never shipped. notebooks/idetc26_work/ + +# Local planning notes are not part of the workshop deliverable. +PLAN.md From 2c5a532bdd4ae904f8a93a5bc3592fd29c425945 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 18:15:50 +0200 Subject: [PATCH 6/9] feat(workshop): add planted model reveal --- engiopt/workshops/idetc26/case.py | 72 +++++++++++++++++++ tests/test_idetc26.py | 65 +++++++++++++++++ .../notebooks/01_find_the_best_model.ipynb | 42 ++++++++--- workshops/idetc26/tools/build_notebook.py | 33 ++++++--- 4 files changed, 194 insertions(+), 18 deletions(-) diff --git a/engiopt/workshops/idetc26/case.py b/engiopt/workshops/idetc26/case.py index 9910a494..c0067041 100644 --- a/engiopt/workshops/idetc26/case.py +++ b/engiopt/workshops/idetc26/case.py @@ -62,6 +62,8 @@ from dataclasses import dataclass from dataclasses import field +import hashlib +import hmac import io from pathlib import Path import textwrap @@ -84,6 +86,26 @@ if TYPE_CHECKING: from engiopt.workshops.idetc26.views import Views +_REVEAL_GATE = "ec6ecbfbd552966130e43013f3c01f9b86e37ed8e9a452dbb7c880fff01b939f" + +_PLANTED_REVEAL = { + "coarse_to_fine_2d": { + "does": "Retrieves a training design, coarsens it, then upsamples it back to full size.", + "tests": "Can constraint checks miss a visible checkerboard artifact?", + "check": "Look at its designs, then compare its constraint and similarity scores.", + }, + "portfolio_2d": { + "does": "Returns a varied set of training designs but pays little attention to which brief each one answers.", + "tests": "Can diversity look good when the model ignores part of the brief?", + "check": "Compare its diversity scores with paired distance and physics results.", + }, + "annealed_2d": { + "does": "Retrieves a training design, adds noise, then adjusts it to meet the volume budget.", + "tests": "Can noise be rewarded as novelty and diversity?", + "check": "Compare its diversity and feasibility scores with latent distance and coverage.", + }, +} + HELP = """ THE CASE FILE -- cheat sheet. First block is all you need; the rest is reference. @@ -235,6 +257,7 @@ class Case: _sample_meta: dict[tuple[str, int], dict[str, Any]] = field(default_factory=dict, repr=False) _published: dict[str, dict[str, Any]] = field(default_factory=dict, repr=False) _views: Views | None = field(default=None, repr=False) + _verdict: dict[str, str] | None = field(default=None, repr=False) # ------------------------------------------------------------------ # Opening the file @@ -886,6 +909,55 @@ def rank(self, frame: pd.DataFrame) -> pd.DataFrame: out.index.name = frame.index.name return out + def commit_verdict(self, model: str, evidence: str, uncertainty: str = "") -> pd.Series: + """Record a participant's choice before the reveal.""" + member = self.bank.resolve(model) + evidence = evidence.strip() + uncertainty = uncertainty.strip() + if not evidence: + raise ValueError("Add one result or observation that supports your choice.") + + self._verdict = { + "model": member.label, + "evidence": evidence, + "uncertainty": uncertainty or "Not stated", + } + print("Verdict recorded. Wait for the facilitator before opening the reveal.") + return pd.Series(self._verdict, name="your verdict") + + def reveal(self, passphrase: str) -> None: + """Show the planted models after a participant has recorded a verdict.""" + if self._verdict is None: + raise RuntimeError("Record your choice with case.commit_verdict(...) before opening the reveal.") + + supplied = hashlib.sha256(passphrase.strip().casefold().encode()).hexdigest() + if not hmac.compare_digest(supplied, _REVEAL_GATE): + raise ValueError("That passphrase does not open the reveal.") + + planted = [member for member in self.bank if member.kind == "planted"] + print("\nTHE REVEAL\n") + print(f"{len(planted)} suspects were planted in the line-up.") + print("Their scores are real. Each one was built to expose a gap in the evaluation.\n") + + planted_labels = set() + for member in planted: + planted_labels.add(member.label) + details = _PLANTED_REVEAL.get(member.identity) + if details is None: + print(f"{member.label}\n {member.summary}\n") + continue + print(member.label) + print(f" What it does: {details['does']}") + print(f" What it tests: {details['tests']}") + print(f" What to check: {details['check']}\n") + + if self._verdict["model"] in planted_labels: + print("Your choice was one of the planted models.") + print("The scores you used were real. Now you can see what those scores left out.") + else: + print("Your choice was not planted.") + print("Now check whether the same evidence would also have supported one of the planted models.") + # ------------------------------------------------------------------ # Internals # ------------------------------------------------------------------ diff --git a/tests/test_idetc26.py b/tests/test_idetc26.py index 8c2f43e5..6d864a7a 100644 --- a/tests/test_idetc26.py +++ b/tests/test_idetc26.py @@ -10,6 +10,7 @@ import os from pathlib import Path +from types import SimpleNamespace import numpy as np import pytest @@ -109,6 +110,70 @@ def test_no_two_suspects_differ_only_by_training_seed() -> None: assert len(identities) == len(set(identities)), f"{problem} has two suspects differing only by seed" +def _case_for_reveal(tmp_path: Path) -> Case: + """A small model bank for testing the reveal without loading any data.""" + members = [ + SimpleNamespace( + label="diffusion_2d_cond", + identity="diffusion_2d_cond", + kind="pretrained", + summary="Conditional diffusion.", + ), + *[ + SimpleNamespace(label=name, identity=name, kind="planted", summary="A planted model.") + for name in ("coarse_to_fine_2d", "portfolio_2d", "annealed_2d") + ], + ] + + class RevealBank(list): + def resolve(self, name: str) -> SimpleNamespace: + matches = [member for member in self if name in member.label] + if len(matches) != 1: + raise KeyError(name) + return matches[0] + + return Case( + config=WorkshopConfig.load(PROBLEM_ID), + evaluator=None, # type: ignore[arg-type] + bank=RevealBank(members), # type: ignore[arg-type] + controls=None, # type: ignore[arg-type] + artifact_dir=tmp_path, + store=None, # type: ignore[arg-type] + ) + + +def test_reveal_requires_a_written_verdict_and_the_facilitator_phrase( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """The reveal stays closed until the participant commits and the facilitator opens it.""" + import engiopt.workshops.idetc26.case as case_module + + reveal_case = _case_for_reveal(tmp_path) + with pytest.raises(RuntimeError, match="Record your choice"): + reveal_case.reveal("anything") + with pytest.raises(ValueError, match="Add one result"): + reveal_case.commit_verdict("portfolio", "") + + reveal_case.commit_verdict("portfolio", "It ranked highly on diversity.", "I did not run the simulator.") + with pytest.raises(ValueError, match="does not open"): + reveal_case.reveal("wrong") + + test_phrase = "open the test reveal" + monkeypatch.setattr( + case_module, + "_REVEAL_GATE", + case_module.hashlib.sha256(test_phrase.casefold().encode()).hexdigest(), + ) + reveal_case.reveal(test_phrase) + output = capsys.readouterr().out + + assert "3 suspects were planted" in output + assert "coarse_to_fine_2d" in output + assert "portfolio_2d" in output + assert "annealed_2d" in output + assert "Your choice was one of the planted models" in output + + def test_a_suspect_that_is_not_a_seed_variant_is_not_named_like_one() -> None: """`cgan_cnn_2d#1` beside `cgan_cnn_2d#42` claims a seed difference that is not there. diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index 8d7b1c9a..5a3469bd 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -533,30 +533,52 @@ }, { "cell_type": "markdown", - "id": "f23ba99f", + "id": "4a884993", "metadata": {}, "source": [ "---\n", "# 6 · Your accusation\n", "\n", - "Choose the model you would use and be ready to explain your choice to the room:\n", - "\n", - "1. **Who did it.** The model you think is the best.\n", - "2. **On what evidence.** Why do you think so?\n", - "3. **What you could not rule out.** What additional evidence would make your decision easier?\n", + "Choose the model you would use. Write down the main result behind your choice\n", + "and one thing you are still unsure about." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "241cfb37", + "metadata": {}, + "outputs": [], + "source": [ + "model = \"diffusion_2d_cond\" # @param [\"knn_retrieval\", \"deconv_regression\", \"vqgan\", \"coarse_to_fine_2d\", \"diffusion_2d_cond\", \"cgan_cnn_2d\", \"portfolio_2d\", \"gan_cnn_2d\", \"annealed_2d\", \"constrained_plvae_2d\"]\n", + "evidence = \"\" # @param {type:\"string\"}\n", + "uncertainty = \"\" # @param {type:\"string\"}\n", "\n", - "The cell below is yours to build with. Get to work.\n", + "case.commit_verdict(model, evidence, uncertainty)" + ] + }, + { + "cell_type": "markdown", + "id": "c50d93e0", + "metadata": {}, + "source": [ + "---\n", + "# 7 · The reveal\n", "\n", - "Remember `case.help()`" + "Wait for the facilitator. Once everyone has recorded a verdict, enter the\n", + "passphrase and run the cell." ] }, { "cell_type": "code", "execution_count": null, - "id": "1271cf25", + "id": "3bfcfcf8", "metadata": {}, "outputs": [], - "source": [] + "source": [ + "passphrase = \"\" # @param {type:\"string\"}\n", + "case.reveal(passphrase)" + ] } ], "metadata": { diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index 76d8c3bf..7c2a3fc2 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -334,18 +334,35 @@ def code(text: str) -> dict: --- # 6 · Your accusation -Choose the model you would use and be ready to explain your choice to the room: - -1. **Who did it.** The model you think is the best. -2. **On what evidence.** Why do you think so? -3. **What you could not rule out.** What additional evidence would make your decision easier? +Choose the model you would use. Write down the main result behind your choice +and one thing you are still unsure about. +""" + ), + code( + """ +model = "diffusion_2d_cond" # @param ["knn_retrieval", "deconv_regression", "vqgan", "coarse_to_fine_2d", "diffusion_2d_cond", "cgan_cnn_2d", "portfolio_2d", "gan_cnn_2d", "annealed_2d", "constrained_plvae_2d"] +evidence = "" # @param {type:"string"} +uncertainty = "" # @param {type:"string"} -The cell below is yours to build with. Get to work. +case.commit_verdict(model, evidence, uncertainty) +""" + ), + # ---- 7 · the reveal ---- + md( + """ +--- +# 7 · The reveal -Remember `case.help()` +Wait for the facilitator. Once everyone has recorded a verdict, enter the +passphrase and run the cell. +""" + ), + code( + """ +passphrase = "" # @param {type:"string"} +case.reveal(passphrase) """ ), - code(""""""), ] From fed598f77873e45229823e63dd1b039f1a58ea61 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 18:25:21 +0200 Subject: [PATCH 7/9] fix(workshop): preserve accusation wording --- .../notebooks/01_find_the_best_model.ipynb | 19 +++++++++++++------ workshops/idetc26/tools/build_notebook.py | 15 +++++++++++---- 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index 5a3469bd..b4a192e3 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -533,14 +533,21 @@ }, { "cell_type": "markdown", - "id": "4a884993", + "id": "f23ba99f", "metadata": {}, "source": [ "---\n", "# 6 · Your accusation\n", "\n", - "Choose the model you would use. Write down the main result behind your choice\n", - "and one thing you are still unsure about." + "Choose the model you would use and be ready to explain your choice to the room:\n", + "\n", + "1. **Who did it.** The model you think is the best.\n", + "2. **On what evidence.** Why do you think so?\n", + "3. **What you could not rule out.** What additional evidence would make your decision easier?\n", + "\n", + "The cell below is yours to build with. Get to work.\n", + "\n", + "Remember `case.help()`" ] }, { @@ -559,14 +566,14 @@ }, { "cell_type": "markdown", - "id": "c50d93e0", + "id": "1b40727e", "metadata": {}, "source": [ "---\n", "# 7 · The reveal\n", "\n", - "Wait for the facilitator. Once everyone has recorded a verdict, enter the\n", - "passphrase and run the cell." + "Do not run this cell yet. Once everyone has recorded a verdict, we will give\n", + "you the passphrase." ] }, { diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index 7c2a3fc2..1c3f4299 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -334,8 +334,15 @@ def code(text: str) -> dict: --- # 6 · Your accusation -Choose the model you would use. Write down the main result behind your choice -and one thing you are still unsure about. +Choose the model you would use and be ready to explain your choice to the room: + +1. **Who did it.** The model you think is the best. +2. **On what evidence.** Why do you think so? +3. **What you could not rule out.** What additional evidence would make your decision easier? + +The cell below is yours to build with. Get to work. + +Remember `case.help()` """ ), code( @@ -353,8 +360,8 @@ def code(text: str) -> dict: --- # 7 · The reveal -Wait for the facilitator. Once everyone has recorded a verdict, enter the -passphrase and run the cell. +Do not run this cell yet. Once everyone has recorded a verdict, we will give +you the passphrase. """ ), code( From 0da3f62cee5253775e399abe09300f2eb0fb8358 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 18:48:39 +0200 Subject: [PATCH 8/9] revert(workshop): remove verdict reveal --- engiopt/workshops/idetc26/case.py | 72 ------------------- tests/test_idetc26.py | 65 ----------------- .../notebooks/01_find_the_best_model.ipynb | 37 ++-------- workshops/idetc26/tools/build_notebook.py | 28 +------- 4 files changed, 6 insertions(+), 196 deletions(-) diff --git a/engiopt/workshops/idetc26/case.py b/engiopt/workshops/idetc26/case.py index c0067041..9910a494 100644 --- a/engiopt/workshops/idetc26/case.py +++ b/engiopt/workshops/idetc26/case.py @@ -62,8 +62,6 @@ from dataclasses import dataclass from dataclasses import field -import hashlib -import hmac import io from pathlib import Path import textwrap @@ -86,26 +84,6 @@ if TYPE_CHECKING: from engiopt.workshops.idetc26.views import Views -_REVEAL_GATE = "ec6ecbfbd552966130e43013f3c01f9b86e37ed8e9a452dbb7c880fff01b939f" - -_PLANTED_REVEAL = { - "coarse_to_fine_2d": { - "does": "Retrieves a training design, coarsens it, then upsamples it back to full size.", - "tests": "Can constraint checks miss a visible checkerboard artifact?", - "check": "Look at its designs, then compare its constraint and similarity scores.", - }, - "portfolio_2d": { - "does": "Returns a varied set of training designs but pays little attention to which brief each one answers.", - "tests": "Can diversity look good when the model ignores part of the brief?", - "check": "Compare its diversity scores with paired distance and physics results.", - }, - "annealed_2d": { - "does": "Retrieves a training design, adds noise, then adjusts it to meet the volume budget.", - "tests": "Can noise be rewarded as novelty and diversity?", - "check": "Compare its diversity and feasibility scores with latent distance and coverage.", - }, -} - HELP = """ THE CASE FILE -- cheat sheet. First block is all you need; the rest is reference. @@ -257,7 +235,6 @@ class Case: _sample_meta: dict[tuple[str, int], dict[str, Any]] = field(default_factory=dict, repr=False) _published: dict[str, dict[str, Any]] = field(default_factory=dict, repr=False) _views: Views | None = field(default=None, repr=False) - _verdict: dict[str, str] | None = field(default=None, repr=False) # ------------------------------------------------------------------ # Opening the file @@ -909,55 +886,6 @@ def rank(self, frame: pd.DataFrame) -> pd.DataFrame: out.index.name = frame.index.name return out - def commit_verdict(self, model: str, evidence: str, uncertainty: str = "") -> pd.Series: - """Record a participant's choice before the reveal.""" - member = self.bank.resolve(model) - evidence = evidence.strip() - uncertainty = uncertainty.strip() - if not evidence: - raise ValueError("Add one result or observation that supports your choice.") - - self._verdict = { - "model": member.label, - "evidence": evidence, - "uncertainty": uncertainty or "Not stated", - } - print("Verdict recorded. Wait for the facilitator before opening the reveal.") - return pd.Series(self._verdict, name="your verdict") - - def reveal(self, passphrase: str) -> None: - """Show the planted models after a participant has recorded a verdict.""" - if self._verdict is None: - raise RuntimeError("Record your choice with case.commit_verdict(...) before opening the reveal.") - - supplied = hashlib.sha256(passphrase.strip().casefold().encode()).hexdigest() - if not hmac.compare_digest(supplied, _REVEAL_GATE): - raise ValueError("That passphrase does not open the reveal.") - - planted = [member for member in self.bank if member.kind == "planted"] - print("\nTHE REVEAL\n") - print(f"{len(planted)} suspects were planted in the line-up.") - print("Their scores are real. Each one was built to expose a gap in the evaluation.\n") - - planted_labels = set() - for member in planted: - planted_labels.add(member.label) - details = _PLANTED_REVEAL.get(member.identity) - if details is None: - print(f"{member.label}\n {member.summary}\n") - continue - print(member.label) - print(f" What it does: {details['does']}") - print(f" What it tests: {details['tests']}") - print(f" What to check: {details['check']}\n") - - if self._verdict["model"] in planted_labels: - print("Your choice was one of the planted models.") - print("The scores you used were real. Now you can see what those scores left out.") - else: - print("Your choice was not planted.") - print("Now check whether the same evidence would also have supported one of the planted models.") - # ------------------------------------------------------------------ # Internals # ------------------------------------------------------------------ diff --git a/tests/test_idetc26.py b/tests/test_idetc26.py index 6d864a7a..8c2f43e5 100644 --- a/tests/test_idetc26.py +++ b/tests/test_idetc26.py @@ -10,7 +10,6 @@ import os from pathlib import Path -from types import SimpleNamespace import numpy as np import pytest @@ -110,70 +109,6 @@ def test_no_two_suspects_differ_only_by_training_seed() -> None: assert len(identities) == len(set(identities)), f"{problem} has two suspects differing only by seed" -def _case_for_reveal(tmp_path: Path) -> Case: - """A small model bank for testing the reveal without loading any data.""" - members = [ - SimpleNamespace( - label="diffusion_2d_cond", - identity="diffusion_2d_cond", - kind="pretrained", - summary="Conditional diffusion.", - ), - *[ - SimpleNamespace(label=name, identity=name, kind="planted", summary="A planted model.") - for name in ("coarse_to_fine_2d", "portfolio_2d", "annealed_2d") - ], - ] - - class RevealBank(list): - def resolve(self, name: str) -> SimpleNamespace: - matches = [member for member in self if name in member.label] - if len(matches) != 1: - raise KeyError(name) - return matches[0] - - return Case( - config=WorkshopConfig.load(PROBLEM_ID), - evaluator=None, # type: ignore[arg-type] - bank=RevealBank(members), # type: ignore[arg-type] - controls=None, # type: ignore[arg-type] - artifact_dir=tmp_path, - store=None, # type: ignore[arg-type] - ) - - -def test_reveal_requires_a_written_verdict_and_the_facilitator_phrase( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] -) -> None: - """The reveal stays closed until the participant commits and the facilitator opens it.""" - import engiopt.workshops.idetc26.case as case_module - - reveal_case = _case_for_reveal(tmp_path) - with pytest.raises(RuntimeError, match="Record your choice"): - reveal_case.reveal("anything") - with pytest.raises(ValueError, match="Add one result"): - reveal_case.commit_verdict("portfolio", "") - - reveal_case.commit_verdict("portfolio", "It ranked highly on diversity.", "I did not run the simulator.") - with pytest.raises(ValueError, match="does not open"): - reveal_case.reveal("wrong") - - test_phrase = "open the test reveal" - monkeypatch.setattr( - case_module, - "_REVEAL_GATE", - case_module.hashlib.sha256(test_phrase.casefold().encode()).hexdigest(), - ) - reveal_case.reveal(test_phrase) - output = capsys.readouterr().out - - assert "3 suspects were planted" in output - assert "coarse_to_fine_2d" in output - assert "portfolio_2d" in output - assert "annealed_2d" in output - assert "Your choice was one of the planted models" in output - - def test_a_suspect_that_is_not_a_seed_variant_is_not_named_like_one() -> None: """`cgan_cnn_2d#1` beside `cgan_cnn_2d#42` claims a seed difference that is not there. diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index b4a192e3..47fb1288 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -533,7 +533,7 @@ }, { "cell_type": "markdown", - "id": "f23ba99f", + "id": "dff6aab5", "metadata": {}, "source": [ "---\n", @@ -545,7 +545,7 @@ "2. **On what evidence.** Why do you think so?\n", "3. **What you could not rule out.** What additional evidence would make your decision easier?\n", "\n", - "The cell below is yours to build with. Get to work.\n", + "The cell below is yours to build with.\n", "\n", "Remember `case.help()`" ] @@ -553,39 +553,10 @@ { "cell_type": "code", "execution_count": null, - "id": "241cfb37", + "id": "1271cf25", "metadata": {}, "outputs": [], - "source": [ - "model = \"diffusion_2d_cond\" # @param [\"knn_retrieval\", \"deconv_regression\", \"vqgan\", \"coarse_to_fine_2d\", \"diffusion_2d_cond\", \"cgan_cnn_2d\", \"portfolio_2d\", \"gan_cnn_2d\", \"annealed_2d\", \"constrained_plvae_2d\"]\n", - "evidence = \"\" # @param {type:\"string\"}\n", - "uncertainty = \"\" # @param {type:\"string\"}\n", - "\n", - "case.commit_verdict(model, evidence, uncertainty)" - ] - }, - { - "cell_type": "markdown", - "id": "1b40727e", - "metadata": {}, - "source": [ - "---\n", - "# 7 · The reveal\n", - "\n", - "Do not run this cell yet. Once everyone has recorded a verdict, we will give\n", - "you the passphrase." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3bfcfcf8", - "metadata": {}, - "outputs": [], - "source": [ - "passphrase = \"\" # @param {type:\"string\"}\n", - "case.reveal(passphrase)" - ] + "source": [] } ], "metadata": { diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index 1c3f4299..f15d3d12 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -340,36 +340,12 @@ def code(text: str) -> dict: 2. **On what evidence.** Why do you think so? 3. **What you could not rule out.** What additional evidence would make your decision easier? -The cell below is yours to build with. Get to work. +The cell below is yours to build with. Remember `case.help()` """ ), - code( - """ -model = "diffusion_2d_cond" # @param ["knn_retrieval", "deconv_regression", "vqgan", "coarse_to_fine_2d", "diffusion_2d_cond", "cgan_cnn_2d", "portfolio_2d", "gan_cnn_2d", "annealed_2d", "constrained_plvae_2d"] -evidence = "" # @param {type:"string"} -uncertainty = "" # @param {type:"string"} - -case.commit_verdict(model, evidence, uncertainty) -""" - ), - # ---- 7 · the reveal ---- - md( - """ ---- -# 7 · The reveal - -Do not run this cell yet. Once everyone has recorded a verdict, we will give -you the passphrase. -""" - ), - code( - """ -passphrase = "" # @param {type:"string"} -case.reveal(passphrase) -""" - ), + code(""""""), ] From d438c2c5f1710d9f599ba16b834d5166854d8a91 Mon Sep 17 00:00:00 2001 From: Soheyl Massoudi Date: Thu, 20 Aug 2026 18:55:32 +0200 Subject: [PATCH 9/9] chore(workshop): target IDETC workshop branch --- workshops/idetc26/README.md | 2 +- workshops/idetc26/notebooks/01_find_the_best_model.ipynb | 6 +++--- workshops/idetc26/tools/build_notebook.py | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/workshops/idetc26/README.md b/workshops/idetc26/README.md index a5324594..f427aaff 100644 --- a/workshops/idetc26/README.md +++ b/workshops/idetc26/README.md @@ -42,7 +42,7 @@ trained models. ## Run it -[Open the participant notebook in Google Colab](https://colab.research.google.com/github/IDEALLab/EngiOpt/blob/codex/idetc26-workshop-participant-rework/workshops/idetc26/notebooks/01_find_the_best_model.ipynb) +[Open the participant notebook in Google Colab](https://colab.research.google.com/github/IDEALLab/EngiOpt/blob/feat/idetc26-workshop/workshops/idetc26/notebooks/01_find_the_best_model.ipynb) The GitHub copy is read-only. Participants should select **File > Save a copy in Drive** before editing. diff --git a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb index 47fb1288..62b89a7f 100644 --- a/workshops/idetc26/notebooks/01_find_the_best_model.ipynb +++ b/workshops/idetc26/notebooks/01_find_the_best_model.ipynb @@ -32,7 +32,7 @@ { "cell_type": "code", "execution_count": null, - "id": "53267677", + "id": "9d0491a1", "metadata": {}, "outputs": [], "source": [ @@ -44,7 +44,7 @@ " if setup_marker.exists():\n", " print(\"Setup already complete.\")\n", " else:\n", - " %pip install -q \"git+https://github.com/IDEALLab/EngiOpt.git@codex/idetc26-workshop-participant-rework\"\n", + " %pip install -q \"git+https://github.com/IDEALLab/EngiOpt.git@feat/idetc26-workshop\"\n", " # PyPI's engibench is 0.2.0 and so is the build from main, so pip calls\n", " # the requirement satisfied and leaves photonics2d at v0. Force the\n", " # GitHub version into the current environment before EngiBench is first\n", @@ -57,7 +57,7 @@ " if not installed.exists():\n", " raise RuntimeError(\"EngiBench installation failed: photonics2d v1 is missing.\")\n", "\n", - " setup_marker.write_text(\"EngiOpt codex/idetc26-workshop-participant-rework; EngiBench main\\n\")\n", + " setup_marker.write_text(\"EngiOpt feat/idetc26-workshop; EngiBench main\\n\")\n", " print(\"Setup complete. Continue with the next cell.\")" ] }, diff --git a/workshops/idetc26/tools/build_notebook.py b/workshops/idetc26/tools/build_notebook.py index f15d3d12..3dcc7215 100644 --- a/workshops/idetc26/tools/build_notebook.py +++ b/workshops/idetc26/tools/build_notebook.py @@ -38,7 +38,7 @@ NOTEBOOK_DIR = Path(__file__).resolve().parents[1] / "notebooks" -BRANCH = "codex/idetc26-workshop-participant-rework" +BRANCH = "feat/idetc26-workshop" """The EngiOpt branch Colab installs from.""" ENGIBENCH_REF = "main"