casebroker — one queue for the v2 campaign: CFD wind and Radiance surface temperatures

A central database of cases plus an HTTP work queue, so that many machines — PACE ICE, PACE Phoenix, the lab workstation, anyone else's box — can each ask "what should I simulate next?" and never collide.

Where things are

Path	What
DOMAIN.md	The nouns — case, lease, worker, fleet, geometry — and the invariants
AGENTS.md	How to change this without breaking the campaign
docs/protocol.md	Client/server interaction, every endpoint, the state machine
docs/operations.md	First run, accounts, tokens, storage, releases, deploying
docs/dashboard.md	The ops UI, deep links, and read-only share links
docs/releases.md	Moving the fleet between builds while a campaign runs: catalog, canary, promote, roll back, the badges
docs/e3d-contract.md	The seam to the CFD: what a node (E3D.exe) is to the broker, where its contract lives, and the identity it declares
docs/thermal.md	The surface-temperature recipe: what a Radiance case computes, its spec, progress, telemetry and archive
docs/mrt.md	The mean-radiant-temperature recipe, built on a site's finished surface-temperature case; the needs gate
docs/dataset.md	casebroker export: the campaign as a training snapshot (Parquet tables, a Zarr store of every field, a card), and how to read it
casebroker/	The service: app.py (API), db.py (both engines, and the schema), auth.py (passwords, sessions, machine tokens), cli.py (casebroker), export.py (casebroker export), ids.py (case identity and splits)
slurm/	E3D node jobs for PACE ICE and Phoenix, and the script that queues them
tests/	The suite: SQLite-only by default, with a Postgres set that runs when CASEBROKER_TEST_PG_DSN is set

Run it

# server. The schema creates itself on the first connection -- there is no
# migration step, and no token needed to start.
CASEBROKER_DB=campaign.sqlite \
  uv run uvicorn casebroker.app:app --host 0.0.0.0 --port 8000

Then open http://localhost:8000 and create the admin account the page asks for. That is what turns auth on: until an account exists (and with no env tokens set) every caller has full access, and /healthz says so with "auth": "OPEN".

A simulation node is E3D.exe (the Eddy3D CLI) and nothing else. On the machine, E3D setup-sim-node https://broker.example.org pairs it -- it prints a code you approve under Machines; the node generates its own token and the broker keeps only its hash -- and E3D node runs it, leasing, solving and updating itself when the broker names a new build. Its side of the contract is Eddy3D's docs/SIMULATION_NODE.md; this side is docs/protocol.md.

No browser? The same first two steps, headless:

uv run casebroker init-db      --db campaign.sqlite     # optional; the server does this too
uv run casebroker account create --db campaign.sqlite --username ada --role admin

Before this faces the internet, set CASEBROKER_SETUP_TOKEN — the setup form cannot require a login, so otherwise whoever reaches it first becomes your permanent admin. Full sequence in docs/operations.md.

Retryable or fatal -- the distinction a node's failure report carries

A node reports a failure as retryable or not (POST /v1/fail, retryable). Fatal means this site is broken and must never be retried -- degenerate geometry that will fail identically everywhere, forever; anything else (a node died, an image pull failed, a host was down) is retryable, and goes to another machine first.

Getting that distinction wrong is expensive in one direction only. A wrongly retryable error costs at most three attempts; a wrongly fatal one removes a site from the campaign permanently. Treating a missing input file as fatal once quarantined 173 perfectly good sites in under a minute.

Why a broker rather than splitting the case list up front

The broker badge pings /healthz (unauthenticated, so uptime checks need no token). It says whether the service is up, not whether the campaign is moving — the dashboard answers that.

• Phoenix's free embers QOS preempts jobs after one hour. A statically assigned case dies with its job. A leased case comes back to the pool by itself, so preemption costs one partial solve instead of a lost slot.
• Machines are wildly unequal. ICE, Phoenix and the workstation differ by more than an order of magnitude in throughput, and Phoenix's availability changes hour to hour. Pull beats push whenever the workers' speeds are unknown.
• The dataset keeps growing (5k → 10k → 30k). Adding cases is one idempotent POST; nothing is renumbered and no worker needs restarting.
• A Windows box behind a firewall can take part. Workers need outbound HTTPS and nothing else — the broker never calls a worker.

Quick reference — the calls you will actually type

uv run casebroker account create --username ada --role admin  # the first admin, headless
uv run casebroker account passwd --username ada               # forgot it -- no old password needed
uv run casebroker account list                                # who exists, and last login
uv run casebroker init-db                                     # create/upgrade the schema alone
uv run casebroker token new                                   # generate a shared env token
uv run casebroker token check --broker URL --expect write     # what can this one do?
uv run casebroker health  --broker URL                        # version and auth posture
uv run casebroker doctor  --broker URL                        # find the broken piece
uv run casebroker fleet   --broker URL --cluster ICE          # report squeue (login node)
uv run casebroker release register release.json --upload . --broker URL  # add a node build, upload its files
uv run casebroker release promote <worker> --broker URL       # canary is good: whole fleet onto it
uv run casebroker release rollback --broker URL [--block]     # back to the previous build (kill switch)
uv run casebroker repro <case_id> --e3d E3D.exe --engine bluecfd  # re-run a failed case HERE
uv run casebroker triage <study_dir> [--node-log node.log]   # name the known failure in its logs
uv run --extra export casebroker export --broker URL --out ds # dataset snapshot; re-run fetches only what changed

Full endpoint table in docs/protocol.md.

Updating the nodes — and testing the auto-updater

A node updates itself only when it runs under the supervisor, E3D node. Started as E3D run-sim-node (or run-simulation-node) it never switches, and says so at start-up: "started directly, so it will not switch builds by itself" (a PACE job runs E3D node-release sync before it starts instead). The broker names the build and its SHA-256, and holds the file once it is uploaded (release register --upload, which Eddy3D's build does for every push to dev once its secrets are set). The node installs that file from its release share, by default %LOCALAPPDATA%\Eddy3D\node\releases (a folder on that machine), or else fetches it from the broker; either way only if it hashes to that SHA-256. The whole model, with the guards and badges, is in docs/releases.md.

To test it on ONE machine before trusting it with the fleet:

1. Run the machine under the supervisor. Stop its node, then start it with the same options, node in place of run-sim-node:

   E3D.exe node --done E:\wind\done --cpus 36

   A case in flight is resumed, not restarted.

2. Put a different build on its share. Every push to Eddy3D dev builds one (gh release download e3d-node-latest -R Eddy3D-Dev/Eddy3D -p E3D.exe). Publish it into the share; this prints the release.json the broker needs:

   E3D.exe node-release publish --file .\E3D.exe --to $env:LOCALAPPDATA\Eddy3D\node\releases > release.json

3. Register it: uv run casebroker release register release.json --broker URL --username <admin> (or paste release.json into Machines → Node builds → Register a published build).

4. Make this one machine the canary. Machines → Node builds: set its Canary target to the new build, and pick when it switches: direction is the right test, since it proves a case survives the switch without waiting for the case to finish.

5. Check that it switched. Within a heartbeat, the node reports that it installed and verified the file and will switch. At the boundary it exits and the supervisor starts the new build, which asks for its own case back and continues at the next direction without spending an attempt. Look for all four:

   • %LOCALAPPDATA%\Eddy3D\node\versions\<build>\ exists and current.txt names it;
   • the dashboard's Build column shows the new build and no → badge;
   • the case still says attempt 1;
   • the finished direction folders carry an .e3d-build stamp from the build that solved each one.

6. Test the way back. Set the canary target back to the old build. The node switches again at the boundary, and nothing is downloaded, because the old build is still installed side by side.

7. Test the refusal. Register a build whose file you did NOT put on the share, and make it the canary target. The node must stay where it is, and the Build column's → tooltip must say the broker wants X and has no file on the share yet. A node that runs something it could not verify is the failure this step is looking for.

8. Then the fleet. uv run casebroker release promote <canary worker> moves everyone to the canary's build in one call. uv run casebroker release rollback goes back to the previous fleet target, and --block also refuses the build being left, at every node's next lease.

Machines that are not under the supervisor yet stay on their build and are counted cannot update by themselves. So step 1 has to be done once on every machine, by hand.

Status

The v2 campaign runs on a self-hosted broker and Postgres (https://casebroker.eddy3d.com, deploy/self-hosted), with E3D nodes on lab workstations and PACE ICE and Phoenix. Two things are worth knowing before trusting a result:

• Every building height is a prediction. GlobalBuildingAtlas covers >97% of buildings, which retired Overture's coverage problem, but it did so by predicting a height for each one from lidar, optical and radar rather than measuring it — published RMSE 1.5–8.9 m by continent. The case inspector draws the per-building variance, and that, not the building count, is what to judge a case on.
• Terrain is effectively complete on land. Gaps are ocean, not missing data — GEDTM30 is a land DTM.
• Trees are modelled where the canopy map reaches. The Meta/WRI 1 m canopy height model covers the inhabited world and has polar and open-ocean gaps; a treeless site and an uncovered one are reported differently, because only one of them means the case is wrong.