Safe submission contract before the leaderboard accepts external checkpoints #78

Raised by @SoheylM in review of #75. The analysis below is his; this issue records it,
explains the mechanics for anyone picking it up later, and states what has to be true
before the leaderboard accepts a checkpoint from outside IDEALLab.

The situation this is about

A leaderboard row records where its weights live: repo, path, revision, content hash.
python -m engiopt.verify follows that address, downloads the package, rebuilds the model
through its registered adapter, and re-scores it. That is what makes a row falsifiable
instead of a self-report.

Today every row comes from IDEALLab, because publishing one needs write access to the
board repo. The moment that changes, the runner downloads and executes bytes chosen by
a stranger, unattended, on a machine that may hold credentials.

Why loading a checkpoint is running a program

This is the part that is easy to miss, so it is worth stating plainly.

A .pth file is not a data format. PyTorch saves with Python's pickle, and a pickle
file is bytecode for a small stack machine. One of its opcodes means "import this module
and call this callable with these arguments." So torch.load(path) on a file you did not
create is exec() on a file you did not create: arbitrary code runs before a single tensor
is read.

A recorded revision and content hash do not help here. They establish which bytes were
loaded. They say nothing about whether loading them was safe.

What #75 already did

Every checkpoint load in the repository now passes weights_only=True, which swaps in a
restricted unpickler that can rebuild tensors and refuses arbitrary imports and calls.
All nine weight files across the four published packages load unchanged under it, VQGAN's
three included — it had opted into the unrestricted form and never needed it. A test in
tests/test_generator_registry.py pins this across core.py and every registered adapter
so a new model cannot quietly reintroduce it.

That closes the obvious hole for free. It is not sufficient.

What it does not cover

• Resource exhaustion. A restricted unpickler still happily rebuilds a tensor whose
  header claims it is 400 GB. Nothing reads the shape before allocating.
• Tensors that are not the model. Nothing checks that what arrived matches what the
  adapter expects — names, shapes, dtypes, count — before load_state_dict sees it.
• Participant-controlled configuration. run_config.json is what build() reconstructs
  from. Values like latent_dim size allocations and are currently trusted as read.
• Preprocessing state. Normalizer statistics recorded with a package change generated
  outputs. They need the same validation as the weights.
• The runner's own blast radius. Nothing today constrains its network access,
  credentials, memory, or wall-clock.

What a safe submission contract needs

1. safetensors as the format for external submissions. A JSON header giving tensor
   names, dtypes, shapes and byte offsets, then a flat blob. No code path exists in the
   format, so loading cannot execute anything — and the header can be read without
   reading the data, which is what makes validation-before-allocation possible.
2. EngiOpt exports safetensors automatically from the training scripts, so it is never
   a manual participant step. Pickle .pth/.pt is rejected on the public path.
3. Validated JSON for reconstruction settings and preprocessing state, schema-checked
   before use rather than trusted as read.
4. Registered adapters only. A submission names an adapter that already exists in this
   repository. New executable model code enters through a reviewed code PR, never through
   a submission.
5. Header validation before loading: tensor names, shapes, supported dtypes, tensor
   count, total element count, file size — checked against adapter-specific expectations.
6. An immutable address: pinned repository revision plus content hash. Already done in
   Generator contract, shared evaluation layer, and HF-hosted leaderboard #75; listed so the contract reads as a whole.
7. A submission route that does not need IDEALLab write access. --push-to currently
   uploads the merged CSV directly. A pending manifest opened as a HuggingFace Community PR
   (create_pr=True) is the straightforward option.
8. A legacy path for existing IDEALLab .pth packages, explicitly allowlisted, with
   migration tracked separately. A user-controlled repository or URL must never be able to
   select it.
9. Participant rows stay verified=false. Only the official runner publishes verified
   scores. Already true in Generator contract, shared evaluation layer, and HF-hosted leaderboard #75.
10. Runner isolation: no credentials, restricted network, CPU/GPU/memory/time limits.
    Needed before unattended operation, not before the format.

Why this was not folded into #75

Not because the risk is theoretical — it is real, and #75 is what makes a remote package
the only way to reach a checkpoint at all.

The argument for including it was that the format cannot be deferred even if the runner
can, because packages accepted now would otherwise need converting later. That is a claim
about a migration cost, and the cost is currently zero: there are no external
submissions, and there cannot be until item 7 exists. The set of packages that would need
converting is empty and stays empty until the board opens.

Meanwhile items 1–5 and 7–8 are a subsystem — safetensors export across 14 training
scripts, per-adapter header validation, a schema layer, a submission flow, an allowlist —
landing on a PR already at 120 files and three review rounds. And a submission format built
before the runner that consumes it is a guess at what the runner needs.

They are one piece of work and should land together. This issue is that piece of work.

Acceptance criteria

• Training scripts write safetensors alongside (or instead of) .pth
• Public submission path rejects pickle-based weights
• Reconstruction settings and preprocessing state validated against a schema before use
• Adapters declare expected tensor names/shapes/dtypes; headers checked before allocation
• Submission possible from an account with no IDEALLab write access
• Legacy IDEALLab .pth packages load only via an explicit allowlist, unreachable from user input
• Runner runs isolated: no credentials, restricted network, memory and time limits
• LEADERBOARD.md updated to describe the contract as built rather than as planned

Gate: items 1–8 must land before the first external submission is accepted. Item 10
must land before verification runs unattended. Neither blocks #75, which keeps submission
internal.

Current documented state: LEADERBOARD.md § Loading someone else's weights.