Skip to content

Verification: a model whose check got no clear answer is left out, not every model measured beside it - #271

Merged
atyrode merged 6 commits into
mainfrom
fix/unchecked-models-excluded
Oct 12, 2026
Merged

atyrode merged 6 commits into
mainfrom
fix/unchecked-models-excluded

Conversation

@atyrode

@atyrode atyrode commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Problem

A model verification probes every candidate, then deriveCatalog builds the list from the receipts. Any result OMP reports as unresolved or unmatched made the derivation refuse code_probe_inconclusive_probe for the whole list. One provider that could not be checked (a key without credit, a timeout, a wording OMP does not classify) therefore discarded every model measured beside it, nothing was staged, and the panel said only "Verification stopped while building the model list: Code refused the step."

Observed on the preview: a first verification in a fresh workspace measured 32 candidates. 19 were reachable, 5 not_found, 2 client_blocked and 6 unresolved (four OpenRouter models, two Codex models). The derivation refused and the workspace stayed without a model list.

Change

  • domain/probe.ts catalogFromObservations: a candidate whose probe is unresolved or unmatched is left out instead of refusing the list. It is named inconclusive, or as it would be had it answered where its answer could not change the list: superseded when a newer version that answered replaces it, separate_quota when it draws a quota of its own. Missing, stale and wrong-API receipts still refuse (code_probe_missing_probe).
  • workflow.ts confirmVerification: OMP reads an outage, a rate limit or a broken credential as unresolved too, so a non-answer may not take away anything already chosen. Before the selection is narrowed, the run stops at derive with code_models_unanswered, naming the models, when an inconclusive model is in the list in use as a verification with this exact account pool made it, or belongs to a family the wanted lane names (one provider, or OpenAI and Anthropic for the mixed lane) that the new list would not have at all. The list in use and the saved selection stay unchanged. A first list, a list verified with other accounts or never verified, and a family that still has an answering rung are saved without the models that gave no answer.
  • The Models sheet names an inconclusive model no answer. After a current verification, a verification again that stopped now reads verification stopped beside back to code, with its reason in the readout; before, the stop was only announced and the sheet looked unchanged.
  • domain/routing.ts: laneFamilies(lane) names the families a lane uses; the mixed lane's two families are one constant shared with lanesFor and defaultSelection.
  • The derive refusals that remain (code_probe_missing_probe, code_probe_insufficient_ladder, code_ladder_regression) have words instead of "Code refused the step."; a ladder regression keeps its identifiers as written.
  • docs/configuration.md describes the rules and the word.

Exclusions are an action result and are not stored. The saved effect of a verification changes only as described above.

Evidence

  • domain/probe.test.ts: one model of a provider that otherwise answered, and the whole of a provider that answered nothing, derive exactly the document and exclusions a refused verdict derives, with the reason inconclusive (fails on main with probe_inconclusive_probe); an old version a newer answering one replaces is named superseded, a separate-quota model separate_quota, and a newest version with no answer is inconclusive while its predecessor takes the rung.
  • test/workflow.test.ts: a first list saves what answered and names the rest; verifying again while a model of a list these accounts verified gives no answer stops at derive with the configuration unchanged; after a pool change and on an imported list the same non-answer only leaves the model out; a mixed-lane first list with no Claude answer and a DeepSeek-only first list with no DeepSeek answer stop with nothing staged. The guard cases fail without the guard.
  • The branch's derivation, run read-only on the preview workspace's own receipts, derives GPT luna/terra, Claude haiku/sonnet/opus/fable and DeepSeek flash/v4-pro and names the six models inconclusive. main refuses the same receipts.
  • Real surface, in the design lab (the branch's packed Code bundles in Chromium, fictional OMP jobs, DeepSeek probes answering unresolved): on a list never verified, verify models saved 8 models and the left-out list read DeepSeek's two models no answer, with "No clear answer when checked; verifying again retries it" in the readout; on a list these accounts verified, verify again stopped, the head kept "10 models · verified 1h ago", Models read verification stopped with the reason and the two addresses, and the main view's launch line wrapped the full stop text. No horizontal overflow in the Code panel at 1280, 390 and 320 px viewports or at a 162 px panel.
  • scripts/gate.sh: see the latest comment for the current head.

…t every model measured beside it

A benchmark result OMP reports as unresolved or unmatched made deriveCatalog refuse
code_probe_inconclusive_probe for the whole list, so one provider that could not be
checked (a key without credit, a timeout, an unclassified wording) discarded every
model measured beside it and nothing was saved. Such a model is now excluded with the
reason inconclusive, exactly as a refused one is, and the Models sheet names it
"no answer". The derive refusals that remain have words instead of "Code refused the
step".
… use or moves the team off its family

A benchmark reads an outage, a rate limit or a broken credential as unresolved, the
same as a model the account cannot use. Leaving such a model out is right for a first
list, but a verification again during an outage would have shortened the list in use
and narrowed the saved selection with it. The run now stops at derive, naming the
models (code_models_unanswered), when an inconclusive model is in the list in use or
when narrowing would move the selection off the provider lane it names because none
of that family answered; the list and the saved selection stay as they were.

The derive refusals read as the receipts and ladders they describe, and a ladder
regression keeps its identifiers as written.
…ane families, and nothing a newer version replaces

The list in use is protected from a non-answer only when a verification with this
exact account pool made it: a list verified with other accounts, or imported, vouches
for nothing these accounts could keep, and a model that never answers would otherwise
block every verification again. A lane names its families through laneFamilies, so
the mixed lane protects both OpenAI and Anthropic, and the rule is judged before the
selection is narrowed. A model with no clear answer that a newer, answering version
supersedes, or that draws a quota of its own, is named as it would be had it answered,
since its answer could not change the list.
A non-answer can be permanent, so the stop also offers choosing a team or accounts
that do without the named models, and the derivation comment names exactly when a
model with no answer is named superseded or separate_quota.
@atyrode

atyrode commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Gate on 698a134523e388d4d5d29843ab6b2431a0494277 (rebased on main 49a3279420): scripts/gate.sh exit 0 — 433 pass, 0 fail; browser PASS (409s); native proofs standalone-native-and-packaged-worker-boundaries and code-omp-native-model-tools ok.

A stop after a current verification was only announced: the sheet kept reading
"back to code · the generator routes on this list", so a verify again that kept the
list in use looked like nothing had happened. The verified step now reads
"verification stopped" beside its action, with the reason in the readout, as the
unverified step already did; the stop message is shorter.
@atyrode

atyrode commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Gate on 44229261ca7731d17c7fcf4362d0f940044a1b18 (on main 49a3279420): scripts/gate.sh exit 0 — 433 pass, 0 fail; browser PASS (413s); native proofs standalone-native-and-packaged-worker-boundaries and code-omp-native-model-tools ok.

@atyrode

atyrode commented Oct 12, 2026

Copy link
Copy Markdown
Owner Author

Verdict: pass

Head 44229261ca7731d17c7fcf4362d0f940044a1b18, on main 49a3279420.

  • Independent review (three rounds plus a final delta): every blocking finding was fixed. The fixes cover protecting only lists verified with the same account pool, both mixed-lane families, judging lost families before the selection is narrowed, superseded and separate-quota naming, and a visible stop in Models. The final verdict was pass. The remaining optional item is a stop that stays shown after the list in use changes elsewhere; the main view's launch line already shows that, and this PR does not change it.
  • Local scripts/gate.sh on this head: exit 0, 433 pass, browser PASS (413s), both native proofs.
  • Real surface checked in Chromium through the design lab: the no answer left-out rows, the stopped verification again in Models and the main view's launch line. No horizontal overflow in the Code panel down to 162 px.
  • Required CI on this head: green (test, plugins / verify, agent-policy).

@atyrode
atyrode merged commit 99ad709 into main Oct 12, 2026
4 checks passed
@atyrode
atyrode deleted the fix/unchecked-models-excluded branch October 12, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant