Repository navigation
Verification: a model whose check got no clear answer is left out, not every model measured beside it - #271
Merged
Merged
Conversation
…t every model measured beside it A benchmark result OMP reports as unresolved or unmatched made deriveCatalog refuse code_probe_inconclusive_probe for the whole list, so one provider that could not be checked (a key without credit, a timeout, an unclassified wording) discarded every model measured beside it and nothing was saved. Such a model is now excluded with the reason inconclusive, exactly as a refused one is, and the Models sheet names it "no answer". The derive refusals that remain have words instead of "Code refused the step".
… use or moves the team off its family A benchmark reads an outage, a rate limit or a broken credential as unresolved, the same as a model the account cannot use. Leaving such a model out is right for a first list, but a verification again during an outage would have shortened the list in use and narrowed the saved selection with it. The run now stops at derive, naming the models (code_models_unanswered), when an inconclusive model is in the list in use or when narrowing would move the selection off the provider lane it names because none of that family answered; the list and the saved selection stay as they were. The derive refusals read as the receipts and ladders they describe, and a ladder regression keeps its identifiers as written.
…ane families, and nothing a newer version replaces The list in use is protected from a non-answer only when a verification with this exact account pool made it: a list verified with other accounts, or imported, vouches for nothing these accounts could keep, and a model that never answers would otherwise block every verification again. A lane names its families through laneFamilies, so the mixed lane protects both OpenAI and Anthropic, and the rule is judged before the selection is narrowed. A model with no clear answer that a newer, answering version supersedes, or that draws a quota of its own, is named as it would be had it answered, since its answer could not change the list.
A non-answer can be permanent, so the stop also offers choosing a team or accounts that do without the named models, and the derivation comment names exactly when a model with no answer is named superseded or separate_quota.
Owner
Author
|
Gate on |
A stop after a current verification was only announced: the sheet kept reading "back to code · the generator routes on this list", so a verify again that kept the list in use looked like nothing had happened. The verified step now reads "verification stopped" beside its action, with the reason in the readout, as the unverified step already did; the stop message is shorter.
Owner
Author
|
Gate on |
Owner
Author
Verdict: passHead
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A model verification probes every candidate, then
deriveCatalogbuilds the list from the receipts. Any result OMP reports asunresolvedorunmatchedmade the derivation refusecode_probe_inconclusive_probefor the whole list. One provider that could not be checked (a key without credit, a timeout, a wording OMP does not classify) therefore discarded every model measured beside it, nothing was staged, and the panel said only "Verification stopped while building the model list: Code refused the step."Observed on the preview: a first verification in a fresh workspace measured 32 candidates. 19 were reachable, 5
not_found, 2client_blockedand 6unresolved(four OpenRouter models, two Codex models). The derivation refused and the workspace stayed without a model list.Change
domain/probe.tscatalogFromObservations: a candidate whose probe isunresolvedorunmatchedis left out instead of refusing the list. It is namedinconclusive, or as it would be had it answered where its answer could not change the list:supersededwhen a newer version that answered replaces it,separate_quotawhen it draws a quota of its own. Missing, stale and wrong-API receipts still refuse (code_probe_missing_probe).workflow.tsconfirmVerification: OMP reads an outage, a rate limit or a broken credential asunresolvedtoo, so a non-answer may not take away anything already chosen. Before the selection is narrowed, the run stops at derive withcode_models_unanswered, naming the models, when aninconclusivemodel is in the list in use as a verification with this exact account pool made it, or belongs to a family the wanted lane names (one provider, or OpenAI and Anthropic for the mixed lane) that the new list would not have at all. The list in use and the saved selection stay unchanged. A first list, a list verified with other accounts or never verified, and a family that still has an answering rung are saved without the models that gave no answer.inconclusivemodel no answer. After a current verification, a verification again that stopped now reads verification stopped beside back to code, with its reason in the readout; before, the stop was only announced and the sheet looked unchanged.domain/routing.ts:laneFamilies(lane)names the families a lane uses; the mixed lane's two families are one constant shared withlanesForanddefaultSelection.code_probe_missing_probe,code_probe_insufficient_ladder,code_ladder_regression) have words instead of "Code refused the step."; a ladder regression keeps its identifiers as written.docs/configuration.mddescribes the rules and the word.Exclusions are an action result and are not stored. The saved effect of a verification changes only as described above.
Evidence
domain/probe.test.ts: one model of a provider that otherwise answered, and the whole of a provider that answered nothing, derive exactly the document and exclusions a refused verdict derives, with the reasoninconclusive(fails onmainwithprobe_inconclusive_probe); an old version a newer answering one replaces is namedsuperseded, a separate-quota modelseparate_quota, and a newest version with no answer isinconclusivewhile its predecessor takes the rung.test/workflow.test.ts: a first list saves what answered and names the rest; verifying again while a model of a list these accounts verified gives no answer stops at derive with the configuration unchanged; after a pool change and on an imported list the same non-answer only leaves the model out; a mixed-lane first list with no Claude answer and a DeepSeek-only first list with no DeepSeek answer stop with nothing staged. The guard cases fail without the guard.inconclusive.mainrefuses the same receipts.unresolved): on a list never verified, verify models saved 8 models and the left-out list read DeepSeek's two models no answer, with "No clear answer when checked; verifying again retries it" in the readout; on a list these accounts verified, verify again stopped, the head kept "10 models · verified 1h ago", Models read verification stopped with the reason and the two addresses, and the main view's launch line wrapped the full stop text. No horizontal overflow in the Code panel at 1280, 390 and 320 px viewports or at a 162 px panel.scripts/gate.sh: see the latest comment for the current head.