Verification: a model whose check got no clear answer is left out, not every model measured beside it

docs/configuration.md

@@ -152,13 +152,24 @@ rungs any other family's listing ladders (DeepSeek, OpenRouter, any other provid
 such a rung that does not answer leaves that family shorter. A family Code requires is
 laddered on whatever of it answers, short below three rungs, and one none of whose models
 answers is left out as a family with no account is; a selection that needs it is refused
-where it is reviewed. `deriveCatalog` returns the document and every other offered model
-with its reason; the document's compiled catalog names its short families. It refuses only a
-family Code requires whose ladder would put a model above one that costs no more and thinks deeper
-(`code_ladder_regression`, naming both), a benchmark that does not answer the charge exactly,
-and nothing laddered at all (`code_probe_insufficient_ladder`). Only the returned `confirm`
+where it is reviewed. A model whose probe got no answer the benchmark could read (OMP's
+`unresolved` or `unmatched`) is left out rather than discarding every model measured beside
+it: as `inconclusive`, or as it would be had it answered where its answer could not change
+the list (`superseded` by a newer version that answered, `separate_quota`). `deriveCatalog` returns the document
+and every other offered model with its reason; the document's compiled catalog names its
+short families. It refuses only a family Code requires whose ladder would put a model above
+one that costs no more and thinks deeper (`code_ladder_regression`, naming both), a benchmark
+that does not answer the charge exactly, and nothing laddered at all
+(`code_probe_insufficient_ladder`). Only the returned `confirm`
 spends: one benchmark job per provider, then derive, stage, review, promote and save
-the selection narrowed to what the verified catalog hosts. Every step re-observes the
+the selection narrowed to what the verified catalog hosts. An `inconclusive` model may be
+left out only where nothing already chosen needs it. When it is in the list in use as a
+verification with this exact account pool made it, or belongs to a family the wanted lane
+names (one provider, or OpenAI and Anthropic for the mixed lane) that the new list would not
+have at all, the run stops at derive (`code_models_unanswered`, naming the models) before the
+selection is narrowed, with the list and the saved selection unchanged, since an outage or a
+rate limit reads the same as a model the account cannot use. A list verified with other
+accounts, or never verified, holds no such claim. Every step re-observes the
 revision, the exact account pool, OMP defaults and OMP itself, and stops, naming itself,
 when one moved, on a refusal or on cancellation. A stop never undoes: probe jobs stay in
 OMP's history and a staged catalog stays staged. A confirmation is used once, and a failed
@@ -780,15 +791,19 @@ verifying, a staged list's changes (field by field, a model added or dropped, th
 value struck), or, after a verification in this panel, the models it left out, each
 with its reason in a word: a model the accounts' plan, their settings or this client
 excludes (`client_blocked`) reads **not served**, one no account finds (`not_found`) **not
-reachable**, a model in OMP's `spark` quota class **retired** and any other class of its
-own **own quota**. Ids, prices, speeds, thinking ranges and reasons
+reachable**, one whose check got no clear answer (`inconclusive`) **no answer**, a model in
+OMP's `spark` quota class **retired** and any other class of its own **own quota**. Ids,
+prices, speeds, thinking ranges and reasons
 appear only in the readout line while a cell, a provider or a left-out model is pointed
 or focused.
 One next action sits where the launch sits and follows the step: **verify models**,
 **checking models…** (drawn with the checking hold the charge keeps), **confirm charge**
 (a single deliberate press of the exact charge shown, as in the launch line), **verifying…**
 with the requests measured, then **back to code** with **verify again** beside it, the
-same Verify a current verification can always renew, its charge confirmed the same way.
+same Verify a current verification can always renew, its charge confirmed the same way;
+a verification again that stopped reads **verification stopped** beside it, with its
+reason in the readout, since what it would have changed (the list, or for a stop while
+saving the profile, the team) stays as it was.
 A staged list offers **use staged list**, which reviews the staged list at its revision
 and promotes exactly that review with the list's own default team, and **discard**. An
 unsaved edit made on the list a write would replace holds it, saying what the write does

plugins/code/generator/models-sheet.tsx

@@ -284,12 +284,19 @@ export function ModelsSheet({ host, model, machine, onBack, backRef, onPlace }:
             : { value: "use staged list", text: draft?.provenance ? "replaces the list in use with this verified one" : "replaces the list in use; it needs verifying after" },
           press: () => { if (writeRefusal) return writeRefusal; useStaged(); return null; } };
       }
-      case "verified":
+      case "verified": {
         // The head already says when the list was verified; beside the action, only its alternative: verifying again, through
-        // the same gate as a first verification, whose charge still waits on its own confirmation.
-        return { label: "back to code", parts: refused, fixes: verifyGate.open ? [{ label: "verify again", run: () => model.actions.verify() }] : [],
-          said: failure ? { value: "back to code", text: failure, warn: true } : { value: "back to code", text: "the generator routes on this list" },
+        // the same gate as a first verification, whose charge still waits on its own confirmation. A verification again that
+        // stopped says so, as a first one does: what it would have changed (the list, or for a stop while saving the profile,
+        // the team) stays as it was, which the head alone cannot show.
+        const stopped = verification.failure;
+        const note: readonly Part[] = stopped ? [[stopped.cancelled ? "verification cancelled" : "verification stopped", stopped.cancelled ? null : "attention"]] : [];
+        return { label: "back to code", parts: [...note, ...refused], fixes: verifyGate.open ? [{ label: "verify again", run: () => model.actions.verify() }] : [],
+          said: failure ? { value: "back to code", text: failure, warn: true }
+            : stopped && !stopped.cancelled ? { value: "back to code", text: stopped.reason, warn: true }
+            : { value: "back to code", text: "the generator routes on this list" },
           press: () => { leave(onBack); return null; } };
+      }
       case "refused": return { label: "enable in setup", parts: [["Discovery is not enabled", "attention"], ["verifying needs it", null]], fixes: [],
         said: { value: "enable in setup", text: `Setup turns on model discovery for ${where}` }, press: () => { leave(() => onPlace("setup")); return null; } };
       case "unverified": {

plugins/code/generator/sheets-model.ts

@@ -93,6 +93,7 @@ const EXCLUSION_WORDS: Readonly<Record<Exclusion["reason"], readonly [string, st
   unstable_id: ["unstable id", "A rolling alias, preview or experiment, which Code never probes"],
   not_found: ["not reachable", "Not found through your accounts"],
   client_blocked: ["not served", "Not served to your accounts: their plan, their settings or this client excludes it"],
+  inconclusive: ["no answer", "No clear answer when checked; verifying again retries it"],
   regression: ["worse than cheaper", "Worse than a cheaper tier"],
   separate_quota: ["own quota", "Draws a quota of its own, which Code does not spend"],
 };

plugins/code/workflow.ts

@@ -1,5 +1,5 @@
 import { createOmpClient, OmpHarnessProfileSchema, OmpSessionRefSchema, OMP_PLUGIN_ID, LAUNCH_OPERATION_ID, MATERIAL_SESSION_OPERATION_ID, RESUME_OPERATION_ID, skillInputBindings, PREPARE_WORKSPACE_OPERATION_ID, VALIDATE_WORKSPACE_OPERATION_ID,
-  INVENTORY_OPERATION_ID, BENCHMARK_OPERATION_ID, ThinkingSelectorSchema,
+  INVENTORY_OPERATION_ID, BENCHMARK_OPERATION_ID, ThinkingSelectorSchema, probeAddress,
   type BenchmarkInput, type BenchmarkReceipt, type InventoryReceipt, type OmpHarnessProfile, type OmpSessionRef, type OmpAction, type RunDials, type ThinkingSelector,
   type ActionInput as OmpInput, type ActionResult as OmpResult } from "@atyrode/manifold-omp";
 import { JobDeploymentApplyArgsSchema, JobDeploymentListArgsSchema, JobDeploymentListResultSchema,
@@ -13,8 +13,9 @@ import { HARNESS_OPERATION_ID, observePermissionPlan, operationReady, type Permi
 import { selectedAccountPool } from "../domain/accounts.ts";
 import { compileCatalog } from "../domain/catalog.ts";
 import { projectUsage } from "../domain/usage.ts";
+import { providerPolicy } from "../domain/providers.ts";
 import type { Exclusion } from "../domain/probe.ts";
-import { clampSelection } from "../domain/routing.ts";
+import { clampSelection, laneFamilies } from "../domain/routing.ts";
 import { DomainError, type AccountChoices, type Selection } from "../domain/contracts.ts";
 
 export type Dispatch = (door: string, input: unknown) => Promise<unknown>;
@@ -35,6 +36,10 @@ const messages: Readonly<Record<string, string>> = {
   code_invalid_catalog: "The model list could not be used.",
   code_preview_changed: "The reviewed model list changed. Review it again.",
   code_starter_candidate_limit: "The bundled model list is too large to verify at once.",
+  code_probe_missing_probe: "OMP's results do not match the models it was asked to check. Verify again.",
+  code_probe_insufficient_ladder: "No model answered its check, so there is no list to build.",
+  code_ladder_regression: "Among the models that answered, a higher tier thinks less than a lower one that costs no more.",
+  code_models_unanswered: "Models the list in use or this team needs gave no clear answer, so the list and team stayed as they were. Verify again later, or choose a team or accounts without them.",
   code_verification_cancelled: "Verification cancelled.",
   code_verification_changed: "The machine, workspace or edit access changed, so the verification stopped.",
   code_verification_confirmed: "That charge was already confirmed.",
@@ -89,6 +94,8 @@ const agentRefusals: Readonly<Record<string, string>> = {
 /** Refusals whose detail is a list of identifiers, with the label that introduces it; the list stays exactly as written. */
 const listedDetails: Readonly<Record<string, string>> = {
   code_model_unpublished: "Not listed:",
+  code_models_unanswered: "No answer:",
+  code_ladder_regression: "In",
 };
 /** How a browser's dispatch reports a refused door (machine-web.ts `codeWorkflow`): `<plugin>.<action>: <denial>. No approval or readiness is assumed.` */
 const DOOR_DENIAL = /^[a-z0-9_-]+(?:\.[A-Za-z0-9_-]+)+: ([^]*?)(?:\. No approval or readiness is assumed\.)?$/;
@@ -117,6 +124,21 @@ export function failureWords(message: string): string {
   return Object.hasOwn(listedDetails, token!) ? `${words} ${listedDetails[token!]} ${detail}` : `${words} ${detail.charAt(0).toUpperCase()}${detail.slice(1)}`;
 }
 
+/**
+ * The left-out models a verification may not lose merely because they gave no clear answer
+ * (`inconclusive`): one the list in use has, and every one of a family the wanted selection's lane
+ * names that the new list (`families`) would not have at all. An outage, a rate limit or a wording
+ * OMP cannot read would otherwise shorten the list in use, or move the team off a family it chose,
+ * which only a provider's own refusal of a model may do. `inUse` is empty where no verification
+ * with this exact pool vouched for the list in use: evidence from other accounts, or a list never
+ * verified, holds nothing these accounts could keep.
+ */
+function unansweredNeeds(exclusions: readonly Exclusion[], inUse: ReadonlySet<string>, wanted: Selection | null, families: readonly string[]): string[] {
+  const lost = wanted ? laneFamilies(wanted.lane).filter(family => !families.includes(family)) : [];
+  return exclusions.filter(exclusion => exclusion.reason === "inconclusive" &&
+    (inUse.has(probeAddress(exclusion)) || lost.includes(providerPolicy(exclusion.provider).family))).map(probeAddress);
+}
+
 /** The steps of a model verification, in the order they run; a failure names the one it stopped at. */
 export type VerificationStep = "observe" | "initialize" | "inventory" | "draft" | "benchmark" | "derive" | "stage" | "review" | "promote" | "select";
 /** Requests a provider's benchmark has answered out of those it was charged. */
@@ -521,6 +543,11 @@ export function createCodeWorkflowClient(dispatch: Dispatch) {
     // before the start: one republished while the inventory runs is then a moved OMP, refused before the charge is answered.
     const [defaults, pool, catalogRevision] = await run.call(() => Promise.all([omp("readDefaults", {}), observePool(target.containerId, revision),
       readCatalogRevision()]));
+    // The models of the list in use, when it was verified with this exact pool: what a model that merely gives no clear
+    // answer may not remove. Spark's retired rung (tier 0) is never routed, so nothing needs it.
+    const active = record.active;
+    const inUse = new Set(active && active.provenance?.poolIdentityDigest === pool.poolIdentityDigest
+      ? active.document.models.filter(model => model.tier !== 0).map(probeAddress) : []);
     // The start is checked by the job's own wait, so a stop that lands now cancels the job rather than abandoning it.
     const job = await run.call(() => omp("startInventory", { ...target, expectedDefaultsRevision: defaults.revision, accountPool: pool.accountPool }), false);
     evidence.inventoryJobId = job.jobId;
@@ -546,11 +573,11 @@ export function createCodeWorkflowClient(dispatch: Dispatch) {
       confirm: (selection, confirmOptions = options) => {
         if (confirmed) return Promise.reject(new WorkflowError("code_verification_confirmed"));
         confirmed = true;
-        return confirmVerification(charge, selection, verificationRun(confirmOptions, { ...evidence, benchmarkJobIds: [] }));
+        return confirmVerification(charge, inUse, selection, verificationRun(confirmOptions, { ...evidence, benchmarkJobIds: [] }));
       },
     };
   }
-  async function confirmVerification(charge: ChargeReview, wanted: Selection | null, run: VerificationRun): Promise<VerifiedModels> {
+  async function confirmVerification(charge: ChargeReview, inUse: ReadonlySet<string>, wanted: Selection | null, run: VerificationRun): Promise<VerifiedModels> {
     const { target } = charge;
     run.enter("benchmark");
     await unchangedCharge(charge, run);
@@ -575,6 +602,13 @@ export function createCodeWorkflowClient(dispatch: Dispatch) {
       inventoryObservedAt: charge.inventory.observedAt, startedAt: Math.min(...receipts.map(receipt => receipt.startedAt)),
       completedAt: Math.max(...receipts.map(receipt => receipt.completedAt)), results: receipts.flatMap(receipt => receipt.results) };
     const derived = await run.call(() => code("deriveCatalog", { inventory: charge.inventory, benchmark, budget: charge.budget }));
+    // A model with no clear answer stays out of the list only where nothing already chosen needs it, so a stop
+    // here keeps the list in use and the saved team exactly as they were. It is judged before the selection is
+    // narrowed, since a team whose family gave no answer could not be formed from this list either.
+    const families = [...new Set(derived.document.models.map(model => providerPolicy(model.provider).family))];
+    const unanswered = unansweredNeeds(derived.exclusions, inUse, wanted, families);
+    if (unanswered.length > 0)
+      throw run.fail(`code_models_unanswered: ${unanswered.slice(0, 8).join(", ")}${unanswered.length > 8 ? ` and ${unanswered.length - 8} more` : ""}`);
     // Narrowed before anything is written, so a selection the verified catalog cannot serve stops
     // here with nothing staged, rather than after a promotion saved its default instead.
     let selection: Selection;