diff --git a/docs/configuration.md b/docs/configuration.md index 2cef145..c45c6a6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -169,16 +169,23 @@ catalog when the inventory started (`catalogRevision`), the list the inventory p models from — the inventory's OMP version, inventory and benchmark times, the account providers and a digest of the exact account pool. A catalog republished while the inventory runs stops the verification before its charge is answered. A catalog stops -being verified when the selected destination's inventory operation is installed from -another artifact (`describeDestination`'s pins), OMP bundles a catalog with another -revision, or the saved choices select another pool. Because the identities are content -digests, an OMP upgrade, its SDK's or its model list's, is seen whatever version it -reports and without a new Code build; no version string is compared. A catalog recorded -before verification carried these identities reads as unverified. An identity not yet -observed, or whose read failed, claims no change: the destination's failure keeps the -launch refused, a failed model list read is named in the launch line with Retry and -Models, and the session door re-checks that every routed model is still published. A -current verification can still be renewed: Verify needs only its own preconditions +being verified when the saved choices select another pool, or when OMP's bundled catalog +has moved to another revision that no longer publishes a model the profile routes to, +leads and fallbacks; the launch line then says OMP no longer lists a model this profile +uses, and Models names it, or counts several. An OMP update alone leaves the verification current: the +selected destination's inventory operation installed from another artifact +(`describeDestination`'s pins), or another catalog revision that still publishes every +routed model. What OMP publishes is read only once its catalog has moved, since the +revision a verification recorded publishes every model it laddered. Because the identities +are content digests, an update is seen whatever version OMP reports and without a new +Code build, and Models says it beside the verification's date; no version string is +compared. A catalog recorded before verification carried these identities reads as +unverified. An identity not yet observed, or whose read failed, claims no change: the +destination's failure keeps the launch refused, a failed model list read is named in the +launch line with Retry and Models, and a failed read of what a moved catalog publishes names +nothing and leaves a dropped model to the session door, which re-checks that every routed +model is still published (`code_model_unpublished`). A current verification can still be +renewed: Verify needs only its own preconditions (write access, the machine, readable accounts that include one, discovery and benchmark readiness), and its charge still waits on Confirm. A model that draws a quota of its own (any OMP class but chat, Spark's for one, as each inventory row states) is @@ -763,9 +770,11 @@ one the family offers names the rung the model row lands on instead, while one a family's top (an elite it lacks) says only that it has no model, as the model row refuses it. A family Code requires that lacks a rung of its own at fast, normal or smart reads **· short** beside its name. The head says -how many models are in use and when they were verified, or that a staged list waits -beside them, that they are verifying, or that they are unverified or the bundled -starter, with **import** (`i`) and **export** (`x`) as two quiet words. Beside the list +how many models are in use and when they were verified, with **OMP updated since** when +OMP's runtime or catalog moved after that, or that accounts changed or OMP dropped a routed +model since they were verified, that a staged list waits beside them, that they are +verifying, or that they are unverified or the bundled starter, with **import** +(`i`) and **export** (`x`) as two quiet words. Beside the list is what the step is about: the charge per provider and the models not probed while verifying, a staged list's changes (field by field, a model added or dropped, the old value struck), or, after a verification in this panel, the models it left out, each diff --git a/docs/status.md b/docs/status.md index a4848de..e4982cd 100644 --- a/docs/status.md +++ b/docs/status.md @@ -211,12 +211,16 @@ files, service state, credentials or backups. `quotaTier`, which the derivation reads directly. OMP's bundled catalog is stamped with its separately pinned SDK's version, now also 18.8.6. No version string decides whether a verification holds: it records the artifact its inventory ran from and the bundled - catalog's revision, and goes stale when the destination pins another artifact or OMP - bundles another revision, with no Code rebuild. A verification recorded before this - change, without those identities, reads as unverified and asks to be verified again, - and a current one can be renewed on purpose. Unit tests and the disposable browser - prove this against a synthetic destination and the real bundled catalog; no - installed OMP upgrade has been observed. This source repin is not a release, preview + catalog's revision. An OMP update alone, another artifact or another catalog revision that + still publishes every routed model, leaves it current and is noted in Models; it goes stale + when a moved catalog no longer publishes a routed model, or the saved choices select another + pool, with no Code rebuild. A verification recorded before these identities, without them, + reads as unverified and asks to be verified again, and a current one can be renewed on + purpose. Unit tests prove the rule against a synthetic destination and catalog, and the + disposable browser against the real bundled catalog: it moves a verified list's runtime + artifact, then serves a synthetic republish of that catalog without the list's default lead. + Applied to the preview workspace's live reads (destination pins, catalog revision and account + pool), the rule gives current: only the runtime artifact moved. This source repin is not a release, preview installation, native deployment review or provider observation. The one-shot-only `agentTools: { runId }` selector carries existing native authority; it does not grant, infer or acknowledge it. Omission stays unbound. diff --git a/plugins/code/generator/launch-step.ts b/plugins/code/generator/launch-step.ts index cbb086c..24cb1c9 100644 --- a/plugins/code/generator/launch-step.ts +++ b/plugins/code/generator/launch-step.ts @@ -152,7 +152,7 @@ const blockerAction: Readonly, string>> = { unverified: "Verify models with your accounts before saving or launching.", "accounts-changed": "Accounts changed since models were verified. Verify again.", - "omp-changed": "OMP changed since models were verified. Verify again.", + unpublished: "OMP no longer lists a model this profile uses. Verify again.", }; function blockerReason(code: LaunchBlockerCode): LaunchBlocker { const action = blockerAction[code]; @@ -215,11 +215,11 @@ export function saveGate(facts: LaunchFacts): LaunchBlockerCode | null { * review → skill problems → `launch` when the review is current, else `review`. * * Verification precedes saving and reviewing: whatever team is shown, its first save and every - * launch need a catalog verified against the present accounts and OMP. While it is not current - * the step is `verify`, or the first failing verification precondition in its order (verifying → - * read-only → conflict → runtime unavailable → accounts unreadable or none included → - * destination status unavailable → discovery/benchmark permission). Below that the precedence is - * unchanged. + * launch need a catalog verified with the present accounts, whose routed models OMP still + * publishes. While it is not current the step is `verify`, or the first failing verification + * precondition in its order (verifying → read-only → conflict → runtime unavailable → accounts + * unreadable or none included → destination status unavailable → discovery/benchmark + * permission). Below that the precedence is unchanged. * * The unsaved branch folds in the Save control: `save` when a save of the local edit may proceed * (`saveGate`), otherwise the first failing Save precondition (read-only → conflict → models), diff --git a/plugins/code/generator/model-verification.ts b/plugins/code/generator/model-verification.ts index 4fabe26..c2a4cac 100644 --- a/plugins/code/generator/model-verification.ts +++ b/plugins/code/generator/model-verification.ts @@ -1,12 +1,13 @@ -import { useEffect, useRef, useState } from "react"; +import { useEffect, useMemo, useRef, useState } from "react"; import type { HostServices } from "@manifold/plugin"; import { BENCHMARK_OPERATION_ID, INVENTORY_OPERATION_ID, type ActionResult as OmpResult } from "@atyrode/manifold-omp"; import type { Selection } from "../../domain/contracts.ts"; +import { unpublishedModels, type ModelAddress } from "../../domain/routing.ts"; import type { Exclusion } from "../../domain/probe.ts"; import type { Configuration, Target, VerificationProvenance } from "../contract.ts"; import { VerificationError, WorkflowError, inventoryArtifact, type ChargeReview, type PendingVerification, type VerificationEvidence, type VerificationProgress, type VerificationStep } from "../workflow.ts"; -import { canWriteCodeWorkspace, codeOperationFailure, codeWorkflow, useWorkflowQuery } from "../machine-web.ts"; +import { canWriteCodeWorkspace, codeOperationFailure, codeWorkflow, useOmpQuery, useWorkflowQuery } from "../machine-web.ts"; import { operationReady } from "../permission-plan.ts"; import { CHECKING_HOLD_MS, confirmsCharge, ownInitialization, verificationState, type ConfirmActivation, type VerificationState } from "./verification.ts"; @@ -20,6 +21,8 @@ export type ModelVerificationInput = { configurationCurrent: boolean; /** The selection the operator sees; a confirmed verification saves it, narrowed to what the verified catalog hosts. */ selection: Selection | null; + /** The models the shown team's routes name, leads and fallbacks; null while no team is formed. */ + routed: readonly ModelAddress[] | null; /** The destination as OMP describes it now; null until it answers. */ setup: OmpResult<"describeDestination"> | null; /** The revision of the model catalog OMP bundles now (`readModelCatalog`); null until it answers. */ @@ -76,14 +79,14 @@ type Session = { /** * The first-use and re-verification flow of `createCodeWorkflowClient().verifyModels`, for the - * workbench. Mounting observes only — OMP's account observation and Code's pool composition, and - * the destination and model catalog the workbench already reads, to compare with the recorded - * verification — and nothing runs until `prepare`. A run is bound to the destination, principal, - * container and write authority it started under; losing any of them stops it, which cancels a - * probe job still spending. + * workbench. Mounting observes only — OMP's account observation and Code's pool composition, the + * destination and model catalog the workbench already reads, and, once that catalog has moved, + * what it publishes for the routed providers, to compare with the recorded verification — and + * nothing runs until `prepare`. A run is bound to the destination, principal, container and write + * authority it started under; losing any of them stops it, which cancels a probe job still spending. */ export function useModelVerification(input: ModelVerificationInput): ModelVerification { - const { host, target, record, revision, configurationCurrent, setup, catalogRevision, writable, available } = input; + const { host, target, record, revision, configurationCurrent, setup, catalogRevision, routed, writable, available } = input; const latest = useRef(input); latest.current = input; // `pool: null` is an observed absence of any selected account; a null `data` is no observation yet. @@ -100,11 +103,19 @@ export function useModelVerification(input: ModelVerificationInput): ModelVerifi return () => { mounted.current = false; active.current?.controller.abort(); }; }, []); const provenance = record?.active?.provenance ?? null; - // The OMP a verification recorded is compared by content: the artifact the destination's inventory - // operation is installed from and the bundled model catalog's revision, so an upgrade of either is - // seen whatever version it reports and without a new Code build. + // A routed model OMP's catalog no longer publishes is refused at launch. The revision the + // verification recorded publishes every model it laddered, so what OMP publishes is read only + // once the catalog has moved past it, and judged only when it answers for that same revision. + const moved = provenance !== null && catalogRevision !== null && catalogRevision !== provenance.catalogRevision; + const providers = useMemo(() => [...new Set((routed ?? []).map(model => model.provider))].sort(), [routed]); + const published = useOmpQuery(host, "readModelCatalog", moved && providers.length > 0 ? { providers } : null); + const unpublished = !moved || !routed?.length ? [] : published.data === null || published.data.revision !== catalogRevision ? null + : unpublishedModels(routed, published.data.models); + // The OMP a verification recorded is compared by content, the artifact the destination's inventory + // operation is installed from and the bundled model catalog's revision, so an update is seen + // whatever version it reports and without a new Code build. const state = verificationState(provenance, { inventoryArtifactSha256: setup === null ? null : inventoryArtifact(setup), catalogRevision, - pool: pool.data === null ? null : pool.data.pool ?? "none" }, run !== null); + unpublished, pool: pool.data === null ? null : pool.data.pool ?? "none" }, run !== null); const ready = operationReady(setup, INVENTORY_OPERATION_ID) && operationReady(setup, BENCHMARK_OPERATION_ID); const canPrepare = run === null && writable && available && configurationCurrent && ready && target !== null && host.containerId !== null; diff --git a/plugins/code/generator/models-sheet.tsx b/plugins/code/generator/models-sheet.tsx index ca44136..b7afa64 100644 --- a/plugins/code/generator/models-sheet.tsx +++ b/plugins/code/generator/models-sheet.tsx @@ -296,7 +296,10 @@ export function ModelsSheet({ host, model, machine, onBack, backRef, onPlace }: if (!verifyGate.open) return verifyRefused(verifyGate.refusal); const stopped = verification.failure; const note: readonly Part[] = stopped ? [[stopped.cancelled ? "verification cancelled" : "verification stopped", stopped.cancelled ? null : "attention"]] : []; - const tail: Part = verification.status === "accounts-changed" ? ["accounts changed", "warn"] : verification.status === "omp-changed" ? ["OMP changed", "warn"] : ["unverified", "warn"]; + // A model OMP dropped is named as the list shows it; several are counted. + const dropped = verification.changes.unpublished, named = inUse.find(row => `${row.provider}/${row.id}` === dropped[0])?.alias ?? dropped[0]; + const tail: Part = verification.status === "accounts-changed" ? ["accounts changed", "warn"] + : verification.status === "unpublished" ? [`OMP dropped ${dropped.length === 1 ? named : count(dropped.length, "model")}`, "warn"] : ["unverified", "warn"]; return { label: "verify models", parts: [...note, ...refused, tail], fixes: [], said: stopped && !stopped.cancelled ? { value: "verify models", text: stopped.reason, warn: true } : { value: "verify models", text: "finds the models your accounts reach and measures them; nothing is spent before you confirm" }, @@ -409,10 +412,12 @@ export function ModelsSheet({ host, model, machine, onBack, backRef, onPlace }: // ------------------------------------------------------------ the sheet const verifiedAt = verification.provenance?.benchmarkCompletedAt ?? null; + // An OMP update the routes survived leaves the list verified; it is said beside the date, since verifying again is the way to measure under it. + const ompUpdated = verification.changes.runtimeChanged || verification.changes.catalogChanged; const [tail, tone]: Part = running ? ["verifying", null] : phase === "staged" ? [active ? "staged list beside it" : "staged, none in use", "warn"] - : verification.status === "current" && verifiedAt !== null ? [`verified ${since(Date.now() - verifiedAt)}`, null] + : verification.status === "current" && verifiedAt !== null ? [`verified ${since(Date.now() - verifiedAt)}${ompUpdated ? " · OMP updated since" : ""}`, null] : verification.status === "accounts-changed" ? ["accounts changed since verified", "warn"] - : verification.status === "omp-changed" ? ["OMP changed since verified", "warn"] + : verification.status === "unpublished" ? [`OMP dropped ${count(verification.changes.unpublished.length, "model")} since verified`, "warn"] : !active ? ["starter · unverified", "warn"] : ["unverified", "warn"]; function cell(family: string, tier: number) { const row = at(family, tier), id = `${family}:${tier}`, tierWord = TIERS[tier - 1]!; diff --git a/plugins/code/generator/sheets-model.ts b/plugins/code/generator/sheets-model.ts index 9378215..e82d5ab 100644 --- a/plugins/code/generator/sheets-model.ts +++ b/plugins/code/generator/sheets-model.ts @@ -109,8 +109,8 @@ export function exclusionWords(exclusion: Exclusion): readonly [string, string] /** * The step the Models sheet is at, which decides its one next action. A run in flight comes first * (`finishing` once its measurements are in), then a staged list waiting beside the one in use, a - * list verified with the present accounts and OMP, and otherwise a list to verify: `refused` when - * verifying needs discovery the machine has not enabled. + * list whose verification holds (verification.ts `current`), and otherwise a list to verify: + * `refused` when verifying needs discovery the machine has not enabled. */ export type ModelsPhase = "inventory" | "charge" | "benchmark" | "finishing" | "staged" | "verified" | "refused" | "unverified"; export function modelsPhase(facts: { diff --git a/plugins/code/generator/verification.ts b/plugins/code/generator/verification.ts index b5ca388..f979fe3 100644 --- a/plugins/code/generator/verification.ts +++ b/plugins/code/generator/verification.ts @@ -5,57 +5,65 @@ import type { VerificationProvenance } from "../contract.ts"; * * `unverified`: no verification is recorded — no active catalog, or one authored, imported or * kept from before verification recorded the OMP it ran against. `verifying`: a verification is - * in flight, including while its charge awaits confirmation. `omp-changed`: the destination's OMP - * now runs from another artifact than the one the catalog's inventory ran from, or OMP bundles - * another model catalog than it did then — an upgrade of the runtime, its SDK or its model list, - * whatever version it reports. `accounts-changed`: the saved choices now select another pool — a + * in flight, including while its charge awaits confirmation. `unpublished`: OMP's model catalog + * moved and no longer publishes a model the profile routes to, which the session door refuses + * (`code_model_unpublished`). `accounts-changed`: the saved choices now select another pool — a * provider added or removed, or the same providers through different accounts. `current`: none * of those. + * + * An OMP update alone leaves a verification current, noted in `changes`: the destination's + * inventory installed from another artifact, or another catalog revision that still publishes + * every routed model. Neither shows that the models these accounts reached have moved, and an + * update would otherwise ask every workspace to spend a verification before its next launch. The + * session door still re-checks every routed model at launch, and Verify renews a current + * verification on purpose. */ -export type VerificationStatus = "unverified" | "verifying" | "current" | "accounts-changed" | "omp-changed"; +export type VerificationStatus = "unverified" | "verifying" | "current" | "accounts-changed" | "unpublished"; /** The present a recorded verification is compared with. A null field has not been observed. */ export type VerificationObservation = { /** The artifact the destination's OMP inventory operation is installed from (workflow.ts `inventoryArtifact`). */ inventoryArtifactSha256: string | null; /** The revision of the model catalog OMP bundles (`readModelCatalog`). */ catalogRevision: string | null; + /** + * The routed models OMP's catalog no longer publishes (domain/routing.ts `unpublishedModels`), + * read only once the catalog has moved: the revision the verification recorded publishes every + * model it laddered. + */ + unpublished: readonly string[] | null; /** The pool the saved choices select now, or `"none"` when they select no account at all. */ pool: { providers: readonly string[]; poolIdentityDigest: string } | "none" | null; }; -/** What differs from the recorded verification, for saying why it no longer holds. */ +/** What differs from the recorded verification: why it no longer holds, or the OMP update it outlived. */ export type VerificationChanges = { /** The destination's OMP runs from another artifact than the inventory did. */ runtimeChanged: boolean; /** OMP bundles another model catalog than it did when the inventory ran. */ catalogChanged: boolean; + /** The routed models, as `provider/id`, that the moved catalog no longer publishes. */ + unpublished: string[]; providersAdded: string[]; providersRemoved: string[]; /** The same providers are covered, by a different exact set of accounts. */ identitiesChanged: boolean; }; -export type VerificationState = { - status: VerificationStatus; - changes: VerificationChanges; - /** - * Every fact the comparison needs has been observed. While one is missing — not yet answered, - * or failed — `current` rests on the record alone: an observation that is slow or failing is - * not evidence that anything changed, its failure is named where that read's failures are, and - * the launch itself re-checks accounts and published models at its owner. - */ - observed: boolean; -}; +export type VerificationState = { status: VerificationStatus; changes: VerificationChanges }; /** - * Compares a recorded verification with the present. An OMP change outranks an account change, - * because a different model list invalidates every provider's answers, not only one pool's. + * Compares a recorded verification with the present. A routed model OMP dropped outranks an account + * change, because the launch fails on it whichever accounts serve it. A fact not yet observed, or + * whose read failed, claims no change, since a slow or failing read is not evidence that anything + * moved. The destination's and the bundled list's failures are named where those reads' failures + * are; the read of what a moved catalog publishes names none and fails open to the session door, + * which re-checks accounts and every routed model at launch. */ export function verificationState(provenance: VerificationProvenance | null, observation: VerificationObservation, verifying: boolean): VerificationState { - const changes: VerificationChanges = { runtimeChanged: false, catalogChanged: false, providersAdded: [], providersRemoved: [], identitiesChanged: false }; - const observed = observation.inventoryArtifactSha256 !== null && observation.catalogRevision !== null && observation.pool !== null; - if (verifying) return { status: "verifying", changes, observed }; - if (provenance === null) return { status: "unverified", changes, observed }; + const changes: VerificationChanges = { runtimeChanged: false, catalogChanged: false, unpublished: [], providersAdded: [], providersRemoved: [], identitiesChanged: false }; + if (verifying) return { status: "verifying", changes }; + if (provenance === null) return { status: "unverified", changes }; changes.runtimeChanged = observation.inventoryArtifactSha256 !== null && observation.inventoryArtifactSha256 !== provenance.inventoryArtifactSha256; changes.catalogChanged = observation.catalogRevision !== null && observation.catalogRevision !== provenance.catalogRevision; + if (changes.catalogChanged) changes.unpublished = [...(observation.unpublished ?? [])]; if (observation.pool !== null) { const providers = observation.pool === "none" ? [] : observation.pool.providers; changes.providersAdded = providers.filter(provider => !provenance.providers.includes(provider)); @@ -64,7 +72,7 @@ export function verificationState(provenance: VerificationProvenance | null, obs observation.pool.poolIdentityDigest !== provenance.poolIdentityDigest; } const accountsChanged = changes.providersAdded.length > 0 || changes.providersRemoved.length > 0 || changes.identitiesChanged; - return { status: changes.runtimeChanged || changes.catalogChanged ? "omp-changed" : accountsChanged ? "accounts-changed" : "current", changes, observed }; + return { status: changes.unpublished.length > 0 ? "unpublished" : accountsChanged ? "accounts-changed" : "current", changes }; } /** diff --git a/plugins/code/generator/workbench-model.ts b/plugins/code/generator/workbench-model.ts index 228a1cc..87bd08e 100644 --- a/plugins/code/generator/workbench-model.ts +++ b/plugins/code/generator/workbench-model.ts @@ -3,7 +3,7 @@ import type { HostServices } from "@manifold/plugin"; import { formatManifoldUri, type MachineSummary } from "@manifold/protocol"; import { compileCatalog, type CompiledCatalog } from "../../domain/catalog.ts"; import { servedProviders } from "../../domain/accounts.ts"; -import { defaultSelection, routeService, type Review, type ServiceGap } from "../../domain/routing.ts"; +import { defaultSelection, routeService, routedModels, type Review, type ServiceGap } from "../../domain/routing.ts"; import { catalogFromMetadata } from "../../domain/probe.ts"; import type { AccountChoiceChange, CatalogDocument, Selection } from "../../domain/contracts.ts"; import type { ActionResult, Configuration, Target } from "../contract.ts"; @@ -331,6 +331,10 @@ export function useWorkbench({ host, target, machine, rosterError, available }: if (!compiled || !selection) return null; return previewSelection(compiled, selection); }, [compiled, selection]); + // The models the shown team's routes name, which the session door requires OMP to still publish. The door first prunes + // fallbacks no included account serves; these are not pruned, because a provider of a verified list that no account + // serves now has changed the pool, which asks to verify again whatever OMP publishes. + const routed = useMemo(() => compiled && localReview ? routedModels(compiled, localReview.routes) : null, [compiled, localReview]); const controlsReview = useMemo(() => { if (localReview || !compiled || !selection || !initialSelection) return localReview; return previewSelection(compiled, { ...initialSelection, budget: selection.budget }); @@ -365,7 +369,7 @@ export function useWorkbench({ host, target, machine, rosterError, available }: // A confirmed verification is itself the save of the shown selection: the bundled preview's // dials, or the active profile's, narrowed to what the verified catalog hosts. const verification = useModelVerification({ host, target, record, revision: profile?.revision ?? observed?.revision ?? 0, - configurationCurrent, selection, setup: setup.data, catalogRevision: metadata.data?.revision ?? null, writable, available, + configurationCurrent, selection, routed, setup: setup.data, catalogRevision: metadata.data?.revision ?? null, writable, available, onVerified: saved => { setSavedPolicy(saved); setDials(null); setPreview(null); setMessage({ text: "Models verified with your accounts, and the profile saved for the workspace.", failed: false }); diff --git a/plugins/code/session.ts b/plugins/code/session.ts index 17ea02e..8863aa0 100644 --- a/plugins/code/session.ts +++ b/plugins/code/session.ts @@ -15,7 +15,7 @@ import { z } from "zod"; import { selectedAccountPool } from "../domain/accounts.ts"; import { compileCatalog } from "../domain/catalog.ts"; import { DomainError, type AccountChoices, type CatalogDocument, type Selection } from "../domain/contracts.ts"; -import { compileOmpOverlay, reviewCatalog, servedRoutes } from "../domain/routing.ts"; +import { compileOmpOverlay, reviewCatalog, routedModels, servedRoutes, unpublishedModels } from "../domain/routing.ts"; import { digest, id, revision, reviewedSessionOptions, sessionInput, type ActionInput, type ActionResult, type Profile, type SessionComposition } from "./contract.ts"; import { CodeRefusal, digestOf, type CodeContext } from "./context.ts"; @@ -65,11 +65,9 @@ export async function composeSession(ctx: CodeContext, args: ActionInput<"compos const catalog = compileCatalog(record.active.document); const reviewed = reviewCatalog(catalog, record.selection, receivedAt); const review = { ...reviewed, routes: servedRoutes(catalog, reviewed.routes, serves) }; - const routed = [...new Set(review.routes.flatMap(route => [route.lead, ...route.fallback]).map(choice => choice.key))] - .map(key => catalog.model(key)); + const routed = routedModels(catalog, review.routes); const published = await ompCall(ctx, "readModelCatalog", { providers: [...new Set(routed.map(model => model.provider))].sort() }); - const listed = new Set(published.models.map(model => `${model.provider}/${model.id}`)); - const missing = routed.map(model => `${model.provider}/${model.id}`).filter(address => !listed.has(address)).sort(); + const missing = unpublishedModels(routed, published.models); if (missing.length > 0) throw new CodeRefusal("model_unpublished", missing.slice(0, 8).join(", ") + (missing.length > 8 ? ` and ${missing.length - 8} more` : "")); const overlay = compileOmpOverlay(catalog, review.selection, review.routes, serves); diff --git a/plugins/code/workflow.ts b/plugins/code/workflow.ts index d2ba49f..48d9fc3 100644 --- a/plugins/code/workflow.ts +++ b/plugins/code/workflow.ts @@ -28,6 +28,7 @@ const messages: Readonly> = { code_account_unavailable: "The selected accounts are unavailable or no longer resolve exactly. Review the account choices.", code_accounts_changed: "The accounts changed. Read them again before continuing.", code_omp_changed: "OMP or its model list changed on the machine since the inventory. Verify again.", + code_model_unpublished: "OMP no longer lists a model this profile uses. Verify models again.", code_invalid_accounts: "The account observation could not be used. Read the accounts again.", code_invalid_selection: "This profile cannot be formed from the current models.", code_budget_unsatisfiable: "No free route serves this profile.", @@ -85,6 +86,10 @@ const agentRefusals: Readonly> = { "run launch already in progress": "That run is already launching.", "harness launch target changed": "The run was created for another machine or workspace.", }; +/** Refusals whose detail is a list of identifiers, with the label that introduces it; the list stays exactly as written. */ +const listedDetails: Readonly> = { + code_model_unpublished: "Not listed:", +}; /** How a browser's dispatch reports a refused door (machine-web.ts `codeWorkflow`): `.: . No approval or readiness is assumed.` */ const DOOR_DENIAL = /^[a-z0-9_-]+(?:\.[A-Za-z0-9_-]+)+: ([^]*?)(?:\. No approval or readiness is assumed\.)?$/; /** A refusal token (`code_account_unavailable`), and the detail for a person that may follow it. */ @@ -98,7 +103,8 @@ export function refusalToken(message: string): string | null { * A workflow failure in a person's words. A refused door's message loses the door name it was * reported under, and its refusal token becomes words: the known ones from `messages` and Manifold's * Agent refusals, any other as who refused, so no raw token reaches a person. A detail written for a - * person after the token stays. + * person after the token stays, as a sentence; a list of identifiers (`listedDetails`) stays exactly + * as written after its label. */ export function failureWords(message: string): string { const denial = DOOR_DENIAL.exec(message)?.[1] ?? message; @@ -107,7 +113,8 @@ export function failureWords(message: string): string { if (!refusal) return denial; const [, token, detail] = refusal; const words = messages[token!] ?? (token!.startsWith("omp_") ? "OMP refused the step." : "Code refused the step."); - return detail ? `${words} ${detail.charAt(0).toUpperCase()}${detail.slice(1)}` : words; + if (!detail) return words; + return Object.hasOwn(listedDetails, token!) ? `${words} ${listedDetails[token!]} ${detail}` : `${words} ${detail.charAt(0).toUpperCase()}${detail.slice(1)}`; } /** The steps of a model verification, in the order they run; a failure names the one it stopped at. */ diff --git a/plugins/domain/catalog-routing.test.ts b/plugins/domain/catalog-routing.test.ts index 539273c..61fb5eb 100644 --- a/plugins/domain/catalog-routing.test.ts +++ b/plugins/domain/catalog-routing.test.ts @@ -2,7 +2,7 @@ import { describe, expect, test } from "bun:test"; import { compileCatalog } from "./catalog.ts"; import { type CatalogDocument, type CatalogModel, type Route, type Selection } from "./contracts.ts"; import { familyPolicy, providerPolicy } from "./providers.ts"; -import { clampSelection, compileOmpOverlay, defaultSelection, reviewCatalog, ReviewSchema, routeService, servedRoutes } from "./routing.ts"; +import { clampSelection, compileOmpOverlay, defaultSelection, reviewCatalog, ReviewSchema, routedModels, routeService, servedRoutes, unpublishedModels } from "./routing.ts"; function model(key: string, provider: string, tier: CatalogModel["tier"], changes: Partial = {}): CatalogModel { return { @@ -446,4 +446,17 @@ describe("a verified catalog and the accounts a launch meets", () => { roles: review.routes.filter(entry => catalog.model(entry.lead.key).provider === "openai-codex").map(entry => entry.role) }); expect(() => servedRoutes(catalog, review.routes, provider => provider !== "openai-codex")).toThrow("code_account_unavailable"); }); + + test("a published list that lacks a routed model, lead or fallback, names it once, by provider and id together", () => { + const catalog = compileCatalog(document()); + const review = reviewCatalog(catalog, selection({ capability: 1, advisor: "audit" }), daytime); + const routed = routedModels(catalog, review.routes); + expect(unpublishedModels(routed, document().models)).toEqual([]); + // The audit advisor falls back to a2 and then a1; a2 dropped from what OMP publishes is named once. + expect(route(review.routes, "advisor").fallback.map(choice => choice.key)).toEqual(["a2", "a1"]); + expect(unpublishedModels(routed, document().models.filter(entry => entry.key !== "a2"))).toEqual(["anthropic/native-a2"]); + // The same id under another provider is another model, and several are named in order. + const opus = { provider: "anthropic", id: "claude-opus" }, terra = { provider: "openai-codex", id: "gpt-terra" }; + expect(unpublishedModels([terra, opus, terra], [{ provider: "openrouter", id: "claude-opus" }])).toEqual(["anthropic/claude-opus", "openai-codex/gpt-terra"]); + }); }); diff --git a/plugins/domain/routing.ts b/plugins/domain/routing.ts index 4200aba..bf35f25 100644 --- a/plugins/domain/routing.ts +++ b/plugins/domain/routing.ts @@ -325,6 +325,23 @@ export function routeService(catalog: CompiledCatalog, routes: readonly Route[], return { lead: first ? gap(...first) : null, pruned: [...pruned].map(([provider, roles]) => gap(provider, roles)) }; } +/** A model as OMP's catalog and a compiled catalog both name it. */ +export type ModelAddress = { readonly provider: string; readonly id: string }; + +/** The models routes name, leads and fallbacks, each once, in route order. */ +export function routedModels(catalog: CompiledCatalog, routes: readonly Route[]): ModelAddress[] { + return [...new Set(routes.flatMap(route => [route.lead, ...route.fallback]).map(choice => choice.key))].map(key => catalog.model(key)); +} + +/** + * The routed models a published catalog lacks, each once as `provider/id` and sorted. The session + * door refuses a composition that names any (`model_unpublished`), so a client says it beforehand. + */ +export function unpublishedModels(routed: readonly ModelAddress[], published: readonly ModelAddress[]): string[] { + const listed = new Set(published.map(model => `${model.provider}/${model.id}`)); + return [...new Set(routed.map(model => `${model.provider}/${model.id}`))].filter(address => !listed.has(address)).sort(); +} + /** * Encode only a complete, exact selection; arbitrary or stale route lists are not overlays. The * routes must be exactly the selection's, less precisely the fallbacks `serves` rules out. diff --git a/plugins/scripts/verify-browser.ts b/plugins/scripts/verify-browser.ts index c75a2de..d854b45 100755 --- a/plugins/scripts/verify-browser.ts +++ b/plugins/scripts/verify-browser.ts @@ -64,8 +64,9 @@ focus kept on it, kept across a lead change and disabled under Mixed, with struc that say why, and Models' pasted imports, discards and the exact review a staged list is used by, through the real Code server. Separate synthetic RPC responses exercise the verification charge and its refused spend; a verified -catalog made stale by an OMP upgrade alone; a Models verification run end to end on fictional probe -jobs that reach no provider, whose derivation, staging and promotion are the real server's; Setup's +catalog that outlives an OMP upgrade alone and goes stale once a republished list drops a routed +model; a Models verification run end to end on fictional probe jobs that reach no provider, whose +derivation, staging and promotion are the real server's; Setup's readiness rows, folder jobs and the owner's classifier (an owner-class identity minted for it alone); and launch/resume review invalidation and refusal. The preparation, configuration and execution they lead to are refused, never native execution or consent success. @@ -2304,10 +2305,12 @@ async function verificationChargeScenario(browser: BrowserInstance, server: Test } /** - * A verified catalog holds for the OMP it ran under. Recorded as the workflow records it — the pinned OMP's artifact and - * the real bundled catalog's revision — on a destination whose synthetic readiness pins that artifact, the verification is - * current and the launch is the step. The same destination upgraded alone, with the same accounts and the same Code - * bundles, makes Verify the step again, ready to run. Nothing is probed, spent, saved or installed. + * A verified catalog outlives an OMP update its routed models survive. Recorded as the workflow records it — the pinned + * OMP's artifact and the real bundled catalog's revision — on a destination whose synthetic readiness pins that artifact, + * the verification is current and the launch is the step. The same destination upgraded alone, with the same accounts, + * catalog and Code bundles, keeps the launch the step, and Models says OMP was updated since. A model list then + * republished under another revision without the default lead makes Verify the step, ready to run, and Models names the + * model OMP dropped. Nothing is probed, spent, saved or installed. */ async function ompUpgradeScenario(browser: BrowserInstance, server: TestServer, writer: TokenGrant, destination: Target): Promise { const workspace = await createContainer(server, "OMP upgrade", "canvas"); @@ -2321,8 +2324,9 @@ async function ompUpgradeScenario(browser: BrowserInstance, server: TestServer, const probe = await callAction(server, writer.token, "atyrode.code.composeProbe", { containerId: workspace.id, expectedRevision: revision, accounts: observation }); assert(probe.ok); let runtime = pinnedOmpArtifact(); + const document = catalogFromMetadata(metadata, "any"); const staged = await callAction(server, writer.token, "atyrode.code.stageCatalog", { containerId: workspace.id, expectedRevision: revision, - document: catalogFromMetadata(metadata, "any"), verification: { ompVersion: OMP_VERSION, inventoryArtifactSha256: runtime, catalogRevision: metadata.revision, + document, verification: { ompVersion: OMP_VERSION, inventoryArtifactSha256: runtime, catalogRevision: metadata.revision, inventoryObservedAt: Date.now() - 3_000, benchmarkCompletedAt: Date.now() - 2_000, accounts: observation, poolIdentityDigest: (probe.result as ActionResult<"composeProbe">).poolIdentityDigest } }); assert(staged.ok, "A verification records the pinned OMP's artifact and the real catalog revision"); @@ -2332,10 +2336,17 @@ async function ompUpgradeScenario(browser: BrowserInstance, server: TestServer, const promoted = await callAction(server, writer.token, "atyrode.code.promoteCatalog", { ...reviewInput, reviewDigest: (reviewed.result as ActionResult<"reviewCatalog">).reviewDigest }); assert(promoted.ok); const saved = await readConfiguration(server, writer, target); + assert(saved.configuration?.selection, "Promotion puts the list in use with its default team"); + // The default lead as the saved team routes it, and the alias the list shows it by. + const catalog = compileCatalog(document); + const leadKey = reviewCatalog(catalog, saved.configuration.selection, Date.now()).routes.find(route => route.role === "default")!.lead.key; + const lead = catalog.model(leadKey), alias = displayAliases(catalog).get(leadKey) ?? leadKey; await arrangeWorkbench(server, writer); let described = 0; + let republished: OmpResult<"readModelCatalog"> | null = null; const fixture = await intercept(browser, server, (name, input) => { if (name === "atyrode.omp.accounts.accounts") return { ok: true, result: fixtureAccounts([1]) }; + if (name === "atyrode.omp.readModelCatalog" && republished) return { ok: true, result: republished }; if (name !== "atyrode.omp.describeDestination" || input.containerId !== target.containerId || input.machineId !== target.machineId) return undefined; described++; const pins = { installationRevision: "synthetic-ui-only", artifactSha256: runtime, resourceBindingDigest: "c".repeat(64) }; @@ -2349,11 +2360,24 @@ async function ompUpgradeScenario(browser: BrowserInstance, server: TestServer, await Bun.sleep(500); // Discovery is ready, so a stale verification would make Verify the step; the launch this machine has no permission for is. await until(browser, "on the OMP it ran under, the verification is current and the next step is the launch", launchIs("launch", "refused")); - // OMP upgraded on the machine, and nothing else: its runtime is another artifact, whatever version it reports. + // OMP upgraded on the machine, and nothing else: its runtime is another artifact, whatever version it reports. Models + // says so once the panel has read it, and the launch stays the step. runtime = "9".repeat(64); await openGenerator(browser, server, workspace.id); - await until(browser, "an OMP upgrade alone makes Verify the step again, ready to run", launchIs("verify models", "ready")); - assert.deepEqual(await readConfiguration(server, writer, target), saved, "Observing an upgraded OMP saves nothing"); + await openFromMore(browser, "models"); + await until(browser, "Models keeps the verification and says OMP was updated since", + `${sheetState(modelsSheet)}.endsWith(' · OMP updated since') && ${goIs(modelsSheet, "back to code")}`); + await closeSheet(browser); + await until(browser, "an OMP upgrade alone keeps the launch the next step", launchIs("launch", "refused")); + // The model list republished under another revision without the default lead: the launch would be refused for it. + republished = { ...metadata, revision: "f".repeat(64), models: metadata.models.filter(model => model.provider !== lead.provider || model.id !== lead.id) }; + await openGenerator(browser, server, workspace.id); + await until(browser, "a model list that dropped a routed model makes Verify the step again, ready to run", launchIs("verify models", "ready")); + await openFromMore(browser, "models"); + await until(browser, "Models names the model OMP dropped beside verify models", + `${goIs(modelsSheet, "verify models")} && JSON.stringify(${goParts(modelsSheet)}) === ${JSON.stringify(JSON.stringify([[`OMP dropped ${alias}`, "warn"]]))}`); + await closeSheet(browser); + assert.deepEqual(await readConfiguration(server, writer, target), saved, "Observing an updated OMP saves nothing"); fixture.check(); } finally { await fixture.stop(); @@ -5426,7 +5450,7 @@ async function run(): Promise { await starterObservationScenario(writerBrowser, server, writer); phase = "verification charge gate"; await verificationChargeScenario(writerBrowser, server, writer, target); - phase = "an OMP upgrade alone makes a verified catalog stale"; + phase = "a verified catalog outlives an OMP upgrade alone and asks for another once a routed model is dropped"; await ompUpgradeScenario(writerBrowser, server, writer, target); phase = "Models: verify, the charge's hold and deliberate confirm, measuring, left out, staged, imported, discarded and verified again"; await modelsSheetScenario(writerBrowser, server, writer, target); diff --git a/plugins/test/action-gate.test.ts b/plugins/test/action-gate.test.ts index 2fe75fa..97d818a 100644 --- a/plugins/test/action-gate.test.ts +++ b/plugins/test/action-gate.test.ts @@ -54,7 +54,7 @@ describe("one gate for every action", () => { expect(code(actionGate(facts({ unsaved: true, localDraft: { source: "active" } }), "save"))).toBe("open"); // The bundled preview is saved only by the verification that replaces it, and nothing saves unverified. expect(code(actionGate(facts({ unsaved: true, profile: { source: "starter" }, localDraft: { source: "starter" } }), "save"))).toBe("unsaved"); - expect(code(actionGate(facts({ unsaved: true, localDraft: { source: "active" } }, "omp-changed"), "save"))).toBe("not-next"); + expect(code(actionGate(facts({ unsaved: true, localDraft: { source: "active" } }, "unpublished"), "save"))).toBe("not-next"); expect(code(actionGate(facts({ unsaved: true, localDraft: { source: "active" }, writable: false }), "save"))).toBe("read-only"); }); diff --git a/plugins/test/verification.test.ts b/plugins/test/verification.test.ts index 447dc9a..85125ee 100644 --- a/plugins/test/verification.test.ts +++ b/plugins/test/verification.test.ts @@ -13,28 +13,36 @@ const runtime = "c".repeat(64), catalog = "e".repeat(64); const provenance: VerificationProvenance = { ompVersion: OMP_VERSION, inventoryArtifactSha256: runtime, catalogRevision: catalog, inventoryObservedAt: 1, benchmarkCompletedAt: 2, providers: ["anthropic", "openai-codex"], poolIdentityDigest: "a".repeat(64) }; const same = { providers: ["anthropic", "openai-codex"], poolIdentityDigest: "a".repeat(64) }; -const present = { inventoryArtifactSha256: runtime, catalogRevision: catalog }; +const present = { inventoryArtifactSha256: runtime, catalogRevision: catalog, unpublished: [] }; +const moved = "f".repeat(64); describe("whether a recorded verification still holds", () => { - test("the present OMP runtime, model catalog and pool decide, and an OMP change outranks an account change", () => { - expect(verificationState(provenance, { ...present, pool: same }, false)).toMatchObject({ status: "current", observed: true }); + test("the present pool and the routed models OMP publishes decide, and a dropped model outranks an account change", () => { + expect(verificationState(provenance, { ...present, pool: same }, false).status).toBe("current"); expect(verificationState(null, { ...present, pool: same }, false).status).toBe("unverified"); expect(verificationState(provenance, { ...present, pool: same }, true).status).toBe("verifying"); - const upgraded = verificationState(provenance, { ...present, inventoryArtifactSha256: "9".repeat(64), pool: { ...same, poolIdentityDigest: "b".repeat(64) } }, false); - expect([upgraded.status, upgraded.changes.runtimeChanged, upgraded.changes.identitiesChanged]).toEqual(["omp-changed", true, true]); + const both = verificationState(provenance, { ...present, catalogRevision: moved, unpublished: ["openai-codex/terra"], pool: { ...same, poolIdentityDigest: "b".repeat(64) } }, false); + expect([both.status, both.changes.unpublished, both.changes.identitiesChanged]).toEqual(["unpublished", ["openai-codex/terra"], true]); }); - test("an OMP-only upgrade with the same accounts makes the verification stale, and Verify is the next step", () => { - // The runtime is upgraded on the destination; Code is the same build, so its compiled OMP version has not moved. + test("an OMP update the routed models survive keeps the verification current, so the launch goes on without verifying", () => { + // The runtime is upgraded on the destination with the same model catalog and the same accounts. const upgraded = verificationState(provenance, { ...present, inventoryArtifactSha256: "9".repeat(64), pool: same }, false); - expect(upgraded).toMatchObject({ status: "omp-changed", observed: true, changes: { runtimeChanged: true, catalogChanged: false, identitiesChanged: false } }); - expect(nextLaunchStep(facts({}, upgraded.status)).step).toBe("verify"); - expect(launchStatusText(facts({}, upgraded.status))).toBe("OMP changed since models were verified. Verify again."); + expect(upgraded).toMatchObject({ status: "current", changes: { runtimeChanged: true, catalogChanged: false, unpublished: [] } }); + expect(nextLaunchStep(facts({}, upgraded.status)).step).toBe("review"); + expect(nextLaunchStep(facts({ previewCurrent: true }, upgraded.status)).step).toBe("launch"); + // A model catalog republished under another revision that still lists every routed model. + const republished = verificationState(provenance, { ...present, catalogRevision: moved, pool: same }, false); + expect(republished).toMatchObject({ status: "current", changes: { runtimeChanged: false, catalogChanged: true } }); }); - test("a new model catalog revision makes the verification stale, whatever version OMP reports", () => { - const republished = verificationState(provenance, { ...present, catalogRevision: "f".repeat(64), pool: same }, false); - expect(republished).toMatchObject({ status: "omp-changed", changes: { runtimeChanged: false, catalogChanged: true } }); + test("a moved catalog that dropped a routed model asks to verify again, and the recorded one never reads as dropping any", () => { + const dropped = verificationState(provenance, { ...present, catalogRevision: moved, unpublished: ["openai-codex/terra"], pool: same }, false); + expect(dropped).toMatchObject({ status: "unpublished", changes: { catalogChanged: true, unpublished: ["openai-codex/terra"] } }); + expect(nextLaunchStep(facts({}, dropped.status)).step).toBe("verify"); + expect(launchStatusText(facts({}, dropped.status))).toBe("OMP no longer lists a model this profile uses. Verify again."); + // The revision the verification recorded publishes every model it laddered, so a list read against it claims nothing. + expect(verificationState(provenance, { ...present, unpublished: ["openai-codex/terra"], pool: same }, false).status).toBe("current"); }); test("a provider added or removed, the same providers through other accounts, or no account at all is an account change", () => { @@ -47,10 +55,12 @@ describe("whether a recorded verification still holds", () => { expect(verificationState(provenance, { ...present, pool: "none" }, false).changes.providersRemoved).toEqual(["anthropic", "openai-codex"]); }); - test("an observation not yet answered or failed never claims a change, and says the comparison is incomplete", () => { - expect(verificationState(provenance, { inventoryArtifactSha256: null, catalogRevision: null, pool: null }, false)).toMatchObject({ status: "current", observed: false }); - expect(verificationState(provenance, { ...present, catalogRevision: null, pool: same }, false)).toMatchObject({ status: "current", observed: false }); - expect(verificationState(provenance, { ...present, inventoryArtifactSha256: "9".repeat(64), pool: null }, false)).toMatchObject({ status: "omp-changed", observed: false }); + test("an observation not yet answered or failed never claims a change", () => { + expect(verificationState(provenance, { inventoryArtifactSha256: null, catalogRevision: null, unpublished: null, pool: null }, false).status).toBe("current"); + // A moved catalog whose published list is still being read claims no dropped model. + expect(verificationState(provenance, { ...present, catalogRevision: moved, unpublished: null, pool: same }, false)) + .toMatchObject({ status: "current", changes: { catalogChanged: true, unpublished: [] } }); + expect(verificationState(provenance, { ...present, pool: null }, false).changes).toMatchObject({ providersAdded: [], providersRemoved: [], identitiesChanged: false }); }); test("while the destination has not answered, the launch is not ready, so a gap in the observation never opens one", () => { @@ -69,7 +79,7 @@ function facts(changes: Partial = {}, status: VerificationStatus = describe("the next launch step with verification", () => { test("verify precedes save and review whenever the verification is not current", () => { - for (const status of ["unverified", "accounts-changed", "omp-changed"] as const) { + for (const status of ["unverified", "accounts-changed", "unpublished"] as const) { expect(nextLaunchStep(facts({}, status)).step).toBe("verify"); // The bundled preview and an unsaved edit both verify before anything saves. expect(nextLaunchStep(facts({ unsaved: true, profile: { source: "starter" }, localDraft: { source: "starter" }, record: null }, status)).step).toBe("verify"); @@ -92,7 +102,7 @@ describe("the next launch step with verification", () => { expect(nextLaunchStep(facts({ queries: { setup: { error: "unreadable" } } }, "unverified"))).toMatchObject({ reason: { code: "verify-status", action: "refresh" } }); expect(nextLaunchStep(facts({}, "unverified", false))).toMatchObject({ reason: { code: "verify-permissions", action: "permissions" } }); // Launch-only preconditions never block a verification: it needs no launch permission, skills or local review. - expect(nextLaunchStep(facts({ launchReady: false, skillProblems: ["x"], localReview: null }, "omp-changed")).step).toBe("verify"); + expect(nextLaunchStep(facts({ launchReady: false, skillProblems: ["x"], localReview: null }, "unpublished")).step).toBe("verify"); // Once current, the launch order is unchanged. expect(nextLaunchStep(facts({ launchReady: false })).reason?.code).toBe("permissions"); }); diff --git a/plugins/test/workflow.test.ts b/plugins/test/workflow.test.ts index 5fff592..26e7def 100644 --- a/plugins/test/workflow.test.ts +++ b/plugins/test/workflow.test.ts @@ -7,7 +7,7 @@ import { PublicJobSchema, type PublicJob, type TerminalSummary } from "@manifold import { actionSchemas, type ActionResult, type Configuration } from "../code/contract.ts"; import { compileCatalog } from "../domain/catalog.ts"; import type { Selection } from "../domain/contracts.ts"; -import { compileOmpOverlay, defaultSelection, reviewCatalog } from "../domain/routing.ts"; +import { compileOmpOverlay, defaultSelection, reviewCatalog, routedModels, unpublishedModels } from "../domain/routing.ts"; import { handlers } from "../code/server.ts"; import type { CodeContext } from "../code/context.ts"; import { z } from "zod"; @@ -224,25 +224,33 @@ test("a model list republished while the inventory runs stops the verification b expect((await f.configuration()).configuration).toMatchObject({ revision: 1, active: null, draft: null }); }); -test("a verification holds for the OMP runtime and model list it ran against, and an upgrade of either alone makes it stale", async () => { +test("a verification outlives an OMP update its routed models survive, and a model list that dropped one asks for another", async () => { const f = verificationFixture(); + // OMP publishes the models its inventory listed, as its bundled catalog does. + f.omp.catalog = { ...f.omp.catalog, models: f.omp.inventory.models }; const { configuration } = await (await f.workflow.verifyModels(target, { expectedRevision: 0, budget: "any" })).confirm(preview); - // What the workbench compares the saved verification with: the destination's pins, the bundled list's revision and the pool. + const catalog = compileCatalog(configuration.active!.document); + const routed = routedModels(catalog, reviewCatalog(catalog, configuration.selection!, 0).routes); + // What the workbench compares the saved verification with: the destination's pins, the bundled list's revision, what that + // list publishes for the routed providers, and the pool. const present = async () => verificationState(configuration.active!.provenance, { inventoryArtifactSha256: inventoryArtifact(await f.workflow.omp("describeDestination", target)), catalogRevision: (await f.workflow.omp("readModelCatalog", { providers: [] })).revision, + unpublished: unpublishedModels(routed, (await f.workflow.omp("readModelCatalog", { providers: [...new Set(routed.map(model => model.provider))] })).models), pool: await f.workflow.observeVerification(target.containerId, configuration.revision, configuration.accounts) ?? "none" }, false); // The command line answered the inventory as `OMP_VERSION` while the bundled list reports another version: one present OMP, // so a fresh verification is current. expect([configuration.active?.provenance?.ompVersion, f.omp.catalog.ompVersion]).toEqual([OMP_VERSION, "18.7.0"]); - expect(await present()).toMatchObject({ status: "current", observed: true }); - // OMP upgraded on the machine, with the same accounts and the same Code build: only the runtime artifact moved. + expect(await present()).toMatchObject({ status: "current" }); + // OMP upgraded on the machine, with the same accounts, model list and Code build: only the runtime artifact moved. f.omp.artifact = "9".repeat(64); - expect(await present()).toMatchObject({ status: "omp-changed", changes: { runtimeChanged: true, catalogChanged: false, identitiesChanged: false } }); - // The model list published under another revision, on the same runtime. - f.omp.artifact = "c".repeat(64); + expect(await present()).toMatchObject({ status: "current", changes: { runtimeChanged: true, catalogChanged: false, identitiesChanged: false } }); + // The model list republished under another revision that still lists every routed model. f.omp.catalog = { ...f.omp.catalog, revision: "b".repeat(64) }; - expect(await present()).toMatchObject({ status: "omp-changed", changes: { runtimeChanged: false, catalogChanged: true } }); + expect(await present()).toMatchObject({ status: "current", changes: { catalogChanged: true, unpublished: [] } }); + // Republished again without Claude Opus, which the saved team routes. + f.omp.catalog = { ...f.omp.catalog, revision: "d".repeat(64), models: f.omp.catalog.models.filter(model => model.id !== "claude-opus-5") }; + expect(await present()).toMatchObject({ status: "unpublished", changes: { catalogChanged: true, unpublished: ["anthropic/claude-opus-5"] } }); }); test("a stop cancels the probe job in flight, and a failed one names its step and leaves its work inspectable", async () => {