diff --git a/AGENTS.md b/AGENTS.md index 0ad718e..7699f71 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ How coding agents should work in this repo. The README covers usage; this file c 8. **One self-contained bin, zero runtime deps.** `dist/main.cjs` bundles everything (citty, picocolors are devDependencies); the library entry (`dist/index.*`) shares the client code and also has no runtime deps. Adding a runtime dependency is a design decision, not a convenience. 9. **`ax journey` namespace is reserved.** Journey-simulation commands beyond the current one land as `ax deep-journey`; do not add subcommands under `journey`. 10. **`process.exitCode`, never `process.exit()`, after writing to stdout.** A large `--json` payload exceeds the 64KB pipe buffer and `process.exit()` truncates it mid-flush. -11. **The WebMCP capture is a copy of ora's engine, not a reimplementation.** `ax webmcp-audit` measures a page here and ora scores it there, and the same site has to get the same answer from both - so everything that decides a value on the wire is copied verbatim into `src/webmcp/vendor/` (the shim, the verdict ladder, the static signals, the check registry, the challenge predicates). Biome does not format that directory, and every file there carries a header saying so. `src/webmcp/{cdp,capture,chrome,ingest,report}.ts` is plumbing only - transport, budgets, error mapping, rendering. A judgement written by hand outside `vendor/` is the drift this arrangement exists to prevent; `deriveIframeAllow`'s tests are ora's own cases, unmodified, for exactly that reason. The version check is NOT sufficient on its own - the engine has changed this code without changing the version - so re-copy the whole directory whenever the engine moves. Four deliberate deviations, all marked inline: `collectStaticSignals` takes its inline JS as a parameter and the two cheerio-only helpers beside it are dropped (ora uses an HTML parser there, which would be a runtime dependency), `projection.ts` carries only the `PublicWebmcpAudit` interface (the projector itself needs zod and ora's scoring module), the capture covers the entry page only, and the engine's comments have their source paths rewritten to point at this directory. `capture.ts` mirrors the pass logic of ora's `capture-server.ts` (Pass A's native `getTools()` probe, the initial-arrival registration wait, `registrationWaitComplete` and `deriveIframeAllow` with ora's own test cases), so re-check it against that file on every re-copy. +11. **The WebMCP capture is a copy of ora's engine, not a reimplementation.** `ax webmcp-audit` measures a page here and ora scores it there, and the same site has to get the same answer from both - so everything that decides a value on the wire is copied verbatim into `src/webmcp/vendor/` (the shim, the verdict ladder, the static signals and the script-type predicate they filter inline scripts with, the declarative-form schema derivation, the check registry, the challenge predicates). Biome does not format that directory, and every file there carries a header saying so. `src/webmcp/{cdp,capture,chrome,ingest,report}.ts` is plumbing only - transport, budgets, error mapping, rendering. A judgement written by hand outside `vendor/` is the drift this arrangement exists to prevent; `deriveIframeAllow`'s tests are ora's own cases, unmodified, for exactly that reason. The version check is NOT sufficient on its own - the engine has changed this code without changing the version - so re-copy the whole directory whenever the engine moves. Four deliberate deviations, all marked inline: `collectStaticSignals` takes its inline JS as a parameter (the caller reads `script:not([src])` off the live DOM and applies the vendored `isJavaScriptScriptType` filter) and the helpers beside it that need an HTML parser (cheerio or htmlparser2) are dropped (ora uses one there, which would be a runtime dependency), `projection.ts` carries only the `PublicWebmcpAudit` interface (the projector itself needs zod and ora's scoring module), the capture covers the entry page only, and the engine's comments have their source paths rewritten to point at this directory. `capture.ts` mirrors the pass logic of ora's `capture-server.ts` (Pass A's native `getTools()` probe, the initial-arrival registration wait, `registrationWaitComplete` and `deriveIframeAllow` with ora's own test cases, the tool mapping - top-level `title`, `outputSchema`, every annotation key `mapAnnotations` keeps, declarative schemas - and a verdict that counts imperative tools only), so re-check it against that file on every re-copy. 12. **The shim version is the contract; pin it from the constant.** `WEBMCP_SHIM_VERSION` (vendored in `checks.ts`) is what the ingest endpoint checks before it does anything else, rejecting a mismatch with `400 SHIM_VERSION_MISMATCH` - which the CLI renders as "upgrade ax", never as a generic API error. A capture taken by a different shim measured something else, so the version is not a label to adjust until a request is accepted: never hand-edit it, and re-copy the whole of `vendor/` from the engine revision the endpoint actually runs. Copying from a newer, unshipped revision produces captures the server refuses and, worse, captures that claim to be something they are not. 13. **Availability is an informational verdict, and it is on the PUBLISHED shape only.** The terminal `done` frame carries the projection (`vendor/projection.ts`), not the stored record: it adds `contractVersion`, a non-null `evidenceCoverage`, `applicableCategories`, and the derived `availability` verdict. Availability never gates the score: a `not-ready` page keeps its score and pillars, and `--min-score` compares that score like any other. The one withholding is a declarative-only page, whose grade is null beside a numeric score; its `declarative-only` availability reason says why, and the report defers to it rather than blaming coverage. A null score with `--min-score` is exit `3` (no answer to compare). A pillar reading `0` is ambiguous in the scores alone (earned nothing vs nothing to measure); the report reads `applicableCategories`, which ora publishes for exactly that, never guessing from the number. 14. **A browser that provides WebMCP does not measure the same thing.** ora captures with a browser that has no WebMCP support, and pass A asks whether `modelContext` is present without the shim having installed it - so on a Chrome started with `--enable-features=WebMCPTesting` that reads true for every page. Three independent guards, all worth keeping: `webmcpParityWarning` reads the finished capture and says so on stderr in both output modes; the CLI launches its own flag-free Chrome rather than attaching to whatever sits on 9222 (which is where the `webmcp` plugin tells people to keep a flag-ENABLED one for `@ora-ai/webmcp-verify`); and the server annotates the affected finding itself, which is why `details` is rendered verbatim. The launched browser also mirrors ora's capture browser where it shows up in a grade: images, media and fonts refused, no host rules (ora makes localhost unresolvable, which would break this command entirely) and no `--no-sandbox` (ora needs it in a container; a developer's machine does not). diff --git a/README.md b/README.md index b65cb51..4b73936 100644 --- a/README.md +++ b/README.md @@ -158,7 +158,7 @@ present in order to call your tools. passes and the same probe code ora's worker runs, plus a viewport screenshot at ora's 1280×720. As ora does, it loads the page with images, media and webfonts refused — stylesheets still apply — so the screenshot the grader sees is the one ora would have taken. That capture is -posted to ora, which runs its 19 checks, two model hops and the tool-selection simulation, and +posted to ora, which runs its 20 checks, two model hops and the tool-selection simulation, and streams back a score. The CLI never calls your tools, so the three checks that need a tool call (`invoke-success-rate`, `visible-effect`, `hint-vs-observed`) read `Not applicable` on a local audit. The CLI scores nothing itself (design decision #1) — grade, findings, diff --git a/scripts/contract-gen.mjs b/scripts/contract-gen.mjs index f1b1ec8..44a1c22 100644 --- a/scripts/contract-gen.mjs +++ b/scripts/contract-gen.mjs @@ -38,7 +38,28 @@ if (!/^\d+\.\d+\.\d+$/.test(version ?? "")) { process.exit(1); } -const types = await openapiTS(spec); +// TypeScript requires an index signature to cover every declared property, +// while JSON Schema lets `additionalProperties` constrain only the EXTRA keys. +// openapi-typescript v6 emits the schema literally, so an object such as +// ScoringManifest.switches (`paymentScoring: string` beside boolean extras) +// becomes a type that does not compile. Widen each such additionalProperties +// to also accept the declared property schemas, on a copy used only for type +// generation: the declared properties keep their exact types, and the index +// signature becomes the union TypeScript needs. +function widenIndexSignatures(node) { + if (Array.isArray(node)) return node.map(widenIndexSignatures); + if (!node || typeof node !== "object") return node; + const out = {}; + for (const [key, value] of Object.entries(node)) out[key] = widenIndexSignatures(value); + const extra = out.additionalProperties; + const declared = out.properties ? Object.values(out.properties) : []; + if (extra && typeof extra === "object" && Object.keys(extra).length > 0 && declared.length > 0) { + out.additionalProperties = { oneOf: [extra, ...declared] }; + } + return out; +} + +const types = await openapiTS(widenIndexSignatures(spec)); const header = [ "// Generated by scripts/contract-gen.mjs - do not edit.", diff --git a/src/contract/types.gen.ts b/src/contract/types.gen.ts index 5b55025..30c9448 100644 --- a/src/contract/types.gen.ts +++ b/src/contract/types.gen.ts @@ -59,6 +59,13 @@ export interface paths { */ get: operations["getScore"]; }; + "/api/deep-checks/{domain}": { + /** + * Poll deep checks for a scan + * @description Returns checks and completion state for the caller's assigned scan. Pass the scanId from the original response to keep reading that scan after an assignment changes. With scanId, domain can be the URL-encoded exact MCP endpoint. Responses are never cached. + */ + get: operations["getDeepChecks"]; + }; "/api/badge/{domain}": { /** * Get SVG badge for a domain @@ -66,10 +73,17 @@ export interface paths { */ get: operations["getBadge"]; }; + "/api/ranker/releases/{version}/rules": { + /** + * Read the rules for a published scoring version + * @description Returns the scored checks, layer weights, grade thresholds and score formula for the requested version. A frozen version returns exactly the rules it was frozen with. Before managed publication, the current default and explicitly available legacy scoring versions are readable with publishedAt set to null. After managed publication, unpublished candidates are not public. No authentication required; responses use Cache-Control: no-store and Access-Control-Allow-Origin: *. + */ + get: operations["getScoringRules"]; + }; "/api/checks": { /** * Get the complete catalog of scanner checks - * @description Returns every check the ora scanner can run - stable id, scored layer, max score, applicability, eligible scan kinds, tier, maturity, and fix guidance per check, plus the four scored layers with their weights. Check ids are stable: gate CI on an explicit id list, not on tiers (the required set can grow on a minor version). Ids are also what POST /api/scan/checks takes, and every check carries a `beta` boolean for building check pickers - a beta check runs but cannot affect any score. The document is static and byte-stable between check-set changes, so diffing it detects catalog updates. One optional parameter: `?include=essentials` adds `essentialsTier`, `essentialsBonusOnly`, and `essentialsExcluded` (the essentials-model classification; excluded checks are ignored by that model outright) to every check; without it the body is byte-identical to previous releases. Sends Access-Control-Allow-Origin: * and is CDN-cached for 1 hour. Rate limited to 60 requests per minute per IP - returns 429 with a Retry-After header if exceeded. + * @description Returns every check the ora scanner can run - stable id, scored layer, max score, applicability, eligible scan kinds, tier, maturity, and fix guidance per check, plus the four scored layers with their weights. Check ids are stable: gate CI on an explicit id list, not on tiers (the required set can grow on a minor version). Ids are also what POST /api/scan/checks takes, and every check carries a `beta` boolean for building check pickers - a beta check runs but cannot affect any score. The catalog follows the public scoring version or the partner API key's assigned version; scoringVersion and rulesUrl identify its rules. One optional parameter: `?include=essentials` adds `essentialsTier`, `essentialsBonusOnly`, and `essentialsExcluded` (the essentials-model classification; excluded checks are ignored by that model outright) to every check; without it those fields are omitted. Sends Access-Control-Allow-Origin: *. Keyed or managed-release responses use private, no-store; anonymous legacy responses are CDN-cached for 1 hour. Rate limited to 60 requests per minute per IP - returns 429 with a Retry-After header if exceeded. */ get: operations["listChecks"]; }; @@ -353,6 +367,11 @@ export interface components { recommendation?: string; /** @description Why this check does not apply to this product - it is skipped, not a deduction */ naReason?: string; + /** + * @description Present only on 'na' checks that DO apply but could not be checked: 'unverified' means a bot shield answered instead of the site, the server requires authentication, our request got no answer, or a source we query (such as web search) did not reply. Skipped, not deducted, like any 'na'. Absent on an 'na' means the check does not apply to this product. + * @enum {string} + */ + naKind?: "unverified"; }; AuditLayer: { /** @description Layer id: discovery | accessibility | usability | payments (historical scans may carry retired ids) */ @@ -391,9 +410,25 @@ export interface components { recommendation?: string; /** @description Why this check does not apply to this product - it is skipped, not a deduction */ naReason?: string; + /** + * @description Present only on 'na' checks that DO apply but could not be checked: 'unverified' means a bot shield answered instead of the site, the server requires authentication, our request got no answer, or a source we query (such as web search) did not reply. Skipped, not deducted, like any 'na'. Absent on an 'na' means the check does not apply to this product. + * @enum {string} + */ + naKind?: "unverified"; })[]; }; AuditScanResult: { + /** @description Stable managed scan ID. Pass it when polling to keep reading this scan after release assignments change. */ + scanId?: number; + /** @description Scoring rules version used for this result, independent of contractVersion. Absent when an older result has no recorded version. */ + scoringVersion?: number; + /** + * Format: uri + * @description Machine-readable rules for scoringVersion. Absent when the scoring version is unknown. + */ + rulesUrl?: string; + /** @description Revision of the shared check code under scoringVersion: a result reads as scoringVersion.scoringRevision. A higher revision means a shared check fix that can move scores under the same rules. Absent when unknown, including results stored before release management. */ + scoringRevision?: number; domain: string; name: string; score: number; @@ -521,6 +556,11 @@ export interface components { recommendation?: string; /** @description Why this check does not apply to this product - it is skipped, not a deduction */ naReason?: string; + /** + * @description Present only on 'na' checks that DO apply but could not be checked: 'unverified' means a bot shield answered instead of the site, the server requires authentication, our request got no answer, or a source we query (such as web search) did not reply. Skipped, not deducted, like any 'na'. Absent on an 'na' means the check does not apply to this product. + * @enum {string} + */ + naKind?: "unverified"; })[]; })[]; /** @description Actionable (fail/warning) checks ranked by ora: non-bonus first, then estimated uplift descending, capped at 6. Render verbatim - do not re-rank. */ @@ -551,7 +591,7 @@ export interface components { * @description The contract version this payload conforms to. SemVer: a major means a response-envelope break - a stable field removed, renamed, or changed in meaning, or the default response format flipping - and is safe to pin. Additive changes and check-catalog membership changes ship on a minor. See docs/api.md -> Contract and versioning. * @enum {string} */ - contractVersion: "1.38.0"; + contractVersion: "1.40.0"; /** * @description Present only when this body is a stored result served by the freshness gate instead of a fresh scan. Absent on a live scan. * @enum {boolean} @@ -599,6 +639,17 @@ export interface components { } | null; }; AuditScoreResult: { + /** @description Stable managed scan ID. Pass it when polling to keep reading this scan after release assignments change. */ + scanId?: number; + /** @description Scoring rules version used for this result, independent of contractVersion. Absent when an older result has no recorded version. */ + scoringVersion?: number; + /** + * Format: uri + * @description Machine-readable rules for scoringVersion. Absent when the scoring version is unknown. + */ + rulesUrl?: string; + /** @description Revision of the shared check code under scoringVersion: a result reads as scoringVersion.scoringRevision. A higher revision means a shared check fix that can move scores under the same rules. Absent when unknown, including results stored before release management. */ + scoringRevision?: number; domain: string; name: string; score: number; @@ -724,6 +775,11 @@ export interface components { recommendation?: string; /** @description Why this check does not apply to this product - it is skipped, not a deduction */ naReason?: string; + /** + * @description Present only on 'na' checks that DO apply but could not be checked: 'unverified' means a bot shield answered instead of the site, the server requires authentication, our request got no answer, or a source we query (such as web search) did not reply. Skipped, not deducted, like any 'na'. Absent on an 'na' means the check does not apply to this product. + * @enum {string} + */ + naKind?: "unverified"; })[]; })[]; /** @description Actionable (fail/warning) checks ranked by ora: non-bonus first, then estimated uplift descending, capped at 6. Render verbatim - do not re-rank. */ @@ -754,7 +810,7 @@ export interface components { * @description The contract version this payload conforms to. SemVer: a major means a response-envelope break - a stable field removed, renamed, or changed in meaning, or the default response format flipping - and is safe to pin. Additive changes and check-catalog membership changes ship on a minor. See docs/api.md -> Contract and versioning. * @enum {string} */ - contractVersion: "1.38.0"; + contractVersion: "1.40.0"; /** @description Wall-clock duration of the scan that produced this stored result */ durationMs: number | null; /** @@ -798,11 +854,20 @@ export interface components { }; /** @description The complete catalog of scanner checks. Stability classes: every field's shape and presence rule is stable within a major version; layer ids are envelope identity, so removing or renaming one is a major version change; check ids never change meaning while they exist, but catalog membership may change on a minor, with a contract changelog entry and a deprecation window; every other value (scores, weights, layer assignments, applicability, tiers, maturity, prose) is advisory and may change on a minor version, with score-relevant changes recorded in the contract changelog. Gate CI on an explicit list of check ids. */ CheckCatalog: { + /** @description Scoring rules version used for this result, independent of contractVersion. Absent when an older result has no recorded version. */ + scoringVersion?: number; + /** + * Format: uri + * @description Machine-readable rules for scoringVersion. Absent when the scoring version is unknown. + */ + rulesUrl?: string; + /** @description Revision of the shared check code under scoringVersion: a result reads as scoringVersion.scoringRevision. A higher revision means a shared check fix that can move scores under the same rules. Absent when unknown, including results stored before release management. */ + scoringRevision?: number; /** * @description The contract version this catalog conforms to - identical to the OpenAPI info.version and the MCP server version. SemVer: a major means a response-envelope break (a stable field, or a layer id, removed or renamed or changed in meaning, or the default response format flipping) and is safe to pin; check-catalog membership changes ship on a minor. The full versioning policy is published in the API description at /api/openapi.json. * @enum {string} */ - contractVersion: "1.38.0"; + contractVersion: "1.40.0"; /** @description The four scored layers in scoring order, with display name and current weight. */ layers: { /** @description Stable layer id: discovery, accessibility, usability, or payments. Removing or renaming a layer id is a major version change. Note one intentional divergence: the id 'accessibility' carries the display name 'Access'. */ @@ -894,14 +959,27 @@ export interface components { checkIds: string[]; /** @description Optional URL of the target's MCP server, matching the same field on POST /api/scan. It pins the MCP endpoint; a failed handshake never substitutes a discovered server. When omitted, ora auto-discovers MCP endpoints. An empty string is treated as absent. */ mcpUrl?: unknown; + /** @description Optional. The published scoring version to use for this request. Omitted, the request uses the version pinned to your API key, else the public version. An unpublished or unknown version is refused, and the error lists the published versions. */ + scoringVersion?: number; }; /** @description The outcome of a selective check run. The run always executes - results are never served from a cache. The response carries no aggregate score of any kind: per-check score and maxScore only, with GET /api/score/{domain} as the score surface. Results carry no tier or layer fields - join with GET /api/checks by id to group or rank them. */ RunChecksResponse: { + /** @description Scoring rules version used for this result, independent of contractVersion. Absent when an older result has no recorded version. */ + scoringVersion?: number; + /** + * Format: uri + * @description Machine-readable rules for scoringVersion. Absent when the scoring version is unknown. + */ + rulesUrl?: string; + /** @description Revision of the shared check code under scoringVersion: a result reads as scoringVersion.scoringRevision. A higher revision means a shared check fix that can move scores under the same rules. Absent when unknown, including results stored before release management. */ + scoringRevision?: number; + /** @description Stable managed scan ID. Pass it when polling to keep reading this scan after release assignments change. */ + scanId?: number; /** * @description The contract version this response conforms to - identical to the OpenAPI info.version and the MCP server version. The full versioning policy is published in the API description at /api/openapi.json. * @enum {string} */ - contractVersion: "1.38.0"; + contractVersion: "1.40.0"; /** @description The apex domain derived from the requested URL. */ domain: string; /** @description The normalized URL the run targeted. */ @@ -936,6 +1014,11 @@ export interface components { recommendation?: string; /** @description Why the check did not apply. Present on 'na' results, including requested ids that cannot apply to the detected kind. */ naReason?: string; + /** + * @description Present only on 'na' results that DO apply but could not be checked: 'unverified' means a bot shield answered instead of the site, the server requires authentication, ora's request got no answer, or a source ora queries (such as web search) did not reply. Absent on an 'na' result means the check does not apply. + * @enum {string} + */ + naKind?: "unverified"; /** * @description The classification of the MCP surface this entry scored against - one of: product | docs | other | app. Present on MCP fan-out entries. * @enum {string} @@ -2071,8 +2154,96 @@ export interface components { /** @description text steps only: the narrative the agent emitted between actions. Advisory - raw model prose. */ text?: string; }; + ScoringRules: { + scoringVersion: number; + /** Format: uri */ + rulesUrl: string; + /** @description The current revision of the shared check code. Results scored now read as scoringVersion.scoringRevision. */ + scoringRevision?: number; + /** @description Content hash of the scoring rules. */ + rulesHash: string; + /** @description Publication timestamp, or null for an available legacy version before managed publication. */ + publishedAt: string | null; + rules: components["schemas"]["ScoringManifest"]; + }; + ScoringManifest: { + version: number; + /** @description Identity of the check evaluation rules. */ + evaluationRules: string; + /** @description Scored checks only. Use the assigned check catalog for unscored checks and fix guidance. */ + checks: { + id: string; + layer: string; + maxScore: number; + bonus: boolean; + }[]; + layers: { + id: string; + weight: number; + }[]; + formula: { + aggregation: string; + notApplicable: string; + emerging: string; + bonus: string; + emptyLayerWeight: string; + normalization: string; + emptyScore: number; + rounding: string; + checkRescaling: string; + }; + rules: { + scoreGate: { + checkIds: string[]; + cap: number; + } | null; + useLayerMinMaxScore: number; + }; + /** @description Scoring behavior switches enabled for this version. */ + switches: { + paymentScoring?: string; + [key: string]: (boolean | string) | undefined; + }; + }; + DeepCheckResponse: { + scanId?: number; + scoringVersion?: number; + /** Format: uri */ + rulesUrl?: string; + /** @description Revision of the shared check code the scan ran with. Absent when unknown. */ + scoringRevision?: number; + allComplete: boolean; + allReady: boolean; + agenticSummary?: string; + checks: { + checkId: string; + status: string; + score: number; + maxScore: number; + details: string; + recommendation?: string; + naReason?: string; + mcpKind?: string; + mcpUrl?: string; + progress?: Record[]; + data?: { + [key: string]: unknown; + }; + }[]; + }; /** @description The default response body of POST /api/scan and GET /api/score/{domain}. Fields not listed below may be present: they are ora internals, are not part of the contract, and may change or disappear in any release without a major version bump. Pass `?format=audit` for the versioned, fully documented shape (AuditScanResult / AuditScoreResult). */ ScanResult: { + /** @description Stable managed scan ID. Preserve it when polling this result. */ + scanId?: number; + /** @description The checks and scoring rules used for this result. Absent when the stored version is unknown. */ + scoringVersion?: number; + /** + * Format: uri + * @description The machine-readable rules for scoringVersion. Absent when the stored version is unknown. + */ + rulesUrl?: string; + /** @description Revision of the shared check code under scoringVersion: the result reads as scoringVersion.scoringRevision. A higher revision means a shared check fix that can move scores under the same rules. Absent when unknown, including results stored before release management. */ + scoringRevision?: number; /** @description The scanned domain (pre-redirect). Compare with new URL(finalUrl).hostname to detect cross-domain redirects. */ domain?: string; /** @description The normalized URL */ @@ -2176,6 +2347,11 @@ export interface components { tier?: "required" | "recommended" | "emerging"; /** @description Optional. Why the check does not apply to this product. Present on 'na' rows; the check is skipped, not deducted. */ naReason?: string; + /** + * @description Optional. Present only on 'na' rows that DO apply but could not be checked: 'unverified' means a bot shield answered instead of the site, the server requires authentication, ora's request got no answer, or a source ora queries (such as web search) did not reply. Skipped, not deducted, like any 'na'. Absent on an 'na' row means the check does not apply to this product. + * @enum {string} + */ + naKind?: "unverified"; /** @description Optional. Estimated points fully fixing this check would add to the overall 0-100 score, already normalized to the layer weight. Present on actionable (fail/warning) checks only; an estimate, not exact. This is the uplift signal - do not read maxScore minus score as score uplift. */ estScoreGain?: number; }; @@ -2656,6 +2832,8 @@ export interface operations { siteType?: "content" | "business" | "app" | "store"; /** @description Optional, defaults to false. A few checks confirm that an endpoint the site declares for agents enforces what it declares by sending it a request that must be refused (for example an unauthenticated POST). Pass true to send those requests for this scan; without it the affected checks reuse the domain's previous result where one exists. */ activeProbes?: boolean; + /** @description Optional. The published scoring version to use for this request. Omitted, the request uses the version pinned to your API key, else the public version. An unpublished or unknown version answers 400 `scoring_version_unavailable` listing the published versions; a value that is not a positive integer answers 400 `invalid_scoring_version`. Until release management is active, a request can name 3 (the same opt-in as the older `rankerVersion: 3`) or the version it would run anyway; any other version answers 400. A version other than the one a domain was last scanned under has no stored result yet, so the first scan of a domain under it runs live. */ + scoringVersion?: number; }; }; }; @@ -2678,7 +2856,7 @@ export interface operations { /** @description Scan accepted and scored, but analysis is still in progress (analysisStatus is 'partial'). The body is the same shape as a 200 (including `competitors` when `?competitors=1` was passed, and `#/components/schemas/AuditScanResult` when `?format=audit` was passed). Poll the Location header URL (GET /api/score/{domain}) until analysisStatus is 'complete' and pendingChecks is empty. */ 202: { headers: { - /** @description URL of the scan result resource to poll, e.g. /api/score/stripe.com */ + /** @description URL of the scan result resource to poll. Managed scans include scanId; preserve the complete Location URL. */ Location?: string; }; content: { @@ -2819,6 +2997,8 @@ export interface operations { ephemeral?: "1"; /** @description Pass 1 to send the requests a few checks send to endpoints the site declares for agents - the query spelling of `activeProbes: true` on POST /api/scan. Absent, they are not sent. */ activeProbes?: "1"; + /** @description Optional. The published scoring version to use for this request. Omitted, the request uses the version pinned to your API key, else the public version. An unpublished or unknown version answers 400 `scoring_version_unavailable` listing the published versions; a value that is not a positive integer answers 400 `invalid_scoring_version`. Until release management is active, a request can name 3 (the same opt-in as the older `rankerVersion: 3`) or the version it would run anyway; any other version answers 400. A version other than the one a domain was last scanned under has no stored result yet, so the first scan of a domain under it runs live. */ + scoringVersion?: number; /** @description Pass `audit` to receive the versioned, allowlisted audit shape (AuditScanResult / AuditScoreResult) instead of the default body: every field is documented, carries a `contractVersion`, and internal fields are dropped. Omit it and the response is unchanged from previous releases. On GET /api/scan/stream, only the `scan_complete` event's `result` is projected - all other events are identical to the default stream. */ format?: "audit"; /** @description Opt-in expansion list (comma-separated). Experimental: the audit schemas declare `siteType` only as a loose object, so its inner fields may change without a version bump, and pinning the contract major does not freeze them. `siteType` adds one response key, `siteType`: the same scan scored against what the site is (`content`, `business`, `app`, or `store`). It carries `type`, `source` (`declared`, `self-declared`, `classified`, `default`, `blocked`, or `unavailable`), `confidence` (`high` or `low`), `basis`, `score` (0-100 or null), per-layer `layers` (`id`, `score`, `maxScore`), per-check `states` (`req`, `bonus`, `gated`, or `emerging`), and `gatedOut` (each excluded check id with the gate that excluded it). A site-type score uses a different rubric from the canonical `score` and can land above or below it: never compare the two, only another `siteType.score` of the same `type`. When no type resolves, the key is `{ type: null, source: "unavailable", confidence: "low", score: null, layers: [] }` with a `basis` saying why. On the stream the key rides the `scan_complete` event beside `result`, never inside it. */ @@ -2828,7 +3008,7 @@ export interface operations { }; }; responses: { - /** @description Server-Sent Events stream of scan progress. If MCP authentication is required, the stream ends with an error event carrying code MCP_AUTH_REQUIRED, mcpAuthRequired: true, mcpUrl and urlKind, without scan_complete or a score. With `?format=audit`, the `scan_complete` event's `result` is `#/components/schemas/AuditScanResult`; every other event is unchanged. When a stored result inside the freshness window answers the request, the stream is a `kind_detected` frame followed by a terminal `scan_complete` event carrying `servedFromCache: true` and `resultAgeSeconds`, and the response carries an `Age` header. After `scan_complete`, a live scan may emit two correction events before the stream closes, both optional: `relevance_assessed` (`{ naCheckIds, reasons, score, grade }`, only when some checks are judged not applicable for this product) carries the corrected `score` and `grade`, and `summary_ready` (`{ agenticSummary }`) carries the summary. Consume events until the stream closes. Closing does not mean the analysis is final: when deep checks are still pending, the stream closes early and a background worker finishes relevance and summary later. Read GET /api/score/{domain} after the stream closes and poll it while `analysisStatus` is `partial`. A freshness-window hit is already final and emits neither event. Event types (the `type` field), in emission order: `kind_detecting` - reachability passed and URL-kind detection is running (no payload); `kind_detected` - detection resolved: `{ kind: 'domain' | 'mcp' | 'mcp-app', mcpUrl?, embeddedMcpUrls?, hint? }`, exactly once, before any `check_start`; `discovery_phase` (optional) - a context-gathering step, or after `scan_complete` a post-scan finalize step: `{ step, label?, stepIndex?, totalSteps? }`, zero or more times; `scan_init` - the check plan: `{ layerMaxScores, totalChecks?, checkRoster?, staticOnly? }`; may be sent twice, the later one is authoritative; `check_start` - one check slot started: `{ layerId, layerName, checkId, checkName, mcpKind?, mcpUrl? }`; `check_complete` - one check slot finished: `{ layerId, layerName, checkId, checkName, status, score, maxScore, details?, bonus?, maturity?, mcpKind?, mcpUrl? }`; `layer_complete` - every check in a layer finished: `{ layerId, layerName }`; `scan_complete` - the scored result, not the last frame: `{ result }` (AuditScanResult under `?format=audit`), plus `siteType` with `?include=siteType` and `servedFromCache` / `resultAgeSeconds` on a freshness hit; `relevance_assessed` (optional) - after `scan_complete`, only when checks were judged not applicable: `{ naCheckIds, reasons, score, grade }` with the corrected score and grade; `summary_ready` (optional) - after `scan_complete`: `{ agenticSummary }`; `error` (terminal, optional) - the scan failed and no `scan_complete` follows: `{ message }`, plus `code: 'MCP_AUTH_REQUIRED'`, `mcpAuthRequired`, `mcpUrl` and `urlKind` when an MCP server requires authentication. Read until the stream closes; only a terminal event is guaranteed to be the last frame. */ + /** @description Server-Sent Events stream of scan progress. If MCP authentication is required, the stream ends with an error event carrying code MCP_AUTH_REQUIRED, mcpAuthRequired: true, mcpUrl and urlKind, without scan_complete or a score. With `?format=audit`, the `scan_complete` event's `result` is `#/components/schemas/AuditScanResult`; every other event is unchanged. When a stored result inside the freshness window answers the request, the stream is a `kind_detected` frame followed by a terminal `scan_complete` event carrying `servedFromCache: true` and `resultAgeSeconds`, and the response carries an `Age` header. After `scan_complete`, a live scan may emit two correction events before the stream closes, both optional: `relevance_assessed` (`{ naCheckIds, reasons, score, grade }`, only when some checks are judged not applicable for this product) carries the corrected `score` and `grade`, and `summary_ready` (`{ agenticSummary }`) carries the summary. Consume events until the stream closes. Closing does not mean the analysis is final: when deep checks are still pending, the stream closes early and a background worker finishes relevance and summary later. Read GET /api/score/{domain} after the stream closes and poll it while `analysisStatus` is `partial`. When present, preserve `result.scanId` as the scanId query parameter. Managed live progress frames carry the same optional scanId after allocation, so early deep-check polling can use it. A freshness-window hit is already final and emits neither event. Event types (the `type` field), in emission order: `kind_detecting` - reachability passed and URL-kind detection is running (no payload); `kind_detected` - detection resolved: `{ kind: 'domain' | 'mcp' | 'mcp-app', mcpUrl?, embeddedMcpUrls?, hint? }`, exactly once, before any `check_start`; `discovery_phase` (optional) - a context-gathering step, or after `scan_complete` a post-scan finalize step: `{ step, label?, stepIndex?, totalSteps? }`, zero or more times; `scan_init` - the check plan: `{ layerMaxScores, totalChecks?, checkRoster?, staticOnly? }`; may be sent twice, the later one is authoritative; `check_start` - one check slot started: `{ layerId, layerName, checkId, checkName, mcpKind?, mcpUrl? }`; `check_complete` - one check slot finished: `{ layerId, layerName, checkId, checkName, status, score, maxScore, details?, bonus?, naKind?, maturity?, mcpKind?, mcpUrl? }`; `layer_complete` - every check in a layer finished: `{ layerId, layerName }`; `scan_complete` - the scored result, not the last frame: `{ result }` (AuditScanResult under `?format=audit`), plus `siteType` with `?include=siteType` and `servedFromCache` / `resultAgeSeconds` on a freshness hit; `relevance_assessed` (optional) - after `scan_complete`, only when checks were judged not applicable: `{ naCheckIds, reasons, score, grade }` with the corrected score and grade; `summary_ready` (optional) - after `scan_complete`: `{ agenticSummary }`; `error` (terminal, optional) - the scan failed and no `scan_complete` follows: `{ message }`, plus `code: 'MCP_AUTH_REQUIRED'`, `mcpAuthRequired`, `mcpUrl` and `urlKind` when an MCP server requires authentication. Read until the stream closes; only a terminal event is guaranteed to be the last frame. */ 200: { content: { "text/event-stream": string; @@ -2894,6 +3074,10 @@ export interface operations { getScore: { parameters: { query?: { + /** @description Read the exact managed scan returned by a scan response. Keep the original target and scanId when polling after a release or partner assignment changes. Only published standard or preparation scans are readable; previews are hidden. A missing or mismatched ID never falls back to the current scan. */ + scanId?: number; + /** @description Optional. The published scoring version to use for this request. Omitted, the request uses the version pinned to your API key, else the public version. An unpublished or unknown version answers 400 `scoring_version_unavailable` listing the published versions; a value that is not a positive integer answers 400 `invalid_scoring_version`. Until release management is active, a request can name 3 (the same opt-in as the older `rankerVersion: 3`) or the version it would run anyway; any other version answers 400. A version other than the one a domain was last scanned under has no stored result yet, so the first scan of a domain under it runs live. */ + scoringVersion?: number; /** @description Pass 1 to include a `competitors` object in the response: the category top-5 leaders plus the neighbor window (2 above / self / 2 below) drawn from the leaderboard. Returned only for domains with a market category - unclassified domains (Community, or no leaderboard row) get `competitors: null`. Omitted by default so the plain response stays lean. */ competitors?: "1"; /** @description Pass `audit` to receive the versioned, allowlisted audit shape (AuditScanResult / AuditScoreResult) instead of the default body: every field is documented, carries a `contractVersion`, and internal fields are dropped. Omit it and the response is unchanged from previous releases. On GET /api/scan/stream, only the `scan_complete` event's `result` is projected - all other events are identical to the default stream. */ @@ -2904,7 +3088,7 @@ export interface operations { siteType?: "content" | "business" | "app" | "store"; }; path: { - /** @description The domain to look up (e.g. stripe.com). URL-encoded full URLs are normalized to their hostname. */ + /** @description The domain to look up (e.g. stripe.com). Without scanId, URL-encoded full URLs are normalized to their hostname. With scanId, use the exact scanned target; MCP endpoints retain their full URL, including a meaningful trailing slash. */ domain: string; }; }; @@ -2925,10 +3109,14 @@ export interface operations { }; }; }; - /** @description No cached score for this domain. Body includes `code: "DOMAIN_NOT_SCANNED"` and a `next_action` pointing at `POST /api/scan` (`nextAction`, camelCase, under `?format=audit`). */ + /** @description The scanId is not a positive integer. */ + 400: { + content: never; + }; + /** @description Without scanId, no cached score for this domain. With scanId, code scan_not_found means the target/ID is missing or not public; scan_result_pending means that exact scan has not written a result yet. Neither falls back to the current row. The domain miss body includes `code: "DOMAIN_NOT_SCANNED"` and a `next_action` pointing at `POST /api/scan` (`nextAction`, camelCase, under `?format=audit`). */ 404: { content: { - "application/json": components["schemas"]["NotScannedResponse"]; + "application/json": components["schemas"]["NotScannedResponse"] | components["schemas"]["ErrorResponse"]; }; }; /** @description MCP authentication is required. No score or grade was produced. The failed attempt does not overwrite a previous measured scan. */ @@ -2951,6 +3139,45 @@ export interface operations { 500: { content: never; }; + /** @description The partner scoring assignment or deployed scoring rules are unavailable. */ + 503: { + content: never; + }; + }; + }; + /** + * Poll deep checks for a scan + * @description Returns checks and completion state for the caller's assigned scan. Pass the scanId from the original response to keep reading that scan after an assignment changes. With scanId, domain can be the URL-encoded exact MCP endpoint. Responses are never cached. + */ + getDeepChecks: { + parameters: { + query?: { + /** @description Read the exact managed scan returned by a scan response. Keep the original target and scanId when polling after a release or partner assignment changes. Only published standard or preparation scans are readable; previews are hidden. A missing or mismatched ID never falls back to the current scan. */ + scanId?: number; + }; + path: { + domain: string; + }; + }; + responses: { + /** @description The selected scan's deep-check state */ + 200: { + content: { + "application/json": components["schemas"]["DeepCheckResponse"]; + }; + }; + /** @description Invalid scanId. */ + 400: { + content: never; + }; + /** @description The exact scan is missing, belongs to another target, or is not public. */ + 404: { + content: never; + }; + /** @description Scoring assignment or deep-check results unavailable. */ + 503: { + content: never; + }; }; }; /** @@ -2977,15 +3204,57 @@ export interface operations { }; }; }; + /** + * Read the rules for a published scoring version + * @description Returns the scored checks, layer weights, grade thresholds and score formula for the requested version. A frozen version returns exactly the rules it was frozen with. Before managed publication, the current default and explicitly available legacy scoring versions are readable with publishedAt set to null. After managed publication, unpublished candidates are not public. No authentication required; responses use Cache-Control: no-store and Access-Control-Allow-Origin: *. + */ + getScoringRules: { + parameters: { + path: { + /** @description The scoringVersion returned by a scan, score or catalog response. */ + version: number; + }; + }; + responses: { + /** @description The scoring rules */ + 200: { + content: { + "application/json": components["schemas"]["ScoringRules"]; + }; + }; + /** @description Invalid scoring version */ + 400: { + content: never; + }; + /** @description Unknown or unpublished scoring version */ + 404: { + content: never; + }; + /** @description Rate limit exceeded: 60 requests per minute per IP */ + 429: { + headers: { + /** @description Seconds until the next allowed request */ + "Retry-After"?: number; + }; + content: never; + }; + /** @description Rules unavailable, including a mismatch between published rules and this deployment */ + 503: { + content: never; + }; + }; + }; /** * Get the complete catalog of scanner checks - * @description Returns every check the ora scanner can run - stable id, scored layer, max score, applicability, eligible scan kinds, tier, maturity, and fix guidance per check, plus the four scored layers with their weights. Check ids are stable: gate CI on an explicit id list, not on tiers (the required set can grow on a minor version). Ids are also what POST /api/scan/checks takes, and every check carries a `beta` boolean for building check pickers - a beta check runs but cannot affect any score. The document is static and byte-stable between check-set changes, so diffing it detects catalog updates. One optional parameter: `?include=essentials` adds `essentialsTier`, `essentialsBonusOnly`, and `essentialsExcluded` (the essentials-model classification; excluded checks are ignored by that model outright) to every check; without it the body is byte-identical to previous releases. Sends Access-Control-Allow-Origin: * and is CDN-cached for 1 hour. Rate limited to 60 requests per minute per IP - returns 429 with a Retry-After header if exceeded. + * @description Returns every check the ora scanner can run - stable id, scored layer, max score, applicability, eligible scan kinds, tier, maturity, and fix guidance per check, plus the four scored layers with their weights. Check ids are stable: gate CI on an explicit id list, not on tiers (the required set can grow on a minor version). Ids are also what POST /api/scan/checks takes, and every check carries a `beta` boolean for building check pickers - a beta check runs but cannot affect any score. The catalog follows the public scoring version or the partner API key's assigned version; scoringVersion and rulesUrl identify its rules. One optional parameter: `?include=essentials` adds `essentialsTier`, `essentialsBonusOnly`, and `essentialsExcluded` (the essentials-model classification; excluded checks are ignored by that model outright) to every check; without it those fields are omitted. Sends Access-Control-Allow-Origin: *. Keyed or managed-release responses use private, no-store; anonymous legacy responses are CDN-cached for 1 hour. Rate limited to 60 requests per minute per IP - returns 429 with a Retry-After header if exceeded. */ listChecks: { parameters: { query?: { /** @description Opt-in expansion list (comma-separated). `essentials` adds one response key, `essentials`: an alternate reading of the same scan carrying its own `score` (0-100 or null - required checks share 80 points, recommended 20, forward-looking signals upside-only), the required/recommended buckets, label copy, per-surface sub-scores, access signals, and a `checks` map keyed by check id. That map holds the essentials INTERPRETATION only (tier, bonus, fraction, occurrences, `essentialsGain`) - name, status, details, ora's recommendation, and `estScoreGain` for the same id stay in `layers[].checks[]`, so nothing serializes twice; join on the id. The pre-sorted `issues` list and `scoreEvidence` are arrays of ids resolving in that map. Omit the parameter and the response is byte-identical to previous releases. On GET /api/checks, `essentials` instead adds `essentialsTier`, `essentialsBonusOnly`, and `essentialsExcluded` to every catalog check (an excluded check never enters the essentials model; since 1.19.1 that is the two robots.txt policy checks). */ include?: "essentials"; + /** @description Optional. The published scoring version to use for this request. Omitted, the request uses the version pinned to your API key, else the public version. An unpublished or unknown version answers 400 `scoring_version_unavailable` listing the published versions; a value that is not a positive integer answers 400 `invalid_scoring_version`. Until release management is active, a request can name 3 (the same opt-in as the older `rankerVersion: 3`) or the version it would run anyway; any other version answers 400. A version other than the one a domain was last scanned under has no stored result yet, so the first scan of a domain under it runs live. */ + scoringVersion?: number; }; }; responses: { diff --git a/src/contract/version.ts b/src/contract/version.ts index 69cd722..8e38b11 100644 --- a/src/contract/version.ts +++ b/src/contract/version.ts @@ -2,4 +2,4 @@ // Regenerate: pnpm contract:gen [--url ] /** The ora contract version (spec info.version) these types were generated from. */ -export const BUILT_AGAINST = "1.38.0"; +export const BUILT_AGAINST = "1.40.0"; diff --git a/src/report/model.test.ts b/src/report/model.test.ts index cfa070a..e7c0a51 100644 --- a/src/report/model.test.ts +++ b/src/report/model.test.ts @@ -7,6 +7,18 @@ import { toReport } from "./model"; const FIXTURE = realAuditScan as unknown as AuditScanResult; +// The fixture predates naKind. brand-search-accuracy is a real na row whose +// cause (web search did not reply) is what ora now marks unverified, so the +// delta adds only that marker. +const withUnverified = (): Partial => ({ + layers: FIXTURE.layers.map((layer) => ({ + ...layer, + checks: layer.checks.map((c) => + c.id === "brand-search-accuracy" ? { ...c, naKind: "unverified" as const } : c, + ), + })), +}); + const outcome = (extra: Partial = {}, verdict?: string) => ({ result: { ...FIXTURE, ...extra }, verdict, @@ -38,9 +50,34 @@ describe("toReport", () => { expect(section.total).toBe(counted.length); expect(section.passed).toBe(counted.filter((c) => c.status === "pass").length); expect(section.skipped).toBe(layer.checks.length - counted.length); + expect(section.unverified).toBe(0); } }); + it("reports an unverified na as couldn't-verify, not as not applicable", () => { + const report = toReport(outcome(withUnverified()), "example.com"); + const baseline = toReport(outcome(), "example.com"); + const i = FIXTURE.layers.findIndex((l) => + l.checks.some((c) => c.id === "brand-search-accuracy"), + ); + const section = report.sections[i]; + const check = section.checks.find((c) => c.id === "brand-search-accuracy"); + + expect(check?.unverified).toBe(true); + // Skipped like any na: never counted for or against the section. + expect(check?.skipped).toBe(true); + expect(check?.passed).toBe(false); + expect(check?.hint).toBeUndefined(); + expect(section.total).toBe(baseline.sections[i].total); + expect(section.passed).toBe(baseline.sections[i].passed); + // Moved out of the not-applicable count into its own. + expect(section.unverified).toBe(1); + expect(section.skipped).toBe(baseline.sections[i].skipped - 1); + // Every other na stays not applicable. + const others = report.sections.flatMap((s) => s.checks).filter((c) => c.unverified); + expect(others).toHaveLength(1); + }); + it("carries specUrl and estScoreGain onto checks that have them", () => { const report = toReport(outcome(), "example.com"); const flat = report.sections.flatMap((s) => s.checks); diff --git a/src/report/model.ts b/src/report/model.ts index f09348e..a0e9cd7 100644 --- a/src/report/model.ts +++ b/src/report/model.ts @@ -11,7 +11,15 @@ export interface ReportCheck { id: string; name: string; passed: boolean; + /** Kept out of every count: not applicable, pending, or could not be checked. */ skipped?: boolean; + /** + * The check applies but ora could not check it (`naKind: "unverified"`): a + * bot shield answered instead of the site, the server wanted credentials, + * or a request went unanswered. Skipped like any `na`, but reported as + * "couldn't verify" rather than as not applicable. + */ + unverified?: boolean; /** The finding — what the scan observed. */ detail?: string; /** The fix — what to implement (the payload's `recommendation`). */ @@ -31,7 +39,10 @@ export interface ReportSection { passed: number; /** Non-skipped check count. */ total: number; + /** Not applicable or pending. Excludes `unverified`. */ skipped: number; + /** Applies but could not be checked (`naKind: "unverified"`). */ + unverified: number; } export interface Report { @@ -56,17 +67,20 @@ export interface Report { export function sectionOf(name: string, checks: ReportCheck[]): ReportSection { const counted = checks.filter((c) => !c.skipped); + const unverified = checks.filter((c) => c.unverified).length; return { name, checks, passed: counted.filter((c) => c.passed).length, total: counted.length, - skipped: checks.length - counted.length, + skipped: checks.length - counted.length - unverified, + unverified, }; } function convertCheck(raw: AuditResult["layers"][number]["checks"][number]): ReportCheck { - // pass → passed; na/pending → skipped (kept out of every count); + // pass → passed; na/pending → skipped (kept out of every count), with an + // na that ora marks `naKind: "unverified"` flagged as couldn't-verify; // fail/warning/error all surface as issues (warning just gets a softer glyph). const passed = raw.status === "pass"; const skipped = raw.status === "na" || raw.status === "pending"; @@ -78,6 +92,7 @@ function convertCheck(raw: AuditResult["layers"][number]["checks"][number]): Rep status: raw.status, }; if (skipped) check.skipped = true; + if (raw.status === "na" && raw.naKind === "unverified") check.unverified = true; if (skipped && raw.naReason) check.detail = raw.naReason; else if (raw.details) check.detail = raw.details; if (typeof raw.estScoreGain === "number") check.estScoreGain = raw.estScoreGain; diff --git a/src/report/terminal.test.ts b/src/report/terminal.test.ts index ff37631..20a47b5 100644 --- a/src/report/terminal.test.ts +++ b/src/report/terminal.test.ts @@ -8,6 +8,17 @@ import { MCP_AUTH_REQUIRED_NOTICE, renderReport } from "./terminal"; const FIXTURE = realAuditScan as unknown as AuditScanResult; +// brand-search-accuracy is a real na row (web search did not reply); the +// delta adds only the marker ora now sends for that cause. +const withUnverified = (): Partial => ({ + layers: FIXTURE.layers.map((layer) => ({ + ...layer, + checks: layer.checks.map((c) => + c.id === "brand-search-accuracy" ? { ...c, naKind: "unverified" as const } : c, + ), + })), +}); + const render = (extra: Partial = {}): string => renderReport(toReport({ result: { ...FIXTURE, ...extra } }, "example.com")).join("\n"); @@ -25,4 +36,19 @@ describe("renderReport", () => { it("says nothing about the MCP handshake for an ordinary result", () => { expect(render()).not.toContain(MCP_AUTH_REQUIRED_NOTICE); }); + + it("lists an unverified check under Couldn't verify by default, apart from skipped", () => { + const out = render(withUnverified()); + expect(out).toContain("Couldn't verify (1)"); + expect(out).toContain("1 couldn't verify"); + expect(out).toContain("Brand search not available"); + expect(render()).not.toContain("Couldn't verify"); + + const withSkipped = renderReport( + toReport({ result: { ...FIXTURE, ...withUnverified() } }, "example.com"), + { showSkipped: true }, + ).join("\n"); + const skippedBlock = withSkipped.slice(withSkipped.indexOf("Skipped (")); + expect(skippedBlock).not.toContain("Brand search not available"); + }); }); diff --git a/src/report/terminal.ts b/src/report/terminal.ts index 8b19b35..f674022 100644 --- a/src/report/terminal.ts +++ b/src/report/terminal.ts @@ -165,9 +165,10 @@ function sectionLines( const tally = section.total > 0 ? `${section.passed}/${section.total} passed` : "no applicable checks"; + const unverifiedTail = section.unverified > 0 ? ` · ${section.unverified} couldn't verify` : ""; const skipTail = section.skipped > 0 ? ` · ${section.skipped} skipped` : ""; lines.push( - ` ${pc.bold(padTo(section.name, label))} ${meter(section.passed, section.total)} ${pc.dim(tally + skipTail)}`, + ` ${pc.bold(padTo(section.name, label))} ${meter(section.passed, section.total)} ${pc.dim(tally + unverifiedTail + skipTail)}`, ); // Payload order throughout: ora already orders checks, and the ranked view @@ -194,6 +195,24 @@ function sectionLines( } } + // Checks that apply but could not be checked (a bot shield answered, the + // server wanted credentials, a request went unanswered). Not counted for or + // against the score, but shown by default: unlike a not-applicable skip, + // the site may be blocking agents. + const unverified = section.checks.filter((c) => c.unverified); + if (unverified.length) { + lines.push(` ${pc.yellow("?")} ${pc.bold(`Couldn't verify (${unverified.length})`)}`); + const grid = tableLines( + pairCols( + term, + unverified.map((c) => c.name), + ["Check", "What happened"], + ), + unverified.map((c) => [c.name, c.detail ?? ""]), + ); + lines.push(...grid.map((row) => ` ${row}`)); + } + if (view.showPassing) { const passing = section.checks.filter((c) => c.passed && !c.skipped); if (passing.length) { @@ -211,7 +230,7 @@ function sectionLines( } if (view.showSkipped) { - const skipped = section.checks.filter((c) => c.skipped); + const skipped = section.checks.filter((c) => c.skipped && !c.unverified); if (skipped.length) { lines.push(` ${pc.dim(`○ Skipped (${skipped.length})`)}`); const grid = tableLines( diff --git a/src/webmcp/capture.live.test.ts b/src/webmcp/capture.live.test.ts index 48c8e5c..e89c26b 100644 --- a/src/webmcp/capture.live.test.ts +++ b/src/webmcp/capture.live.test.ts @@ -48,9 +48,11 @@ const PAGE = ` if (mc) { mc.registerTool({ name: "search_books", + title: "Search books", description: "Search the catalogue by title or author and return matches.", inputSchema: { type: "object", properties: { query: { type: "string" } }, required: ["query"] }, - annotations: { readOnlyHint: true }, + outputSchema: { type: "object", properties: { matches: { type: "array" } } }, + annotations: { readOnlyHint: true, consequentialHint: false, debugging: false }, async execute({ query }) { return { content: [{ type: "text", text: "no results for " + query }] }; }, }); } @@ -124,6 +126,15 @@ describe.skipIf(!LIVE)("captureLocally against a real Chrome", () => { expect(tool.entryPoint).toBe("document"); expect(tool.hasExecute).toBe(true); expect(tool.annotations.readOnlyHint).toBe(true); + // The spec's newer annotation keys, top-level title and outputSchema all + // survive the trip from the shim to the contract, as on ora's worker. + expect(tool.annotations.consequentialHint).toBe(false); + expect(tool.annotations.debugging).toBe(false); + expect(tool.title).toBe("Search books"); + expect(tool.outputSchema).toEqual({ + type: "object", + properties: { matches: { type: "array" } }, + }); // Timed from navigation, so a registration that happened is a number. expect(typeof tool.registrationMs).toBe("number"); diff --git a/src/webmcp/capture.test.ts b/src/webmcp/capture.test.ts index 4d21367..de10dc3 100644 --- a/src/webmcp/capture.test.ts +++ b/src/webmcp/capture.test.ts @@ -1,5 +1,19 @@ import { describe, expect, it } from "vitest"; -import { deriveIframeAllow, registrationWaitComplete } from "./capture"; +import realCapture from "./__fixtures__/local-catalogue-capture.json"; +import { + deriveIframeAllow, + mapAnnotations, + mapDeclarativeTools, + mapShimTools, + registrationWaitComplete, +} from "./capture"; +import { declarativeInputSchemas } from "./vendor/declarative-schema"; +import { + WEBMCP_MAX_DECLARATIVE_FORMS, + type WebmcpShimDeclarativeForm, + type WebmcpShimToolRecord, +} from "./vendor/shim"; +import type { WebmcpCaptureResult } from "./vendor/types"; // The cases below are ora's own, copied alongside the logic they cover. That is // the point: this file is not checking that our implementation is reasonable, it @@ -74,3 +88,122 @@ describe("registrationWaitComplete", () => { ).toBe(true); }); }); + +// The shim record behind the real capture's one tool: the same name, +// description, schema and annotations, plus the bookkeeping fields only the +// shim carries. Variations below are explicit deltas on it. +const capturedTool = (realCapture as unknown as WebmcpCaptureResult).pages[0].tools[0]; +const shimRecord = (delta: Partial = {}): WebmcpShimToolRecord => ({ + name: capturedTool.name, + description: capturedTool.description, + descriptionTruncated: false, + inputSchema: capturedTool.inputSchema, + annotations: capturedTool.annotations, + hasExecute: capturedTool.hasExecute, + entryPoint: "document", + source: "registerTool", + registrationMs: capturedTool.registrationMs ?? 0, + unregistered: false, + unregisteredMs: null, + ...delta, +}); + +describe("mapAnnotations", () => { + it("keeps WebMCP's consequentialHint and debugging beside the older keys", () => { + expect( + mapAnnotations({ + readOnlyHint: false, + untrustedContentHint: true, + consequentialHint: true, + debugging: false, + title: "Book a table", + }), + ).toEqual({ + readOnlyHint: false, + untrustedContentHint: true, + consequentialHint: true, + debugging: false, + title: "Book a table", + }); + }); + + it("copies by shape: a non-boolean hint is not a declaration", () => { + expect(mapAnnotations({ consequentialHint: "yes", debugging: 1 })).toEqual({}); + }); +}); + +describe("mapShimTools", () => { + it("maps the real capture's tool to what the capture recorded", () => { + const [tool] = mapShimTools([shimRecord()], capturedTool.pageUrl); + expect(tool).toEqual({ ...capturedTool, title: null }); + }); + + it("reads the spec's top-level title, capped like an annotation title", () => { + const [titled] = mapShimTools([shimRecord({ title: "Search books" })], capturedTool.pageUrl); + expect(titled.title).toBe("Search books"); + const [long] = mapShimTools([shimRecord({ title: "t".repeat(600) })], capturedTool.pageUrl); + expect(long.title).toHaveLength(512); + // A shim that did not record it, or a page that declared none: null. + const [absent] = mapShimTools([shimRecord({ title: undefined })], capturedTool.pageUrl); + expect(absent.title).toBeNull(); + }); + + it("carries outputSchema when the page declared one, and omits it otherwise", () => { + const outputSchema = { type: "object", properties: { results: { type: "array" } } }; + const [declared] = mapShimTools([shimRecord({ outputSchema })], capturedTool.pageUrl); + expect(declared.outputSchema).toEqual(outputSchema); + for (const missing of [undefined, null]) { + const [tool] = mapShimTools([shimRecord({ outputSchema: missing })], capturedTool.pageUrl); + expect("outputSchema" in tool).toBe(false); + } + }); +}); + +describe("mapDeclarativeTools", () => { + const form: WebmcpShimDeclarativeForm = { + toolName: "search_books", + toolDescription: "Search the catalogue by title or author.", + action: "/search", + method: "get", + fieldCount: 1, + fields: [ + { + tag: "input", + type: "text", + name: "q", + disabled: false, + readOnly: false, + required: true, + multiple: false, + paramDescription: null, + labels: ["Query"], + ariaDescription: null, + min: null, + max: null, + step: null, + value: null, + pattern: null, + checkedValue: null, + options: [], + }, + ], + fieldsTruncated: false, + }; + + it("takes each form's schema from the engine's own derivation", () => { + const [tool] = mapDeclarativeTools([form], "https://example.com/"); + expect(tool.inputSchema).toEqual(declarativeInputSchemas([form])[0]); + expect(tool.inputSchema).not.toBeNull(); + expect(tool.via).toBe("declarative"); + }); + + it("stops at the shim's per-page form cap", () => { + const forms = Array.from({ length: WEBMCP_MAX_DECLARATIVE_FORMS + 3 }, (_, i) => ({ + ...form, + toolName: `form_${i}`, + })); + expect(mapDeclarativeTools(forms, "https://example.com/")).toHaveLength( + WEBMCP_MAX_DECLARATIVE_FORMS, + ); + }); +}); diff --git a/src/webmcp/capture.ts b/src/webmcp/capture.ts index 6131e75..80defae 100644 --- a/src/webmcp/capture.ts +++ b/src/webmcp/capture.ts @@ -41,15 +41,17 @@ import { type WebmcpPageFacts, } from "./vendor/capture-verdict"; import { WEBMCP_SHIM_VERSION } from "./vendor/checks"; +import { declarativeInputSchemas } from "./vendor/declarative-schema"; import { isSecurityChallenge } from "./vendor/security-challenge"; import { DECLARATIVE_SCAN_SNIPPET, WEBMCP_CAPTURE_SHIM, + WEBMCP_MAX_DECLARATIVE_FORMS, type WebmcpShimDeclarativeScan, type WebmcpShimSnapshot, type WebmcpShimToolRecord, } from "./vendor/shim"; -import { collectStaticSignals } from "./vendor/static-signals"; +import { collectStaticSignals, isJavaScriptScriptType } from "./vendor/static-signals"; import type { WebmcpCaptureResult, WebmcpPageCapture, @@ -81,8 +83,8 @@ const MAX_QUIET_WAIT_MS = { fast: 12_000, deep: 20_000 } as const; const MAX_HTML_CHARS = 2 * 1024 * 1024; /** Tool names are page-controlled and reach the payload verbatim. */ const MAX_TOOL_NAME_CHARS = 512; -/** Shim v1 carries a tool's human label only inside `annotations`; there is no - * `title` on the tool itself. Same ceiling ora's capture service applies. */ +/** Ceiling on a tool's `title`, top-level and inside `annotations` alike. Same + * ceiling ora's capture service applies. */ const MAX_ANNOTATION_TITLE_CHARS = 512; const SCREENSHOT_JPEG_QUALITY = 60; @@ -178,11 +180,13 @@ const FINGERPRINT_EXPRESSION = `(function () { * dropping it, but `WebmcpTool` has no slot for that state, and listing a * retired tool as registered would misreport what an agent would find. */ -function mapShimTools(records: WebmcpShimToolRecord[], pageUrl: string): WebmcpTool[] { +export function mapShimTools(records: WebmcpShimToolRecord[], pageUrl: string): WebmcpTool[] { return records .filter((record) => !record.unregistered) .map((record) => ({ name: record.name.slice(0, MAX_TOOL_NAME_CHARS), + title: + typeof record.title === "string" ? record.title.slice(0, MAX_ANNOTATION_TITLE_CHARS) : null, description: record.description, via: "imperative" as const, // The contract's four-value entry point is the pair the shim records: @@ -192,26 +196,28 @@ function mapShimTools(records: WebmcpShimToolRecord[], pageUrl: string): WebmcpT record.source === "provideContext" ? ("provideContext" as const) : record.entryPoint, pageUrl, inputSchema: record.inputSchema, + ...(record.outputSchema === null || record.outputSchema === undefined + ? {} + : { outputSchema: record.outputSchema }), annotations: mapAnnotations(record.annotations), hasExecute: record.hasExecute, registrationMs: record.registrationMs, })); } -function mapDeclarativeTools( +export function mapDeclarativeTools( forms: WebmcpShimDeclarativeScan["forms"], pageUrl: string, ): WebmcpTool[] { - return forms.map((form) => ({ + const capped = forms.slice(0, WEBMCP_MAX_DECLARATIVE_FORMS); + const schemas = declarativeInputSchemas(capped); + return capped.map((form, index) => ({ name: form.toolName.slice(0, MAX_TOOL_NAME_CHARS), description: form.toolDescription, via: "declarative" as const, entryPoint: "form" as const, pageUrl, - // A declarative tool declares no JSON Schema; its inputs are the form's - // own fields. Reporting `null` keeps the schema checks from grading a - // schema the page never claimed to have. - inputSchema: null, + inputSchema: schemas[index], annotations: {}, // Declarative execution is the browser submitting the element, not a JS // callback, so there is no stub axis here to fail. @@ -224,7 +230,7 @@ function mapDeclarativeTools( /** Page-supplied annotations, narrowed to the contract's keys. Untrusted * metadata: copied by shape, never trusted as a claim about behaviour. */ -function mapAnnotations(raw: unknown): WebmcpToolAnnotations { +export function mapAnnotations(raw: unknown): WebmcpToolAnnotations { if (!raw || typeof raw !== "object" || Array.isArray(raw)) return {}; const record = raw as Record; const out: WebmcpToolAnnotations = {}; @@ -234,6 +240,8 @@ function mapAnnotations(raw: unknown): WebmcpToolAnnotations { "idempotentHint", "openWorldHint", "untrustedContentHint", + "consequentialHint", + "debugging", ] as const) { if (typeof record[key] === "boolean") out[key] = record[key] as boolean; } @@ -369,8 +377,12 @@ async function runPassA( const status = response?.status ?? 0; const html = (await page.content()).slice(0, MAX_HTML_CHARS); // ora derives this with cheerio inside collectStaticSignals; we read the - // same selector off the live DOM. See vendor/static-signals.ts. - const inlineJs = await page.inlineScripts(); + // same selector off the live DOM and apply the same vendored type filter. + // See vendor/static-signals.ts. + const inlineJs = (await page.inlineScripts()) + .filter((script) => isJavaScriptScriptType(script.type)) + .map((script) => script.text) + .join("\n"); const probe = ((await page .evaluate<{ nativeApi?: unknown; metaTokens?: unknown }>(PASS_A_PROBE) @@ -616,7 +628,10 @@ async function capturePage( nativeApi: passA.nativeApi, nativeToolCount: passA.nativeToolCount, declarativeToolCount: declarativeTools.length, - toolCount: tools.length, + // Imperative only: declarative forms are recorded on the page as + // observations, but the spec does not define them, so they never make a + // page `active` or `testing-only`. + toolCount: imperativeTools.length, shimTakenOver: (snapshot?.takenOver.length ?? 0) > 0, staticSignals, }; diff --git a/src/webmcp/cdp.ts b/src/webmcp/cdp.ts index 76a329c..8c9ad31 100644 Binary files a/src/webmcp/cdp.ts and b/src/webmcp/cdp.ts differ diff --git a/src/webmcp/vendor/capture-verdict.ts b/src/webmcp/vendor/capture-verdict.ts index ddcf5e1..7de27db 100644 --- a/src/webmcp/vendor/capture-verdict.ts +++ b/src/webmcp/vendor/capture-verdict.ts @@ -243,9 +243,11 @@ export interface WebmcpPageFacts { /** Number of tools the native getTools() surface returned. `null` means the * API could not enumerate tools, which is unverified rather than empty. */ nativeToolCount: number | null; - /** Declarative `[toolname]` tools found in the DOM. */ + /** Declarative `[toolname]` forms found in the DOM: observed, never + * counted as tools (the 2026-10-16 draft removed the declarative API). */ declarativeToolCount: number; - /** Live tools in the capture, declarative and imperative together. */ + /** WebMCP tools in the capture: imperative registrations only. A page with + * declarative forms and nothing else is not a WebMCP page. */ toolCount: number; /** The page redefined `modelContext` over the shim on some entry point, so * an empty imperative capture is not evidence of absence. */ @@ -294,6 +296,27 @@ export function deriveVerdict(facts: WebmcpPageFacts): WebmcpVerdict { return "absent"; } +/** + * The audit verdict once declarative forms stop counting as tools. A worker + * verdict of `active` or `testing-only` over a capture with no imperative + * tool cannot stand: the native registry can list forms (Chrome's preview + * implements them) and a worker older than the 2026-10-19 change counted + * them, but the spec does not define forms, so the page has no WebMCP tools. + * It falls to the empty verdict its other facts support. Applied app-side + * so the answer does not depend on which worker produced the capture. + */ +export function verdictWithoutForms( + verdict: WebmcpVerdict, + webmcpToolCount: number, + observed: { nativeApi: boolean; staticEvidence: boolean }, +): WebmcpVerdict { + if (webmcpToolCount > 0) return verdict; + if (verdict !== "active" && verdict !== "testing-only") return verdict; + if (observed.nativeApi) return "api-empty"; + if (observed.staticEvidence) return "declared-inactive"; + return "absent"; +} + /** * Page-level status for a verdict. Exported so the audit engine maps the * audit-level status from the same table instead of restating it. diff --git a/src/webmcp/vendor/checks.ts b/src/webmcp/vendor/checks.ts index dbb1bda..35c3b32 100644 --- a/src/webmcp/vendor/checks.ts +++ b/src/webmcp/vendor/checks.ts @@ -190,16 +190,22 @@ export const WEBMCP_CHECKS: Record = { // non-GET traffic (after the GraphQL and baseline-noise filters) gets a // warning, never a fail - an HTTP method is not a write. The 2a rebalance // scaled the four siblings (6/6/4/4 -> 5/5/3/3) to make room for it. + // + // consequential-hint (2026-10-19) scores WebMCP's consequentialHint on tools + // whose names read like significant or irreversible actions - the spec's + // own mitigation for misrepresented intent (6.4.5), the sibling of + // untrusted-content-hint (6.4.4), so it takes the same 3. The two annotation + // checks (5/5 -> 4/4) and hint-vs-observed (4 -> 3) made room. "annotations-present": { title: "Annotations present", category: "trust", - weight: 5, + weight: 4, docSlug: "annotations-present", }, "annotation-mismatch": { title: "Annotation mismatch", category: "trust", - weight: 5, + weight: 4, docSlug: "annotation-mismatch", }, "injection-surface": { @@ -217,9 +223,15 @@ export const WEBMCP_CHECKS: Record = { "hint-vs-observed": { title: "Hint vs. observed", category: "trust", - weight: 4, + weight: 3, docSlug: "hint-vs-observed", }, + "consequential-hint": { + title: "Consequential hint", + category: "trust", + weight: 3, + docSlug: "consequential-hint", + }, }; // The WORKER's capture stamp: `capture-server.ts` (deployed separately, on @@ -245,11 +257,55 @@ export const WEBMCP_CHECKS: Record = { // and the registration wait gives late hydration an initial arrival window // before applying the quiet timer. Stored v2 captures cannot answer either // question, so they are deliberately incompatible rather than merely old. +// +// Not bumped on 2026-10-19, deliberately: the capture shim began recording the +// spec's top-level `title`, the invoke shim began firing toolchange / +// toolactivated / toolcancel, and declarative forms stopped counting as tools. +// All of it is additive or re-derived app-side - an older capture simply +// lacks `title` (read as "not recorded", never "not declared"), the events +// only affect invoking captures, and the forms change is applied when a +// capture is scored: the worker's verdict is worker-side, so the engine +// re-derives it (`verdictWithoutForms`) instead of trusting an older worker +// that still called a forms-only page `active` or `testing-only`. A v3 +// capture from before the change therefore still answers the questions v3 +// asks. +// Bumping would have discarded every stored audit for no measurement gain. export const WEBMCP_SHIM_VERSION = "3"; // The date the pinned spec read was actually taken (see the protocol-webmcp -// skill, which pins the same 2026-08-26 review); moves only when the read +// skill, which pins the same 2026-11-04 review); moves only when the read // date moves. -export const WEBMCP_SPEC_SNAPSHOT = "2026-08-26"; +export const WEBMCP_SPEC_SNAPSHOT = "2026-11-04"; + +/** + * Every spec snapshot whose stored audits still measure what the scanner's + * `webmcp` check reads from them (`webmcpRuntimeEvidence` in + * the scanner's own layer: the verdict and the tool count). + * Moving `WEBMCP_SPEC_SNAPSHOT` alone would refuse every stored audit as + * runtime evidence until each domain was re-audited, dropping the whole + * corpus to the static ladder for up to the 7-day fresh window. A snapshot + * belongs here only when the delta leaves the capture contract alone: + * + * - 2026-08-26 -> 2026-10-16: new annotation keys (consequentialHint, + * debugging), specified toolactivated / toolcancel events, specific + * DOMException names in place of the old catch-all, the origin-keyed agent + * cluster requirement dropped, and the declarative section removed. None + * of it changes which tools register or how the verdict is reached. + * - 2026-10-16 -> 2026-11-04: the optional outputSchema tool member and + * security notes on cross-origin exposure. Neither changes registration + * or the verdict. + * + * A delta that does change it (a new registration path, a renamed entry + * point) leaves the old snapshot out, and those audits age out on purpose. + * + * Verdict and tool count only: 2026-08-26 captures went through the old + * annotation normalizer, which dropped consequentialHint and debugging, so + * this set must never back a consumer that reads annotations. + */ +export const WEBMCP_RUNTIME_COMPATIBLE_SPEC_SNAPSHOTS: ReadonlySet = new Set([ + "2026-08-26", + "2026-10-16", + WEBMCP_SPEC_SNAPSHOT, +]); /** * Git blob SHA of the normative spec source (index.bs in @@ -260,7 +316,7 @@ export const WEBMCP_SPEC_SNAPSHOT = "2026-08-26"; * silently treated every commit on the snapshot date as reviewed). On * drift, re-verify and advance this pin together with the skill. */ -export const WEBMCP_SPEC_SOURCE_SHA = "241bb2c014b10a646649cd2a0d147498925aa63a"; +export const WEBMCP_SPEC_SOURCE_SHA = "847677e03d3e61aaddf65ad328147129083168c0"; /** Grade bands for the WebMCP audit score. Deliberately NOT the scanner's * `GRADE_THRESHOLDS` (`src/config/scoring.ts`) - this is its own domain even @@ -313,11 +369,13 @@ export function fixSnippetFor( case "annotations-present": return `{\n name: "${name}",\n // Declare readOnlyHint: true on tools that truly have no side effects -\n // that is the claim that lets an agent relax confirmation on reads.\n // (false is the default, so declaring it adds no information.)\n annotations: { readOnlyHint: true }\n}`; case "annotation-mismatch": - return `// readOnlyHint must match what the tool actually does - drop it (or set false) on a tool that writes.\n{ name: "${name}", annotations: { readOnlyHint: false } }`; + return `// readOnlyHint must match what the tool actually does - drop it (or set false) on a tool that writes,\n// and never pair readOnlyHint: true with consequentialHint: true. If the write is significant or\n// hard to undo (a purchase, a booking, a payment), declare consequentialHint: true so the agent\n// confirms with the person first.\n{ name: "${name}", annotations: { readOnlyHint: false } }`; case "injection-surface": return `{\n name: "${name}",\n // Metadata DESCRIBES the tool - it never addresses the agent reading it.\n // Rewrite any name, description, title, or schema field description that\n // instructs the agent (which tool to prefer, what to output, rules to\n // ignore) so it states what the tool does and what it returns instead.\n description: "Searches the catalog and returns matching items with prices."\n}`; case "untrusted-content-hint": return `{ name: "${name}", annotations: { untrustedContentHint: true } }`; + case "consequential-hint": + return `// consequentialHint: true marks a significant or irreversible action (a purchase,\n// a booking, a payment) so the agent confirms with the person before running it.\n{ name: "${name}", annotations: { consequentialHint: true } }`; case "human-parity": return `// Make the tool's effect visible on the page the person is looking at:\n// update the same UI your existing human flow uses when ${name.toLowerCase().replace(/\s+/g, "_")} runs,\n// so the person co-browsing sees what the agent just did.\nexecute: async (args, { signal }) => {\n const result = await performAction(args, signal);\n renderResultInPage(result); // the human-visible half of the same action\n return result;\n}`; case "registration-errors": @@ -325,7 +383,7 @@ export function fixSnippetFor( case "visible-effect": return `// A read-only tool should still show its answer on the page: render what\n// ${name.toLowerCase().replace(/\s+/g, "_")} returns where the person co-browsing can see it, and return\n// the same data to the agent.\nexecute: async (args, { signal }) => {\n const result = await fetchResult(args, signal);\n renderResultInPage(result);\n return result;\n}`; case "invoke-success-rate": - return `{\n name: "${name}",\n execute: async (args, { signal }) => {\n // Settle promptly and never throw: a rejection reaches the agent as an\n // opaque UnknownError, so put recovery hints in a successful return.\n try {\n return await doWork(args, signal);\n } catch (err) {\n return { error: "could not complete", hint: String(err && err.message) };\n }\n }\n}`; + return `{\n name: "${name}",\n execute: async (args, { signal }) => {\n // Settle promptly and never throw: a rejection reaches the agent as an\n // opaque OperationError with your message dropped, so put recovery hints\n // in a successful return.\n try {\n return await doWork(args, signal);\n } catch (err) {\n return { error: "could not complete", hint: String(err && err.message) };\n }\n }\n}`; case "hint-vs-observed": return `// readOnlyHint: true is a promise that calling the tool changes nothing.\n// Keep it on tools that only read; drop it (or set false) on one that writes.\n{ name: "${name}", annotations: { readOnlyHint: true } }`; case "tool-selection": diff --git a/src/webmcp/vendor/declarative-schema.ts b/src/webmcp/vendor/declarative-schema.ts new file mode 100644 index 0000000..53a5ac0 --- /dev/null +++ b/src/webmcp/vendor/declarative-schema.ts @@ -0,0 +1,255 @@ +// Copied from ora's WebMCP audit engine. DO NOT EDIT BY HAND. +// +// `ax webmcp-audit` captures a page here and ora scores it there. Both sides +// have to measure the same thing or a developer gets one answer locally and a +// different one from the published audit, so this file is a COPY of the +// engine's own code rather than a reimplementation of it. +// +// What pins the two together is the capture shim version - `WEBMCP_SHIM_VERSION` +// in ./checks.ts. The ingest endpoint refuses a capture taken by any other +// version. Note what that does NOT catch: the engine can change this code +// without changing the version, and it has. Re-copy the whole directory +// whenever the engine moves; do not trust the version check to notice. +// +// To update: re-copy from the engine and rewrite the import paths. Never patch +// it here - a change that belongs in the engine has to be made there first, or +// the two stop agreeing. +// +// One thing the re-copy must redo: the engine's comments cite its own source +// paths, and those are rewritten to point at files in THIS directory. Nothing +// executable differs. +// +// Verbatim: no edits beyond import paths. +// +import { + WEBMCP_MAX_DECLARATIVE_FIELDS, + WEBMCP_MAX_DECLARATIVE_FORMS, + WEBMCP_MAX_DECLARATIVE_PAGE_FIELDS, + WEBMCP_MAX_DECLARATIVE_PAGE_OPTIONS, + WEBMCP_MAX_DECLARATIVE_TEXT, + WEBMCP_MAX_SCHEMA_CHARS, + type WebmcpShimDeclarativeField, + type WebmcpShimDeclarativeForm, +} from "./shim"; + +type JsonSchema = Record; +type PendingPatterns = Array<[JsonSchema, string]>; + +const EXCLUDED_INPUT_TYPES = new Set(["hidden", "file", "submit", "image", "reset", "button"]); +const PATTERN_INPUT_TYPES = new Set(["text", "search", "url", "tel", "email", "password"]); +const DATE_HINT = "Dates MUST be provided in 'YYYY-MM-DD' format."; +// Chrome 151 puts these regexes under `format`, not `pattern`; copied verbatim from its getTools() output. +const FORMAT_BY_INPUT_TYPE = new Map([ + ["time", "^([01][0-9]|2[0-3]):[0-5][0-9]$"], + ["month", "^[0-9]{4}-(0[1-9]|1[0-2])$"], + ["week", "^[0-9]{4}-W(0[1-9]|[1-4][0-9]|5[0-3])$"], + ["datetime-local", "^[0-9]{4}-(0[1-9]|1[0-2])-[0-9]{2}T([01][0-9]|2[0-3]):[0-5][0-9]$"], + ["color", "^#[0-9a-zA-Z]{6}$"], +]); +const HTML_FLOAT = /^-?(?:\d+(?:\.\d+)?|\.\d+)(?:[eE][-+]?\d+)?$/; +const STEP_TOLERANCE = 1e-9; +const PATTERN_KEY_CHARS = ',"pattern":'.length; +// Costliest shape measured: (?i:\p{RGI_Emoji}...) at about 100 ms per 1,000 characters (bare plus anchored), so this +// holds a page to about 0.6 s of synchronous compile on the shared capture worker; real pages carry under 50. +export const WEBMCP_MAX_PAGE_PATTERN_CHARS = 6_000; + +interface PatternBudget { + charsLeft: number; +} + +/** The HTML "valid floating-point number" grammar, which is what Chrome accepts for min, max, step and value. */ +export function parseHtmlFloat(value: string | null): number | null { + if (value === null || !HTML_FLOAT.test(value)) return null; + const parsed = Number(value); + return Number.isFinite(parsed) ? parsed : null; +} + +/** One schema per form, with the scan's form, field and option caps applied again here: page scripts can + * override the built-ins the in-page caps rely on. */ +export function declarativeInputSchemas(forms: readonly WebmcpShimDeclarativeForm[]): JsonSchema[] { + let fieldsLeft = WEBMCP_MAX_DECLARATIVE_PAGE_FIELDS; + let optionsLeft = WEBMCP_MAX_DECLARATIVE_PAGE_OPTIONS; + const patterns: PatternBudget = { charsLeft: WEBMCP_MAX_PAGE_PATTERN_CHARS }; + return forms.slice(0, WEBMCP_MAX_DECLARATIVE_FORMS).map((form) => { + const fields: unknown[] = Array.isArray(form.fields) ? form.fields : []; + const options = fields.reduce( + (sum, field) => sum + (isRecord(field) && Array.isArray(field.options) ? field.options.length : 0), + 0, + ); + const withinCaps = + fields.length <= WEBMCP_MAX_DECLARATIVE_FIELDS && fields.length <= fieldsLeft && options <= optionsLeft; + fieldsLeft -= Math.min(fields.length, fieldsLeft); + optionsLeft -= Math.min(options, optionsLeft); + return withinCaps ? declarativeInputSchema(form, patterns) : { __truncated: true }; + }); +} + +/** The input schema Chrome 151 synthesizes from a declarative `[toolname]` form, a Chrome preview the WebMCP spec does + * not define. A form the scan could not read in full is `{ __truncated: true }`. */ +export function declarativeInputSchema( + form: Pick, + patterns: PatternBudget, +): JsonSchema { + if (!Array.isArray(form.fields) || form.fieldsTruncated !== false) return { __truncated: true }; + + const groups = new Map(); + for (const field of form.fields) { + if (!isFieldRecord(field) || !isAgentFillable(field)) continue; + const group = groups.get(field.name); + if (group) group.push(field); + else groups.set(field.name, [field]); + } + + const properties: Array<[string, JsonSchema]> = []; + const required: string[] = []; + const pendingPatterns: PendingPatterns = []; + for (const [name, group] of groups) { + const schema = groupSchema(group, pendingPatterns); + if (!schema) continue; + properties.push([name, schema]); + if (group.some((field) => field.required === true)) required.push(name); + } + + const schema = { type: "object", properties: Object.fromEntries(properties), required }; + const withEveryPattern = pendingPatterns.reduce( + (size, [, pattern]) => size + PATTERN_KEY_CHARS + JSON.stringify(pattern).length, + JSON.stringify(schema).length, + ); + if (withEveryPattern > WEBMCP_MAX_SCHEMA_CHARS) return { __truncated: true }; + const compiled = pendingPatterns.filter(([, pattern]) => pattern.length <= WEBMCP_MAX_DECLARATIVE_TEXT); + const compileChars = compiled.reduce((sum, [, pattern]) => sum + pattern.length, 0); + if (compileChars > patterns.charsLeft) return { __truncated: true }; + patterns.charsLeft -= compileChars; + for (const [property, pattern] of compiled) { + if (compilesAsHtmlPattern(pattern)) property.pattern = pattern; + } + return schema; +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function isFieldRecord(value: unknown): value is WebmcpShimDeclarativeField { + return ( + isRecord(value) && + (value.tag === "input" || value.tag === "select" || value.tag === "textarea") && + typeof value.type === "string" && + typeof value.name === "string" && + Array.isArray(value.labels) && + Array.isArray(value.options) + ); +} + +function isAgentFillable(field: WebmcpShimDeclarativeField): boolean { + if (field.disabled === true || field.readOnly === true) return false; + return field.tag !== "input" || !EXCLUDED_INPUT_TYPES.has(field.type); +} + +/** Radios and checkboxes sharing a name become one choice; any other shared name is dropped, as Chrome drops it. */ +function groupSchema(group: WebmcpShimDeclarativeField[], pendingPatterns: PendingPatterns): JsonSchema | null { + const ofInputType = (type: string) => group.every((field) => field.tag === "input" && field.type === type); + if (ofInputType("radio")) return choiceSchema(group); + if (ofInputType("checkbox")) { + return group.length === 1 + ? describe({ type: "boolean" }, group[0]) + : { type: "array", items: choiceSchema(group), uniqueItems: true }; + } + return group.length === 1 ? controlSchema(group[0], pendingPatterns) : null; +} + +function choiceSchema(group: WebmcpShimDeclarativeField[]): JsonSchema { + const values = group.map((field) => text(field.checkedValue) ?? "on"); + return { + type: "string", + anyOf: group.map((field, index) => { + const title = labelText(field); + return title ? { type: "string", const: values[index], title } : { type: "string", const: values[index] }; + }), + enum: values, + }; +} + +function controlSchema(field: WebmcpShimDeclarativeField, pendingPatterns: PendingPatterns): JsonSchema { + if (field.tag === "select") { + const options = field.options.filter( + (option) => typeof option?.value === "string" && typeof option?.text === "string", + ); + const choice = { + type: "string", + anyOf: options.map((option) => ({ type: "string", const: option.value, title: option.text })), + enum: options.map((option) => option.value), + }; + return describe(field.multiple === true ? { type: "array", items: choice, uniqueItems: true } : choice, field); + } + if (field.tag === "textarea") return describe({ type: "string" }, field); + if (field.type === "number" || field.type === "range") { + return describe(numberSchema(field, field.type === "range"), field); + } + if (field.type === "date") { + const own = description(field); + return { type: "string", format: "date", description: own ? `${own} (${DATE_HINT})` : DATE_HINT }; + } + const format = FORMAT_BY_INPUT_TYPE.get(field.type); + if (format) return describe({ type: "string", format }, field); + const schema = describe({ type: "string" }, field); + const pattern = PATTERN_INPUT_TYPES.has(field.type) ? text(field.pattern) : null; + if (pattern !== null) pendingPatterns.push([schema, pattern]); + return schema; +} + +/** HTML compiles `pattern` with the v flag, bare and anchored; one that fails either is ignored by the browser. */ +function compilesAsHtmlPattern(pattern: string): boolean { + if (pattern.length > WEBMCP_MAX_DECLARATIVE_TEXT) return false; + try { + RegExp(pattern, "v"); + RegExp(`^(?:${pattern})$`, "v"); + return true; + } catch (error) { + if (error instanceof SyntaxError) return false; + throw error; + } +} + +function numberSchema(field: WebmcpShimDeclarativeField, range: boolean): JsonSchema { + const min = parseHtmlFloat(text(field.min)); + const max = parseHtmlFloat(text(field.max)); + const minimum = range ? (min ?? 0) : min; + const maximum = range ? Math.max(max ?? 100, minimum ?? 0) : max; + const schema: JsonSchema = { type: "number" }; + if (minimum !== null) schema.minimum = minimum; + if (maximum !== null) schema.maximum = maximum; + const step = allowedStep(text(field.step), range); + const stepBase = min ?? parseHtmlFloat(text(field.value)) ?? 0; + if (step !== null && isMultipleOf(stepBase, step)) schema.multipleOf = step; + return schema; +} + +function allowedStep(step: string | null, range: boolean): number | null { + if (step !== null && step.toLowerCase() === "any") return range ? 1 : null; + const parsed = parseHtmlFloat(step); + return parsed !== null && parsed > 0 ? parsed : 1; +} + +function isMultipleOf(value: number, step: number): boolean { + const quotient = value / step; + return Math.abs(quotient - Math.round(quotient)) < STEP_TOLERANCE; +} + +function describe(schema: JsonSchema, field: WebmcpShimDeclarativeField): JsonSchema { + const own = description(field); + return own === null ? schema : { ...schema, description: own }; +} + +/** toolparamdescription, else the associated label text, else aria-description: the order Chrome documents. */ +function description(field: WebmcpShimDeclarativeField): string | null { + return text(field.paramDescription) || labelText(field) || text(field.ariaDescription) || null; +} + +function labelText(field: WebmcpShimDeclarativeField): string { + return field.labels.filter((label) => typeof label === "string" && label !== "").join("; "); +} + +function text(value: unknown): string | null { + return typeof value === "string" ? value : null; +} diff --git a/src/webmcp/vendor/script-type.ts b/src/webmcp/vendor/script-type.ts new file mode 100644 index 0000000..de4b5fe --- /dev/null +++ b/src/webmcp/vendor/script-type.ts @@ -0,0 +1,59 @@ +// Copied from ora's WebMCP audit engine. DO NOT EDIT BY HAND. +// +// `ax webmcp-audit` captures a page here and ora scores it there. Both sides +// have to measure the same thing or a developer gets one answer locally and a +// different one from the published audit, so this file is a COPY of the +// engine's own code rather than a reimplementation of it. +// +// What pins the two together is the capture shim version - `WEBMCP_SHIM_VERSION` +// in ./checks.ts. The ingest endpoint refuses a capture taken by any other +// version. Note what that does NOT catch: the engine can change this code +// without changing the version, and it has. Re-copy the whole directory +// whenever the engine moves; do not trust the version check to notice. +// +// To update: re-copy from the engine and rewrite the import paths. Never patch +// it here - a change that belongs in the engine has to be made there first, or +// the two stop agreeing. +// +// One thing the re-copy must redo: the engine's comments cite its own source +// paths, and those are rewritten to point at files in THIS directory. Nothing +// executable differs. +// +// Verbatim: no edits beyond import paths. +// +// Which