Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ How coding agents should work in this repo. The README covers usage; this file c
8. **One self-contained bin, zero runtime deps.** `dist/main.cjs` bundles everything (citty, picocolors are devDependencies); the library entry (`dist/index.*`) shares the client code and also has no runtime deps. Adding a runtime dependency is a design decision, not a convenience.
9. **`ax journey` namespace is reserved.** Journey-simulation commands beyond the current one land as `ax deep-journey`; do not add subcommands under `journey`.
10. **`process.exitCode`, never `process.exit()`, after writing to stdout.** A large `--json` payload exceeds the 64KB pipe buffer and `process.exit()` truncates it mid-flush.
11. **The WebMCP capture is a copy of ora's engine, not a reimplementation.** `ax webmcp-audit` measures a page here and ora scores it there, and the same site has to get the same answer from both - so everything that decides a value on the wire is copied verbatim into `src/webmcp/vendor/` (the shim, the verdict ladder, the static signals, the check registry, the challenge predicates). Biome does not format that directory, and every file there carries a header saying so. `src/webmcp/{cdp,capture,chrome,ingest,report}.ts` is plumbing only - transport, budgets, error mapping, rendering. A judgement written by hand outside `vendor/` is the drift this arrangement exists to prevent; `deriveIframeAllow`'s tests are ora's own cases, unmodified, for exactly that reason. The version check is NOT sufficient on its own - the engine has changed this code without changing the version - so re-copy the whole directory whenever the engine moves. Four deliberate deviations, all marked inline: `collectStaticSignals` takes its inline JS as a parameter and the two cheerio-only helpers beside it are dropped (ora uses an HTML parser there, which would be a runtime dependency), `projection.ts` carries only the `PublicWebmcpAudit` interface (the projector itself needs zod and ora's scoring module), the capture covers the entry page only, and the engine's comments have their source paths rewritten to point at this directory. `capture.ts` mirrors the pass logic of ora's `capture-server.ts` (Pass A's native `getTools()` probe, the initial-arrival registration wait, `registrationWaitComplete` and `deriveIframeAllow` with ora's own test cases), so re-check it against that file on every re-copy.
11. **The WebMCP capture is a copy of ora's engine, not a reimplementation.** `ax webmcp-audit` measures a page here and ora scores it there, and the same site has to get the same answer from both - so everything that decides a value on the wire is copied verbatim into `src/webmcp/vendor/` (the shim, the verdict ladder, the static signals and the script-type predicate they filter inline scripts with, the declarative-form schema derivation, the check registry, the challenge predicates). Biome does not format that directory, and every file there carries a header saying so. `src/webmcp/{cdp,capture,chrome,ingest,report}.ts` is plumbing only - transport, budgets, error mapping, rendering. A judgement written by hand outside `vendor/` is the drift this arrangement exists to prevent; `deriveIframeAllow`'s tests are ora's own cases, unmodified, for exactly that reason. The version check is NOT sufficient on its own - the engine has changed this code without changing the version - so re-copy the whole directory whenever the engine moves. Four deliberate deviations, all marked inline: `collectStaticSignals` takes its inline JS as a parameter (the caller reads `script:not([src])` off the live DOM and applies the vendored `isJavaScriptScriptType` filter) and the helpers beside it that need an HTML parser (cheerio or htmlparser2) are dropped (ora uses one there, which would be a runtime dependency), `projection.ts` carries only the `PublicWebmcpAudit` interface (the projector itself needs zod and ora's scoring module), the capture covers the entry page only, and the engine's comments have their source paths rewritten to point at this directory. `capture.ts` mirrors the pass logic of ora's `capture-server.ts` (Pass A's native `getTools()` probe, the initial-arrival registration wait, `registrationWaitComplete` and `deriveIframeAllow` with ora's own test cases, the tool mapping - top-level `title`, `outputSchema`, every annotation key `mapAnnotations` keeps, declarative schemas - and a verdict that counts imperative tools only), so re-check it against that file on every re-copy.
12. **The shim version is the contract; pin it from the constant.** `WEBMCP_SHIM_VERSION` (vendored in `checks.ts`) is what the ingest endpoint checks before it does anything else, rejecting a mismatch with `400 SHIM_VERSION_MISMATCH` - which the CLI renders as "upgrade ax", never as a generic API error. A capture taken by a different shim measured something else, so the version is not a label to adjust until a request is accepted: never hand-edit it, and re-copy the whole of `vendor/` from the engine revision the endpoint actually runs. Copying from a newer, unshipped revision produces captures the server refuses and, worse, captures that claim to be something they are not.
13. **Availability is an informational verdict, and it is on the PUBLISHED shape only.** The terminal `done` frame carries the projection (`vendor/projection.ts`), not the stored record: it adds `contractVersion`, a non-null `evidenceCoverage`, `applicableCategories`, and the derived `availability` verdict. Availability never gates the score: a `not-ready` page keeps its score and pillars, and `--min-score` compares that score like any other. The one withholding is a declarative-only page, whose grade is null beside a numeric score; its `declarative-only` availability reason says why, and the report defers to it rather than blaming coverage. A null score with `--min-score` is exit `3` (no answer to compare). A pillar reading `0` is ambiguous in the scores alone (earned nothing vs nothing to measure); the report reads `applicableCategories`, which ora publishes for exactly that, never guessing from the number.
14. **A browser that provides WebMCP does not measure the same thing.** ora captures with a browser that has no WebMCP support, and pass A asks whether `modelContext` is present without the shim having installed it - so on a Chrome started with `--enable-features=WebMCPTesting` that reads true for every page. Three independent guards, all worth keeping: `webmcpParityWarning` reads the finished capture and says so on stderr in both output modes; the CLI launches its own flag-free Chrome rather than attaching to whatever sits on 9222 (which is where the `webmcp` plugin tells people to keep a flag-ENABLED one for `@ora-ai/webmcp-verify`); and the server annotates the affected finding itself, which is why `details` is rendered verbatim. The launched browser also mirrors ora's capture browser where it shows up in a grade: images, media and fonts refused, no host rules (ora makes localhost unresolvable, which would break this command entirely) and no `--no-sandbox` (ora needs it in a container; a developer's machine does not).
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ present in order to call your tools.
passes and the same probe code ora's worker runs, plus a viewport screenshot at ora's 1280×720. As
ora does, it loads the page with images, media and webfonts refused — stylesheets still apply — so
the screenshot the grader sees is the one ora would have taken. That capture is
posted to ora, which runs its 19 checks, two model hops and the tool-selection simulation, and
posted to ora, which runs its 20 checks, two model hops and the tool-selection simulation, and
streams back a score. The CLI never calls your tools, so the three checks that need a tool call
(`invoke-success-rate`, `visible-effect`, `hint-vs-observed`) read `Not applicable` on a local
audit. The CLI scores nothing itself (design decision #1) — grade, findings,
Expand Down
23 changes: 22 additions & 1 deletion scripts/contract-gen.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,28 @@ if (!/^\d+\.\d+\.\d+$/.test(version ?? "")) {
process.exit(1);
}

const types = await openapiTS(spec);
// TypeScript requires an index signature to cover every declared property,
// while JSON Schema lets `additionalProperties` constrain only the EXTRA keys.
// openapi-typescript v6 emits the schema literally, so an object such as
// ScoringManifest.switches (`paymentScoring: string` beside boolean extras)
// becomes a type that does not compile. Widen each such additionalProperties
// to also accept the declared property schemas, on a copy used only for type
// generation: the declared properties keep their exact types, and the index
// signature becomes the union TypeScript needs.
function widenIndexSignatures(node) {
if (Array.isArray(node)) return node.map(widenIndexSignatures);
if (!node || typeof node !== "object") return node;
const out = {};
for (const [key, value] of Object.entries(node)) out[key] = widenIndexSignatures(value);
const extra = out.additionalProperties;
const declared = out.properties ? Object.values(out.properties) : [];
if (extra && typeof extra === "object" && Object.keys(extra).length > 0 && declared.length > 0) {
out.additionalProperties = { oneOf: [extra, ...declared] };
}
return out;
}

const types = await openapiTS(widenIndexSignatures(spec));

const header = [
"// Generated by scripts/contract-gen.mjs - do not edit.",
Expand Down
Loading
Loading