diff --git a/CHANGELOG.md b/CHANGELOG.md index 5717652..d658747 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ The version is the same for the package `@refact/a11y` and for the plugin `accessibility-audit`. A machine gets a new copy of the plugin only when this version changes. +## 1.1.1, 2026-11-01 + +Fixes from the first dry submit on a real site, CRE Daily, on 2026-10-28. + +- A form that sends to its own page, for example a Gravity Forms form, is the same form on each page. Before, the engine tested the footer form of CRE Daily on 47 pages. The names of the hidden fields now tell two forms with the same visible fields apart. +- The engine skips a form in a fixed bar or panel that is outside the screen, for example a closed slide-up bar. It also skips a form that a new render does not show yet, for example in a popup that opens 10 seconds after the load. The coverage names that form. Such forms show only after an action or a delay. Before, their buttons did not respond, and the coverage said "not tested" for 11 pages of CRE Daily. +- The audit records the version of the package as the engine version. Before, it recorded 0.1.0. The first audit after this update is a baseline audit: an issue that disappears in it gets "resolved, rule changed", not "fixed". +- A criterion that the processes test, for example 3.3.1 on staging, shows the process steps in its coverage (`coverage.process_units`). Before, it showed "no issue found" with 0 tested units. +- `tests/summary.mjs` tests the engine version and the coverage of the process steps. `npm test` runs it. + ## 1.1.0, 2026-10-26 The dry submit: form errors are tested on production, and no data leaves the browser. See `FORMS.md`. diff --git a/FORMS.md b/FORMS.md index 6630661..7da0bfc 100644 --- a/FORMS.md +++ b/FORMS.md @@ -36,7 +36,7 @@ Two other methods are not in this plan: 1. On each page of the sample, at the first width, the engine finds the forms that are visible after the load. 2. A form is a `form` element with at least one text field and a submit control. The engine skips a search form, a form with only a select or checkboxes, and a form with no submit control. -3. The engine tests a maximum of three forms on each page. A form that has the same action, the same fields, and the same submit control as a tested form is the same form. The engine tests it one time in each audit, on the first page in a fixed order. +3. The engine tests a maximum of three forms on each page. A form that has the same action, the same fields, the same hidden field names, and the same submit control as a tested form is the same form. A form that sends to its own page, for example a Gravity Forms form, has the same action on each page. The engine tests it one time in each audit, on the first page in a fixed order. 4. Each form gets one or two submissions. Each submission is a state of the page, in its own render and in a new browser context. - Submission A: the form empty. The engine clicks the submit control. - Submission B: each text field filled with `a11y-dry-submit`, which is not a correct email, date, phone, or address. The engine clicks the submit control. Submission B runs only when submission A showed errors on the page. @@ -75,6 +75,7 @@ The guard records each blocked request. The load of the page before the guard is | No change, and the browser marks a field as invalid | The browser shows its own message | No finding. The criteria count as tested | | No change, and the guard blocked a request after the click | The site checks the fields on the server only | No finding. The page gets "not tested" for 3.3.1 and 3.3.3. The coverage lists the form: its errors need part 2 | | No change, and no blocked request | The form needs more than one action, or does nothing | No finding. The coverage lists the form | +| The form is not visible in a new render | The form shows only after an action or a delay, for example in a popup | No submission. The coverage lists the form. Its page can still count as tested | ### 3.4 The settings @@ -91,13 +92,13 @@ The guard records each blocked request. The load of the page before the guard is ### 3.6 What the dry submit does not test - The errors of the server and the success message. Part 2 tests them. -- A form that shows only after an action, for example in a dialog or a tab. +- A form that shows only after an action, for example in a dialog, a tab, or a fixed bar that is outside the screen until it opens. - A form in a frame from another domain, for example an older HubSpot embed. - The steps after the first step of a form with more than one step. ### 3.7 The proof -1. `tests/dry-submit.mjs` serves a local site whose forms send data in eleven ways. The ways are: a fetch POST, a POST form, a GET form, and a fetch GET. Also a JSONP request to the site and to another host, a beacon, and a WebSocket message. Also a form in a new window, a form in a frame, and a pixel to another host. Each page has a form with no check and a form that checks only that the fields are not empty. The server records each request. The test passes when the server gets no submission and no typed value, and when the coverage lists the forms with no check. +1. `tests/dry-submit.mjs` serves a local site whose forms send data in eleven ways. The ways are: a fetch POST, a POST form, a GET form, and a fetch GET. Also a JSONP request to the site and to another host, a beacon, and a WebSocket message. Also a form in a new window, a form in a frame, and a pixel to another host. Each page has a form with no check and a form that checks only that the fields are not empty. The server records each request. The test passes when the server gets no submission and no typed value, and when the coverage lists the forms with no check. Since 1.1.1, two more pages have a form that sends to its own page, as Gravity Forms does, and a form in a closed fixed bar. A third form is in a popup that opens 3 seconds after the load. The engine must test the first form one time for the two pages, and skip the form in the bar and the form in the popup. 2. The same test audits the seeded faults F59 (3.3.1) and F61 (3.3.3) with production settings and no process. The engine must find both. The base site must give no form finding. 3. `npm test` runs the test. The seeded detection runs again for all faults. 4. One short audit of the contact page of refact.co, the site of the company, shows the dry submit on a real form. diff --git a/README.md b/README.md index 2b3dcc9..0c9edf2 100644 --- a/README.md +++ b/README.md @@ -108,6 +108,7 @@ npm test the type check, the formats, and the seeded site: recurren npm run detect the detection of all 74 seeded faults npm run cycle-eval the whole cycle on one seeded site, and the solution success rate npm run dry-submit the dry submit sends nothing, and finds the form errors (also in npm test) +npm run summary the engine version and the coverage of the process steps (also in npm test) ``` The seeded site is a small correct site with 74 known faults, in `tests/seeded-faults/`. The numbers of the last measurements are in `tests/seeded-faults/*.md`: 74 of 74 faults found, 29 of 29 source files, 42 of 42 solutions that fixed their issue. diff --git a/formats/summary.schema.json b/formats/summary.schema.json index fed2351..656b335 100644 --- a/formats/summary.schema.json +++ b/formats/summary.schema.json @@ -252,6 +252,25 @@ "process" ], "description": "What 'all units' means for this criterion." + }, + "process_units": { + "type": "object", + "additionalProperties": false, + "required": [ + "tested", + "all" + ], + "description": "For a criterion that the processes also test: the steps of the processes, apart from the units of the definition.", + "properties": { + "tested": { + "type": "integer", + "minimum": 0 + }, + "all": { + "type": "integer", + "minimum": 0 + } + } } } }, diff --git a/package-lock.json b/package-lock.json index e352a56..9eaa353 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@refact/a11y", - "version": "1.0.2", + "version": "1.1.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@refact/a11y", - "version": "1.0.2", + "version": "1.1.1", "dependencies": { "ajv": "^8.20.0", "ajv-formats": "^3.0.1", diff --git a/package.json b/package.json index cc0e502..6203c48 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@refact/a11y", - "version": "1.1.0", + "version": "1.1.1", "private": true, "description": "Accessibility Audit: finds what blocks people with disabilities on a website, where each problem comes from, and how to fix it. It does not fix.", "type": "module", @@ -14,7 +14,7 @@ "build": "tsc -p tsconfig.build.json", "prepare": "npm run build", "typecheck": "tsc -p tsconfig.json", - "test": "npm run typecheck && npm run validate-formats && npm run seed-build && node tests/recurrence.mjs --serve && node tests/review.mjs --serve && node tests/seeded-faults/detect.mjs --serve --only F01,F05,F18,F74 && node tests/dry-submit.mjs --serve", + "test": "npm run typecheck && npm run validate-formats && npm run seed-build && node tests/recurrence.mjs --serve && node tests/review.mjs --serve && node tests/seeded-faults/detect.mjs --serve --only F01,F05,F18,F74 && node tests/dry-submit.mjs --serve && node tests/summary.mjs", "audit": "node src/cli.ts audit", "build-criteria": "python3 -I criteria/build.py criteria/criteria.base.json criteria/sources/axe/rule-descriptions.md criteria/sources/act/testcases.json criteria/criteria.json", "validate-formats": "node formats/validate.mjs", @@ -27,7 +27,8 @@ "regroup": "node tools/regroup.mjs", "review": "node tests/review.mjs", "cycle-eval": "node tests/cycle-eval.mjs", - "dry-submit": "node tests/dry-submit.mjs" + "dry-submit": "node tests/dry-submit.mjs", + "summary": "node tests/summary.mjs" }, "dependencies": { "ajv": "^8.20.0", diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index e8bb10f..08411cc 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "accessibility-audit", - "version": "1.1.0", + "version": "1.1.1", "description": "The agent skill of Accessibility Audit: runs the WCAG 2.2 AA audit engine on a site, answers the review questions from the evidence, finds the origin and the source file of each issue, writes a specific solution, and records the decisions of a person. It never fixes the site.", "author": { "name": "Refact", diff --git a/src/audit.ts b/src/audit.ts index 6aee2c1..3f6e3f5 100644 --- a/src/audit.ts +++ b/src/audit.ts @@ -379,10 +379,11 @@ export async function runAudit(options: AuditOptions): Promise { }; await Promise.all(Array.from({ length: concurrency }, () => dryWorker())); // A sample page counts as tested for the form criteria, unless one of its forms checks on the server only or did not run. + // A form that is hidden in a new render shows only after an action or a delay, as a form in a closed dialog: it does not count. const formResult = new Map(dryRuns.map((r) => [r.form.signature, r])); for (const [p, forms] of formsOf) { if (!p.tested) continue; - if (forms.some((f) => { const r = formResult.get(f.signature); return !r || !r.tested || r.serverOnly; })) continue; + if (forms.some((f) => { const r = formResult.get(f.signature); return !r || (!r.tested && !r.hidden) || r.serverOnly; })) continue; for (const c of DRY_SUBMIT_CRITERIA) p.testedCriteria[first]?.add(c); } @@ -444,6 +445,7 @@ export async function runAudit(options: AuditOptions): Promise { const where = `the form '${r.form.name}' on ${pathOf(r.url)}`; if (r.serverOnly) testsNotRun.push(`dry submit: ${where} checks its fields on the server only. Its errors and its success message need a process on staging`); else if (r.unchanged) testsNotRun.push(`dry submit: ${where} did not change after the submission`); + else if (r.hidden) testsNotRun.push(`dry submit: ${where} shows only after an action or a delay, for example in a popup. The dry submit skipped it`); else if (!r.tested) testsNotRun.push(`dry submit: ${where} was not tested: ${r.error}`); } if (!config.interaction.processes.length) testsNotRun.push("processes: the settings have none. The success message of a form and the redundant entry test (3.3.7) run in the processes only"); diff --git a/src/own/dry-submit.ts b/src/own/dry-submit.ts index 44eec92..44e9524 100644 --- a/src/own/dry-submit.ts +++ b/src/own/dry-submit.ts @@ -24,11 +24,20 @@ export interface FormCandidate { selector: string; submit: string; name: string; const TEXT = "input:not([type]), input[type=text], input[type=email], input[type=tel], input[type=url], input[type=password], textarea"; const USABLE = `const usable = (el) => visible(el) && !el.disabled && !el.readOnly && !el.closest("[aria-hidden=true], [inert]");`; const SUBMIT_TYPES = "button[type=submit], button:not([type]), input[type=submit], input[type=image]"; +/** A control in a fixed bar or panel that is outside the screen. A scroll cannot bring it into view, so a click cannot reach it. */ +const UNREACHABLE = `const unreachable = (el) => { + let fixed = false; + for (let a = el; a && a !== document.documentElement; a = a.parentElement) if (getComputedStyle(a).position === "fixed") { fixed = true; break; } + if (!fixed) return false; + const r = el.getBoundingClientRect(); + return r.bottom > innerHeight + 1 || r.top < -1 || r.right > innerWidth + 1 || r.left < -1; +};`; /** The forms of the page: a form element with a text field and a submit control. Not a search form. */ const FIND_FORMS = `(() => { ${HELPERS} ${USABLE} + ${UNREACHABLE} const out = []; for (const form of Array.from(document.forms)) { if (form.getAttribute("role") === "search" || form.closest("[role=search]")) continue; @@ -41,10 +50,20 @@ const FIND_FORMS = `(() => { let submit = [...Array.from(form.querySelectorAll(${JSON.stringify(SUBMIT_TYPES)})), ...owned].find((b) => visible(b) && !b.disabled); if (!submit) submit = Array.from(form.querySelectorAll("button, [role=button], input[type=button]")).find((b) => visible(b) && !b.disabled && /send|submit|subscribe|sign ?up|sign ?in|log ?in|join|register|continue|next|request|get started|apply/i.test(nameOf(b) || b.value || "")); if (!submit) continue; + // A form in a fixed bar or panel that is outside the screen shows only after an action, as a form in a closed dialog. + if (unreachable(submit)) continue; const name = (nameOf(submit) || submit.value || "").trim().replace(/\\s+/g, " ").slice(0, 60); - let action = ""; - if (form.getAttribute("action")) { try { action = new URL(form.action).pathname; } catch { action = form.getAttribute("action"); } } - const signature = [action, (form.getAttribute("method") || "get").toLowerCase(), all.map((e) => (e.type || e.localName) + ":" + (e.getAttribute("name") || e.id || nameOf(e))).sort().join(","), name].join("|"); + // A form that sends to its own page, for example a Gravity Forms form, is the same form on each page. + // The attribute, not form.action: a field with the name "action", common in WordPress forms, hides that property. + let action = "self"; + const raw = form.getAttribute("action"); + if (raw) { + try { const u = new URL(raw, location.href); action = u.origin === location.origin && u.pathname.replace(/\\/$/, "") === location.pathname.replace(/\\/$/, "") ? "self" : u.pathname; } + catch { action = raw; } + } + // The names of the hidden fields tell two forms with the same visible fields apart, for example two lists of one newsletter. + const hidden = Array.from(form.querySelectorAll("input[type=hidden]")).map((e) => e.getAttribute("name") || "").filter(Boolean).sort().join(","); + const signature = [action, (form.getAttribute("method") || "get").toLowerCase(), all.map((e) => (e.type || e.localName) + ":" + (e.getAttribute("name") || e.id || nameOf(e))).sort().join(","), hidden, name].join("|"); out.push({ selector: sel(form), submit: sel(submit), name: name || form.getAttribute("aria-label") || form.id || "form", signature, html: html(form).slice(0, 300), item: item(form), textFields: fields.length }); } return out; @@ -146,12 +165,13 @@ export async function guardedContext(browser: Browser, config: Config, width: nu } /** What a submission showed. browser: the page checked the fields. native: the browser checked them. server: the form tried to - * send, so the site checks on the server only. none: nothing changed. error: the engine could not do the submission. */ -export type SubmissionKind = "browser" | "native" | "server" | "none" | "error"; + * send, so the site checks on the server only. none: nothing changed. hidden: the form is not visible in a new render, because it + * shows only after an action or a delay, for example in a popup. error: the engine could not do the submission. */ +export type SubmissionKind = "browser" | "native" | "server" | "none" | "hidden" | "error"; export interface Submission { variant: "empty" | "incorrect"; kind: SubmissionKind; state: string[]; blocked: string[]; rules: string[]; reason?: string } -export interface DryRun { form: FormCandidate; url: string; submissions: Submission[]; renders: number; states: number; tested: boolean; serverOnly: boolean; unchanged: boolean; error?: string } +export interface DryRun { form: FormCandidate; url: string; submissions: Submission[]; renders: number; states: number; tested: boolean; serverOnly: boolean; unchanged: boolean; hidden: boolean; error?: string } export interface DryOptions { contextOptions?: Record; @@ -194,6 +214,9 @@ async function submitOnce(browser: Browser, config: Config, record: PageRecord, guard.values.push(DRY_VALUE); const present = await page.evaluate(`!!document.querySelector(${JSON.stringify(form.selector)}) && !!document.querySelector(${JSON.stringify(form.submit)})`) as boolean; if (!present) return { variant, kind: "error", state, blocked: [], rules: [], reason: "the form is not on the page in a new render" }; + // A form that the page record found later, for example in a popup that opens after 10 seconds, is hidden in a new render. + const shown = await page.evaluate(`(() => { ${HELPERS} ${UNREACHABLE} const b = document.querySelector(${JSON.stringify(form.submit)}); return !!b && visible(b) && !unreachable(b); })()`) as boolean; + if (!shown) return { variant, kind: "hidden", state, blocked: [], rules: [], reason: "the form is not visible in a new render" }; if (variant === "incorrect") { for (const f of await page.evaluate(TEXT_FIELDS(form.selector)) as string[]) await page.locator(f).first().fill(DRY_VALUE, { timeout: 3000 }).catch(() => {}); } @@ -240,7 +263,7 @@ async function submitOnce(browser: Browser, config: Config, record: PageRecord, /** The dry submit of one form: submission A empty, then submission B with incorrect values when A showed errors on the page. */ export async function runDrySubmit(browser: Browser, config: Config, record: PageRecord, width: number, form: FormCandidate, options: DryOptions): Promise { - const run: DryRun = { form, url: record.url, submissions: [], renders: 0, states: 0, tested: false, serverOnly: false, unchanged: false }; + const run: DryRun = { form, url: record.url, submissions: [], renders: 0, states: 0, tested: false, serverOnly: false, unchanged: false, hidden: false }; const a = await submitOnce(browser, config, record, width, form, "empty", options); run.submissions.push(a); run.renders++; if (a.kind === "browser" && form.textFields > 0) { @@ -250,7 +273,8 @@ export async function runDrySubmit(browser: Browser, config: Config, record: Pag } record.renders += run.renders; run.states = run.submissions.filter((s) => s.kind === "browser").length; - run.tested = a.kind !== "error"; + run.hidden = a.kind === "hidden"; + run.tested = a.kind !== "error" && a.kind !== "hidden"; run.serverOnly = a.kind === "server"; run.unchanged = a.kind === "none"; if (a.kind === "error") run.error = a.reason; diff --git a/src/own/page.ts b/src/own/page.ts index beea438..8c702c6 100644 --- a/src/own/page.ts +++ b/src/own/page.ts @@ -1,11 +1,14 @@ // Own page tests that read the page at rest: language, media, moving content, bypass blocks, target size, // contrast of placeholders and borders, orientation lock, and the content checks that make review questions. +import { readFileSync } from "node:fs"; import type { Page } from "playwright"; import { HELPERS } from "../browser.ts"; import { deciderOf } from "../criteria.ts"; import type { Outcome, Severity } from "../types.ts"; -export const ENGINE_VERSION = "0.1.0"; +/** The version of the package, from src/ and from dist/ alike. A new version makes the next audit a baseline audit, so that + * an issue that a rule change removes is "resolved, rule changed", not a fix. */ +export const ENGINE_VERSION: string = JSON.parse(readFileSync(new URL("../../package.json", import.meta.url), "utf8")).version; export interface Found { criterion: string; rule: string; kind: Outcome["kind"]; selector: string; html?: string; item: string; diff --git a/src/report.ts b/src/report.ts index a7d660c..5ce1647 100644 --- a/src/report.ts +++ b/src/report.ts @@ -40,7 +40,7 @@ export function renderReport(r: ReportInput): string { for (const c of CRITERIA) { const s = r.criteria[c.id]; const q = r.questions.filter((x) => x.criterion === c.id && !x.person_answer && !(x.ai_answer && x.ai_answer.value !== "unsure")).length; - lines.push(`| ${c.id} ${c.name} | ${c.level} | ${s.result} | ${s.method} | ${s.coverage.tested_units} of ${s.coverage.all_units} ${s.coverage.definition} units | ${s.issues} | ${q} |`); + lines.push(`| ${c.id} ${c.name} | ${c.level} | ${s.result} | ${s.method} | ${s.coverage.tested_units} of ${s.coverage.all_units} ${s.coverage.definition} units${s.coverage.process_units ? `, ${s.coverage.process_units.tested} of ${s.coverage.process_units.all} process steps` : ""} | ${s.issues} | ${q} |`); } lines.push(""); diff --git a/src/results.ts b/src/results.ts index f664ed2..2485dae 100644 --- a/src/results.ts +++ b/src/results.ts @@ -40,7 +40,7 @@ export function pageResults(page: PageRecord, widths: number[], decisions: Decis export interface CriterionSummary { result: UnitResult; method: "auto" | "auto+review" | "person"; - coverage: { tested_units: number; all_units: number; definition: "page" | "template" | "site" | "process" }; + coverage: { tested_units: number; all_units: number; definition: "page" | "template" | "site" | "process"; process_units?: { tested: number; all: number } }; issues: number; open_questions: number; ai_only_answers: number; } @@ -59,8 +59,17 @@ export function siteSummary(pages: PageRecord[], widths: number[], processes: { else if (definition === "template") { all = sample.length * widths.length + (crawled.length - tested.length) * widths.length; done = sample.reduce((n, p) => n + widths.filter((w) => p.testedCriteria[w]?.has(c.id)).length, 0); } else if (definition === "site") { all = 1; done = tested.some((p) => widths.some((w) => p.testedCriteria[w]?.has(c.id))) ? 1 : 0; } else { all = processes.all; done = processes.tested; } - out[c.id] = { result: done === 0 && definition !== "process" ? rollUp(results.filter((r) => r !== "not tested")) === "not tested" ? "not tested" : rollUp(results) : rollUp(results), method: c.method, - coverage: { tested_units: done, all_units: all, definition }, issues: issueCounts.get(c.id) || 0, open_questions: questionCounts.get(c.id) || 0, ai_only_answers: aiOnly.get(c.id) || 0 }; + // A criterion that the processes also test, for example 3.3.1 on staging: the steps of the processes are units too. + // They stay apart from the units of the definition, so that the report names both. + let processUnits: { tested: number; all: number } | undefined; + if (definition !== "process" && c.test_groups.includes("process")) { + processUnits = { tested: 0, all: 0 }; + for (const p of pages) if (p.process) for (const w of widths) if (p.testedCriteria[w]) { processUnits.all++; if (p.testedCriteria[w].has(c.id)) processUnits.tested++; } + if (!processUnits.all) processUnits = undefined; + } + const anyDone = done + (processUnits?.tested || 0); + out[c.id] = { result: anyDone === 0 && definition !== "process" ? rollUp(results.filter((r) => r !== "not tested")) === "not tested" ? "not tested" : rollUp(results) : rollUp(results), method: c.method, + coverage: { tested_units: done, all_units: all, definition, ...(processUnits ? { process_units: processUnits } : {}) }, issues: issueCounts.get(c.id) || 0, open_questions: questionCounts.get(c.id) || 0, ai_only_answers: aiOnly.get(c.id) || 0 }; if (definition === "process" && processes.all === 0) out[c.id].result = rollUp(results); } return out; diff --git a/tests/dry-submit.mjs b/tests/dry-submit.mjs index 668265e..444c12a 100644 --- a/tests/dry-submit.mjs +++ b/tests/dry-submit.mjs @@ -77,6 +77,27 @@ ${w.extra || ""} `; } +/** Forms that send to their own page, as Gravity Forms does. "list-1" is on both pages and must get one dry submit. "list-2" has + * the same visible fields and other hidden fields, so it is another form. "bar" is in a fixed bar below the screen, as a closed + * slide-up bar: the engine must skip it. "popup" opens 3 seconds after the load, as a newsletter popup: the page record finds it, + * but a new render does not show it yet, so the dry submit skips it. */ +function selfPage(name) { + const own = `/w/${name}.html`; + const form = (id, n, action, field = "input_1", button = "Subscribe") => `
+ +

+ +
`; + return `Dry submit ${name}
+

Newsletter ${name}

+${form("list-1", 1, name === "self-a" ? own : own + "#gf_1")} +${name === "self-a" ? form("list-2", 2, own + "#gf_2") : ""} +
${form("bar", 3, own, "input_3", "Join the list")}
+${name === "self-b" ? ` +` : ""} +
`; +} + /** Two forms that send nothing: "silent" shows its errors with no live region and keeps the focus, "alert" puts them in role=alert. */ const ANNOUNCE = `Dry submit announce

Announce

@@ -105,6 +126,7 @@ const server = http.createServer((req, res) => { const way = url.pathname.match(/^\/w\/(.+)\.html$/)?.[1]; if (way && WAYS[way]) { res.writeHead(200, { "content-type": "text/html; charset=utf-8" }); res.end(pageOf(way)); return; } if (way === "announce") { res.writeHead(200, { "content-type": "text/html; charset=utf-8" }); res.end(ANNOUNCE); return; } + if (way === "self-a" || way === "self-b") { res.writeHead(200, { "content-type": "text/html; charset=utf-8" }); res.end(selfPage(way)); return; } if (url.pathname === "/api/collect") { res.writeHead(200, { "content-type": url.searchParams.has("c") ? "application/javascript" : "text/plain" }); res.end(url.searchParams.has("c") ? "/* ok */" : "ok"); return; } res.writeHead(404, { "content-type": "text/plain" }); res.end("not found"); }); @@ -164,13 +186,20 @@ config.crawl.random_sample_share = 1; config.crawl.rate_limit = 10; const lines = []; fs.rmSync(path.join(out, "leaks"), { recursive: true, force: true }); -const result = await runAudit({ config, outDir: path.join(out, "leaks"), only: [...pages, `${base}/announce.html`], log: (l) => lines.push(l) }); +const result = await runAudit({ config, outDir: path.join(out, "leaks"), only: [...pages, `${base}/announce.html`, `${base}/self-a.html`, `${base}/self-b.html`], log: (l) => lines.push(l) }); for (const l of lines.filter((l) => l.includes("dry submit"))) console.error(" " + l.trim()); -const sent = record.filter((r) => (r.kind === "http" && r.path === "/api/collect") || r.kind === "ws-message"); +const sent = record.filter((r) => (r.kind === "http" && (r.path === "/api/collect" || r.method === "POST")) || r.kind === "ws-message"); check(sent.length === 0, `the server got no submission (${sent.length}${sent.length ? ": " + sent.map((r) => `${r.method || r.kind} ${r.url || r.body}`).join("; ") : ""})`); check(!record.some((r) => text(r).includes(DRY_VALUE)), "no request carried the typed value"); check(record.some((r) => r.kind === "ws-open"), "the WebSocket of the page opened before the guard"); -check(result.dryRuns.length === 24, `24 forms got a dry submit (${result.dryRuns.length})`); +check(result.dryRuns.filter((r) => r.tested).length === 26, `26 forms got a dry submit (${result.dryRuns.filter((r) => r.tested).length})`); +const popup = result.dryRuns.find((r) => r.form.selector === "#popup"); +check(!!popup && popup.hidden && !popup.tested && popup.submissions[0].kind === "hidden", `the form in a popup that opens later is skipped (${popup ? popup.submissions.map((s) => s.kind).join("/") : "not found by the page record"})`); +const lists = result.dryRuns.filter((r) => r.form.name === "Subscribe"); +check(lists.filter((r) => r.form.selector === "#list-1").length === 1, `the form that sends to its own page got one dry submit for two pages (${lists.map((r) => r.form.selector + " on " + r.url.split("/").pop()).join(", ")})`); +check(lists.filter((r) => r.form.selector === "#list-2").length === 1, "a form with the same visible fields and other hidden fields is another form"); +check(!result.dryRuns.some((r) => r.form.selector === "#bar"), "the form in a fixed bar below the screen is skipped"); +check(lists.every((r) => r.serverOnly && r.submissions[0].blocked.length > 0), `the guard blocked the forms that send to their own page (${lists.map((r) => r.submissions.map((s) => s.kind + " " + s.blocked.join(", ")).join("; ")).join(" | ")})`); const announce = result.pages.find((p) => p.url.endsWith("/announce.html"))?.outcomes.filter((o) => o.rule === "error-not-announced") || []; check(announce.some((o) => o.selector === "#silent"), "4.1.3: the errors of the silent form are not announced"); check(!announce.some((o) => o.selector === "#alert"), "4.1.3: the errors in role=alert count as announced"); @@ -181,8 +210,10 @@ for (const way of Object.keys(WAYS)) { check(!!checked && checked.submissions[0].kind === "browser" && checked.submissions[1]?.kind === "server" && checked.submissions[1].blocked.length > 0, `${way}: errors shown when empty, the guard blocked the send with values (${checked ? checked.submissions.map((s) => s.kind + " " + s.blocked.join(", ")).join("; ") : "not run"})`); } const summary = JSON.parse(fs.readFileSync(path.join(out, "leaks", "summary.json"), "utf8")); -check(summary.coverage.forms_tested === 24, `the coverage gives 24 forms (${summary.coverage.forms_tested})`); -check(summary.coverage.tests_not_run.filter((t) => t.includes("checks its fields on the server only")).length === 11, "the coverage lists the 11 forms that check on the server only"); +check(summary.coverage.forms_tested === 26, `the coverage gives 26 forms (${summary.coverage.forms_tested})`); +check(summary.coverage.tests_not_run.filter((t) => t.includes("checks its fields on the server only")).length === 13, "the coverage lists the 13 forms that check on the server only"); +check(!summary.coverage.tests_not_run.some((t) => t.includes("was not tested")), `no form is "not tested" (${summary.coverage.tests_not_run.filter((t) => t.includes("was not tested")).join("; ") || "none"})`); +check(summary.coverage.tests_not_run.some((t) => t.includes("the form 'Get the guide' on /w/self-b.html shows only after an action or a delay")), "the coverage names the popup form that the dry submit skipped"); await new Promise((r) => server.close(r)); // Part 2: the seeded form faults with production settings and no process. diff --git a/tests/seeded-faults/detection.md b/tests/seeded-faults/detection.md index debbb4a..b06e909 100644 --- a/tests/seeded-faults/detection.md +++ b/tests/seeded-faults/detection.md @@ -1,6 +1,6 @@ # Detection of the seeded faults -Generated by `detect.mjs` on 2026-10-26. "Found" means that the audit reported the criterion of the fault on its page, as the issue or the question that the decider of the fault expects. "Found as" means another kind of finding for the same criterion. "Related" means a finding for a criterion in the "also" list only. +Generated by `detect.mjs` on 2026-11-01. "Found" means that the audit reported the criterion of the fault on its page, as the issue or the question that the decider of the fault expects. "Found as" means another kind of finding for the same criterion. "Related" means a finding for a criterion in the "also" list only. | Faults | Found | Found as another kind | Related only | Missed | |---|---|---|---|---| diff --git a/tests/summary.mjs b/tests/summary.mjs new file mode 100644 index 0000000..ba06e90 --- /dev/null +++ b/tests/summary.mjs @@ -0,0 +1,34 @@ +// The summary: the engine version, and the coverage of a criterion that only the processes tested. +// +// Usage: node tests/summary.mjs +import fs from "node:fs"; +import { ENGINE_VERSION } from "../src/own/page.ts"; +import { siteSummary } from "../src/results.ts"; + +const failures = []; +const check = (ok, message) => { console.error(`${ok ? "ok " : "FAIL"} ${message}`); if (!ok) failures.push(message); }; + +// The version that the audit records is the version of the package. Before 1.1.1 it stayed 0.1.0. +const pkg = JSON.parse(fs.readFileSync(new URL("../package.json", import.meta.url), "utf8")); +check(ENGINE_VERSION === pkg.version, `the engine version is the package version (${ENGINE_VERSION}, ${pkg.version})`); + +// One sample page whose form checks on the server only: the dry submit did not test 3.3.1 there. One process step tested it. +const W = 390; +const page = (url, extra) => ({ url, template: "page", set: "site", states: [], tested: true, links: [], renders: 1, facts: {}, outcomes: [], ...extra }); +const crawled = page("https://example.com/contact/", { sample: true, testedCriteria: { [W]: new Set(["1.1.1", "4.1.2"]) } }); +const step = page("https://example.com/contact/", { sample: true, process: "contact", testedCriteria: { [W]: new Set(["1.1.1", "3.3.1", "3.3.3", "3.3.7"]) } }); +const summary = siteSummary([crawled, step], [W], { all: 1, tested: 1 }, new Map(), new Map()); + +const c331 = summary["3.3.1"]; +check(c331.result === "no issue found", `3.3.1: the process gives "no issue found" (${c331.result})`); +check(c331.coverage.tested_units === 0 && c331.coverage.all_units === 1, `3.3.1: the template units stay 0 of 1 (${c331.coverage.tested_units} of ${c331.coverage.all_units})`); +check(c331.coverage.process_units?.tested === 1 && c331.coverage.process_units?.all === 1, `3.3.1: the coverage names the process step (${JSON.stringify(c331.coverage.process_units)})`); +check(!summary["1.1.1"].coverage.process_units, "1.1.1: a criterion with no process test has no process units"); +check(summary["3.3.7"].coverage.definition === "process" && !summary["3.3.7"].coverage.process_units, "3.3.7: a process criterion counts the processes, as before"); + +// With no process, a criterion that the dry submit could not test stays "not tested". +const alone = siteSummary([crawled], [W], { all: 0, tested: 0 }, new Map(), new Map()); +check(alone["3.3.1"].result === "not tested" && !alone["3.3.1"].coverage.process_units, `3.3.1 with no process: "not tested" (${alone["3.3.1"].result})`); + +if (failures.length) { console.error(`\n${failures.length} checks failed`); process.exit(1); } +console.error("\nall checks passed");