diff --git a/plugins/stardust/.claude-plugin/plugin.json b/plugins/stardust/.claude-plugin/plugin.json index 7d56fd920..234d702fc 100644 --- a/plugins/stardust/.claude-plugin/plugin.json +++ b/plugins/stardust/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "stardust", "description": "Redesign an existing website to make it better. Higher-level guided flow on top of impeccable.", - "version": "0.25.2", + "version": "0.26.0", "author": { "name": "Adobe" }, diff --git a/plugins/stardust/CHANGELOG.md b/plugins/stardust/CHANGELOG.md index 1399087ea..9a8b7ad0f 100644 --- a/plugins/stardust/CHANGELOG.md +++ b/plugins/stardust/CHANGELOG.md @@ -4,6 +4,77 @@ This file starts at 0.14.0. Prior versions (0.3.0 – 0.13.1) are documented in git history only (plus the branch-scoped notes in `CHANGELOG-redesign-adobecom.md` and `CHANGELOG-delivery-media-fidelity.md`). +## 0.26.0 — published-origin gate hardened: element-level criteria beside the pixel number (#125) + +A 96-page rollout of a pharmacy retailer gated every deployed page on stitched captures and +pixel-compare; its coupon-listing page PASSED at 6.7 % / Δh 69 px while all 284 cards were broken — +a fixed-height card with `overflow: hidden`, the body ~30 px low, the description clipped mid-glyph, +the details link pushed under the primary button, the sort control on another default. The links +were in the served DOM: pure CSS geometry, which pixelmatch underweights (16.5 % over the text +boxes alone). "PASS" meant "right shapes at the right places", not "every element present and +legible". Element presence and legibility are checkable — so the gate checks them. Cross-references +#115 (chrome crop gate), #124 (content-cap row) and the run's feedback items A1–A3. + +- **New `replica/scripts/gate-all.mjs` (D0)** — the all-pages published-origin gate: per deployed + page in `state.json`, stitched captures of both sides, `pixel-compare`, then the two DOM probes + below in the same window-free real-Chrome tier; per-page dir + `summary.{json,md}`, `runs/` for + `--only`. **Verdict = pixel % ≤ 10 AND |Δh| ≤ 5 % of origin height AND clipped ≤ 0 (+ documented + allowance) AND content MISSING + HIDDEN links / headings = 0** (+ required repeated units within + 4 px); the pixel-only verdict is recorded beside it — every run is calibration data. Sidecars, each + entry documented: `masks.json`, `overrides.json` (shown beside the number, never replacing it), + `clip-allow.json`, `presence.json`, `units.json`. Origin fallback live → previous → crawl shot + (asymmetric, flagged); `--eds-host` gates a code branch; a union (white-padded) pixel metric was + tried as the height guard and rejected (white gaps score as matches) — the guard is explicit. +- **New `diff/scripts/clip-probe.mjs` (D1)** — text lines and controls against every overflow- + clipping ancestor: TEXT CLIPPED / HIDDEN, CONTROL HIDDEN / CLIPPED counted (exit 2); line-clamp, + "read more" collapsibles (`--more-words`), scrollable containers, horizontal cuts advisory; + collapsed menus, sr-only, hidden subtrees, off-page boxes never reported. Emits the visible text + line boxes. Recorded page: 379; the live origin and two known-good pages: 0. +- **New `diff/scripts/content-presence.mjs` (D2)** — + visible headings, links (visible text), buttons, images, text blocks per band (h1–h3 aligned by + LCS) + control state (`--count-words`); MISSING / HIDDEN LINK ×n and HEADING 🔴, buttons and + CONTROL STATE 🟠, MOVED / EXTRA / COUNT / HEADING AS TEXT 🟡; symmetric scope, chrome left to + the crop gate unless `--chrome`, `--variable` regions as counts (HIDDEN still counts), origin + HTTP ≥ 400 fails loud (exit 4). Recorded page: HIDDEN 284, CONTROL STATE 2. +- **New `diff/scripts/unit-geometry.mjs` (D3) + `diff/scripts/measure-live.mjs`** (library) — the shared + settle / session / shadow-DOM measurement lifted from the project measure scripts; per-element + Δx / Δy / Δw / Δh of the first N repeated units relative to the unit, `hidden` where clipped, + origin cached per slug. Recorded card: body +28…+47 px, details link +28 and hidden, badge + 70×70 at (−10,−10) vs 64×64 at (−27,−30), card 260 → 238. +- **`pixel-compare.mjs --text-boxes` (D4, auxiliary)**; `textBoxPct` exported. (`--pad`, the rejected + union metric, is not shipped.) +- **The pixel-table rule.** Every crafted prototype and every deployed page is a row: + `gate-all --stage prototype --proto-base ` writes `gates/prototypes-/summary.{json,md}` at the + end of Phase 4 (reusing each archetype's cached `live.png`), the default stage writes `gates/all-/`; + `gate-evidence.mjs` reads the tables as the source of record (`--tables`; a page without a row is + `OPEN: no table row`, `--check` names a missing table), the eval requires the prototype table, rollout + Phase H reports both totals. `gate.sh --full` runs clip-probe on the build side in every regime, + content-presence in the published regime, unit-geometry when `stardust/replica/units.json` declares a + repeated-unit family for the slug (`{ family: { origin, build, n, required, pages[], templates[] } }` + — replica recreation-procedure § Repeated-unit families); element lines fail the round and outrank a + pixel PASS in gate-evidence the way the overflow assert does. +- **Coverage carries the verdict; the roster run is a recorded unit** (review notes on the PR). + `update-coverage.mjs --gate ` writes each row into `delivery.gate` and flips a failing + page (no documented override) to `failed`; `verify.mjs` never marks a page `verified` while + `delivery.gate.pass` is false — completion derives from one place. Handoff contract § 3: each cluster + subagent runs `gate-all --only` over its own pages inside the fan-out; C-final's roster run is unit + `gate-all` (recorded `running` → `done`, resumable at its boundary) followed by `--gate`. Setup probes + the deps before installing them (a harness that resolves them leaves the code repo untouched). +- **Upstreamed feedback.** A1 `launchStealthHeaded` is WINDOW-FREE by default + (`STARDUST_HEADED_WINDOW=1` opts in). The tier is the BEST AVAILABLE and REGISTERED: Chrome when + installed, else bundled Chromium with a warning, and every probe records `tier` (`chrome` | + `chromium-fallback` | `chromium`) in its evidence; gate-all's summary has a `browser` column and + counts degraded pages. A2 Akamai's HTTP + 400 escalation is a challenge marker. A3 setup steps install `playwright pixelmatch pngjs cheerio` + as devDependencies, never `--no-save`. `deploy-batch.mjs` appends the admin `x-error` header to + 4xx (the DA 200-images-per-document cap; protocol step 3a'). +- **Docs** — gate doc § The all-pages published-origin gate; replica Phase 5 + Setup; deploy + Step 10 item 7; rollout Phase E; diff SKILL § The published-origin probes; scripts index. **Tests** + for the four instruments (pure parts browser-free; clip-probe end-to-end on a fixture where + playwright resolves). Calibration on the recorded roster: pixel-only 66 PASS → full 28 PASS. +- **Open** — default tolerances (`--clip-max 0`, `--unit-tol 4`, 25 % counts), how projects declare + repeated-unit block families, whether buttons / control state should block. + ## 0.25.2 — content cap: the container sizing model is measured, persisted and gated at a derived wide width (#124) A hands-off replica run delivered a site whose live pages centre their content in two nested caps — diff --git a/plugins/stardust/evals/lint/script-help.mjs b/plugins/stardust/evals/lint/script-help.mjs index 0a4dbec05..f4172f2d4 100644 --- a/plugins/stardust/evals/lint/script-help.mjs +++ b/plugins/stardust/evals/lint/script-help.mjs @@ -34,6 +34,7 @@ const EXEMPT = { 'diff/scripts/content-inventory.mjs': 'library: the diff skill\'s copy of the inventory classifier, imported by content-diff; no CLI', 'diff/scripts/diff-profiles.mjs': 'library: the diff skill\'s copy of the stack profiles, imported by content-diff and visual-diff; no CLI', 'diff/scripts/live-session.mjs': 'library: live-site browser session helpers (UA, stealth launch, overlay dismissal); no CLI', + 'diff/scripts/measure-live.mjs': 'library: settle + rect/type measurement shared by clip-probe, content-presence, unit-geometry; no CLI', 'dynamics/scripts/lib.mjs': 'library: shared arg/io/playwright helpers for the dynamics instruments; no CLI', 'qa/scripts/lib.mjs': 'library: shared helpers for qa.mjs and its checks; no CLI', 'rollout/scripts/lib.mjs': 'library: shared IO + roll-up helpers for the rollout scripts; no CLI', diff --git a/plugins/stardust/evals/replica-source-fidelity/criteria.json b/plugins/stardust/evals/replica-source-fidelity/criteria.json index 70a08d6fe..ab876577f 100644 --- a/plugins/stardust/evals/replica-source-fidelity/criteria.json +++ b/plugins/stardust/evals/replica-source-fidelity/criteria.json @@ -25,7 +25,7 @@ { "name": "source_fidelity_gate_runs", "max_score": 20, - "description": "The source-fidelity gate runs all three probes (content-diff and visual-diff with --profile generic against the live URL, plus stitch-shot + pixel-compare) and its evidence (per-iteration metrics, band breakdown, height delta, diff images) is written under stardust/replica/." + "description": "The source-fidelity gate runs all three probes (content-diff and visual-diff with --profile generic against the live URL, plus stitch-shot + pixel-compare) and its evidence (per-iteration metrics, band breakdown, height delta, diff images) is written under stardust/replica/; --full rounds carry the clip-probe line, and every crafted prototype is a row in stardust/replica/gates/prototypes-/summary.json (gate-all --stage prototype) — a prototype without a row is ungated." }, { "name": "both_breakpoints_gated", diff --git a/plugins/stardust/evals/replica-source-fidelity/task.md b/plugins/stardust/evals/replica-source-fidelity/task.md index b771aa422..ed335d006 100644 --- a/plugins/stardust/evals/replica-source-fidelity/task.md +++ b/plugins/stardust/evals/replica-source-fidelity/task.md @@ -57,6 +57,8 @@ The stardust `replica` skill is invoked. It: - "Improving" the design outside the inconsistency register. - DOM-copying the source page instead of re-authoring. - Declaring fidelity without gate evidence on disk (metrics, diff artifacts). +- A crafted prototype with no row in `stardust/replica/gates/prototypes-/summary.json` + (the pixel table is the source of record; a sidecar claim alone is not evidence). - Only gating desktop (mobile is not free — the 360 pass is required). - Skipping the content-cap row — a container cap wider than 1440 is invisible to both pixel gates. - Rehosting a licensed brand font. diff --git a/plugins/stardust/skills/deploy/SKILL.md b/plugins/stardust/skills/deploy/SKILL.md index 00d87eeb4..c03f5888f 100644 --- a/plugins/stardust/skills/deploy/SKILL.md +++ b/plugins/stardust/skills/deploy/SKILL.md @@ -967,6 +967,8 @@ After deploy, reconcile the EDS page against the source prototype on the **DEPLO **6. Geometry-fix verification hygiene (#117).** One field "parity verified" claim was wrong three ways at once; when verifying any geometry fix: (a) probe the **rule-bearing element**, not a heuristic match ("white column wider than 400px" happened to select a different box than the one carrying the lifted rule) — pair the element the fixed rule targets on the build with the element whose source rule was lifted on live; (b) verify **serving cache-free** — a reviewer's DevTools showed the old rule at its old line number while both hosts already served the fix; check out-of-band with `curl --compressed | grep ''` (block CSS is gzip-encoded — a bare `curl | grep` scans binary and silently matches nothing) and re-render in a **fresh headless context**, telling any human reviewer to hard-refresh before re-judging; (c) a reviewer's screenshot encodes their zoom — **back-compute their CSS viewport** from any element with a known percentage rule (a card at 851px under `width: 50%` → viewport 1702px) and reproduce THAT viewport headlessly; judged at face value their numbers contradicted a correct fix, reproduced at 1702 live and build matched to the pixel. +**7. All-pages published-origin gate at delivery (#125).** Step 10 is per page; once the roster is deployed, run `node stardust/scripts/replica/gate-all.mjs` (project root, through `run-bg.mjs`). DELIVERED = pixel % ≤ 10 AND |Δh| ≤ 5 % AND 0 clipped text / controls AND 0 MISSING / HIDDEN links / headings — a recorded page passed the pixel bar with every card clipped and its links hidden by a fixed-height `overflow: hidden` block. `clip.json` groups name the clipper (fix that block's CSS); `content.json` HIDDEN = CSS, MISSING = encoder. Rules: `../replica/reference/source-fidelity-gate.md` § The all-pages published-origin gate. + The retired `visual-diff` classes are covered elsewhere: stretched images by the `img { height:auto }` reset (#36) + eyeball; dropped max-width wraps and the outer cap by the qa-gate wide pass (#13/#124); blank/broken renders by the computed-style guard; imagery gaps by the `.plain.html` img/alt count (#75) + eyeball. `content-diff` is stack-agnostic via `skills/deploy/scripts/diff-profiles.mjs` (`--profile eds | generic`); it shares the role classifier with `block-roundtrip`/`section-schema` in `skills/deploy/scripts/content-inventory.mjs`. diff --git a/plugins/stardust/skills/deploy/da-deploy-protocol.md b/plugins/stardust/skills/deploy/da-deploy-protocol.md index bcbdba698..12d1f0a8d 100644 --- a/plugins/stardust/skills/deploy/da-deploy-protocol.md +++ b/plugins/stardust/skills/deploy/da-deploy-protocol.md @@ -56,6 +56,9 @@ done curl -sS -X POST -H "Authorization: Bearer $TOKEN" \ "https://admin.hlx.page/preview/$ORG/$REPO/$BRANCH/$P" # expect 200 +# 3a'. ANY admin 4xx: the reason is in the `x-error` response header, not the body +# (deploy-batch appends it to lastError). `… N of 200 images` = the DA pipeline caps a +# document at 200 images (#125) → split long grids into /fragments/ documents. # 3a. preview 409 "error from content-bus" — the error is OPAQUE (no per-asset # detail); do NOT dead-end on it. Two cheap diagnostics, in order: # (i) upload a known-good doc to the SAME path and re-preview — separates diff --git a/plugins/stardust/skills/deploy/scripts/deploy-batch.mjs b/plugins/stardust/skills/deploy/scripts/deploy-batch.mjs index 6c7ae714c..267acf351 100644 --- a/plugins/stardust/skills/deploy/scripts/deploy-batch.mjs +++ b/plugins/stardust/skills/deploy/scripts/deploy-batch.mjs @@ -179,7 +179,10 @@ async function call(method, url, { token, body } = {}, { retries = 4, backoffMs try { const res = await fetch(url, { method, headers: { Authorization: `Bearer ${token}` }, body }); status = res.status; - text = status >= 400 ? (await res.text()).slice(0, 200) : ''; + // The admin API puts the REASON of a 4xx in the `x-error` header, not the body ("… N of 200 images": + // the DA pipeline caps a document at 200 images — long grids ride /fragments/ documents; #125). + const xErr = status >= 400 ? (res.headers.get('x-error') || '') : ''; + text = status >= 400 ? `${(await res.text()).slice(0, 200)}${xErr ? ` [x-error: ${xErr.slice(0, 200)}]` : ''}` : ''; } catch (err) { status = 0; text = String(err.message || err); diff --git a/plugins/stardust/skills/diff/SKILL.md b/plugins/stardust/skills/diff/SKILL.md index e8dbd9ae6..4f5001cf2 100644 --- a/plugins/stardust/skills/diff/SKILL.md +++ b/plugins/stardust/skills/diff/SKILL.md @@ -40,9 +40,10 @@ root carries the `placeholder` / `aria-label` / `title` values and the icons. ```bash # Prereq 0: playwright importable from the project root — probe # node -e "import('playwright').then(()=>process.exit(0))" -# and re-install (npm i -D playwright --no-save --legacy-peer-deps) on failure: -# a --no-save install from extract is PRUNED by any later real npm i -# (extract SKILL.md § Setup). Run the copied scripts from the project, not the plugin. +# and on failure install it AS A devDependency (npm i -D playwright pixelmatch pngjs cheerio +# --legacy-peer-deps — never --no-save: a --no-save install is PRUNED by the next real npm i, +# recorded twice in one run; extract SKILL.md § Setup). Run the copied scripts from the project +# root, not the plugin: ESM resolves `playwright` from the script's own location. # Copy the WHOLE skills/diff/scripts/ dir: content-diff imports its local diff-profiles.mjs # AND content-inventory.mjs. (The deploy gates #93/#94 now use their OWN synced copies in # skills/deploy/scripts/ — A6/A2 are independent of this skill; the two copies must stay in @@ -110,6 +111,30 @@ measures the error page, and the flags (BLANK RENDER / content asymmetry) carry signal with **exit 0**. That is the probes' advisory contract: 0 = ran (flags advisory), 1 = probe error, 3 = bot challenge. +## The published-origin probes (#125) + +`content-diff` reconciles a prototype with its build; against a LIVE commerce origin its per-node +findings were false, and the pixel gate passed a page whose every card was clipped +(`../replica/reference/source-fidelity-gate.md` § The all-pages published-origin gate). Three +probes ask the checkable questions; all load both sides in the same window-free real-Chrome tier +and settle them the same way (`scripts/measure-live.mjs`). + +```bash +ORIGIN="https://www.example.com/"; SERVED="https://main--repo--owner.aem.live/" +node stardust/scripts/diff/clip-probe.mjs "$SERVED" [--json clip.json] # D1: exit 2 on cut / hidden text or controls +node stardust/scripts/diff/content-presence.mjs "$ORIGIN" "$SERVED" [--variable "="] # D2: exit 2 on MISSING/HIDDEN link or heading +node stardust/scripts/diff/unit-geometry.mjs "$ORIGIN" "$SERVED" --unit "=" --n 2 # D3: exit 2 on an element off / hidden / missing +``` + +Reading: a 🔴 clip group names the clipper (a fixed-height box with `overflow: hidden`) — fix +that block's CSS, never the content. HIDDEN LINK = in the DOM but clipped (CSS); MISSING LINK = +not served (encoder); CONTROL STATE = same control, another value; COUNT … = session-variable +region, confirm by eye. The origin side fails loud on HTTP ≥ 400 (exit 4) and a bot challenge +(exit 3). Traps: an infinite-scroll origin keeps loading under the settle — mark the region +`--variable`; a live origin without `
` compares whole page against whole page (the scope +line says so), header / footer left to the chrome crop gate unless `--chrome`; the count-phrase +and "read more" heuristics are English word lists (`--count-words`, `--more-words`). + ## Reading content-diff - 🔴 **MISSING CTA / HEADING / EYEBROW** — real dropped content. FIX. A missing eyebrow is most often a segmentation drop where the eyebrow precedes its heading; a missing CTA means the component never rendered the link. These are exactly what the pixel probe cannot see. diff --git a/plugins/stardust/skills/diff/scripts/clip-probe.mjs b/plugins/stardust/skills/diff/scripts/clip-probe.mjs new file mode 100644 index 000000000..94f026dba --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/clip-probe.mjs @@ -0,0 +1,369 @@ +#!/usr/bin/env node +/* eslint-disable import/no-extraneous-dependencies, import/extensions, no-await-in-loop, no-restricted-syntax, brace-style, object-curly-newline, max-len, no-console, no-continue, no-nested-ternary, no-plusplus, no-underscore-dangle, no-restricted-globals */ +/* global __clipChain, __controlOf, __isControl, __lineRects, __norm, __pageRect, __path, __rendered, __sel, __srOnly, __walk */ +/** + * skills/diff/scripts/clip-probe.mjs — the CLIPPING probe of the published-origin gate (#125, D1). + * + * Why: a recorded page passed the pixel gate at 6.7 % with every card broken — a fixed-height card + * with `overflow: hidden` clipped the description mid-glyph and pushed the details link under the + * primary button. The links were in the DOM; pixelmatch underweights small text inside matching + * shapes. Geometry is checkable, so this probe checks it: every text node's line rects and every + * control's box against EVERY overflow-clipping ancestor (nearest first) and the page width. + * + * TEXT CLIPPED / TEXT HIDDEN / CONTROL HIDDEN / CONTROL CLIPPED counted (exit 2) + * TEXT CLAMPED (line-clamp), TEXT|CONTROL COLLAPSED ("read more" toggle), SCROLL-HIDDEN, + * X-CUT / X-HIDDEN (horizontal: carousels, ellipsis) advisory + * collapsed containers (≤ 2 px), sr-only boxes, hidden / opacity-0 subtrees, off-page boxes, + * text inside a control already reported never reported + * + * content-presence.mjs runs the same inventory on both sides; gate-all reads the served count as + * criterion 3. The visible text LINE boxes come out as `textBoxes` (pixel-compare --text-boxes). + * + * Usage: node skills/diff/scripts/clip-probe.mjs [--width 1440] [--min-cut 2] [--main ] + * [--json []] [--max-findings 400] [--advisory] [--more-words ] [--plain] + * [--warmup ] [--locale en-US] + * Exit: 0 nothing counted, 2 counted findings, 1 error, 3 bot challenge. Requires playwright. + * `clipInventoryInPage`, `IN_PAGE_LIB`, `inPage`, `summarize`, `formatTable` are exported. + */ +import { mkdirSync, realpathSync, writeFileSync } from 'node:fs'; +import { dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { browserTier, openBrowser, openPage, visit } from './measure-live.mjs'; + +const HELP = `clip-probe — text and controls cut or hidden by an overflow ancestor on a served page (#125 D1) + +Usage: node clip-probe.mjs [options] + --width viewport width (default 1440) + --min-cut ignore partial cuts below this (default 2) + --main restrict to this root (default whole document) + --json [] JSON (counts, findings, groups, textBoxes) on stdout or to + --max-findings cap the finding list (default 400) + --advisory print the advisory kinds too (CLAMPED, COLLAPSED, SCROLL-HIDDEN, X-CUT, X-HIDDEN) + --more-words "read more" toggle labels (default English: read|show|see|view|load + more|all|full|less) + --plain bundled Chromium instead of the window-free real-Chrome tier + --warmup visit this URL first (bot-managed sites) + --locale default en-US + --help this text +Counted (exit 2): TEXT CLIPPED, TEXT HIDDEN, CONTROL HIDDEN, CONTROL CLIPPED. Exit 3 = bot challenge.`; + +// ---- in-page library -------------------------------------------------------------------------------- +// Helper declarations injected in front of every in-page function of the gate probes (content-presence +// and unit-geometry reuse them), so the clipping model is ONE piece of code. Playwright serialises a +// function's source only, so shared helpers must travel as text: inPage(fn, arg) builds the expression. +export const IN_PAGE_LIB = ` +const __cs = new WeakMap(); +function __style(el) { let s = __cs.get(el); if (!s) { s = getComputedStyle(el); __cs.set(el, s); } return s; } +function __parent(el) { return el.parentElement || (el.parentNode && el.parentNode.host) || null; } +function __rendered(el) { + for (let a = el; a && a.nodeType === 1; a = __parent(a)) { + const s = __style(a); + if (s.display === 'none' || s.visibility === 'hidden' || s.visibility === 'collapse' || parseFloat(s.opacity) === 0) return false; + if (a.tagName === 'BODY') break; + } + return true; +} +function __pageRect(r) { return { l: r.left + window.scrollX, t: r.top + window.scrollY, r: r.right + window.scrollX, b: r.bottom + window.scrollY, w: r.width, h: r.height }; } +// nearest clipping ancestor of el (el itself excluded): { el, rect, mode: 'hidden'|'scroll'|'collapsed'|'srOnly', clamp } or null +function __srOnly(el) { + const s = __style(el); const br = el.getBoundingClientRect(); + return (s.clipPath && s.clipPath !== 'none') || (s.clip && s.clip !== 'auto') || (br.width <= 1 && br.height <= 1 && s.overflow === 'hidden'); +} +// every clipping ancestor, nearest first (el itself included when inclusive — a text node's own element clips +// its lines: a 30 px 'p.desc { overflow: hidden }' inside a 170 px body). Stops at a collapsed / sr-only box +// (the whole subtree is a state, not a clip defect) by returning { skip: true }. +function __clipChain(el, inclusive) { + const chain = []; + for (let a = inclusive ? el : __parent(el); a && a.nodeType === 1 && a.tagName !== 'BODY' && a.tagName !== 'HTML'; a = __parent(a)) { + const s = __style(a); + const clips = (v) => v === 'hidden' || v === 'clip' || v === 'scroll' || v === 'auto'; + if (!clips(s.overflowX) && !clips(s.overflowY)) continue; + const c = __clipOf(a, true); + if (!c) continue; + if (c.mode === 'collapsed' || c.mode === 'srOnly') return { skip: true, chain }; + chain.push(c); + } + return { skip: false, chain }; +} +function __clipOf(el, inclusive) { + for (let a = inclusive ? el : __parent(el); a && a.nodeType === 1 && a.tagName !== 'BODY' && a.tagName !== 'HTML'; a = __parent(a)) { + const s = __style(a); + const ox = s.overflowX, oy = s.overflowY; + const clips = (v) => v === 'hidden' || v === 'clip' || v === 'scroll' || v === 'auto'; + if (!clips(ox) && !clips(oy)) continue; + const br = a.getBoundingClientRect(); + const cw = a.clientWidth || br.width, ch = a.clientHeight || br.height; + const l = br.left + (a.clientLeft || 0) + window.scrollX, t = br.top + (a.clientTop || 0) + window.scrollY; + const rect = { l, t, r: l + cw, b: t + ch, w: cw, h: ch }; + const srOnly = (s.clipPath && s.clipPath !== 'none') || (s.clip && s.clip !== 'auto') || (cw <= 1 && ch <= 1); + const mode = srOnly ? 'srOnly' : (cw <= 2 || ch <= 2) ? 'collapsed' : (ox === 'scroll' || ox === 'auto' || oy === 'scroll' || oy === 'auto') ? 'scroll' : 'hidden'; + const clamp = s.webkitLineClamp && s.webkitLineClamp !== 'none'; + const toggle = mode === 'hidden' && !clamp && __collapsible(a); + return { el: a, rect, mode, clamp, toggle, axes: { x: clips(ox), y: clips(oy) } }; + } + return null; +} +// a clipper that is a "Read more" / "Show all" collapsible: it (or its parent) holds or is followed by a +// control with aria-expanded="false" or a show-more label. Whole lines behind it are a STATE the visitor can +// open, not a defect — recorded: 13 product pages read 4–46 TEXT HIDDEN inside their collapsed accordions. +const __MORE = typeof __MORE_OVERRIDE !== 'undefined' ? __MORE_OVERRIDE : /\\b(read|show|see|view|load)\\s+(more|all|full|less)\\b|\\bmore\\b\\s*$|expand/i; +function __collapsible(clipEl) { + const isToggle = (c) => c.getAttribute('aria-expanded') === 'false' || c.getAttribute('aria-controls') || __MORE.test(__norm(c.textContent || c.getAttribute('aria-label') || '').slice(0, 40)); + const CTRL = 'button, [role="button"], a[href], summary, [aria-expanded]'; + // the toggle sits inside the box, right AFTER it, or in a small wrapper around it (≤ 3 children) — never + // before it or anywhere in a large ancestor, or every box on a page with one "Read more" reads collapsible + if ([...clipEl.querySelectorAll(CTRL)].some(isToggle)) return true; + const next = clipEl.nextElementSibling; + if (next && (isToggle(next) || [...next.querySelectorAll(CTRL)].some(isToggle))) return true; + const parent = __parent(clipEl); + if (parent && parent.children.length <= 3 && [...parent.querySelectorAll(CTRL)].some(isToggle)) return true; + return false; +} +function __lineRects(node) { + const range = document.createRange(); range.selectNodeContents(node); + return [...range.getClientRects()].filter((r) => r.width > 0 && r.height > 0).map(__pageRect); +} +function __sel(el) { + if (!el || el.nodeType !== 1) return ''; + const tag = el.tagName.toLowerCase(); + if (el.id) return tag + '#' + el.id; + const cls = (typeof el.className === 'string' ? el.className : '').trim().split(/\\s+/).filter(Boolean).slice(0, 2); + return tag + (cls.length ? '.' + cls.join('.') : ''); +} +function __path(el, depth) { + const parts = []; + for (let a = el, i = 0; a && a.nodeType === 1 && a.tagName !== 'BODY' && i < (depth || 4); a = __parent(a), i += 1) parts.unshift(__sel(a)); + return parts.join(' > '); +} +function __norm(s) { return (s || '').replace(/\\s+/g, ' ').trim().toLowerCase(); } +// text a sighted visitor can read inside el: text nodes whose element is rendered, not sr-only, on-page +function __visibleText(el) { + const pageW = document.documentElement.clientWidth; const parts = []; + const stack = [el]; + while (stack.length) { + const n = stack.pop(); + if (n.nodeType === 3) { if (n.textContent.trim()) parts.push(n.textContent); continue; } + if (n.nodeType !== 1) continue; + const tag = n.tagName; + if (tag === 'SCRIPT' || tag === 'STYLE' || tag === 'NOSCRIPT' || tag === 'TEMPLATE' || tag === 'SVG' || tag === 'svg') continue; + const s = __style(n); + if (s.display === 'none' || s.visibility === 'hidden' || parseFloat(s.opacity) === 0 || __srOnly(n)) continue; + const br = n.getBoundingClientRect(); + if (br.width > 0 && (br.right + window.scrollX <= 0 || br.left + window.scrollX >= pageW)) continue; + if (n.shadowRoot) for (let i = n.shadowRoot.childNodes.length - 1; i >= 0; i -= 1) stack.push(n.shadowRoot.childNodes[i]); + for (let i = n.childNodes.length - 1; i >= 0; i -= 1) stack.push(n.childNodes[i]); + } + return parts.join(' ').replace(/\\s+/g, ' ').trim(); +} +function __isControl(el) { + if (!el || el.nodeType !== 1) return false; + const t = el.tagName; + if (t === 'A') return el.hasAttribute('href'); + if (t === 'BUTTON' || t === 'SUMMARY' || t === 'SELECT') return true; + if (t === 'INPUT') return /^(button|submit|reset)$/i.test(el.type); + const role = el.getAttribute('role'); + return role === 'button' || role === 'link' || role === 'tab' || role === 'menuitem'; +} +function __controlOf(el) { for (let a = el; a && a.nodeType === 1; a = __parent(a)) { if (__isControl(a)) return a; if (a.tagName === 'BODY') break; } return null; } +function* __walk(root) { + const stack = [root]; + while (stack.length) { + const n = stack.pop(); + yield n; + if (n.nodeType === 1) { + if (n.shadowRoot) for (let i = n.shadowRoot.childNodes.length - 1; i >= 0; i -= 1) stack.push(n.shadowRoot.childNodes[i]); + const tag = n.tagName; + if (tag === 'SCRIPT' || tag === 'STYLE' || tag === 'NOSCRIPT' || tag === 'TEMPLATE' || tag === 'SVG' || tag === 'svg' || tag === 'SELECT' || tag === 'TEXTAREA' || tag === 'TITLE') continue; + for (let i = n.childNodes.length - 1; i >= 0; i -= 1) stack.push(n.childNodes[i]); + } + } +} +`; + +/** Build a page.evaluate expression: the library, then fn applied to a JSON-inlined arg. */ +export function inPage(fn, arg = {}, prelude = '') { return `(() => { ${prelude}\n${IN_PAGE_LIB}\n return (${fn.toString()})(${JSON.stringify(arg)}); })()`; } +/** The "read more" word list is English by default; --more-words replaces it (both probes). */ +export const morePrelude = (re) => (re ? `const __MORE_OVERRIDE = new RegExp(${JSON.stringify(re)}, 'i');` : ''); + +/** + * In-page (needs IN_PAGE_LIB): page.evaluate(inPage(clipInventoryInPage, { minCut, rootSel, maxFindings })) + * → { docH, pageW, counts, findings, textBoxes } + */ +export function clipInventoryInPage({ minCut = 2, rootSel = null, maxFindings = 400 } = {}) { + const root = (rootSel && document.querySelector(rootSel)) || document.body; + const pageW = document.documentElement.clientWidth; + const docH = document.documentElement.scrollHeight; + const counts = { textClipped: 0, textHidden: 0, controlHidden: 0, controlClipped: 0, clamped: 0, collapsed: 0, scrollHidden: 0, horizontal: 0, offPage: 0, textNodes: 0, controls: 0, total: 0 }; + const findings = []; + const textBoxes = []; + const reported = new Set(); + const push = (f) => { if (findings.length < maxFindings) findings.push(f); }; + const overlap = (a0, a1, b0, b1) => Math.max(0, Math.min(a1, b1) - Math.max(a0, b0)); + const rnd = (o) => Object.fromEntries(Object.entries(o).map(([k, v]) => [k, typeof v === 'number' ? Math.round(v) : v])); + // per-rect visibility against EVERY clipping ancestor, nearest first (vertical = the defect axis) and the + // page width (horizontal). A line hidden by the nearest clipper is attributed to it (a line-clamp there is + // design); a line no single clipper hides but one cuts across is a partial cut by that clipper. + const judge = (rects, chain) => { + let partialCut = 0; let hiddenV = 0; let visibleV = 0; let hcut = 0; let hiddenH = 0; let offPage = 0; let by = null; let hiddenBy = null; + for (const r of rects) { + if (r.r <= 0 || r.l >= pageW) { offPage += 1; continue; } + if (r.r > pageW + minCut) hcut = Math.max(hcut, r.r - pageW); + let hidden = null; let cut = 0; let cutBy = null; let hHidden = false; + for (const clip of chain) { + const vis = clip.axes.y ? overlap(r.t, r.b, clip.rect.t, clip.rect.b) : r.h; + const hvis = clip.axes.x ? overlap(r.l, r.r, clip.rect.l, clip.rect.r) : r.w; + if (vis <= 0) { hidden = clip; break; } + if (r.h - vis > cut + 0.5 && r.h - vis > minCut) { cut = r.h - vis; cutBy = clip; } + if (hvis <= 0) hHidden = true; + else if (r.w - hvis > minCut) hcut = Math.max(hcut, r.w - hvis); + } + if (hidden) { hiddenV += 1; if (!hiddenBy) hiddenBy = hidden; } + else { visibleV += 1; if (cut > partialCut) { partialCut = cut; by = cutBy; } if (hHidden) hiddenH += 1; } + } + return { partialCut, hiddenV, visibleV, hcut, hiddenH, offPage, n: rects.length, by, hiddenBy }; + }; + for (const node of __walk(root)) { + if (node.nodeType === 1 && __isControl(node)) { + if (!__rendered(node) || __srOnly(node)) continue; + const br = node.getBoundingClientRect(); + if (br.width <= 0 || br.height <= 0) continue; + counts.controls += 1; + const r = __pageRect(br); + const cc = __clipChain(node, false); + if (cc.skip || !cc.chain.length) continue; + const j = judge([r], cc.chain); + if (j.offPage) { counts.offPage += 1; continue; } + const text = __norm(node.textContent || node.getAttribute('aria-label') || node.getAttribute('title') || (node.querySelector('img') && node.querySelector('img').alt) || '').slice(0, 60); + const clip = j.hiddenBy || j.by || cc.chain[0]; + const base = { path: __path(node), text, clipper: __sel(clip.el), rect: rnd(r), clip: rnd(clip.rect) }; + if (j.hiddenV) { + if (clip.mode === 'scroll') { counts.scrollHidden += 1; push({ kind: 'CONTROL SCROLL-HIDDEN', advisory: true, ...base }); } + else if (clip.toggle) { counts.collapsed += 1; reported.add(node); push({ kind: 'CONTROL COLLAPSED', advisory: true, ...base }); } + else { counts.controlHidden += 1; reported.add(node); push({ kind: 'CONTROL HIDDEN', cut: Math.round(r.h), ...base }); } + } else if (j.partialCut) { + if (clip.mode === 'scroll') { counts.scrollHidden += 1; push({ kind: 'CONTROL SCROLL-HIDDEN', advisory: true, cut: Math.round(j.partialCut), ...base }); } + else { counts.controlClipped += 1; reported.add(node); push({ kind: 'CONTROL CLIPPED', cut: Math.round(j.partialCut), ...base }); } + } else if (j.hiddenH) { counts.horizontal += 1; push({ kind: 'CONTROL X-HIDDEN', advisory: true, ...base }); } + else if (j.hcut) { counts.horizontal += 1; push({ kind: 'CONTROL X-CUT', advisory: true, cut: Math.round(j.hcut), ...base }); } + continue; + } + if (node.nodeType !== 3) continue; + const text = node.textContent; + if (!text || !text.trim()) continue; + const parent = node.parentElement || (node.parentNode && node.parentNode.host) || null; + if (!parent || !__rendered(parent)) continue; + const ptag = parent.tagName; + if (ptag === 'OPTION' || ptag === 'OPTGROUP' || ptag === 'TEXTAREA') continue; + if (__srOnly(parent)) continue; // the sr-only pattern carried by the text's own element + const lines = __lineRects(node); + if (!lines.length) continue; + counts.textNodes += 1; + const cc = __clipChain(parent, true); + if (cc.skip) continue; + const ctrl = __controlOf(parent); + if (ctrl && reported.has(ctrl)) continue; // the control already carries the finding + const j = judge(lines, cc.chain); + if (j.offPage === j.n) { counts.offPage += 1; continue; } + // visible line boxes → textBoxes (D4 input) + for (const r of lines) { + if (r.r <= 0 || r.l >= pageW) continue; + if (cc.chain.some((clip) => clip.axes.y && overlap(r.t, r.b, clip.rect.t, clip.rect.b) <= 0)) continue; + textBoxes.push({ x: Math.round(r.l), y: Math.round(r.t), w: Math.round(r.w), h: Math.round(r.h) }); + } + const clip = j.by || j.hiddenBy || cc.chain[0] || null; + const base = { path: __path(parent), text: __norm(text).slice(0, 60), clipper: clip ? __sel(clip.el) : null, rect: rnd(lines[0]), clip: clip ? rnd(clip.rect) : null, lines: lines.length }; + if (j.partialCut) { + if (j.by.mode === 'scroll') { counts.scrollHidden += 1; push({ kind: 'TEXT SCROLL-HIDDEN', advisory: true, cut: Math.round(j.partialCut), ...base }); } + else { counts.textClipped += 1; push({ kind: 'TEXT CLIPPED', cut: Math.round(j.partialCut), hiddenLines: j.hiddenV, ...base }); } + } else if (j.hiddenV) { + if (j.hiddenBy.mode === 'scroll') { counts.scrollHidden += 1; push({ kind: 'TEXT SCROLL-HIDDEN', advisory: true, hiddenLines: j.hiddenV, ...base }); } + else if (j.hiddenBy.clamp) { counts.clamped += 1; push({ kind: 'TEXT CLAMPED', advisory: true, hiddenLines: j.hiddenV, ...base }); } + else if (j.hiddenBy.toggle) { counts.collapsed += 1; push({ kind: 'TEXT COLLAPSED', advisory: true, hiddenLines: j.hiddenV, ...base }); } + else { counts.textHidden += 1; push({ kind: 'TEXT HIDDEN', hiddenLines: j.hiddenV, cut: Math.round(lines[0].h), ...base }); } + } else if (j.hiddenH || j.hcut) { counts.horizontal += 1; push({ kind: j.hiddenH ? 'TEXT X-HIDDEN' : 'TEXT X-CUT', advisory: true, cut: Math.round(j.hcut), ...base }); } + } + counts.total = counts.textClipped + counts.textHidden + counts.controlHidden + counts.controlClipped; + return { docH, pageW, counts, findings, textBoxes }; +} + +// ---- pure: grouping + formatting ---------------------------------------------------------------------- + +/** Group findings by kind + clipper + parent selector: [{ kind, advisory, n, clipper, path, text, cutMin, cutMax }] sorted by count. */ +export function summarize(findings) { + const groups = new Map(); + for (const f of findings) { + const key = `${f.kind}|${f.clipper || ''}|${(f.path || '').split(' > ').pop()}`; + let g = groups.get(key); + if (!g) { g = { kind: f.kind, advisory: !!f.advisory, n: 0, clipper: f.clipper, path: f.path, text: f.text, cutMin: Infinity, cutMax: -Infinity }; groups.set(key, g); } + g.n += 1; + if (Number.isFinite(f.cut)) { g.cutMin = Math.min(g.cutMin, f.cut); g.cutMax = Math.max(g.cutMax, f.cut); } + } + return [...groups.values()].map((g) => ({ ...g, cutMin: Number.isFinite(g.cutMin) ? g.cutMin : null, cutMax: Number.isFinite(g.cutMax) ? g.cutMax : null })).sort((a, b) => Number(a.advisory) - Number(b.advisory) || b.n - a.n); +} + +export function formatTable(groups, { advisory = false } = {}) { + const rows = groups.filter((g) => advisory || !g.advisory); + if (!rows.length) return ' (none)'; + const cut = (g) => (g.cutMin == null ? '-' : g.cutMin === g.cutMax ? `${g.cutMin}px` : `${g.cutMin}–${g.cutMax}px`); + return rows.map((g) => ` ${g.advisory ? '🟡' : '🔴'} ${g.kind.padEnd(22)} ×${String(g.n).padEnd(5)} in ${g.clipper || '(page)'} ${g.path} "${g.text}" cut ${cut(g)}`).join('\n'); +} + +export const verdictLine = (counts) => `Clipped: ${counts.total} (text clipped ${counts.textClipped}, text hidden ${counts.textHidden}, controls hidden ${counts.controlHidden}, controls clipped ${counts.controlClipped}; advisory: clamped ${counts.clamped}, collapsed ${counts.collapsed || 0}, scroll-hidden ${counts.scrollHidden}, horizontal ${counts.horizontal})`; + +// ---- CLI --------------------------------------------------------------------------------------------- + +export function parseArgs(argv) { + const rest = argv.slice(2); + if (!rest.length || rest.includes('--help') || rest.includes('-h')) { console.log(HELP); process.exit(0); } + const opts = { url: null, width: 1440, minCut: 2, main: null, json: false, jsonFile: null, maxFindings: 400, advisory: false, plain: false, warmup: null, locale: 'en-US', moreWords: null }; + for (let i = 0; i < rest.length; i += 1) { + const a = rest[i]; + if (a === '--width') opts.width = Number(rest[++i]); + else if (a === '--min-cut') opts.minCut = Number(rest[++i]); + else if (a === '--main') opts.main = rest[++i]; + else if (a === '--json') { opts.json = true; if (rest[i + 1] && !rest[i + 1].startsWith('--')) opts.jsonFile = rest[++i]; } + else if (a === '--max-findings') opts.maxFindings = Number(rest[++i]); + else if (a === '--advisory') opts.advisory = true; + else if (a === '--more-words') opts.moreWords = rest[++i]; + else if (a === '--plain') opts.plain = true; + else if (a === '--warmup') opts.warmup = rest[++i]; + else if (a === '--locale') opts.locale = rest[++i]; + else if (a.startsWith('--')) { console.error(`unknown flag ${a}\n\n${HELP}`); process.exit(1); } + else if (!opts.url) opts.url = a; + } + if (!opts.url) { console.error(`need \n\n${HELP}`); process.exit(1); } + return opts; +} + +/** Load + settle a page and run the inventory (shared with gate-all's --no-content path). */ +export async function probe(page, url, opts) { + const v = await visit(page, url, { warmup: opts.warmup }); + const inv = await page.evaluate(inPage(clipInventoryInPage, { minCut: opts.minCut, rootSel: opts.main, maxFindings: opts.maxFindings }, morePrelude(opts.moreWords))); + return { url, at: new Date().toISOString(), width: opts.width, status: v.status, settlePasses: v.passes, ...inv, groups: summarize(inv.findings) }; +} + +async function main() { + const opts = parseArgs(process.argv); + const { chromium } = await import('playwright'); + const browser = await openBrowser(chromium, { tier: opts.plain ? 'plain' : 'stealth' }); + let res; + try { + const { ctx, page } = await openPage(browser, { width: opts.width, locale: opts.locale }); + res = await probe(page, opts.url, opts); + res.tier = browserTier(browser); + await ctx.close(); + } finally { await browser.close(); } + const out = { ...res, textBoxes: res.textBoxes }; + if (opts.json && !opts.jsonFile) { console.log(JSON.stringify(out, null, 1)); } + else { + console.log(`clip-probe ${opts.url} @ ${opts.width}px — docH ${res.docH}, ${res.counts.textNodes} text nodes, ${res.counts.controls} controls (HTTP ${res.status}, ${res.tier})`); + console.log(formatTable(res.groups, { advisory: opts.advisory })); + console.log(verdictLine(res.counts)); + if (opts.jsonFile) { mkdirSync(dirname(opts.jsonFile) || '.', { recursive: true }); writeFileSync(opts.jsonFile, JSON.stringify(out, null, 1)); console.log(`json → ${opts.jsonFile}`); } + } + process.exitCode = res.counts.total > 0 ? 2 : 0; +} + +function safeRealpath(p) { try { return realpathSync(p); } catch { return p; } } +if (process.argv[1] && fileURLToPath(import.meta.url) === safeRealpath(process.argv[1])) { + main().catch((e) => { console.error(`clip-probe error: ${String(e.message).split('\n')[0]}`); process.exit(e.name === 'BotChallengeError' ? 3 : 1); }); +} diff --git a/plugins/stardust/skills/diff/scripts/content-presence.mjs b/plugins/stardust/skills/diff/scripts/content-presence.mjs new file mode 100644 index 000000000..717d63106 --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/content-presence.mjs @@ -0,0 +1,370 @@ +#!/usr/bin/env node +/* eslint-disable import/no-extraneous-dependencies, import/extensions, no-await-in-loop, no-restricted-syntax, brace-style, object-curly-newline, max-len, no-console, no-continue, no-nested-ternary, no-plusplus, no-underscore-dangle, no-restricted-globals, newline-per-chained-call, object-property-newline */ +/* global __clipChain, __controlOf, __isControl, __norm, __pageRect, __parent, __path, __rendered, __sel, __srOnly, __visibleText, __walk */ +/** + * skills/diff/scripts/content-presence.mjs — the CONTENT-PRESENCE gate of the published-origin gate + * (#125, D2): live origin vs served page, same minute, same settle, compared per band. + * + * content-diff.mjs's role classifier is tuned to a prototype's DOM; on live commerce origins its + * per-node findings were false. This probe asks the smaller question the pixel gate cannot: is every + * VISIBLE heading, link, button, image and text block of the origin present AND visible on the served + * page, band by band, and does every control show the same state? Both pages load in the same + * window-free real-Chrome tier and settle the same way (measure-live.mjs). Visibility = rendered, + * non-zero box, on-page, not clipped past 50 % (clip-probe's model): in the DOM but clipped = HIDDEN. + * Bands: the visible h1–h3 sequences aligned by text (LCS); items fall into bands by y. + * + * MISSING / HIDDEN HEADING, MISSING / HIDDEN LINK ×n 🔴 (exit 2) + * MISSING / HIDDEN BUTTON, CONTROL STATE 🟠 + * COUNT TEXT|IMAGES|LINKS, MOVED LINK, EXTRA …, HEADING AS TEXT 🟡 + * + * Links and buttons are one pool on the served side; text-less image anchors count as images. Scope + * is symmetric (root only when BOTH sides have one, else whole page); header / footer and everything + * above / below them are left to the chrome crop gate unless --chrome. `--variable ` + * marks session-variable regions (counts only, HIDDEN still counts). The origin side fails loud on + * HTTP ≥ 400 (exit 4) and on a bot challenge (exit 3). Trap: an infinite-scroll origin keeps loading + * under the settle — mark that region --variable. + * + * Usage: node skills/diff/scripts/content-presence.mjs [--width 1440] + * [--main [=]] [--variable ] [--chrome] [--json []] [--min-cut 2] + * [--settle-passes 4] [--max-findings 40] [--count-words ] [--plain] [--warmup ] + * `presenceInventoryInPage`, `alignHeadings`, `diffPresence`, `formatReport`, `norm` are exported. + */ +import { mkdirSync, realpathSync, writeFileSync } from 'node:fs'; +import { dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { clipInventoryInPage, inPage, morePrelude, summarize } from './clip-probe.mjs'; +import { browserTier, openBrowser, openPage, visit } from './measure-live.mjs'; + +const HELP = `content-presence — origin vs served page: visible headings / links / buttons / images / text per band + control state (#125 D2) + +Usage: node content-presence.mjs [options] + --width viewport width (default 1440) + --main [=] content root(s) (default main, else body) + --variable session-variable subtrees, counts only (selO=selE pairs allowed) + --json [] JSON (inventories, bands, findings, totals, clip, textBoxes) + --min-cut clip partial-cut floor (default 2) + --chrome include header / footer items (default: left to the chrome crop gate) + --settle-passes slow-scroll passes until the height is stable (default 4; infinite-scroll + origins keep loading — mark those regions --variable) + --max-findings printed findings per kind (default 40) + --count-words result-count nouns (default English: results, items, products, coupons, offers, reviews, stores, matches) + --more-words "read more" toggle labels for the clip model (default English) + --plain bundled Chromium instead of the window-free real-Chrome tier + --warmup visit this URL first on the ORIGIN side (bot-managed sites) + --locale default en-US + --help this text +Exit: 0 clean, 2 any MISSING/HIDDEN link or heading, 1 error, 3 bot challenge, 4 origin HTTP >= 400.`; + +// ---- in-page inventory (needs IN_PAGE_LIB — run via inPage()) ----------------------------------------- +/** page.evaluate(inPage(presenceInventoryInPage, { rootSel, variableSels })) → { docH, pageW, root, items[] } */ +export function presenceInventoryInPage({ rootSel = null, variableSels = [], countWords = 'results?|items?|products?|coupons?|offers?|reviews?|stores?|matches' } = {}) { + const COUNT_RE = new RegExp(`\\b(\\d[\\d,]*)\\s+(${countWords})\\b`, 'i'); + // Always walk the whole body; each item carries `inRoot`. The differ picks the scope SYMMETRICALLY: root + // only when BOTH sides have one, else the whole page (a live origin without
against a build with + // one would otherwise compare chrome against content). + const root = document.body; + const rootEl = (rootSel && document.querySelector(rootSel)) || (!rootSel && document.querySelector('main')) || null; + // chrome landmarks: items inside them are flagged so the differ can leave the header / footer to the chrome + // crop gate (the chrome repeats on every page and its live promo strips / account state are session-variable) + // Chrome = the landmarks AND everything above the header's bottom edge / below the footer's top edge + // (promo strips often sit OUTSIDE
); the chrome crop gate judges that region, --chrome includes it. + const chromeEls = [...document.querySelectorAll('header, [role="banner"], footer, [role="contentinfo"]')].filter((c) => !rootEl || !rootEl.contains(c)); + const banner = chromeEls.filter((c) => c.matches('header, [role="banner"]')).map((c) => __pageRect(c.getBoundingClientRect())).filter((r) => r.h > 0); + const contentinfo = chromeEls.filter((c) => c.matches('footer, [role="contentinfo"]')).map((c) => __pageRect(c.getBoundingClientRect())).filter((r) => r.h > 0); + const chromeTop = banner.length ? Math.max(...banner.map((r) => r.b)) : 0; + const chromeBottom = contentinfo.length ? Math.min(...contentinfo.map((r) => r.t)) : Infinity; + const inChrome = (el) => { if (chromeEls.some((c) => c.contains(el))) return true; const r = __pageRect(el.getBoundingClientRect()); return r.b <= chromeTop || r.t >= chromeBottom; }; + const pageW = document.documentElement.clientWidth; + const variableEls = variableSels.flatMap((s) => { try { return [...document.querySelectorAll(s)]; } catch { return []; } }); + const inVariable = (el) => variableEls.some((v) => v.contains(el)); + const overlap = (a0, a1, b0, b1) => Math.max(0, Math.min(a1, b1) - Math.max(a0, b0)); + // 'visible' | 'hidden' (clipped past 50 % by the nearest overflow ancestor) | null (not rendered / off-page / collapsed) + const state = (el) => { + if (!__rendered(el) || __srOnly(el)) return null; + const br = el.getBoundingClientRect(); + if (br.width <= 0 || br.height <= 0) return null; + const r = __pageRect(br); + if (r.r <= 0 || r.l >= pageW) return null; + const cc = __clipChain(el, false); + if (cc.skip) return null; + for (const clip of cc.chain) { + if (clip.mode === 'scroll') continue; + const vis = (clip.axes.y ? overlap(r.t, r.b, clip.rect.t, clip.rect.b) : r.h) * (clip.axes.x ? overlap(r.l, r.r, clip.rect.l, clip.rect.r) : r.w); + if (vis / (r.w * r.h) < 0.5) return 'hidden'; + } + return 'visible'; + }; + const rectOf = (el) => { const r = __pageRect(el.getBoundingClientRect()); return { x: Math.round(r.l), y: Math.round(r.t), w: Math.round(r.w), h: Math.round(r.h) }; }; + const label = (el) => __norm(__visibleText(el)) || __norm((el.querySelector('img') || {}).alt) || __norm(el.getAttribute('aria-label')) || __norm(el.getAttribute('title')) || ''; + const file = (src) => { try { return new URL(src, window.location.href).pathname.split('/').pop().replace(/\.[a-z0-9]+$/i, '').slice(0, 40); } catch { return ''; } }; + const items = []; + const push = (kind, el, extra) => { const st = state(el); if (!st) return; items.push({ kind, state: st, variable: inVariable(el), inRoot: rootEl ? rootEl.contains(el) : null, inChrome: inChrome(el), path: __path(el, 3), ...rectOf(el), ...extra }); }; + const textBlocks = new Set(); + const ownText = (el) => [...el.childNodes].filter((n) => n.nodeType === 3).map((n) => n.textContent).join(' ').replace(/\s+/g, ' ').trim(); + const hasCountedAncestor = (el) => { for (let a = __parent(el); a && a !== root; a = __parent(a)) if (textBlocks.has(a)) return true; return false; }; + const controlLabel = (el) => { + const al = __norm(el.getAttribute('aria-label')); + if (al) return al; + if (el.labels && el.labels.length) return __norm(el.labels[0].textContent); + const lb = el.closest && el.closest('label'); + if (lb) return __norm(lb.textContent).slice(0, 60); + const id = el.getAttribute('aria-labelledby'); if (id) { const l = document.getElementById(id); if (l) return __norm(l.textContent); } + return __norm(el.getAttribute('name')) || ''; + }; + for (const el of __walk(root)) { + if (el.nodeType !== 1) continue; + const tag = el.tagName; + if (/^H[1-6]$/.test(tag) || el.getAttribute('role') === 'heading') { + const level = /^H[1-6]$/.test(tag) ? Number(tag[1]) : Number(el.getAttribute('aria-level') || 2); + const text = label(el); if (text) push('heading', el, { level, text: text.slice(0, 120) }); + continue; + } + if (tag === 'A' && el.hasAttribute('href')) { const text = label(el); let href = ''; try { href = new URL(el.getAttribute('href'), window.location.href).pathname; } catch { href = el.getAttribute('href'); } push('link', el, { text: text.slice(0, 80), href }); } + else if (tag === 'BUTTON' || (tag === 'INPUT' && /^(button|submit|reset)$/i.test(el.type)) || el.getAttribute('role') === 'button') { + const text = tag === 'INPUT' ? __norm(el.value) : label(el); + push('button', el, { text: text.slice(0, 80) }); + const pop = el.getAttribute('aria-haspopup'); const exp = el.getAttribute('aria-expanded'); + if ((pop && pop !== 'false') || exp !== null || el.getAttribute('role') === 'combobox') push('control', el, { label: controlLabel(el) || 'trigger', value: text.slice(0, 60), via: 'trigger' }); + } + else if (tag === 'IMG') { const br = el.getBoundingClientRect(); if (br.width >= 16 && br.height >= 16) push('image', el, { text: __norm(el.alt).slice(0, 60), file: file(el.currentSrc || el.src) }); } + else if (tag === 'SELECT') { const o = el.options[el.selectedIndex]; push('control', el, { label: controlLabel(el) || 'select', value: o ? __norm(o.textContent).slice(0, 60) : '', via: 'select' }); } + else if (tag === 'INPUT' && /^(radio|checkbox)$/i.test(el.type)) { if (el.checked) push('control', el, { label: controlLabel(el) || el.type, value: controlLabel(el) || 'checked', via: el.type }); } + else if (el.getAttribute('role') === 'combobox') push('control', el, { label: controlLabel(el) || 'combobox', value: (__norm(el.value) || label(el)).slice(0, 60), via: 'combobox' }); + else if ((el.getAttribute('role') === 'tab' || el.getAttribute('role') === 'option') && el.getAttribute('aria-selected') === 'true') push('control', el, { label: `${el.getAttribute('role')} selected`, value: label(el).slice(0, 60), via: 'aria-selected' }); + else if (el.getAttribute('aria-pressed') === 'true') push('control', el, { label: 'pressed', value: label(el).slice(0, 60), via: 'aria-pressed' }); + else if (el.hasAttribute('aria-current') && el.getAttribute('aria-current') !== 'false') push('control', el, { label: 'current', value: label(el).slice(0, 60), via: 'aria-current' }); + // text blocks: own text ≥ 3 chars, not inside an already counted block, not a control / heading + if (tag !== 'A' && tag !== 'BUTTON' && !__isControl(el)) { + const own = ownText(el); + if (own.length >= 3 && !hasCountedAncestor(el) && !__controlOf(el)) { + textBlocks.add(el); + push('text', el, { text: own.slice(0, 80) }); + const m = own.match(COUNT_RE); + if (m) push('control', el, { label: `count ${m[2].toLowerCase().replace(/s$/, '')}`, value: m[1].replace(/,/g, ''), via: 'count' }); + } + } + } + return { docH: document.documentElement.scrollHeight, pageW, root: rootEl ? __sel(rootEl) : null, items }; +} + +// ---- pure: alignment + diff -------------------------------------------------------------------------- +// Matching key: lower-case; trademark / footnote glyphs and a trailing footnote digit dropped ("no annual +// fee †", "$50/month. 1"); a bare "tm" token dropped (a TM serialises as text on one side and as +// the glyph on the other — 17 false MISSING on one recorded page); punctuation runs collapsed. +export const norm = (s) => (s || '') + .toLowerCase() + .replace(/[®™©†‡§*]/g, ' ') + .replace(/\b(tm|sm)\b/g, ' ') + .replace(/[\u2018\u2019]/g, "'") + .replace(/[\u201c\u201d]/g, '"') + .replace(/\s+/g, ' ') + .replace(/(\D)\s+\d{1,2}\s*$/g, '$1') + .replace(/\s*([.,;:!?])\s*/g, '$1 ') + .replace(/\s+/g, ' ') + .trim(); +const sameHeading = (a, b) => a === b || (a.length >= 8 && b.length >= 8 && (a.startsWith(b) || b.startsWith(a))); + +/** Longest common subsequence over the two visible h1–h3 sequences → [{ oi, ei }] index pairs. */ +export function alignHeadings(origin, eds) { + const n = origin.length; const m = eds.length; + const dp = Array.from({ length: n + 1 }, () => new Array(m + 1).fill(0)); + for (let i = n - 1; i >= 0; i -= 1) for (let j = m - 1; j >= 0; j -= 1) dp[i][j] = sameHeading(origin[i], eds[j]) ? dp[i + 1][j + 1] + 1 : Math.max(dp[i + 1][j], dp[i][j + 1]); + const pairs = []; let i = 0; let j = 0; + while (i < n && j < m) { + if (sameHeading(origin[i], eds[j])) { pairs.push({ oi: i, ei: j }); i += 1; j += 1; } + else if (dp[i + 1][j] >= dp[i][j + 1]) i += 1; else j += 1; + } + return pairs; +} + +const countBy = (arr, key) => { const m = new Map(); for (const it of arr) { const k = key(it); m.set(k, (m.get(k) || 0) + 1); } return m; }; +const linkKey = (it) => norm(it.text) || `href:${it.href}`; + +/** + * diffPresence(originInv, edsInv, { countTol }) → { bands, findings, totals } + * Inventories are the in-page result ({ items }); items hidden on the origin are dropped first. + */ +export function diffPresence(originInv, edsInv, { countTol = 0.25, chrome = false } = {}) { + const scope = originInv.root && edsInv.root ? 'root' : 'page'; + const inScope = (it) => (scope === 'page' || it.inRoot) && (chrome || !it.inChrome); + const O = originInv.items.filter((it) => it.state === 'visible' && inScope(it)); + const E = edsInv.items.filter(inScope); + const bandHeads = (items) => items.filter((it) => it.kind === 'heading' && it.level <= 3 && it.state === 'visible').sort((a, b) => a.y - b.y); + const oh = bandHeads(O); const eh = bandHeads(E); + const pairs = alignHeadings(oh.map((h) => norm(h.text)), eh.map((h) => norm(h.text))); + const findings = []; + const add = (sev, kind, msg, extra = {}) => findings.push({ sev, kind, msg, ...extra }); + // unaligned headings + const alignedO = new Set(pairs.map((p) => p.oi)); const alignedE = new Set(pairs.map((p) => p.ei)); + const edsHeadingsAll = E.filter((it) => it.kind === 'heading'); + const edsVisibleTexts = new Set(E.filter((it) => it.state === 'visible' && (it.kind === 'text' || it.kind === 'link' || it.kind === 'heading')).map((it) => norm(it.text))); + oh.forEach((h, i) => { + if (alignedO.has(i)) return; + const hidden = edsHeadingsAll.find((x) => sameHeading(norm(x.text), norm(h.text)) && x.state === 'hidden'); + if (hidden) add('🔴', 'HIDDEN HEADING', `h${h.level} "${h.text}" is in the served DOM but clipped away (${hidden.path})`, { band: i, text: h.text }); + else if (edsVisibleTexts.has(norm(h.text))) add('🟡', 'HEADING AS TEXT', `origin h${h.level} "${h.text}" is served as plain text / a link, not a heading (visually present — semantics only)`, { band: i, text: h.text }); + else add('🔴', 'MISSING HEADING', `h${h.level} "${h.text}" (origin y ${h.y}) has no served heading`, { band: i, text: h.text }); + }); + eh.forEach((h, j) => { if (!alignedE.has(j)) add('🟡', 'EXTRA HEADING', `served h${h.level} "${h.text}" (y ${h.y}) has no origin heading`, { text: h.text }); }); + // bands: one per aligned heading pair, plus band 0 (above the first aligned heading) + const bounds = (heads, idxs) => { const ys = idxs.map((i) => heads[i].y); return [0, ...ys, Infinity]; }; + const ob = bounds(oh, pairs.map((p) => p.oi)); const eb = bounds(eh, pairs.map((p) => p.ei)); + const bandOf = (b, y) => { let k = 0; while (k + 1 < b.length - 1 && y >= b[k + 1]) k += 1; return k; }; + const bands = []; + for (let k = 0; k < ob.length - 1; k += 1) { + const name = k === 0 ? '(top)' : oh[pairs[k - 1].oi].text; + const oItems = O.filter((it) => bandOf(ob, it.y) === k); const eItems = E.filter((it) => bandOf(eb, it.y) === k); + const variable = oItems.some((it) => it.variable) || eItems.some((it) => it.variable); + const band = { index: k, heading: name, variable, origin: {}, eds: {}, findings: [] }; + for (const kind of ['link', 'button', 'text', 'image']) { + band.origin[kind] = oItems.filter((it) => it.kind === kind).length; + band.eds[kind] = eItems.filter((it) => it.kind === kind && it.state === 'visible').length; + band.eds[`${kind}Hidden`] = eItems.filter((it) => it.kind === kind && it.state === 'hidden').length; + } + bands.push(band); + // links + buttons by text + for (const [kind, sev, missingKind, hiddenKind] of [['link', '🔴', 'MISSING LINK', 'HIDDEN LINK'], ['button', '🟠', 'MISSING BUTTON', 'HIDDEN BUTTON']]) { + // links and buttons are ONE pool on the served side (a "Sign in" button served as a link is the same + // visible control); the ORIGIN kind decides the severity. + const action = (it) => it.kind === 'link' || it.kind === 'button'; + // a text-less link (an image-only anchor without alt) is the image's presence (counted there); its href + // does not survive a migration's path rewrite — text-keyed only. + const oFixed = oItems.filter((it) => it.kind === kind && !it.variable && norm(it.text)); const oVar = oItems.filter((it) => it.kind === kind && it.variable && norm(it.text)); + const eVis = countBy(eItems.filter((it) => action(it) && it.state === 'visible'), linkKey); + const eHid = countBy(eItems.filter((it) => action(it) && it.state === 'hidden'), linkKey); + const eVisAll = countBy(E.filter((it) => action(it) && it.state === 'visible'), linkKey); + const oAll = countBy(O.filter(action), linkKey); + for (const [key, cO] of countBy(oFixed, linkKey)) { + const vE = eVis.get(key) || 0; if (vE >= cO) continue; + let deficit = cO - vE; + const hid = Math.min(deficit, eHid.get(key) || 0); + const sample = oFixed.find((it) => linkKey(it) === key); + if (hid) { add(sev, hiddenKind, `${hid > 1 ? `×${hid} ` : ''}"${sample.text || sample.href}" in band "${name}" — in the served DOM but clipped away (${(eItems.find((it) => action(it) && it.state === 'hidden' && linkKey(it) === key) || {}).path || ''})`, { band: k, n: hid, text: sample.text, href: sample.href }); deficit -= hid; } + if (deficit > 0) { + if ((eVisAll.get(key) || 0) >= (oAll.get(key) || 0)) add('🟡', kind === 'link' ? 'MOVED LINK' : 'MOVED BUTTON', `${deficit > 1 ? `×${deficit} ` : ''}"${sample.text || sample.href}" short in band "${name}" but visible elsewhere on the served page`, { band: k, n: deficit, text: sample.text }); + else add(sev, missingKind, `${deficit > 1 ? `×${deficit} ` : ''}"${sample.text || sample.href}"${sample.href ? ` → ${sample.href}` : ''} in band "${name}" (origin y ${sample.y}) not on the served page`, { band: k, n: deficit, text: sample.text, href: sample.href }); + } + } + if (oVar.length) { const eVarVis = eItems.filter((it) => it.kind === kind && it.variable && it.state === 'visible').length; const eVarHid = eItems.filter((it) => it.kind === kind && it.variable && it.state === 'hidden').length; if (eVarHid) add(sev, hiddenKind, `×${eVarHid} ${kind}s clipped away inside the session-variable region of band "${name}"`, { band: k, n: eVarHid, variable: true }); if (Math.abs(eVarVis - oVar.length) > Math.max(2, countTol * oVar.length)) add('🟡', `COUNT ${kind.toUpperCase()}S`, `session-variable region in band "${name}": ${oVar.length} origin → ${eVarVis} served`, { band: k }); } + // extra + if (kind === 'link') for (const [key, cE] of eVis) { const cO = oAll.get(key) || 0; if (cO === 0 && !key.startsWith('href:')) { const s = eItems.find((it) => action(it) && linkKey(it) === key); add('🟡', 'EXTRA LINK', `${cE > 1 ? `×${cE} ` : ''}"${s.text || s.href}" served in band "${name}", no origin link or button`, { band: k, n: cE, text: s.text }); } } + } + for (const kind of ['text', 'image']) { + const cO = band.origin[kind]; const cE = band.eds[kind]; const hid = band.eds[`${kind}Hidden`]; + if (Math.abs(cE - cO) > Math.max(kind === 'text' ? 3 : 2, countTol * cO) || hid > Math.max(2, countTol * cO)) add('🟡', `COUNT ${kind.toUpperCase()}${kind === 'text' ? '' : 'S'}`, `band "${name}": ${cO} origin → ${cE} served visible${hid ? ` (+${hid} clipped away)` : ''}`, { band: k, origin: cO, eds: cE, hidden: hid }); + } + // control state: by label, else by ordinal among generic triggers + const oC = oItems.filter((it) => it.kind === 'control'); const eC = eItems.filter((it) => it.kind === 'control' && it.state === 'visible'); + const used = new Set(); + for (const c of oC) { + const lab = norm(c.label); const generic = /^(trigger|combobox|select|checked|radio|checkbox|pressed|current|tab selected|option selected)$/.test(lab); + let m = null; + if (!generic) m = eC.find((x, idx) => !used.has(idx) && norm(x.label) === lab && (used.add(idx) || true)); + // ordinal fallback only for GENERIC labels: a specifically labelled trigger with no served counterpart is + // CONTROL MISSING, never paired with an unrelated trigger + if (!m && generic) { const sameVia = eC.map((x, idx) => ({ x, idx })).filter(({ x, idx }) => !used.has(idx) && x.via === c.via); const exact = sameVia.find(({ x }) => norm(x.value) === norm(c.value)); const pick = exact || sameVia[0]; if (pick) { used.add(pick.idx); m = pick.x; } } + if (!m) { + if (/^count/.test(lab)) { add('🟠', 'CONTROL STATE', `${c.label}: origin "${c.value}", served: none`, { band: k, label: c.label, value: c.value }); continue; } + const dup = findings.find((f) => f.kind === 'CONTROL MISSING' && f.band === k && f.label === c.label && f.value === c.value); + if (dup) { dup.n = (dup.n || 1) + 1; dup.msg = dup.msg.replace(/^(×\d+ )?/, `×${dup.n} `); } else add('🟡', 'CONTROL MISSING', `${c.label} = "${c.value}" (band "${name}") has no served counterpart`, { band: k, label: c.label, value: c.value }); + continue; + } + if (norm(m.value) !== norm(c.value)) add('🟠', 'CONTROL STATE', `${c.label}: origin "${c.value}" → served "${m.value}" (band "${name}")`, { band: k, label: c.label, origin: c.value, eds: m.value }); + } + band.findings = findings.filter((f) => f.band === k).map((f) => f.kind); + } + const order = { '🔴': 0, '🟠': 1, '🟡': 2 }; + findings.sort((a, b) => order[a.sev] - order[b.sev]); + const sum = (kind) => findings.filter((f) => f.kind === kind).reduce((a, f) => a + (f.n || 1), 0); + const totals = { + missingHeadings: sum('MISSING HEADING'), hiddenHeadings: sum('HIDDEN HEADING'), extraHeadings: sum('EXTRA HEADING'), + missingLinks: sum('MISSING LINK'), hiddenLinks: sum('HIDDEN LINK'), movedLinks: sum('MOVED LINK'), extraLinks: sum('EXTRA LINK'), + missingButtons: sum('MISSING BUTTON'), hiddenButtons: sum('HIDDEN BUTTON'), + controlState: findings.filter((f) => f.kind === 'CONTROL STATE').length, countDeltas: findings.filter((f) => f.kind.startsWith('COUNT')).length, + findings: findings.length, structural: findings.filter((f) => f.sev === '🔴').length, + }; + totals.missing = totals.missingHeadings + totals.missingLinks; totals.hidden = totals.hiddenHeadings + totals.hiddenLinks; + return { scope, chrome, bands, findings, totals }; +} + +export function formatReport(res, { maxPerKind = 40 } = {}) { + const lines = []; + lines.push('| band | heading | links O/E (+hidden) | buttons O/E (+hidden) | text O/E | images O/E | findings |', '|---|---|---|---|---|---|---|'); + for (const b of res.bands) lines.push(`| ${b.index} | ${b.heading.slice(0, 40)}${b.variable ? ' (variable)' : ''} | ${b.origin.link}/${b.eds.link}${b.eds.linkHidden ? ` (+${b.eds.linkHidden})` : ''} | ${b.origin.button}/${b.eds.button}${b.eds.buttonHidden ? ` (+${b.eds.buttonHidden})` : ''} | ${b.origin.text}/${b.eds.text} | ${b.origin.image}/${b.eds.image} | ${[...new Set(b.findings)].join(', ') || '-'} |`); + const seen = new Map(); + for (const f of res.findings) { const n = (seen.get(f.kind) || 0) + 1; seen.set(f.kind, n); if (n <= maxPerKind) lines.push(` ${f.sev} ${f.kind}: ${f.msg}`); else if (n === maxPerKind + 1) lines.push(` … more ${f.kind} (${res.findings.filter((x) => x.kind === f.kind).length} total)`); } + const t = res.totals; + lines.push(`Content: MISSING ${t.missing} (headings ${t.missingHeadings}, links ${t.missingLinks}) / HIDDEN ${t.hidden} (headings ${t.hiddenHeadings}, links ${t.hiddenLinks}) / control-state ${t.controlState} / buttons missing ${t.missingButtons} hidden ${t.hiddenButtons} / moved ${t.movedLinks} extra ${t.extraLinks} / count deltas ${t.countDeltas}`); + lines.push(`Findings: ${t.findings ? `${t.findings} (${t.structural} structural 🔴)` : 'none — every visible origin heading and link is visible on the served page'}`); + return lines.join('\n'); +} + +// ---- CLI --------------------------------------------------------------------------------------------- +export function parseArgs(argv) { + const rest = argv.slice(2); + if (rest.length < 2 || rest.includes('--help') || rest.includes('-h')) { console.log(HELP); process.exit(rest.includes('--help') || rest.includes('-h') || !rest.length ? 0 : 1); } + const opts = { origin: null, eds: null, width: 1440, main: null, mainEds: null, variable: [], variableEds: [], json: false, jsonFile: null, minCut: 2, maxFindings: 40, plain: false, warmup: null, locale: 'en-US', chrome: false, settlePasses: 4, countWords: null, moreWords: null }; + for (let i = 0; i < rest.length; i += 1) { + const a = rest[i]; + if (a === '--width') opts.width = Number(rest[++i]); + else if (a === '--main') { const [o, e] = rest[++i].split('='); opts.main = o; opts.mainEds = e || o; } + else if (a === '--variable') { for (const s of rest[++i].split(',').map((x) => x.trim()).filter(Boolean)) { const [o, e] = s.split('='); opts.variable.push(o); opts.variableEds.push(e || o); } } + else if (a === '--json') { opts.json = true; if (rest[i + 1] && !rest[i + 1].startsWith('--')) opts.jsonFile = rest[++i]; } + else if (a === '--min-cut') opts.minCut = Number(rest[++i]); + else if (a === '--chrome') opts.chrome = true; + else if (a === '--settle-passes') opts.settlePasses = Number(rest[++i]); + else if (a === '--count-words') opts.countWords = rest[++i].split(',').map((w) => w.trim()).filter(Boolean).join('|'); + else if (a === '--more-words') opts.moreWords = rest[++i]; + else if (a === '--max-findings') opts.maxFindings = Number(rest[++i]); + else if (a === '--plain') opts.plain = true; + else if (a === '--warmup') opts.warmup = rest[++i]; + else if (a === '--locale') opts.locale = rest[++i]; + else if (a.startsWith('--')) { console.error(`unknown flag ${a}\n\n${HELP}`); process.exit(1); } + else if (!opts.origin) opts.origin = a; else if (!opts.eds) opts.eds = a; + } + if (!opts.origin || !opts.eds) { console.error(`need \n\n${HELP}`); process.exit(1); } + return opts; +} + +/** Inventory one side: visit + settle, presence inventory, clip inventory. */ +export async function inventorySide(browser, url, { width, locale, warmup, rootSel, variableSels, minCut, settlePasses = 4, httpError = 'measure', countWords = null, moreWords = null }) { + const { ctx, page } = await openPage(browser, { width, locale }); + try { + const v = await visit(page, url, { warmup, settle: { passes: settlePasses }, httpError }); + const inv = await page.evaluate(inPage(presenceInventoryInPage, countWords ? { rootSel, variableSels, countWords } : { rootSel, variableSels }, morePrelude(moreWords))); + const clip = await page.evaluate(inPage(clipInventoryInPage, { minCut, rootSel: null, maxFindings: 400 }, morePrelude(moreWords))); + return { url, at: new Date().toISOString(), status: v.status, settlePasses: v.passes, ...inv, clip: { counts: clip.counts, groups: summarize(clip.findings), findings: clip.findings }, textBoxes: clip.textBoxes }; + } finally { await ctx.close(); } +} + +async function main() { + const opts = parseArgs(process.argv); + const { chromium } = await import('playwright'); + const browser = await openBrowser(chromium, { tier: opts.plain ? 'plain' : 'stealth' }); + let o; let e; let tier; + try { + tier = browserTier(browser); + // ORIGIN side fails loud on any HTTP ≥ 400 (a 403 page measured as the origin reads "100 EXTRA on the + // build"); the SERVED side is measured with a warning (a 404 build before preview propagation is the + // advisory contract). + o = await inventorySide(browser, opts.origin, { width: opts.width, locale: opts.locale, warmup: opts.warmup, rootSel: opts.main, variableSels: opts.variable, minCut: opts.minCut, settlePasses: opts.settlePasses, httpError: 'throw', countWords: opts.countWords, moreWords: opts.moreWords }); + e = await inventorySide(browser, opts.eds, { width: opts.width, locale: opts.locale, warmup: null, rootSel: opts.mainEds, variableSels: opts.variableEds, minCut: opts.minCut, settlePasses: opts.settlePasses, countWords: opts.countWords, moreWords: opts.moreWords }); + } finally { await browser.close(); } + const res = diffPresence(o, e, { chrome: opts.chrome }); + const strip = (side) => ({ url: side.url, at: side.at, status: side.status, settlePasses: side.settlePasses, docH: side.docH, root: side.root, items: side.items, clip: { counts: side.clip.counts, groups: side.clip.groups } }); + const out = { _provenance: { writtenBy: 'content-presence.mjs', at: new Date().toISOString(), width: opts.width, variable: opts.variable, tier }, origin: strip(o), eds: { ...strip(e), clip: e.clip }, bands: res.bands, findings: res.findings, totals: res.totals, textBoxes: { origin: o.textBoxes, eds: e.textBoxes } }; + if (opts.json && !opts.jsonFile) console.log(JSON.stringify(out, null, 1)); + else { + console.log(`content-presence @ ${opts.width}px (${tier}) — origin ${opts.origin} (docH ${o.docH}, ${o.items.length} items, HTTP ${o.status}) vs served ${opts.eds} (docH ${e.docH}, ${e.items.length} items, HTTP ${e.status})`); + console.log(`scope: ${res.scope === 'root' ? `${o.root} vs ${e.root}` : 'whole page (one side has no
/ --main root)'}${res.chrome ? ' + chrome' : ' (header/footer left to the chrome crop gate; --chrome to include)'}`); + console.log(formatReport(res, { maxPerKind: opts.maxFindings })); + console.log(`Clipped (served side): ${e.clip.counts.total} — origin side ${o.clip.counts.total}`); + if (opts.jsonFile) { mkdirSync(dirname(opts.jsonFile) || '.', { recursive: true }); writeFileSync(opts.jsonFile, JSON.stringify(out, null, 1)); console.log(`json → ${opts.jsonFile}`); } + } + process.exitCode = res.totals.missing + res.totals.hidden > 0 ? 2 : 0; +} + +function safeRealpath(p) { try { return realpathSync(p); } catch { return p; } } +if (process.argv[1] && fileURLToPath(import.meta.url) === safeRealpath(process.argv[1])) { + main().catch((err) => { console.error(`content-presence error: ${String(err.message).split('\n')[0]}`); process.exit(err.name === 'BotChallengeError' ? 3 : err.name === 'LiveHTTPError' ? 4 : 1); }); +} diff --git a/plugins/stardust/skills/diff/scripts/live-session.mjs b/plugins/stardust/skills/diff/scripts/live-session.mjs index 4b0efe3e3..4b57ea7f4 100644 --- a/plugins/stardust/skills/diff/scripts/live-session.mjs +++ b/plugins/stardust/skills/diff/scripts/live-session.mjs @@ -225,6 +225,13 @@ function challengeMarker(resp) { if (server.includes('big-ip') || server.includes('imperva') || h['x-iinfo']) return `HTTP ${status} + F5/Imperva edge signature`; // no edge signature — a genuine app-level status, not a challenge. } + // Akamai escalates to HTTP 400 {"result":"Bad Request"} after a burst of headless + // probes (#125): the instruments refused to measure it but never suggested + // escalation. A 400 stamped by AkamaiGHost is a challenge marker. + if (status === 400) { + const server = (h.server || '').toLowerCase(); + if (server.includes('akamaighost') || server.includes('akamai') || h['x-akamai-transformed']) return 'HTTP 400 + Akamai edge signature (bot-management escalation)'; + } return null; } @@ -306,20 +313,46 @@ export async function gotoLive(page, url, { waitUntil = 'domcontentloaded', time } /** - * Headed stealth escalation tier (crawl.mjs launchHeadedStealth semantics): - * headed real Chrome clears TLS/H2-fingerprint blocks, and the stealth args - * strip the automation signals Cloudflare's managed challenge probes for. - * Pair with newLiveContext so the navigator.webdriver spoof lands on every - * context. Takes the caller's `chromium` so this module stays import-free. + * Stealth escalation tier (crawl.mjs launchHeadedStealth semantics): the REAL + * Chrome binary (`channel: 'chrome'`) clears TLS/H2-fingerprint blocks, and the + * stealth args strip the automation signals Cloudflare's managed challenge + * probes for. Pair with newLiveContext so the navigator.webdriver spoof lands + * on every context. Takes the caller's `chromium` so this module stays import-free. + * + * WINDOW-FREE BY DEFAULT (#125). The window was never what cleared the block, the + * binary was: `channel:'chrome'` + headless gets HTTP 200 from Akamai where bundled + * headless Chromium gets 400. Visible windows blocked an operator's desktop under + * four parallel agents. A window is opt-in: `STARDUST_HEADED_WINDOW=1`, for a + * challenge that genuinely needs a human. Every `--headed` flag means THIS tier. + * + * `channel: 'chrome'` needs Google Chrome installed; when the launch fails the + * bundled Chromium is launched instead with the same stealth args and a loud + * stderr warning (the instrument is then the lesser tier — say so in the log). */ export async function launchStealthHeaded(chromium) { - return chromium.launch({ - headless: false, + const visible = process.env.STARDUST_HEADED_WINDOW === '1'; + const opts = { + headless: !visible, channel: 'chrome', args: ['--disable-blink-features=AutomationControlled'], ignoreDefaultArgs: ['--enable-automation'], - }); + }; + try { + const b = await chromium.launch(opts); + b.stardustTier = visible ? 'chrome-window' : 'chrome'; + return b; + } catch (e) { + console.error(`[live-session] WARNING: real Chrome (channel: chrome) failed to launch (${String(e.message).split('\n')[0]}) — falling back to bundled Chromium with the stealth args; bot-managed origins may still block this tier`); + const { channel, ...rest } = opts; + const b = await chromium.launch(rest); + b.stardustTier = 'chromium-fallback'; + return b; + } } +/** The tier a browser from launchStealthHeaded / chromium.launch actually runs: 'chrome' | 'chrome-window' | + * 'chromium-fallback' (Chrome elected, not installed) | 'chromium' (elected). Every probe records it in its + * evidence so a run that degraded is visible in the artifact, not only in a log line. */ +export const browserTier = (b) => b.stardustTier || 'chromium'; // Consent-accept candidates (clicked, never DOM-removed, so consent-gated // layout settles the way a real visit does) — stitch-shot's proven list. diff --git a/plugins/stardust/skills/diff/scripts/measure-live.mjs b/plugins/stardust/skills/diff/scripts/measure-live.mjs new file mode 100644 index 000000000..c5a6952bf --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/measure-live.mjs @@ -0,0 +1,139 @@ +#!/usr/bin/env node +/* eslint-disable import/no-extraneous-dependencies, import/extensions, no-await-in-loop, no-restricted-syntax, brace-style, object-curly-newline, max-len, no-console, no-continue, no-nested-ternary, no-plusplus, object-property-newline */ +/** + * skills/diff/scripts/measure-live.mjs — measure a live page the way the converging passes did (#125): + * window-free real Chrome (live-session), optional home warm-up for bot-managed sites, a slow-scroll + * settle repeated until the document height is stable, then rect + computed type per element, + * shadow-DOM aware, cached per slug under stardust/current/measure/.json. + * + * Library for clip-probe, content-presence and unit-geometry (every gate probe settles both sides + * with THIS routine — the instrument stays symmetric): openBrowser, openPage, visit, settle, + * measureInPage, serializeInPage, cachePath / readCache / writeCache. + * + * Library only (no CLI): selectors are measured through the probes or `replica/measure.mjs`. + * Requires playwright (a project devDependency). + */ +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { REAL_CHROME_UA, browserTier, defaultWaitUntil, dismissOverlays, gotoLive, isLiveHttpUrl, launchStealthHeaded, newLiveContext } from './live-session.mjs'; + +export { browserTier }; + +export const MEASURE_DIR = 'stardust/current/measure'; +export const cachePath = (slug) => join(MEASURE_DIR, `${slug}.json`); +export function readCache(slug) { const p = cachePath(slug); return existsSync(p) ? JSON.parse(readFileSync(p, 'utf8')) : null; } +export function writeCache(slug, data) { mkdirSync(MEASURE_DIR, { recursive: true }); writeFileSync(cachePath(slug), JSON.stringify(data)); return cachePath(slug); } + +/** Launch the measurement browser. 'stealth' = the window-free real-Chrome tier (default on BOTH sides + * of a compare so the instrument is symmetric); 'plain' = bundled Chromium. */ +export async function openBrowser(chromium, { tier = 'stealth' } = {}) { + if (tier === 'plain') { const b = await chromium.launch(); b.stardustTier = 'chromium'; return b; } + return launchStealthHeaded(chromium); // the best available: Chrome, else Chromium (tagged 'chromium-fallback') +} + +export async function openPage(browser, { width = 1440, height = 900, locale = 'en-US', ua = REAL_CHROME_UA } = {}) { + const ctx = await newLiveContext(browser, { ua, locale, viewport: { width, height } }); + const page = await ctx.newPage(); + return { ctx, page }; +} + +/** + * Slow-scroll settle until the document height is stable (hydrate-dom.mjs semantics): scroll the whole + * page in `step` px increments dwelling `dwell` ms each, wait `quietMs`, re-read the height; stop when + * the height did not change AND no `emptySel` shells remain, or after `passes`. Ends scrolled to top. + */ +export async function settle(page, { passes = 4, step = 400, dwell = 250, emptySel = null, quietMs = 2000 } = {}) { + let prevH = -1; let docH = 0; let pendingImgs = 0; let i = 0; + for (; i < passes; i += 1) { + await page.evaluate(async ({ s, d }) => { + const max = () => Math.max(document.documentElement.scrollHeight, document.body ? document.body.scrollHeight : 0); + for (let y = 0; y <= max(); y += s) { window.scrollTo(0, y); await new Promise((r) => { setTimeout(r, d); }); } + }, { s: step, d: dwell }); + await page.waitForTimeout(quietMs); + const st = await page.evaluate((sel) => ({ + h: Math.max(document.documentElement.scrollHeight, document.body ? document.body.scrollHeight : 0), + empty: sel ? document.querySelectorAll(sel).length : 0, + pending: [...document.images].filter((im) => im.getBoundingClientRect().width > 10 && (!im.complete || im.naturalWidth === 0)).length, + }), emptySel); + docH = st.h; pendingImgs = st.pending; + if (st.h === prevH && st.empty === 0) break; + prevH = st.h; + } + await page.evaluate(() => window.scrollTo(0, 0)); + await page.waitForTimeout(600); + return { docH, passes: Math.min(i + 1, passes), pendingImgs }; +} + +/** + * Navigate (fail-loud live-session contract: a challenge throws BotChallengeError, a 404 build is measured + * with a warning), dismiss overlays, settle. `warmup` visits another URL first in the same context. + */ +export async function visit(page, url, { warmup = null, settle: settleOpts = {}, dismiss = true, solveWindow = true, timeoutMs = 90000, httpError = 'measure' } = {}) { + if (warmup) { + await gotoLive(page, warmup, { waitUntil: 'domcontentloaded', timeoutMs, settleMs: 0, solveWindow, httpError: 'measure' }); + await page.waitForTimeout(2500); + } + const resp = await gotoLive(page, url, { waitUntil: defaultWaitUntil(url), timeoutMs, settleMs: 0, solveWindow, httpError }); + await page.waitForTimeout(2500); + if (dismiss) { try { await dismissOverlays(page, { lateWindowMs: isLiveHttpUrl(url) ? 4000 : 0 }); } catch { /* none */ } } + const s = await settle(page, settleOpts); + if (dismiss) { try { await dismissOverlays(page, { lateWindowMs: 0 }); } catch { /* none */ } } + return { status: resp ? resp.status() : null, ...s }; +} + +// ---- in-page functions (Playwright-serialised; ONE argument object) -------------------------------- + +/** page.evaluate(measureInPage, { sels, all }) → { docH, items: [{ sel, tag, cls, x, y, w, h, fs, lh, fw, ff, color, bg, ta, pad, mar, br, display, text }] } */ +export function measureInPage({ sels, all }) { + const px = (v) => Math.round(parseFloat(v) || 0); + const out = { docH: document.documentElement.scrollHeight, items: [] }; + sels.forEach((sel) => { + let els; try { els = [...document.querySelectorAll(sel)]; } catch { out.items.push({ sel, error: 'bad selector' }); return; } + els = all ? els.slice(0, 60) : els.slice(0, 1); + if (!els.length) out.items.push({ sel, missing: true }); + els.forEach((el) => { + const cs = getComputedStyle(el); const r = el.getBoundingClientRect(); + out.items.push({ + sel, tag: el.tagName.toLowerCase(), cls: (typeof el.className === 'string' ? el.className : '').trim().slice(0, 60), + x: Math.round(r.left + window.scrollX), y: Math.round(r.top + window.scrollY), w: Math.round(r.width), h: Math.round(r.height), + fs: px(cs.fontSize), lh: px(cs.lineHeight), fw: cs.fontWeight, ff: cs.fontFamily.split(',')[0].replace(/"/g, ''), + color: cs.color, bg: cs.backgroundColor, ta: cs.textAlign, + pad: [cs.paddingTop, cs.paddingRight, cs.paddingBottom, cs.paddingLeft].map(px).join(' '), + mar: [cs.marginTop, cs.marginRight, cs.marginBottom, cs.marginLeft].map(px).join(' '), + br: cs.borderRadius, display: cs.display, text: el.textContent.replace(/\s+/g, ' ').trim().slice(0, 60), + }); + }); + }); + return out; +} + +/** page.evaluate(serializeInPage, { sel, budget }) → deep HTML of the first match: shadow roots expanded, + * data-r="x,y,w,h" + data-t="fs/lh/fw/color[/bg]" on every element (pdp-measure.mjs's serialiser, lifted). */ +export function serializeInPage({ sel, budget = 4000000 }) { + const root = document.querySelector(sel); + if (!root) return null; + const px = (v) => Math.round(parseFloat(v) || 0); + const rect = (el) => { const r = el.getBoundingClientRect(); return `${Math.round(r.left + window.scrollX)},${Math.round(r.top + window.scrollY)},${Math.round(r.width)},${Math.round(r.height)}`; }; + const esc = (s) => s.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); + const KEEP = ['id', 'class', 'href', 'src', 'alt', 'aria-label', 'type', 'value', 'placeholder', 'name', 'for', 'title', 'aria-expanded', 'aria-hidden', 'style']; + let left = budget; + const ser = (node) => { + if (left <= 0) return ''; + if (node.nodeType === 3) { const t = node.textContent; left -= t.length; return esc(t); } + if (node.nodeType !== 1) return ''; + const tag = node.tagName.toLowerCase(); + if (['script', 'style', 'noscript', 'template', 'link', 'meta'].includes(tag)) return ''; + const attrs = KEEP.filter((k) => node.hasAttribute(k)).map((k) => ` ${k}="${esc(node.getAttribute(k).slice(0, k === 'style' ? 200 : 400))}"`).join(''); + const cs = getComputedStyle(node); + const meta = ` data-r="${rect(node)}" data-t="${px(cs.fontSize)}/${px(cs.lineHeight)}/${cs.fontWeight}/${cs.color.replace(/\s/g, '')}${cs.backgroundColor !== 'rgba(0, 0, 0, 0)' ? `/${cs.backgroundColor.replace(/\s/g, '')}` : ''}"`; + if (tag === 'svg') return ``; + if (['img', 'input', 'br', 'hr'].includes(tag)) return `<${tag}${attrs}${meta}>`; + if (tag === 'iframe') return ``; + let inner = ''; + if (node.shadowRoot) inner += [...node.shadowRoot.childNodes].map(ser).join(''); + inner += [...node.childNodes].map(ser).join(''); + left -= 20; + return `<${tag}${attrs}${meta}>${inner}`; + }; + return ser(root); +} diff --git a/plugins/stardust/skills/diff/scripts/test/clip-probe.test.mjs b/plugins/stardust/skills/diff/scripts/test/clip-probe.test.mjs new file mode 100644 index 000000000..2900ad5d7 --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/test/clip-probe.test.mjs @@ -0,0 +1,113 @@ +#!/usr/bin/env node +// skills/diff/scripts/test/clip-probe.test.mjs — the clip-probe.mjs contract (#125 D1). Part (a), no browser: +// summarize groups findings by kind + clipper + parent and keeps advisory kinds apart, formatTable hides advisory +// rows unless asked, verdictLine, parseArgs, --help in an empty cwd. Part (b), where playwright is importable: +// a fixture page served from this process — a 238 px card with overflow:hidden whose description is cut +// mid-line and whose "View details" link sits below the box (the recorded card defect), a line-clamped +// paragraph (advisory only), a collapsed mega-menu (max-height 0 — must not count), an sr-only span (must not +// count), a horizontal carousel track (advisory only), a "Read more" collapsible (advisory TEXT COLLAPSED) — and +// a clean page (exit 0). Run: node . +import assert from 'node:assert/strict'; +import { spawn, spawnSync } from 'node:child_process'; +import { mkdtempSync, readdirSync, rmSync } from 'node:fs'; +import { createServer } from 'node:http'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { formatTable, parseArgs, summarize, verdictLine } from '../clip-probe.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const SCRIPT = join(HERE, '..', 'clip-probe.mjs'); +let failed = 0; +const check = async (name, fn) => { try { await fn(); console.log(`✓ ${name}`); } catch (e) { failed += 1; console.log(`✗ ${name}\n ${String(e.message).split('\n').join('\n ')}`); } }; +const runAsync = (args, cwd) => new Promise((done) => { const ch = spawn(process.execPath, [SCRIPT, ...args], { cwd }); let out = ''; let err = ''; ch.stdout.on('data', (d) => { out += d; }); ch.stderr.on('data', (d) => { err += d; }); ch.on('close', (code) => done({ code, out, err })); }); + +const findings = [ + { kind: 'CONTROL HIDDEN', path: 'div.card-body > p.details > a', text: 'view details', clipper: 'div.card-body', cut: 20 }, + { kind: 'CONTROL HIDDEN', path: 'div.card-body > p.details > a', text: 'view details', clipper: 'div.card-body', cut: 20 }, + { kind: 'TEXT CLIPPED', path: 'div.card-body > p.desc', text: 'multi use offer', clipper: 'div.card-body', cut: 4 }, + { kind: 'TEXT CLIPPED', path: 'div.card-body > p.desc', text: 'valid online', clipper: 'div.card-body', cut: 9 }, + { kind: 'TEXT CLAMPED', advisory: true, path: 'div.card > p.brand', text: 'neutrogena', clipper: 'p.brand' }, +]; +await check('summarize groups by kind + clipper + parent, counts, cut range, advisory last', () => { + const g = summarize(findings); + assert.equal(g.length, 3); + assert.deepEqual(g.map((x) => [x.kind, x.n]), [['CONTROL HIDDEN', 2], ['TEXT CLIPPED', 2], ['TEXT CLAMPED', 1]]); + assert.equal(g[1].cutMin, 4); assert.equal(g[1].cutMax, 9); assert.equal(g[2].advisory, true); assert.equal(g[2].cutMin, null); +}); +await check('formatTable hides advisory rows by default and shows them with { advisory: true }', () => { + const g = summarize(findings); + assert.ok(!formatTable(g).includes('CLAMPED')); + assert.ok(formatTable(g, { advisory: true }).includes('CLAMPED')); + assert.ok(formatTable(g).includes('×2')); + assert.equal(formatTable([]), ' (none)'); +}); +await check('verdictLine names every counter', () => { + const l = verdictLine({ total: 3, textClipped: 1, textHidden: 0, controlHidden: 2, controlClipped: 0, clamped: 4, scrollHidden: 0, horizontal: 1 }); + assert.ok(l.startsWith('Clipped: 3')); assert.ok(l.includes('controls hidden 2')); assert.ok(l.includes('clamped 4')); +}); +await check('parseArgs: url, flags, --json with and without a file', () => { + const o = parseArgs(['node', 'x', 'https://a.test/', '--width', '360', '--json', 'out.json', '--min-cut', '3', '--advisory']); + assert.equal(o.url, 'https://a.test/'); assert.equal(o.width, 360); assert.equal(o.jsonFile, 'out.json'); assert.equal(o.minCut, 3); assert.equal(o.advisory, true); + const p = parseArgs(['node', 'x', 'https://a.test/', '--json', '--plain']); + assert.equal(p.json, true); assert.equal(p.jsonFile, null); assert.equal(p.plain, true); +}); +await check('--help exits 0 with the usage and writes nothing', () => { + const cwd = mkdtempSync(join(tmpdir(), 'clip-probe-help-')); + const r = spawnSync(process.execPath, [SCRIPT, '--help'], { cwd, encoding: 'utf8' }); + assert.equal(r.status, 0); assert.match(r.stdout, /Usage: node clip-probe\.mjs/); assert.deepEqual(readdirSync(cwd), []); + rmSync(cwd, { recursive: true, force: true }); +}); + +let playwright = true; +try { await import('playwright'); } catch { playwright = false; } +if (!playwright) { console.log('skip end-to-end (playwright not importable here)'); } +else { + const BROKEN = `
+

Cards

+

First line of the description
second line of the description is cut mid-glyph by the 30px box
third

View details

+

A long clamped paragraph that keeps going and going until the second line ends and a third line is hidden by the clamp, which is design, not a defect.

+

Visible paragraph screen reader only.

+
slide oneslide two off to the right
+

Collapsed accordion first line
second line
third line behind the toggle

+
`; + const CLEAN = '

Clean

All visible text.

A link
'; + const server = createServer((req, res) => { res.setHeader('content-type', 'text/html'); res.end(req.url.startsWith('/clean') ? CLEAN : BROKEN); }); + await new Promise((r) => { server.listen(0, '127.0.0.1', r); }); + const base = `http://127.0.0.1:${server.address().port}`; + const cwd = mkdtempSync(join(tmpdir(), 'clip-probe-e2e-')); + await check('e2e: the broken card counts one hidden control + one clipped line + one hidden line; clamp / menu / sr-only / carousel do not count (exit 2)', async () => { + const r = await runAsync([`${base}/broken`, '--json', '--plain', '--width', '900'], cwd); + assert.equal(r.code, 2, r.err); + const j = JSON.parse(r.out); + assert.equal(j.counts.controlHidden, 1, JSON.stringify(j.findings)); + assert.equal(j.counts.textClipped, 1, 'the second line is cut mid-glyph by p.desc'); + assert.equal(j.counts.textHidden, 1, 'the third line (its own text node after
) is wholly hidden by p.desc — no clamp'); + assert.equal(j.counts.total, 3); + assert.equal(j.counts.clamped, 1); + assert.ok(j.counts.horizontal >= 1, 'carousel slide is a horizontal advisory'); + assert.equal(j.counts.collapsed, 1, 'lines behind a "Read more" collapsible are advisory TEXT COLLAPSED'); + assert.ok(j.findings.some((f) => f.kind === 'TEXT COLLAPSED')); + assert.ok(!j.findings.some((f) => /collapsed menu|hidden menu/i.test(f.text)), 'collapsed menu must not be reported'); + assert.ok(!j.findings.some((f) => /screen reader/i.test(f.text)), 'sr-only must not be reported'); + assert.ok(j.textBoxes.length > 3, 'visible text line boxes are emitted for --text-boxes'); + assert.ok(!j.textBoxes.some((b) => b.h <= 1), 'no sr-only boxes among the text boxes'); + }); + await check('e2e: a clean page exits 0 with Clipped: 0', async () => { + const r = await runAsync([`${base}/clean`, '--plain', '--width', '900'], cwd); + assert.equal(r.code, 0, r.err); assert.match(r.out, /Clipped: 0 /); + }); + server.close(); rmSync(cwd, { recursive: true, force: true }); +} +console.log(failed ? `\n${failed} failed` : '\nall passed'); +process.exit(failed ? 1 : 0); diff --git a/plugins/stardust/skills/diff/scripts/test/content-presence.test.mjs b/plugins/stardust/skills/diff/scripts/test/content-presence.test.mjs new file mode 100644 index 000000000..798d7db7c --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/test/content-presence.test.mjs @@ -0,0 +1,97 @@ +#!/usr/bin/env node +// skills/diff/scripts/test/content-presence.test.mjs — the content-presence.mjs differ (#125 D2), no browser: +// alignHeadings (LCS by text, prefix tolerance), diffPresence — HIDDEN LINK ×n for links present but clipped, +// MISSING LINK for absent ones, MOVED LINK when present in another band, cross-kind matching (an origin +// button served as a link is not missing), session-variable regions compared as counts (never MISSING) while +// HIDDEN still counts, CONTROL STATE for a sort trigger, HEADING AS TEXT when the text survives as plain +// text, scope = whole page when one side lacks
, chrome items excluded by default; formatReport lines; +// --help in an empty cwd. Run: node . +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, readdirSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { alignHeadings, diffPresence, formatReport } from '../content-presence.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const SCRIPT = join(HERE, '..', 'content-presence.mjs'); +let failed = 0; +const check = (name, fn) => { try { fn(); console.log(`✓ ${name}`); } catch (e) { failed += 1; console.log(`✗ ${name}\n ${String(e.message).split('\n').join('\n ')}`); } }; +const it = (kind, text, y, extra = {}) => ({ kind, text, y, x: 0, w: 100, h: 20, state: 'visible', variable: false, inRoot: true, inChrome: false, path: 'p', ...extra }); +const inv = (items, root = 'main') => ({ docH: 3000, pageW: 1440, root, items }); + +check('alignHeadings: LCS in order, prefix-tolerant for long texts', () => { + assert.deepEqual(alignHeadings(['a', 'b', 'c'], ['a', 'c']), [{ oi: 0, ei: 1 }, { oi: 2, ei: 1 }].map((p, i) => [{ oi: 0, ei: 0 }, { oi: 2, ei: 1 }][i])); + assert.deepEqual(alignHeadings(['help center topics'], ['help center topics and more']), [{ oi: 0, ei: 0 }]); + assert.deepEqual(alignHeadings(['x'], ['y']), []); +}); +check('HIDDEN LINK ×n when the served links exist but are clipped; exit criterion counts them', () => { + const o = inv([it('heading', 'Coupons', 100, { level: 1 }), ...Array.from({ length: 3 }, (_, i) => it('link', 'View details', 200 + i * 10, { href: '/d' }))]); + const e = inv([it('heading', 'Coupons', 100, { level: 1 }), ...Array.from({ length: 3 }, (_, i) => it('link', 'View details', 200 + i * 10, { href: '/d', state: 'hidden' }))]); + const r = diffPresence(o, e); + const f = r.findings.find((x) => x.kind === 'HIDDEN LINK'); + assert.ok(f && f.n === 3, JSON.stringify(r.findings)); assert.equal(r.totals.hidden, 3); assert.equal(r.totals.missing, 0); assert.equal(r.bands[1].eds.linkHidden, 3); +}); +check('MISSING LINK when absent, MOVED LINK when visible in another band, EXTRA LINK for build-only', () => { + const o = inv([it('heading', 'A', 100, { level: 2 }), it('link', 'Contact', 150, { href: '/c' }), it('heading', 'B', 500, { level: 2 }), it('link', 'Careers', 550, { href: '/j' })]); + const e = inv([it('heading', 'A', 100, { level: 2 }), it('heading', 'B', 500, { level: 2 }), it('link', 'Contact', 550, { href: '/c' }), it('link', 'Newsroom', 560, { href: '/n' })]); + const r = diffPresence(o, e); + assert.deepEqual(r.findings.map((x) => x.kind).sort(), ['EXTRA LINK', 'MISSING LINK', 'MOVED LINK']); + assert.equal(r.totals.missing, 1); +}); +check('cross-kind: an origin button served as a link is present; an origin link served as a button is present', () => { + const o = inv([it('heading', 'H', 10, { level: 1 }), it('button', 'Sign in', 20), it('link', 'Join', 30, { href: '/j' })]); + const e = inv([it('heading', 'H', 10, { level: 1 }), it('link', 'Sign in', 20, { href: '/s' }), it('button', 'Join', 30)]); + const r = diffPresence(o, e); + assert.equal(r.findings.length, 0, JSON.stringify(r.findings)); +}); +check('session-variable region: counts only (never MISSING), HIDDEN still 🔴', () => { + const o = inv([it('heading', 'H', 10, { level: 1 }), ...Array.from({ length: 6 }, (_, i) => it('link', `Coupon ${i}`, 100 + i, { href: '/c', variable: true }))]); + const e = inv([it('heading', 'H', 10, { level: 1 }), ...Array.from({ length: 2 }, (_, i) => it('link', `Other ${i}`, 100 + i, { href: '/o', variable: true })), it('link', 'Clipped one', 120, { href: '/x', variable: true, state: 'hidden' })]); + const r = diffPresence(o, e); + assert.ok(!r.findings.some((x) => x.kind === 'MISSING LINK'), JSON.stringify(r.findings)); + assert.ok(r.findings.some((x) => x.kind === 'COUNT LINKS')); + assert.equal(r.totals.hidden, 1); assert.equal(r.bands[1].variable, true); +}); +check('CONTROL STATE for a trigger with the same label and a different value; COUNT control matched by via', () => { + const o = inv([it('heading', 'H', 10, { level: 1 }), it('control', '', 20, { label: 'sort by', value: 'Recommended', via: 'trigger' }), it('control', '', 30, { label: 'count coupon', value: '285', via: 'count' })]); + const e = inv([it('heading', 'H', 10, { level: 1 }), it('control', '', 20, { label: 'sort by', value: 'Expiration Date', via: 'trigger' }), it('control', '', 30, { label: 'count coupon', value: '284', via: 'count' })]); + const r = diffPresence(o, e); + assert.equal(r.findings.filter((x) => x.kind === 'CONTROL STATE').length, 2, JSON.stringify(r.findings)); assert.equal(r.totals.controlState, 2); assert.equal(r.totals.structural, 0); +}); +check('a specifically labelled control with no counterpart is CONTROL MISSING (never paired by ordinal); generic ones pair by ordinal', () => { + const o = inv([it('heading', 'H', 10, { level: 1 }), it('control', '', 20, { label: 'rebate help', value: 'rebate help', via: 'trigger' }), it('control', '', 25, { label: 'trigger', value: 'Menu A', via: 'trigger' })]); + const e = inv([it('heading', 'H', 10, { level: 1 }), it('control', '', 20, { label: 'trigger', value: 'Menu B', via: 'trigger' })]); + const r = diffPresence(o, e); + assert.ok(r.findings.some((x) => x.kind === 'CONTROL MISSING' && /rebate/.test(x.msg))); + assert.ok(r.findings.some((x) => x.kind === 'CONTROL STATE' && /Menu A.*Menu B/.test(x.msg))); +}); +check('HEADING AS TEXT (🟡) when the text is served as plain text; MISSING HEADING (🔴) when it is gone', () => { + const o = inv([it('heading', 'Rx information', 100, { level: 3 }), it('heading', 'Gone', 300, { level: 2 })]); + const e = inv([it('text', 'Rx information', 100)]); + const r = diffPresence(o, e); + assert.ok(r.findings.some((x) => x.kind === 'HEADING AS TEXT' && x.sev === '🟡')); + assert.ok(r.findings.some((x) => x.kind === 'MISSING HEADING' && x.sev === '🔴')); assert.equal(r.totals.missingHeadings, 1); +}); +check('scope: whole page when one side has no root; chrome items excluded unless { chrome: true }', () => { + const o = inv([it('link', 'Promo strip', 5, { href: '/p', inRoot: null, inChrome: true }), it('link', 'Body link', 400, { href: '/b', inRoot: null })], null); + const e = inv([it('link', 'Body link', 400, { href: '/b', inRoot: true })], 'main'); + assert.equal(diffPresence(o, e).scope, 'page'); + assert.equal(diffPresence(o, e).findings.length, 0); + assert.equal(diffPresence(o, e, { chrome: true }).totals.missing, 1); +}); +check('formatReport: band table + Content/Findings lines', () => { + const o = inv([it('heading', 'H', 10, { level: 1 }), it('link', 'A', 20, { href: '/a' })]); + const r = diffPresence(o, inv([it('heading', 'H', 10, { level: 1 })])); + const s = formatReport(r); + assert.match(s, /\| band \| heading/); assert.match(s, /MISSING LINK: "A"/); assert.match(s, /Content: MISSING 1/); assert.match(s, /Findings: 1 \(1 structural/); +}); +check('--help exits 0 in an empty cwd and writes nothing', () => { + const cwd = mkdtempSync(join(tmpdir(), 'content-presence-help-')); + const r = spawnSync(process.execPath, [SCRIPT, '--help'], { cwd, encoding: 'utf8' }); + assert.equal(r.status, 0); assert.match(r.stdout, /Usage: node content-presence\.mjs/); assert.deepEqual(readdirSync(cwd), []); + rmSync(cwd, { recursive: true, force: true }); +}); +console.log(failed ? `\n${failed} failed` : '\nall passed'); +process.exit(failed ? 1 : 0); diff --git a/plugins/stardust/skills/diff/scripts/test/unit-geometry.test.mjs b/plugins/stardust/skills/diff/scripts/test/unit-geometry.test.mjs new file mode 100644 index 000000000..63e4b21c3 --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/test/unit-geometry.test.mjs @@ -0,0 +1,70 @@ +#!/usr/bin/env node +// skills/diff/scripts/test/unit-geometry.test.mjs — the unit-geometry.mjs alignment (#125 D3), no browser: +// keyed() stable keys (role:text, role:selector#ordinal, #2 for duplicates), alignUnit pairs by key, by text +// prefix, across relaxed roles (link ↔ button), images by order whatever their alt, leftover text by position +// (the session-variable badge), reports Δx/Δy/Δw/Δh relative to the unit, flags off / hidden / missing / extra +// and the unit size; verdictOf totals; parseArgs; --help in an empty cwd. Run: node . +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, readdirSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { alignUnit, compareUnits, formatUnits, keyed, parseArgs, verdictOf } from '../unit-geometry.mjs'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const SCRIPT = join(HERE, '..', 'unit-geometry.mjs'); +let failed = 0; +const check = (name, fn) => { try { fn(); console.log(`✓ ${name}`); } catch (e) { failed += 1; console.log(`✗ ${name}\n ${String(e.message).split('\n').join('\n ')}`); } }; +const el = (role, text, x, y, w, h, extra = {}) => ({ role, tag: role === 'image' ? 'img' : 'p', sel: role === 'image' ? 'img' : 'p.x', text, x, y, w, h, fs: 16, lh: 22, fw: '400', ...extra }); + +check('keyed: role:text keys, selector#ordinal for text-less, #2 for duplicates', () => { + const k = keyed([el('text', 'Expires', 0, 0, 10, 10), el('image', '', 0, 0, 10, 10), el('image', '', 0, 0, 10, 10), el('text', 'Expires', 0, 0, 10, 10)]); + assert.deepEqual(k.map((x) => x.key), ['text:expires', 'image:img#1', 'image:img#2', 'text:expires #2']); +}); +check('alignUnit: the recorded card — body 28 px low, details link hidden, badge paired by position, image by order', () => { + const o = { rect: { x: 165, y: 500, w: 350, h: 260 }, elements: [el('text', '1 day left', -10, -10, 70, 70), el('image', '', 17, 76, 70, 70), el('text', 'Expires 09/26/26', 112, 12, 202, 15), el('text', 'Earn $5 W Cash rewards when you spend', 112, 27, 202, 54), el('link', 'View details', 112, 176, 91, 24), el('button', 'Clip', 12, 204, 326, 44)] }; + const e = { rect: { x: 165, y: 522, w: 355, h: 238 }, elements: [el('text', '2 days left', -27, -30, 64, 64), el('image', 'coupon', 11, 46, 65, 65), el('text', 'Expires 09/26/26', 101, 16, 242, 20), el('text', 'Earn $5 W Cash rewards when yo…', 101, 55, 215, 48), el('link', 'View details', 101, 204, 91, 20, { hidden: true }), el('link', 'Clip', 12, 182, 331, 44)] }; + const r = alignUnit(o, e, 4); + assert.equal(r.summary.missing, 0, JSON.stringify(r.missing)); assert.equal(r.summary.extra, 0); + const by = Object.fromEntries(r.rows.map((x) => [x.key, x])); + assert.equal(by['text:1 day left'].byPosition, true); assert.equal(by['text:1 day left'].dy, -20); + assert.equal(by['image:img#1'].dy, -30); + assert.equal(by['text:earn $5 w cash rewards when you spend'].dy, 28, 'prefix-matched summary line is 28 px low'); + assert.equal(by['link:view details'].hidden, true); assert.equal(by['link:view details'].dy, 28); + assert.equal(by['button:clip'].eds.role, 'link', 'relaxed role pairing'); assert.equal(by['button:clip'].dy, -22); + assert.equal(r.unit.dh, -22); assert.equal(r.summary.unitOff, true); + assert.equal(r.summary.within, 0); assert.equal(r.summary.hidden, 1); +}); +check('alignUnit: identical units read all within, nothing off', () => { + const u = { rect: { x: 0, y: 0, w: 100, h: 100 }, elements: [el('heading', 'Title', 0, 0, 100, 20), el('text', 'Body', 0, 30, 100, 20)] }; + const r = alignUnit(u, { ...u, rect: { x: 300, y: 0, w: 100, h: 100 } }, 4); + assert.equal(r.summary.within, 2); assert.equal(r.summary.off, 0); assert.equal(r.unit.dx, 300); assert.equal(r.summary.unitOff, false); +}); +check('missing + extra when nothing pairs; verdictOf totals; formatUnits marks rows', () => { + const o = { rect: { x: 0, y: 0, w: 10, h: 10 }, elements: [el('link', 'Only here', 0, 0, 10, 10)] }; + const e = { rect: { x: 0, y: 0, w: 10, h: 10 }, elements: [el('button', 'Something else', 0, 0, 10, 10)] }; + const r = alignUnit(o, e, 4); + assert.equal(r.summary.missing, 1); assert.equal(r.summary.extra, 1); + const res = [{ sel: 'a=b', units: [{ index: 0, ...r }] }, { sel: 'x=y', units: [], error: 'no visible unit' }]; + assert.deepEqual(verdictOf(res), { units: 1, within: 0, off: 0, hidden: 0, missing: 1, errors: 1, requiredOff: 1 }); + assert.equal(verdictOf([{ sel: 'a=b', required: false, units: [{ index: 0, ...r }] }]).requiredOff, 0, 'an advisory family never feeds the exit code'); + const s = formatUnits(res, 4); assert.match(s, /✗ missing/); assert.match(s, /🟡 extra/); assert.match(s, /no visible unit/); +}); +check('compareUnits pairs units by index and reports a side without visible units', () => { + const inv = (n) => ({ bySel: { '.c': { matches: n, units: Array.from({ length: n }, (_, i) => ({ index: i, path: 'div', rect: { x: 0, y: i * 100, w: 10, h: 10 }, elements: [] })) } } }); + assert.equal(compareUnits(inv(2), inv(2), { origin: '.c', eds: '.c' }, 4).units.length, 2); + assert.match(compareUnits(inv(2), inv(0), { origin: '.c', eds: '.c' }, 4).error, /served side/); +}); +check('parseArgs: repeatable --unit with and without the served selector', () => { + const o = parseArgs(['node', 'x', 'https://o/', 'https://e/', '--unit', '.a=.b', '--unit', '.c', '--n', '3', '--tol', '2', '--slug', 's']); + assert.deepEqual(o.units, [{ origin: '.a', eds: '.b' }, { origin: '.c', eds: '.c' }]); assert.equal(o.n, 3); assert.equal(o.tol, 2); assert.equal(o.slug, 's'); +}); +check('--help exits 0 in an empty cwd and writes nothing', () => { + const cwd = mkdtempSync(join(tmpdir(), 'unit-geometry-help-')); + const r = spawnSync(process.execPath, [SCRIPT, '--help'], { cwd, encoding: 'utf8' }); + assert.equal(r.status, 0); assert.match(r.stdout, /Usage: node unit-geometry\.mjs/); assert.deepEqual(readdirSync(cwd), []); + rmSync(cwd, { recursive: true, force: true }); +}); +console.log(failed ? `\n${failed} failed` : '\nall passed'); +process.exit(failed ? 1 : 0); diff --git a/plugins/stardust/skills/diff/scripts/unit-geometry.mjs b/plugins/stardust/skills/diff/scripts/unit-geometry.mjs new file mode 100644 index 000000000..41a73a840 --- /dev/null +++ b/plugins/stardust/skills/diff/scripts/unit-geometry.mjs @@ -0,0 +1,278 @@ +#!/usr/bin/env node +/* eslint-disable import/no-extraneous-dependencies, import/extensions, no-await-in-loop, no-restricted-syntax, brace-style, object-curly-newline, max-len, no-console, no-continue, no-nested-ternary, no-plusplus, no-underscore-dangle, no-restricted-globals, newline-per-chained-call, object-property-newline, no-use-before-define */ +/* global __clipChain, __norm, __pageRect, __path, __rendered, __sel, __srOnly, __style, __visibleText, __walk */ +/** + * skills/diff/scripts/unit-geometry.mjs — per-unit GEOMETRY compare for repeated units (#125, D3). + * + * A card, a rail item, an FAQ row: the pixel gate sees the grid of shapes, not whether the badge sits + * on the corner or the body starts 30 px low. The workstreams that converged measured live element + * rects; this makes that a gate row: for the first N matching units on both sides, the rects of the + * inner elements (role + text, shadow-DOM aware) relative to the unit's corner, aligned (key, text + * prefix, relaxed link ↔ button, images by order, leftover text by position) and reported as + * Δx / Δy / Δw / Δh against --tol, plus the unit's own size / position delta; `hidden` where the + * served page clips. Both sides settle the same way; the origin inventory caches per slug under + * stardust/current/measure/-units.json. + * + * Repeated-unit FAMILIES are declared once per project in stardust/replica/units.json — + * { "": { "origin": "", "build": "", "n": 2, "required": true, "pages": [""], "templates": [""] } } + * — and resolved per page by `--families --slug [--template ]` (gate.sh and gate-all both do); + * a family the page does not belong to is skipped, a non-`required` family is advisory (never exit 2). + * + * Usage: node skills/diff/scripts/unit-geometry.mjs (--unit "=" … | --families --slug ) + * [--n 1] [--tol 4] [--width 1440] [--json []] [--slug ] [--force] [--advisory] + * [--plain] [--warmup ] [--locale en-US] + * Exit: 0 within tolerance (or nothing declared), 2 a required unit off / hidden / missing, 1 error, + * 3 bot challenge. `unitInventoryInPage`, `alignUnit`, `keyed`, `compareUnits`, `formatUnits`, + * `verdictOf`, `unitsFor` are exported. + */ +import { existsSync, mkdirSync, readFileSync, realpathSync, writeFileSync } from 'node:fs'; +import { dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { inPage } from './clip-probe.mjs'; +import { browserTier, openBrowser, openPage, readCache, visit, writeCache } from './measure-live.mjs'; + +const HELP = `unit-geometry — per-element Δx/Δy/Δw/Δh of the first N repeated units, origin vs served (#125 D3) + +Usage: node unit-geometry.mjs (--unit "=" … | --families --slug ) [options] + --unit [=] repeatable + --families stardust/replica/units.json: { family: { origin, build, n, required, pages[], templates[] } } + --template the page's template, for family matching (read from stardust/state.json when absent) + --n units per selector (default 1) + --tol per-element tolerance (default 4) + --width viewport width (default 1440) + --json [] JSON on stdout or to + --slug cache the origin inventory at stardust/current/measure/-units.json (--force to redo) + --advisory report only, exit 0 + --plain bundled Chromium instead of the window-free real-Chrome tier + --warmup visit this URL first on the origin side + --locale default en-US + --help this text +Exit: 0 within tolerance, 2 any element off / missing / hidden, 1 error, 3 bot challenge.`; + +// ---- in-page (needs IN_PAGE_LIB) ------------------------------------------------------------------------- +/** page.evaluate(inPage(unitInventoryInPage, { sel, n })) → { sel, matches, units: [{ index, path, rect, elements[] }] } */ +export function unitInventoryInPage({ sel, n = 1 } = {}) { + let all; try { all = [...document.querySelectorAll(sel)]; } catch { return { sel, error: 'bad selector', matches: 0, units: [] }; } + const pageW = document.documentElement.clientWidth; + const overlap = (a0, a1, b0, b1) => Math.max(0, Math.min(a1, b1) - Math.max(a0, b0)); + const onPage = (el) => { if (!__rendered(el) || __srOnly(el)) return false; const r = __pageRect(el.getBoundingClientRect()); return r.w > 0 && r.h > 0 && r.r > 0 && r.l < pageW; }; + const visible = all.filter(onPage).slice(0, n); + const px = (v) => Math.round(parseFloat(v) || 0); + const roleOf = (el) => { + const t = el.tagName; + if (/^H[1-6]$/.test(t) || el.getAttribute('role') === 'heading') return 'heading'; + if (t === 'A' && el.hasAttribute('href')) return 'link'; + if (t === 'BUTTON' || el.getAttribute('role') === 'button' || (t === 'INPUT' && /^(button|submit|reset)$/i.test(el.type))) return 'button'; + if (t === 'IMG' || t === 'PICTURE' || t === 'SVG' || t === 'svg' || t === 'VIDEO' || t === 'IFRAME') return 'image'; + if (t === 'INPUT' || t === 'SELECT' || t === 'TEXTAREA') return 'input'; + return null; + }; + const ownText = (el) => [...el.childNodes].filter((x) => x.nodeType === 3).map((x) => x.textContent).join(' ').replace(/\s+/g, ' ').trim(); + const units = visible.map((u, index) => { + const ur = __pageRect(u.getBoundingClientRect()); + const elements = []; + const seenText = new Set(); + for (const el of __walk(u)) { + if (el.nodeType !== 1 || el === u) continue; + if (!__rendered(el) || __srOnly(el)) continue; + const br = el.getBoundingClientRect(); if (br.width <= 0 || br.height <= 0) continue; + const s = __style(el); + let role = roleOf(el); + const own = ownText(el); + const boxed = s.backgroundColor !== 'rgba(0, 0, 0, 0)' || s.borderTopWidth !== '0px' || s.boxShadow !== 'none' || (s.backgroundImage && s.backgroundImage !== 'none'); + if (!role) { if (own) role = 'text'; else if (boxed) role = 'box'; else continue; } + // a link/button's text lives in descendants — take the visible text; other roles take own text + const text = role === 'link' || role === 'button' || role === 'heading' ? __norm(__visibleText(el)).slice(0, 60) : role === 'image' ? __norm(el.getAttribute('alt') || el.getAttribute('aria-label') || '').slice(0, 60) : __norm(own).slice(0, 60); + if (role === 'text' && seenText.has(text) && !boxed) continue; // nested wrappers repeating the same own text + if (text) seenText.add(text); + const r = __pageRect(br); + const cc = __clipChain(el, false); + if (cc.skip) continue; + let hidden = false; + for (const clip of cc.chain) { if (clip.mode === 'scroll') continue; const vis = (clip.axes.y ? overlap(r.t, r.b, clip.rect.t, clip.rect.b) : r.h) * (clip.axes.x ? overlap(r.l, r.r, clip.rect.l, clip.rect.r) : r.w); if (vis / (r.w * r.h) < 0.5) { hidden = true; break; } } + elements.push({ + role, tag: el.tagName.toLowerCase(), sel: __sel(el), text, boxed, + x: Math.round(r.l - ur.l), y: Math.round(r.t - ur.t), w: Math.round(r.w), h: Math.round(r.h), absY: Math.round(r.t), hidden, + fs: px(s.fontSize), lh: px(s.lineHeight), fw: s.fontWeight, color: s.color, bg: s.backgroundColor !== 'rgba(0, 0, 0, 0)' ? s.backgroundColor : null, position: s.position, + }); + } + return { index, path: __path(u, 3), rect: { x: Math.round(ur.l), y: Math.round(ur.t), w: Math.round(ur.w), h: Math.round(ur.h) }, elements }; + }); + return { sel, matches: all.length, visibleMatches: all.filter(onPage).length, units }; +} + +// ---- pure: alignment ------------------------------------------------------------------------------------ +const norm = (s) => (s || '').replace(/\s+/g, ' ').trim().toLowerCase(); +const keyOf = (el, ord) => (el.text ? `${el.role}:${norm(el.text).slice(0, 40)}` : `${el.role}:${el.sel}#${ord}`); + +/** Assign stable keys (role:text, or role:selector#ordinal for text-less elements; duplicates get #2, #3 …). */ +export function keyed(elements) { + const ordBySel = new Map(); const seen = new Map(); + return elements.map((el) => { + const o = (ordBySel.get(`${el.role}:${el.sel}`) || 0) + 1; ordBySel.set(`${el.role}:${el.sel}`, o); + let key = keyOf(el, o); + const n = (seen.get(key) || 0) + 1; seen.set(key, n); + if (n > 1) key = `${key} #${n}`; + return { ...el, key }; + }); +} + +/** alignUnit(originUnit, edsUnit, tol) → { rows, missing, extra, unit, summary } */ +export function alignUnit(o, e, tol = 4) { + const O = keyed(o.elements); const E = keyed(e.elements); + const usedE = new Set(); const rows = []; const missing = []; + const relaxed = { link: ['button', 'text'], button: ['link', 'text'], heading: ['text'], text: ['heading', 'link', 'button'] }; + const find = (oe) => { + let idx = E.findIndex((ee, i) => !usedE.has(i) && ee.key === oe.key); + if (idx < 0 && oe.text) idx = E.findIndex((ee, i) => !usedE.has(i) && ee.text && ee.role === oe.role && (norm(ee.text) === norm(oe.text) || (norm(oe.text).length >= 10 && (norm(ee.text).startsWith(norm(oe.text).slice(0, 10)) || norm(oe.text).startsWith(norm(ee.text).slice(0, 10)))))); + if (idx < 0 && oe.text) idx = E.findIndex((ee, i) => !usedE.has(i) && ee.text && (relaxed[oe.role] || []).includes(ee.role) && norm(ee.text) === norm(oe.text)); + if (idx < 0 && !oe.text) idx = E.findIndex((ee, i) => !usedE.has(i) && !ee.text && ee.role === oe.role && (ee.tag === oe.tag || oe.role === 'image')); + // images pair by role + order whatever their alt text says (an origin without alt vs a served one with) + if (idx < 0 && oe.role === 'image') idx = E.findIndex((ee, i) => !usedE.has(i) && ee.role === 'image'); + return idx; + }; + // second pass for what text could not pair: same role, nearest position within 40 px (session-variable text + // such as a "1 day left" vs "2 days left" badge is the same element) + const byPosition = (oe) => { + let best = -1; let bestD = 41; + E.forEach((ee, i) => { if (usedE.has(i) || ee.role !== oe.role) return; const d = Math.max(Math.abs(ee.x - oe.x), Math.abs(ee.y - oe.y)); if (d < bestD) { bestD = d; best = i; } }); + return best; + }; + const pending = []; + for (const oe of O) { + const idx = find(oe); + if (idx < 0) { pending.push(oe); continue; } + usedE.add(idx); rows.push(rowFor(oe, E[idx], tol, false)); + } + for (const oe of pending) { + const idx = oe.role === 'text' || oe.role === 'box' || oe.role === 'heading' ? byPosition(oe) : -1; + if (idx < 0) { missing.push(oe); continue; } + usedE.add(idx); rows.push(rowFor(oe, E[idx], tol, true)); + } + rows.sort((a, b) => a.origin.y - b.origin.y || a.origin.x - b.origin.x); + const extra = E.filter((_, i) => !usedE.has(i)); + const unit = { origin: o.rect, eds: e.rect, dx: e.rect.x - o.rect.x, dy: e.rect.y - o.rect.y, dw: e.rect.w - o.rect.w, dh: e.rect.h - o.rect.h }; + const summary = { elements: O.length, within: rows.filter((r) => !r.off && !r.hidden).length, off: rows.filter((r) => r.off).length, hidden: rows.filter((r) => r.hidden).length, missing: missing.length, extra: extra.length, unitOff: Math.abs(unit.dw) > tol || Math.abs(unit.dh) > tol }; + return { rows, missing, extra, unit, summary }; +} + +function rowFor(oe, ee, tol, byPosition) { + const d = { dx: ee.x - oe.x, dy: ee.y - oe.y, dw: ee.w - oe.w, dh: ee.h - oe.h }; + const off = Math.abs(d.dx) > tol || Math.abs(d.dy) > tol || Math.abs(d.dw) > tol || Math.abs(d.dh) > tol; + return { key: oe.key, role: oe.role, text: oe.text, edsText: ee.text, byPosition, origin: { x: oe.x, y: oe.y, w: oe.w, h: oe.h }, eds: { x: ee.x, y: ee.y, w: ee.w, h: ee.h, sel: ee.sel, role: ee.role }, ...d, off, hidden: !!ee.hidden, type: { origin: `${oe.fs}/${oe.lh}/${oe.fw}`, eds: `${ee.fs}/${ee.lh}/${ee.fw}` } }; +} + +const fmt = (r) => `${r.x},${r.y} ${r.w}×${r.h}`; +const sgn = (n) => (n > 0 ? `+${n}` : String(n)); +export function formatUnits(results, tol) { + const lines = []; + for (const res of results) { + for (const u of res.units) { + const { unit, summary } = u; + lines.push(`unit ${u.index + 1} of ${res.family ? `${res.family} (${res.required ? 'required' : 'advisory'}) ` : ''}${res.sel}: origin ${fmt(unit.origin)} vs served ${fmt(unit.eds)} → Δx ${sgn(unit.dx)} Δy ${sgn(unit.dy)} Δw ${sgn(unit.dw)} Δh ${sgn(unit.dh)}${summary.unitOff ? ' ✗ unit size' : ''}; ${summary.elements} elements: ${summary.within} within ${tol}px, ${summary.off} off, ${summary.hidden} hidden, ${summary.missing} missing, ${summary.extra} extra`); + lines.push('| element | origin x,y w×h | served x,y w×h | Δx | Δy | Δw | Δh | |', '|---|---|---|---|---|---|---|---|'); + for (const r of u.rows) lines.push(`| ${r.role} "${(r.text || r.key).slice(0, 36)}"${r.byPosition ? ` ≈ "${(r.edsText || '').slice(0, 20)}" (by position)` : ''} | ${fmt(r.origin)} | ${fmt(r.eds)}${r.eds.role !== r.role ? ` (${r.eds.role})` : ''} | ${sgn(r.dx)} | ${sgn(r.dy)} | ${sgn(r.dw)} | ${sgn(r.dh)} | ${r.hidden ? '✗ hidden (clipped)' : r.off ? '✗' : '✓'} |`); + for (const m of u.missing) lines.push(`| ${m.role} "${(m.text || m.key).slice(0, 36)}" | ${fmt(m)} | — | | | | | ✗ missing |`); + for (const x of u.extra) lines.push(`| ${x.role} "${(x.text || x.key).slice(0, 36)}" | — | ${fmt(x)} | | | | | 🟡 extra |`); + } + if (res.error) lines.push(`${res.sel}: ${res.error}`); + } + return lines.join('\n'); +} + +// ---- CLI ---------------------------------------------------------------------------------------------- +export function parseArgs(argv) { + const rest = argv.slice(2); + if (!rest.length || rest.includes('--help') || rest.includes('-h')) { console.log(HELP); process.exit(0); } + const opts = { origin: null, eds: null, units: [], n: 1, tol: 4, width: 1440, json: false, jsonFile: null, slug: null, force: false, advisory: false, plain: false, warmup: null, locale: 'en-US', families: null, template: null }; + for (let i = 0; i < rest.length; i += 1) { + const a = rest[i]; + if (a === '--unit') { const [o, e] = rest[++i].split('='); opts.units.push({ origin: o.trim(), eds: (e || o).trim() }); } + else if (a === '--families') opts.families = rest[++i]; + else if (a === '--template') opts.template = rest[++i]; + else if (a === '--n') opts.n = Number(rest[++i]); + else if (a === '--tol') opts.tol = Number(rest[++i]); + else if (a === '--width') opts.width = Number(rest[++i]); + else if (a === '--json') { opts.json = true; if (rest[i + 1] && !rest[i + 1].startsWith('--')) opts.jsonFile = rest[++i]; } + else if (a === '--slug') opts.slug = rest[++i]; + else if (a === '--force') opts.force = true; + else if (a === '--advisory') opts.advisory = true; + else if (a === '--plain') opts.plain = true; + else if (a === '--warmup') opts.warmup = rest[++i]; + else if (a === '--locale') opts.locale = rest[++i]; + else if (a.startsWith('--')) { console.error(`unknown flag ${a}\n\n${HELP}`); process.exit(1); } + else if (!opts.origin) opts.origin = a; else if (!opts.eds) opts.eds = a; + } + if (opts.families) { + if (!opts.slug) { console.error(`--families needs --slug\n\n${HELP}`); process.exit(1); } + const fam = JSON.parse(readFileSync(opts.families, 'utf8')); + let { template } = opts; + if (!template && existsSync('stardust/state.json')) { const pg = (JSON.parse(readFileSync('stardust/state.json', 'utf8')).pages || []).find((p) => p.slug === opts.slug); template = pg ? pg.template : null; } + opts.units.push(...unitsFor(fam, opts.slug, template)); + opts.n = Math.max(opts.n, ...opts.units.map((u) => u.n || 1)); + } + if (!opts.origin || !opts.eds || (!opts.units.length && !opts.families)) { console.error(`need and --unit = or --families --slug \n\n${HELP}`); process.exit(1); } + return opts; +} + +export async function measureUnits(browser, url, sels, { width, locale, warmup, n }) { + const { ctx, page } = await openPage(browser, { width, locale }); + try { + const v = await visit(page, url, { warmup }); + const bySel = {}; + for (const sel of sels) bySel[sel] = await page.evaluate(inPage(unitInventoryInPage, { sel, n })); + return { url, at: new Date().toISOString(), status: v.status, docH: v.docH, bySel }; + } finally { await ctx.close(); } +} + +/** Compare two measurement results for one --unit pair. */ +export function compareUnits(oInv, eInv, pair, tol) { + const o = oInv.bySel[pair.origin]; const e = eInv.bySel[pair.eds]; + const res = { sel: `${pair.origin}=${pair.eds}`, family: pair.family || null, required: pair.required !== false, originMatches: o.matches, edsMatches: e.matches, units: [] }; + if (o.error || e.error) { res.error = o.error ? `origin: ${o.error}` : `served: ${e.error}`; return res; } + if (!o.units.length || !e.units.length) { res.error = `no visible unit on the ${!o.units.length ? 'origin' : 'served'} side (${o.matches} / ${e.matches} DOM matches)`; return res; } + for (let i = 0; i < Math.min(o.units.length, e.units.length); i += 1) res.units.push({ index: i, origin: { path: o.units[i].path }, eds: { path: e.units[i].path }, ...alignUnit(o.units[i], e.units[i], tol) }); + return res; +} + +// `required` counters only feed the exit code; advisory families are reported, never blocking. +export const verdictOf = (results) => results.reduce((a, r) => { if (r.error) a.errors += 1; for (const u of r.units) { a.off += u.summary.off; a.hidden += u.summary.hidden; a.missing += u.summary.missing; a.within += u.summary.within; a.units += 1; if (r.required !== false) a.requiredOff += u.summary.off + u.summary.hidden + u.summary.missing; } return a; }, { units: 0, within: 0, off: 0, hidden: 0, missing: 0, errors: 0, requiredOff: 0 }); + +/** Families a page belongs to (by slug or template) → [{ family, origin, eds, n, required }]. */ +export function unitsFor(families, slug, template = null) { + return Object.entries(families || {}).filter(([, f]) => (f.pages || []).includes(slug) || (template && (f.templates || []).includes(template))) + .map(([family, f]) => ({ family, origin: f.origin, eds: f.build || f.eds || f.origin, n: f.n || 1, required: f.required !== false })); +} + +async function main() { + const opts = parseArgs(process.argv); + if (!opts.units.length) { console.log(`unit-geometry: none declared for ${opts.slug} in ${opts.families} → n/a`); return; } + const { chromium } = await import('playwright'); + const browser = await openBrowser(chromium, { tier: opts.plain ? 'plain' : 'stealth' }); + let oInv; let eInv; let tier; + try { + tier = browserTier(browser); + const cacheKey = opts.slug ? `${opts.slug}-units` : null; + const cached = cacheKey && !opts.force ? readCache(cacheKey) : null; + const oSels = opts.units.map((u) => u.origin); + if (cached && oSels.every((s) => cached.bySel && cached.bySel[s])) { oInv = cached; console.log(`origin: cached ${cacheKey} (${cached.at})`); } + else { oInv = await measureUnits(browser, opts.origin, oSels, { width: opts.width, locale: opts.locale, warmup: opts.warmup, n: opts.n }); if (cacheKey) writeCache(cacheKey, oInv); } + eInv = await measureUnits(browser, opts.eds, opts.units.map((u) => u.eds), { width: opts.width, locale: opts.locale, warmup: null, n: opts.n }); + } finally { await browser.close(); } + const results = opts.units.map((pair) => compareUnits(oInv, eInv, pair, opts.tol)); + const v = verdictOf(results); + const out = { _provenance: { writtenBy: 'unit-geometry.mjs', at: new Date().toISOString(), tol: opts.tol, n: opts.n, width: opts.width, tier }, origin: { url: opts.origin, at: oInv.at, docH: oInv.docH }, eds: { url: opts.eds, at: eInv.at, docH: eInv.docH }, results, verdict: v }; + if (opts.json && !opts.jsonFile) console.log(JSON.stringify(out, null, 1)); + else { + console.log(`unit-geometry @ ${opts.width}px, tol ${opts.tol}px — ${opts.origin} vs ${opts.eds}`); + console.log(formatUnits(results, opts.tol)); + console.log(`Units: ${v.units} compared; elements within ${v.within}, off ${v.off}, hidden ${v.hidden}, missing ${v.missing}${v.errors ? `; ${v.errors} selector error(s)` : ''} → ${v.requiredOff + v.errors ? 'FAIL' : v.off + v.hidden + v.missing ? 'PASS (advisory deltas)' : 'PASS'}`); + if (opts.jsonFile) { mkdirSync(dirname(opts.jsonFile) || '.', { recursive: true }); writeFileSync(opts.jsonFile, JSON.stringify(out, null, 1)); console.log(`json → ${opts.jsonFile}`); } + } + process.exitCode = !opts.advisory && (v.requiredOff + v.errors) > 0 ? 2 : 0; +} + +function safeRealpath(p) { try { return realpathSync(p); } catch { return p; } } +if (process.argv[1] && fileURLToPath(import.meta.url) === safeRealpath(process.argv[1])) { + main().catch((e) => { console.error(`unit-geometry error: ${String(e.message).split('\n')[0]}`); process.exit(e.name === 'BotChallengeError' ? 3 : 1); }); +} diff --git a/plugins/stardust/skills/extract/SKILL.md b/plugins/stardust/skills/extract/SKILL.md index c740e0bb9..e50cc95a3 100644 --- a/plugins/stardust/skills/extract/SKILL.md +++ b/plugins/stardust/skills/extract/SKILL.md @@ -89,8 +89,9 @@ Additional checks for this sub-command: `npx playwright --version` succeeds. Verify the module is import-resolvable from the project root (probe: `node -e "import('playwright').then(()=>process.exit(0))"`); if it - isn't, run `npm i -D playwright --no-save --legacy-peer-deps` (or - use the Playwright MCP server) before crawling. The + isn't, run `npm i -D playwright pixelmatch pngjs cheerio --legacy-peer-deps` + (devDependencies, never `--no-save` — #125) or use the Playwright MCP + server, before crawling. The `--legacy-peer-deps` flag is required on `aem-boilerplate` targets (their pinned `eslint@8` makes a plain `npm i` exit `ERESOLVE` before playwright is even considered). Don't trust the CLI diff --git a/plugins/stardust/skills/replica/SKILL.md b/plugins/stardust/skills/replica/SKILL.md index 7a60fd970..2986eef23 100644 --- a/plugins/stardust/skills/replica/SKILL.md +++ b/plugins/stardust/skills/replica/SKILL.md @@ -51,11 +51,16 @@ eyeballing. § Flow keys): invoking `replica` is the choice. 2. Verify Playwright is importable from the project root (extract needs it; so do the gate scripts). -3. Install the gate's pixel deps in the project: - `npm i -D playwright pixelmatch pngjs --no-save --legacy-peer-deps`. - Same trap as diff's prereq 0: a `--no-save` install is PRUNED by any later - real `npm i` — re-probe before every gate run - (`node -e "import('pixelmatch').then(()=>process.exit(0))"`). +3. Probe the gate's deps from the project root first — + `node -e "import('pixelmatch').then(()=>process.exit(0))"` (and playwright, + pngjs, cheerio) — and only on failure install them AS devDependencies, + never `--no-save`: `npm i -D playwright pixelmatch pngjs cheerio + --legacy-peer-deps` (a `--no-save` install is pruned by the next real + `npm i` — recorded twice in one run, #125). A harness that already + resolves them leaves the delivered code repo untouched; the + devDependencies otherwise land in the repo the skills push. Run every + probe from the project root: ESM resolves `playwright` from the script's + own location. 4. Copy scripts into the project and run them from there, not from the plugin: this skill's whole `scripts/` dir to `stardust/scripts/replica/`, the master skill's `../stardust/scripts/` @@ -253,6 +258,8 @@ node stardust/scripts/replica/anchor.mjs "$LIVE" --width 1440 --cache $G/anchor node stardust/scripts/replica/anchor.mjs "$PROTO" --width 1440 # build-side runs are free # Chrome: computed-style parity BEFORE any pixel round on header/footer/strips node stardust/scripts/replica/chrome-parity.mjs "$LIVE" "$PROTO" --width 1440 --live-cache $G/chrome-live.json # exit 0 = quiet, then crop-compare +# --headed anywhere in these instruments = the WINDOW-FREE real-Chrome stealth tier (live-session +# launchStealthHeaded, #125): it never opens a window; STARDUST_HEADED_WINDOW=1 is the only way to get one. # gate.sh: live.png cached, every step under a deadline (exit 124 = re-run, not FAIL). Rounds run in # the BACKGROUND: start every round at once (the slots pace the Chromiums — no `sleep N;` staggering), # then `wait` prints verdict lines only; exit 75 = still going → `wait` again as your NEXT step, never @@ -278,6 +285,10 @@ a verdict; 0 only when all four ran and passed). - **content-cap row: `cap-probe.mjs … --against` prints `cap-probe: PASS`** (gate doc § Pass bar item 6 — every live cap held within ±20 px by kind, nothing capped only on the prototype; a ✗ names the sizing rule, no pixel iteration); +- **clip-probe: `Clipped: 0`** on the build side (a `--full` round runs it; cut or hidden + text / controls fail the round like an over-threshold pixel diff — #125); a + declared repeated-unit family (`stardust/replica/units.json`, gate doc item 7) + within 4 px; - and, outside the bar and outside the cap, the horizontal-overflow assert: `document.documentElement.scrollWidth` within 4 px of the viewport (integer rounding; `GATE_OVERFLOW_TOLERANCE`) at every breakpoint on the @@ -285,6 +296,12 @@ a verdict; 0 only when all four ran and passed). says; a `capture failed (exit 1)` round (after gate.sh's one retry) is re-queued, never counted. +**Every prototype is a row (#125).** When the archetypes pass, one run per width +writes the prototype table — `node stardust/scripts/replica/gate-all.mjs --stage +prototype --proto-base "$PROTO_BASE" --width 1440` (then 360) through `run-bg.mjs`; +it reuses each archetype's cached `live.png`. `gate-evidence.mjs` reads it as the +source of record: a prototype without a row is ungated. + **Iteration discipline: hard cap 3 iterations per breakpoint.** Each iteration's fixes come off the instruments, never off eyeballing. After 3, log the residuals in the ledger and move on — a documented 2% residual beats @@ -415,11 +432,29 @@ had one section for the whole run). `stardust/replica/gates/-/` dir under the `pub` label (a new dir would force a fresh live capture). Only the published number counts. +- **The delivery gate is the ALL-PAGES run, four criteria — not the pixel + number alone** (`reference/source-fidelity-gate.md` § The all-pages + published-origin gate, #125): `node stardust/scripts/replica/gate-all.mjs + [--only ] [--skip-existing] [--eds-host ]` through `run-bg.mjs`. + DELIVERED = pixel % ≤ 10 AND |Δh| ≤ 5 % AND 0 clipped text / controls + (`clip-probe`) AND 0 MISSING / HIDDEN links / headings (`content-presence`); + a recorded page passed the pixel bar with all of its cards clipped. Evidence + and sidecars: `stardust/replica/gates/all-/`; with the Phase 4 + `prototypes-/` table it is the pair `gate-evidence.mjs` reads (a page + without a row is ungated). Each cluster subagent runs `--only` over its own + pages inside the fan-out; C-final's roster run is the recorded unit + `gate-all` (`--skip-existing`, captures reused) followed by + `update-coverage.mjs --gate` (handoff contract § 3, row C). **State:** replica writes its own state under `stardust/replica/` — the inconsistency register, `progress.json` (per page type: archetype slug, iterations used, per-breakpoint gate results, residuals, motion inventory), `motion/.json`, and `gates/-/` evidence. +The delivery gate (#125) adds `gates/all-/` — one dir per deployed +page (`origin.png`, `eds.png`, `diff.png`, `pixel.json`, `content.json`, +`clip.json`, `units.json`), `summary.{json,md}`, `runs/` for `--only` runs and +the documented sidecars (`masks.json`, `overrides.json`, `clip-allow.json`, +`presence.json`, `units.json`). Phase 5 adds three files under `stardust/rollout/`: `progress.json` (the C-deliver unit ledger — status, gates and verdict per unit), `foundation-freeze.json` (the sha256 manifest of the frozen foundation) and diff --git a/plugins/stardust/skills/replica/reference/handoff-contract.md b/plugins/stardust/skills/replica/reference/handoff-contract.md index 476e8d13f..e41b3e6a7 100644 --- a/plugins/stardust/skills/replica/reference/handoff-contract.md +++ b/plugins/stardust/skills/replica/reference/handoff-contract.md @@ -62,11 +62,11 @@ Ledger phase strings are NOT named in `../../rollout/SKILL.md` or its scripts (s | A — Inventory | `A-inventory` | `coverage/pages.json`, `coverage/templates.json`, `rollout.json` (fill `site.da.org`, `site.site`, `site.da.ref`, `site.liveHost`) | `inventory.mjs --site-url [--state stardust/state.json]` | | B — Block dedup plan | `B-block` | `coverage/blocks.json` (distinct blocks → `edsBlockName`), `plan.json` (representative-first; per page `convert` / `reuse` = deploy's brief input) | `blocks.mjs`, then `plan.mjs` | | B2 — Dynamic surface | `B2-dynamic` | `stardust/dynamic-features.md` verified against fresh evidence — pre-import gate, blocks C | `stardust/scripts/dynamics/dynamics-detect.mjs`, `stardust/scripts/dynamics/dynamics-plan.mjs --target-origin ` (dynamics skill, § 4 copy step) | -| C — Deliver the site | `C-deliver` | ONE ledger phase (a single `C-deliver` start/end pair) run as RECORDED UNITS in `stardust/rollout/progress.json` (shape below), so a session that has to end (a hard blocker, an owner decision, a wait longer than the master's wait discipline allows) ends at a unit end and a resumed one runs only the units not `done` — a recorded run spent 118 turns in one C-deliver session and grew its context from 177k to 520k while its four cluster reports were about 1k characters each. Every page still goes through § 2's chain in this per-page order: convert (brief carries the plan step AND EW1–EW10) → `delivery-lint.mjs` + boilerplate lint → `media-reconcile.mjs` → `davids-model-lint.mjs` → `sanitise.js` → decoration asserts on the local harness (`qa-gate.mjs`, `block-roundtrip.mjs --ew`; structural exit codes only) → PUT → preview → `.plain.html` checks → `deployed` → THEN the visual gates against the PUBLISHED origin — replica's source-fidelity gate (pixel), `crop-compare.mjs` header/footer bands, chrome parity (§ 2, final paragraph; `../../deploy/SKILL.md § Step 10 — Reconcile on the DEPLOYED URL`) → `update-coverage.mjs`. The units: **C0 foundation** — ONE foundation subagent authors AND deploys the foundation under the same fan-out rules (below): `stardust/runtime-contract.json`, `styles/`, fonts, favicon, the header and footer blocks, the per-locale nav/footer documents and locale shells when the site has them; lint; PUT → preview (→ publish when § 2's publish decision says so); `.plain.html` checks — and hands back ONE verdict line plus the list of foundation files. The MAIN agent then runs the foundation-first gate on the shell template against the published origin — pixel at both widths, `crop-compare` header/footer bands, chrome parity (the gate runs after PUT + preview, its probes against the preview URL — `../../rollout/SKILL.md § Phase C`); it fixes nothing itself — a failing gate goes back to the foundation subagent as one message carrying the numbers — then `foundation-freeze.mjs freeze`, the checkpoint commit, unit `foundation` recorded `done` (a recorded run's main agent authored the foundation itself and grew from 107k to 337k tokens of context inside this one unit, more than its four cluster units together). **C1…Cn, one unit per template cluster** — ONE subagent per cluster, clusters spawned concurrently (disjoint blocks + pages): the subagent runs the WHOLE chain above for its blocks and pages — through the local structural asserts, then PUT → preview → `.plain.html` checks → the published-origin gates on the preview URLs (pixel on the archetype + one sibling per width, `crop-compare`, chrome parity) → its coverage rows — writing its page list to `stardust/rollout/units/.paths` (one DA path per line) and driving `deploy-batch.mjs --paths stardust/rollout/units/.paths --ledger stardust/deploy/ledger-.json --log stardust/.work/rollout//deploy.log --concurrency 2` through `run-bg.mjs` (`--concurrency 1` when more than three clusters run at once — several clusters PUT together and the admin API tolerates about 4–6 PUTs in flight site-wide; `--no-publish` until § 2's publish decision is made). The ledgers are lock-safe, so clusters deploy at once without clobbering each other (fan-out discipline below); `DA_TOKEN` is read from its file at each call and never copied into a brief; a 401 halt is recovered by re-running the same command once the token file is fresh — the batch ledger resumes it (the deploy card's `DA_TOKEN` lifecycle rule, `../../deploy/SKILL.md § Deploy (DA Source API, from a local agent)`). The subagent reports ONE verdict line: `: pages n/n live, blocks k, gate 1440 x% / 360 y%, requests r` (a recorded 36-page hands-off run delivered 36/36 this way with zero 401s, its coordinator at or below 131k tokens of context across three cluster units). The main agent is a coordinator — spawn, wait, record the unit in progress.json, commit the checkpoint — and never runs a page instrument or edits CSS during the wave. **After EACH unit is recorded and committed**, C0 included, the run has a safe resume point; it continues with the next unit — a unit end is never by itself a reason to end the session (the master's turn-end rule: end only on completion, a hard blocker, an owner decision, or a wait longer than ~45 minutes). A session resumed inside C-deliver reads progress.json and runs only the units not `done`; a unit that was in flight is re-run from its recorded state — pages already live are re-verified, not re-authored. **C-final** (main agent) — `foundation-freeze.mjs check`; the queued `stardust/rollout/foundation-requests.md` lines applied ONCE; the published gate re-run on each affected template's archetype at both widths; `update-coverage.mjs`; then the single `C-deliver end` ledger line. Rollout's own pre-PUT gates are static, not pixel: `section-fidelity.mjs` outline (informs, never decides), image (= media-reconcile), path-safety, hygiene, tier declared (`../../rollout/reference/delivery-gates.md`). No pixel or visual gate runs on the local harness before the first PUT — the harness serves the structural asserts only; its pixel numbers are not a verdict (`../../deploy/SKILL.md § Local QA before deploy`). A recorded delivery session iterated CSS against a local harness diff for its whole budget and delivered no page. | `delivery-lint.mjs`, `media-reconcile.mjs`, `section-fidelity.mjs`, `update-coverage.mjs`, `stardust/scripts/deploy/deploy-batch.mjs`, `stardust/scripts/replica/foundation-freeze.mjs` | +| C — Deliver the site | `C-deliver` | ONE ledger phase (a single `C-deliver` start/end pair) run as RECORDED UNITS in `stardust/rollout/progress.json` (shape below), so a session that has to end (a hard blocker, an owner decision, a wait longer than the master's wait discipline allows) ends at a unit end and a resumed one runs only the units not `done` — a recorded run spent 118 turns in one C-deliver session and grew its context from 177k to 520k while its four cluster reports were about 1k characters each. Every page still goes through § 2's chain in this per-page order: convert (brief carries the plan step AND EW1–EW10) → `delivery-lint.mjs` + boilerplate lint → `media-reconcile.mjs` → `davids-model-lint.mjs` → `sanitise.js` → decoration asserts on the local harness (`qa-gate.mjs`, `block-roundtrip.mjs --ew`; structural exit codes only) → PUT → preview → `.plain.html` checks → `deployed` → THEN the visual gates against the PUBLISHED origin — replica's source-fidelity gate (pixel), `crop-compare.mjs` header/footer bands, chrome parity (§ 2, final paragraph; `../../deploy/SKILL.md § Step 10 — Reconcile on the DEPLOYED URL`) → `update-coverage.mjs`. The units: **C0 foundation** — ONE foundation subagent authors AND deploys the foundation under the same fan-out rules (below): `stardust/runtime-contract.json`, `styles/`, fonts, favicon, the header and footer blocks, the per-locale nav/footer documents and locale shells when the site has them; lint; PUT → preview (→ publish when § 2's publish decision says so); `.plain.html` checks — and hands back ONE verdict line plus the list of foundation files. The MAIN agent then runs the foundation-first gate on the shell template against the published origin — pixel at both widths, `crop-compare` header/footer bands, chrome parity (the gate runs after PUT + preview, its probes against the preview URL — `../../rollout/SKILL.md § Phase C`); it fixes nothing itself — a failing gate goes back to the foundation subagent as one message carrying the numbers — then `foundation-freeze.mjs freeze`, the checkpoint commit, unit `foundation` recorded `done` (a recorded run's main agent authored the foundation itself and grew from 107k to 337k tokens of context inside this one unit, more than its four cluster units together). **C1…Cn, one unit per template cluster** — ONE subagent per cluster, clusters spawned concurrently (disjoint blocks + pages): the subagent runs the WHOLE chain above for its blocks and pages — through the local structural asserts, then PUT → preview → `.plain.html` checks → the published-origin gates on the preview URLs (pixel on the archetype + one sibling per width, `crop-compare`, chrome parity) → `gate-all.mjs --only --eds-host ` over its own pages, in parallel with the other clusters (#125: the added wall time stays inside the fan-out) → its coverage rows — writing its page list to `stardust/rollout/units/.paths` (one DA path per line) and driving `deploy-batch.mjs --paths stardust/rollout/units/.paths --ledger stardust/deploy/ledger-.json --log stardust/.work/rollout//deploy.log --concurrency 2` through `run-bg.mjs` (`--concurrency 1` when more than three clusters run at once — several clusters PUT together and the admin API tolerates about 4–6 PUTs in flight site-wide; `--no-publish` until § 2's publish decision is made). The ledgers are lock-safe, so clusters deploy at once without clobbering each other (fan-out discipline below); `DA_TOKEN` is read from its file at each call and never copied into a brief; a 401 halt is recovered by re-running the same command once the token file is fresh — the batch ledger resumes it (the deploy card's `DA_TOKEN` lifecycle rule, `../../deploy/SKILL.md § Deploy (DA Source API, from a local agent)`). The subagent reports ONE verdict line: `: pages n/n live, blocks k, gate 1440 x% / 360 y%, requests r` (a recorded 36-page hands-off run delivered 36/36 this way with zero 401s, its coordinator at or below 131k tokens of context across three cluster units). The main agent is a coordinator — spawn, wait, record the unit in progress.json, commit the checkpoint — and never runs a page instrument or edits CSS during the wave. **After EACH unit is recorded and committed**, C0 included, the run has a safe resume point; it continues with the next unit — a unit end is never by itself a reason to end the session (the master's turn-end rule: end only on completion, a hard blocker, an owner decision, or a wait longer than ~45 minutes). A session resumed inside C-deliver reads progress.json and runs only the units not `done`; a unit that was in flight is re-run from its recorded state — pages already live are re-verified, not re-authored. **C-final** (main agent) — `foundation-freeze.mjs check`; the queued `stardust/rollout/foundation-requests.md` lines applied ONCE; the published gate re-run on each affected template's archetype at both widths; then unit **`gate-all`** (kind `final`, recorded `running` → `done` in progress.json like every unit): `gate-all.mjs --skip-existing` over the whole roster through `run-bg.mjs` — the captures the clusters took are reused, so the pass is compares + probes only; still 30–45 min per 64 pages, the longest single step of a run, so a session may end at its unit boundary and resume after it; then `update-coverage.mjs --gate stardust/replica/gates/all-/summary.json` (a page the table failed reads `failed`, never `verified`, until it passes or carries a documented override); then the single `C-deliver end` ledger line. Rollout's own pre-PUT gates are static, not pixel: `section-fidelity.mjs` outline (informs, never decides), image (= media-reconcile), path-safety, hygiene, tier declared (`../../rollout/reference/delivery-gates.md`). No pixel or visual gate runs on the local harness before the first PUT — the harness serves the structural asserts only; its pixel numbers are not a verdict (`../../deploy/SKILL.md § Local QA before deploy`). A recorded delivery session iterated CSS against a local harness diff for its whole budget and delivered no page. | `delivery-lint.mjs`, `media-reconcile.mjs`, `section-fidelity.mjs`, `update-coverage.mjs`, `stardust/scripts/deploy/deploy-batch.mjs`, `stardust/scripts/replica/foundation-freeze.mjs` | | D — Site assembly | `D-site` | `site/{sitemap.xml,robots.txt,manifest.json}`; wire `stardust/redirects.tsv` into the redirects sheet at the content root (on a DA-backed site `/redirects.json`, columns Source / Destination; PUT via the admin API, preview + publish); the root MUST answer: when the source root serves a page, that page is delivered as the root `index` document (the pipeline serves `/` from it — `/index` ≡ `/`; never also a `/` row in the sheet, which would shadow it); only when the source root itself redirects (`curl -sI ` and follow the Location chain) does the sheet carry `/` and `/index.html` → the landing page; verify `curl -sIL https://----.aem.page/` ends in 200 — a 404 on `/` fails the whole delivery after every phase has passed (recorded: a sheet with a row per `.html` path and none for `/`); `/nav` + `/footer` published; the assembled `site/sitemap.xml` is the EXPECTED set only — `stardust/rollout/` is in `.hlxignore`, the platform serves its own `/sitemap.xml` from its index of published documents (a recorded hands-off run reported "sitemap 36 urls" from the local file while the served one listed 58: every page plus 22 chrome documents); every chrome/fragment document (`/nav`, `/footer`, per-locale `nav-*` / `footer-*`, locale shells that are not pages) carries a `Robots \| noindex` metadata row and is re-published when it lacked one; then `assemble.mjs --verify-origin https://----.aem.live` MUST exit 0 (served `` path set == coverage rows; an extra path is a chrome document without `noindex` or a D2-built page without a coverage row, a missing path is a page not published live); the `D-site end` line's `--detail` names the SERVED count — `"sitemap served 36 = assembled 36; redirects wired; / 200"` — never the assembled one | `assemble.mjs`, `assemble.mjs --verify-origin ` | | D2 — Dynamic features | `D2-dynamic` | `stardust/dynamics/parity.json` — one replayable row per feature; a 404 on `/query-index.json` is answered by `helix-query.yaml` in the code branch FIRST (skeleton `../../dynamics/reference/listings.md § Getting an index at all`): commit, push, publish the pages live, poll no more often than every 5 s for at most 10 minutes until `total` settles — no configuration-service write is involved (a 403 there with the migration token is expected), the sheet-backed interim index only when the code branch is not writable (recorded `interim`, decision named); every `search-query` row carries the source's `expectCount` + `expectTitles` (+ `expectIncludes`) read at detect time and the replay fails on a count mismatch; every page built here (the search results page) gets a coverage row — `update-coverage.mjs --new --path --template --origin dynamics [--title …]`, kept across `inventory.mjs` re-runs | dynamics skill Phases 4–5, `stardust/scripts/dynamics/dynamics-check.mjs --origin ` | | D3 — Multilingual (optional) | `D3-multilingual` | per-language content trees reusing the block library | `../../rollout/reference/multilingual.md` | -| E — Full-site verify | `E-full-site` | each page `verified` / `failed`; headless render check on the first page of each template | `verify.mjs [--base ]` | +| E — Full-site verify | `E-full-site` | each page `verified` / `failed` (a `delivery.gate.pass: false` from the pixel table never reads verified); headless render check on the first page of each template | `verify.mjs [--base ]`; the table: `update-coverage.mjs --gate stardust/replica/gates/all-/summary.json` | | E2 — Link-audit completeness | `E2-link-audit` | nav/footer/landing targets deployed + published + verified; internal hrefs root-relative, extensionless; every href GET against the live tree. Internal targets missing from the capture split two ways, both recorded in `direction.md` as a named decision with the list: a target the direction's caps meant to include is a CAPTURE GAP — crawl it first (it enters the roster through capture → migrate → gate, never authored from a live read) and deliver when the gap is ≤ 12 pages, else repoint to the source site and list it as scope debt; a target outside the declared caps is SCOPE EXTENSION → repoint regardless of count; external targets → repoint; other-locale targets → repoint ONLY when that locale root was never captured; any captured page (a `stardust/state.json` row, including `duplicateOf` shells and locale roots) is in scope and is delivered, never repointed — the ≤ 12 default applies to UNcaptured in-scope pages only (a recorded run dropped ten captured locale roots by citing this row; another delivered 10 uncaptured in-scope pages — the right call); never leave a 404; the locale-root subtrees the crawl never captured are already listed in `stardust/state.json` `site.captureGaps` (extract copies them from `_crawl-log.json#captureGaps` and names the counts in its end ledger line) — a gap listed there was decided at Phase 1 and crawled before C-deliver, never discovered here | `stardust/scripts/deploy/localize-links.mjs --source-host --content content --redirects stardust/redirects.tsv`, then `--check` | | F — Optimize | `F-optimize` | `optimize/findings.json`, `optimize/scorecard.json`; gate exits non-zero on any open in-scope P1 | `optimize.mjs`, `findings.mjs record` / `resolve` | | G — AEM autofix | `G-aem` | EDS project edits for fixable findings → re-deploy → re-run verify + optimize | `autofix-aem.mjs --project --dry-run`, then without | diff --git a/plugins/stardust/skills/replica/reference/recreation-procedure.md b/plugins/stardust/skills/replica/reference/recreation-procedure.md index 0dfeabe37..8cbf151da 100644 --- a/plugins/stardust/skills/replica/reference/recreation-procedure.md +++ b/plugins/stardust/skills/replica/reference/recreation-procedure.md @@ -70,6 +70,17 @@ the chrome crop gate (`source-fidelity-gate.md` § Pass bar, item 5) against the NEW page's live chrome, and flag any page-level compensation for back-port into the canon files so later archetypes don't re-discover it. +### Repeated-unit families (#125) + +A block that repeats a unit — cards, rail items, list rows, FAQ rows — is declared once in +`stardust/replica/units.json` when it is authored: `{ "": { "origin": "", "build": "", "n": 2, "required": true, "pages": [""], +"templates": ["