Pan to offscreen elements in video mode

[mmkal]

When a test waits for or clicks an element outside the viewport, the rendered video now pans to it — a smooth eased camera travel, the usual pointer/outline hold, then onward — without the test ever scrolling the page.

await page.getByText("Deploy succeeded").waitFor(); // no real scroll → video pans down, holds, pans back
await page.getByRole("button", { name: "View summary" }).click(); // real scroll → video pans down and stays

What really scrolls in Playwright, and what the video does

Operation	Live page	Rendered video
waitFor(), expect(...).toBeVisible() and other queries	never scrolls — they only observe	pan to the element, hold, pan back
click(), check(), hover() and other interactions	really scrolls — actionability runs scrollIntoViewIfNeeded before acting	pan to the element, hold, stay where the post-action footage resumes

Two refinements keep multi-scroll tests watchable:

• Adjacent pans hand over instead of yo-yoing. A waitFor pan directly followed by another pan (the extremely common wait-then-click) skips its return leg; the next pan enters from the previous destination — zero travel when both point at the same view, so the hold simply continues into the action.
• Pan destinations center the element, matching Chromium's own scroll-for-action alignment — so wait-then-act pairs land on the same view, and the held element stays clear of the caption band.

The stay-pan adopts the browser's actual scroll destination after the action, so the pan lands pixel-exact on the resumed footage — replacing what used to be an instant jump-cut. fill() keeps its reveal pipeline unpanned, and elements clipped by inner scroll containers keep plain highlighting.

How it stays true to the record-facts/render-effects philosophy: at highlight time video mode captures a beyond-viewport screenshot (fullPage + document-coordinate clip — Chromium renders offscreen pixels without scrolling: no scroll event, scroll position untouched, IntersectionObservers never fire, so lazy-loading can't trigger). The renderer animates a viewport-sized crop over that tall still with cosine-eased ffmpeg time expressions.

Known costs, handled: Chromium implements beyond-viewport capture by momentarily resizing the renderer, leaking one zoomed-out frame into the raw screencast; the pan piece consumes that span from the source so it never reaches the rendered video. The page observes a single no-op resize event with unchanged dimensions.

Also fixes a pre-existing race this work exposed: a selector-driven trimStart resolves over the protocol and could land 1–2ms after a highlight recorded at effectively the same moment, silently dropping that highlight from the render. A trim start within one source frame after a highlight start now moves back to it (the flaky text-cursor spec went from 2/3 failing to 8/8 passing).

Raw vs rendered

The checked-in demo — spec/scroll-pan-demo.spec.ts — uses the same narrow 480×720 viewport as the todo-app spec. It has targets both below the fold (success card, "View summary" button) and above it (header badge, "Copy status link" button), and asserts the live scroll position plus every recorded pan mode. The clip is caption-free so none of the page or motion is obscured.

Raw (left): waits never move; the two clicks cause instant browser scrolls. Rendered (right): waits pan smoothly, adjacent wait/click pans hand over without yo-yoing, and click pans meet the live footage at its real scroll destination. The raw stream ends first and holds its final frame while the rendered timeline completes.

scroll-pan-raw-vs-rendered.mp4

Todo-app baseline

Standard visual baseline — no pans trigger here (everything fits the viewport), pacing unchanged.

todo-app.mp4

Raw recording (recording/rendering behavior changed on this branch, per repo guidance):

todo-app-raw.mp4

Validation

108 passed / 4 provider-gated skips. After merging current main: 8 focused pan/fill-ordering checks passed, plus typecheck, build, and publint. Caption-free narrow demo: 1/1 passed. Task file: tasks/video-mode-scroll-pan.md

🤖 Generated with Claude Code

Session: c46b4c3b-ccee-4225-aa58-7d8d22fcba39

---

Note

Medium Risk
Large changes to video-mode recording and ffmpeg rendering paths; behavior is heavily tested but regressions in highlight timing, trim, or non-pan highlights are possible.

Overview
video mode now pans the rendered video to offscreen highlight targets using a beyond-viewport screenshot and ffmpeg crop animation, while the live page never scrolls for waitFor (and similar queries).

waitFor highlights use pan down, hold, pan back; actions like click use pan down and stay, with finalizePanHighlightAfterAction aligning the synthetic pan to Playwright’s real post-scroll position so live footage resumes without a jump-cut. fill stays unpanned; targets clipped by inner scroll containers skip panning. Adjacent pans hand over (suppress return legs) so wait-then-click does not yo-yo.

Metadata adds VideoModePan; the renderer consumes Chromium’s capture-flash span so it never appears in the output. A trimStart race fix keeps highlights that land within one frame after selector trim start. README documents behavior and limitations; spec/scroll-pan-demo.spec.ts and FFmpeg/metadata specs cover pans.

^{Reviewed by Cursor Bugbot for commit 50c8e66. Bugbot is set up for automated code reviews on this repo. Configure here.}

[pkg-pr-new]

Open in StackBlitz

pnpm add https://pkg.pr.new/middlewright@22

commit: 50c8e66

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 50c8e66. Configure here.}

[cursor]

Stay pan breaks when scroll unchanged

Medium Severity

finalizePanHighlightAfterAction always adopts the post-action scroll, but when the action does not move the page (blur, click({ force: true }), or a handler that scrolls back), pan.to collapses onto pan.from. panDurationMs still enforces PAN_MIN_MS, and the hold rect is rewritten to offscreen coordinates, so the render inserts a idle pan and an invisible highlight before live footage resumes.

Additional Locations (2)

• src/plugins/video-mode.ts#L1059-L1069
• src/plugins/video-mode.ts#L4226-L4227

 

^{Reviewed by Cursor Bugbot for commit 50c8e66. Configure here.}