Render video dialogs entirely with ASS

[mmkal]

Summary

Render alert(), confirm(), and prompt() wholly in video post-production with ASS shapes and text. The raw Playwright recording stays clean: video mode freezes the last app frame before the native dialog, draws a deterministic dialog over that frame, then resumes after Playwright resolves the real dialog.

page.once("dialog", (dialog) => dialog.accept("secret"));
await page.getByRole("button", { name: "Sign in" }).click();

// Rendered video: neutral prompt → typed text → selected OK.
// Raw video: application page only.

This removes the dialog DOM overlay, init script, browser screenshots, and per-dialog image layers. Playwright's real Dialog still owns listener order, automatic dismissal, accepted text, and accept/dismiss behavior.

What changes

• Freeze one clean raw frame for the full synthetic-dialog interval.
• Draw dimming, panel, wrapped message, prompt input, and buttons in video-mode-dialogs.ass.
• Reveal prompt responses one Unicode grapheme at a time.
• Keep both actions neutral until typing ends; select the resolved action blue once, with the pointer composited above it.
• Preserve alert, confirm, prompt, back-to-back dialog, no-listener auto-dismiss, dead-air, pointer, and final-hold behavior.

Trade-offs

Area	ASS approach
Code	src/plugins/video-mode.ts is +565/−303 lines versus main (+262 net). Layout and ASS helpers replace page lifecycle and screenshot-layer code.
Render cost	Adds one libass filter inside the existing FFmpeg graph, not another encode. It removes browser DOM mutation and per-dialog screenshot inputs. No benchmark is claimed.
Layout	Deterministic rectangles and explicit wrapping are easy to reason about, but text metrics are approximate and complex/native browser layout is intentionally out of scope.
Portability	No new package or browser-context requirement. It uses FFmpeg's existing ASS/libass support, already used by video-mode captions and address bars.
Fidelity	Stable across browsers and immune to live-capture frame leaks, but it is a designed synthetic dialog rather than a pixel copy of any browser's native prompt.

Review videos

Focused prompt — rendered:

prompt-rendered.mp4

Focused prompt — raw:

prompt-raw.mp4

Todo app from #15 — rendered:

todo-rendered.mp4

Todo app from #15 — raw:

todo-raw.mp4

Validation

• 31 FFmpeg integration specs, including exact 25 fps frame-order and raw-artwork regressions
• Prompt sequence stress: 10/10 passes across 5 workers
• Full suite: 98 passed, 3 provider-gated skipped
• pnpm typecheck
• pnpm build
• pnpm exec publint

Coding agent session: 019fc7f7-de88-7690-8d0a-31790c4dbc29

---

Note

Medium Risk
Touches core video-mode FFmpeg rendering and dialog timing/metadata; behavior is heavily tested but regressions could affect rendered artifact fidelity or highlight ordering.

Overview
videoMode no longer injects synthetic dialog DOM, init scripts, or per-dialog screenshots. Dialog scenes are drawn in post with ASS (video-mode-dialogs.ass): freeze the last clean app frame before the native dialog, overlay panel/message/buttons, then resume raw footage after resolution.

Prompt handling is tightened: metadata can carry defaultValue separately from promptText (e.g. explicit empty accept vs Unicode default). Rendered prompts reveal text by grapheme, keep OK/Cancel neutral until typing finishes, then highlight the resolved action in blue—ordering is covered by new FFmpeg pixel/frame regressions and raw-video checks that synthetic white panels never appear.

Playwright’s real Dialog still drives accept/dismiss, listener order, and auto-dismiss; README documents the ASS-only path. Large refactor in video-mode.ts (~+565/−303) replacing overlay lifecycle with layout helpers and an FFmpeg ass filter step.

^{Reviewed by Cursor Bugbot for commit 95d27a5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[pkg-pr-new]

Open in StackBlitz

pnpm add https://pkg.pr.new/middlewright@19

commit: 95d27a5

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 95d27a5. Configure here.}

[cursor]

Dialog pointer misses ASS buttons

Medium Severity

Dialog hit targets are laid out in page-viewport space at capture time, while ASS redraws the dialog with videoModeDialogLayout on the scaled video viewport. That layout scale is clamped to [0.75, 1.5], so it is not homogeneous with Playwright’s linear video downscale. Pointer overlays then use scaleHighlight on the capture-time rects and can land off the ASS OK/Cancel controls whenever the clamp bites, for example a 1920×1080 viewport with the default fit-within-800 video size.

Additional Locations (2)

• src/plugins/video-mode.ts#L1551-L1561
• src/plugins/video-mode.ts#L2337-L2349

 

^{Reviewed by Cursor Bugbot for commit 95d27a5. Configure here.}