Highlight visible waitFor results in video mode - #15
Merged
Conversation
commit: |
…itfor-highlight # Conflicts: # src/plugins/video-mode.ts
mmkal
added a commit
that referenced
this pull request
Aug 3, 2026
## Summary
Render `alert()`, `confirm()`, and `prompt()` wholly in video
post-production with ASS shapes and text. The raw Playwright recording
stays clean: video mode freezes the last app frame before the native
dialog, draws a deterministic dialog over that frame, then resumes after
Playwright resolves the real dialog.
```ts
page.once("dialog", (dialog) => dialog.accept("secret"));
await page.getByRole("button", { name: "Sign in" }).click();
// Rendered video: neutral prompt → typed text → selected OK.
// Raw video: application page only.
```
This removes the dialog DOM overlay, init script, browser screenshots,
and per-dialog image layers. Playwright's real `Dialog` still owns
listener order, automatic dismissal, accepted text, and accept/dismiss
behavior.
## What changes
- Freeze one clean raw frame for the full synthetic-dialog interval.
- Draw dimming, panel, wrapped message, prompt input, and buttons in
`video-mode-dialogs.ass`.
- Reveal prompt responses one Unicode grapheme at a time.
- Keep both actions neutral until typing ends; select the resolved
action blue once, with the pointer composited above it.
- Preserve alert, confirm, prompt, back-to-back dialog, no-listener
auto-dismiss, dead-air, pointer, and final-hold behavior.
## Trade-offs
| Area | ASS approach |
| --- | --- |
| Code | `src/plugins/video-mode.ts` is +565/−303 lines versus `main`
(+262 net). Layout and ASS helpers replace page lifecycle and
screenshot-layer code. |
| Render cost | Adds one libass filter inside the existing FFmpeg graph,
not another encode. It removes browser DOM mutation and per-dialog
screenshot inputs. No benchmark is claimed. |
| Layout | Deterministic rectangles and explicit wrapping are easy to
reason about, but text metrics are approximate and complex/native
browser layout is intentionally out of scope. |
| Portability | No new package or browser-context requirement. It uses
FFmpeg's existing ASS/libass support, already used by video-mode
captions and address bars. |
| Fidelity | Stable across browsers and immune to live-capture frame
leaks, but it is a designed synthetic dialog rather than a pixel copy of
any browser's native prompt. |
## Review videos
Focused prompt — rendered:
https://github.com/user-attachments/assets/0b33eb3a-ba6d-4b15-972a-66bcdf90d932
Focused prompt — raw:
https://github.com/user-attachments/assets/162a8e53-3d2d-4994-adf5-00ce8e666309
Todo app from #15 — rendered:
https://github.com/user-attachments/assets/edd3c617-42cd-4e3d-9126-535bcd7aab8f
Todo app from #15 — raw:
https://github.com/user-attachments/assets/83eeae7e-d389-4ea7-9daf-7439a7640f2c
## Validation
- 31 FFmpeg integration specs, including exact 25 fps frame-order and
raw-artwork regressions
- Prompt sequence stress: 10/10 passes across 5 workers
- Full suite: 98 passed, 3 provider-gated skipped
- `pnpm typecheck`
- `pnpm build`
- `pnpm exec publint`
Coding agent session: `019fc7f7-de88-7690-8d0a-31790c4dbc29`
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Touches core video-mode FFmpeg rendering and dialog timing/metadata;
behavior is heavily tested but regressions could affect rendered
artifact fidelity or highlight ordering.
>
> **Overview**
> **videoMode** no longer injects synthetic dialog DOM, init scripts, or
per-dialog screenshots. Dialog scenes are drawn in post with ASS
(`video-mode-dialogs.ass`): freeze the last clean app frame before the
native dialog, overlay panel/message/buttons, then resume raw footage
after resolution.
>
> Prompt handling is tightened: metadata can carry **`defaultValue`**
separately from **`promptText`** (e.g. explicit empty accept vs Unicode
default). Rendered prompts reveal text by grapheme, keep OK/Cancel
neutral until typing finishes, then highlight the resolved action in
blue—ordering is covered by new FFmpeg pixel/frame regressions and
raw-video checks that synthetic white panels never appear.
>
> Playwright’s real **`Dialog`** still drives accept/dismiss, listener
order, and auto-dismiss; README documents the ASS-only path. Large
refactor in `video-mode.ts` (~+565/−303) replacing overlay lifecycle
with layout helpers and an FFmpeg **ass** filter step.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
95d27a5. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
…itfor-highlight # Conflicts: # src/plugins/video-mode.ts
…eo-mode-waitfor-highlight
mmkal
marked this pull request as ready for review
August 3, 2026 16:26
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 73633c7. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Visible
locator.waitFor()results now receive the same post-rendered pointer/outline hold as clicks and other meaningful actions. The elapsed wait remains dead air, and the synthetic hold does not add its configured duration to test runtime.The existing
highlight.durationcontrols the hold. Callers can retain the old behavior withskipMethods: ["waitFor"]. Hidden or detached terminal states do not produce an empty highlight.Video mode's own selector and dialog-overlay waits bypass action middleware, so internal bookkeeping cannot appear as user-facing highlights.
Rendered holds no longer calibrate their source timeline by matching the final page screenshot. That heuristic could select an identical state from seconds earlier—or miss a final paint that Playwright never encoded—and replay old footage. The settled recorder endpoint is now the sole clock marker, with a frame-level regression for a final state that repeats an earlier state.
videoMode()now uses the showcase pacing without extra options: dead air is capped at 300ms, pointer highlights hold for 1000ms, the final frame holds for 1000ms, and startup trimming remains automatic.Showcase
todo-app.spec.tsis a client-only kitchen-sink example with an in-memoryTodoDB. The explicit test flow changesdb.setDelay(ms)between interactions, then:The reviewed raw/rendered comparison is attached below. The deliberately slow raw sign-in makes the dead-air compression easy to compare.
Validation
Codex session:
019fc75f-5c86-7f43-90c6-aaafd1ebb46eVideos
Rendered
todo-app.webm
Raw Playwright recording
todo-app-raw.webm
Raw vs rendered
Raw is left; rendered is right.
todo-raw-vs-rendered.mp4
Note
Medium Risk
Changes default video pacing and rendering clock logic in
video-mode.ts, which can alter every rendered artifact; behavior is heavily tested but regressions would be visual/timing rather than test-runtime failures.Overview
Video mode now treats successful visible
waitFor()like other locator actions: the wait stays dead air (still compressible viadeadAirThreshold), then the rendered video points at and holds the resolved element using the shared highlight mode and duration. Opt out withskipMethods: ["waitFor"]; hidden/detached results do not get an empty highlight. Internal trim-start and overlay waits usewaitFor_originalso bookkeeping does not show up as user highlights.Defaults shift to review-friendly pacing without extra options: dead-air cap 300ms, final hold 1000ms (was 3000ms), and
waitForis no longer skipped by default.Timeline calibration drops final-page screenshot matching (it could latch an earlier identical frame and replay stale footage). Source timing now relies on the settled recorder endpoint; post-resolution wait highlights advance the source cursor by a frame before resuming live footage.
Docs & process: README/AGENTS describe the new behavior and require attaching the
todo-app.spec.tsrendered video (and raw when recording changes) on every PR.Fixture: New
spec/todo-app.spec.tskitchen-sink demo (TodoDB, spinner UI, prompt login) as the shared visual baseline. A skipped spec documents a future-fill compositing bug for follow-up.Reviewed by Cursor Bugbot for commit ec7c933. Bugbot is set up for automated code reviews on this repo. Configure here.