Skip to content

fix(browser): deliver real screencast frames — #491 P0 实时画面在真实 Chromium 上从未工作过 - #522

Open
Qiyuanqiii wants to merge 2 commits into
mainfrom
Qiyuanqiii/browser-screencast-delivery-fix
Open

Qiyuanqiii wants to merge 2 commits into
mainfrom
Qiyuanqiii/browser-screencast-delivery-fix

Conversation

@Qiyuanqiii

Copy link
Copy Markdown
Member

动机 / Motivation

#491 的 P0(只读 screencast 实时画面)在 main 上已有完整实现(泵、白名单、httpapi、前端 canvas),但从未做过真浏览器端到端验证——四个泵单测全部使用带 string sessionId 的脚本事件。本次用 opt-in 真浏览器(TRAVERSE_TEST_AGENT_BROWSER=1,本机已安装 Edge)实测时发现:实时画面在真实 Chromium 上一帧都从未送达过。

根因(探针实证)

wire 层 Page.screencastFrame 事件正常到达(envelope session 归属正确),但 json.Unmarshal 报:

cannot unmarshal number into Go struct field .sessionId of type string

新版 Chromium 的 params.sessionId 类型从 string 变成了 number,typed 解码使整个事件 unmarshal 失败,deliverScreencastLocked 静默丢帧。泵、预算、ack 逻辑全部正常,帧根本到不了它们。

修复 / Changes

  • SessionID 改为 json.RawMessage 原样接收:会话归属继续由 wire 信封保证(onEvent 已按 event.SessionID == r.session 过滤),params token 只用于 ack 回传。
  • ack 原样回显 token:string 与 number 两种 screencast 会话 id 都按原始 JSON 类型往返(评审做了逐字节等价实证);字面 null 视为缺席不 ack。
  • 缺席 sessionId 的帧从"整帧丢弃"改为"交付但不 ack"。
  • 新增 opt-in 真浏览器 E2E TestAgentBrowserRealScreencastLiveFrames:动画页 → 解码 JPEG 帧(魔数 + viewport 边界校验)→ Stop 后 channel 正常关闭。
  • 旧版 Chromium(string sessionId)行为完全不变。

验证 / Verification

  • gofmt -l 无输出;go vet 无告警;go test -race -count=1 ./internal/browserruntime/ 全通过。
  • 真浏览器 E2E:修复前复现 0 帧;修复后 5 帧 / 1.2s、1250×805、JPEG 正确(本机 Edge,TRAVERSE_TEST_AGENT_BROWSER=1)。
  • 评审记录(1 条 P2 + 2 条 P3,全部修复并复测;生产代码 No blocking findings)与覆盖明细见 docs/reviews/browser-screencast-delivery-fix/。

已知限制(如实记录):实测帧率由 Chromium 损伤/采样与 ack 流控共同决定,动画页约数 fps;#491 的 "≥5fps" 验收需在真实桌面会话进一步观测。

关联 #491(P0 可见性的交付修复)。同系列:#520(P1 下载隔离地基)。

… longer a string

Real-browser verification (TRAVERSE_TEST_AGENT_BROWSER=1) exposed that
the P0 live view never delivered a single frame against real Chromium:
screencastFrame.params.sessionId has changed type across Chromium
versions (now a number, previously a string, sometimes absent), and
typing it as a string failed the whole event unmarshal, silently
dropping every frame before the pump. The four pump tests passed only
because their scripted events carried string ids.

- Receive SessionID as a raw JSON token; session attribution keeps coming
  from the wire envelope, which onEvent already binds to the owning
  target session.
- Ack echoes the raw token verbatim so string and numeric screencast
  session ids both round-trip with their original JSON type.
- Add an opt-in real-browser E2E: an animated page yields decoded JPEG
  frames with sane viewport bounds, and Stop tears the tap down.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants