Skip to content

Video frames: convert without QVideoFrame::toImage() once the colors … - #1906

Closed
MajorLOL wants to merge 1 commit into
PokemonAutomation:mainfrom
MajorLOL:video-frame-conversion
Closed

MajorLOL wants to merge 1 commit into
PokemonAutomation:mainfrom
MajorLOL:video-frame-conversion

Conversation

@MajorLOL

@MajorLOL MajorLOL commented Nov 12, 2026 •

Copy link
Copy Markdown

…match

The process memory grows with each QVideoFrame::toImage() on some computers and is only given back by restarting PA: 10-13 KB a conversion, 0.3-1.6 GB an hour while a program with screen watchers runs (Windows, Intel UHD 630, a 1280 x 720 NV12 capture card, Qt 6.10.2). With PA open and no program running the memory stays flat for hours. Isolation tests of 10 minutes (about 8,000 conversions each): toImage() +77 MB, on the computation_realtime pool +101 MB, map() + toImage() +104 MB; the CPU conversion below -7 MB, SnapshotManager using it -27 MB.

VideoFrameConverter (used by SnapshotManager::convert()) converts NV12, NV21, YUV420P, YV12, YUYV and UYVY on the CPU. On the first 8 frames with colors (at most one a second) it compares 8 ways (BT.601/BT.709, limited/full range, nearest/bilinear chroma) with toImage() and uses the best one from then on if its mean color difference is at most 1.5 of 255; otherwise, and for other formats, toImage() stays. The decision is logged. PA_QT_FRAME_CONVERSION=1 keeps toImage().

On the computer above toImage() matched BT.709 limited range with bilinear chroma (its GPU path): mean difference 0.17, largest 2 of 255. A 720p frame takes about 9 ms, as toImage() did.

…match

The process memory grows with each QVideoFrame::toImage() on some
computers and is only given back by restarting PA: 10-13 KB a conversion,
0.3-1.6 GB an hour while a program with screen watchers runs (Windows,
Intel UHD 630, a 1280 x 720 NV12 capture card, Qt 6.10.2). With PA open
and no program running the memory stays flat for hours. Isolation tests
of 10 minutes (about 8,000 conversions each): toImage() +77 MB, on the
computation_realtime pool +101 MB, map() + toImage() +104 MB; the CPU
conversion below -7 MB, SnapshotManager using it -27 MB.

VideoFrameConverter (used by SnapshotManager::convert()) converts NV12,
NV21, YUV420P, YV12, YUYV and UYVY on the CPU. On the first 8 frames with
colors (at most one a second) it compares 8 ways (BT.601/BT.709,
limited/full range, nearest/bilinear chroma) with toImage() and uses the
best one from then on if its mean color difference is at most 1.5 of
255; otherwise, and for other formats, toImage() stays. The decision is
logged. PA_QT_FRAME_CONVERSION=1 keeps toImage().

On the computer above toImage() matched BT.709 limited range with
bilinear chroma (its GPU path): mean difference 0.17, largest 2 of 255.
A 720p frame takes about 9 ms, as toImage() did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AFpKyhKsEg9bJNoPWQzjJJ
@MajorLOL MajorLOL closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant