Skip to content

[CI only] Stream history: ffmpeg ring buffer - #1

Open
jw098 wants to merge 96 commits into
mainfrom
stream-history-ffmpeg-ring
Open

jw098 wants to merge 96 commits into
mainfrom
stream-history-ffmpeg-ring

Conversation

@jw098

@jw098 jw098 commented Nov 12, 2026

Copy link
Copy Markdown
Owner

This PR is only for running CI on the fork. Don't merge it.

The workflows only run on pushes to main and on pull requests, so pushes to this branch don't start CI by themselves. While this PR is open, every push to stream-history-ffmpeg-ring runs the full CI. Close it once the real PR is open upstream.

It stays out of draft on purpose: the workflows skip draft PRs.


Generated by Claude Code

pa-devbot and others added 30 commits November 11, 2026 20:51
…mation#1895)

* Command-line test runner: filter tests by name substring

Add a "--test-substring <value>" argument to the command-line unit test
runner (--command-line-test-mode): it only runs the tests whose name contains
the given substring, using the same case-sensitive matching as the GUI's
"Run All Tests Containing This Substring" mode. Without the argument (or
with an empty value) all tests run, as before.

- Main.cpp: parse the argument (error + exit 1 if it has no value).
- CommandLineTests.h/.cpp: pass the substring through to the runner.
- CommandLineUnitTestRunner::run(): add only the matching tests; log how many
  will run, and warn in orange if nothing matches.

* Dedup the GUI and command-line unit test runners

The two runners (UnitTestRunner::program and CommandLineUnitTestRunner::run)
each had their own copy of the "add the tests whose name contains this
substring" loop (and the GUI's "Tests to Run" preview had a third copy),
and each of their on_test_finished listeners re-implemented the same result
classification into passed/failed/skipped/OOM buckets.

- Add a file-local add_tests_matching(filter, add_test) helper and use it
  in the GUI's run-substring-match / run-all modes, the GUI's "Tests to
  Run" preview, and the command-line runner.
- Add UnitTestResultCounters, shared by both runners: one place defines
  which bucket each result kind goes into (OOM is a failure and has its own
  counter) and returns the result's 1-based index within its bucket, which
  the GUI uses for its 1000-entry panel caps, exactly like the old
  fetch_add(1) + 1 sequences. The write-only m_total_failed goes away.

* Revert the shared result counters; keep the shared test selection

Per review: UnitTestResultCounters/m_results was redundant for the GUI,
whose Stats already track the same passed/failed/skipped/OOM buckets, and
its one unique use (the combined FAILED+OOM cap sequence) was just the old
m_total_failed. Revert both runners to their original atomics and
per-case counting, keeping only the shared add_tests_matching() helper
from the previous commit.

---------

Co-authored-by: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com>
* Exit menu simplification

* cleanup
* StorageView

* Read the box number at the bottom left
…HISTORY_FFMPEG).

Replaces the JPEG-frame ring with an in-memory ring of encoded H.264 video + AAC
audio packets when the new CMake option PA_STREAM_HISTORY_FFMPEG is enabled.
Nothing is written to storage until save() is called, at which point the ring
is muxed into an MP4 (or MKV for .mkv filenames) via libavformat.

- New StreamHistoryTracker_Ring (Recording/StreamHistoryTracker_Ring.{h,cpp}):
  - Video: dedicated encoder thread with the same bounded-queue / drop-oldest
    scheme as the JPEG tracker. Hardware H.264 encoders preferred
    (qsv/nvenc/amf on Windows, videotoolbox on macOS, v4l2m2m on Linux),
    libx264 veryfast as fallback, no B-frames, keyframe every 2 s, NV12 fast
    path with a Qt RGB888 fallback for other pixel formats.
  - Audio: resampled to 48 kHz (swresample) and encoded as AAC inline in
    on_samples(); deinterleaves the DUAL_44100 input, which the session
    declares as mono but the pipeline delivers as interleaved stereo.
  - The ring is evicted by time window (HISTORY_SECONDS) plus a ~90 MB cap.
  - save() snapshots the ring (shared packets, no deep copy), rewinds to the
    last keyframe, and muxes H.264+AAC. Encoder init / capture failures log
    and disable the affected stream instead of crashing.
- StreamHistorySession picks the ring tracker when PA_STREAM_HISTORY_FFMPEG is
  defined (same #if shape as the existing Qt version check).
- CMake: option(PA_STREAM_HISTORY_FFMPEG OFF) - OFF leaves every existing
  build untouched. When ON: Windows links the ffmpeg import libs from
  3rdPartyBinaries/ + headers from 3rdParty/ffmpeg/ (mirroring the OpenCV
  precedent; DLLs placed in 3rdPartyBinaries/ are auto-deployed by the
  existing THIRD_PARTY_DLLS glob), macOS/Linux use pkg-config
  (avcodec/avformat/avutil/swscale/swresample, requires ffmpeg >= 5.1).
  Missing ffmpeg fails at configure time with an explicit FATAL_ERROR.

Note: the design discussion called the audio codec Opus; AAC was chosen
instead because the save target is 'Video.mp4' (Discord attachments) and
H.264+AAC MP4 plays everywhere, while Opus-in-MP4 has spotty player support.
- save() rewound the video to the LAST keyframe, so saving a 30 s window
  returned only the ~2 s since the newest keyframe. Rewind to the FIRST
  keyframe in the ring instead (loses at most one keyframe interval from
  the head of the window).
- Video packet PTS were computed as us * time_base.den / time_base.num,
  which for the 1/1e6 timebase multiplies timestamps by 1e6, producing
  unplayable files. Use av_rescale_q from a 1/1000000 timebase.
- Muxed packets never set packet->stream_index, so real libavformat would
  route every packet (video and audio) into stream 0.
- swr_alloc_set_opts2() was called with 9 args including &swr as the first
  parameter; the ffmpeg >= 5.1 API takes 8 (out channel layout first, no
  SwrContext**). This did not compile against real ffmpeg headers.
- avcodec_send_frame() EAGAIN (encoder input queue full - realistic with
  hardware encoders) was treated as a hard failure without draining, which
  clogs the encoder permanently and logs every frame after. Drain one
  output packet and retry the send.
- Video encoder failure paths wrote video_enabled without the ring lock
  (raced with on_frame reading it under the lock).
- Link the ffmpeg libraries PRIVATE on POSIX (matches the PRIVATE compile
  definition and the Windows side).
- Include libavutil/error.h and libavutil/rational.h explicitly.
…encoder.

The project is LGPL-only, so the encoder candidate list no longer relies
on the GPL libx264. The final candidate is now 'mpeg4' (MPEG-4 Part 2,
built into libavcodec in LGPL builds), with the libx264-specific
preset/tune/bframes options removed along with the now-unneeded <cstring>
include. The CMake comment now points at the LGPL shared ffmpeg builds
(gyan.dev '...-lgpl-shared' / BtbN 'win64-lgpl-shared').
…g 5.1-8.x

- Save bug: the AAC encoder was opened without AV_CODEC_FLAG_GLOBAL_HEADER,
  so it emitted ADTS frames with no AudioSpecificConfig extradata and
  mux_to_file() failed to set up the audio stream - every save that
  included audio failed. Set the flag (both MP4 and MKV need it).
- MP4 output: pass movflags=+faststart to the mp4 muxer so the moov atom
  (file index) is written at the front of the file instead of at the end.
  MKV saves are unchanged.
- Portability: jw's build fixes targeted only the ffmpeg 8.0 API surface
  (void av_channel_layout_default + av_channel_layout_check,
  avcodec_get_supported_config, swr_alloc_set_opts2 taking a SwrContext**),
  which does not compile against the ffmpeg that mac/linux package managers
  provide. Guard each version-sensitive call on the library version macros
  so the tracker builds against ffmpeg 5.1 through 8.x (AVCodec.pix_fmts
  for pre-7.0, the old swr_alloc_set_opts2 signature for pre-8.0, the
  int-returning av_channel_layout_default for pre-8.0).
- Fix an AVFrame leak in on_samples() (early return without
  av_frame_free) and stop retrying a permanently broken audio path by
  disabling audio instead.
- Restore <cstring> (still needed for memcpy) and add
  <libavutil/version.h> for the guard macros.
… detection

* StreamHistoryTracker_Ring.cpp: ffmpeg's public API headers do not carry
  their own extern "C" guards, so from C++ every ffmpeg declaration was
  C++ name-mangled and none of the symbols resolved against the ffmpeg
  shared libraries (Windows: LNK2019 unresolved externals at the link
  stage after a full ~31 min build; mac/linux: undefined symbols).
  Wrap all ffmpeg includes in an extern "C" block.

* Restore the version-conditional include: ffmpeg 8.0 (libavutil 61+)
  renamed err.h to error.h, but Ubuntu 24.04's apt ships ffmpeg 6.1
  (libavutil 58) which only has err.h - the unconditional error.h include
  fails to compile there.

* Re-apply the C4458-safe parameter/local names in mux_to_file() and
  save() (video/audio/have_audio/video_start hide Impl members, which is
  a /W4 warning that becomes an error under /WX on Windows).

* Restore the #ifndef EAGAIN fallback for MSVC.

* CMakeLists (POSIX): pkg-config module names differ between platforms -
  Homebrew ships avcodec.pc etc. while Debian/Ubuntu ship libavcodec.pc
  etc. - so the previous 'pkg_check_modules(FFMPEG REQUIRED avcodec ...)'
  failed configure on Ubuntu, and the 'libavcodec>=59' version check
  failed configure on macOS (brew has no libavcodec.pc). Try both
  namings, check the minimum version with whichever was found, and add
  target_link_directories (like the OpenCV/Tesseract/DPP blocks above)
  so -lavcodec etc. are found under /opt/homebrew/lib on macOS.
…al API

The CI Ubuntu runner's ffmpeg (7.1-era) already has the 'new' API
signatures (void av_channel_layout_default, int
swr_alloc_set_opts2(SwrContext **ps, ...)) while its libavutil/libswresample
major numbers are still below the 8.0 ones the #if guards expected, so the
old-API branches were selected against new-API headers and failed to
compile.

- av_channel_layout_default / swr_alloc_set_opts2: select the API
  generation at compile time from the actual declarations
  (std::is_same on the function-pointer types, C language linkage). The
  generation-specific calls are routed through function templates so they
  are only instantiated when their branch is taken - names/expressions in a
  discarded if-constexpr branch are still looked up and checked, which is
  why the old 8-argument swr_alloc_set_opts2 call (missing the
  AVDictionary** that real ffmpeg 5.1-7.0 takes) also failed against the
  real headers.
- error.h vs err.h: pick with __has_include instead of a version macro.
- avcodec_get_supported_config stays a preprocessor gate
  (LIBAVCODEC_VERSION_MAJOR >= 61): the symbol is not declared at all in
  older headers, so the code must be removed before name lookup, not merely
  branch-discarded. The threshold is confirmed by this CI (its libavcodec
  compiles the new branch fine) and by ffmpeg 6.1 (libavcodec 60, which has
  AVCodec.pix_fmts instead).
- CMake: log which pkg-config module set / version was found.

Verified: syntax-checks against the committed ffmpeg 8.0 headers, and full
behavioral harness runs against both an ffmpeg 5.1-7.0 stub set and a
7.1-runner-model stub set (new signatures, old version numbers).
…ffmpeg 7.x

Per jw: we won't support the old ffmpeg API, so remove the
compile-time detection of the API generation in
StreamHistoryTracker_Ring.cpp and use the current signatures directly.

- error.h/err.h: just include <libavutil/error.h>. The __has_include
  fallback to err.h was dead either way - error.h exists in every
  ffmpeg release, and ffmpeg 8.x removed err.h by merging it into
  error.h.
- Removed kChannelLayoutDefaultReturnsVoid /
  kSwrAllocSetOpts2TakesContextPtr and the old-API template shims; call
  av_channel_layout_default() (void, validated with
  av_channel_layout_check()) and swr_alloc_set_opts2() taking a
  SwrContext** directly. These signatures exist in ffmpeg 7.1+ (the
  pre-7.1 releases pass an AVDictionary** to swr_alloc_set_opts2 and
  get an int back from av_channel_layout_default).
- CMake: enforce the floor for real. macOS/Linux: avcodec>=61
  (libavcodec 61 = ffmpeg 7.x; previously avcodec>=59 / 'ffmpeg 5.1').
  Windows: new check parsing LIBAVCODEC_VERSION_MAJOR out of the
  vendored 3rdParty/ffmpeg/ headers (previously there was no version
  check at all). The vendored 8.x headers pass.
The ring's unguarded ffmpeg API usage (channel layout API:
av_channel_layout_default/check/copy/uninit, AVCodecContext.ch_layout,
AVFrame.ch_layout; swr_alloc_set_opts2; everything else is pre-5.x API)
is available in ffmpeg 5.1 with unchanged signatures, so 5.1 is the
minimum supported version.

libavcodec's major version cannot express that floor as a number (4.x
and 5.x are both libavcodec 58), so the CMake floor check now detects
the 5.1 API generation in the headers (vendored tree on Windows,
pkg-config include dirs elsewhere) instead of comparing
LIBAVCODEC_VERSION_MAJOR against 61.

The pix_fmts/avcodec_get_supported_config split in
StreamHistoryTracker_Ring.cpp is untouched: it is an API-generation
gate for 7.0+ headers, and with 5.1-7.0 headers the AVCodec.pix_fmts
path (still present in those releases) compiles instead.
…ntime

Hardware encoders (h264_amf & co.) can open successfully and still fail
on every frame - e.g. h264_amf whose D3D11 device cannot create
textures (SubmitInput() failed with error 18 / AMF_DIRECTX_FAILED,
CreateTexture2D -> 0x887A0005 DXGI_ERROR_GPU_NOT_FOUND). Previously that
logged 'Video encode failed' on every frame forever and the saved file
had no video.

After 3 consecutive hard avcodec_send_frame() failures the current
candidate is abandoned and init_video_encoder() retries starting from
the next candidate (typically the CPU mpeg4 encoder), so the recording
still gets video and the error spam stops. Candidate rotation is
remembered (failed candidates are not retried), and the scaler is
rebuilt when the new encoder prefers a different pixel format.
On some AMD iGPUs (observed: Ryzen 5 5625U), a hardware H.264 encoder
can open successfully and then fail on every frame because its GPU
device cannot allocate textures (h264_amf: CreateTexture2D fails with
DXGI_ERROR_GPU_NOT_FOUND). Releasing such a context with
avcodec_free_context() crashes the AMD display driver with an access
violation in atidxx64.dll, taking the whole process down - observed
both at the encoder fallback point and in the tracker destructor.

Therefore, when a hardware encoder fails at runtime (or at open), its
context is now deliberately leaked (a bounded, one-time leak) instead
of freed, and the encoder is remembered as poisoned for the rest of
the process so that later sessions skip it instead of re-initializing
it. Fallback to the next candidate now happens on the first hard
avcodec_send_frame() failure instead of after three, to minimize the
number of calls into the broken driver. The CPU mpeg4 fallback is
still released normally.
…on open failures

Two fixes for the encoder fallback path:

- mpeg4 failed to open with 'Invalid argument' on ffmpeg 8.x: since
  7.0, AVCodec.pix_fmts is gone and the supported formats must come
  from avcodec_get_supported_config(); when that query yields no list,
  ctx->pix_fmt was left at AV_PIX_FMT_NONE, which mpeg4 (and likely
  other encoders) reject at open time. The chosen input pixel format
  now always falls back to YUV420P - supported by every encoder in
  the candidate list - when no list is available, and the selected
  format is logged when the encoder opens.

- Encoders that merely fail to open (e.g. h264_qsv/h264_nvenc on a
  machine without that hardware) are now released normally instead of
  being leaked and poisoned for the whole process: the driver was
  never exercised through a context that failed at open. Only a
  context that failed at runtime (after pushing frames into a possibly
  broken GPU driver) is leaked, as before.
Two encoder-setup fixes, both observed on an AMD Ryzen 5 5625U iGPU
(ffmpeg 8.1):

- mpeg4 failed to open with EINVAL: 'timebase 1/1000000 not supported
  by MPEG 4 standard, the maximum admitted value for the timebase
  denominator is 65535' (the 16-bit time_incr_length of MPEG-4 Part 2).
  The video timebase is now 1/1000 (milliseconds) for all candidates;
  the hardware H.264 wrappers accept any timebase, and the muxer
  derives packet timestamps from the ring's microsecond WallClock
  timestamps anyway.

- h264_amf failed at the first SubmitInput: the AMD driver logged
  'unexpected frame rate' and removed the D3D11 device
  (AMF_NO_DEVICE). The frame rate is now set explicitly
  (ctx->framerate = target_fps/1), which the AMF wrapper passes to the
  driver; it also feeds rate control on the other hardware encoders.
Saving the video crashed in avformat_write_header() (access violation
in avformat-63.dll) on an AMD iGPU with the h264_amf encoder. The
muxer ran on the caller's thread while the video worker and the audio
pipeline thread were still encoding into the same AVCodecContexts, and
ffmpeg is not thread-safe on a live context: the hardware encoders
populate their extradata lazily (sometimes only after the first
encoded packet, and possibly mid-stream), so the cross-thread
avcodec_parameters_from_context() in mux_to_file() could capture a torn
state (e.g. a nonzero extradata_size with a null extradata pointer),
which is what made the mp4 muxer walk off a null pointer.

The muxer now works exclusively from codec-parameter snapshots
(video_params/audio_params + video_time_base) that are captured on
each encoder's own thread - the video worker after every encoded
packet, the audio pipeline thread after every encoded AAC frame -
under the ring lock, and are refreshed whenever the encoder's
extradata size changes. A new encoder invalidates the stale video
snapshot. If no snapshot (or no extradata) exists yet, the save fails
with the existing 'missing encoder state' error instead of crashing.

Also logs the video extradata size and first four bytes at mux time
to make any future extradata-format issue (AVCC vs Annex B) visible
in the log.
The h264_amf wrapper emits its H.264 stream in Annex B form (SPS/PPS
extradata and packets with 00 00 00 01 start codes), even with
AV_CODEC_FLAG_GLOBAL_HEADER. The MP4 muxer in the vendored ffmpeg 8.x
(avformat-63) crashes with an access violation inside
avformat_write_header() when it is given H.264 in that form (older
ffmpeg, e.g. 6.x, tolerated it). Now that h264_amf is the first
encoder that actually works on AMD iGPUs, saving the video hit this
crash deterministically.

Since MP4 (ISO BMFF) canonically stores H.264 as AVCC - an avcC
configuration record in the stream header plus NAL units with
4-byte big-endian length prefixes - mux_to_file() now converts the
stream to that form when the video is H.264, the target is MP4, and
the extradata is Annex B: it builds the avcC record from the SPS/PPS
NAL units in the extradata snapshot and rewrites each ring packet
with NAL length prefixes at write time. Matroska output is left in
Annex B, and encoders that already produce AVCC (nvenc, qsv) are
untouched.

Also fixes a regression introduced by the parameter-snapshot commit:
the 'missing encoder state' check required extradata for every video
codec, but MPEG-4 Part 2 (the CPU fallback) has no extradata at all;
the requirement now applies to H.264 only (AAC audio still requires
its AudioSpecificConfig).
avformat_write_header() does not open format->url: the lavf docs
require the caller to provide an opened AVIOContext (format->pb),
and the muxer's write_header() dereferences s->pb unconditionally.
With pb still NULL the MP4 muxer faults in mov_write_header
(NULL+0x88 read, avformat-63.dll). Open the file with avio_open2()
just before writing the header and close it on the failure path
(the success path already closed it via avio_closep).
The avcC configuration record built by h264_avcc_from_annexb_extradata()
was malformed: it wrote a stray 5th byte (sps[4], the first byte of the
SPS RBSP) before the reserved/lengthSizeMinusOne byte, and used 0xE0
(numOfSequenceParameterSets = 0) instead of 0xE1 (= 1). Players that
take the SPS from the stream header therefore found none: the saved MP4
had the right duration and container structure, but decoded to a blank
screen, while MKV (which keeps the Annex B SPS/PPS in-band) played fine.

The record now matches the ISO/IEC 14496-15 avcC layout
01 <profile> <compat> <level> FF E1 <spsLen16> <SPS> 01 <ppsLen16> <PPS>,
byte-for-byte the same layout ffmpeg's own mov muxer writes from the
same SPS/PPS (verified by muxing and decoding the result).
Ring video packets are stamped with the wall-clock time of the frame they
were drained from, and a hardware encoder can emit several packets for one
frame (or flush two or more at once), so consecutive ring packets can share
a timestamp. The MP4 muxer rejects that:
  [mp4] Application provided invalid, non monotonically increasing dts to
  muxer in stream 0: 596 >= 596
and the whole save failed.

mux_to_file() now clamps a non-increasing video dts to last+1 (1 ms at the
1/1000 video timebase), and skips empty ring packets instead of feeding an
empty sample to the muxer (which would also fail the Annex B -> AVCC
conversion).

Verified with a standalone lavf harness: writing two consecutive H.264
packets with the same dts fails with AVERROR(EINVAL) exactly like the
reported error, and the clamped sequence muxes and decodes cleanly.
claude and others added 28 commits November 12, 2026 12:00
PA_STREAM_HISTORY_FFMPEG is on by default and configure stops without the
ffmpeg development packages, but the macOS and Ubuntu x64 guides did not
install them. Add them (with how to opt out on Ubuntu), and say in the
ARM64 guide that the packages it already installs are now needed for the
stream history too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Revert the per-version extraction folder, stamp file and configure
dependency on the zip: the DLLs are again extracted once into
3rdPartyBinaries/ffmpeg/ when missing, mirroring the OpenCV DLLs. The fix
for the blank ffmpeg version in the configure log is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- The AVCodec.pix_fmts / avcodec_get_supported_config() switch was gated on
  libavcodec >= 61, but the function was added in libavcodec 61.13
  (ffmpeg 7.1); ffmpeg 7.0 (61.3) failed to compile. Gate on 61.13.100.
- Drop a frame whose millisecond pts would not advance instead of clamping
  it to a repeat, which mpeg4 rejects as a hard error (and so would
  trigger an encoder fallback). Unreachable with the current throttling.
- Replace qt_pixel_format_to_av_pix_fmt() with an NV12 check: only its
  NV12 result was used, and its RGB mappings were wrong.
- Comments: describe the thread ownership and what m_lock guards, the
  audio clock's delivery-time assumption, encoder fallback on send or
  receive failures, the encoders that emit Annex B vs AVCC, the
  keyframe-headroom guarantee (give or take a frame, memory cap aside),
  and the dts bump; fix stale references (Impl, the resampler, the
  ffmpeg version range, the video time base, the bitrate clamp).
- Use the Mutex wrapper for the poisoned-encoder set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- CMake: the ffmpeg 5.1 check no longer claims the ring calls
  swr_alloc_set_opts2() (the audio pipeline resamples; libswresample is
  still required and linked), and the version note now says libavcodec
  cannot tell 5.0 from 5.1 (both 59), not 4.x from 5.x.
- Session: start() and m_audio_choice no longer describe nonexistent
  parameters or a shape the ffmpeg ring ignores, and
  attach_working_listener() states the fixed 48 kHz stereo attach.
- Ring tracker: document on_frame(), save() and set_window(); drop the
  single-channel pass-through case from push_samples() (the attach is
  always stereo); list the members each thread owns by name rather than
  by prefix (m_last_* and m_audio_* also matched lock-guarded members).
- H.264 helpers, tests: tidy the start-code scan comments; describe what
  the sync tolerance covers (AAC pre-echo), the Feeder's non-default
  delivery, and both bounds of the eviction test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
sws_getContext() fails the same way for the same input/output shape, so
retrying on every frame only logged an error at the frame rate. Disable
video like the other unrecoverable setup failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
The "could not convert video frame" flag was never reset, so after a
recovery a later failure went unlogged. Reset it once a frame converts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
- Video: an encoder that refuses a frame (EAGAIN) with no packet to drain
  is stuck, so fall back to the next candidate instead of dropping every
  frame silently. A packet allocation failure no longer looks like
  "nothing to drain".
- Video: log the per-frame skips (map, frame allocation, scaling) once per
  failing stretch; they used to be silent. They share one flag with the
  RGB conversion failure, cleared when a frame reaches the encoder.
- Audio: stop encoding once a receive error disables audio (the loop kept
  feeding the failed encoder), and disable audio with a clear message when
  a packet cannot be allocated instead of failing later with EAGAIN.
- Log a failed codec parameter snapshot and drop it so the next packet
  retries, instead of keeping a half-reset snapshot that breaks saves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
Every return path freed the AVFrame/AVPacket by hand, and an exception
(e.g. bad_alloc from the logger or the ring packet) leaked them. Use
FramePtr/PacketPtr, like the existing CodecParametersPtr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
encode_video_frame() unmapped the frame by hand before each early return,
and an exception (e.g. from the RGB conversion or the logger) left it
mapped. QVideoFrame is explicitly shared with reference-counted maps, so
that kept the frame mapped for every holder. ReadOnlyFrameMap maps in its
constructor, unmaps in its destructor, and keeps unmap() for the two
deliberate early unmaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
Add StreamHistory_FfmpegPtr.h: FfmpegPtr, a unique_ptr alias template
whose deleter calls an ffmpeg free function (taking the pointer or its
address, picked at compile time), the Frame/Packet/CodecParameters/
CodecContext/SwsContext aliases, an output format context owner (closes
the file, then frees), and an option dictionary owner.

mux_to_file() repeated its cleanup (free the options, close the file,
free the context) on nine return paths and leaked all three when an
exception escaped. Use the owners instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
m_video_context, m_audio_context, m_sws and the two codec parameter
snapshots were raw pointers freed in ~Data() and by hand wherever they
were replaced. ~Data() never runs when the constructor throws, so an
exception after init_audio_encoder() leaked the AAC context. Make them
FfmpegPtr members; ~Data() goes away, and the deliberate leak of a
failed hardware encoder in abandon_video_encoder() becomes an explicit
release(). The per-candidate context in init_video_encoder() is owned
too, so a throw from the open-failure log no longer leaks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
…oder lag

The video encoder holds the newest frames for a while (h264_amf held up to
~1.4 s when many trackers shared the GPU), so the newest video packet in
the ring trails the newest audio packet. evict() measured the window back
from the newest packet of either stream, so video was evicted early by
that lag, eating into the keyframe headroom; with enough lag the ring held
no keyframe and save() wrote a file starting on a non-keyframe. Measure
the video from its own newest packet (capped at MAX_VIDEO_LAG_US behind
the audio, so stopped video is still evicted), and log when save() finds
no keyframe instead of silently starting at the first packet.

Tests: the save tests assumed the encoder holds at most 0.5 s. They now
feed a 3 s tail so the frames before the checked point leave the encoder,
and the lower bound gets a small slack (4.9 - 0.1 - 0.5 rounded above the
measured 4.3). The eviction test no longer needs a backlog allowance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
encode_video_frame() forces a keyframe every 2 s with pict_type = I, but
h264_amf, h264_nvenc and h264_qsv turn that into a plain I-frame unless
their forced_idr option is set. Those frames are not marked as keyframes
and a decoder cannot start from them, so once the encoder's first frame
was evicted the ring held no start point and save() wrote a file whose
video does not decode (the Eviction test failed on h264_amf). Set
forced_idr; encoders without the option ignore it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
… throttling

New checks and tests:
- Every saved video must have a keyframe at least every keyframe interval
  plus a frame (guards the forced_idr fix: without forced keyframes, seven
  tests fail).
- Eviction and the new window-shrink test check that the audio starts
  within one AAC frame of the video.
- "Shrink the window while recording": set_window() evicts at once.
- "Video stops while the audio continues": the old video is evicted by
  the audio's clock once it trails by more than MAX_VIDEO_LAG_US.
- "Frames faster than the target FPS": 50 fps input is throttled to the
  target frame interval.

The audio-start check found that evict() dropped audio from before the
video's eviction point when the video trailed the audio, so a saved file
could start with up to the encoder's lag of video without sound. Keep the
audio back to the video's eviction point.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
The frames the video encoder still holds are not in a saved file. Log the
gap between the newest video and audio packets in save() to measure how
much video a save loses in real sessions, before deciding whether the
encoders' low-latency modes are worth enabling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Up to one frame interval of lag between the newest video and audio is
normal (the next frame is not due yet); only lag beyond that means the
encoder is holding frames. Log the interval so the number reads on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Muxing:
- Give every muxed packet a duration. Without it the MP4's last video
  sample was stored 0 long: the stream ended a frame early and some
  readers dropped the newest frame.

H.264 helpers (StreamHistory_H264):
- One Annex B NAL scanner for both helpers. Two start codes in a row
  (an empty NAL unit) no longer swallow the rest of the packet as one
  NAL, and 00 00 01 followed by 00 00 00 01 no longer fails the whole
  conversion (and with it the save).
- Write the avcC High profile extension (chroma format, bit depths, SPS
  extension count) that ISO/IEC 14496-15 requires for every profile but
  Baseline, Main and Extended, parsed from the SPS after removing the
  emulation prevention bytes. Matches ffmpeg's own avcC byte for byte.

Ring tracker:
- StreamHistorySession calls the ring's push_samples() with an owning
  reference instead of under the session spinlock, so the video thread
  and the UI no longer spin while AAC is encoded.
- Accumulate the audio per channel (planar, like the working stream and
  the encoder) instead of interleaving it and de-interleaving it again.
- Open the video encoder before mapping the frame.
- Share the packet-store / parameter-snapshot code between audio and
  video, and the hex formatting of the save-time log.

Tests:
- H.264: empty NAL units, a 3-byte start code before a 4-byte one,
  trailing zero padding; avcC for Main (no extension), High, High 4:2:2
  10-bit, High 4:4:4, an SPS with an emulation prevention byte, a
  libx264 SPS/PPS against ffmpeg's avcC, and an SPS cut short.
- Ring: the same ring saved as MP4 and MKV holds the same packets and
  the MP4 video stream lasts through its last frame; audio delivered in
  channel buffers longer than the samples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Session (StreamHistorySession):
- initialize() takes the generation its caller bumped, instead of
  reading it after the lock was released, so a clear() in between is no
  longer undone by installing a tracker built from the old settings.
- push_samples() (ffmpeg) pushes under a dedicated mutex and releases
  its tracker reference before that mutex; clear() and initialize()
  take the mutex before destroying a replaced tracker. The audio thread
  is never left with the last reference, whose destructor waits for
  the video encoder thread and would stall the audio pipeline.

Ring tracker:
- Test AVFMT_NOFILE on the output format's flags, not on the context's
  flags (where 0x0001 is AVFMT_FLAG_GENPTS).
- Convert audio sample counts to time in whole seconds plus remainder;
  scaling the whole count to nanoseconds overflowed int64 about 53
  hours after the last re-anchor.
- Drop a frame whose pts would not advance before mapping, converting
  and scaling it. The keyframe state still only advances for a frame
  that is sent.
- Reuse one AVFrame and AVPacket per encoder instead of allocating them
  for every frame and packet. The video frame's pict_type is now set on
  every frame (a reused frame would otherwise stay an I-frame).
- Decide MP4 vs Matroska once.
- Remove the EAGAIN fallback #define here and in the tests: MSVC's
  errno.h defines it.

Tests:
- Every saved file must not be mostly keyframes. This catches a reused
  frame keeping its forced I (verified: all 50 of 50 packets became
  keyframes, failing 20 ring tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
StreamHistorySession is main's version again, plus what the ffmpeg ring
needs (picking the tracker, passing the planar audio through, its
constructor, the fixed 48 kHz stereo attach shape) and one change:
clear() and initialize() still run under the session lock, as on main,
but hand back the tracker they replaced, and the caller destroys it
after releasing the lock (destroying the ring can wait on its encoder
thread). They move into Data, the only place that can name the tracker
type.

This drops the generation counter, building the tracker outside the
lock, pushing audio outside the lock, and the push mutex. Building the
tracker is cheap (the video encoder opens lazily on its own thread),
an AAC encode under the lock is well under a millisecond, and these
layers had already needed two race fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
…enable encoders; one FPS helper

- The video scaler was only rebuilt when the input or the encoder's
  pixel format changed, not when the encoder's size did. Resolution can
  be changed while recording, and an encoder reopened after a runtime
  failure takes the new size: a smaller one made sws_scale write past
  the end of the frame, a larger one dropped every frame from then on.
  sws_getCachedContext() now takes the output size on every frame,
  replacing the hand-kept m_sws_* fields.
- Encoders that fail to open are remembered for the process, by name
  and output size, so trackers rebuilt on input, resolution, startup or
  shutdown changes no longer initialize those devices and log the same
  failure again.
- StreamHistoryOption::target_fps() replaces the three copies of the
  VideoFPS switch (ring tracker, SaveFrames tracker, tests).

Tests: a test hook, reopen_video_encoder(), closes and reopens the
video encoder as after a runtime failure (which no encoder available to
a test can be made to do). "Encoder reopened at a new size" (RGB and
NV12) switch the frames from 16:9 to 4:3, then reopen the encoder: both
failed before the scaler fix (no video in the saved file).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
The stream history settings cannot change during a tracker's life:
opening the Settings panel closes the program panel, and with it the
console session that owns the stream history (and the Settings button
is disabled while a program runs). So a reopened encoder always gets the
same size, and the "Encoder reopened at a new size" tests covered a case
the app cannot produce, at the cost of a test-only method in the
tracker's header. sws_getCachedContext() stays: it is less code than
tracking the scaler's shape by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
h264_nvenc spells the option "forced-idr" while h264_qsv and h264_amf
spell it "forced_idr", so the forced keyframes on NVENC were plain
I-frames without AV_PKT_FLAG_KEY. Set both spellings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
StreamHistorySession is main's version again, with only what the ffmpeg
ring needs, each behind PA_STREAM_HISTORY_FFMPEG: picking the tracker,
its constructor, passing the planar audio through, and the fixed 48 kHz
stereo attach shape.

Destroying the replaced tracker after releasing the session lock is
dropped: main's tracker is destroyed under the lock too, and the swaps
it covered (input, resolution, startup and shutdown changes) are rare.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from 8798dce to f9ecab9 Compare November 12, 2026 19:01
The session calls the tracker (inline AAC encode, tracker construction and
teardown) while holding its SpinLock. Note that this is known and deferred,
and how to fix it if it ever matters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants