Skip to content

Stream history: in-memory H.264 + AAC ring (PA_STREAM_HISTORY_FFMPEG) - #1907

Draft
jw098 wants to merge 3 commits into
PokemonAutomation:mainfrom
jw098:stream-history2
Draft

jw098 wants to merge 3 commits into
PokemonAutomation:mainfrom
jw098:stream-history2

Conversation

@jw098

@jw098 jw098 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Stream history. squashed

Keeps the stream history as an in-memory ring of encoded H.264 (hardware
encoders, with ffmpeg's LGPL mpeg4 as a fallback) and AAC packets, and
muxes it to MP4/MKV only when saving. Enabled by the CMake option
PA_STREAM_HISTORY_FFMPEG (on by default); requires ffmpeg >= 5.1.

Squashed from 91 commits (branch stream-history-ffmpeg-ring at f9ecab95e).
Original messages, oldest first:

---- 683512180  Gin890  2026-10-12

Add ffmpeg-based in-memory stream history ring (opt-in via PA_STREAM_HISTORY_FFMPEG).

Replaces the JPEG-frame ring with an in-memory ring of encoded H.264 video + AAC
audio packets when the new CMake option PA_STREAM_HISTORY_FFMPEG is enabled.
Nothing is written to storage until save() is called, at which point the ring
is muxed into an MP4 (or MKV for .mkv filenames) via libavformat.

- New StreamHistoryTracker_Ring (Recording/StreamHistoryTracker_Ring.{h,cpp}):
  - Video: dedicated encoder thread with the same bounded-queue / drop-oldest
    scheme as the JPEG tracker. Hardware H.264 encoders preferred
    (qsv/nvenc/amf on Windows, videotoolbox on macOS, v4l2m2m on Linux),
    libx264 veryfast as fallback, no B-frames, keyframe every 2 s, NV12 fast
    path with a Qt RGB888 fallback for other pixel formats.
  - Audio: resampled to 48 kHz (swresample) and encoded as AAC inline in
    on_samples(); deinterleaves the DUAL_44100 input, which the session
    declares as mono but the pipeline delivers as interleaved stereo.
  - The ring is evicted by time window (HISTORY_SECONDS) plus a ~90 MB cap.
  - save() snapshots the ring (shared packets, no deep copy), rewinds to the
    last keyframe, and muxes H.264+AAC. Encoder init / capture failures log
    and disable the affected stream instead of crashing.
- StreamHistorySession picks the ring tracker when PA_STREAM_HISTORY_FFMPEG is
  defined (same #if shape as the existing Qt version check).
- CMake: option(PA_STREAM_HISTORY_FFMPEG OFF) - OFF leaves every existing
  build untouched. When ON: Windows links the ffmpeg import libs from
  3rdPartyBinaries/ + headers from 3rdParty/ffmpeg/ (mirroring the OpenCV
  precedent; DLLs placed in 3rdPartyBinaries/ are auto-deployed by the
  existing THIRD_PARTY_DLLS glob), macOS/Linux use pkg-config
  (avcodec/avformat/avutil/swscale/swresample, requires ffmpeg >= 5.1).
  Missing ffmpeg fails at configure time with an explicit FATAL_ERROR.

Note: the design discussion called the audio codec Opus; AAC was chosen
instead because the save target is 'Video.mp4' (Discord attachments) and
H.264+AAC MP4 plays everywhere, while Opus-in-MP4 has spotty player support.

---- a464e307d  Gin890  2026-10-12

Fix several bugs in the ffmpeg stream history ring found during review.

- save() rewound the video to the LAST keyframe, so saving a 30 s window
  returned only the ~2 s since the newest keyframe. Rewind to the FIRST
  keyframe in the ring instead (loses at most one keyframe interval from
  the head of the window).
- Video packet PTS were computed as us * time_base.den / time_base.num,
  which for the 1/1e6 timebase multiplies timestamps by 1e6, producing
  unplayable files. Use av_rescale_q from a 1/1000000 timebase.
- Muxed packets never set packet->stream_index, so real libavformat would
  route every packet (video and audio) into stream 0.
- swr_alloc_set_opts2() was called with 9 args including &swr as the first
  parameter; the ffmpeg >= 5.1 API takes 8 (out channel layout first, no
  SwrContext**). This did not compile against real ffmpeg headers.
- avcodec_send_frame() EAGAIN (encoder input queue full - realistic with
  hardware encoders) was treated as a hard failure without draining, which
  clogs the encoder permanently and logs every frame after. Drain one
  output packet and retry the send.
- Video encoder failure paths wrote video_enabled without the ring lock
  (raced with on_frame reading it under the lock).
- Link the ffmpeg libraries PRIVATE on POSIX (matches the PRIVATE compile
  definition and the Windows side).
- Include libavutil/error.h and libavutil/rational.h explicitly.

---- ca5fbcf57  Gin890  2026-10-14

Replace the libx264 video fallback with ffmpeg's built-in LGPL mpeg4 encoder.

The project is LGPL-only, so the encoder candidate list no longer relies
on the GPL libx264. The final candidate is now 'mpeg4' (MPEG-4 Part 2,
built into libavcodec in LGPL builds), with the libx264-specific
preset/tune/bframes options removed along with the now-unneeded <cstring>
include. The CMake comment now points at the LGPL shared ffmpeg builds
(gyan.dev '...-lgpl-shared' / BtbN 'win64-lgpl-shared').

---- 09daad5d5  jw098  2026-10-13

add ffmpeg library files

---- cccc92f44  jw098  2026-10-14

fix build

---- 22d47fc0c  Gin890  2026-10-14

Fix the stream history ring save path and make it build against ffmpeg 5.1-8.x

- Save bug: the AAC encoder was opened without AV_CODEC_FLAG_GLOBAL_HEADER,
  so it emitted ADTS frames with no AudioSpecificConfig extradata and
  mux_to_file() failed to set up the audio stream - every save that
  included audio failed. Set the flag (both MP4 and MKV need it).
- MP4 output: pass movflags=+faststart to the mp4 muxer so the moov atom
  (file index) is written at the front of the file instead of at the end.
  MKV saves are unchanged.
- Portability: jw's build fixes targeted only the ffmpeg 8.0 API surface
  (void av_channel_layout_default + av_channel_layout_check,
  avcodec_get_supported_config, swr_alloc_set_opts2 taking a SwrContext**),
  which does not compile against the ffmpeg that mac/linux package managers
  provide. Guard each version-sensitive call on the library version macros
  so the tracker builds against ffmpeg 5.1 through 8.x (AVCodec.pix_fmts
  for pre-7.0, the old swr_alloc_set_opts2 signature for pre-8.0, the
  int-returning av_channel_layout_default for pre-8.0).
- Fix an AVFrame leak in on_samples() (early return without
  av_frame_free) and stop retrying a permanently broken audio path by
  disabling audio instead.
- Restore <cstring> (still needed for memcpy) and add
  <libavutil/version.h> for the guard macros.

---- 792ecbbbd  jw098  2026-10-14

fix build

---- 78a362c68  jw098  2026-10-14

fix ffmpeg EAGAIN sign convention

---- fe6efc75f  jw098  2026-10-14

install ffmpeg libraries for mac/linux CI

---- a86c16fd3  jw098  2026-10-14

attempt to fix mac CI

---- c11acd0fd  jw098  2026-10-14

fix CI

---- 9818fb05c  Gin890  2026-10-14

Fix ffmpeg C++ linkage, C4458 under /W4 /WX, and mac/linux pkg-config detection

* StreamHistoryTracker_Ring.cpp: ffmpeg's public API headers do not carry
  their own extern "C" guards, so from C++ every ffmpeg declaration was
  C++ name-mangled and none of the symbols resolved against the ffmpeg
  shared libraries (Windows: LNK2019 unresolved externals at the link
  stage after a full ~31 min build; mac/linux: undefined symbols).
  Wrap all ffmpeg includes in an extern "C" block.

* Restore the version-conditional include: ffmpeg 8.0 (libavutil 61+)
  renamed err.h to error.h, but Ubuntu 24.04's apt ships ffmpeg 6.1
  (libavutil 58) which only has err.h - the unconditional error.h include
  fails to compile there.

* Re-apply the C4458-safe parameter/local names in mux_to_file() and
  save() (video/audio/have_audio/video_start hide Impl members, which is
  a /W4 warning that becomes an error under /WX on Windows).

* Restore the #ifndef EAGAIN fallback for MSVC.

* CMakeLists (POSIX): pkg-config module names differ between platforms -
  Homebrew ships avcodec.pc etc. while Debian/Ubuntu ship libavcodec.pc
  etc. - so the previous 'pkg_check_modules(FFMPEG REQUIRED avcodec ...)'
  failed configure on Ubuntu, and the 'libavcodec>=59' version check
  failed configure on macOS (brew has no libavcodec.pc). Try both
  namings, check the minimum version with whichever was found, and add
  target_link_directories (like the OpenCV/Tesseract/DPP blocks above)
  so -lavcodec etc. are found under /opt/homebrew/lib on macOS.

---- 671af3433  jw098  2026-10-14

fix build

---- 3a59cc26f  Gin890  2026-10-15

Stop keying ffmpeg API differences on version macros; detect the actual API

The CI Ubuntu runner's ffmpeg (7.1-era) already has the 'new' API
signatures (void av_channel_layout_default, int
swr_alloc_set_opts2(SwrContext **ps, ...)) while its libavutil/libswresample
major numbers are still below the 8.0 ones the #if guards expected, so the
old-API branches were selected against new-API headers and failed to
compile.

- av_channel_layout_default / swr_alloc_set_opts2: select the API
  generation at compile time from the actual declarations
  (std::is_same on the function-pointer types, C language linkage). The
  generation-specific calls are routed through function templates so they
  are only instantiated when their branch is taken - names/expressions in a
  discarded if-constexpr branch are still looked up and checked, which is
  why the old 8-argument swr_alloc_set_opts2 call (missing the
  AVDictionary** that real ffmpeg 5.1-7.0 takes) also failed against the
  real headers.
- error.h vs err.h: pick with __has_include instead of a version macro.
- avcodec_get_supported_config stays a preprocessor gate
  (LIBAVCODEC_VERSION_MAJOR >= 61): the symbol is not declared at all in
  older headers, so the code must be removed before name lookup, not merely
  branch-discarded. The threshold is confirmed by this CI (its libavcodec
  compiles the new branch fine) and by ffmpeg 6.1 (libavcodec 60, which has
  AVCodec.pix_fmts instead).
- CMake: log which pkg-config module set / version was found.

Verified: syntax-checks against the committed ffmpeg 8.0 headers, and full
behavioral harness runs against both an ffmpeg 5.1-7.0 stub set and a
7.1-runner-model stub set (new signatures, old version numbers).

---- 090fd71d8  Gin890  2026-10-15

Stream history: drop the dual ffmpeg API-generation support; enforce ffmpeg 7.x

Per jw: we won't support the old ffmpeg API, so remove the
compile-time detection of the API generation in
StreamHistoryTracker_Ring.cpp and use the current signatures directly.

- error.h/err.h: just include <libavutil/error.h>. The __has_include
  fallback to err.h was dead either way - error.h exists in every
  ffmpeg release, and ffmpeg 8.x removed err.h by merging it into
  error.h.
- Removed kChannelLayoutDefaultReturnsVoid /
  kSwrAllocSetOpts2TakesContextPtr and the old-API template shims; call
  av_channel_layout_default() (void, validated with
  av_channel_layout_check()) and swr_alloc_set_opts2() taking a
  SwrContext** directly. These signatures exist in ffmpeg 7.1+ (the
  pre-7.1 releases pass an AVDictionary** to swr_alloc_set_opts2 and
  get an int back from av_channel_layout_default).
- CMake: enforce the floor for real. macOS/Linux: avcodec>=61
  (libavcodec 61 = ffmpeg 7.x; previously avcodec>=59 / 'ffmpeg 5.1').
  Windows: new check parsing LIBAVCODEC_VERSION_MAJOR out of the
  vendored 3rdParty/ffmpeg/ headers (previously there was no version
  check at all). The vendored 8.x headers pass.

---- 464b000e6  Gin890  2026-10-15

Stream history: enforce the ffmpeg 5.1 floor instead of 7.x

The ring's unguarded ffmpeg API usage (channel layout API:
av_channel_layout_default/check/copy/uninit, AVCodecContext.ch_layout,
AVFrame.ch_layout; swr_alloc_set_opts2; everything else is pre-5.x API)
is available in ffmpeg 5.1 with unchanged signatures, so 5.1 is the
minimum supported version.

libavcodec's major version cannot express that floor as a number (4.x
and 5.x are both libavcodec 58), so the CMake floor check now detects
the 5.1 API generation in the headers (vendored tree on Windows,
pkg-config include dirs elsewhere) instead of comparing
LIBAVCODEC_VERSION_MAJOR against 61.

The pix_fmts/avcodec_get_supported_config split in
StreamHistoryTracker_Ring.cpp is untouched: it is an API-generation
gate for 7.0+ headers, and with 5.1-7.0 headers the AVCodec.pix_fmts
path (still present in those releases) compiles instead.

---- 5e06fd7f0  Gin890  2026-10-15

Stream history: fall back to the next video encoder if it fails at runtime

Hardware encoders (h264_amf & co.) can open successfully and still fail
on every frame - e.g. h264_amf whose D3D11 device cannot create
textures (SubmitInput() failed with error 18 / AMF_DIRECTX_FAILED,
CreateTexture2D -> 0x887A0005 DXGI_ERROR_GPU_NOT_FOUND). Previously that
logged 'Video encode failed' on every frame forever and the saved file
had no video.

After 3 consecutive hard avcodec_send_frame() failures the current
candidate is abandoned and init_video_encoder() retries starting from
the next candidate (typically the CPU mpeg4 encoder), so the recording
still gets video and the error spam stops. Candidate rotation is
remembered (failed candidates are not retried), and the scaler is
rebuilt when the new encoder prefers a different pixel format.

---- adfb6b026  Gin890  2026-10-15

Stream history: never free a failed hardware encoder context

On some AMD iGPUs (observed: Ryzen 5 5625U), a hardware H.264 encoder
can open successfully and then fail on every frame because its GPU
device cannot allocate textures (h264_amf: CreateTexture2D fails with
DXGI_ERROR_GPU_NOT_FOUND). Releasing such a context with
avcodec_free_context() crashes the AMD display driver with an access
violation in atidxx64.dll, taking the whole process down - observed
both at the encoder fallback point and in the tracker destructor.

Therefore, when a hardware encoder fails at runtime (or at open), its
context is now deliberately leaked (a bounded, one-time leak) instead
of freed, and the encoder is remembered as poisoned for the rest of
the process so that later sessions skip it instead of re-initializing
it. Fallback to the next candidate now happens on the first hard
avcodec_send_frame() failure instead of after three, to minimize the
number of calls into the broken driver. The CPU mpeg4 fallback is
still released normally.

---- 5776b17c0  jw098  2026-10-15

fix build

---- 18d6660b2  Gin890  2026-10-15

Stream history: fix pixel format selection on ffmpeg 7.0+, don't poison open failures

Two fixes for the encoder fallback path:

- mpeg4 failed to open with 'Invalid argument' on ffmpeg 8.x: since
  7.0, AVCodec.pix_fmts is gone and the supported formats must come
  from avcodec_get_supported_config(); when that query yields no list,
  ctx->pix_fmt was left at AV_PIX_FMT_NONE, which mpeg4 (and likely
  other encoders) reject at open time. The chosen input pixel format
  now always falls back to YUV420P - supported by every encoder in
  the candidate list - when no list is available, and the selected
  format is logged when the encoder opens.

- Encoders that merely fail to open (e.g. h264_qsv/h264_nvenc on a
  machine without that hardware) are now released normally instead of
  being leaked and poisoned for the whole process: the driver was
  never exercised through a context that failed at open. Only a
  context that failed at runtime (after pushing frames into a possibly
  broken GPU driver) is leaked, as before.

---- 24202f8a1  Gin890  2026-10-15

Stream history: use a mpeg4-compatible timebase and set the frame rate

Two encoder-setup fixes, both observed on an AMD Ryzen 5 5625U iGPU
(ffmpeg 8.1):

- mpeg4 failed to open with EINVAL: 'timebase 1/1000000 not supported
  by MPEG 4 standard, the maximum admitted value for the timebase
  denominator is 65535' (the 16-bit time_incr_length of MPEG-4 Part 2).
  The video timebase is now 1/1000 (milliseconds) for all candidates;
  the hardware H.264 wrappers accept any timebase, and the muxer
  derives packet timestamps from the ring's microsecond WallClock
  timestamps anyway.

- h264_amf failed at the first SubmitInput: the AMD driver logged
  'unexpected frame rate' and removed the D3D11 device
  (AMF_NO_DEVICE). The frame rate is now set explicitly
  (ctx->framerate = target_fps/1), which the AMF wrapper passes to the
  driver; it also feeds rate control on the other hardware encoders.

---- 284126b2e  Gin890  2026-10-16

Stream history: mux from codec-parameter snapshots, not live contexts

Saving the video crashed in avformat_write_header() (access violation
in avformat-63.dll) on an AMD iGPU with the h264_amf encoder. The
muxer ran on the caller's thread while the video worker and the audio
pipeline thread were still encoding into the same AVCodecContexts, and
ffmpeg is not thread-safe on a live context: the hardware encoders
populate their extradata lazily (sometimes only after the first
encoded packet, and possibly mid-stream), so the cross-thread
avcodec_parameters_from_context() in mux_to_file() could capture a torn
state (e.g. a nonzero extradata_size with a null extradata pointer),
which is what made the mp4 muxer walk off a null pointer.

The muxer now works exclusively from codec-parameter snapshots
(video_params/audio_params + video_time_base) that are captured on
each encoder's own thread - the video worker after every encoded
packet, the audio pipeline thread after every encoded AAC frame -
under the ring lock, and are refreshed whenever the encoder's
extradata size changes. A new encoder invalidates the stale video
snapshot. If no snapshot (or no extradata) exists yet, the save fails
with the existing 'missing encoder state' error instead of crashing.

Also logs the video extradata size and first four bytes at mux time
to make any future extradata-format issue (AVCC vs Annex B) visible
in the log.

---- 346fe0134  Gin890  2026-10-16

Stream history: convert H.264 from Annex B to AVCC when muxing MP4

The h264_amf wrapper emits its H.264 stream in Annex B form (SPS/PPS
extradata and packets with 00 00 00 01 start codes), even with
AV_CODEC_FLAG_GLOBAL_HEADER. The MP4 muxer in the vendored ffmpeg 8.x
(avformat-63) crashes with an access violation inside
avformat_write_header() when it is given H.264 in that form (older
ffmpeg, e.g. 6.x, tolerated it). Now that h264_amf is the first
encoder that actually works on AMD iGPUs, saving the video hit this
crash deterministically.

Since MP4 (ISO BMFF) canonically stores H.264 as AVCC - an avcC
configuration record in the stream header plus NAL units with
4-byte big-endian length prefixes - mux_to_file() now converts the
stream to that form when the video is H.264, the target is MP4, and
the extradata is Annex B: it builds the avcC record from the SPS/PPS
NAL units in the extradata snapshot and rewrites each ring packet
with NAL length prefixes at write time. Matroska output is left in
Annex B, and encoders that already produce AVCC (nvenc, qsv) are
untouched.

Also fixes a regression introduced by the parameter-snapshot commit:
the 'missing encoder state' check required extradata for every video
codec, but MPEG-4 Part 2 (the CPU fallback) has no extradata at all;
the requirement now applies to H.264 only (AAC audio still requires
its AudioSpecificConfig).

---- d790c5c3f  Gin890  2026-10-16

Stream history: open the output file before avformat_write_header

avformat_write_header() does not open format->url: the lavf docs
require the caller to provide an opened AVIOContext (format->pb),
and the muxer's write_header() dereferences s->pb unconditionally.
With pb still NULL the MP4 muxer faults in mov_write_header
(NULL+0x88 read, avformat-63.dll). Open the file with avio_open2()
just before writing the header and close it on the failure path
(the success path already closed it via avio_closep).

---- 855c59c8e  Gin890  2026-10-17

Stream history: write a valid avcC record when muxing MP4

The avcC configuration record built by h264_avcc_from_annexb_extradata()
was malformed: it wrote a stray 5th byte (sps[4], the first byte of the
SPS RBSP) before the reserved/lengthSizeMinusOne byte, and used 0xE0
(numOfSequenceParameterSets = 0) instead of 0xE1 (= 1). Players that
take the SPS from the stream header therefore found none: the saved MP4
had the right duration and container structure, but decoded to a blank
screen, while MKV (which keeps the Annex B SPS/PPS in-band) played fine.

The record now matches the ISO/IEC 14496-15 avcC layout
01 <profile> <compat> <level> FF E1 <spsLen16> <SPS> 01 <ppsLen16> <PPS>,
byte-for-byte the same layout ffmpeg's own mov muxer writes from the
same SPS/PPS (verified by muxing and decoding the result).

---- d3a95e686  Gin890  2026-10-17

Stream history: keep video dts strictly increasing when muxing to file

Ring video packets are stamped with the wall-clock time of the frame they
were drained from, and a hardware encoder can emit several packets for one
frame (or flush two or more at once), so consecutive ring packets can share
a timestamp. The MP4 muxer rejects that:
  [mp4] Application provided invalid, non monotonically increasing dts to
  muxer in stream 0: 596 >= 596
and the whole save failed.

mux_to_file() now clamps a non-increasing video dts to last+1 (1 ms at the
1/1000 video timebase), and skips empty ring packets instead of feeding an
empty sample to the muxer (which would also fail the Annex B -> AVCC
conversion).

Verified with a standalone lavf harness: writing two consecutive H.264
packets with the same dts fails with AVERROR(EINVAL) exactly like the
reported error, and the clamped sequence muxes and decodes cleanly.

---- 6b12b5b03  Gin890  2026-10-17

Stream history: set the AAC encoder's sample format (FLTP) and de-interleave into planes

init_audio_encoder() never set actx->sample_fmt, so avcodec_open2() failed
with EINVAL ("Invalid audio sample format: -1") and audio was disabled on
every platform - the log showed 'Failed to open AAC encoder. Disabling
audio.' even though everything else in the configuration was valid.

The native AAC encoder only accepts AV_SAMPLE_FMT_FLTP, so:
- set actx->sample_fmt = AV_SAMPLE_FMT_FLTP (and frame_size = 1024) before
  avcodec_open2(), and
- on_samples() now builds each 1024-sample encode frame as FLTP by
  de-interleaving the accumulated packed resampler output into the frame's
  per-channel planes instead of memcpy'ing it as one packed plane.

Also split the open-failure logging so the channel-layout step and
avcodec_open2() failures are reported separately, with the ffmpeg error
string for the latter.

Verified with a standalone harness that replicates the full pipeline
(resample 44.1/48 kHz -> packed FLT -> de-interleave -> FLTP frames ->
AAC encode with AV_CODEC_FLAG_GLOBAL_HEADER): the encoder opens, produces
packets, and exposes the 5-byte AudioSpecificConfig extradata that the
MP4/MKV muxers need.

---- d029a40c4  Gin890  2026-10-17

Stream history: initialize the audio resampler before use

make_audio_resampler() only called swr_alloc_set_opts2(), which allocates
the SwrContext and fills in the options but does NOT initialize it; the
context must be passed through swr_init() before swr_convert() will work.

On every audio callback swr_convert() therefore logged 'Context has not
been initialized' and returned an error, so no audio samples were ever
accumulated and saved videos had a video track but no audio track - even
after the AAC encoder itself was fixed to open (sample_fmt/FLTP).

Call swr_init() after allocation and free the context if it fails; the
existing 'Failed to create audio resampler' error path now reports real
initialization failures too.

---- 372b23aca  Gin890  2026-10-17

Stream history: stamp MKV packets in milliseconds and interleave the writes

The Matroska muxer writes packet timestamps into the file verbatim: its
write_header() rewrites every stream timebase to 1/1000 (the native
TimestampScale is milliseconds) but does not rescale the dts/pts the
application submits. The audio stream was submitted with a 1/48000
timebase and sample-count timestamps, so every audio block landed 48x too
late in the file timeline (a 30 s clip's audio blocks spanned 0..24 min)
and the segment Duration element became ~24 min for a 30 s clip.

Symptom: the MKV played its 30 s of video with no audible audio (each
audio block carried only 21 ms of sound, spaced 1.024 s apart on the
timeline), then sat on the last frame for the remaining ~19 minutes the
player believed the file still had. MP4 was unaffected because the mov
muxer rescales from the stream timebase itself.

Fix in mux_to_file():
- for .mkv, use a 1/1000 stream timebase for both video and audio and
  rescale the packet dts into it (video: us -> ms; audio: samples -> ms),
  while .mp4 keeps the encoder's video timebase and the 1/48000 audio
  timebase it already handled correctly;
- add a strictly-increasing dts guard to the audio packets, mirroring the
  video one;
- submit the video and audio packets interleaved in timestamp order
  instead of writing the whole video stream and then the whole audio
  stream, which laid the Matroska clusters out of time order (all audio
  data physically after all of the video).

Verified with a standalone harness that muxes real H.264 + AAC packets
through the same avformat calls: with sample-count audio dts the MKV
Duration element came out 1441813 ms (24 min) for a 30 s clip, with the
new millisecond stamping the duration is ~30 s and the audio packets
decode at 0..30 s; the MP4 path is unchanged.

---- f522f96a4  Gin890  2026-10-17

Stream history: remember every runtime-failed video encoder, not just the last

poisoned_encoder held a single name, so if two different hardware
encoders hard-failed in one process, the second overwrote the first and
a later session would re-initialize the first (leaked, possibly still
broken) encoder. A set keeps the poison for every encoder that failed.

---- 26be07beb  Gin890  2026-10-17

Stream history: drop the unused <libavutil/opt.h> include

Only av_dict_* is used here, which is declared in dict.h; both
avcodec.h and avformat.h include dict.h directly, so opt.h was
redundant. pixdesc.h stays (needed by the Windows build).

---- d555e91ce  PA Dev Bot  2026-10-17

Stream history: log the first video packet's bytes at mux time

Every known hardware encoder emits the extradata and the packets in the
same form (Annex B), and the MP4 conversion is driven by the extradata
form. If a driver ever reported one form while emitting the other, the
MP4 save would be a silent blank file. Logging the first 4 bytes of the
first ring packet next to the extradata makes any such mismatch
immediately visible (NVENC's SPS/PPS payload, in particular, is
driver-produced and unverified).

---- 415e0ef01  jw098  2026-10-17

adjust visibility of StreamHistory Options, depending on whether ffmpeg is enabled.

---- 32aa983ba  jw098  2026-10-17

minor change

---- 59d6c8be6  PA Dev Bot  2026-10-17

Stream history: add the Media Foundation encoder (h264_mf) to the Windows candidates

h264_mf is the OS-native Media Foundation H.264 encoder (typically the
built-in Microsoft encoder). It ships in the vendored LGPL build (verified
in avcodec-63.dll) and is present on essentially all Windows 8+. When none
of the GPU vendors' encoders (qsv/nvenc/amf) is available, it now yields
real, universally playable H.264 instead of dropping to MPEG-4 Part 2.

It joins the candidate list after h264_amf and before mpeg4. Like the
hardware wrappers it is device-backed, so a runtime failure follows the
same leak-and-poison path as abandon_video_encoder() (only the mpeg4 CPU
fallback is released normally); the existing rotation, poison, and
fallback logic all handle it without changes. The mux-time log lines
(extradata hex + first packet bytes) report its H.264 form on first use.

---- 7e5afb09d  jw098  2026-10-18

add info about the ffmpeg library

---- dce0deaf9  PA Dev Bot  2026-10-22

Stream history: fix two documentation inaccuracies (comments only)

1. The AVCC-conversion block comment claimed 'the other hardware
   encoders (nvenc, qsv) produce AVCC natively'. The ffmpeg 8.1 wrapper
   sources show the opposite: qsv and videotoolbox pass the driver's
   start-coded SPS/PPS and Annex B bitstream through without any
   conversion, and nvenc's packets are the raw start-coded bitstream.
   State that, and note that nvenc's driver-produced SPS/PPS payload is
   the one unverified case - which is what the mux-time log lines
   (extradata hex + first packet bytes) are for.

2. The audio ring packet's back-dated timestamp (now - AUDIO_FRAME_US)
   had no comment; explain that it aligns the packet with the start of
   the samples it contains, keeping audio/video aligned for eviction
   and the save() trim.

No code changes.

---- 7cdf45e90  PA Dev Bot  2026-10-23

Stream history: deduplicate the ffmpeg >= 5.1 API check into pa_check_ffmpeg_51_api()

The WIN32 and mac/Linux PA_STREAM_HISTORY_FFMPEG blocks each carried
their own copy of the ffmpeg 5.1 API-generation detection (grep
channel_layout.h for av_channel_layout_default and swresample.h for
swr_alloc_set_opts2) plus a near-identical FATAL_ERROR. The WIN32
version was just the mac/Linux per-include-dir scan with a one-element
list.

- New pa_check_ffmpeg_51_api(FFMPEG_INC_DIRS) function: scans the
  candidate include dirs for the 5.1 API surface and FATAL_ERRORs if
  it is missing. No return value - it either passes or dies.
- The WIN32 block now only does its discovery (vendored headers +
  import libraries) and records FFMPEG_INC_DIRS; the mac/Linux block
  keeps its pkg-config discovery + linking and records
  FFMPEG_INC_DIRS from the found module set.
- The shared block after the WIN32/else() split now applies
  target_include_directories and runs the check, so the requirement
  (and its rationale comment) lives in one place instead of three.

Discovery stays per-platform: Windows vendors import libraries in the
repo (no pkg-config), mac/Linux use pkg-config with dual module
naming. Symbol-grepping instead of version numbers is unchanged
(libavcodec 58 spans both ffmpeg 4.x and 5.x).

Net: 41 insertions, 62 deletions. Behavior-preserving; verified with
the Linux pkg-config configure path, a standalone success/failure
test of the function, and the full command-line suite.

---- 9cdef2423  jw098  2026-10-23

move ffmpeg binary into own folder. zip the dlls.

---- 68d96d53f  PA Dev Bot  2026-10-23

Stream history: keep ffmpeg binaries in their own 3rdPartyBinaries/ffmpeg/ folder

Move the Windows ffmpeg binaries out of the 3rdPartyBinaries/ top level
into 3rdPartyBinaries/ffmpeg/, following the discord_social_sdk_win
pattern:

- The import libraries (.lib) are checked in unzipped under
  3rdPartyBinaries/ffmpeg/ (jw's commit fb025a692 adds them), so the
  add_library(IMPORTED)/set_target_properties declarations move into
  the early imported-library cluster (next to ONNX/dpp/discord), with
  no configure-time existence checks - like discord_lib.
- The .dll files start zipped in
  3rdPartyBinaries/ffmpeg/ffmpeg-n9.0-win64-lgpl-shared-9.0.zip (the
  BtbN LGPL shared build, DLLs at the zip root, per the folder's
  README). The deploy section at the bottom of the file extracts them
  into 3rdPartyBinaries/ffmpeg/ if no .dll is present yet (like the
  OpenCV extraction) and adds POST_BUILD copy_if_different commands for
  them, since the top-level THIRD_PARTY_DLLS glob no longer sees them.

The old configure-time header/.lib existence checks are dropped (the
shared pa_check_ffmpeg_51_api() call after the platform split still
guards the header generation); a missing .lib now fails at link time,
matching the discord precedent. mac/linux pkg-config wiring is
unchanged. The old top-level ffmpeg .lib/.dll files remain until they
are deleted in a follow-up.

---- 489915dbe  jw098  2026-10-23

remove duplicate ffmpeg binaries

---- 25fefc117  jw098  2026-10-23

update comments

---- 91b71d984  PA Dev Bot  2026-10-24

Stream history: consume the audio pipeline's working stream instead of the legacy float path

Per the audio pipeline hand-off doc (PR #1672): do not pull in an ffmpeg
audio resampler - use the WorkingAudioStreamListener path and let ffmpeg
only encode/mux.

- StreamHistoryTracker_Ring now implements push_samples() and consumes
  the planar working stream directly: the pipeline tree's per-listener
  node delivers it already mixed + resampled to the ring's fixed
  48 kHz / stereo attach shape, so the ffmpeg swresample
  (make_audio_resampler/SwrContext), the DUAL_44100 left-channel hack,
  and the input-rate plumbing are gone. Audio is always recorded as
  2ch AAC @ 48 kHz (mono inputs upmix to dual-mono in the tree's
  mixer); the encoder now unconditionally uses the 96 kbps stereo
  bitrate. The constructor's audio args only gate whether audio is
  enabled at all.
- StreamHistorySession attaches the working listener at that fixed
  48 kHz / stereo shape in the ffmpeg build (the hand-off's suggested
  shape) and forwards the planar data to the tracker without
  re-interleaving; the non-ffmpeg builds keep the preset-shape attach +
  re-interleave for the SaveFrames tracker.
- The legacy AudioFloatStreamListener path is dropped from stream
  history: the session no longer implements on_samples, and
  ConsoleSystemSession no longer add/remove_stream_listener()s it, so
  the double-feed (legacy float + working stream) goes away.

CMake still links/checks swresample (unused by code now; the 5.1
API-generation check greps it) - a possible follow-up. Verified with
the full command-line suite: 18,272 passed / 0 failed / 3 skipped
(skips = no audio decode backend in this container).

---- 7fe9e40da  PA Dev Bot  2026-10-24

Stream history: decouple the ring tracker from the input's shape

The ring records audio at a fixed 48 kHz / stereo regardless of the
input, so the shape parameters in its constructor (the input preset's
framing) were only ever used as an audio-enabled gate. Make that
explicit: the constructor now takes 'bool has_audio'.

In the ffmpeg build the session no longer reads the resolved input
choice's rate / channels at all - it only uses its validity: whether a
device input is active gates whether audio is captured, and the
working-stream attach is unconditionally 2 x 48k. (The non-ffmpeg
build keeps the shape: the SaveFrames tracker records at the
delivered shape, so the session still re-interleaves at the choice's
framing there.)

---- a3578259e  PA Dev Bot  2026-10-25

Stream history: keep one keyframe interval of headroom in the ring

save() starts the file at the first keyframe in the ring, but keyframes
arrive only every 2 s and evict() dropped everything older than
"latest - window", so the first keyframe could fall up to a full
keyframe interval after the window's start. A 30 s window therefore
saved only 28-30 s of video (27-28 s in practice, since the file also
ends at the last drained packet, i.e. behind the encoder's internal
delay).

Keep one extra keyframe interval of headroom in the ring: the eviction
threshold is now "latest - window - keyframe_interval", so the ring
always contains a keyframe at or before the window's start and the
saved file always covers the most recent "window" seconds (it may
start up to one keyframe interval earlier, ~30-32 s for a 30 s
window). Memory cost is bounded by one extra interval of encoded
packets (~10% of the ring; well under the 90 MB cap at the 8 Mbps
bitrate clamp).

---- 01a384d84  jw098  2026-10-25

use Pimpl

---- 792cfaa0c  jw098  2026-10-25

m_-prefix the ring tracker's Data members

---- 342e0d9b2  PA Dev Bot  2026-10-25

Stream history: document the ring tracker's functions

Brief step-level comments for the functions that had none: the Data
ctor/dtor, init_audio_encoder, init_video_encoder, worker_loop,
encode_video_frame, on_frame, push_samples, save, the tracker
destructor (worker stop ordering), and one-liners for the settings
helpers and the Qt pixel-format mapping. mux_to_file, evict,
drain_one_video_packet, abandon_video_encoder and the H.264 AVCC
converters already had them.

---- d52329e83  jw098  2026-10-25

add comments

---- 871264bb9  jw098  2026-10-25

add comments

---- e41644eeb  Claude  2026-10-26

Stream history: fix a null-encoder crash, a shutdown hang, and saves after an encoder fallback

- init_video_encoder(): when every candidate has been used up (e.g. the
  last candidate opened and then failed at runtime), disable video
  instead of returning with m_vctx null, which the next frame
  dereferenced.
- ~StreamHistoryTracker_Ring(): set the stop flag under the queue lock
  so the worker cannot miss the wakeup and block forever. On stop the
  worker now drops queued frames instead of encoding them.
- When a new video encoder opens after a runtime fallback, drop the
  previous encoder's packets from the ring along with its parameter
  snapshot; they cannot be muxed with the new encoder's parameters.

---- f74d42566  Claude  2026-10-26

Stream history: fix A/V sync, a save/fallback race, and receive-side encoder failures

- save(): copy the codec parameters in the same critical section as the
  ring packets. An encoder fallback replaces both together, so reading the
  parameters later under a separate lock could mux one encoder's packets
  with another encoder's parameters (or fail with missing encoder state).
- Video packets are stamped from their own pts (ms since the first frame)
  instead of the most recently sent frame's timestamp. Hardware encoders
  pipeline several frames, which delayed the video against the audio.
- Audio packets are stamped from a sample clock anchored to the wall clock
  (re-anchored when the two diverge by more than 100 ms) and taking the AAC
  encoder's delay into account via the packet pts. The muxer now places
  audio by these timestamps instead of by sample count from zero, so a gap
  in the delivered audio no longer shifts all later audio against video.
- A hard failure from avcodec_receive_packet() now falls back to the next
  encoder candidate, like a hard failure from avcodec_send_frame() does.

---- 4e689178d  Claude  2026-10-26

Stream history: move the H.264 bitstream helpers into their own file

h264_annexb_to_avcc() and h264_avcc_from_annexb_extradata() only work on
byte buffers (no ffmpeg, Qt or tracker state), so they now live in
StreamHistory_H264.h/.cpp. That makes them testable and lets them build in
every configuration, not just with PA_STREAM_HISTORY_FFMPEG. No behavior
change: the function bodies are moved verbatim.

---- b140f973f  Claude  2026-10-26

Stream history: add unit tests for the H.264 bitstream helpers

Table-driven tests for h264_annexb_to_avcc() and
h264_avcc_from_annexb_extradata() that check the exact output bytes:
3- and 4-byte start codes, several NAL units, leading garbage, inputs with
no NAL unit, extra and repeated NAL types in the extradata, and missing or
truncated parameter sets.

Three cases fail on this commit and are fixed by the next one: a zero byte
of a following 4-byte start code is kept as part of the previous NAL unit,
and an extradata without a PPS produces an avcC that claims one.

---- 3d1190e61  Claude  2026-10-26

Stream history: fix two H.264 bitstream bugs found by the new tests

- A zero byte belonging to a following 4-byte start code (00 00 00 01)
  was kept as the last byte of the previous NAL unit, so the AVCC length
  prefixes and the SPS/PPS in the avcC record carried a stray trailing
  zero. A NAL unit never ends in a zero byte, so trailing zeros are now
  left with the start code.
- An extradata without a PPS produced an avcC record claiming one PPS of
  length 0. It now records zero picture parameter sets.

---- d91ea93e6  Claude  2026-10-26

Stream history: add test hooks to the ring tracker

- push_samples() overload taking the capture time of the last sample, so
  tests can drive audio and video on one simulated timeline. The existing
  overload passes current_time(), so production behavior is unchanged.
- wait_until_idle(): blocks until every queued video frame has been
  encoded. on_frame() only queues frames for the encoder thread (dropping
  the oldest beyond a small backlog), so tests wait on this between frames.

---- 346488ad5  Claude  2026-10-26

Stream history: add end-to-end tests for the ring tracker

In PA_STREAM_HISTORY_FFMPEG builds, StreamHistory_Tests.cpp now also feeds
StreamHistoryTracker_Ring synthetic frames (black, then white) and audio
(silence, then a 1 kHz tone) on a simulated timeline, saves to MP4/MKV, and
reads the file back with ffmpeg. Covered: stream layout, a keyframe first,
strictly increasing timestamps, file length, window eviction, A/V sync
(also after a one-second audio gap), saving with nothing recorded or to an
unwritable path, destruction with frames still queued, and saving from
another thread while recording. Files go under DEBUG_PATH() and are
removed when the test passes.

---- 879edbf44  Claude  2026-10-26

Stream history: build and destroy the tracker outside the session lock

clear() and initialize() ran under the session's spin lock, so every input,
resolution or startup/shutdown change constructed the new tracker (opening
the AAC encoder, starting the encoder thread) and destroyed the old one
(joining its encoder thread, which may be opening a hardware encoder) while
the audio and video threads spun on that lock in push_samples() and
on_frame().

The lock is now held only to read the settings and to swap the tracker
pointer; the old tracker is destroyed after the lock is released. A
generation counter, bumped on every settings change and on clear(), keeps
a tracker built from stale settings from replacing a newer one.

---- c62cc408e  Claude  2026-10-26

Stream history: recognize 3-byte start codes in H.264 Annex B extradata

The MP4 path converts Annex B H.264 to AVCC only when the extradata starts
with a 4-byte start code. Extradata starting with a 3-byte start code
(00 00 01) skipped the conversion and reached the MP4 muxer as Annex B.
The check now accepts both, and is limited to H.264: MPEG-4 Part 2
extradata also starts with a 00 00 01 start code.

---- 8ac841c3a  Claude  2026-10-26

Stream history: test NV12 frames, audio jitter and drift, resolution changes

- Save and sync from NV12 frames (the common capture card format, scaled
  directly from its planes; the other tests use RGB32).
- Sync with audio delivered 0-50 ms late: no re-sync may happen.
- Sync with the audio device clock running 0.1% fast or slow over 200
  simulated seconds: the re-sync keeps the audio within its threshold.
- Input resolution change while recording (RGB32 and NV12).
- The sync tests without a gap now also check that the audio packets are
  evenly spaced, which catches a re-sync that should not have happened.

---- 46679a38c  Claude  2026-10-27

Stream history: drop audio overlapping a re-sync, report the memory cap

- When the audio device clock runs fast, the re-sync moves the audio
  timeline about 100 ms back, so the following packets overlap audio
  already written. The muxer bumped them to strictly increasing
  timestamps, squeezing them into near-zero-length packets (an audible
  glitch). It now drops the overlapping packets and logs how many.
- The 90 MB memory cap shortened saves silently (about 175 s of history at
  the Very High quality setting). The first time it drops packets the
  window would have kept, the tracker now logs it once, outside the lock.
- Dropped frames (the encoder falling behind) are reported at most every
  10 seconds with a count, instead of one red log line per frame.
- The ffmpeg log level is process-wide; it is now set once, by the first
  tracker, instead of being reset by every tracker.

---- cb4660cc4  Claude  2026-10-27

Stream history: test that a backwards audio re-sync drops the overlap

Every sync test now also checks that no two audio packets are closer than
one AAC frame, and a new test (device clock 0.1% fast, 110 s, saved window
covering the re-sync at about 100 s) exercises the overlap case directly.

---- 1b4378b52  Claude  2026-10-27

Stream history: extract the Windows ffmpeg DLLs per version; fix the version log

- The Windows DLLs were extracted into 3rdPartyBinaries/ffmpeg/ only when
  no .dll was there yet. Since the DLLs are git-ignored, a machine that had
  extracted them once kept them after the repo moved to a new zip and new
  import libraries: stale DLLs with the same names fail to load ("entry
  point not found"), and ones with new names are never extracted. The zip
  is now extracted into a folder named after it, completed extractions are
  marked with a stamp file (an interrupted one is redone, a failed one is
  a configure error), and changing the zip re-runs the configure step.
- The configure log printed a blank ffmpeg version: with several modules,
  pkg_check_modules() sets <prefix>_<module>_VERSION, not
  <prefix>_VERSION. It now prints the libavcodec version.

---- aae71d3b4  Claude  2026-10-27

Build instructions: install ffmpeg for the stream history

PA_STREAM_HISTORY_FFMPEG is on by default and configure stops without the
ffmpeg development packages, but the macOS and Ubuntu x64 guides did not
install them. Add them (with how to opt out on Ubuntu), and say in the
ARM64 guide that the packages it already installs are now needed for the
stream history too.

---- 9268f49f5  Claude  2026-10-27

Stream history: extract the Windows ffmpeg DLLs once, like OpenCV

Revert the per-version extraction folder, stamp file and configure
dependency on the zip: the DLLs are again extracted once into
3rdPartyBinaries/ffmpeg/ when missing, mirroring the OpenCV DLLs. The fix
for the blank ffmpeg version in the configure log is kept.

---- 67a2a86b1  Claude  2026-10-27

Stream history: fix the ffmpeg 7.0 build and correct the ring's comments

- The AVCodec.pix_fmts / avcodec_get_supported_config() switch was gated on
  libavcodec >= 61, but the function was added in libavcodec 61.13
  (ffmpeg 7.1); ffmpeg 7.0 (61.3) failed to compile. Gate on 61.13.100.
- Drop a frame whose millisecond pts would not advance instead of clamping
  it to a repeat, which mpeg4 rejects as a hard error (and so would
  trigger an encoder fallback). Unreachable with the current throttling.
- Replace qt_pixel_format_to_av_pix_fmt() with an NV12 check: only its
  NV12 result was used, and its RGB mappings were wrong.
- Comments: describe the thread ownership and what m_lock guards, the
  audio clock's delivery-time assumption, encoder fallback on send or
  receive failures, the encoders that emit Annex B vs AVCC, the
  keyframe-headroom guarantee (give or take a frame, memory cap aside),
  and the dts bump; fix stale references (Impl, the resampler, the
  ffmpeg version range, the video time base, the bitrate clamp).
- Use the Mutex wrapper for the poisoned-encoder set.

---- 1c2db20c6  Claude  2026-10-28

Stream history: correct and clarify comments

- CMake: the ffmpeg 5.1 check no longer claims the ring calls
  swr_alloc_set_opts2() (the audio pipeline resamples; libswresample is
  still required and linked), and the version note now says libavcodec
  cannot tell 5.0 from 5.1 (both 59), not 4.x from 5.x.
- Session: start() and m_audio_choice no longer describe nonexistent
  parameters or a shape the ffmpeg ring ignores, and
  attach_working_listener() states the fixed 48 kHz stereo attach.
- Ring tracker: document on_frame(), save() and set_window(); drop the
  single-channel pass-through case from push_samples() (the attach is
  always stereo); list the members each thread owns by name rather than
  by prefix (m_last_* and m_audio_* also matched lock-guarded members).
- H.264 helpers, tests: tidy the start-code scan comments; describe what
  the sync tolerance covers (AAC pre-echo), the Feeder's non-default
  delivery, and both bounds of the eviction test.

---- 458dc2cf2  Claude  2026-11-11

Stream history: disable video when the scaler cannot be created

sws_getContext() fails the same way for the same input/output shape, so
retrying on every frame only logged an error at the frame rate. Disable
video like the other unrecoverable setup failures.

---- f85e93865  Claude  2026-11-11

Stream history: log each stretch of failed frame conversions

The "could not convert video frame" flag was never reset, so after a
recovery a later failure went unlogged. Reset it once a frame converts.

---- 92891ba60  jw098  2026-11-05

add comments. rename member variables for clarity

---- 166f06f1f  jw098  2026-11-11

add comments

---- b8975c6da  Claude  2026-11-11

Stream history: tighten the encoder failure handling

- Video: an encoder that refuses a frame (EAGAIN) with no packet to drain
  is stuck, so fall back to the next candidate instead of dropping every
  frame silently. A packet allocation failure no longer looks like
  "nothing to drain".
- Video: log the per-frame skips (map, frame allocation, scaling) once per
  failing stretch; they used to be silent. They share one flag with the
  RGB conversion failure, cleared when a frame reaches the encoder.
- Audio: stop encoding once a receive error disables audio (the loop kept
  feeding the failed encoder), and disable audio with a clear message when
  a packet cannot be allocated instead of failing later with EAGAIN.
- Log a failed codec parameter snapshot and drop it so the next packet
  retries, instead of keeping a half-reset snapshot that breaks saves.

---- 8e8f48feb  jw098  2026-11-11

add comments

---- 87b8567b2  Claude  2026-11-11

Stream history: own ffmpeg frames and packets with unique_ptr

Every return path freed the AVFrame/AVPacket by hand, and an exception
(e.g. bad_alloc from the logger or the ring packet) leaked them. Use
FramePtr/PacketPtr, like the existing CodecParametersPtr.

---- e96ea18e4  Claude  2026-11-11

Stream history: unmap the video frame with an RAII guard

encode_video_frame() unmapped the frame by hand before each early return,
and an exception (e.g. from the RGB conversion or the logger) left it
mapped. QVideoFrame is explicitly shared with reference-counted maps, so
that kept the frame mapped for every holder. ReadOnlyFrameMap maps in its
constructor, unmaps in its destructor, and keeps unmap() for the two
deliberate early unmaps.

---- 0a4a76c57  Claude  2026-11-11

Stream history: ffmpeg RAII header; RAII in mux_to_file()

Add StreamHistory_FfmpegPtr.h: FfmpegPtr, a unique_ptr alias template
whose deleter calls an ffmpeg free function (taking the pointer or its
address, picked at compile time), the Frame/Packet/CodecParameters/
CodecContext/SwsContext aliases, an output format context owner (closes
the file, then frees), and an option dictionary owner.

mux_to_file() repeated its cleanup (free the options, close the file,
free the context) on nine return paths and leaked all three when an
exception escaped. Use the owners instead.

---- 485a55695  Claude  2026-11-11

Stream history: own the encoder state with RAII members

m_video_context, m_audio_context, m_sws and the two codec parameter
snapshots were raw pointers freed in ~Data() and by hand wherever they
were replaced. ~Data() never runs when the constructor throws, so an
exception after init_audio_encoder() leaked the AAC context. Make them
FfmpegPtr members; ~Data() goes away, and the deliberate leak of a
failed hardware encoder in abandon_video_encoder() becomes an explicit
release(). The per-candidate context in init_video_encoder() is owned
too, so a throw from the open-failure log no longer leaks it.

---- 9c5f7abf2  jw098  2026-11-11

add comments. minor refactor to the loop draining packets from the encoder.

---- 04fa2f270  jw098  2026-11-11

update comments

---- 896f12f77  Claude  2026-11-12

Stream history: evict video by its own clock; fix the tests under encoder lag

The video encoder holds the newest frames for a while (h264_amf held up to
~1.4 s when many trackers shared the GPU), so the newest video packet in
the ring trails the newest audio packet. evict() measured the window back
from the newest packet of either stream, so video was evicted early by
that lag, eating into the keyframe headroom; with enough lag the ring held
no keyframe and save() wrote a file starting on a non-keyframe. Measure
the video from its own newest packet (capped at MAX_VIDEO_LAG_US behind
the audio, so stopped video is still evicted), and log when save() finds
no keyframe instead of silently starting at the first packet.

Tests: the save tests assumed the encoder holds at most 0.5 s. They now
feed a 3 s tail so the frames before the checked point leave the encoder,
and the lower bound gets a small slack (4.9 - 0.1 - 0.5 rounded above the
measured 4.3). The eviction test no longer needs a backlog allowance.

---- e76e3ded9  Claude  2026-11-12

Stream history: force IDR keyframes on the hardware encoders

encode_video_frame() forces a keyframe every 2 s with pict_type = I, but
h264_amf, h264_nvenc and h264_qsv turn that into a plain I-frame unless
their forced_idr option is set. Those frames are not marked as keyframes
and a decoder cannot start from them, so once the encoder's first frame
was evicted the ring held no start point and save() wrote a file whose
video does not decode (the Eviction test failed on h264_amf). Set
forced_idr; encoders without the option ignore it.

---- 51c5a6772  Claude  2026-11-12

Stream history: test keyframe spacing, window changes, stopped video, throttling

New checks and tests:
- Every saved video must have a keyframe at least every keyframe interval
  plus a frame (guards the forced_idr fix: without forced keyframes, seven
  tests fail).
- Eviction and the new window-shrink test check that the audio starts
  within one AAC frame of the video.
- "Shrink the window while recording": set_window() evicts at once.
- "Video stops while the audio continues": the old video is evicted by
  the audio's clock once it trails by more than MAX_VIDEO_LAG_US.
- "Frames faster than the target FPS": 50 fps input is throttled to the
  target frame interval.

The audio-start check found that evict() dropped audio from before the
video's eviction point when the video trailed the audio, so a saved file
could start with up to the encoder's lag of video without sound. Keep the
audio back to the video's eviction point.

---- 6297e4802  Claude  2026-11-12

Stream history: log how far the video trails the audio at save time

The frames the video encoder still holds are not in a saved file. Log the
gap between the newest video and audio packets in save() to measure how
much video a save loses in real sessions, before deciding whether the
encoders' low-latency modes are worth enabling.

---- 923448bb5  Claude  2026-11-12

Stream history: include the frame interval in the save-time lag log

Up to one frame interval of lag between the newest video and audio is
normal (the next frame is not due yet); only lag beyond that means the
encoder is holding frames. Log the interval so the number reads on its own.

---- 5183b47a7  Claude  2026-11-12

Stream history: fix the MP4 tail, H.264 parsing and avcC; tests for each

Muxing:
- Give every muxed packet a duration. Without it the MP4's last video
  sample was stored 0 long: the stream ended a frame early and some
  readers dropped the newest frame.

H.264 helpers (StreamHistory_H264):
- One Annex B NAL scanner for both helpers. Two start codes in a row
  (an empty NAL unit) no longer swallow the rest of the packet as one
  NAL, and 00 00 01 followed by 00 00 00 01 no longer fails the whole
  conversion (and with it the save).
- Write the avcC High profile extension (chroma format, bit depths, SPS
  extension count) that ISO/IEC 14496-15 requires for every profile but
  Baseline, Main and Extended, parsed from the SPS after removing the
  emulation prevention bytes. Matches ffmpeg's own avcC byte for byte.

Ring tracker:
- StreamHistorySession calls the ring's push_samples() with an owning
  reference instead of under the session spinlock, so the video thread
  and the UI no longer spin while AAC is encoded.
- Accumulate the audio per channel (planar, like the working stream and
  the encoder) instead of interleaving it and de-interleaving it again.
- Open the video encoder before mapping the frame.
- Share the packet-store / parameter-snapshot code between audio and
  video, and the hex formatting of the save-time log.

Tests:
- H.264: empty NAL units, a 3-byte start code before a 4-byte one,
  trailing zero padding; avcC for Main (no extension), High, High 4:2:2
  10-bit, High 4:4:4, an SPS with an emulation prevention byte, a
  libx264 SPS/PPS against ffmpeg's avcC, and an SPS cut short.
- Ring: the same ring saved as MP4 and MKV holds the same packets and
  the MP4 video stream lasts through its last frame; audio delivered in
  channel buffers longer than the samples.

---- 780d88a63  Claude  2026-11-12

Stream history: fix the session's tracker races; second review fixes

Session (StreamHistorySession):
- initialize() takes the generation its caller bumped, instead of
  reading it after the lock was released, so a clear() in between is no
  longer undone by installing a tracker built from the old settings.
- push_samples() (ffmpeg) pushes under a dedicated mutex and releases
  its tracker reference before that mutex; clear() and initialize()
  take the mutex before destroying a replaced tracker. The audio thread
  is never left with the last reference, whose destructor waits for
  the video encoder thread and would stall the audio pipeline.

Ring tracker:
- Test AVFMT_NOFILE on the output format's flags, not on the context's
  flags (where 0x0001 is AVFMT_FLAG_GENPTS).
- Convert audio sample counts to time in whole seconds plus remainder;
  scaling the whole count to nanoseconds overflowed int64 about 53
  hours after the last re-anchor.
- Drop a frame whose pts would not advance before mapping, converting
  and scaling it. The keyframe state still only advances for a frame
  that is sent.
- Reuse one AVFrame and AVPacket per encoder instead of allocating them
  for every frame and packet. The video frame's pict_type is now set on
  every frame (a reused frame would otherwise stay an I-frame).
- Decide MP4 vs Matroska once.
- Remove the EAGAIN fallback #define here and in the tests: MSVC's
  errno.h defines it.

Tests:
- Every saved file must not be mostly keyframes. This catches a reused
  frame keeping its forced I (verified: all 50 of 50 packets became
  keyframes, failing 20 ring tests).

---- c2ef88591  Claude  2026-11-12

Stream history: simplify the session back to main's locking

StreamHistorySession is main's version again, plus what the ffmpeg ring
needs (picking the tracker, passing the planar audio through, its
constructor, the fixed 48 kHz stereo attach shape) and one change:
clear() and initialize() still run under the session lock, as on main,
but hand back the tracker they replaced, and the caller destroys it
after releasing the lock (destroying the ring can wait on its encoder
thread). They move into Data, the only place that can name the tracker
type.

This drops the generation counter, building the tracker outside the
lock, pushing audio outside the lock, and the push mutex. Building the
tracker is cheap (the video encoder opens lazily on its own thread),
an AAC encode under the lock is well under a millisecond, and these
layers had already needed two race fixes.

---- 8586c436e  Claude  2026-11-12

Stream history: rebuild the scaler on output size changes; cache unopenable encoders; one FPS helper

- The video scaler was only rebuilt when the input or the encoder's
  pixel format changed, not when the encoder's size did. Resolution can
  be changed while recording, and an encoder reopened after a runtime
  failure takes the new size: a smaller one made sws_scale write past
  the end of the frame, a larger one dropped every frame from then on.
  sws_getCachedContext() now takes the output size on every frame,
  replacing the hand-kept m_sws_* fields.
- Encoders that fail to open are remembered for the process, by name
  and output size, so trackers rebuilt on input, resolution, startup or
  shutdown changes no longer initialize those devices and log the same
  failure again.
- StreamHistoryOption::target_fps() replaces the three copies of the
  VideoFPS switch (ring tracker, SaveFrames tracker, tests).

Tests: a test hook, reopen_video_encoder(), closes and reopens the
video encoder as after a runtime failure (which no encoder available to
a test can be made to do). "Encoder reopened at a new size" (RGB and
NV12) switch the frames from 16:9 to 4:3, then reopen the encoder: both
failed before the scaler fix (no video in the saved file).

---- b579e6b7f  Claude  2026-11-12

Stream history: remove the encoder reopen test hook and its tests

The stream history settings cannot change during a tracker's life:
opening the Settings panel closes the program panel, and with it the
console session that owns the stream history (and the Settings button
is disabled while a program runs). So a reopened encoder always gets the
same size, and the "Encoder reopened at a new size" tests covered a case
the app cannot produce, at the cost of a test-only method in the
tracker's header. sws_getCachedContext() stays: it is less code than
tracking the scaler's shape by hand.

---- 990aa84a5  Claude  2026-11-12

Stream history: set NVENC's forced-idr option too

h264_nvenc spells the option "forced-idr" while h264_qsv and h264_amf
spell it "forced_idr", so the forced keyframes on NVENC were plain
I-frames without AV_PKT_FLAG_KEY. Set both spellings.

---- f9ecab95e  Claude  2026-11-12

Stream history: reduce the session to main plus the ring's hooks

StreamHistorySession is main's version again, with only what the ffmpeg
ring needs, each behind PA_STREAM_HISTORY_FFMPEG: picking the tracker,
its constructor, passing the planar audio through, and the fixed 48 kHz
stereo attach shape.

Destroying the replaced tracker after releasing the session lock is
dropped: main's tracker is destroyed under the lock too, and the swaps
it covered (input, resolution, startup and shutdown changes) are rare.

Co-authored-by: Gin890 <79114795+Gin890@users.noreply.github.com>
Co-authored-by: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
@jw098 jw098 closed this Nov 12, 2026
@jw098

jw098 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Other PR had too many commits, so I squashed them, to make it easier to rebase.

The session calls the tracker (inline AAC encode, tracker construction and
teardown) while holding its SpinLock. Note that this is known and deferred,
and how to fix it if it ever matters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
encode_video_frame() recorded m_last_keyframe_us and m_last_pts_us before
avcodec_send_frame(). When the send returned early (the NO_MEMORY path), the
forced keyframe never reached the encoder but the next one was still
scheduled 2 s later, so the keyframe gap could exceed evict()'s headroom.
Update both only after the send succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants