Repository navigation
Conversation
Keeps the stream history as an in-memory ring of encoded H.264 (hardware
encoders, with ffmpeg's LGPL mpeg4 as a fallback) and AAC packets, and
muxes it to MP4/MKV only when saving. Enabled by the CMake option
PA_STREAM_HISTORY_FFMPEG (on by default); requires ffmpeg >= 5.1.
Squashed from 91 commits (branch stream-history-ffmpeg-ring at f9ecab95e).
Original messages, oldest first:
---- 683512180 Gin890 2026-10-12
Add ffmpeg-based in-memory stream history ring (opt-in via PA_STREAM_HISTORY_FFMPEG).
Replaces the JPEG-frame ring with an in-memory ring of encoded H.264 video + AAC
audio packets when the new CMake option PA_STREAM_HISTORY_FFMPEG is enabled.
Nothing is written to storage until save() is called, at which point the ring
is muxed into an MP4 (or MKV for .mkv filenames) via libavformat.
- New StreamHistoryTracker_Ring (Recording/StreamHistoryTracker_Ring.{h,cpp}):
- Video: dedicated encoder thread with the same bounded-queue / drop-oldest
scheme as the JPEG tracker. Hardware H.264 encoders preferred
(qsv/nvenc/amf on Windows, videotoolbox on macOS, v4l2m2m on Linux),
libx264 veryfast as fallback, no B-frames, keyframe every 2 s, NV12 fast
path with a Qt RGB888 fallback for other pixel formats.
- Audio: resampled to 48 kHz (swresample) and encoded as AAC inline in
on_samples(); deinterleaves the DUAL_44100 input, which the session
declares as mono but the pipeline delivers as interleaved stereo.
- The ring is evicted by time window (HISTORY_SECONDS) plus a ~90 MB cap.
- save() snapshots the ring (shared packets, no deep copy), rewinds to the
last keyframe, and muxes H.264+AAC. Encoder init / capture failures log
and disable the affected stream instead of crashing.
- StreamHistorySession picks the ring tracker when PA_STREAM_HISTORY_FFMPEG is
defined (same #if shape as the existing Qt version check).
- CMake: option(PA_STREAM_HISTORY_FFMPEG OFF) - OFF leaves every existing
build untouched. When ON: Windows links the ffmpeg import libs from
3rdPartyBinaries/ + headers from 3rdParty/ffmpeg/ (mirroring the OpenCV
precedent; DLLs placed in 3rdPartyBinaries/ are auto-deployed by the
existing THIRD_PARTY_DLLS glob), macOS/Linux use pkg-config
(avcodec/avformat/avutil/swscale/swresample, requires ffmpeg >= 5.1).
Missing ffmpeg fails at configure time with an explicit FATAL_ERROR.
Note: the design discussion called the audio codec Opus; AAC was chosen
instead because the save target is 'Video.mp4' (Discord attachments) and
H.264+AAC MP4 plays everywhere, while Opus-in-MP4 has spotty player support.
---- a464e307d Gin890 2026-10-12
Fix several bugs in the ffmpeg stream history ring found during review.
- save() rewound the video to the LAST keyframe, so saving a 30 s window
returned only the ~2 s since the newest keyframe. Rewind to the FIRST
keyframe in the ring instead (loses at most one keyframe interval from
the head of the window).
- Video packet PTS were computed as us * time_base.den / time_base.num,
which for the 1/1e6 timebase multiplies timestamps by 1e6, producing
unplayable files. Use av_rescale_q from a 1/1000000 timebase.
- Muxed packets never set packet->stream_index, so real libavformat would
route every packet (video and audio) into stream 0.
- swr_alloc_set_opts2() was called with 9 args including &swr as the first
parameter; the ffmpeg >= 5.1 API takes 8 (out channel layout first, no
SwrContext**). This did not compile against real ffmpeg headers.
- avcodec_send_frame() EAGAIN (encoder input queue full - realistic with
hardware encoders) was treated as a hard failure without draining, which
clogs the encoder permanently and logs every frame after. Drain one
output packet and retry the send.
- Video encoder failure paths wrote video_enabled without the ring lock
(raced with on_frame reading it under the lock).
- Link the ffmpeg libraries PRIVATE on POSIX (matches the PRIVATE compile
definition and the Windows side).
- Include libavutil/error.h and libavutil/rational.h explicitly.
---- ca5fbcf57 Gin890 2026-10-14
Replace the libx264 video fallback with ffmpeg's built-in LGPL mpeg4 encoder.
The project is LGPL-only, so the encoder candidate list no longer relies
on the GPL libx264. The final candidate is now 'mpeg4' (MPEG-4 Part 2,
built into libavcodec in LGPL builds), with the libx264-specific
preset/tune/bframes options removed along with the now-unneeded <cstring>
include. The CMake comment now points at the LGPL shared ffmpeg builds
(gyan.dev '...-lgpl-shared' / BtbN 'win64-lgpl-shared').
---- 09daad5d5 jw098 2026-10-13
add ffmpeg library files
---- cccc92f44 jw098 2026-10-14
fix build
---- 22d47fc0c Gin890 2026-10-14
Fix the stream history ring save path and make it build against ffmpeg 5.1-8.x
- Save bug: the AAC encoder was opened without AV_CODEC_FLAG_GLOBAL_HEADER,
so it emitted ADTS frames with no AudioSpecificConfig extradata and
mux_to_file() failed to set up the audio stream - every save that
included audio failed. Set the flag (both MP4 and MKV need it).
- MP4 output: pass movflags=+faststart to the mp4 muxer so the moov atom
(file index) is written at the front of the file instead of at the end.
MKV saves are unchanged.
- Portability: jw's build fixes targeted only the ffmpeg 8.0 API surface
(void av_channel_layout_default + av_channel_layout_check,
avcodec_get_supported_config, swr_alloc_set_opts2 taking a SwrContext**),
which does not compile against the ffmpeg that mac/linux package managers
provide. Guard each version-sensitive call on the library version macros
so the tracker builds against ffmpeg 5.1 through 8.x (AVCodec.pix_fmts
for pre-7.0, the old swr_alloc_set_opts2 signature for pre-8.0, the
int-returning av_channel_layout_default for pre-8.0).
- Fix an AVFrame leak in on_samples() (early return without
av_frame_free) and stop retrying a permanently broken audio path by
disabling audio instead.
- Restore <cstring> (still needed for memcpy) and add
<libavutil/version.h> for the guard macros.
---- 792ecbbbd jw098 2026-10-14
fix build
---- 78a362c68 jw098 2026-10-14
fix ffmpeg EAGAIN sign convention
---- fe6efc75f jw098 2026-10-14
install ffmpeg libraries for mac/linux CI
---- a86c16fd3 jw098 2026-10-14
attempt to fix mac CI
---- c11acd0fd jw098 2026-10-14
fix CI
---- 9818fb05c Gin890 2026-10-14
Fix ffmpeg C++ linkage, C4458 under /W4 /WX, and mac/linux pkg-config detection
* StreamHistoryTracker_Ring.cpp: ffmpeg's public API headers do not carry
their own extern "C" guards, so from C++ every ffmpeg declaration was
C++ name-mangled and none of the symbols resolved against the ffmpeg
shared libraries (Windows: LNK2019 unresolved externals at the link
stage after a full ~31 min build; mac/linux: undefined symbols).
Wrap all ffmpeg includes in an extern "C" block.
* Restore the version-conditional include: ffmpeg 8.0 (libavutil 61+)
renamed err.h to error.h, but Ubuntu 24.04's apt ships ffmpeg 6.1
(libavutil 58) which only has err.h - the unconditional error.h include
fails to compile there.
* Re-apply the C4458-safe parameter/local names in mux_to_file() and
save() (video/audio/have_audio/video_start hide Impl members, which is
a /W4 warning that becomes an error under /WX on Windows).
* Restore the #ifndef EAGAIN fallback for MSVC.
* CMakeLists (POSIX): pkg-config module names differ between platforms -
Homebrew ships avcodec.pc etc. while Debian/Ubuntu ship libavcodec.pc
etc. - so the previous 'pkg_check_modules(FFMPEG REQUIRED avcodec ...)'
failed configure on Ubuntu, and the 'libavcodec>=59' version check
failed configure on macOS (brew has no libavcodec.pc). Try both
namings, check the minimum version with whichever was found, and add
target_link_directories (like the OpenCV/Tesseract/DPP blocks above)
so -lavcodec etc. are found under /opt/homebrew/lib on macOS.
---- 671af3433 jw098 2026-10-14
fix build
---- 3a59cc26f Gin890 2026-10-15
Stop keying ffmpeg API differences on version macros; detect the actual API
The CI Ubuntu runner's ffmpeg (7.1-era) already has the 'new' API
signatures (void av_channel_layout_default, int
swr_alloc_set_opts2(SwrContext **ps, ...)) while its libavutil/libswresample
major numbers are still below the 8.0 ones the #if guards expected, so the
old-API branches were selected against new-API headers and failed to
compile.
- av_channel_layout_default / swr_alloc_set_opts2: select the API
generation at compile time from the actual declarations
(std::is_same on the function-pointer types, C language linkage). The
generation-specific calls are routed through function templates so they
are only instantiated when their branch is taken - names/expressions in a
discarded if-constexpr branch are still looked up and checked, which is
why the old 8-argument swr_alloc_set_opts2 call (missing the
AVDictionary** that real ffmpeg 5.1-7.0 takes) also failed against the
real headers.
- error.h vs err.h: pick with __has_include instead of a version macro.
- avcodec_get_supported_config stays a preprocessor gate
(LIBAVCODEC_VERSION_MAJOR >= 61): the symbol is not declared at all in
older headers, so the code must be removed before name lookup, not merely
branch-discarded. The threshold is confirmed by this CI (its libavcodec
compiles the new branch fine) and by ffmpeg 6.1 (libavcodec 60, which has
AVCodec.pix_fmts instead).
- CMake: log which pkg-config module set / version was found.
Verified: syntax-checks against the committed ffmpeg 8.0 headers, and full
behavioral harness runs against both an ffmpeg 5.1-7.0 stub set and a
7.1-runner-model stub set (new signatures, old version numbers).
---- 090fd71d8 Gin890 2026-10-15
Stream history: drop the dual ffmpeg API-generation support; enforce ffmpeg 7.x
Per jw: we won't support the old ffmpeg API, so remove the
compile-time detection of the API generation in
StreamHistoryTracker_Ring.cpp and use the current signatures directly.
- error.h/err.h: just include <libavutil/error.h>. The __has_include
fallback to err.h was dead either way - error.h exists in every
ffmpeg release, and ffmpeg 8.x removed err.h by merging it into
error.h.
- Removed kChannelLayoutDefaultReturnsVoid /
kSwrAllocSetOpts2TakesContextPtr and the old-API template shims; call
av_channel_layout_default() (void, validated with
av_channel_layout_check()) and swr_alloc_set_opts2() taking a
SwrContext** directly. These signatures exist in ffmpeg 7.1+ (the
pre-7.1 releases pass an AVDictionary** to swr_alloc_set_opts2 and
get an int back from av_channel_layout_default).
- CMake: enforce the floor for real. macOS/Linux: avcodec>=61
(libavcodec 61 = ffmpeg 7.x; previously avcodec>=59 / 'ffmpeg 5.1').
Windows: new check parsing LIBAVCODEC_VERSION_MAJOR out of the
vendored 3rdParty/ffmpeg/ headers (previously there was no version
check at all). The vendored 8.x headers pass.
---- 464b000e6 Gin890 2026-10-15
Stream history: enforce the ffmpeg 5.1 floor instead of 7.x
The ring's unguarded ffmpeg API usage (channel layout API:
av_channel_layout_default/check/copy/uninit, AVCodecContext.ch_layout,
AVFrame.ch_layout; swr_alloc_set_opts2; everything else is pre-5.x API)
is available in ffmpeg 5.1 with unchanged signatures, so 5.1 is the
minimum supported version.
libavcodec's major version cannot express that floor as a number (4.x
and 5.x are both libavcodec 58), so the CMake floor check now detects
the 5.1 API generation in the headers (vendored tree on Windows,
pkg-config include dirs elsewhere) instead of comparing
LIBAVCODEC_VERSION_MAJOR against 61.
The pix_fmts/avcodec_get_supported_config split in
StreamHistoryTracker_Ring.cpp is untouched: it is an API-generation
gate for 7.0+ headers, and with 5.1-7.0 headers the AVCodec.pix_fmts
path (still present in those releases) compiles instead.
---- 5e06fd7f0 Gin890 2026-10-15
Stream history: fall back to the next video encoder if it fails at runtime
Hardware encoders (h264_amf & co.) can open successfully and still fail
on every frame - e.g. h264_amf whose D3D11 device cannot create
textures (SubmitInput() failed with error 18 / AMF_DIRECTX_FAILED,
CreateTexture2D -> 0x887A0005 DXGI_ERROR_GPU_NOT_FOUND). Previously that
logged 'Video encode failed' on every frame forever and the saved file
had no video.
After 3 consecutive hard avcodec_send_frame() failures the current
candidate is abandoned and init_video_encoder() retries starting from
the next candidate (typically the CPU mpeg4 encoder), so the recording
still gets video and the error spam stops. Candidate rotation is
remembered (failed candidates are not retried), and the scaler is
rebuilt when the new encoder prefers a different pixel format.
---- adfb6b026 Gin890 2026-10-15
Stream history: never free a failed hardware encoder context
On some AMD iGPUs (observed: Ryzen 5 5625U), a hardware H.264 encoder
can open successfully and then fail on every frame because its GPU
device cannot allocate textures (h264_amf: CreateTexture2D fails with
DXGI_ERROR_GPU_NOT_FOUND). Releasing such a context with
avcodec_free_context() crashes the AMD display driver with an access
violation in atidxx64.dll, taking the whole process down - observed
both at the encoder fallback point and in the tracker destructor.
Therefore, when a hardware encoder fails at runtime (or at open), its
context is now deliberately leaked (a bounded, one-time leak) instead
of freed, and the encoder is remembered as poisoned for the rest of
the process so that later sessions skip it instead of re-initializing
it. Fallback to the next candidate now happens on the first hard
avcodec_send_frame() failure instead of after three, to minimize the
number of calls into the broken driver. The CPU mpeg4 fallback is
still released normally.
---- 5776b17c0 jw098 2026-10-15
fix build
---- 18d6660b2 Gin890 2026-10-15
Stream history: fix pixel format selection on ffmpeg 7.0+, don't poison open failures
Two fixes for the encoder fallback path:
- mpeg4 failed to open with 'Invalid argument' on ffmpeg 8.x: since
7.0, AVCodec.pix_fmts is gone and the supported formats must come
from avcodec_get_supported_config(); when that query yields no list,
ctx->pix_fmt was left at AV_PIX_FMT_NONE, which mpeg4 (and likely
other encoders) reject at open time. The chosen input pixel format
now always falls back to YUV420P - supported by every encoder in
the candidate list - when no list is available, and the selected
format is logged when the encoder opens.
- Encoders that merely fail to open (e.g. h264_qsv/h264_nvenc on a
machine without that hardware) are now released normally instead of
being leaked and poisoned for the whole process: the driver was
never exercised through a context that failed at open. Only a
context that failed at runtime (after pushing frames into a possibly
broken GPU driver) is leaked, as before.
---- 24202f8a1 Gin890 2026-10-15
Stream history: use a mpeg4-compatible timebase and set the frame rate
Two encoder-setup fixes, both observed on an AMD Ryzen 5 5625U iGPU
(ffmpeg 8.1):
- mpeg4 failed to open with EINVAL: 'timebase 1/1000000 not supported
by MPEG 4 standard, the maximum admitted value for the timebase
denominator is 65535' (the 16-bit time_incr_length of MPEG-4 Part 2).
The video timebase is now 1/1000 (milliseconds) for all candidates;
the hardware H.264 wrappers accept any timebase, and the muxer
derives packet timestamps from the ring's microsecond WallClock
timestamps anyway.
- h264_amf failed at the first SubmitInput: the AMD driver logged
'unexpected frame rate' and removed the D3D11 device
(AMF_NO_DEVICE). The frame rate is now set explicitly
(ctx->framerate = target_fps/1), which the AMF wrapper passes to the
driver; it also feeds rate control on the other hardware encoders.
---- 284126b2e Gin890 2026-10-16
Stream history: mux from codec-parameter snapshots, not live contexts
Saving the video crashed in avformat_write_header() (access violation
in avformat-63.dll) on an AMD iGPU with the h264_amf encoder. The
muxer ran on the caller's thread while the video worker and the audio
pipeline thread were still encoding into the same AVCodecContexts, and
ffmpeg is not thread-safe on a live context: the hardware encoders
populate their extradata lazily (sometimes only after the first
encoded packet, and possibly mid-stream), so the cross-thread
avcodec_parameters_from_context() in mux_to_file() could capture a torn
state (e.g. a nonzero extradata_size with a null extradata pointer),
which is what made the mp4 muxer walk off a null pointer.
The muxer now works exclusively from codec-parameter snapshots
(video_params/audio_params + video_time_base) that are captured on
each encoder's own thread - the video worker after every encoded
packet, the audio pipeline thread after every encoded AAC frame -
under the ring lock, and are refreshed whenever the encoder's
extradata size changes. A new encoder invalidates the stale video
snapshot. If no snapshot (or no extradata) exists yet, the save fails
with the existing 'missing encoder state' error instead of crashing.
Also logs the video extradata size and first four bytes at mux time
to make any future extradata-format issue (AVCC vs Annex B) visible
in the log.
---- 346fe0134 Gin890 2026-10-16
Stream history: convert H.264 from Annex B to AVCC when muxing MP4
The h264_amf wrapper emits its H.264 stream in Annex B form (SPS/PPS
extradata and packets with 00 00 00 01 start codes), even with
AV_CODEC_FLAG_GLOBAL_HEADER. The MP4 muxer in the vendored ffmpeg 8.x
(avformat-63) crashes with an access violation inside
avformat_write_header() when it is given H.264 in that form (older
ffmpeg, e.g. 6.x, tolerated it). Now that h264_amf is the first
encoder that actually works on AMD iGPUs, saving the video hit this
crash deterministically.
Since MP4 (ISO BMFF) canonically stores H.264 as AVCC - an avcC
configuration record in the stream header plus NAL units with
4-byte big-endian length prefixes - mux_to_file() now converts the
stream to that form when the video is H.264, the target is MP4, and
the extradata is Annex B: it builds the avcC record from the SPS/PPS
NAL units in the extradata snapshot and rewrites each ring packet
with NAL length prefixes at write time. Matroska output is left in
Annex B, and encoders that already produce AVCC (nvenc, qsv) are
untouched.
Also fixes a regression introduced by the parameter-snapshot commit:
the 'missing encoder state' check required extradata for every video
codec, but MPEG-4 Part 2 (the CPU fallback) has no extradata at all;
the requirement now applies to H.264 only (AAC audio still requires
its AudioSpecificConfig).
---- d790c5c3f Gin890 2026-10-16
Stream history: open the output file before avformat_write_header
avformat_write_header() does not open format->url: the lavf docs
require the caller to provide an opened AVIOContext (format->pb),
and the muxer's write_header() dereferences s->pb unconditionally.
With pb still NULL the MP4 muxer faults in mov_write_header
(NULL+0x88 read, avformat-63.dll). Open the file with avio_open2()
just before writing the header and close it on the failure path
(the success path already closed it via avio_closep).
---- 855c59c8e Gin890 2026-10-17
Stream history: write a valid avcC record when muxing MP4
The avcC configuration record built by h264_avcc_from_annexb_extradata()
was malformed: it wrote a stray 5th byte (sps[4], the first byte of the
SPS RBSP) before the reserved/lengthSizeMinusOne byte, and used 0xE0
(numOfSequenceParameterSets = 0) instead of 0xE1 (= 1). Players that
take the SPS from the stream header therefore found none: the saved MP4
had the right duration and container structure, but decoded to a blank
screen, while MKV (which keeps the Annex B SPS/PPS in-band) played fine.
The record now matches the ISO/IEC 14496-15 avcC layout
01 <profile> <compat> <level> FF E1 <spsLen16> <SPS> 01 <ppsLen16> <PPS>,
byte-for-byte the same layout ffmpeg's own mov muxer writes from the
same SPS/PPS (verified by muxing and decoding the result).
---- d3a95e686 Gin890 2026-10-17
Stream history: keep video dts strictly increasing when muxing to file
Ring video packets are stamped with the wall-clock time of the frame they
were drained from, and a hardware encoder can emit several packets for one
frame (or flush two or more at once), so consecutive ring packets can share
a timestamp. The MP4 muxer rejects that:
[mp4] Application provided invalid, non monotonically increasing dts to
muxer in stream 0: 596 >= 596
and the whole save failed.
mux_to_file() now clamps a non-increasing video dts to last+1 (1 ms at the
1/1000 video timebase), and skips empty ring packets instead of feeding an
empty sample to the muxer (which would also fail the Annex B -> AVCC
conversion).
Verified with a standalone lavf harness: writing two consecutive H.264
packets with the same dts fails with AVERROR(EINVAL) exactly like the
reported error, and the clamped sequence muxes and decodes cleanly.
---- 6b12b5b03 Gin890 2026-10-17
Stream history: set the AAC encoder's sample format (FLTP) and de-interleave into planes
init_audio_encoder() never set actx->sample_fmt, so avcodec_open2() failed
with EINVAL ("Invalid audio sample format: -1") and audio was disabled on
every platform - the log showed 'Failed to open AAC encoder. Disabling
audio.' even though everything else in the configuration was valid.
The native AAC encoder only accepts AV_SAMPLE_FMT_FLTP, so:
- set actx->sample_fmt = AV_SAMPLE_FMT_FLTP (and frame_size = 1024) before
avcodec_open2(), and
- on_samples() now builds each 1024-sample encode frame as FLTP by
de-interleaving the accumulated packed resampler output into the frame's
per-channel planes instead of memcpy'ing it as one packed plane.
Also split the open-failure logging so the channel-layout step and
avcodec_open2() failures are reported separately, with the ffmpeg error
string for the latter.
Verified with a standalone harness that replicates the full pipeline
(resample 44.1/48 kHz -> packed FLT -> de-interleave -> FLTP frames ->
AAC encode with AV_CODEC_FLAG_GLOBAL_HEADER): the encoder opens, produces
packets, and exposes the 5-byte AudioSpecificConfig extradata that the
MP4/MKV muxers need.
---- d029a40c4 Gin890 2026-10-17
Stream history: initialize the audio resampler before use
make_audio_resampler() only called swr_alloc_set_opts2(), which allocates
the SwrContext and fills in the options but does NOT initialize it; the
context must be passed through swr_init() before swr_convert() will work.
On every audio callback swr_convert() therefore logged 'Context has not
been initialized' and returned an error, so no audio samples were ever
accumulated and saved videos had a video track but no audio track - even
after the AAC encoder itself was fixed to open (sample_fmt/FLTP).
Call swr_init() after allocation and free the context if it fails; the
existing 'Failed to create audio resampler' error path now reports real
initialization failures too.
---- 372b23aca Gin890 2026-10-17
Stream history: stamp MKV packets in milliseconds and interleave the writes
The Matroska muxer writes packet timestamps into the file verbatim: its
write_header() rewrites every stream timebase to 1/1000 (the native
TimestampScale is milliseconds) but does not rescale the dts/pts the
application submits. The audio stream was submitted with a 1/48000
timebase and sample-count timestamps, so every audio block landed 48x too
late in the file timeline (a 30 s clip's audio blocks spanned 0..24 min)
and the segment Duration element became ~24 min for a 30 s clip.
Symptom: the MKV played its 30 s of video with no audible audio (each
audio block carried only 21 ms of sound, spaced 1.024 s apart on the
timeline), then sat on the last frame for the remaining ~19 minutes the
player believed the file still had. MP4 was unaffected because the mov
muxer rescales from the stream timebase itself.
Fix in mux_to_file():
- for .mkv, use a 1/1000 stream timebase for both video and audio and
rescale the packet dts into it (video: us -> ms; audio: samples -> ms),
while .mp4 keeps the encoder's video timebase and the 1/48000 audio
timebase it already handled correctly;
- add a strictly-increasing dts guard to the audio packets, mirroring the
video one;
- submit the video and audio packets interleaved in timestamp order
instead of writing the whole video stream and then the whole audio
stream, which laid the Matroska clusters out of time order (all audio
data physically after all of the video).
Verified with a standalone harness that muxes real H.264 + AAC packets
through the same avformat calls: with sample-count audio dts the MKV
Duration element came out 1441813 ms (24 min) for a 30 s clip, with the
new millisecond stamping the duration is ~30 s and the audio packets
decode at 0..30 s; the MP4 path is unchanged.
---- f522f96a4 Gin890 2026-10-17
Stream history: remember every runtime-failed video encoder, not just the last
poisoned_encoder held a single name, so if two different hardware
encoders hard-failed in one process, the second overwrote the first and
a later session would re-initialize the first (leaked, possibly still
broken) encoder. A set keeps the poison for every encoder that failed.
---- 26be07beb Gin890 2026-10-17
Stream history: drop the unused <libavutil/opt.h> include
Only av_dict_* is used here, which is declared in dict.h; both
avcodec.h and avformat.h include dict.h directly, so opt.h was
redundant. pixdesc.h stays (needed by the Windows build).
---- d555e91ce PA Dev Bot 2026-10-17
Stream history: log the first video packet's bytes at mux time
Every known hardware encoder emits the extradata and the packets in the
same form (Annex B), and the MP4 conversion is driven by the extradata
form. If a driver ever reported one form while emitting the other, the
MP4 save would be a silent blank file. Logging the first 4 bytes of the
first ring packet next to the extradata makes any such mismatch
immediately visible (NVENC's SPS/PPS payload, in particular, is
driver-produced and unverified).
---- 415e0ef01 jw098 2026-10-17
adjust visibility of StreamHistory Options, depending on whether ffmpeg is enabled.
---- 32aa983ba jw098 2026-10-17
minor change
---- 59d6c8be6 PA Dev Bot 2026-10-17
Stream history: add the Media Foundation encoder (h264_mf) to the Windows candidates
h264_mf is the OS-native Media Foundation H.264 encoder (typically the
built-in Microsoft encoder). It ships in the vendored LGPL build (verified
in avcodec-63.dll) and is present on essentially all Windows 8+. When none
of the GPU vendors' encoders (qsv/nvenc/amf) is available, it now yields
real, universally playable H.264 instead of dropping to MPEG-4 Part 2.
It joins the candidate list after h264_amf and before mpeg4. Like the
hardware wrappers it is device-backed, so a runtime failure follows the
same leak-and-poison path as abandon_video_encoder() (only the mpeg4 CPU
fallback is released normally); the existing rotation, poison, and
fallback logic all handle it without changes. The mux-time log lines
(extradata hex + first packet bytes) report its H.264 form on first use.
---- 7e5afb09d jw098 2026-10-18
add info about the ffmpeg library
---- dce0deaf9 PA Dev Bot 2026-10-22
Stream history: fix two documentation inaccuracies (comments only)
1. The AVCC-conversion block comment claimed 'the other hardware
encoders (nvenc, qsv) produce AVCC natively'. The ffmpeg 8.1 wrapper
sources show the opposite: qsv and videotoolbox pass the driver's
start-coded SPS/PPS and Annex B bitstream through without any
conversion, and nvenc's packets are the raw start-coded bitstream.
State that, and note that nvenc's driver-produced SPS/PPS payload is
the one unverified case - which is what the mux-time log lines
(extradata hex + first packet bytes) are for.
2. The audio ring packet's back-dated timestamp (now - AUDIO_FRAME_US)
had no comment; explain that it aligns the packet with the start of
the samples it contains, keeping audio/video aligned for eviction
and the save() trim.
No code changes.
---- 7cdf45e90 PA Dev Bot 2026-10-23
Stream history: deduplicate the ffmpeg >= 5.1 API check into pa_check_ffmpeg_51_api()
The WIN32 and mac/Linux PA_STREAM_HISTORY_FFMPEG blocks each carried
their own copy of the ffmpeg 5.1 API-generation detection (grep
channel_layout.h for av_channel_layout_default and swresample.h for
swr_alloc_set_opts2) plus a near-identical FATAL_ERROR. The WIN32
version was just the mac/Linux per-include-dir scan with a one-element
list.
- New pa_check_ffmpeg_51_api(FFMPEG_INC_DIRS) function: scans the
candidate include dirs for the 5.1 API surface and FATAL_ERRORs if
it is missing. No return value - it either passes or dies.
- The WIN32 block now only does its discovery (vendored headers +
import libraries) and records FFMPEG_INC_DIRS; the mac/Linux block
keeps its pkg-config discovery + linking and records
FFMPEG_INC_DIRS from the found module set.
- The shared block after the WIN32/else() split now applies
target_include_directories and runs the check, so the requirement
(and its rationale comment) lives in one place instead of three.
Discovery stays per-platform: Windows vendors import libraries in the
repo (no pkg-config), mac/Linux use pkg-config with dual module
naming. Symbol-grepping instead of version numbers is unchanged
(libavcodec 58 spans both ffmpeg 4.x and 5.x).
Net: 41 insertions, 62 deletions. Behavior-preserving; verified with
the Linux pkg-config configure path, a standalone success/failure
test of the function, and the full command-line suite.
---- 9cdef2423 jw098 2026-10-23
move ffmpeg binary into own folder. zip the dlls.
---- 68d96d53f PA Dev Bot 2026-10-23
Stream history: keep ffmpeg binaries in their own 3rdPartyBinaries/ffmpeg/ folder
Move the Windows ffmpeg binaries out of the 3rdPartyBinaries/ top level
into 3rdPartyBinaries/ffmpeg/, following the discord_social_sdk_win
pattern:
- The import libraries (.lib) are checked in unzipped under
3rdPartyBinaries/ffmpeg/ (jw's commit fb025a692 adds them), so the
add_library(IMPORTED)/set_target_properties declarations move into
the early imported-library cluster (next to ONNX/dpp/discord), with
no configure-time existence checks - like discord_lib.
- The .dll files start zipped in
3rdPartyBinaries/ffmpeg/ffmpeg-n9.0-win64-lgpl-shared-9.0.zip (the
BtbN LGPL shared build, DLLs at the zip root, per the folder's
README). The deploy section at the bottom of the file extracts them
into 3rdPartyBinaries/ffmpeg/ if no .dll is present yet (like the
OpenCV extraction) and adds POST_BUILD copy_if_different commands for
them, since the top-level THIRD_PARTY_DLLS glob no longer sees them.
The old configure-time header/.lib existence checks are dropped (the
shared pa_check_ffmpeg_51_api() call after the platform split still
guards the header generation); a missing .lib now fails at link time,
matching the discord precedent. mac/linux pkg-config wiring is
unchanged. The old top-level ffmpeg .lib/.dll files remain until they
are deleted in a follow-up.
---- 489915dbe jw098 2026-10-23
remove duplicate ffmpeg binaries
---- 25fefc117 jw098 2026-10-23
update comments
---- 91b71d984 PA Dev Bot 2026-10-24
Stream history: consume the audio pipeline's working stream instead of the legacy float path
Per the audio pipeline hand-off doc (PR #1672): do not pull in an ffmpeg
audio resampler - use the WorkingAudioStreamListener path and let ffmpeg
only encode/mux.
- StreamHistoryTracker_Ring now implements push_samples() and consumes
the planar working stream directly: the pipeline tree's per-listener
node delivers it already mixed + resampled to the ring's fixed
48 kHz / stereo attach shape, so the ffmpeg swresample
(make_audio_resampler/SwrContext), the DUAL_44100 left-channel hack,
and the input-rate plumbing are gone. Audio is always recorded as
2ch AAC @ 48 kHz (mono inputs upmix to dual-mono in the tree's
mixer); the encoder now unconditionally uses the 96 kbps stereo
bitrate. The constructor's audio args only gate whether audio is
enabled at all.
- StreamHistorySession attaches the working listener at that fixed
48 kHz / stereo shape in the ffmpeg build (the hand-off's suggested
shape) and forwards the planar data to the tracker without
re-interleaving; the non-ffmpeg builds keep the preset-shape attach +
re-interleave for the SaveFrames tracker.
- The legacy AudioFloatStreamListener path is dropped from stream
history: the session no longer implements on_samples, and
ConsoleSystemSession no longer add/remove_stream_listener()s it, so
the double-feed (legacy float + working stream) goes away.
CMake still links/checks swresample (unused by code now; the 5.1
API-generation check greps it) - a possible follow-up. Verified with
the full command-line suite: 18,272 passed / 0 failed / 3 skipped
(skips = no audio decode backend in this container).
---- 7fe9e40da PA Dev Bot 2026-10-24
Stream history: decouple the ring tracker from the input's shape
The ring records audio at a fixed 48 kHz / stereo regardless of the
input, so the shape parameters in its constructor (the input preset's
framing) were only ever used as an audio-enabled gate. Make that
explicit: the constructor now takes 'bool has_audio'.
In the ffmpeg build the session no longer reads the resolved input
choice's rate / channels at all - it only uses its validity: whether a
device input is active gates whether audio is captured, and the
working-stream attach is unconditionally 2 x 48k. (The non-ffmpeg
build keeps the shape: the SaveFrames tracker records at the
delivered shape, so the session still re-interleaves at the choice's
framing there.)
---- a3578259e PA Dev Bot 2026-10-25
Stream history: keep one keyframe interval of headroom in the ring
save() starts the file at the first keyframe in the ring, but keyframes
arrive only every 2 s and evict() dropped everything older than
"latest - window", so the first keyframe could fall up to a full
keyframe interval after the window's start. A 30 s window therefore
saved only 28-30 s of video (27-28 s in practice, since the file also
ends at the last drained packet, i.e. behind the encoder's internal
delay).
Keep one extra keyframe interval of headroom in the ring: the eviction
threshold is now "latest - window - keyframe_interval", so the ring
always contains a keyframe at or before the window's start and the
saved file always covers the most recent "window" seconds (it may
start up to one keyframe interval earlier, ~30-32 s for a 30 s
window). Memory cost is bounded by one extra interval of encoded
packets (~10% of the ring; well under the 90 MB cap at the 8 Mbps
bitrate clamp).
---- 01a384d84 jw098 2026-10-25
use Pimpl
---- 792cfaa0c jw098 2026-10-25
m_-prefix the ring tracker's Data members
---- 342e0d9b2 PA Dev Bot 2026-10-25
Stream history: document the ring tracker's functions
Brief step-level comments for the functions that had none: the Data
ctor/dtor, init_audio_encoder, init_video_encoder, worker_loop,
encode_video_frame, on_frame, push_samples, save, the tracker
destructor (worker stop ordering), and one-liners for the settings
helpers and the Qt pixel-format mapping. mux_to_file, evict,
drain_one_video_packet, abandon_video_encoder and the H.264 AVCC
converters already had them.
---- d52329e83 jw098 2026-10-25
add comments
---- 871264bb9 jw098 2026-10-25
add comments
---- e41644eeb Claude 2026-10-26
Stream history: fix a null-encoder crash, a shutdown hang, and saves after an encoder fallback
- init_video_encoder(): when every candidate has been used up (e.g. the
last candidate opened and then failed at runtime), disable video
instead of returning with m_vctx null, which the next frame
dereferenced.
- ~StreamHistoryTracker_Ring(): set the stop flag under the queue lock
so the worker cannot miss the wakeup and block forever. On stop the
worker now drops queued frames instead of encoding them.
- When a new video encoder opens after a runtime fallback, drop the
previous encoder's packets from the ring along with its parameter
snapshot; they cannot be muxed with the new encoder's parameters.
---- f74d42566 Claude 2026-10-26
Stream history: fix A/V sync, a save/fallback race, and receive-side encoder failures
- save(): copy the codec parameters in the same critical section as the
ring packets. An encoder fallback replaces both together, so reading the
parameters later under a separate lock could mux one encoder's packets
with another encoder's parameters (or fail with missing encoder state).
- Video packets are stamped from their own pts (ms since the first frame)
instead of the most recently sent frame's timestamp. Hardware encoders
pipeline several frames, which delayed the video against the audio.
- Audio packets are stamped from a sample clock anchored to the wall clock
(re-anchored when the two diverge by more than 100 ms) and taking the AAC
encoder's delay into account via the packet pts. The muxer now places
audio by these timestamps instead of by sample count from zero, so a gap
in the delivered audio no longer shifts all later audio against video.
- A hard failure from avcodec_receive_packet() now falls back to the next
encoder candidate, like a hard failure from avcodec_send_frame() does.
---- 4e689178d Claude 2026-10-26
Stream history: move the H.264 bitstream helpers into their own file
h264_annexb_to_avcc() and h264_avcc_from_annexb_extradata() only work on
byte buffers (no ffmpeg, Qt or tracker state), so they now live in
StreamHistory_H264.h/.cpp. That makes them testable and lets them build in
every configuration, not just with PA_STREAM_HISTORY_FFMPEG. No behavior
change: the function bodies are moved verbatim.
---- b140f973f Claude 2026-10-26
Stream history: add unit tests for the H.264 bitstream helpers
Table-driven tests for h264_annexb_to_avcc() and
h264_avcc_from_annexb_extradata() that check the exact output bytes:
3- and 4-byte start codes, several NAL units, leading garbage, inputs with
no NAL unit, extra and repeated NAL types in the extradata, and missing or
truncated parameter sets.
Three cases fail on this commit and are fixed by the next one: a zero byte
of a following 4-byte start code is kept as part of the previous NAL unit,
and an extradata without a PPS produces an avcC that claims one.
---- 3d1190e61 Claude 2026-10-26
Stream history: fix two H.264 bitstream bugs found by the new tests
- A zero byte belonging to a following 4-byte start code (00 00 00 01)
was kept as the last byte of the previous NAL unit, so the AVCC length
prefixes and the SPS/PPS in the avcC record carried a stray trailing
zero. A NAL unit never ends in a zero byte, so trailing zeros are now
left with the start code.
- An extradata without a PPS produced an avcC record claiming one PPS of
length 0. It now records zero picture parameter sets.
---- d91ea93e6 Claude 2026-10-26
Stream history: add test hooks to the ring tracker
- push_samples() overload taking the capture time of the last sample, so
tests can drive audio and video on one simulated timeline. The existing
overload passes current_time(), so production behavior is unchanged.
- wait_until_idle(): blocks until every queued video frame has been
encoded. on_frame() only queues frames for the encoder thread (dropping
the oldest beyond a small backlog), so tests wait on this between frames.
---- 346488ad5 Claude 2026-10-26
Stream history: add end-to-end tests for the ring tracker
In PA_STREAM_HISTORY_FFMPEG builds, StreamHistory_Tests.cpp now also feeds
StreamHistoryTracker_Ring synthetic frames (black, then white) and audio
(silence, then a 1 kHz tone) on a simulated timeline, saves to MP4/MKV, and
reads the file back with ffmpeg. Covered: stream layout, a keyframe first,
strictly increasing timestamps, file length, window eviction, A/V sync
(also after a one-second audio gap), saving with nothing recorded or to an
unwritable path, destruction with frames still queued, and saving from
another thread while recording. Files go under DEBUG_PATH() and are
removed when the test passes.
---- 879edbf44 Claude 2026-10-26
Stream history: build and destroy the tracker outside the session lock
clear() and initialize() ran under the session's spin lock, so every input,
resolution or startup/shutdown change constructed the new tracker (opening
the AAC encoder, starting the encoder thread) and destroyed the old one
(joining its encoder thread, which may be opening a hardware encoder) while
the audio and video threads spun on that lock in push_samples() and
on_frame().
The lock is now held only to read the settings and to swap the tracker
pointer; the old tracker is destroyed after the lock is released. A
generation counter, bumped on every settings change and on clear(), keeps
a tracker built from stale settings from replacing a newer one.
---- c62cc408e Claude 2026-10-26
Stream history: recognize 3-byte start codes in H.264 Annex B extradata
The MP4 path converts Annex B H.264 to AVCC only when the extradata starts
with a 4-byte start code. Extradata starting with a 3-byte start code
(00 00 01) skipped the conversion and reached the MP4 muxer as Annex B.
The check now accepts both, and is limited to H.264: MPEG-4 Part 2
extradata also starts with a 00 00 01 start code.
---- 8ac841c3a Claude 2026-10-26
Stream history: test NV12 frames, audio jitter and drift, resolution changes
- Save and sync from NV12 frames (the common capture card format, scaled
directly from its planes; the other tests use RGB32).
- Sync with audio delivered 0-50 ms late: no re-sync may happen.
- Sync with the audio device clock running 0.1% fast or slow over 200
simulated seconds: the re-sync keeps the audio within its threshold.
- Input resolution change while recording (RGB32 and NV12).
- The sync tests without a gap now also check that the audio packets are
evenly spaced, which catches a re-sync that should not have happened.
---- 46679a38c Claude 2026-10-27
Stream history: drop audio overlapping a re-sync, report the memory cap
- When the audio device clock runs fast, the re-sync moves the audio
timeline about 100 ms back, so the following packets overlap audio
already written. The muxer bumped them to strictly increasing
timestamps, squeezing them into near-zero-length packets (an audible
glitch). It now drops the overlapping packets and logs how many.
- The 90 MB memory cap shortened saves silently (about 175 s of history at
the Very High quality setting). The first time it drops packets the
window would have kept, the tracker now logs it once, outside the lock.
- Dropped frames (the encoder falling behind) are reported at most every
10 seconds with a count, instead of one red log line per frame.
- The ffmpeg log level is process-wide; it is now set once, by the first
tracker, instead of being reset by every tracker.
---- cb4660cc4 Claude 2026-10-27
Stream history: test that a backwards audio re-sync drops the overlap
Every sync test now also checks that no two audio packets are closer than
one AAC frame, and a new test (device clock 0.1% fast, 110 s, saved window
covering the re-sync at about 100 s) exercises the overlap case directly.
---- 1b4378b52 Claude 2026-10-27
Stream history: extract the Windows ffmpeg DLLs per version; fix the version log
- The Windows DLLs were extracted into 3rdPartyBinaries/ffmpeg/ only when
no .dll was there yet. Since the DLLs are git-ignored, a machine that had
extracted them once kept them after the repo moved to a new zip and new
import libraries: stale DLLs with the same names fail to load ("entry
point not found"), and ones with new names are never extracted. The zip
is now extracted into a folder named after it, completed extractions are
marked with a stamp file (an interrupted one is redone, a failed one is
a configure error), and changing the zip re-runs the configure step.
- The configure log printed a blank ffmpeg version: with several modules,
pkg_check_modules() sets <prefix>_<module>_VERSION, not
<prefix>_VERSION. It now prints the libavcodec version.
---- aae71d3b4 Claude 2026-10-27
Build instructions: install ffmpeg for the stream history
PA_STREAM_HISTORY_FFMPEG is on by default and configure stops without the
ffmpeg development packages, but the macOS and Ubuntu x64 guides did not
install them. Add them (with how to opt out on Ubuntu), and say in the
ARM64 guide that the packages it already installs are now needed for the
stream history too.
---- 9268f49f5 Claude 2026-10-27
Stream history: extract the Windows ffmpeg DLLs once, like OpenCV
Revert the per-version extraction folder, stamp file and configure
dependency on the zip: the DLLs are again extracted once into
3rdPartyBinaries/ffmpeg/ when missing, mirroring the OpenCV DLLs. The fix
for the blank ffmpeg version in the configure log is kept.
---- 67a2a86b1 Claude 2026-10-27
Stream history: fix the ffmpeg 7.0 build and correct the ring's comments
- The AVCodec.pix_fmts / avcodec_get_supported_config() switch was gated on
libavcodec >= 61, but the function was added in libavcodec 61.13
(ffmpeg 7.1); ffmpeg 7.0 (61.3) failed to compile. Gate on 61.13.100.
- Drop a frame whose millisecond pts would not advance instead of clamping
it to a repeat, which mpeg4 rejects as a hard error (and so would
trigger an encoder fallback). Unreachable with the current throttling.
- Replace qt_pixel_format_to_av_pix_fmt() with an NV12 check: only its
NV12 result was used, and its RGB mappings were wrong.
- Comments: describe the thread ownership and what m_lock guards, the
audio clock's delivery-time assumption, encoder fallback on send or
receive failures, the encoders that emit Annex B vs AVCC, the
keyframe-headroom guarantee (give or take a frame, memory cap aside),
and the dts bump; fix stale references (Impl, the resampler, the
ffmpeg version range, the video time base, the bitrate clamp).
- Use the Mutex wrapper for the poisoned-encoder set.
---- 1c2db20c6 Claude 2026-10-28
Stream history: correct and clarify comments
- CMake: the ffmpeg 5.1 check no longer claims the ring calls
swr_alloc_set_opts2() (the audio pipeline resamples; libswresample is
still required and linked), and the version note now says libavcodec
cannot tell 5.0 from 5.1 (both 59), not 4.x from 5.x.
- Session: start() and m_audio_choice no longer describe nonexistent
parameters or a shape the ffmpeg ring ignores, and
attach_working_listener() states the fixed 48 kHz stereo attach.
- Ring tracker: document on_frame(), save() and set_window(); drop the
single-channel pass-through case from push_samples() (the attach is
always stereo); list the members each thread owns by name rather than
by prefix (m_last_* and m_audio_* also matched lock-guarded members).
- H.264 helpers, tests: tidy the start-code scan comments; describe what
the sync tolerance covers (AAC pre-echo), the Feeder's non-default
delivery, and both bounds of the eviction test.
---- 458dc2cf2 Claude 2026-11-11
Stream history: disable video when the scaler cannot be created
sws_getContext() fails the same way for the same input/output shape, so
retrying on every frame only logged an error at the frame rate. Disable
video like the other unrecoverable setup failures.
---- f85e93865 Claude 2026-11-11
Stream history: log each stretch of failed frame conversions
The "could not convert video frame" flag was never reset, so after a
recovery a later failure went unlogged. Reset it once a frame converts.
---- 92891ba60 jw098 2026-11-05
add comments. rename member variables for clarity
---- 166f06f1f jw098 2026-11-11
add comments
---- b8975c6da Claude 2026-11-11
Stream history: tighten the encoder failure handling
- Video: an encoder that refuses a frame (EAGAIN) with no packet to drain
is stuck, so fall back to the next candidate instead of dropping every
frame silently. A packet allocation failure no longer looks like
"nothing to drain".
- Video: log the per-frame skips (map, frame allocation, scaling) once per
failing stretch; they used to be silent. They share one flag with the
RGB conversion failure, cleared when a frame reaches the encoder.
- Audio: stop encoding once a receive error disables audio (the loop kept
feeding the failed encoder), and disable audio with a clear message when
a packet cannot be allocated instead of failing later with EAGAIN.
- Log a failed codec parameter snapshot and drop it so the next packet
retries, instead of keeping a half-reset snapshot that breaks saves.
---- 8e8f48feb jw098 2026-11-11
add comments
---- 87b8567b2 Claude 2026-11-11
Stream history: own ffmpeg frames and packets with unique_ptr
Every return path freed the AVFrame/AVPacket by hand, and an exception
(e.g. bad_alloc from the logger or the ring packet) leaked them. Use
FramePtr/PacketPtr, like the existing CodecParametersPtr.
---- e96ea18e4 Claude 2026-11-11
Stream history: unmap the video frame with an RAII guard
encode_video_frame() unmapped the frame by hand before each early return,
and an exception (e.g. from the RGB conversion or the logger) left it
mapped. QVideoFrame is explicitly shared with reference-counted maps, so
that kept the frame mapped for every holder. ReadOnlyFrameMap maps in its
constructor, unmaps in its destructor, and keeps unmap() for the two
deliberate early unmaps.
---- 0a4a76c57 Claude 2026-11-11
Stream history: ffmpeg RAII header; RAII in mux_to_file()
Add StreamHistory_FfmpegPtr.h: FfmpegPtr, a unique_ptr alias template
whose deleter calls an ffmpeg free function (taking the pointer or its
address, picked at compile time), the Frame/Packet/CodecParameters/
CodecContext/SwsContext aliases, an output format context owner (closes
the file, then frees), and an option dictionary owner.
mux_to_file() repeated its cleanup (free the options, close the file,
free the context) on nine return paths and leaked all three when an
exception escaped. Use the owners instead.
---- 485a55695 Claude 2026-11-11
Stream history: own the encoder state with RAII members
m_video_context, m_audio_context, m_sws and the two codec parameter
snapshots were raw pointers freed in ~Data() and by hand wherever they
were replaced. ~Data() never runs when the constructor throws, so an
exception after init_audio_encoder() leaked the AAC context. Make them
FfmpegPtr members; ~Data() goes away, and the deliberate leak of a
failed hardware encoder in abandon_video_encoder() becomes an explicit
release(). The per-candidate context in init_video_encoder() is owned
too, so a throw from the open-failure log no longer leaks it.
---- 9c5f7abf2 jw098 2026-11-11
add comments. minor refactor to the loop draining packets from the encoder.
---- 04fa2f270 jw098 2026-11-11
update comments
---- 896f12f77 Claude 2026-11-12
Stream history: evict video by its own clock; fix the tests under encoder lag
The video encoder holds the newest frames for a while (h264_amf held up to
~1.4 s when many trackers shared the GPU), so the newest video packet in
the ring trails the newest audio packet. evict() measured the window back
from the newest packet of either stream, so video was evicted early by
that lag, eating into the keyframe headroom; with enough lag the ring held
no keyframe and save() wrote a file starting on a non-keyframe. Measure
the video from its own newest packet (capped at MAX_VIDEO_LAG_US behind
the audio, so stopped video is still evicted), and log when save() finds
no keyframe instead of silently starting at the first packet.
Tests: the save tests assumed the encoder holds at most 0.5 s. They now
feed a 3 s tail so the frames before the checked point leave the encoder,
and the lower bound gets a small slack (4.9 - 0.1 - 0.5 rounded above the
measured 4.3). The eviction test no longer needs a backlog allowance.
---- e76e3ded9 Claude 2026-11-12
Stream history: force IDR keyframes on the hardware encoders
encode_video_frame() forces a keyframe every 2 s with pict_type = I, but
h264_amf, h264_nvenc and h264_qsv turn that into a plain I-frame unless
their forced_idr option is set. Those frames are not marked as keyframes
and a decoder cannot start from them, so once the encoder's first frame
was evicted the ring held no start point and save() wrote a file whose
video does not decode (the Eviction test failed on h264_amf). Set
forced_idr; encoders without the option ignore it.
---- 51c5a6772 Claude 2026-11-12
Stream history: test keyframe spacing, window changes, stopped video, throttling
New checks and tests:
- Every saved video must have a keyframe at least every keyframe interval
plus a frame (guards the forced_idr fix: without forced keyframes, seven
tests fail).
- Eviction and the new window-shrink test check that the audio starts
within one AAC frame of the video.
- "Shrink the window while recording": set_window() evicts at once.
- "Video stops while the audio continues": the old video is evicted by
the audio's clock once it trails by more than MAX_VIDEO_LAG_US.
- "Frames faster than the target FPS": 50 fps input is throttled to the
target frame interval.
The audio-start check found that evict() dropped audio from before the
video's eviction point when the video trailed the audio, so a saved file
could start with up to the encoder's lag of video without sound. Keep the
audio back to the video's eviction point.
---- 6297e4802 Claude 2026-11-12
Stream history: log how far the video trails the audio at save time
The frames the video encoder still holds are not in a saved file. Log the
gap between the newest video and audio packets in save() to measure how
much video a save loses in real sessions, before deciding whether the
encoders' low-latency modes are worth enabling.
---- 923448bb5 Claude 2026-11-12
Stream history: include the frame interval in the save-time lag log
Up to one frame interval of lag between the newest video and audio is
normal (the next frame is not due yet); only lag beyond that means the
encoder is holding frames. Log the interval so the number reads on its own.
---- 5183b47a7 Claude 2026-11-12
Stream history: fix the MP4 tail, H.264 parsing and avcC; tests for each
Muxing:
- Give every muxed packet a duration. Without it the MP4's last video
sample was stored 0 long: the stream ended a frame early and some
readers dropped the newest frame.
H.264 helpers (StreamHistory_H264):
- One Annex B NAL scanner for both helpers. Two start codes in a row
(an empty NAL unit) no longer swallow the rest of the packet as one
NAL, and 00 00 01 followed by 00 00 00 01 no longer fails the whole
conversion (and with it the save).
- Write the avcC High profile extension (chroma format, bit depths, SPS
extension count) that ISO/IEC 14496-15 requires for every profile but
Baseline, Main and Extended, parsed from the SPS after removing the
emulation prevention bytes. Matches ffmpeg's own avcC byte for byte.
Ring tracker:
- StreamHistorySession calls the ring's push_samples() with an owning
reference instead of under the session spinlock, so the video thread
and the UI no longer spin while AAC is encoded.
- Accumulate the audio per channel (planar, like the working stream and
the encoder) instead of interleaving it and de-interleaving it again.
- Open the video encoder before mapping the frame.
- Share the packet-store / parameter-snapshot code between audio and
video, and the hex formatting of the save-time log.
Tests:
- H.264: empty NAL units, a 3-byte start code before a 4-byte one,
trailing zero padding; avcC for Main (no extension), High, High 4:2:2
10-bit, High 4:4:4, an SPS with an emulation prevention byte, a
libx264 SPS/PPS against ffmpeg's avcC, and an SPS cut short.
- Ring: the same ring saved as MP4 and MKV holds the same packets and
the MP4 video stream lasts through its last frame; audio delivered in
channel buffers longer than the samples.
---- 780d88a63 Claude 2026-11-12
Stream history: fix the session's tracker races; second review fixes
Session (StreamHistorySession):
- initialize() takes the generation its caller bumped, instead of
reading it after the lock was released, so a clear() in between is no
longer undone by installing a tracker built from the old settings.
- push_samples() (ffmpeg) pushes under a dedicated mutex and releases
its tracker reference before that mutex; clear() and initialize()
take the mutex before destroying a replaced tracker. The audio thread
is never left with the last reference, whose destructor waits for
the video encoder thread and would stall the audio pipeline.
Ring tracker:
- Test AVFMT_NOFILE on the output format's flags, not on the context's
flags (where 0x0001 is AVFMT_FLAG_GENPTS).
- Convert audio sample counts to time in whole seconds plus remainder;
scaling the whole count to nanoseconds overflowed int64 about 53
hours after the last re-anchor.
- Drop a frame whose pts would not advance before mapping, converting
and scaling it. The keyframe state still only advances for a frame
that is sent.
- Reuse one AVFrame and AVPacket per encoder instead of allocating them
for every frame and packet. The video frame's pict_type is now set on
every frame (a reused frame would otherwise stay an I-frame).
- Decide MP4 vs Matroska once.
- Remove the EAGAIN fallback #define here and in the tests: MSVC's
errno.h defines it.
Tests:
- Every saved file must not be mostly keyframes. This catches a reused
frame keeping its forced I (verified: all 50 of 50 packets became
keyframes, failing 20 ring tests).
---- c2ef88591 Claude 2026-11-12
Stream history: simplify the session back to main's locking
StreamHistorySession is main's version again, plus what the ffmpeg ring
needs (picking the tracker, passing the planar audio through, its
constructor, the fixed 48 kHz stereo attach shape) and one change:
clear() and initialize() still run under the session lock, as on main,
but hand back the tracker they replaced, and the caller destroys it
after releasing the lock (destroying the ring can wait on its encoder
thread). They move into Data, the only place that can name the tracker
type.
This drops the generation counter, building the tracker outside the
lock, pushing audio outside the lock, and the push mutex. Building the
tracker is cheap (the video encoder opens lazily on its own thread),
an AAC encode under the lock is well under a millisecond, and these
layers had already needed two race fixes.
---- 8586c436e Claude 2026-11-12
Stream history: rebuild the scaler on output size changes; cache unopenable encoders; one FPS helper
- The video scaler was only rebuilt when the input or the encoder's
pixel format changed, not when the encoder's size did. Resolution can
be changed while recording, and an encoder reopened after a runtime
failure takes the new size: a smaller one made sws_scale write past
the end of the frame, a larger one dropped every frame from then on.
sws_getCachedContext() now takes the output size on every frame,
replacing the hand-kept m_sws_* fields.
- Encoders that fail to open are remembered for the process, by name
and output size, so trackers rebuilt on input, resolution, startup or
shutdown changes no longer initialize those devices and log the same
failure again.
- StreamHistoryOption::target_fps() replaces the three copies of the
VideoFPS switch (ring tracker, SaveFrames tracker, tests).
Tests: a test hook, reopen_video_encoder(), closes and reopens the
video encoder as after a runtime failure (which no encoder available to
a test can be made to do). "Encoder reopened at a new size" (RGB and
NV12) switch the frames from 16:9 to 4:3, then reopen the encoder: both
failed before the scaler fix (no video in the saved file).
---- b579e6b7f Claude 2026-11-12
Stream history: remove the encoder reopen test hook and its tests
The stream history settings cannot change during a tracker's life:
opening the Settings panel closes the program panel, and with it the
console session that owns the stream history (and the Settings button
is disabled while a program runs). So a reopened encoder always gets the
same size, and the "Encoder reopened at a new size" tests covered a case
the app cannot produce, at the cost of a test-only method in the
tracker's header. sws_getCachedContext() stays: it is less code than
tracking the scaler's shape by hand.
---- 990aa84a5 Claude 2026-11-12
Stream history: set NVENC's forced-idr option too
h264_nvenc spells the option "forced-idr" while h264_qsv and h264_amf
spell it "forced_idr", so the forced keyframes on NVENC were plain
I-frames without AV_PKT_FLAG_KEY. Set both spellings.
---- f9ecab95e Claude 2026-11-12
Stream history: reduce the session to main plus the ring's hooks
StreamHistorySession is main's version again, with only what the ffmpeg
ring needs, each behind PA_STREAM_HISTORY_FFMPEG: picking the tracker,
its constructor, passing the planar audio through, and the fixed 48 kHz
stereo attach shape.
Destroying the replaced tracker after releasing the session lock is
dropped: main's tracker is destroyed under the lock too, and the swaps
it covered (input, resolution, startup and shutdown changes) are rare.
Co-authored-by: Gin890 <79114795+Gin890@users.noreply.github.com>
Co-authored-by: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
The session calls the tracker (inline AAC encode, tracker construction and teardown) while holding its SpinLock. Note that this is known and deferred, and how to fix it if it ever matters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
encode_video_frame() recorded m_last_keyframe_us and m_last_pts_us before avcodec_send_frame(). When the send returned early (the NO_MEMORY path), the forced keyframe never reached the encoder but the next one was still scheduled 2 s later, so the keyframe gap could exceed evict()'s headroom. Update both only after the send succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keeps the stream history as an in-memory ring of encoded packets instead of JPEG frames, and only writes to storage when saving.
h264_qsv,h264_nvenc,h264_amfandh264_mfon Windows,h264_videotoolboxon macOS,h264_v4l2m2mon Linux), then falls back to ffmpeg's LGPLmpeg4. It forces an IDR keyframe every 2 s. An encoder that fails at runtime is replaced by the next one in that list..mkvfilenames). The file starts at the first keyframe, and the audio is trimmed to match. Annex B H.264 is converted to AVCC for MP4 (StreamHistory_H264.{h,cpp}).PA_STREAM_HISTORY_FFMPEGis on by default and requires ffmpeg >= 5.1. On Windows it uses vendored headers, import libs and zipped DLLs (LGPL build). On macOS and Linux it finds ffmpeg through pkg-config, and CI installs it. With the option off, the existingStreamHistoryTracker_SaveFramesis used, unchanged.StreamHistorySession: this is main's version plus four#if defined(PA_STREAM_HISTORY_FFMPEG)blocks: picking the tracker, its constructor, passing the planar audio through, and the fixed 48 kHz stereo attach shape.StreamHistoryOption::target_fps()replaces the FPS switch inSaveFrames.This is a squash of 91 commits from
stream-history-ffmpeg-ring. The commit message keeps every original message, in order.Testing
--command-line-test-mode --test-substring StreamHistory: 50/50 pass. That is 25 H.264 bitstream tests and 25 end-to-end ring tests, which save MP4/MKV files and read them back. The container has no hardware encoder, so the ring tests ran on thempeg4fallback.🤖 Generated with Claude Code
https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9