Skip to content

Stream history: in-memory H.264+AAC ring via ffmpeg (opt-in CMake toggle) - #1563

Closed
Gin890 wants to merge 79 commits into
mainfrom
stream-history-ffmpeg-ring
Closed

Gin890 wants to merge 79 commits into
mainfrom
stream-history-ffmpeg-ring

Conversation

@Gin890

@Gin890 Gin890 commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Needs testing with Mac, Linux, and Windows with Nvidia GPU.
For Mac, Linux, ffmpeg needs to be installed. See the updates to the CI, for an example installation.

Implements the in-memory ffmpeg stream-history ring discussed in #Stream (option 1): video + audio are encoded in the background into an in-memory ring, and nothing touches storage until save() is called, when the ring is muxed into a single MP4/MKV file.

Toggle (compile-time)

-DPA_STREAM_HISTORY_FFMPEG=ON
  • OFF (default) — every existing build/CI path is byte-for-byte unaffected (the new .cpp isn't even compiled; StreamHistorySession.cpp keeps the old #if shape).
  • ON, Windows — headers from 3rdParty/ffmpeg/ (the ffmpeg include/ tree) + import libs avcodec/lib avformat/lib avutil/lib swscale/lib swresample/lib in 3rdPartyBinaries/ (mirrors the OpenCV import-lib precedent). The existing THIRD_PARTY_DLLS glob auto-deploys any ffmpeg DLLs placed in 3rdPartyBinaries/.
  • ON, macOS/Linux — pkg-config (avcodec avformat avutil swscale swresample); Ubuntu: sudo apt install libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libswresample-dev, macOS: brew install ffmpeg.
  • ON but ffmpeg missing / too old — FATAL_ERROR at configure time with an explicit message (ffmpeg ≥ 5.1 / libavcodec 59+ required for the channel-layout API).

What the tracker does (Recording/StreamHistoryTracker_Ring.{h,cpp})

  • Video — same worker pattern as the JPEG tracker (bounded queue of 10, drop-oldest when lagging, AsyncTask on unlimited_normal()). Replaces JPEG with the ffmpeg C API: hardware H.264 encoders first (h264_qsv/h264_nvenc/h264_amf on Windows, h264_videotoolbox on macOS, h264_v4l2m2m on Linux) with libx264 veryfast fallback; no B-frames, forced keyframe every 2 s, NV12 fast path straight into sws_scale with a Qt toImage() RGB888 fallback for other pixel formats. Honors the existing StreamHistoryOption settings (RESOLUTION / VIDEO_FPS / ENCODING_MODE / VIDEO_QUALITY / VIDEO_BITRATE).
  • Audio — on_samples → swresample to 48 kHz → native aac encoder, 1024-sample frames, inline (sub-millisecond). Includes a deinterleave for DUAL_44100, which the session declares as 1ch but the pipeline actually delivers as interleaved stereo.
  • Ring — (WallClock, encoded packet) deques evicted by the HISTORY_SECONDS window plus a hard ~90 MB byte cap (≈23 min at the default 512 kbps). Zero I/O until prompted.
  • save(filename) — snapshot the ring (shared packets, no deep copy), rewind to the first keyframe in the ring (oldest decodable start; loses at most one ~2 s keyframe interval from the head of the window), trim audio to the same start point, mux to MP4 (or MKV when the filename ends .mkv) with libavformat using copy-only semantics. ErrorReports saves to Video.mp4, so MP4 is the default target.

One deviation from the discussion worth calling out: the plan said Opus, but I used AAC — the save target is Video.mp4 (Discord attachment) and H.264+AAC MP4 plays everywhere, whereas Opus-in-MP4 has spotty player support. The ring design (compressed audio packets, 48 kHz, ~20 ms frames) is otherwise as discussed; swapping to libopus is a ~30-line change if you'd rather have it.

Failure behavior

Encoder init, scaler, resampler, or open failures log and disable just that stream (video or audio) — the bot never crashes, matching the existing tracker's failure style. save() with an empty ring logs and returns false.

Verification

No ffmpeg/Qt toolchain in this sandbox, so this is not a full production build. What was done instead:

  • Compiled the real StreamHistoryTracker_Ring.cpp (unmodified, straight from the repo) with g++ 13 -std=c++23 -Wall -Wextra against signature-faithful ffmpeg ≥5.1 API stubs + a minimal Qt QVideoFrame/QImage model, and ran it behaviorally: full video+audio pipeline → MP4 mux (packet counts, keyframe cadence, µs timestamps, A/V alignment), HISTORY_SECONDS eviction, DUAL_44100 deinterleave/resample (44100 → 48000, exact frame counts), empty-ring save, .mkv muxer selection, and x264 veryfast/zerolatency/bframes=0 options. All pass, repeated runs included.
  • The stub x264 encoder deliberately models a hardware-encoder-style one-deep output queue (every send_frame hits EAGAIN until drained), which exercises the EAGAIN/retry path.
  • ASan+UBSan build of the same harness passes all tests.

The stub review caught and fixed (commit 7cd8981), against real ffmpeg 5.1 semantics:

  • save() rewound to the last keyframe → a 30 s window saved only the final ~2 s. Now rewinds to the first keyframe.
  • Video mux timestamps off by a factor of 1e6 (us * den / num on a 1/1e6 timebase) → unplayable files. Now av_rescale_q from 1/1000000.
  • Muxed packets never set stream_index → real libavformat would route audio into the video stream.
  • swr_alloc_set_opts2(&swr, ...) — 9-arg hybrid that does not exist in ffmpeg ≥5.1 (8 params, out layout first); would not compile against real headers.
  • avcodec_send_frame EAGAIN (full encoder input queue — realistic on QSV/NVENC/AMF) permanently clogged the encoder instead of draining + retrying.
  • Unlocked video_enabled writes racing on_frame; ffmpeg libs linked PRIVATE on POSIX; explicit libavutil/error.h/rational.h includes.

Please still do a real build with the toggle ON before relying on it — in particular the hardware-encoder paths (QSV/NVENC/AMF) are the part most likely to need a first-pass tweak.

Files

  • SerialPrograms/Source/CommonFramework/Recording/StreamHistoryTracker_Ring.{h,cpp} — new tracker
  • SerialPrograms/Source/CommonFramework/Recording/StreamHistorySession.cpp — 4-line #if selection
  • SerialPrograms/CMakeLists.txt — option, per-platform link blocks, compile definition
  • SerialPrograms/cmake/SourceFiles.cmake — new header in the list; new .cpp appended only when the option is ON

@jw098
jw098 marked this pull request as draft October 12, 2026 23:31
@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from 63eac29 to 4f3491f Compare October 14, 2026 19:13
@Gin890

Gin890 commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

MP4 blank screen: malformed avcC fixed (commit 93253e8)

Symptom: saved MP4 had the correct duration but decoded to a blank screen, while the same ring saved as MKV played fine.

Root cause: h264_avcc_from_annexb_extradata() (added in b6e2221) built a malformed avcC record for the MP4 stream header:

  1. a stray 5th byte (sps[4], the first byte of the SPS RBSP) was written before the reserved/lengthSizeMinusOne byte, and
  2. the numOfSequenceParameterSets byte was 0xE0 (= 0 SPS) instead of 0xE1 (= 1 SPS).

MP4 players take the SPS from the stream header (avcC); with a zero SPS count they get none, so every sample fails to decode -> blank screen, while the container metadata (duration, size) is still correct. MKV is unaffected because it stores the Annex B SPS/PPS in-band/in CodecPrivate.

The record now follows the ISO/IEC 14496-15 avcC layout 01 <profile> <compat> <level> FF E1 <spsLen16> <SPS> 01 <ppsLen16> <PPS> — byte-for-byte the same layout ffmpeg's own mov muxer writes from the same SPS/PPS. Verified by building the record with this code, muxing it into an MP4, and decoding (clean, non-blank output). Linux build + full command-line test suite pass (1962/1962).

@Gin890

Gin890 commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

MP4 save now fails with "non monotonically increasing dts" — fixed (commit 4d40e92)

Root cause (separate from the avcC bug): ring video packets are stamped with the wall-clock time of the frame they were drained from, and a hardware encoder can emit several packets for one frame (or flush two or more at once) — so consecutive ring packets share a timestamp. The MP4 muxer requires strictly increasing dts and fails the whole save:

[mp4 @ ...] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 596 >= 596

mux_to_file() now clamps a non-increasing video dts to last + 1 (1 ms at the 1/1000 video timebase), and skips empty ring packets instead of passing an empty sample to the muxer (which would also break the Annex B → AVCC packet conversion).

Verified with a standalone lavf harness: two consecutive H.264 packets with equal dts → AVERROR(EINVAL) from av_interleaved_write_frame() (the reported error); with the clamp the same sequence muxes and decodes cleanly (60/60 frames). Linux build + full command-line test suite pass (1962/1962).

On keeping the Annex B → AVCC conversion for MP4: yes, it is still needed.

  • The h264_amf wrapper emits Annex B (SPS/PPS extradata and 00 00 00 01-coded packets) even with AV_CODEC_FLAG_GLOBAL_HEADER — established empirically on AMD iGPUs, where AMF is the first encoder that works.
  • The vendored ffmpeg 8.x MP4 muxer crashes (access violation in avformat_write_header) when handed H.264 in that form — that crash is exactly why the conversion was added in b6e2221.
  • With the avcC record fixed in 93253e8 the conversion now produces a spec-conformant AVCC MP4 (verified by muxing and decoding its output).
  • The only way to drop it would be an encoder that natively emits AVCC (nvenc/qsv do, but AMF does not, and ffmpeg's AMF wrapper exposes no option to force it).

@Gin890

Gin890 commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

Audio never opened: AAC encoder missing sample_fmt — fixed (commit aa6809c)

"Stream history: Failed to open AAC encoder. Disabling audio." was not a Windows/vendored-ffmpeg issue — it reproduces on any ffmpeg build. init_audio_encoder() set rate, timebase, bitrate, channel layout and flags, but never actx->sample_fmt, so avcodec_open2() failed with EINVAL ("Invalid audio sample format: -1") every time. Audio has been silently disabled on all platforms since the ring tracker was added.

The native AAC encoder only accepts AV_SAMPLE_FMT_FLTP (planar float), so the fix:

  • init_audio_encoder() now sets actx->sample_fmt = AV_SAMPLE_FMT_FLTP (and frame_size = 1024) before opening,
  • on_samples() now builds each 1024-sample encode frame as FLTP by de-interleaving the accumulated packed resampler output into the frame's per-channel planes (previously it memcpy'd packed data into data[0] of a frame declared as AV_SAMPLE_FMT_FLT, which the encoder would have rejected anyway),
  • the open-failure log is split so a channel-layout failure and an avcodec_open2() failure are reported separately, the latter with the ffmpeg error string.

Verified with a standalone harness replicating the full pipeline (resample 44.1/48 kHz → packed FLT → de-interleave → FLTP frames → AAC encode with AV_CODEC_FLAG_GLOBAL_HEADER): encoder opens, produces packets, and exposes the 5-byte AudioSpecificConfig extradata that the MP4/MKV muxers need; AAC-with-ASC muxes and decodes cleanly in both MP4 and MKV. Linux build + full command-line test suite pass (1962/1962).

@Gin890

Gin890 commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

Audio resampler never initialized — fixed (commit 886a710)

The AAC encoder now opens (the sample_fmt/FLTP fix is working — see the audio = AAC 1ch @ 48 kHz line), but make_audio_resampler() only called swr_alloc_set_opts2(), which allocates the SwrContext and fills in the options — it does not initialize it. The header is explicit: "Once all values have been set, it must be initialized with swr_init()."

So every swr_convert() call logged [SWR] Context has not been initialized and failed, no samples were ever accumulated, and saved files had a video track but no audio.

Fix: call swr_init(swr) after allocation in make_audio_resampler(), and swr_free() it if initialization fails (the existing "Failed to create audio resampler. Disabling audio." path now reports real failures).

Verified the exact case from the log (48 kHz mono → 48 kHz mono, packed FLT in/out) in a standalone harness: swr_init = 0, swr_convert returns the full 1024 samples with clean data. Linux build + full command-line test suite pass (1962/1962).

@Gin890

Gin890 commented Oct 17, 2026

Copy link
Copy Markdown
Collaborator Author

MKV audio/duration: Matroska needs millisecond timestamps — fixed (commit e49393d)

"MKV: no audio, plays 30 s then freezes on the last frame for >20 min" traced to a Matroska-specific timestamp issue, reproduced with a standalone harness muxing real H.264+AAC packets through the same avformat calls:

The matroska muxer writes packet timestamps into the file verbatim. Its write_header() rewrites every stream's timebase to 1/1000 (Matroska's native TimestampScale is milliseconds) but does not rescale the dts/pts the application submits. We submitted audio with a 1/48000 timebase and sample-count timestamps, so:

  • every audio SimpleBlock landed 48× too late on the file timeline (a 30 s clip's 1024-sample audio blocks were spaced 1.024 s apart, each carrying only 21 ms of sound) → the player had no continuous audio to play during the video → "no audio"
  • the segment Duration element became 1441813 ms ≈ 24 min for a 30 s clip → after the video ends, the player waits out the remaining ~19 min on the last frame → "freeze on the last frame"

MP4 was unaffected because the mov muxer rescales from the stream timebase itself (hence MP4 audio worked).

Fix in mux_to_file():

  1. For .mkv only: use a 1/1000 stream timebase for both streams and rescale the packet dts into it (video: µs → ms; audio: samples → ms). .mp4 keeps the encoder's video timebase and the 1/48000 audio timebase it already handled correctly.
  2. A strictly-increasing dts guard for audio packets, mirroring the video one.
  3. Interleave the video/audio writes in timestamp order instead of writing all video then all audio, which laid the Matroska clusters out of time order (all audio data physically after all video data).

Harness verification: old behavior → MKV Duration element 1441813 (24 min) for a 30 s clip; new behavior → duration ≈ 30 s, audio packets decode at 0…30 s, MP4 path unchanged. Linux build + full command-line test suite pass (1962/1962).

@pa-devbot

Copy link
Copy Markdown
Collaborator

Follow-up to the encoder-format audit (nvenc/qsv/amf/videotoolbox/v4l2m2m wrappers reviewed against the ffmpeg 8.1 sources): no new encoder-specific conversion code is needed - every candidate emits its extradata and packets in the same form. This commit adds one log line at mux time: the first 4 bytes of the first video ring packet, printed next to the existing extradata hex. If a driver ever reported one H.264 form (AVCC extradata) while emitting the other (Annex B packets), the MP4 save would otherwise be a silent blank file; this makes the mismatch immediately visible in the log. Log-only, no behavior change. (aedba6c)

@pa-devbot

Copy link
Copy Markdown
Collaborator

Added h264_mf (the OS-native Media Foundation H.264 encoder) to the Windows candidate list, after h264_amf and before mpeg4. It is compiled into the vendored LGPL build (verified in avcodec-63.dll) and present on essentially all Windows 8+: when none of the GPU vendors' encoders (qsv/nvenc/amf) is available, the session now falls back to real, universally playable H.264 instead of MPEG-4 Part 2. Device-backed like the hardware wrappers, so a runtime failure follows the existing leak-and-poison path; rotation/poison/fallback logic needed no changes. The mux-time log lines report its extradata/packet form on first use. (9508c51, rebased onto dc8129c)

@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from 9508c51 to 927370c Compare October 18, 2026 04:58
@jw098
jw098 marked this pull request as ready for review October 18, 2026 04:59
@Mysticial

Copy link
Copy Markdown
Collaborator

This is GPL safe right? We're using the LGPL part of ffmpeg because we're only including headers and their pre-compiled binaries?

@jw098

jw098 commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

I believe so. I specifically used a LGPL build of ffmpeg. So, I think it should be LGPL?

@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from 927370c to dc747ed Compare October 18, 2026 22:23
@Mysticial
Mysticial self-requested a review October 22, 2026 04:10
@jw098
jw098 marked this pull request as draft October 22, 2026 06:45
@pa-devbot

Copy link
Copy Markdown
Collaborator

Comments-only follow-up (1653b37): (1) corrected the AVCC-conversion block comment - the ffmpeg 8.1 wrapper sources show qsv and videotoolbox pass the driver's start-coded SPS/PPS and Annex B bitstream through unconverted (nvenc packets are the raw start-coded bitstream too), so "the other hardware encoders produce AVCC natively" was wrong; the comment now states that and points at the mux-time log lines as the check for nvenc's one unverified driver-produced payload. (2) documented the audio ring packet's back-dated timestamp (now - AUDIO_FRAME_US): it aligns each packet with the start of the samples it contains, keeping audio/video aligned for eviction and the save() trim. No code changes; full test suite green on the rebased branch.

Mysticial pushed a commit that referenced this pull request Oct 22, 2026
…on (#1671)

* Audio pipeline: rebase the working-stream path onto the new node API.

Rebased #1671 onto main (incl. #1674's AudioFormat /
WorkingAudio2WorkingAudio and #1676's tree simplification), and rewrote
the attach path around the per-listener conversion node:

- AudioPassthroughPair gains
  add_listener(const AudioFormat&, WorkingAudioStreamListener&) /
  remove_listener(WorkingAudioStreamListener&). The Qt pair builds one
  WorkingAudio2WorkingAudio per attached listener (source shape -> the
  requested shape), plus a small forwarder that carries the requested
  shape as its nominal one (the node validates its listeners against
  that; the user's listener may have any nominal shape of its own).
  Entries survive tree rebuilds: the nodes are rebuilt for each new
  source shape in build_tree() and detached in teardown_tree(). The
  INTERLEAVE_RL_96000 channel swap is applied to the node's matrix, the
  same way as for the sink branch. Unresolvable layouts are logged and
  skipped rather than thrown from the calling thread.
- StreamHistorySession now implements both listener interfaces: the
  legacy AudioFloatStreamListener::on_samples path stays exactly as it
  was (kept until #1563 switches off it), and the new
  WorkingAudioStreamListener::push_samples re-interleaves the planar
  working stream at the active preset's physical shape (the shape its
  tracker records at; the RL swap already applied by the pair's node).
  The session attaches at start() and re-attaches in
  post_input_change(), removing the old-shape attach first so no push in
  the old shape can ever race the shape update. While both paths are
  attached the tracker is fed by both (double-record) - inert in the
  active tracker, and #1563 drops the legacy attach when it switches.
- The console wires it up: set_audio_session() before start(), and the
  working-listener remove in try_shutdown().

Verified: -Werror clean on aarch64; full command-line suite green.

* Audio pipeline: define InternalWorkingListener before its first destruction.

remove_listener() destroys a WorkingListenerEntry (m_working_entries.erase)
at its call site, which instantiates
~unique_ptr<InternalWorkingListener>() there. The nested class was defined
further down in the .cpp, so on GCC 15 (x86 CI) the deleter's
sizeof static_assert fired on the incomplete type. aarch64's older GCC
instantiated it lazily enough to miss it.

Moved the definition above the first member-function body so the type is
complete at every destruction site in the translation unit. Pure
reordering, no behavior change.

---------

Co-authored-by: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com>
@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from 1653b37 to d635166 Compare October 23, 2026 05:56
@pa-devbot

Copy link
Copy Markdown
Collaborator

CMake refactor (44810f3): the ffmpeg >= 5.1 API-generation detection that was duplicated in the WIN32 and mac/Linux PA_STREAM_HISTORY_FFMPEG blocks (grep channel_layout.h for av_channel_layout_default, grep swresample.h for swr_alloc_set_opts2, near-identical FATAL_ERROR) is now one function, pa_check_ffmpeg_51_api(FFMPEG_INC_DIRS), defined before the platform split. Each platform block keeps only its discovery (vendored import libs on Windows, pkg-config dual-naming on mac/Linux) and records FFMPEG_INC_DIRS; the shared block after the WIN32/else() split applies target_include_directories and runs the check, so the requirement comment now lives once instead of three times. Discovery stays per-platform; symbol-grepping instead of version numbers is unchanged. Net -21 lines. Verified: Linux pkg-config configure path, standalone success/failure test of the function, full suite green.

@pa-devbot

Copy link
Copy Markdown
Collaborator

CMake change for the new 3rdPartyBinaries/ffmpeg/ folder (6e39b23, built on your fb025a6):

  • Import libraries: the add_library(IMPORTED)/set_target_properties declarations for the 5 ffmpeg libs now sit in the early imported-library cluster (next to ONNX/dpp/discord), pointing at 3rdPartyBinaries/ffmpeg/*.lib, with no configure-time existence checks - the discord_social_sdk_win pattern. The old header/.lib checks are dropped; the shared pa_check_ffmpeg_51_api() call after the platform split still guards the header generation, and a missing .lib fails at link time (same as discord when its zip is missing).
  • DLLs: the deploy section at the bottom sets FFMPEG_ZIP/FFMPEG_DLL_DIR, extracts ffmpeg-n9.0-win64-lgpl-shared-9.0.zip into 3rdPartyBinaries/ffmpeg/ at configure time if no .dll is present yet (like the OpenCV extraction - I verified cmake -E tar xf on that zip: the 5 DLLs land at the folder root, and the no-DLLs glob guard makes re-extraction idempotent), then adds POST_BUILD copy_if_different commands for 3rdPartyBinaries/ffmpeg/*.dll - the top-level THIRD_PARTY_DLLS glob no longer sees them since they live in the subfolder.
  • mac/linux pkg-config wiring untouched. The old top-level .lib/.dll files stay until your follow-up deletion; until then they are simply unreferenced (the top-level glob will still copy the old DLLs during the transition, which is harmless - copy_if_different).

Linux configure + build + full suite green here; the Windows path needs CI/jw hardware.

@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from a5ac055 to 5a42691 Compare October 23, 2026 22:20
@pa-devbot

Copy link
Copy Markdown
Collaborator

Migrated StreamHistory onto the new audio pipeline (f72b51c, per the PR #1672 hand-off; built on your b8f186e):

  • StreamHistoryTracker_Ring now implements push_samples() and consumes the planar working stream directly. The tree's per-listener node delivers it already mixed + resampled to the ring's fixed 48 kHz / stereo attach shape, so the ffmpeg swresample (make_audio_resampler / SwrContext), the DUAL_44100 left-channel hack, and the input-rate plumbing are gone. ffmpeg only encodes + muxes now, as the hand-off prescribes.
  • StreamHistorySession attaches the working listener at that fixed 48 kHz / 2ch shape in the ffmpeg build (the hand-off's suggested shape) and forwards the planar data to the tracker without re-interleaving; non-ffmpeg builds keep the preset-shape attach + re-interleave for the SaveFrames tracker.
  • The legacy AudioFloatStreamListener path is dropped from stream history: the session no longer implements on_samples and ConsoleSystemSession no longer add/remove_stream_listener()s it, so the double-feed (legacy float + working stream) goes away.

Behavior change to be aware of: audio is now always recorded as 2ch AAC @ 48 kHz / 96 kbps - mono presets (MONO_48000, MONO_96000, DUAL_44100) upmix to dual-mono in the tree's mixer instead of recording mono AAC (previously 1ch / 64 kbps).

Two follow-ups for later: CMake still links + checks swresample (unused by code now, and it's one of the 5.1 API markers the check greps); and the pipeline-side legacy float path (AudioSession::add_stream_listener etc.) now has zero consumers, so the hand-off's removal checklist can run in its own cleanup PR.

Verified: full command-line suite 18,272 passed / 0 failed / 3 skipped (the skips are the decode-backend tests, unavailable in this container), on the rebased tip.

@pa-devbot
pa-devbot force-pushed the stream-history-ffmpeg-ring branch 2 times, most recently from f7d7790 to 866ed89 Compare October 24, 2026 21:13
claude and others added 27 commits November 11, 2026 20:25
h264_annexb_to_avcc() and h264_avcc_from_annexb_extradata() only work on
byte buffers (no ffmpeg, Qt or tracker state), so they now live in
StreamHistory_H264.h/.cpp. That makes them testable and lets them build in
every configuration, not just with PA_STREAM_HISTORY_FFMPEG. No behavior
change: the function bodies are moved verbatim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Table-driven tests for h264_annexb_to_avcc() and
h264_avcc_from_annexb_extradata() that check the exact output bytes:
3- and 4-byte start codes, several NAL units, leading garbage, inputs with
no NAL unit, extra and repeated NAL types in the extradata, and missing or
truncated parameter sets.

Three cases fail on this commit and are fixed by the next one: a zero byte
of a following 4-byte start code is kept as part of the previous NAL unit,
and an extradata without a PPS produces an avcC that claims one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- A zero byte belonging to a following 4-byte start code (00 00 00 01)
  was kept as the last byte of the previous NAL unit, so the AVCC length
  prefixes and the SPS/PPS in the avcC record carried a stray trailing
  zero. A NAL unit never ends in a zero byte, so trailing zeros are now
  left with the start code.
- An extradata without a PPS produced an avcC record claiming one PPS of
  length 0. It now records zero picture parameter sets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- push_samples() overload taking the capture time of the last sample, so
  tests can drive audio and video on one simulated timeline. The existing
  overload passes current_time(), so production behavior is unchanged.
- wait_until_idle(): blocks until every queued video frame has been
  encoded. on_frame() only queues frames for the encoder thread (dropping
  the oldest beyond a small backlog), so tests wait on this between frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
In PA_STREAM_HISTORY_FFMPEG builds, StreamHistory_Tests.cpp now also feeds
StreamHistoryTracker_Ring synthetic frames (black, then white) and audio
(silence, then a 1 kHz tone) on a simulated timeline, saves to MP4/MKV, and
reads the file back with ffmpeg. Covered: stream layout, a keyframe first,
strictly increasing timestamps, file length, window eviction, A/V sync
(also after a one-second audio gap), saving with nothing recorded or to an
unwritable path, destruction with frames still queued, and saving from
another thread while recording. Files go under DEBUG_PATH() and are
removed when the test passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
clear() and initialize() ran under the session's spin lock, so every input,
resolution or startup/shutdown change constructed the new tracker (opening
the AAC encoder, starting the encoder thread) and destroyed the old one
(joining its encoder thread, which may be opening a hardware encoder) while
the audio and video threads spun on that lock in push_samples() and
on_frame().

The lock is now held only to read the settings and to swap the tracker
pointer; the old tracker is destroyed after the lock is released. A
generation counter, bumped on every settings change and on clear(), keeps
a tracker built from stale settings from replacing a newer one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
The MP4 path converts Annex B H.264 to AVCC only when the extradata starts
with a 4-byte start code. Extradata starting with a 3-byte start code
(00 00 01) skipped the conversion and reached the MP4 muxer as Annex B.
The check now accepts both, and is limited to H.264: MPEG-4 Part 2
extradata also starts with a 00 00 01 start code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
…changes

- Save and sync from NV12 frames (the common capture card format, scaled
  directly from its planes; the other tests use RGB32).
- Sync with audio delivered 0-50 ms late: no re-sync may happen.
- Sync with the audio device clock running 0.1% fast or slow over 200
  simulated seconds: the re-sync keeps the audio within its threshold.
- Input resolution change while recording (RGB32 and NV12).
- The sync tests without a gap now also check that the audio packets are
  evenly spaced, which catches a re-sync that should not have happened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- When the audio device clock runs fast, the re-sync moves the audio
  timeline about 100 ms back, so the following packets overlap audio
  already written. The muxer bumped them to strictly increasing
  timestamps, squeezing them into near-zero-length packets (an audible
  glitch). It now drops the overlapping packets and logs how many.
- The 90 MB memory cap shortened saves silently (about 175 s of history at
  the Very High quality setting). The first time it drops packets the
  window would have kept, the tracker now logs it once, outside the lock.
- Dropped frames (the encoder falling behind) are reported at most every
  10 seconds with a count, instead of one red log line per frame.
- The ffmpeg log level is process-wide; it is now set once, by the first
  tracker, instead of being reset by every tracker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Every sync test now also checks that no two audio packets are closer than
one AAC frame, and a new test (device clock 0.1% fast, 110 s, saved window
covering the re-sync at about 100 s) exercises the overlap case directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
…version log

- The Windows DLLs were extracted into 3rdPartyBinaries/ffmpeg/ only when
  no .dll was there yet. Since the DLLs are git-ignored, a machine that had
  extracted them once kept them after the repo moved to a new zip and new
  import libraries: stale DLLs with the same names fail to load ("entry
  point not found"), and ones with new names are never extracted. The zip
  is now extracted into a folder named after it, completed extractions are
  marked with a stamp file (an interrupted one is redone, a failed one is
  a configure error), and changing the zip re-runs the configure step.
- The configure log printed a blank ffmpeg version: with several modules,
  pkg_check_modules() sets <prefix>_<module>_VERSION, not
  <prefix>_VERSION. It now prints the libavcodec version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
PA_STREAM_HISTORY_FFMPEG is on by default and configure stops without the
ffmpeg development packages, but the macOS and Ubuntu x64 guides did not
install them. Add them (with how to opt out on Ubuntu), and say in the
ARM64 guide that the packages it already installs are now needed for the
stream history too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
Revert the per-version extraction folder, stamp file and configure
dependency on the zip: the DLLs are again extracted once into
3rdPartyBinaries/ffmpeg/ when missing, mirroring the OpenCV DLLs. The fix
for the blank ffmpeg version in the configure log is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- The AVCodec.pix_fmts / avcodec_get_supported_config() switch was gated on
  libavcodec >= 61, but the function was added in libavcodec 61.13
  (ffmpeg 7.1); ffmpeg 7.0 (61.3) failed to compile. Gate on 61.13.100.
- Drop a frame whose millisecond pts would not advance instead of clamping
  it to a repeat, which mpeg4 rejects as a hard error (and so would
  trigger an encoder fallback). Unreachable with the current throttling.
- Replace qt_pixel_format_to_av_pix_fmt() with an NV12 check: only its
  NV12 result was used, and its RGB mappings were wrong.
- Comments: describe the thread ownership and what m_lock guards, the
  audio clock's delivery-time assumption, encoder fallback on send or
  receive failures, the encoders that emit Annex B vs AVCC, the
  keyframe-headroom guarantee (give or take a frame, memory cap aside),
  and the dts bump; fix stale references (Impl, the resampler, the
  ffmpeg version range, the video time base, the bitrate clamp).
- Use the Mutex wrapper for the poisoned-encoder set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
- CMake: the ffmpeg 5.1 check no longer claims the ring calls
  swr_alloc_set_opts2() (the audio pipeline resamples; libswresample is
  still required and linked), and the version note now says libavcodec
  cannot tell 5.0 from 5.1 (both 59), not 4.x from 5.x.
- Session: start() and m_audio_choice no longer describe nonexistent
  parameters or a shape the ffmpeg ring ignores, and
  attach_working_listener() states the fixed 48 kHz stereo attach.
- Ring tracker: document on_frame(), save() and set_window(); drop the
  single-channel pass-through case from push_samples() (the attach is
  always stereo); list the members each thread owns by name rather than
  by prefix (m_last_* and m_audio_* also matched lock-guarded members).
- H.264 helpers, tests: tidy the start-code scan comments; describe what
  the sync tolerance covers (AAC pre-echo), the Feeder's non-default
  delivery, and both bounds of the eviction test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B7TXvAbVhXAdjGKZbrMgT9
sws_getContext() fails the same way for the same input/output shape, so
retrying on every frame only logged an error at the frame rate. Disable
video like the other unrecoverable setup failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
The "could not convert video frame" flag was never reset, so after a
recovery a later failure went unlogged. Reset it once a frame converts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
- Video: an encoder that refuses a frame (EAGAIN) with no packet to drain
  is stuck, so fall back to the next candidate instead of dropping every
  frame silently. A packet allocation failure no longer looks like
  "nothing to drain".
- Video: log the per-frame skips (map, frame allocation, scaling) once per
  failing stretch; they used to be silent. They share one flag with the
  RGB conversion failure, cleared when a frame reaches the encoder.
- Audio: stop encoding once a receive error disables audio (the loop kept
  feeding the failed encoder), and disable audio with a clear message when
  a packet cannot be allocated instead of failing later with EAGAIN.
- Log a failed codec parameter snapshot and drop it so the next packet
  retries, instead of keeping a half-reset snapshot that breaks saves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
Every return path freed the AVFrame/AVPacket by hand, and an exception
(e.g. bad_alloc from the logger or the ring packet) leaked them. Use
FramePtr/PacketPtr, like the existing CodecParametersPtr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
encode_video_frame() unmapped the frame by hand before each early return,
and an exception (e.g. from the RGB conversion or the logger) left it
mapped. QVideoFrame is explicitly shared with reference-counted maps, so
that kept the frame mapped for every holder. ReadOnlyFrameMap maps in its
constructor, unmaps in its destructor, and keeps unmap() for the two
deliberate early unmaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
Add StreamHistory_FfmpegPtr.h: FfmpegPtr, a unique_ptr alias template
whose deleter calls an ffmpeg free function (taking the pointer or its
address, picked at compile time), the Frame/Packet/CodecParameters/
CodecContext/SwsContext aliases, an output format context owner (closes
the file, then frees), and an option dictionary owner.

mux_to_file() repeated its cleanup (free the options, close the file,
free the context) on nine return paths and leaked all three when an
exception escaped. Use the owners instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
m_video_context, m_audio_context, m_sws and the two codec parameter
snapshots were raw pointers freed in ~Data() and by hand wherever they
were replaced. ~Data() never runs when the constructor throws, so an
exception after init_audio_encoder() leaked the AAC context. Make them
FfmpegPtr members; ~Data() goes away, and the deliberate leak of a
failed hardware encoder in abandon_video_encoder() becomes an explicit
release(). The per-candidate context in init_video_encoder() is owned
too, so a throw from the open-failure log no longer leaks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014GzhHzE332dv7vKrCvfkqw
@jw098
jw098 force-pushed the stream-history-ffmpeg-ring branch from aa405c2 to 5b1c650 Compare November 12, 2026 03:59
@jw098

jw098 commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Replaced with PR #1907

@jw098 jw098 closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants