From a0958ec89e8696c95fe3e841f7ee674778eb8588 Mon Sep 17 00:00:00 2001 From: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com> Date: Thu, 22 Oct 2026 07:05:36 +0000 Subject: [PATCH 1/3] Docs: audio pipeline working-path hand-off for the PR #1563 session Hand-off doc for the session taking over PR #1563 (ffmpeg resampler for video output encoding): use the WorkingAudioStreamListener path (AudioSession::add_working_stream_listener, PR #1671) and the existing Kernels::AudioResampler instead of an ffmpeg audio resampler; documents the post-migration architecture, the working-stream data layout / shape semantics, resampler caveats, and the legacy float-path removal checklist to run once #1563 is off it. --- .../AudioPipelineWorkingPathHandoff.md | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 CodingAgentContext/AudioPipelineWorkingPathHandoff.md diff --git a/CodingAgentContext/AudioPipelineWorkingPathHandoff.md b/CodingAgentContext/AudioPipelineWorkingPathHandoff.md new file mode 100644 index 0000000000..c5375def96 --- /dev/null +++ b/CodingAgentContext/AudioPipelineWorkingPathHandoff.md @@ -0,0 +1,135 @@ +# Audio Pipeline — Working-Stream Path Hand-Off + +Hand-off for the session taking over **PR #1563** (ffmpeg resampler for video output +encoding), from the session that did the audio pipeline migration. + +## Read this first + +- **Do not pull in an ffmpeg audio resampler.** The audio pipeline already has a + kernel resampler (`Kernels::AudioResampler`, SIMD-dispatched, gate-tested) and a + channel mixer. Use them; let ffmpeg do only what it does for video (decode/encode/ + mux). +- Audio is exposed to consumers through a **`WorkingAudioStreamListener` path** + (added in PR #1671). Subscribe with + `AudioSession::add_working_stream_listener(listener)` / + `remove_working_stream_listener(listener)`. +- The legacy `AudioFloatStreamListener` path still exists but has **no consumers** + left. It is being kept in place (punted) until #1563 moves off of it. Once that + is done, remove it per the checklist at the bottom. + +## What's already in main (the migration) + +| PR | What | +|----|------| +| #1620 | Split `IO/AudioInputFile` / `IO/AudioInputDevice`; raw-listener roots on the reader. | +| #1623 | Pipeline tree built inside `AudioPassthroughPairQt` (decoder + sink/FFT branches). | +| #1647 / #1652 / #1654 | `WorkingAudioToFFT` node; FFT endpoint switched to the tree; legacy FFT path deleted. | +| #1655 | Sink gate tests (`SinkPipeline_Tests.cpp`): no-op bit-exactness, 1-LSB sint16, resample/mix invariants. | +| #1658 | Playback endpoint switched to the tree; `IO/AudioOutputDevice` (raw `RawAudioStreamListener` → `QAudioSink`); `AudioSink` deleted. Volume rides the sink encoder's multiplier (device opens at full volume). | +| #1665 | Sink opens in its **native** format (no more forced 44.1/48k presets); tree resamples/mixes/converts to it. | +| #1668 | Sink-side cleanups: dead per-output-device format enumeration removed (duplicate collapse now first-occurrence-wins per display name); `AudioFloatToStream` (float→raw) deleted; gate tests made self-contained. | +| #1671 | **This path**: `WorkingAudioStreamListener` listener API on the pair/session; `StreamHistorySession` now consumes it. | + +## Architecture + +``` +AudioSession (UI thread) ── audio() on ConsoleSystemSession +└─ AudioPassthroughPairQtThread → AudioPassthroughPairQt (event thread, SpinLock m_lock) + ├─ m_reader: AudioSource + │ ├─ raw stream → m_source_decoder (RawAudio2WorkingAudio) [tree root] + │ │ ├─ sink branch: [mixer/swap] → [resampler] → m_sink_encoder → m_sink_device (QAudioSink) + │ │ ├─ FFT branch: [mixer 0.5/0.5] → [resampler] → m_fft_node (WorkingAudioToFFT) → InternalFFTListener + │ │ └─ working tap: InternalWorkingListener → m_working_listeners ← THE NEW PATH + │ └─ float stream → SampleListener → m_stream_listeners (legacy, PUNTED, no consumers) + └─ m_fft_listeners (AudioSession itself is an FFTListener) +``` + +- The tree is rebuilt under `m_lock` whenever the source/sink changes; producers push + lock-free, and `ListenerSet::remove` blocks on in-flight pushes, so rebuilds never + race a live push. +- Threading: file input produces on the event thread (QTimer); device input on the + Qt audio backend thread. **Your listener callbacks run on whichever of those is + producing** — no extra hop. + +## The working-stream API (reference) + +```cpp +// AudioSession (thread-safe; safe to call from any thread) +void add_working_stream_listener(Kernels::AudioPipeline::WorkingAudioStreamListener&); +void remove_working_stream_listener(Kernels::AudioPipeline::WorkingAudioStreamListener&); +``` + +- Tap point: the source decoder's output — the tree's working float stream. +- Data layout: **split (planar) channels**. `push_samples(samples, stride, data)`: + channel `c` starts at `data[c * stride]` and runs `samples` floats. `stride` is the + per-channel buffer length (≥ `samples`); it is `0` for the single-channel float + pass-through. Interleaved: `out[i*ch + c] = data[c*stride + i]`. +- Shape = the source's **physical** rate/channels, with the **input volume already + applied**. For device presets: MONO_48000 → 48k/1ch, DUAL_44100 → 44.1k/2ch, + DUAL_48000 → 48k/2ch, MONO_96000 → 96k/1ch, INTERLEAVE_LR/RL_96000 → 48k/2ch + (the card's 1ch×96k reinterpreted as two 48k channels). File input currently + delivers 48k/2ch (the loader still forces it); a follow-up PR makes file input + native rate/channels — this path adapts automatically (the tap is rebuilt with the + tree), nothing for you to handle. +- The capture-card `INTERLEAVE_RL_96000` channel swap is **not** applied on this path + (physical channel order); `StreamHistorySession` applies it itself. +- The pair's listener set does not validate shapes (the per-tree tap node does), so a + persistent listener with a nominal `(rate, channels)` base is safe across input + changes — derive the live shape from `AudioChannelFormat` state callbacks if you + need it. + +## The resampler (you already have one) + +```cpp +// Kernels/AudioResampler/Kernels_AudioResampler.h +AudioResampler(Logger&, size_t in_rate, size_t out_rate, size_t channels, + size_t buffer_size = 4096 + 256, uint32_t taps = 32, double beta = 9.0); +// Is itself a WorkingAudioStreamListener: add_listener(WorkingAudioStreamListener&). +// validate_listener() matches (rate, channels) exactly. +``` + +- Window-sinc FIR, SIMD-dispatched; emits in output blocks. Gate-tested in + `SinkPipeline_Tests.cpp` (resample-up 44.1k→48k, decimate 96k→48k: output length + within ±2%/±1%, tone amplitudes at gain ≈1 bins). +- Caveats: block-buffered (first output lag is bounded), and input tail carry is + bounded by `m_carry_capacity` (worst case one kernel block's input span; up to 128 + output samples on x86 AVX-512) — a stream that ends abruptly can lose a bounded + tail. For continuous capture this is irrelevant; for file-ended streams pad the + input or accept it. +- Channel conversion: `Kernels::AudioPipeline::ChannelMixer(rate, in_ch, out_ch, + weight_matrix)`; `build_channel_mixer_matrix(QAudioFormat in, QAudioFormat out)` + (`Tools/AudioFormatUtils.h`) gives the standard matrix (mono→stereo is gain-1.0 + dual-mono; stereo→mono is 0.5/0.5). `SinkPipeline_Tests.cpp::run_tree()` shows the + exact construction/attachment pattern for chains of these nodes. + +**Suggested shape for #1563:** own a small chain in the encoding component — +your `WorkingAudioStreamListener` → optional `ChannelMixer` → `AudioResampler` +(source rate → your target rate) → your ffmpeg audio encoder input. These are plain +thread-affine kernel objects (no Qt, no locks needed if only one producer feeds +them), exactly as the gate tests use them. + +## Legacy float path — removal checklist (after #1563 is off it) + +The legacy reader still runs with no consumers; remove all of it in one cleanup PR: + +- `AudioStream.h/.cpp`: `AudioFloatStreamListener`, `AudioStreamToFloat`. +- `IO/AudioSource.{h,cpp}`: `m_reader`, `m_listeners`, `InternalListener`, + `add_listener(AudioFloatStreamListener&)`, `samples_per_frame()`, `m_multiplier`, + and the preset switch in `init()` (the raw-shape accessors + `raw_sample_rate()/raw_channels()/raw_sample_format()` stay). +- `Backends/AudioPassthroughPairQt.{h,cpp}`: `SampleListener`, `m_sample_listener`, + `m_stream_listeners`, the float `add/remove_listener` overloads (and every + `m_sample_listener.reset()` in the lifecycle paths). +- `AudioPassthroughPair.h` + `AudioPassthroughPairQtThread.{h,cpp}`: the float + listener overloads. +- `AudioSession.{h,cpp}`: `add_stream_listener` / `remove_stream_listener`. + +## Build & test + +- `pa-build` (CMake + Ninja, Qt 6.10.2 aarch64, `-Wall -Wextra -Wpedantic -Werror`; + incremental after the first build), then `pa-test .` (full command-line suite, + ~18.2k tests; exit 0 on success). +- The command-line suite does no real audio I/O: device/sink behavior is verified + `-Werror` + CI (x86) + live hardware. Gate tests + (`SinkPipeline_Tests.cpp`, `Spectrum/FFTStreamer_Tests.cpp`) pin the kernel math. +- Audio-tree source files are CRLF; keep them that way when editing. From ae1e4d4a75e6470c7e9d943117c9ba433082c382 Mon Sep 17 00:00:00 2001 From: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com> Date: Thu, 22 Oct 2026 17:44:20 +0000 Subject: [PATCH 2/3] Docs: update the working-path hand-off for the per-listener node API. The #1671 path is now attached with add_working_stream_listener(const AudioFormat&, listener): the pair builds a WorkingAudio2WorkingAudio per listener (mix + resample to the requested shape, RL preset swap applied in the matrix), so the #1563 session no longer owns a manual mixer/resampler chain. Updates the PR table (#1674, #1676, rebased #1671), the architecture diagram, the API semantics (fixed-at-attach shape, re-attach for preset-following shapes, entries survive tree rebuilds), the StreamHistorySession transition state (both interfaces, double-feed caveat), and the #1563 suggested shape (just attach with the target format). --- .../AudioPipelineWorkingPathHandoff.md | 132 +++++++++++++----- 1 file changed, 94 insertions(+), 38 deletions(-) diff --git a/CodingAgentContext/AudioPipelineWorkingPathHandoff.md b/CodingAgentContext/AudioPipelineWorkingPathHandoff.md index c5375def96..a7b106d3a8 100644 --- a/CodingAgentContext/AudioPipelineWorkingPathHandoff.md +++ b/CodingAgentContext/AudioPipelineWorkingPathHandoff.md @@ -10,12 +10,17 @@ encoding), from the session that did the audio pipeline migration. channel mixer. Use them; let ffmpeg do only what it does for video (decode/encode/ mux). - Audio is exposed to consumers through a **`WorkingAudioStreamListener` path** - (added in PR #1671). Subscribe with - `AudioSession::add_working_stream_listener(listener)` / - `remove_working_stream_listener(listener)`. -- The legacy `AudioFloatStreamListener` path still exists but has **no consumers** - left. It is being kept in place (punted) until #1563 moves off of it. Once that - is done, remove it per the checklist at the bottom. + (added in PR #1671, reworked onto #1674's all-in-one node). Subscribe with + `AudioSession::add_working_stream_listener(format, listener)` / + `remove_working_stream_listener(listener)`, where `format` is the shape you + want delivered (sample rate + channel count; the channel layout in + `format.native_format()` drives the channel mixing). The pair builds a + `WorkingAudio2WorkingAudio` node per listener (mix + resample, source shape + -> your shape), so you get the converted stream for free. +- The legacy `AudioFloatStreamListener` path still exists and still has one + consumer (`StreamHistorySession`'s old interface, kept next to the new one + until #1563 switches off it). Once that is done, remove it per the + checklist at the bottom. ## What's already in main (the migration) @@ -28,7 +33,9 @@ encoding), from the session that did the audio pipeline migration. | #1658 | Playback endpoint switched to the tree; `IO/AudioOutputDevice` (raw `RawAudioStreamListener` → `QAudioSink`); `AudioSink` deleted. Volume rides the sink encoder's multiplier (device opens at full volume). | | #1665 | Sink opens in its **native** format (no more forced 44.1/48k presets); tree resamples/mixes/converts to it. | | #1668 | Sink-side cleanups: dead per-output-device format enumeration removed (duplicate collapse now first-occurrence-wins per display name); `AudioFloatToStream` (float→raw) deleted; gate tests made self-contained. | -| #1671 | **This path**: `WorkingAudioStreamListener` listener API on the pair/session; `StreamHistorySession` now consumes it. | +| #1674 | `AudioFormat` (Pimpl wrapper over `QAudioFormat`) and `WorkingAudio2WorkingAudio` (all-in-one mix + resample node); `ChannelMixer` / `AudioResampler` moved to the `WorkingAudioNode` base. | +| #1676 | Tree simplified onto the all-in-one node (one `WorkingAudio2WorkingAudio` per branch, direct connection when no conversion is needed). Also fixed a latent OOB write in `ChannelMixer::push_samples` for pushes larger than its buffer (it now chunks internally). | +| #1671 (rebased) | **This path**: per-listener `add_listener(const AudioFormat&, WorkingAudioStreamListener&)` on the pair/session; `StreamHistorySession` now consumes it next to its legacy interface. | ## Architecture @@ -37,10 +44,10 @@ AudioSession (UI thread) ── audio() on ConsoleSystemSession └─ AudioPassthroughPairQtThread → AudioPassthroughPairQt (event thread, SpinLock m_lock) ├─ m_reader: AudioSource │ ├─ raw stream → m_source_decoder (RawAudio2WorkingAudio) [tree root] - │ │ ├─ sink branch: [mixer/swap] → [resampler] → m_sink_encoder → m_sink_device (QAudioSink) - │ │ ├─ FFT branch: [mixer 0.5/0.5] → [resampler] → m_fft_node (WorkingAudioToFFT) → InternalFFTListener - │ │ └─ working tap: InternalWorkingListener → m_working_listeners ← THE NEW PATH - │ └─ float stream → SampleListener → m_stream_listeners (legacy, PUNTED, no consumers) + │ │ ├─ sink branch: [m_sink_conv: WorkingAudio2WorkingAudio] → m_sink_encoder → m_sink_device (QAudioSink) + │ │ ├─ FFT branch: [m_fft_conv: WorkingAudio2WorkingAudio] → m_fft_node (WorkingAudioToFFT) → InternalFFTListener + │ │ └─ per working listener: [its WorkingAudio2WorkingAudio node] → internal forwarder → YOUR listener ← THE NEW PATH + │ └─ float stream → SampleListener → m_stream_listeners (legacy, one consumer: StreamHistorySession's old interface, PUNTED) └─ m_fft_listeners (AudioSession itself is an FFTListener) ``` @@ -55,30 +62,65 @@ AudioSession (UI thread) ── audio() on ConsoleSystemSession ```cpp // AudioSession (thread-safe; safe to call from any thread) -void add_working_stream_listener(Kernels::AudioPipeline::WorkingAudioStreamListener&); +void add_working_stream_listener( + const AudioFormat& format, + Kernels::AudioPipeline::WorkingAudioStreamListener& listener); void remove_working_stream_listener(Kernels::AudioPipeline::WorkingAudioStreamListener&); ``` - Tap point: the source decoder's output — the tree's working float stream. -- Data layout: **split (planar) channels**. `push_samples(samples, stride, data)`: - channel `c` starts at `data[c * stride]` and runs `samples` floats. `stride` is the - per-channel buffer length (≥ `samples`); it is `0` for the single-channel float - pass-through. Interleaved: `out[i*ch + c] = data[c*stride + i]`. -- Shape = the source's **physical** rate/channels, with the **input volume already - applied**. For device presets: MONO_48000 → 48k/1ch, DUAL_44100 → 44.1k/2ch, - DUAL_48000 → 48k/2ch, MONO_96000 → 96k/1ch, INTERLEAVE_LR/RL_96000 → 48k/2ch - (the card's 1ch×96k reinterpreted as two 48k channels). File input currently - delivers 48k/2ch (the loader still forces it); a follow-up PR makes file input - native rate/channels — this path adapts automatically (the tap is rebuilt with the - tree), nothing for you to handle. -- The capture-card `INTERLEAVE_RL_96000` channel swap is **not** applied on this path - (physical channel order); `StreamHistorySession` applies it itself. -- The pair's listener set does not validate shapes (the per-tree tap node does), so a - persistent listener with a nominal `(rate, channels)` base is safe across input - changes — derive the live shape from `AudioChannelFormat` state callbacks if you - need it. - -## The resampler (you already have one) +- The pair builds one `WorkingAudio2WorkingAudio` node per attached listener + (source shape -> `format`), plus an internal forwarder carrying the + requested shape as its nominal one (the node validates its listeners + against that; **your listener's own nominal (rate, channels) is never + validated** — it's metadata, pick anything sensible). +- Data layout: **split (planar) channels** in the requested shape. + `push_samples(samples, stride, data)`: channel `c` starts at + `data[c * stride]` and runs `samples` floats. `stride` is the + per-channel buffer length (≥ `samples`); it is `0` for the + single-channel float pass-through. Interleaved: + `out[i*ch + c] = data[c*stride + i]`. +- The requested shape is **fixed at attach time**. If the shape you want + follows the input preset (like the history session's), re-attach on input + change: `post_input_change` -> remove + add with the new format. A + consumer that requests one fixed shape (like a video encoder targeting + 48 kHz stereo) never has to. +- The input volume is already applied (it rides the source decoder). +- The capture-card `INTERLEAVE_RL_96000` channel swap **is applied** by the + pair (literal swap matrix in the node, same as the sink branch), when the + source is the RL preset and the requested shape is 2 channels. +- Re-adding a listener (e.g. with a new format) rebuilds its node; the + remove quiesces in-flight pushes first, so no push can run through a node + that is about to be destroyed. The entries survive source changes: + `build_tree()` rebuilds each node for the new source shape, so a + persistent listener keeps working across input changes. +- WAV file input is native rate/channels (#1673); compressed formats still + decode to 48k/2ch. Either way the node adapts automatically. + +## `StreamHistorySession` (state of the transition) + +It now implements **both** listener interfaces: + +- `AudioFloatStreamListener::on_samples` — the legacy pass-through, kept + exactly as it was. Attached via `add_stream_listener` in + `ConsoleSystemSession`. +- `WorkingAudioStreamListener::push_samples` — the new path. Attached via + `add_working_stream_listener(preset's physical shape, ...)` at + `start()`, re-attached in `post_input_change()` (old-shape attach removed + first, so no push in the old shape can race the shape update). It + re-interleaves the planar working stream at the active preset's physical + shape — the shape its tracker records at (the tracker's audio shape was + corrected in the same PR: DUAL_44100 now 2 ch @ 44.1 kHz, MONO_96000 now + 1 ch @ 96 kHz). + +While both are attached, the tracker is fed by **both** paths (double-record); +the legacy feed is shape-inconsistent with the corrected tracker shape. That +is inert today (the active tracker's audio is `#if 0`) — when #1563 switches +the video encoder over, drop the legacy attach (`add_stream_listener` / +`remove_stream_listener` in `ConsoleSystemSession`) and the double-feed goes +away with it. + +## The resampler & mixer kernels (if you ever own the chain yourself) ```cpp // Kernels/AudioResampler/Kernels_AudioResampler.h @@ -102,16 +144,30 @@ AudioResampler(Logger&, size_t in_rate, size_t out_rate, size_t channels, dual-mono; stereo→mono is 0.5/0.5). `SinkPipeline_Tests.cpp::run_tree()` shows the exact construction/attachment pattern for chains of these nodes. -**Suggested shape for #1563:** own a small chain in the encoding component — -your `WorkingAudioStreamListener` → optional `ChannelMixer` → `AudioResampler` -(source rate → your target rate) → your ffmpeg audio encoder input. These are plain -thread-affine kernel objects (no Qt, no locks needed if only one producer feeds -them), exactly as the gate tests use them. +## Suggested shape for #1563 -## Legacy float path — removal checklist (after #1563 is off it) +Just attach your encoder's listener to the requested shape — the pair does +the mixing and resampling for you: + +```cpp +AudioFormat target; +target.native_format().setSampleRate(48000); +target.native_format().setChannelCount(2); +target.native_format().setChannelConfig(QAudioFormat::ChannelConfigStereo); +audio_session.add_working_stream_listener(target, my_encoder_listener); +``` + +Your `WorkingAudioStreamListener` then receives 48 kHz / 2 ch planar float +regardless of the input source, and can feed the ffmpeg audio encoder +directly. No manual `ChannelMixer` / `AudioResampler` chain needed (those +kernels are what you'd use to own the chain yourself, e.g. for a custom +target layout). -The legacy reader still runs with no consumers; remove all of it in one cleanup PR: +## Legacy float path — removal checklist (after #1563 is off it) +The legacy reader still runs with one consumer (the history session's old +interface, which is dropped when #1563 switches); remove all of it in one +cleanup PR: - `AudioStream.h/.cpp`: `AudioFloatStreamListener`, `AudioStreamToFloat`. - `IO/AudioSource.{h,cpp}`: `m_reader`, `m_listeners`, `InternalListener`, `add_listener(AudioFloatStreamListener&)`, `samples_per_frame()`, `m_multiplier`, From 694ad73363df3babf80cee3914cad574e59285c6 Mon Sep 17 00:00:00 2001 From: PA Dev Bot <344001113+pa-devbot@users.noreply.github.com> Date: Thu, 22 Oct 2026 20:02:58 +0000 Subject: [PATCH 3/3] Docs: #1671 is merged; note #1673 is still in review. --- CodingAgentContext/AudioPipelineWorkingPathHandoff.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/CodingAgentContext/AudioPipelineWorkingPathHandoff.md b/CodingAgentContext/AudioPipelineWorkingPathHandoff.md index a7b106d3a8..af0af2f765 100644 --- a/CodingAgentContext/AudioPipelineWorkingPathHandoff.md +++ b/CodingAgentContext/AudioPipelineWorkingPathHandoff.md @@ -35,7 +35,7 @@ encoding), from the session that did the audio pipeline migration. | #1668 | Sink-side cleanups: dead per-output-device format enumeration removed (duplicate collapse now first-occurrence-wins per display name); `AudioFloatToStream` (float→raw) deleted; gate tests made self-contained. | | #1674 | `AudioFormat` (Pimpl wrapper over `QAudioFormat`) and `WorkingAudio2WorkingAudio` (all-in-one mix + resample node); `ChannelMixer` / `AudioResampler` moved to the `WorkingAudioNode` base. | | #1676 | Tree simplified onto the all-in-one node (one `WorkingAudio2WorkingAudio` per branch, direct connection when no conversion is needed). Also fixed a latent OOB write in `ChannelMixer::push_samples` for pushes larger than its buffer (it now chunks internally). | -| #1671 (rebased) | **This path**: per-listener `add_listener(const AudioFormat&, WorkingAudioStreamListener&)` on the pair/session; `StreamHistorySession` now consumes it next to its legacy interface. | +| #1671 | **This path**: per-listener `add_listener(const AudioFormat&, WorkingAudioStreamListener&)` on the pair/session; `StreamHistorySession` now consumes it next to its legacy interface. | ## Architecture @@ -94,8 +94,9 @@ void remove_working_stream_listener(Kernels::AudioPipeline::WorkingAudioStreamLi that is about to be destroyed. The entries survive source changes: `build_tree()` rebuilds each node for the new source shape, so a persistent listener keeps working across input changes. -- WAV file input is native rate/channels (#1673); compressed formats still - decode to 48k/2ch. Either way the node adapts automatically. +- WAV file input is native rate/channels (#1673, in review); compressed + formats still decode to 48k/2ch. Either way the node adapts + automatically. ## `StreamHistorySession` (state of the transition)