Skip to content

feat: stereo publish option, application playout delay, synchronous unbuffered audio capture - #13

Merged
tiredIsa merged 1 commit into
native-v2from
feat/stereo-playout-delay
Oct 11, 2026
Merged

tiredIsa merged 1 commit into
native-v2from
feat/stereo-playout-delay

Conversation

@tiredIsa

@tiredIsa tiredIsa commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

For syrnike13/syrnike13-app#157 (screen-share audio: crackling, lag, mono, audio/video sync).

Changes

  • Stereo publish option. TrackPublishOptions::stereo (FFI TrackPublishOptions.stereo = 14) adds TF_STEREO to the AddTrackRequest. The server answers the publisher and offers subscribers stereo=1. Before this, two-channel screen audio was encoded as mono Opus.
  • Application playout delay. livekit::setAudioPlayoutDelay(ms) (C ABI livekit_ffi_set_audio_playout_delay) is a process-wide value that the synthetic audio device returns as PlayoutDelay. WebRTC's ChannelReceive subtracts it from the audio playout position, so audio/video sync (RtpStreamsSynchronizer) accounts for audio the application renders itself after the SDK. It is clamped to 0..1000 ms.
  • Synchronous unbuffered capture. For queue_size_ms = 0 sources, AudioSource::captureFrame now sends CaptureAudioFrameRequest.synchronous = true. The FFI calls the new NativeAudioSource::capture_frame_direct in the calling thread, without a runtime task or completion callback. Previously, with every core busy, a realtime capture thread waited up to 440 ms for the tokio task and the FFI event thread, and 395 of 2,022 screen-audio packets were dropped in 20 s. Errors now surface as the call's own exception.

Validation (local Windows x64 build)

  • livekit_unit_tests 377/377, including the new TrackPublishOptionsTest.StereoRoundTrip and AudioSourceTest.UnbufferedCaptureCompletesInTheCallingThread.
  • cargo test -p livekit-ffi --lib conversion::room, including stereo_publish_option_round_trips. cargo fmt --check is clean.
  • 0001-native-v2-runtime.patch applies with git apply --check to a fresh 1a477bc4 checkout.
  • App call-bench against the project's LiveKit fork with sync_streams:
    • subscribers negotiate opus/2 stereo=1;
    • the screen-audio/video playout offset is 7–38 ms with 150 ms of injected sender skew;
    • with every core busy, the engine's screen audio has 0 drops and a maximum captureFrame of 0.12 ms (it was 440 ms).

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…nbuffered audio capture

- TrackPublishOptions::stereo sets TF_STEREO, so the server negotiates
  stereo Opus with the publisher and its subscribers.
- livekit::setAudioPlayoutDelay(): applications that render remote audio
  themselves report the delay they add after the SDK. The synthetic audio
  device returns it as PlayoutDelay, which WebRTC's audio/video sync
  subtracts from the audio playout position.
- AudioSource::captureFrame on an unbuffered source hands the frame to
  WebRTC in the calling thread (CaptureAudioFrameRequest.synchronous,
  NativeAudioSource::capture_frame_direct). It no longer waits for an async
  runtime task and an FFI callback, which stalled realtime capture threads
  for up to 440 ms under CPU load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tiredIsa
tiredIsa merged commit d86197b into native-v2 Oct 11, 2026
2 of 4 checks passed
@coderabbitai

coderabbitai Bot commented Oct 11, 2026

Copy link
Copy Markdown

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: bba2f4a6-ad75-408c-9226-f7fb3707e977

📥 Commits

Reviewing files that changed from the base of the PR and between 1719e9b and d5bf74e.


📒 Files selected for processing (9)
  • include/livekit/livekit.h
  • include/livekit/room_event_types.h
  • patches/client-sdk-rust/0001-native-v2-runtime.patch
  • src/audio_source.cpp
  • src/livekit.cpp
  • src/room_proto_converter.cpp
  • src/tests/unit/test_audio_source.cpp
  • src/tests/unit/test_room_event_types.cpp
  • src/tests/unit/test_video_frame_metadata.cpp

 _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
< I think a lot of new programmers like to use advanced data structures and advanced language features as a way of demonstrating their ability. I call it the lion-tamer syndrome. Such demonstrations are impressive, but unless they actually translate into real wins for the project, avoid them. - Glyn Williams >
 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR

🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

tiredIsa added a commit to syrnike13/syrnike13-app that referenced this pull request Oct 11, 2026
Stereo publish option, application playout delay and synchronous
unbuffered audio capture (syrnike13/client-sdk-cpp#13).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tiredIsa added a commit to syrnike13/syrnike13-app that referenced this pull request Oct 11, 2026
…nd A/V sync (#157)

* fix(native-media): send screen audio in direct capture mode

The screen-audio AudioSource used the SDK's buffered mode (10 ms queue),
which runs its own 10 ms WebRTC timer. Every late loopback packet made the
timer inject 10 ms of silence, and every burst blocked captureFrame (up to
98 ms in a real session), so the backlog guard discarded queued PCM. Viewers
heard crackling and up to ~120 ms of extra delay.

Use direct capture like the microphone: the loopback device clock paces
every 10 ms packet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): apply listener volume and mute per user id

The renderer keys participant and stream volume/mute by user id, but the
native mixer compared that value with the full LiveKit identity
("voice:v1|engine|instance|epoch|operation|userId"), so no control ever
matched: the context-menu volume, the stream volume slider and the stream
mute button had no effect in the desktop app.

Rename AudioMixSetting.identity to userId and match it against the user id
part of each Room identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep listener volumes across reloads and runtime replacement

The renderer's remote-audio revision restarts at 1 after a reload while the
main-process adapter drops settings older than the last one it saw, so
volume changes were ignored until the counter caught up. A runtime replaced
on account switch also started with no listener settings.

The voice service now owns the revision and replays the latest settings to
a replacement runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): ramp listener gain changes across one frame

Volume, mute and deafen changes stepped the gain between two 10 ms frames,
which clicks while a slider is dragged. Ramp each input's gain across the
frame; new and re-primed inputs fade in from silence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(voice): coalesce listener volume updates while dragging

Every pointer move of a volume slider wrote local settings to disk and sent
a full desired-state update to the native engine. Persist once the value
settles (300 ms, flushed on pagehide) and send at most one native mix
update per 40 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(livekit-server): group each sender's audio and video for sync by default

Enable room.sync_streams by default in the fork. Subscribers then see a
sender's screen video and screen audio (and camera and microphone) in one
WebRTC sync group, so receivers lip-sync them; without it WebRTC never
synchronizes the pair (estimatedPlayoutTimestamp stays empty). LiveKit
stops reusing subscriber transceivers in this mode. An explicit
`room.sync_streams: false` in livekit.yml still overrides it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): keep screen audio continuous, stereo and on time under load

- Queue is FIFO: the sender hands PCM to WebRTC synchronously (SDK
  syrnike.17), so a backlog after a stall drains at once. Discarding it
  stopped the RTP clock, and the sender reports then skewed screen audio
  against its video at every receiver (2.5 s after 395 drops under load).
- Gaps in capture timestamps are sent as silent frames (at most 50), so the
  RTP clock stays in step with capture time; `silence_filled` metric.
- Loopback capture and sender threads run in MMCSS "Audio", like the
  microphone path.
- Publish as stereo Opus (TF_STEREO); it was negotiated mono.

Under every-core load the engine's screen audio now drops 0 packets with a
0.12 ms maximum SDK call (was 395 drops, 440 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(native-media): synchronize screen video with screen audio at receivers

WebRTC's own lip sync (RtpStreamsSynchronizer, as in browsers and
Discord) runs once the server groups the streams (sync_streams). Feed it
correct inputs:

- Screen capture times reach the SDK in WebRTC's clock. On Windows WebRTC
  uses timeGetTime(), measured ~30 ms off QPC; a QPC time it saw as
  "future" was clamped to submit time, hiding capture/encode latency from
  the video sender reports.
- Remote playout reports the delay it adds after the SDK (ingress queue,
  jitter buffer, device) through livekit::setAudioPlayoutDelay, so the
  synthetic audio device no longer claims 0 ms and pictures wait for the
  sound actually heard. remote_audio_output_metrics.latency_ms is now that
  measured value.

Bench: a bot stamps its frames 150 ms early; viewers' playout offset is
7–38 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(native-media): bench stereo screen audio and audio/video sync

bench_bot publishes stereo screen audio (--screen-audio 1), can stamp
screen frames early (--screen-skew-ms N) and reports the watched sharer's
screen-audio codec and playout offset. call-bench uses both while viewing a
bot and while the engine shares the monitor with system audio, and reports
the engine's screen-audio sender counters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(native-v2): screen audio direct capture, stereo and A/V sync

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(native-media): pin LiveKit SDK v1.10.0-syrnike.17

Stereo publish option, application playout delay and synchronous
unbuffered audio capture (syrnike13/client-sdk-cpp#13).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
tiredIsa added a commit to syrnike13/syrnike13-app that referenced this pull request Oct 12, 2026
* refactor: migrate TypeScript runtime to Effect

* fix: harden voice media recovery

* fix(voice): restore and expand RTC telemetry

* fix: prevent long-running screen stream degradation

* fix: prevent VoiceStage viewer degradation

* fix: restore nightly native verification

* test: isolate remote video static frame verification

* fix(desktop): restore preload bridge on Windows

* fix(voice): republish microphone after server unmute

* fix(native): drop H264 AU until first complete SPS/PPS/IDR

Late subscribers were feeding partial access units into FFmpeg before
the first keyframe, which produced decoder corruption. Hold those
frames at the decoder wrapper until a complete keyframe arrives.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(native): stabilize Windows media contention soaks and texture leases

Keep Electron NT handles alive across probe recycle so the renderer cannot
import a recycled handle while a fence is still held, and count skipped
old-epoch releases as injected recovery without treating the hold as RPC
latency. CI contention and microphone soaks pass their assertions; elevated
ETW selection stays environment-blocked without Profile system performance.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(native): complete media stability validation

* test(native): wait for cooperative voice cancellation

* fix(native): avoid remote video stream churn on transient fences

* fix(native): harden H264 recovery and contention evidence

* fix(native): cancel screen cleanup at shutdown deadline

* test(native): drain renderer releases before probe shutdown

* test(native): sustain GPU rollover fault pressure

* fix(native): resume renderer flow beside pending GPU work

* fix(native): absorb bounded remote decode jitter

* fix(native): serialize retired D3D resource release

* fix(desktop): retire stale renderer releases with owner host

* test(desktop): account for bounded video queue drops

* test(native): warm production soak timer handle

* test(native): schedule microphone soak as pro audio

* test(native): bound microphone queue recoveries

* test(native): bound microphone runtime handle warmup

* test(desktop): report resilience addon ABI exactly

* test(native): bound utility host handles

* fix(native): bound display source visual probes

* fix(native-media): stabilize Windows validation

* test(desktop): make Windows CI contracts portable

* test(native): await contention renderer readiness

* test(native): align Windows smoke contracts

* fix(native): measure receiver audio playout coverage

* fix(desktop): close media reconcile completion race

* test(native): stabilize cleanup retirement races

* feat(desktop): establish native v2 boundary

Remove the legacy native media and vendored LiveKit implementation, retain the hook-only runtime, and add the unavailable engine boundary for issue #112.

* feat(native): bootstrap isolated media lifecycle boundary (#114)

* feat(native): define v2 media control protocol

Implements the bounded Electron/utility/N-API/C++ control, state, event, diagnostics, revision, and lifecycle contract for #115.

* feat(native): add LiveKit media lab (#116)

* docs: record Docker Desktop recovery notes

* feat(desktop): complete native v2 phase A boundary

* fix(media): harden native room boundary

* feat(native): add bounded Windows source enumeration (#117)

* feat(native): add bounded WGC monitor capture (#118)

* feat(windows-media): add WGC window capture lifecycle (#119)

* feat(native): add CPU screen publication reference (#120)

* refactor(native): harden Phase B boundaries

* feat(native): add GPU hardware H.264 screen sender (#121)

* fix(native-v2): address epic #112 review findings

* feat(native): add bounded remote video and Electron texture bridge (#122)

* build(native): pin published SDK and clear stale local overrides

* build(native): preserve canonical protocol bytes on Windows checkout

* test(native): distinguish stall settling from held GPU leases

* ci(native): separate GPU-required validation from hosted runner checks

* test(native): require actual SDK connection continuity in viewer acceptance

* fix: order native callback teardown before environment finalizers

* test: isolate voice navigation from asynchronous icon loading

* fix: use an allowed screen bitrate in the CPU reference lab

* fix: treat Room removal as terminal in the CPU reference lab

* feat: isolate local screen preview from publication (#123)

* ci: replace Blacksmith with GitHub-hosted runners

* fix: finalize terminal WGC capture only after frame leases drain

* fix: pin CPU screen oracle to software H264 for late receivers

* test: budget CPU screen lifecycle acceptance per cycle

* test: await decoded screen video before Room disconnect

* fix: drop WGC resize frames that exceed their actual surface

* feat(native-media): add bounded adaptive screen quality (#124)

* feat(native): add independent Windows screen audio (#125)

* feat(native-media): DXGI monitor capture and deterministic backend selection (#126) (#138)

* feat(native-media): add deterministic DXGI backend selection (#126)

* test(native-media): select the CI lab interface and preserve diagnostic evidence

* fix(native): drain minimized window frames before resize or delivery

* build(native): pin SDK with port-owned ICE sockets

* docs(native): record SDK 8 release and focused qualification

* docs(native): record actual UAC pause and recovery proof

* fix(native-media): pin SDK with joined negotiation shutdown

* docs(native-media): record accepted phase C qualification [skip ci]

* docs: update repository guidelines by removing outdated instructions and clarifying best practices

* fix(native-media): preserve screen preset during bitrate adaptation (#139) (#140)

* fix(native-media): keep screen preset fixed during bitrate adaptation (#139)

* fix: bound raw screen admission and preserve live encoder rate control

* test: exercise real encoder overload and static late subscription

* test(native-media): retain full candidate qualification diagnostics

* test(native-media): verify capture marker payload integrity

* build(native-media): pin released preencoded SDK fix

* test(native-media): qualify encoder pressure with bounded compute workload

* docs(native-media): retain focused SDK 11 qualification results

* docs(native-media): retain keyframe pacing candidate evidence

* build(native-media): pin released keyframe pacing SDK

* docs(native-media): retain published SDK 12 qualification results

* docs(native-media): retain rejected SDK transport diagnostics

* docs(native-media): record latency exception and final focused checks

* fix(native-audio): recover PCM freshness after short sender stalls

* test(native-media): retain audio recovery follow-ups and log lab termination

* test(native-media): isolate static scene from cursor composition

* docs(native-media): record final audio and fixed-preset qualification

* feat(native-v2): warm microphone capture, DSP and publication (#127) (#141)

* feat(native-media): add warm microphone capture and publication (#127)

* Pin released realtime microphone SDK for #127

* Document user waiver of physical microphone unplug test

* feat(native-media): bounded remote audio mixing and WASAPI output (#128) (#142)

* feat(native-media): add bounded remote audio mixing and WASAPI output (#128)

* build(native-media): pin verified LiveKit SDK 14 release

* feat(native-media): independent camera capture, preview and publication (#129) (#143)

* feat(native-media): add independent asynchronous camera pipeline (#129)

* fix(camera): drain preview GPU work and record published SDK validation

* docs(camera): verify combined Phase D build and publication

* fix(ci): select Media Lab IPv4 without adapter aggregation

* test(media-lab): report host pressure and failed ICE transports

* feat(native-media): cut Windows desktop over to Engine v2 (#130) (#144)

* feat(native-media): cut Windows product over to Engine v2 (#130)

* fix(native): preserve generated media model line endings

* test(native): await fatal event delivery before asserting retirement

* test(native): identify capture resource threads in CI

* fix(native-v2): finish bounded recovery acceptance

Merge PR #145 after green hosted CI. The user-approved resource/full-suite exception remains documented in the PR and Phase E matrix.

* chore(deploy): import per-project Caddyfile fragments

Другие проекты на этом сервере держат свои vhost'ы отдельными файлами в
/etc/caddy/sites (каталог монтируется через compose.override.yml, который
деплой исключает из архива). Строка import в основном Caddyfile позволяет
таким проектам жить независимо: выкат syrnike13 больше не сносит их домены.

* fix(native-v2): wait for window capture worker retirement

Hosted CI is green. Keep the strict zero-growth resource budget while allowing asynchronous Windows thread-pool retirement to settle before the final probe sample.

* fix(deploy): use pinned MinIO images from Quay (#147)

* fix(ci): skip absent hosted media lab report (#148)

* fix(native-media): stabilize local voice and screen sharing

* fix(native-media): make camera watchdog and WGC probe deterministic

* fix(native-media): make microphone failures recoverable and diagnosable

- Fall back from WASAPI RAW mode when the driver rejects it; RAW was the
  default and made capture fail on every device for affected drivers.
- Treat MMCSS registration as an optional scheduling hint.
- Fall back to the default input when the selected device is missing and
  report it as a path warning.
- Report specific microphone failure codes with the platform HRESULT and
  add capture state, failure, HRESULT, RAW and MMCSS to microphone metrics.
- Configure the startup voice runtime on first login instead of replacing
  the media utility process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(native-media): remove media lab, probes and fault-injection harness

Drop the experimental Media Lab (native src/lab, media_probe,
packages/native-media-lab, desktop preview/video-bridge labs and
product-fault harness), test-only fault gates and probe hooks from
production sources, acceptance JSON dumps from docs/native-v2 and the
CI steps that ran them. Unit tests keep their fixtures under
native/tests/support.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(voice): disable WASAPI RAW microphone capture by default

RAW capture bypasses driver microphone processing and is rejected by
many drivers. Desktop settings v4 turns it off once for existing
installs; a later explicit opt-in is preserved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(desktop): drop dead v1 diagnostics and journal only transitions

- Incident classification lists only events the media runtime records;
  v1 event names and their filters are removed.
- Remove never-called recordNativeDiagnosticMetrics and its metric names.
- Voice snapshots and per-track states are journaled only when they
  change, instead of on every meter/revision update.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(desktop): decode media command results once

The utility host dispatches commands through one handler table and
forwards addon results; the supervisor decodes each reply once against
its command-specific schema instead of three times.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(native-media): emit media metric samples through one helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(native-media): push video frames instead of polling on timers

Frame producers (remote tracks, camera and screen previews) wake the
export lane as soon as a frame is available, and the addon notifies the
utility host through a coalesced frames-ready callback. Main pulls
frames on that signal and whenever releases are queued, instead of a
16 ms timer stacked on an 8 ms native poll. The main timer now only
paces inventory and retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(native-media): describe pushed frames and microphone capture policy

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: skip native test suites in release pipelines

Release workflows rebuilt and re-ran the full Windows native test suite
(ctest, ASan, 50-cycle Electron smoke) that pull requests already run.
Timing-sensitive tests there flaked on shared runners and blocked desktop
packaging. Release callers now pass skip_tests to build, verify, and
stage native artifacts only; pull requests keep the full suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(native-media): device-clocked audio, lossless microphone and stable screen sharing (#149)

* fix(native-media): keep echo cancellation and live state across calls

- Renderer epochs are process-unique and a new render timeline keeps AEC
  active, so echo cancellation no longer stops after the first call or an
  output device switch.
- Echo references are a short FIFO instead of latest-wins, so frames rendered
  between two capture frames still reach the canceller.
- Voice gate holds 300 ms, fades on close, and drives the speaking indicator.
- Microphone reports Starting only when capture or publication starts.
- LiveKit statistics are sampled every 150 ms instead of every 20 ms.
- MMCSS is optional for remote audio output and mixing.
- Video frames and inventory are fenced by renderer and Room session, not by
  the global desired-state revision bumped by mute/PTT/volume.
- A transient desired-state failure is retried, so a mute cannot be lost.
- The utility restart budget is replenished after 60 s of stable host uptime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(native-media): one device-clocked playout thread and lossless audio

Remote audio: replace the SDK reader thread (5 ms polling), the timer-driven
mixer worker, the output owner and the WASAPI render worker with one
RemoteAudioPlayout thread. The output device event clocks reading, mixing,
rendering and the echo reference; per-track jitter buffers prime 20 ms,
re-prime after underruns and trim drift. Under full CPU load SDK drops fell
from 1104 to 0 frames per 15 s and voice CPU from 14.8% to 6-8% of a core.

Microphone: PCM ports are bounded FIFOs drained on every wake instead of
latest-wins slots, the sender runs under MMCSS, and the LiveKit source is
unbuffered as the SDK recommends for live capture. Under load every captured
frame is now sent (previously ~15-22% were lost).

Screen: pacing a 1080p keyframe no longer counts as congestion (150 ms
packet-queue threshold instead of 30 ms), and the hardware encoder uses a
60 s GOP so viewers are not stalled by an IDR every second (keyframes per
14 s: 15 -> 3, viewer gaps over 100 ms: 13 -> 1, 22.8 -> 30.1 fps).

Reconnect: remote audio readers and video decoders survive SDK
reconnection. Speaking indicators poll inventory at 10 Hz during a call.

Tests: drop the encoder fault-injection suite (217-650 s, fails on a known
driver handle leak) and the 100x repeat/handle accounting harness; the native
suite now runs in 25 s. Add bench_bot and bench/call-bench.mjs for local
end-to-end measurements against a dev LiveKit server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(native-media): recover microphone failures automatically

A retryable microphone input or publication failure is retried every two
seconds, recreating a retired pipeline, instead of waiting for a manual
Retry. The call benchmark gains device switching, a delayed signaling proxy
and fatal-failure capture. Microphone and remote audio docs describe the
current design instead of removed lab tooling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): leave a channel instantly on a dead network

LiveKit's graceful Room disconnect blocks for over 30 s when signaling is
dead. The transport waited for it, so leaving or moving after a network loss
hit the 2 s operation deadline and the fatal room_operation_unresponsive path
restarted the whole media process. Observers and publications are already
released when the transport disconnects, so the Room is now handed to a
teardown lane and the Engine leaves at once (66 ms; previously a fatal
failure after 2 s). At most four stuck teardowns are retained; beyond that
the previous in-place disconnect and its deadline apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): fall back to WGC when desktop duplication fails

Monitor sharing forced DXGI desktop duplication, so it could not start where
duplication is unavailable (hybrid GPUs, some remote sessions) or after it
failed repeatedly. DXGI is now the preferred backend with the existing WGC
fallback. WGC sessions request a borderless capture where Windows supports
it. The benchmark also shares an animated fixture window over WGC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): trim media lifecycle smoke repetitions

The process-guard check ran 100 warmups and 100 measured repetitions and
the host lifecycle 50 cycles; 10, 25 and 10 keep the same coverage of every
path. Electron-integration resource deltas were already informational.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(native-media): remove the unused CPU screen reference path and stale docs

CpuScreenConverter was only reachable from its own reference test; production
screen sharing converts on the GPU. Rewrite the architecture overview around
the current threads, failure rules and the call benchmark, update resource
ownership for the playout and teardown lanes, and drop the historical SDK
blocker log and a link to a removed baseline document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(native-media): report expected microphone frames in the call benchmark

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): fall back to the default output when the selected one is gone

Jitter counters of released readers are retained so totals never go back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): remote audio playout stack overflow under ASan (#150)

* fix(native-media): keep remote audio readers off the playout thread stack

The 64 readers hold ~1 MB of jitter buffers, the whole default thread
stack. Release fit by a margin; the ASan Debug product-runtime test
overflowed it. Allocate them once on the heap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(native-media): build the shared texture pool test again

It went out with the media lab block in daa37f0, but it covers the
product SharedTexturePool behind the Electron remote video bridge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(native-media): drop evidence logs of the removed media lab

The native-v2 docs still described how to run native-media-lab,
media_probe and build:lab, linked acceptance JSON reports that are no
longer in the repository and recorded one-off qualification runs. Keep
the contracts, point verification at the CTest targets and the call
benchmark that exist, and record the final benchmark numbers for the
microphone and remote audio. The DXGI doc now says the product prefers
DXGI with a WGC fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* chore: remove unused web and desktop code (#151)

* chore(web): remove modules nothing imports

Only their own tests referenced them: the old friends view, the
renderer PCM jitter worklet from before native playout, screen share
tuning/limits helpers, voice operation, gate calibration, preference
effects, participant media, transition rate limit and watch helpers,
the skeleton component, the message-format barrel and the unused
React Query devtools panel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(web,desktop): remove declarations nothing references

Exported functions, constants and types with no reference anywhere in
the repository, tests included: unused promise wrappers next to the
Effect requests the app calls, authorization helpers, selectors,
mention helpers, icons, desktop contract types. Imports and private
helpers left unused by that are removed as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(web,desktop): drop promise wrappers only tests called

Tests now run the Effect the app uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: describe desktop-native and windows-media-engine in the repository map

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(web,admin): pin TanStack versions and drop unused devtools

`latest` specifiers made every package.json change silently upgrade
router, start, query and form; they are pinned to the versions the
lockfile already resolved. The four devtools packages had no import.
Ignore local build-* scratch directories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(desktop): run input and overlay hooks in the main process (#152)

Hotkeys and overlay foreground detection ran in two utility processes,
each with a supervisor, handshake, host epochs, restart backoff, an
artifact manifest check and its own copy of the addon. The hooks
already run on the addon's message-loop thread, so the main process
only relays events; Discord loads its hooks in-process the same way.

- desktop-native builds one syrnike_hooks.node with createHooksRuntime;
  its runtime already served hotkeys and overlay together.
- main/native-hooks.ts loads it and correlates replies; hotkeys and the
  overlay detector keep their calls. A hook runtime error releases held
  hotkey actions, as runtime loss did before.
- Removed the hooks supervisor, utility adapter, contract, artifact
  verification, utility hosts and their tests.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): hide speaking indicators while deafened (#153)

A deafened listener hears nobody, so remote speakers are no longer
reported as speaking. Detection keeps tracking the noise floor, so the
indicators are accurate again as soon as deafen is lifted.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): screen share — borderless, static-screen keyframes, source aspect ratio, receive slots, real window list (#154)

* fix(native-media): borderless capture, static-screen keyframes, no cropping, enough receive slots

- Request GraphicsCaptureAccess Borderless before IsBorderRequired(false):
  Windows ignored the flag, so window shares and thumbnails had a yellow border.
- Re-encode a GPU copy of the newest capture after 250 ms without a new one.
  A viewer who joined a static window got no frame until it changed.
- Letterbox sources of another aspect ratio instead of center-cropping them.
- Remote video pool: 12 slots instead of 4, reuse same-size textures first,
  and return a quarantined slot once its timed-out GPU write completes.
- Bench: --port, a late viewer of a frozen window, firstFrameMs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(native-media): stream follows the source aspect ratio; list only real windows

- The preset is a box: the encoded size keeps the source's aspect ratio
  (never upscaled, multiples of 8). The first frame sets it before publish;
  a later size change that holds for 500 ms replaces the converter and
  encoder on the same publication, so viewers keep their track.
- Source enumeration skips tool windows, owned windows without
  WS_EX_APPWINDOW, and hidden/cloaked windows never seen shown
  (258 candidates on this desktop, 13 real; the list was truncated).
- Bench: a shared window turns portrait while a viewer watches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): Discord-like capture details

- Closing the shared window ends the share (source_closed/source_replaced
  turn the screen intent off) instead of a failed path and an error banner.
- WGC MinUpdateInterval at half the stream frame interval: a 165 Hz display
  no longer delivers 165 frames a second for a 30 fps stream.
- Skip content-protected windows (display affinity), which capture as black.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(native-media): recover stale window capture; paused stream while minimized

- Like Discord: a shown window without frames for 2 s is asked to repaint
  (asynchronous invalidate); no frame 1 s later recreates the frame pool and
  session (at most every 10 s, no leased frames). Recreating only the session
  on the old pool left the capture dead.
- A minimized/hidden shared window mutes the screen track and stops repeats;
  restoring unmutes and requests a keyframe. Native viewers get `muted` in the
  video inventory; viewers show "Трансляция на паузе" instead of a frozen frame
  (LiveKit browser viewers through the same track mute).
- Bench/fixture: `stall` ignores repaints; the bot counts mute/unmute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* feat(native-media): decode remote H.264 on the GPU (#156)

* feat(native-media): decode remote H.264 on the GPU

Remote H.264 is decoded by the Media Foundation MFT through DXVA on the
process D3D11 device (any GPU vendor), converted NV12 -> BGRA by the D3D11
video processor and copied into the shared texture pool on the GPU. The
SDK's external decoder hook keeps WebRTC's FFmpeg decoder as the fallback
for machines without DXVA. Pool writes wait on a D3D11 fence instead of
spinning with the device context locked.

Also fixes the frame protocol: since the pool grew to twelve slots, frames
in slots 4-11 failed the release/exported-frame bounds, invalidating the
whole frames reply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(native-media): pin LiveKit SDK v1.10.0-syrnike.16 (external video decoder)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): stream/participant volume, screen audio crackle, stereo and A/V sync (#157)

* fix(native-media): send screen audio in direct capture mode

The screen-audio AudioSource used the SDK's buffered mode (10 ms queue),
which runs its own 10 ms WebRTC timer. Every late loopback packet made the
timer inject 10 ms of silence, and every burst blocked captureFrame (up to
98 ms in a real session), so the backlog guard discarded queued PCM. Viewers
heard crackling and up to ~120 ms of extra delay.

Use direct capture like the microphone: the loopback device clock paces
every 10 ms packet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): apply listener volume and mute per user id

The renderer keys participant and stream volume/mute by user id, but the
native mixer compared that value with the full LiveKit identity
("voice:v1|engine|instance|epoch|operation|userId"), so no control ever
matched: the context-menu volume, the stream volume slider and the stream
mute button had no effect in the desktop app.

Rename AudioMixSetting.identity to userId and match it against the user id
part of each Room identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep listener volumes across reloads and runtime replacement

The renderer's remote-audio revision restarts at 1 after a reload while the
main-process adapter drops settings older than the last one it saw, so
volume changes were ignored until the counter caught up. A runtime replaced
on account switch also started with no listener settings.

The voice service now owns the revision and replays the latest settings to
a replacement runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): ramp listener gain changes across one frame

Volume, mute and deafen changes stepped the gain between two 10 ms frames,
which clicks while a slider is dragged. Ramp each input's gain across the
frame; new and re-primed inputs fade in from silence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* perf(voice): coalesce listener volume updates while dragging

Every pointer move of a volume slider wrote local settings to disk and sent
a full desired-state update to the native engine. Persist once the value
settles (300 ms, flushed on pagehide) and send at most one native mix
update per 40 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(livekit-server): group each sender's audio and video for sync by default

Enable room.sync_streams by default in the fork. Subscribers then see a
sender's screen video and screen audio (and camera and microphone) in one
WebRTC sync group, so receivers lip-sync them; without it WebRTC never
synchronizes the pair (estimatedPlayoutTimestamp stays empty). LiveKit
stops reusing subscriber transceivers in this mode. An explicit
`room.sync_streams: false` in livekit.yml still overrides it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(native-media): keep screen audio continuous, stereo and on time under load

- Queue is FIFO: the sender hands PCM to WebRTC synchronously (SDK
  syrnike.17), so a backlog after a stall drains at once. Discarding it
  stopped the RTP clock, and the sender reports then skewed screen audio
  against its video at every receiver (2.5 s after 395 drops under load).
- Gaps in capture timestamps are sent as silent frames (at most 50), so the
  RTP clock stays in step with capture time; `silence_filled` metric.
- Loopback capture and sender threads run in MMCSS "Audio", like the
  microphone path.
- Publish as stereo Opus (TF_STEREO); it was negotiated mono.

Under every-core load the engine's screen audio now drops 0 packets with a
0.12 ms maximum SDK call (was 395 drops, 440 ms).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(native-media): synchronize screen video with screen audio at receivers

WebRTC's own lip sync (RtpStreamsSynchronizer, as in browsers and
Discord) runs once the server groups the streams (sync_streams). Feed it
correct inputs:

- Screen capture times reach the SDK in WebRTC's clock. On Windows WebRTC
  uses timeGetTime(), measured ~30 ms off QPC; a QPC time it saw as
  "future" was clamped to submit time, hiding capture/encode latency from
  the video sender reports.
- Remote playout reports the delay it adds after the SDK (ingress queue,
  jitter buffer, device) through livekit::setAudioPlayoutDelay, so the
  synthetic audio device no longer claims 0 ms and pictures wait for the
  sound actually heard. remote_audio_output_metrics.latency_ms is now that
  measured value.

Bench: a bot stamps its frames 150 ms early; viewers' playout offset is
7–38 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(native-media): bench stereo screen audio and audio/video sync

bench_bot publishes stereo screen audio (--screen-audio 1), can stamp
screen frames early (--screen-skew-ms N) and reports the watched sharer's
screen-audio codec and playout offset. call-bench uses both while viewing a
bot and while the engine shares the monitor with system audio, and reports
the engine's screen-audio sender counters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(native-v2): screen audio direct capture, stereo and A/V sync

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(native-media): pin LiveKit SDK v1.10.0-syrnike.17

Stereo publish option, application playout delay and synchronous
unbuffered audio capture (syrnike13/client-sdk-cpp#13).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(deploy): run MinIO from the server's local images (#158)

MinIO no longer publishes images: Quay answers 401 and Docker Hub removed
minio/minio, so every nightly deploy failed at `docker compose pull`. The
server already has the exact images both stacks run. Reference them by
their local docker.io digests with `pull_policy: never`, so pull skips them
and up reuses them. Verified on the server: `compose config` passes and
`compose pull minio createbuckets` reports "Skipped" for nightly and
production. The image IDs do not change.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(deploy): pin MongoDB 8.2.12; unpinned mongo moved to 9.0 (#159)

The unpinned docker.io/mongo now resolves to MongoDB 9.0.2. Its mongod
exits with code 62 on our data (featureCompatibilityVersion 8.2), so the
nightly deploy recreated the database container into a restart loop.
Pin mongo:8.2.12 by its index digest (e0ce8c35…), the exact image
production runs, in both compose files. Production would have hit the same
failure on its next deploy.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(deploy): drop MinIO; files live in external S3 (#160)

MinIO no longer publishes images, so a fresh server could not start it.
Production already stores files in an immers.cloud bucket and never used
its MinIO container; nightly now uses its own bucket there. The compose
files no longer run minio or createbuckets; secrets.env.example documents
the [files.s3] settings a new instance needs.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): publish the microphone at the channel's voice bitrate (#161) (#195)

The channel setting `voice.audio_bitrate_kbps` (8-96 kbps) was saved but never
used: the browser always published at 64 kbps and the desktop engine left
the Opus bitrate at the SDK default.

- Voice desired state carries `microphoneBitrate`; the renderer sets it from
  the channel before joining and whenever the channel or its setting changes
  (`setMicrophoneBitrate`). DM and group calls keep 64 kbps.
- Browser: published with that bitrate; a change retargets the sender with
  `RTCRtpSender.setParameters`, without republishing.
- Desktop: `MicrophoneIntent.bitrate` reaches the native sender as
  `audio_encoding`. The SDK fixes Opus at publication, so a change
  republishes the microphone once.
- call-bench: a peer counts the engine's microphone republishes; bench_bot
  `--mic-bitrate` reports the encoder target the SDK applies.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* test(voice): measure RED/FEC on native microphones in the call bench (#171) (#197)

bench_bot reports, for a watched microphone, the inbound codec, TrackInfo
codec, packets/bytes received, loss, FEC and concealment, and for its own
microphone the outbound packets/bytes (`--red 0|1` sets the publish option).
call-bench adds `voiceResilience`: the engine and a plain SDK publisher, plus
a matched RED on/off pair with one watcher each.

Findings: native publishers already send RED (193 vs 94 payload B/packet at
48 kb/s) and the SFU forwards RED to native subscribers (192 vs 95 B/packet,
DownTrack.Bind audio/red). codecId reports the inner Opus on both sides, so
mime type alone cannot show RED. No product change is needed.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): clock remote audio from the output device (#164) (#198)

* fix(voice): clock remote audio from the output device (#164)

Remote audio had two clocks: the SDK's synthetic audio device pulled WebRTC
every 10 ms on a normal-priority timer, and our MMCSS thread rendered at the
device's pace. Their contention and drift dropped decoded frames and needed
20 ms of jitter priming.

The playout thread now enables the SDK's external playout mode and calls
livekit::pullAudioPlayout() right before mixing each 10 ms frame (also in the
no-device tick), then drains the track sinks that the pull filled
synchronously. The jitter buffer primes one frame. Ingress diagnostics split
dropped frames, discontinuities and underruns by microphone and screen-share
audio.

Bench (15 s phases): playout latency 33→22 ms idle, 72→22 ms under CPU load
and while viewing a screen share; dropped frames 1→0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* build(native): pin LiveKit SDK v1.10.0-syrnike.19

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(voice): WebRTC AGC2 instead of the hand-written 6x microphone gain (#165) (#199)

Automatic gain now runs through the SDK's AGC2 adaptive digital controller
(RealtimeAudioProcessing::processGain). The gate still measures the level
after AEC/NS and before gain, so threshold semantics do not change; the
silence guard still wins for a closed gate, mute and push-to-talk.

Offline measurement with SAPI speech (NS on, AEC off): quiet input speech
-27 -> -22 dBFS; loud input -21 -> -17 dBFS; signal-to-noise unchanged; with
the gate on pauses stay digital silence. microphone_dsp_bench runs a WAV
through the DSP for such measurements.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* fix(video): wake the remote video worker for frames that arrive during upload (#200)

The worker waited 16 ms on its condition variable without a predicate, so a
frame published while it was uploading the previous one was only noticed on
timeout and could be overwritten by the next frame. It now waits for pending
work (stop, a new frame, a revision change).

Diagnostics to find the remaining viewer frame and audio gaps:
- remote_video_metrics: superseded_before_upload, superseded_unsent,
  maximum/total upload time, slow_uploads (> 33 ms). Lifecycle counters move
  to remote_video_lifecycle_metrics.
- Every 5 s, inbound network stats per subscribed track: packets lost,
  NetEq concealment and time stretching for microphone and screen audio;
  loss, dropped frames, freezes and pauses for video.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* chore: bump app version to 0.7.0

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: tiredIsa <tiredisa@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant