Repository navigation
feat: stereo publish option, application playout delay, synchronous unbuffered audio capture - #13
Merged
Merged
Conversation
…nbuffered audio capture - TrackPublishOptions::stereo sets TF_STEREO, so the server negotiates stereo Opus with the publisher and its subscribers. - livekit::setAudioPlayoutDelay(): applications that render remote audio themselves report the delay they add after the SDK. The synthetic audio device returns it as PlayoutDelay, which WebRTC's audio/video sync subtracts from the audio playout position. - AudioSource::captureFrame on an unbuffered source hands the frame to WebRTC in the calling thread (CaptureAudioFrameRequest.synchronous, NativeAudioSource::capture_frame_direct). It no longer waits for an async runtime task and an FFI callback, which stalled realtime capture threads for up to 440 ms under CPU load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
📒 Files selected for processing (9)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
tiredIsa
added a commit
to syrnike13/syrnike13-app
that referenced
this pull request
Oct 11, 2026
Stereo publish option, application playout delay and synchronous unbuffered audio capture (syrnike13/client-sdk-cpp#13). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tiredIsa
added a commit
to syrnike13/syrnike13-app
that referenced
this pull request
Oct 11, 2026
…nd A/V sync (#157) * fix(native-media): send screen audio in direct capture mode The screen-audio AudioSource used the SDK's buffered mode (10 ms queue), which runs its own 10 ms WebRTC timer. Every late loopback packet made the timer inject 10 ms of silence, and every burst blocked captureFrame (up to 98 ms in a real session), so the backlog guard discarded queued PCM. Viewers heard crackling and up to ~120 ms of extra delay. Use direct capture like the microphone: the loopback device clock paces every 10 ms packet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): apply listener volume and mute per user id The renderer keys participant and stream volume/mute by user id, but the native mixer compared that value with the full LiveKit identity ("voice:v1|engine|instance|epoch|operation|userId"), so no control ever matched: the context-menu volume, the stream volume slider and the stream mute button had no effect in the desktop app. Rename AudioMixSetting.identity to userId and match it against the user id part of each Room identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep listener volumes across reloads and runtime replacement The renderer's remote-audio revision restarts at 1 after a reload while the main-process adapter drops settings older than the last one it saw, so volume changes were ignored until the counter caught up. A runtime replaced on account switch also started with no listener settings. The voice service now owns the revision and replays the latest settings to a replacement runtime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): ramp listener gain changes across one frame Volume, mute and deafen changes stepped the gain between two 10 ms frames, which clicks while a slider is dragged. Ramp each input's gain across the frame; new and re-primed inputs fade in from silence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(voice): coalesce listener volume updates while dragging Every pointer move of a volume slider wrote local settings to disk and sent a full desired-state update to the native engine. Persist once the value settles (300 ms, flushed on pagehide) and send at most one native mix update per 40 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(livekit-server): group each sender's audio and video for sync by default Enable room.sync_streams by default in the fork. Subscribers then see a sender's screen video and screen audio (and camera and microphone) in one WebRTC sync group, so receivers lip-sync them; without it WebRTC never synchronizes the pair (estimatedPlayoutTimestamp stays empty). LiveKit stops reusing subscriber transceivers in this mode. An explicit `room.sync_streams: false` in livekit.yml still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): keep screen audio continuous, stereo and on time under load - Queue is FIFO: the sender hands PCM to WebRTC synchronously (SDK syrnike.17), so a backlog after a stall drains at once. Discarding it stopped the RTP clock, and the sender reports then skewed screen audio against its video at every receiver (2.5 s after 395 drops under load). - Gaps in capture timestamps are sent as silent frames (at most 50), so the RTP clock stays in step with capture time; `silence_filled` metric. - Loopback capture and sender threads run in MMCSS "Audio", like the microphone path. - Publish as stereo Opus (TF_STEREO); it was negotiated mono. Under every-core load the engine's screen audio now drops 0 packets with a 0.12 ms maximum SDK call (was 395 drops, 440 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(native-media): synchronize screen video with screen audio at receivers WebRTC's own lip sync (RtpStreamsSynchronizer, as in browsers and Discord) runs once the server groups the streams (sync_streams). Feed it correct inputs: - Screen capture times reach the SDK in WebRTC's clock. On Windows WebRTC uses timeGetTime(), measured ~30 ms off QPC; a QPC time it saw as "future" was clamped to submit time, hiding capture/encode latency from the video sender reports. - Remote playout reports the delay it adds after the SDK (ingress queue, jitter buffer, device) through livekit::setAudioPlayoutDelay, so the synthetic audio device no longer claims 0 ms and pictures wait for the sound actually heard. remote_audio_output_metrics.latency_ms is now that measured value. Bench: a bot stamps its frames 150 ms early; viewers' playout offset is 7–38 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(native-media): bench stereo screen audio and audio/video sync bench_bot publishes stereo screen audio (--screen-audio 1), can stamp screen frames early (--screen-skew-ms N) and reports the watched sharer's screen-audio codec and playout offset. call-bench uses both while viewing a bot and while the engine shares the monitor with system audio, and reports the engine's screen-audio sender counters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(native-v2): screen audio direct capture, stereo and A/V sync Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(native-media): pin LiveKit SDK v1.10.0-syrnike.17 Stereo publish option, application playout delay and synchronous unbuffered audio capture (syrnike13/client-sdk-cpp#13). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
tiredIsa
added a commit
to syrnike13/syrnike13-app
that referenced
this pull request
Oct 12, 2026
* refactor: migrate TypeScript runtime to Effect * fix: harden voice media recovery * fix(voice): restore and expand RTC telemetry * fix: prevent long-running screen stream degradation * fix: prevent VoiceStage viewer degradation * fix: restore nightly native verification * test: isolate remote video static frame verification * fix(desktop): restore preload bridge on Windows * fix(voice): republish microphone after server unmute * fix(native): drop H264 AU until first complete SPS/PPS/IDR Late subscribers were feeding partial access units into FFmpeg before the first keyframe, which produced decoder corruption. Hold those frames at the decoder wrapper until a complete keyframe arrives. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(native): stabilize Windows media contention soaks and texture leases Keep Electron NT handles alive across probe recycle so the renderer cannot import a recycled handle while a fence is still held, and count skipped old-epoch releases as injected recovery without treating the hold as RPC latency. CI contention and microphone soaks pass their assertions; elevated ETW selection stays environment-blocked without Profile system performance. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(native): complete media stability validation * test(native): wait for cooperative voice cancellation * fix(native): avoid remote video stream churn on transient fences * fix(native): harden H264 recovery and contention evidence * fix(native): cancel screen cleanup at shutdown deadline * test(native): drain renderer releases before probe shutdown * test(native): sustain GPU rollover fault pressure * fix(native): resume renderer flow beside pending GPU work * fix(native): absorb bounded remote decode jitter * fix(native): serialize retired D3D resource release * fix(desktop): retire stale renderer releases with owner host * test(desktop): account for bounded video queue drops * test(native): warm production soak timer handle * test(native): schedule microphone soak as pro audio * test(native): bound microphone queue recoveries * test(native): bound microphone runtime handle warmup * test(desktop): report resilience addon ABI exactly * test(native): bound utility host handles * fix(native): bound display source visual probes * fix(native-media): stabilize Windows validation * test(desktop): make Windows CI contracts portable * test(native): await contention renderer readiness * test(native): align Windows smoke contracts * fix(native): measure receiver audio playout coverage * fix(desktop): close media reconcile completion race * test(native): stabilize cleanup retirement races * feat(desktop): establish native v2 boundary Remove the legacy native media and vendored LiveKit implementation, retain the hook-only runtime, and add the unavailable engine boundary for issue #112. * feat(native): bootstrap isolated media lifecycle boundary (#114) * feat(native): define v2 media control protocol Implements the bounded Electron/utility/N-API/C++ control, state, event, diagnostics, revision, and lifecycle contract for #115. * feat(native): add LiveKit media lab (#116) * docs: record Docker Desktop recovery notes * feat(desktop): complete native v2 phase A boundary * fix(media): harden native room boundary * feat(native): add bounded Windows source enumeration (#117) * feat(native): add bounded WGC monitor capture (#118) * feat(windows-media): add WGC window capture lifecycle (#119) * feat(native): add CPU screen publication reference (#120) * refactor(native): harden Phase B boundaries * feat(native): add GPU hardware H.264 screen sender (#121) * fix(native-v2): address epic #112 review findings * feat(native): add bounded remote video and Electron texture bridge (#122) * build(native): pin published SDK and clear stale local overrides * build(native): preserve canonical protocol bytes on Windows checkout * test(native): distinguish stall settling from held GPU leases * ci(native): separate GPU-required validation from hosted runner checks * test(native): require actual SDK connection continuity in viewer acceptance * fix: order native callback teardown before environment finalizers * test: isolate voice navigation from asynchronous icon loading * fix: use an allowed screen bitrate in the CPU reference lab * fix: treat Room removal as terminal in the CPU reference lab * feat: isolate local screen preview from publication (#123) * ci: replace Blacksmith with GitHub-hosted runners * fix: finalize terminal WGC capture only after frame leases drain * fix: pin CPU screen oracle to software H264 for late receivers * test: budget CPU screen lifecycle acceptance per cycle * test: await decoded screen video before Room disconnect * fix: drop WGC resize frames that exceed their actual surface * feat(native-media): add bounded adaptive screen quality (#124) * feat(native): add independent Windows screen audio (#125) * feat(native-media): DXGI monitor capture and deterministic backend selection (#126) (#138) * feat(native-media): add deterministic DXGI backend selection (#126) * test(native-media): select the CI lab interface and preserve diagnostic evidence * fix(native): drain minimized window frames before resize or delivery * build(native): pin SDK with port-owned ICE sockets * docs(native): record SDK 8 release and focused qualification * docs(native): record actual UAC pause and recovery proof * fix(native-media): pin SDK with joined negotiation shutdown * docs(native-media): record accepted phase C qualification [skip ci] * docs: update repository guidelines by removing outdated instructions and clarifying best practices * fix(native-media): preserve screen preset during bitrate adaptation (#139) (#140) * fix(native-media): keep screen preset fixed during bitrate adaptation (#139) * fix: bound raw screen admission and preserve live encoder rate control * test: exercise real encoder overload and static late subscription * test(native-media): retain full candidate qualification diagnostics * test(native-media): verify capture marker payload integrity * build(native-media): pin released preencoded SDK fix * test(native-media): qualify encoder pressure with bounded compute workload * docs(native-media): retain focused SDK 11 qualification results * docs(native-media): retain keyframe pacing candidate evidence * build(native-media): pin released keyframe pacing SDK * docs(native-media): retain published SDK 12 qualification results * docs(native-media): retain rejected SDK transport diagnostics * docs(native-media): record latency exception and final focused checks * fix(native-audio): recover PCM freshness after short sender stalls * test(native-media): retain audio recovery follow-ups and log lab termination * test(native-media): isolate static scene from cursor composition * docs(native-media): record final audio and fixed-preset qualification * feat(native-v2): warm microphone capture, DSP and publication (#127) (#141) * feat(native-media): add warm microphone capture and publication (#127) * Pin released realtime microphone SDK for #127 * Document user waiver of physical microphone unplug test * feat(native-media): bounded remote audio mixing and WASAPI output (#128) (#142) * feat(native-media): add bounded remote audio mixing and WASAPI output (#128) * build(native-media): pin verified LiveKit SDK 14 release * feat(native-media): independent camera capture, preview and publication (#129) (#143) * feat(native-media): add independent asynchronous camera pipeline (#129) * fix(camera): drain preview GPU work and record published SDK validation * docs(camera): verify combined Phase D build and publication * fix(ci): select Media Lab IPv4 without adapter aggregation * test(media-lab): report host pressure and failed ICE transports * feat(native-media): cut Windows desktop over to Engine v2 (#130) (#144) * feat(native-media): cut Windows product over to Engine v2 (#130) * fix(native): preserve generated media model line endings * test(native): await fatal event delivery before asserting retirement * test(native): identify capture resource threads in CI * fix(native-v2): finish bounded recovery acceptance Merge PR #145 after green hosted CI. The user-approved resource/full-suite exception remains documented in the PR and Phase E matrix. * chore(deploy): import per-project Caddyfile fragments Другие проекты на этом сервере держат свои vhost'ы отдельными файлами в /etc/caddy/sites (каталог монтируется через compose.override.yml, который деплой исключает из архива). Строка import в основном Caddyfile позволяет таким проектам жить независимо: выкат syrnike13 больше не сносит их домены. * fix(native-v2): wait for window capture worker retirement Hosted CI is green. Keep the strict zero-growth resource budget while allowing asynchronous Windows thread-pool retirement to settle before the final probe sample. * fix(deploy): use pinned MinIO images from Quay (#147) * fix(ci): skip absent hosted media lab report (#148) * fix(native-media): stabilize local voice and screen sharing * fix(native-media): make camera watchdog and WGC probe deterministic * fix(native-media): make microphone failures recoverable and diagnosable - Fall back from WASAPI RAW mode when the driver rejects it; RAW was the default and made capture fail on every device for affected drivers. - Treat MMCSS registration as an optional scheduling hint. - Fall back to the default input when the selected device is missing and report it as a path warning. - Report specific microphone failure codes with the platform HRESULT and add capture state, failure, HRESULT, RAW and MMCSS to microphone metrics. - Configure the startup voice runtime on first login instead of replacing the media utility process. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(native-media): remove media lab, probes and fault-injection harness Drop the experimental Media Lab (native src/lab, media_probe, packages/native-media-lab, desktop preview/video-bridge labs and product-fault harness), test-only fault gates and probe hooks from production sources, acceptance JSON dumps from docs/native-v2 and the CI steps that ran them. Unit tests keep their fixtures under native/tests/support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(voice): disable WASAPI RAW microphone capture by default RAW capture bypasses driver microphone processing and is rejected by many drivers. Desktop settings v4 turns it off once for existing installs; a later explicit opt-in is preserved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(desktop): drop dead v1 diagnostics and journal only transitions - Incident classification lists only events the media runtime records; v1 event names and their filters are removed. - Remove never-called recordNativeDiagnosticMetrics and its metric names. - Voice snapshots and per-track states are journaled only when they change, instead of on every meter/revision update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(desktop): decode media command results once The utility host dispatches commands through one handler table and forwards addon results; the supervisor decodes each reply once against its command-specific schema instead of three times. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(native-media): emit media metric samples through one helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(native-media): push video frames instead of polling on timers Frame producers (remote tracks, camera and screen previews) wake the export lane as soon as a frame is available, and the addon notifies the utility host through a coalesced frames-ready callback. Main pulls frames on that signal and whenever releases are queued, instead of a 16 ms timer stacked on an 8 ms native poll. The main timer now only paces inventory and retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(native-media): describe pushed frames and microphone capture policy Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: skip native test suites in release pipelines Release workflows rebuilt and re-ran the full Windows native test suite (ctest, ASan, 50-cycle Electron smoke) that pull requests already run. Timing-sensitive tests there flaked on shared runners and blocked desktop packaging. Release callers now pass skip_tests to build, verify, and stage native artifacts only; pull requests keep the full suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(native-media): device-clocked audio, lossless microphone and stable screen sharing (#149) * fix(native-media): keep echo cancellation and live state across calls - Renderer epochs are process-unique and a new render timeline keeps AEC active, so echo cancellation no longer stops after the first call or an output device switch. - Echo references are a short FIFO instead of latest-wins, so frames rendered between two capture frames still reach the canceller. - Voice gate holds 300 ms, fades on close, and drives the speaking indicator. - Microphone reports Starting only when capture or publication starts. - LiveKit statistics are sampled every 150 ms instead of every 20 ms. - MMCSS is optional for remote audio output and mixing. - Video frames and inventory are fenced by renderer and Room session, not by the global desired-state revision bumped by mute/PTT/volume. - A transient desired-state failure is retried, so a mute cannot be lost. - The utility restart budget is replenished after 60 s of stable host uptime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(native-media): one device-clocked playout thread and lossless audio Remote audio: replace the SDK reader thread (5 ms polling), the timer-driven mixer worker, the output owner and the WASAPI render worker with one RemoteAudioPlayout thread. The output device event clocks reading, mixing, rendering and the echo reference; per-track jitter buffers prime 20 ms, re-prime after underruns and trim drift. Under full CPU load SDK drops fell from 1104 to 0 frames per 15 s and voice CPU from 14.8% to 6-8% of a core. Microphone: PCM ports are bounded FIFOs drained on every wake instead of latest-wins slots, the sender runs under MMCSS, and the LiveKit source is unbuffered as the SDK recommends for live capture. Under load every captured frame is now sent (previously ~15-22% were lost). Screen: pacing a 1080p keyframe no longer counts as congestion (150 ms packet-queue threshold instead of 30 ms), and the hardware encoder uses a 60 s GOP so viewers are not stalled by an IDR every second (keyframes per 14 s: 15 -> 3, viewer gaps over 100 ms: 13 -> 1, 22.8 -> 30.1 fps). Reconnect: remote audio readers and video decoders survive SDK reconnection. Speaking indicators poll inventory at 10 Hz during a call. Tests: drop the encoder fault-injection suite (217-650 s, fails on a known driver handle leak) and the 100x repeat/handle accounting harness; the native suite now runs in 25 s. Add bench_bot and bench/call-bench.mjs for local end-to-end measurements against a dev LiveKit server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(native-media): recover microphone failures automatically A retryable microphone input or publication failure is retried every two seconds, recreating a retired pipeline, instead of waiting for a manual Retry. The call benchmark gains device switching, a delayed signaling proxy and fatal-failure capture. Microphone and remote audio docs describe the current design instead of removed lab tooling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): leave a channel instantly on a dead network LiveKit's graceful Room disconnect blocks for over 30 s when signaling is dead. The transport waited for it, so leaving or moving after a network loss hit the 2 s operation deadline and the fatal room_operation_unresponsive path restarted the whole media process. Observers and publications are already released when the transport disconnects, so the Room is now handed to a teardown lane and the Engine leaves at once (66 ms; previously a fatal failure after 2 s). At most four stuck teardowns are retained; beyond that the previous in-place disconnect and its deadline apply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): fall back to WGC when desktop duplication fails Monitor sharing forced DXGI desktop duplication, so it could not start where duplication is unavailable (hybrid GPUs, some remote sessions) or after it failed repeatedly. DXGI is now the preferred backend with the existing WGC fallback. WGC sessions request a borderless capture where Windows supports it. The benchmark also shares an animated fixture window over WGC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): trim media lifecycle smoke repetitions The process-guard check ran 100 warmups and 100 measured repetitions and the host lifecycle 50 cycles; 10, 25 and 10 keep the same coverage of every path. Electron-integration resource deltas were already informational. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(native-media): remove the unused CPU screen reference path and stale docs CpuScreenConverter was only reachable from its own reference test; production screen sharing converts on the GPU. Rewrite the architecture overview around the current threads, failure rules and the call benchmark, update resource ownership for the playout and teardown lanes, and drop the historical SDK blocker log and a link to a removed baseline document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(native-media): report expected microphone frames in the call benchmark Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): fall back to the default output when the selected one is gone Jitter counters of released readers are retained so totals never go back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): remote audio playout stack overflow under ASan (#150) * fix(native-media): keep remote audio readers off the playout thread stack The 64 readers hold ~1 MB of jitter buffers, the whole default thread stack. Release fit by a margin; the ASan Debug product-runtime test overflowed it. Allocate them once on the heap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(native-media): build the shared texture pool test again It went out with the media lab block in daa37f0, but it covers the product SharedTexturePool behind the Electron remote video bridge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(native-media): drop evidence logs of the removed media lab The native-v2 docs still described how to run native-media-lab, media_probe and build:lab, linked acceptance JSON reports that are no longer in the repository and recorded one-off qualification runs. Keep the contracts, point verification at the CTest targets and the call benchmark that exist, and record the final benchmark numbers for the microphone and remote audio. The DXGI doc now says the product prefers DXGI with a WGC fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * chore: remove unused web and desktop code (#151) * chore(web): remove modules nothing imports Only their own tests referenced them: the old friends view, the renderer PCM jitter worklet from before native playout, screen share tuning/limits helpers, voice operation, gate calibration, preference effects, participant media, transition rate limit and watch helpers, the skeleton component, the message-format barrel and the unused React Query devtools panel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(web,desktop): remove declarations nothing references Exported functions, constants and types with no reference anywhere in the repository, tests included: unused promise wrappers next to the Effect requests the app calls, authorization helpers, selectors, mention helpers, icons, desktop contract types. Imports and private helpers left unused by that are removed as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(web,desktop): drop promise wrappers only tests called Tests now run the Effect the app uses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: describe desktop-native and windows-media-engine in the repository map Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(web,admin): pin TanStack versions and drop unused devtools `latest` specifiers made every package.json change silently upgrade router, start, query and form; they are pinned to the versions the lockfile already resolved. The four devtools packages had no import. Ignore local build-* scratch directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * refactor(desktop): run input and overlay hooks in the main process (#152) Hotkeys and overlay foreground detection ran in two utility processes, each with a supervisor, handshake, host epochs, restart backoff, an artifact manifest check and its own copy of the addon. The hooks already run on the addon's message-loop thread, so the main process only relays events; Discord loads its hooks in-process the same way. - desktop-native builds one syrnike_hooks.node with createHooksRuntime; its runtime already served hotkeys and overlay together. - main/native-hooks.ts loads it and correlates replies; hotkeys and the overlay detector keep their calls. A hook runtime error releases held hotkey actions, as runtime loss did before. - Removed the hooks supervisor, utility adapter, contract, artifact verification, utility hosts and their tests. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): hide speaking indicators while deafened (#153) A deafened listener hears nobody, so remote speakers are no longer reported as speaking. Detection keeps tracking the noise floor, so the indicators are accurate again as soon as deafen is lifted. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): screen share — borderless, static-screen keyframes, source aspect ratio, receive slots, real window list (#154) * fix(native-media): borderless capture, static-screen keyframes, no cropping, enough receive slots - Request GraphicsCaptureAccess Borderless before IsBorderRequired(false): Windows ignored the flag, so window shares and thumbnails had a yellow border. - Re-encode a GPU copy of the newest capture after 250 ms without a new one. A viewer who joined a static window got no frame until it changed. - Letterbox sources of another aspect ratio instead of center-cropping them. - Remote video pool: 12 slots instead of 4, reuse same-size textures first, and return a quarantined slot once its timed-out GPU write completes. - Bench: --port, a late viewer of a frozen window, firstFrameMs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(native-media): stream follows the source aspect ratio; list only real windows - The preset is a box: the encoded size keeps the source's aspect ratio (never upscaled, multiples of 8). The first frame sets it before publish; a later size change that holds for 500 ms replaces the converter and encoder on the same publication, so viewers keep their track. - Source enumeration skips tool windows, owned windows without WS_EX_APPWINDOW, and hidden/cloaked windows never seen shown (258 candidates on this desktop, 13 real; the list was truncated). - Bench: a shared window turns portrait while a viewer watches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): Discord-like capture details - Closing the shared window ends the share (source_closed/source_replaced turn the screen intent off) instead of a failed path and an error banner. - WGC MinUpdateInterval at half the stream frame interval: a 165 Hz display no longer delivers 165 frames a second for a 30 fps stream. - Skip content-protected windows (display affinity), which capture as black. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(native-media): recover stale window capture; paused stream while minimized - Like Discord: a shown window without frames for 2 s is asked to repaint (asynchronous invalidate); no frame 1 s later recreates the frame pool and session (at most every 10 s, no leased frames). Recreating only the session on the old pool left the capture dead. - A minimized/hidden shared window mutes the screen track and stops repeats; restoring unmutes and requests a keyframe. Native viewers get `muted` in the video inventory; viewers show "Трансляция на паузе" instead of a frozen frame (LiveKit browser viewers through the same track mute). - Bench/fixture: `stall` ignores repaints; the bot counts mute/unmute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * feat(native-media): decode remote H.264 on the GPU (#156) * feat(native-media): decode remote H.264 on the GPU Remote H.264 is decoded by the Media Foundation MFT through DXVA on the process D3D11 device (any GPU vendor), converted NV12 -> BGRA by the D3D11 video processor and copied into the shared texture pool on the GPU. The SDK's external decoder hook keeps WebRTC's FFmpeg decoder as the fallback for machines without DXVA. Pool writes wait on a D3D11 fence instead of spinning with the device context locked. Also fixes the frame protocol: since the pool grew to twelve slots, frames in slots 4-11 failed the release/exported-frame bounds, invalidating the whole frames reply. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(native-media): pin LiveKit SDK v1.10.0-syrnike.16 (external video decoder) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(voice): stream/participant volume, screen audio crackle, stereo and A/V sync (#157) * fix(native-media): send screen audio in direct capture mode The screen-audio AudioSource used the SDK's buffered mode (10 ms queue), which runs its own 10 ms WebRTC timer. Every late loopback packet made the timer inject 10 ms of silence, and every burst blocked captureFrame (up to 98 ms in a real session), so the backlog guard discarded queued PCM. Viewers heard crackling and up to ~120 ms of extra delay. Use direct capture like the microphone: the loopback device clock paces every 10 ms packet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): apply listener volume and mute per user id The renderer keys participant and stream volume/mute by user id, but the native mixer compared that value with the full LiveKit identity ("voice:v1|engine|instance|epoch|operation|userId"), so no control ever matched: the context-menu volume, the stream volume slider and the stream mute button had no effect in the desktop app. Rename AudioMixSetting.identity to userId and match it against the user id part of each Room identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep listener volumes across reloads and runtime replacement The renderer's remote-audio revision restarts at 1 after a reload while the main-process adapter drops settings older than the last one it saw, so volume changes were ignored until the counter caught up. A runtime replaced on account switch also started with no listener settings. The voice service now owns the revision and replays the latest settings to a replacement runtime. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): ramp listener gain changes across one frame Volume, mute and deafen changes stepped the gain between two 10 ms frames, which clicks while a slider is dragged. Ramp each input's gain across the frame; new and re-primed inputs fade in from silence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * perf(voice): coalesce listener volume updates while dragging Every pointer move of a volume slider wrote local settings to disk and sent a full desired-state update to the native engine. Persist once the value settles (300 ms, flushed on pagehide) and send at most one native mix update per 40 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(livekit-server): group each sender's audio and video for sync by default Enable room.sync_streams by default in the fork. Subscribers then see a sender's screen video and screen audio (and camera and microphone) in one WebRTC sync group, so receivers lip-sync them; without it WebRTC never synchronizes the pair (estimatedPlayoutTimestamp stays empty). LiveKit stops reusing subscriber transceivers in this mode. An explicit `room.sync_streams: false` in livekit.yml still overrides it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(native-media): keep screen audio continuous, stereo and on time under load - Queue is FIFO: the sender hands PCM to WebRTC synchronously (SDK syrnike.17), so a backlog after a stall drains at once. Discarding it stopped the RTP clock, and the sender reports then skewed screen audio against its video at every receiver (2.5 s after 395 drops under load). - Gaps in capture timestamps are sent as silent frames (at most 50), so the RTP clock stays in step with capture time; `silence_filled` metric. - Loopback capture and sender threads run in MMCSS "Audio", like the microphone path. - Publish as stereo Opus (TF_STEREO); it was negotiated mono. Under every-core load the engine's screen audio now drops 0 packets with a 0.12 ms maximum SDK call (was 395 drops, 440 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(native-media): synchronize screen video with screen audio at receivers WebRTC's own lip sync (RtpStreamsSynchronizer, as in browsers and Discord) runs once the server groups the streams (sync_streams). Feed it correct inputs: - Screen capture times reach the SDK in WebRTC's clock. On Windows WebRTC uses timeGetTime(), measured ~30 ms off QPC; a QPC time it saw as "future" was clamped to submit time, hiding capture/encode latency from the video sender reports. - Remote playout reports the delay it adds after the SDK (ingress queue, jitter buffer, device) through livekit::setAudioPlayoutDelay, so the synthetic audio device no longer claims 0 ms and pictures wait for the sound actually heard. remote_audio_output_metrics.latency_ms is now that measured value. Bench: a bot stamps its frames 150 ms early; viewers' playout offset is 7–38 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(native-media): bench stereo screen audio and audio/video sync bench_bot publishes stereo screen audio (--screen-audio 1), can stamp screen frames early (--screen-skew-ms N) and reports the watched sharer's screen-audio codec and playout offset. call-bench uses both while viewing a bot and while the engine shares the monitor with system audio, and reports the engine's screen-audio sender counters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(native-v2): screen audio direct capture, stereo and A/V sync Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(native-media): pin LiveKit SDK v1.10.0-syrnike.17 Stereo publish option, application playout delay and synchronous unbuffered audio capture (syrnike13/client-sdk-cpp#13). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(deploy): run MinIO from the server's local images (#158) MinIO no longer publishes images: Quay answers 401 and Docker Hub removed minio/minio, so every nightly deploy failed at `docker compose pull`. The server already has the exact images both stacks run. Reference them by their local docker.io digests with `pull_policy: never`, so pull skips them and up reuses them. Verified on the server: `compose config` passes and `compose pull minio createbuckets` reports "Skipped" for nightly and production. The image IDs do not change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(deploy): pin MongoDB 8.2.12; unpinned mongo moved to 9.0 (#159) The unpinned docker.io/mongo now resolves to MongoDB 9.0.2. Its mongod exits with code 62 on our data (featureCompatibilityVersion 8.2), so the nightly deploy recreated the database container into a restart loop. Pin mongo:8.2.12 by its index digest (e0ce8c35…), the exact image production runs, in both compose files. Production would have hit the same failure on its next deploy. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(deploy): drop MinIO; files live in external S3 (#160) MinIO no longer publishes images, so a fresh server could not start it. Production already stores files in an immers.cloud bucket and never used its MinIO container; nightly now uses its own bucket there. The compose files no longer run minio or createbuckets; secrets.env.example documents the [files.s3] settings a new instance needs. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(voice): publish the microphone at the channel's voice bitrate (#161) (#195) The channel setting `voice.audio_bitrate_kbps` (8-96 kbps) was saved but never used: the browser always published at 64 kbps and the desktop engine left the Opus bitrate at the SDK default. - Voice desired state carries `microphoneBitrate`; the renderer sets it from the channel before joining and whenever the channel or its setting changes (`setMicrophoneBitrate`). DM and group calls keep 64 kbps. - Browser: published with that bitrate; a change retargets the sender with `RTCRtpSender.setParameters`, without republishing. - Desktop: `MicrophoneIntent.bitrate` reaches the native sender as `audio_encoding`. The SDK fixes Opus at publication, so a change republishes the microphone once. - call-bench: a peer counts the engine's microphone republishes; bench_bot `--mic-bitrate` reports the encoder target the SDK applies. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * test(voice): measure RED/FEC on native microphones in the call bench (#171) (#197) bench_bot reports, for a watched microphone, the inbound codec, TrackInfo codec, packets/bytes received, loss, FEC and concealment, and for its own microphone the outbound packets/bytes (`--red 0|1` sets the publish option). call-bench adds `voiceResilience`: the engine and a plain SDK publisher, plus a matched RED on/off pair with one watcher each. Findings: native publishers already send RED (193 vs 94 payload B/packet at 48 kb/s) and the SFU forwards RED to native subscribers (192 vs 95 B/packet, DownTrack.Bind audio/red). codecId reports the inner Opus on both sides, so mime type alone cannot show RED. No product change is needed. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(voice): clock remote audio from the output device (#164) (#198) * fix(voice): clock remote audio from the output device (#164) Remote audio had two clocks: the SDK's synthetic audio device pulled WebRTC every 10 ms on a normal-priority timer, and our MMCSS thread rendered at the device's pace. Their contention and drift dropped decoded frames and needed 20 ms of jitter priming. The playout thread now enables the SDK's external playout mode and calls livekit::pullAudioPlayout() right before mixing each 10 ms frame (also in the no-device tick), then drains the track sinks that the pull filled synchronously. The jitter buffer primes one frame. Ingress diagnostics split dropped frames, discontinuities and underruns by microphone and screen-share audio. Bench (15 s phases): playout latency 33→22 ms idle, 72→22 ms under CPU load and while viewing a screen share; dropped frames 1→0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(native): pin LiveKit SDK v1.10.0-syrnike.19 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(voice): WebRTC AGC2 instead of the hand-written 6x microphone gain (#165) (#199) Automatic gain now runs through the SDK's AGC2 adaptive digital controller (RealtimeAudioProcessing::processGain). The gate still measures the level after AEC/NS and before gain, so threshold semantics do not change; the silence guard still wins for a closed gate, mute and push-to-talk. Offline measurement with SAPI speech (NS on, AEC off): quiet input speech -27 -> -22 dBFS; loud input -21 -> -17 dBFS; signal-to-noise unchanged; with the gate on pauses stay digital silence. microphone_dsp_bench runs a WAV through the DSP for such measurements. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * fix(video): wake the remote video worker for frames that arrive during upload (#200) The worker waited 16 ms on its condition variable without a predicate, so a frame published while it was uploading the previous one was only noticed on timeout and could be overwritten by the next frame. It now waits for pending work (stop, a new frame, a revision change). Diagnostics to find the remaining viewer frame and audio gaps: - remote_video_metrics: superseded_before_upload, superseded_unsent, maximum/total upload time, slow_uploads (> 33 ms). Lifecycle counters move to remote_video_lifecycle_metrics. - Every 5 s, inbound network stats per subscribed track: packets lost, NetEq concealment and time stretching for microphone and screen audio; loss, dropped frames, freezes and pauses for video. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * chore: bump app version to 0.7.0 --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: tiredIsa <tiredisa@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For syrnike13/syrnike13-app#157 (screen-share audio: crackling, lag, mono, audio/video sync).
Changes
TrackPublishOptions::stereo(FFITrackPublishOptions.stereo = 14) addsTF_STEREOto theAddTrackRequest. The server answers the publisher and offers subscribersstereo=1. Before this, two-channel screen audio was encoded as mono Opus.livekit::setAudioPlayoutDelay(ms)(C ABIlivekit_ffi_set_audio_playout_delay) is a process-wide value that the synthetic audio device returns asPlayoutDelay. WebRTC'sChannelReceivesubtracts it from the audio playout position, so audio/video sync (RtpStreamsSynchronizer) accounts for audio the application renders itself after the SDK. It is clamped to 0..1000 ms.queue_size_ms = 0sources,AudioSource::captureFramenow sendsCaptureAudioFrameRequest.synchronous = true. The FFI calls the newNativeAudioSource::capture_frame_directin the calling thread, without a runtime task or completion callback. Previously, with every core busy, a realtime capture thread waited up to 440 ms for the tokio task and the FFI event thread, and 395 of 2,022 screen-audio packets were dropped in 20 s. Errors now surface as the call's own exception.Validation (local Windows x64 build)
livekit_unit_tests377/377, including the newTrackPublishOptionsTest.StereoRoundTripandAudioSourceTest.UnbufferedCaptureCompletesInTheCallingThread.cargo test -p livekit-ffi --lib conversion::room, includingstereo_publish_option_round_trips.cargo fmt --checkis clean.0001-native-v2-runtime.patchapplies withgit apply --checkto a fresh1a477bc4checkout.call-benchagainst the project's LiveKit fork withsync_streams:opus/2 stereo=1;captureFrameof 0.12 ms (it was 440 ms).🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.