Conversation
pblazej
changed the title
Exposes the telemetry core's public API through UniFFI. Example app displays the Rust core build version as an FFI smoke test.
Process-wide export queue with device state monitoring and RTC stats collection. Serves OTLP/HTTP pulled export with persistent storage and fail-open behavior.
Record Engine connect checkpoints on the lk.connect span that Room wiring starts, add reconnect spans, and propagate the mapped disconnect reason.
Track lk.publish spans for local participants, lk.subscribe for remote. Capture getUserMedia failures with mapped reason.
Add LiveKitClient.disableTelemetry() to opt out process-wide. Room.emitTelemetryEvent() sends custom.<name> events with attributes. Room.setTelemetryAttribute() sets correlation attributes per session.
Unit tests for SDK logger, telemetry core errors, isolate safety, opt-out. E2E test against local collector validates the full pipeline.
OTEL collector service provides HTTP endpoint for E2E validation.
Changesets for client telemetry and uniffi-rust-core dev wiring. AGENTS.md documents UniFFI setup and local development: Native Assets, overrides and build hooks.
6c5d055 to
8755fbe
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| return result! as LocalTrackPublication<LocalAudioTrack>; | ||
| } | ||
|
|
||
| return room.telemetry?.publish(track, publish) ?? publish(); |
There was a problem hiding this comment.
🔴 Device toggles omit publish spans
When apps enable a microphone, camera, or screen share, publishAudioTrack and publishVideoTrack are bypassed. setTrackEnabled calls the private publishers directly, so these publications have no lk.publish span.
Learn more
The public track publishers wrap their serialized work in telemetry, but the SDK's device-toggle path invokes _publishAudioTrack and _publishVideoTrack directly inside _publishRunner in setTrackEnabled. This affects microphone, camera, screen-share video, and captured screen-share audio. Those publishes still reach the server, but never reach the telemetry publish wrapper.
Example: Calling room.localParticipant!.setMicrophoneEnabled(true) captures and publishes the microphone through _publishAudioTrack, but records no lk.publish span. Publishing an already-created microphone via publishAudioTrack records one.
Recommended fix: Instrument the shared private publish operation or the serialized toggle path as well, without calling a public method from inside _publishRunner and deadlocking its queue. Preserve one span per published track, including screen-share audio.
Was this helpful? React with 👍 or 👎 to provide feedback.
| _stats = Timer(due - _clock.elapsed, () async { | ||
| if (_disabled || _room.connectionState == ConnectionState.disconnected) return; |
There was a problem hiding this comment.
🔴 RTC statistics stop after reconnect
If the stats timer fires while signaling is disconnected, _poll exits without rearming. A successful resume emits RoomReconnectedEvent, not RoomConnectedEvent, so RTC statistics stay silent until another track event.
Learn more
A connected Room schedules _poll according to the core's interval. During a resume, SignalClient.connect temporarily calls cleanUp, which sets its state to disconnected, before opening the new socket. If the timer fires in that window, it returns without scheduling another timer. The existing RoomReconnectedEvent does not restart the telemetry timer, and a resume does not emit RoomConnectedEvent.
Example: A Room has a one-second poll due at 12:00:01. The socket drops at 12:00:00.5 and reconnects at 12:00:02. The scheduled poll sees disconnected at 12:00:01 and stops; a quiet call collects no further RTC stats.
Recommended fix: Resume _poll() on successful resume/full restart, or keep scheduling while reconnection is pending and stop only on a final RoomDisconnectedEvent or disposal.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (_room.connectOptions.autoSubscribe) { | ||
| for (final participant in _room.remoteParticipants.values.toList()) { | ||
| for (final publication in participant.trackPublications.values.toList()) { | ||
| if (publication.track == null) subscribeStarted(publication); |
There was a problem hiding this comment.
🟡 Early media loses subscribe spans
If a pre-join track attaches before RoomConnectedEvent, subscribeStarted skips its publication because track is non-null. TrackSubscribedEvent only marks an existing span, so that subscription has no lk.subscribe span.
Learn more
Tracks received before their metadata are queued by EngineTrackAddedEvent. During join, _getOrCreateRemoteParticipant flushes that queue before the handler emits RoomConnectedEvent. When the track is attached in time, the new telemetry handler excludes its publication on RoomConnectedEvent, even though no TrackPublishedEvent was emitted during join. The TrackSubscribedEvent handler calls _scope.subscribed but never starts the subscribe intent.
Example: A remote audio track arrives while the Room is connecting. Joining creates the remote participant and flushes the queued audio before RoomConnectedEvent. The publication already has a track when the telemetry handler enumerates it, so its lk.subscribe span never starts.
Recommended fix: Register auto-subscribe intent for all eligible join publications, including ones already carrying media; account for an already subscribed track when signaling its completion to the core.
Was this helpful? React with 👍 or 👎 to provide feedback.
This branch has not been deployed
1 participant
Client telemetry for Flutter, on top of the shared Rust core (livekit/rust-sdks#1396). On iOS, Android, macOS, Windows and Linux every Room reports its spans, RTC statistics, SDK warnings/errors and device state to its LiveKit Cloud project (only when the token carries the observability grant), for about 1,200 lines of Dart and no new public types; web stays a compiled no-op.
Carries the
livekit_uniffifacade from #1160 (uniffi-rust-core) until that merges, so GitHub's diff againstmainalso shows its lines. The counts below are this PR's own.Public API
LiveKitClient.disableTelemetry()getStats()or stats submission begins afterwards; another live isolate stops at its next check of the core's process-wide flag (made right before each read and each submission), so at most onegetStats()per Room that had already passed the check can still start there, and whatever it submits is refused by the core (nothing is uploaded); unsent data is deleted in the background; TODO: final shape pending the token/consent discussionRoom.emitTelemetryEvent(String name, {Map<String, String> attributes})Room.setTelemetryAttribute(String key, String? value)nullremovesNothing else is public. Configuration, instruments, transport and the UniFFI types stay internal (nothing new is exported from
livekit_client.dart;Room.telemetryandEngine.telemetryare@internalfields, like the existing@internal Room.engine); every tuning value (60 s export, 60 s windows, stats poll interval) is the core's default.Platform code
What this platform adds on top of Rust (everything else — destination, token handling, retries, cache, holds, stats mapping, span state — is in the core).
Four files, 1,030 lines; about 1,200 lines in total including the wiring in existing files.
Files and responsibilities
telemetry/telemetry_io.darttryNext()— every second while a Room is in a call or requests keep coming, else every 5 s (uniffi-dart callbacks are isolate-bound; the SDK leaves the core no callback and no pending Rust future into the isolate, so hot restart and recreated engines are safe) withpackage:http: the raw answer comes back, a redirect is never followed, an unanswered request is aborted after the export timeout; SDK log records (the live ambient span, else the Room, else the process); device and log instruments started and stopped from Dart (app lifecycle, memory pressure, connectivity, audio outputs → one change stream →DeviceState/ events); per-Room session: token forwarding,lk.connect/lk.reconnect/lk.publishspans, remote track lifecycle for the core'slk.subscribe, onegetStats()per peer connection everystatsPollIntervalMs()→recordPeerStats; the on-disk cache location (the app's temporary directory on mobile, a per-user directory per app on desktop: an absolute$XDG_CACHE_HOME, else$HOME/.cacheon Linux, memory only without either); one guard around the core calls made from SDK paths (connect, publish, subscribe, spans, teardown, opt-out, app events, log and device capture), so a core error is dropped and never fails a connect, publish, teardown or log call; a disposed Room leaves no listener on its signal client, and an ended span drops its native handle and its Room reference, so a zone that outlives the Room (the signal client's existing connectivity subscription) keeps only an empty span object; enum mappingstelemetry/telemetry.darttelemetry/telemetry_web.dartsupport/sdk_logger.dartlogger: every call goes to thelivekitLoggerunchanged when it would emit (telemetry reads its records); only a warning/error its level filters out is built once for telemetry, guarded, and never thrown to the calleruniffi/uniffi_io.dartlivekit_uniffiChanges in existing code
Wiring only: every existing public API and the SDK's logging behave as on main.
Changed files
Roomconnect()delegates to a private_connect()(body unchanged) run insidelk.connect; its engine/signal listeners are set up in the Room's telemetry zone (re-indent only); the two public methods; the session follows room updates and a room move, and ends ondispose()withoutdisconnect()Enginelk.reconnectspan per reconnect cycle, attempts as checkpoints; the internalEngineClosingEventcarries the disconnect reasonLocalParticipant.publishAudioTrack/publishVideoTracklk.publishspan per attempt, nested under the ambient span or the open connect span (pre-connect microphone); the sid is set on the span once knownRemoteTrackPublication.subscribe()lk.subscribeLocalTrack.createStreamlk.device.capture.failed, then rethrownLiveKitClientdisableTelemetry()logger.dartloggeris thelivekitLoggerwrapped bySdkLogger; telemetry reads its warning/error records, and gets the ones its level filters out (disableLogging(), a level above WARNING) from the wrapper.gitignore,AGENTS.mdpubspec.yamlis untouched by this PR. A local rust-sdks build is wired per checkout with a git-ignoredpubspec_overrides.yamlnext to the package's and the example'spubspec.yaml, each path relative to its own file (overrides do not propagate from a dependency, hence both);AGENTS.md's local loop shows the two commands; without the files the published package resolveslivekit_unifficonstraint is bumped to it (a cross-platform release step), a build without the override fails to compilebuild.yamlflutter test, one more step runs a checksum-verifiedotelcol-contribandflutter test test/telemetry/withLK_TELEMETRY_ENDPOINT; a collector that does not start fails the jobtest/mock/peerconnection_mock.dart,e2e_container.dartgetStats()reports its sent tracks and mock inbound tracks;addTransceiver/restartIceimplemented;connectRoom(otherParticipants:)joins a room others are already inEvents
9 of 19 SPEC signals fully covered, 5 partially, 5 skipped (no plugin exposes them to the SDK, or smoke-test only).
error.typeis the Dart exception's type name (TrackPublishException); in--obfuscatebuilds that name is obfuscated — how every platform names errors is a pending cross-platform decision.Event coverage table
lk.connectspan (+ checkpoints)Room.connect:ws_open·signal·join_recv·pc_created·offer_sent·answer_sent·engine·pc_connected·room_connected; the app'sdisconnect()meanwhile ends it as cancelled, a server close as failedlk.reconnectspanEnginereconnect cycle;attempt <n> quick|fullper attemptlk.publishspanpublishAudioTrack/publishVideoTracklk.subscribespansubscribe(); subscribed / failed / unsubscribed / unpublished (cancelled before first media); first media seen by the corelk.rtc.stats.samplegetStats()per peer connection (publisher + subscriber), paced by the core; track signals only bring the next poll forward, one poll at a time, each read bounded to 5 slk.room.disconnectedRoomDisconnectedEventreasonlk.telemetry.reportcustom.<name>Room.emitTelemetryEventlivekitlogger, whatever its console level; the Rust core copies its own warnings/errors once a platform installs its log forwarder, which the Flutter SDK does not (no Rust log forwarding today); native WebRTC logs are not capturedlk.device.thermal.changedlk.device.low_power.changedlk.device.app_state.changedWidgetsBindingObserverlifecycle, starting from the binding's current statelk.device.memory.changeddidHaveMemoryPressure→ warning; the OS never signals relief, so back to normal after a quiet minute; no critical levellk.device.network.changedlk.device.battery.changedlk.device.audio_route.changedHardware.onDeviceChange), outputs from device labels; the platform gives no reasonlk.device.audio.interruptionlk.device.capture.failedlk.pingLocal testing
Try it against a local OTel backend (LGTM) in a few minutes; the e2e test needs no server (signalling and media are mocked).
Commands
Then open http://localhost:3000 → Explore: logs (Loki:
{service_name="livekit-client-flutter"} | app_call_id="<marker>",otel_event_name) and traces (Tempo: the records'trace_id;lk.connect,lk.publish,lk.subscribe, …) for the session's trace id.