Conversation
|
|
pblazej
changed the title
Introduce telemetry infrastructure built on livekit-telemetry Rust crate. Add Telemetry singleton managing OTLP export pipeline, opt-out control, and tracing context helpers. Pipeline ships spans to configured collector; defaults to no-op when unconfigured. Rename UniFFI error field from message to reason to avoid Kotlin bindgen conflict. Forward SDK warnings and errors to telemetry core.
Add tracing spans for connect, publish, subscribe and reconnect, and record disconnect events with their mapped reason. Connect span wraps full connection setup; publish and subscribe spans track track operations; reconnect spans measure recovery time. Add SDK-internal observer support to MulticastDelegate.
Report device thermal state, power state, memory warnings, network reachability changes, battery level and state, audio route changes, and app foreground/background transitions as telemetry events. Bounded stream prevents unbounded memory growth from rapid device changes.
Poll getStats() on each peer connection; submit codecs, inbound/outbound RTP streams, ICE candidates, and connection quality metrics. Respects opt-out; keeps an idle timer while disconnected, without reading stats.
Add disableTelemetry() for opt-out, emitTelemetryEvent() for custom events, and setTelemetryAttribute() for context propagation. Opt-out stops collection when the call returns.
Cover connect, publish, subscribe, reconnect, late join, device state, RTC stats, and opt-out. Include local OTLP collector config; tests skip when collector unavailable.
Update CI workflow for telemetry tests and benchmark support.
e4acf0e to
96be295
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 5 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
There was a problem hiding this comment.
🟡 Room moves lose subscription start times
When a room moves with existing remote tracks, joined never reports their subscription intent. The move rebuilds participants without publishing callbacks, so their subscribe spans start only when media arrives.
(Refers to this code)
Was this helpful? React with 👍 or 👎 to provide feedback.
| let next = DispatchTime.now().uptimeNanoseconds + wait | ||
| self?.polling.mutate { if !Task.isCancelled { $0.next = next } } |
There was a problem hiding this comment.
🟡 Track polls restart on stale deadlines
When poll replaces a sleeping loop, that loop can overwrite polling.next after its stats request completes. A new subscription can then inherit an incorrect deadline and delay its first-media reading.
Was this helpful? React with 👍 or 👎 to provide feedback.
| for case let publication as RemoteTrackPublication in participant.trackPublications.values where publication.track == nil { | ||
| guard let track = SpanTrack(publication, remoteIdentity: participant.identity?.stringValue) else { continue } | ||
| scope.subscribeStarted(track: track) |
There was a problem hiding this comment.
🟡 Unwanted tracks open subscribe spans
With auto-subscribe enabled, joined treats every unattached track as wanted, including tracks explicitly unsubscribed through set(subscribed: false). A full reconnect then records a subscribe wait for media the client never requested.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if level >= .warning { | ||
| Telemetry.log(LogRecord(severity: level.severity, source: .sdk, body: message?.description ?? "", | ||
| logger: String(describing: Self.self), function: "\(function)", file: "\(file)", | ||
| line: UInt32(clamping: line)), | ||
| scope: scope) |
There was a problem hiding this comment.
🟡 Warning messages are evaluated twice
For warning and error logs, log evaluates message for both the app logger and telemetry. Side-effecting or changing descriptions produce inconsistent records and duplicate work.
Was this helpful? React with 👍 or 👎 to provide feedback.
| guard let url = URL(string: request.url) else { | ||
| throw ExportError.Rejected(reason: "invalid url \(request.url)") | ||
| } | ||
| var urlRequest = URLRequest(url: url) | ||
| urlRequest.httpMethod = "POST" | ||
| urlRequest.httpBody = request.body | ||
| for (name, value) in request.headers { | ||
| urlRequest.setValue(value, forHTTPHeaderField: name) |
There was a problem hiding this comment.
🟨 Cleartext telemetry can expose participant tokens
When the supplied server URL uses HTTP, telemetry can send its bearer token over cleartext HTTP. The redirect guard checks same-origin redirects but never requires an encrypted transport.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Client telemetry for Swift, on top of the shared Rust core (livekit/rust-sdks#1396). Every Room reports its spans, RTC statistics, SDK warnings/errors and device state to its LiveKit Cloud project (only when the token carries the observability grant), for about 1.2k lines of Swift and no new public types.
Public API
LiveKitSDK.disableTelemetry()Room.emitTelemetryEvent(_:attributes:)Room.setTelemetryAttribute(_:value:)nilremovesNothing else is public. Configuration, instruments, transport and the UniFFI types stay internal; every tuning value (60 s export, 60 s windows, stats poll interval) is the core's default.
Platform code
What this platform adds on top of Rust (everything else — destination, token handling, retries, cache, holds, stats mapping, span state — is in the core).
Three files, 838 lines; about 1,210 lines in total including the wiring in existing files.
Files and responsibilities
Telemetry/Telemetry.swiftURLSessiontransport that returns the raw answer and never follows a redirect to another origin; enum mappingsTelemetry/DeviceTelemetry.swiftDeviceState; audio route / interruption → device events; oneAsyncStreamfor every OS callbackTelemetry/RTCTelemetry.swiftlk.subscribe(intents at join and after a full reconnect for tracks already in the Room); onegetStats()per peer connection everystatsPollIntervalMs()→recordPeerStats; report flatteningRoom/Room+Engine)Changes in existing code
Wiring only: every existing public API, the console logging and custom tracers behave as on main; one visible side effect, battery monitoring (below).
Changed files
Room,Room+SignalClientDelegatesetRoomat join, full reconnect, move and room update; join-time subscribe intents;disconnectedat clean-up, with the protocol reason of a server Leave; connect runs inside the telemetry span (re-indent only), the pre-connect microphone publish outside itRoom+Engine,Room+TransportDelegatelk.reconnectspan per reconnect cycle, attempts as checkpoints; subscribe intents re-reconciled after a full reconnect; a cycle that gives up reportsreconnect_failedLocalParticipant._publishlk.publishspan per attempt, parented to the ambient span only while it is open (its sid lets the core poll the new track sooner); a capture error during publish is reported aslk.device.capture.failedRemoteTrackPublication.set(subscribed:)lk.subscribeSupport/Tracing.swiftSpanoptionally carries the core's span:record(_:)adds its checkpoint,end()ends itSpan,Tracing,LoggingTracer,setTracing,Room.connectSpanas on main; custom tracers (Benchmarks) work as beforeSupport/Logger.swiftLoggablewarnings/errors also go to the coreLoggergets exactly what it got beforeMulticastDelegateremoveAllDelegates()(the RTC instrument uses it)TransporttelemetryStatistics(), initiated on the RTC executor only while the opt-out lock admits it, bounded to 5 s (a late answer is dropped)LiveKit.swiftdisableTelemetry(): once it returns, no new Room gets a scope, the core accepts nothing more, and no Room starts anothergetStats()or submits a report (a request already in flight is dropped); OS observers and the WebRTC log sink are released shortly after; everything unsent, including a cache from an earlier launch, is deleted; not remembered across launchesLiveKit+DeviceHelpers.swiftUIApplicationlookup also returns the app state (initiallk.device.app_state)UIDevice.isBatteryMonitoringEnabled = trueand leaves it on for the process, also after opt-out; restoring it could switch off monitoring the app enabled meanwhileDataTrackE2EE, test support, Benchmarks tokenFailed(reason:),VideoGrants.agentPackage.swift,Package@swift-6.2.swift,Benchmarks/Package.swiftLIVEKIT_UNIFFI_PATHswitches livekit-uniffi to a local build (cargo make swift-package-debug), else the released packageci.yamlotelcol-contribnext to the dev server for the telemetry e2e testEvents
13 of 19 SPEC signals fully covered, 5 partially (not exposed by the OS on some platforms, or capture failures outside publish), 1 skipped (smoke-test only).
Event coverage table
lk.connectspan (+ checkpoints)Room.connect, via the existingSpancheckpoints:ws_open·signal·join_recv·pc_created·offer_sent·answer_sent·engine·pc_connected·room_connected(+ customearly_pc_created)lk.reconnectspanattempt <n> <mode>per attempt; reasonpublisher_failed/subscriber_failedfor a failed peer connection,signal_disconnected,network_changed, elsetransport_failedlk.publishspanLocalParticipant._publish, nested under the ambient span while that span is openlk.subscribespanset(subscribed: true); subscribed / failed / unsubscribed / unpublished; first media seen by the core at its 1 s pollslk.rtc.stats.samplegetStats()per peer connection (publisher + subscriber), paced by the core; a connection with no answer within 5 s is skippedlk.room.disconnectedreconnect_failed, otherwise theLiveKitErrorlk.telemetry.reportcustom.<name>Room.emitTelemetryEventLoggable) warnings/errors and WebRTC errors from Swift; the Rust core copies its own warnings and errors with the default logger (OSLoggerwithffi: true, which starts the core's log forwarder); Swift never forwards those entries itself, so nothing is counted twicelk.device.thermal.changedProcessInfo.thermalStateDidChangeNotificationlk.device.low_power.changedNSProcessInfoPowerStateDidChange(macOS 12+)lk.device.app_state.changedAppStateListener: background/foreground; sleep/wake on macOSlk.device.memory.changedDispatchSourcememory pressurelk.device.network.changedNWPathMonitor: wifi / cell / wired / other / unavailable, expensive, constrained; VPN and Bluetooth tethering are not told apart by the pathlk.device.battery.changedUIDeviceon iOS/visionOS; not exposed by the OS on macOS/tvOSlk.device.audio_route.changedAVAudioSessionon iOS/tvOS/visionOS; no audio session on macOSlk.device.audio.interruptionAVAudioSessionon iOS/tvOS/visionOS; no audio session on macOSlk.device.capture.failedlk.pingLocal testing
Try it against a local OTel backend (LGTM) in a few minutes: the e2e test runs a whole call and prints its Rooms' trace ids.
Commands
Then open http://localhost:3000 → Explore:
lk.connect(with its checkpoints),lk.reconnect,lk.publishon the publisher;lk.connect,lk.subscribe(first_media) on the subscribers.{service_name="livekit-client-swift"} | trace_id="<id>"for that Room's stats windows (lk.rtc.stats.sample),lk.room.disconnected,custom.e2e.checkpointand SDK warnings/errors;{service_name="livekit-client-swift"} | otel_event_name=~"lk.device.+"for the device events.Without
LK_TELEMETRY_ENDPOINTthe same test posts to the collector CI runs (otelcol-contrib --config Tests/LiveKitCoreTests/Telemetry/otelcol.yaml) and asserts on what it wrote.