Skip to content

CLAUDE.md

Latest commit

 

History

History
183 lines (157 loc) · 11.6 KB

CLAUDE.md

File metadata and controls

183 lines (157 loc) · 11.6 KB
Raw

ClientiOS — Swift iOS Device Client

Native iOS device client for the Splintercom system — plays the same device role as ClientPython (OAuth2 device flow + WebRTC camera streaming), so an iPhone/iPad can be repurposed as a CCTV/doorbell camera instead of needing a Raspberry Pi.

Tech Stack

Layer Technology
Language Swift 6 (strict concurrency)
UI SwiftUI (iOS 15+)
WebRTC stasel/WebRTC (prebuilt Google libwebrtc xcframework via SPM)
Toolchain xtool (github.com/xtool-org/xtool) — builds/installs without Xcode/macOS
Package Mgmt Swift Package Manager
Persistence Keychain (OAuth tokens), UserDefaults (server settings + device preferences)

Directory Structure

ClientiOS/
├── Package.swift                 # swift-tools-version 6.0; two targets, see below
├── xtool.yml                     # bundleID, infoPath (Info.plist fragment)
├── Info.plist                    # NSCameraUsageDescription, orientation, etc.
├── Sources/
│   ├── ClientiOSCore/            # Portable Foundation logic — builds/tests on Linux
│   │   ├── Models/                # DeviceSettings, TokenSet, DeviceAuthorization DTOs,
│   │   │                          # TelemetryEvent, TurnCredentials, SignalingMessage,
│   │   │                          # DevicePreferences
│   │   └── Services/              # DeviceAuthorizationClient, TelemetryClient,
│   │                              # TurnCredentialsClient, SignalingClient, HTTPTransport
│   └── ClientiOS/                 # App target — SwiftUI, WebRTC, Keychain (Apple-only)
│       ├── ClientiOSApp.swift      # @main, scenePhase-driven start/stop, idle timer
│       ├── ContentView.swift       # Routes Settings -> Pairing -> Status/FullScreen
│       ├── DeviceClient.swift      # Orchestrator — mirrors ClientPython's PiClient
│       ├── DeviceIdentity.swift    # Hardware identifier + OS description (device_type/os)
│       ├── Services/               # TokenStore (Keychain), SettingsStore, DevicePreferencesStore,
│       │                          # CameraController, PeerConnectionCoordinator, WebRTCSendable
│       └── Views/                  # SettingsView, PairingView, StatusView, FullScreenView,
│                                  # CameraPreviewView, QRScannerView
└── Tests/ClientiOSCoreTests/      # swift-testing; runs on plain Linux Swift

Every file under Sources/ClientiOS/ is wrapped in #if canImport(SwiftUI) (or canImport(UIKit)/canImport(Security) where more specific) so swift test can still build the whole package graph on Linux without choking on Apple-only APIs — swift test builds one combined test product regardless of which target's tests you're running, so the whole graph must compile even on platforms that can't run the app itself.

Key Architecture

DeviceClient (DeviceClient.swift)

Direct Swift port of ClientPython/main.py's PiClient:

  • start()/stop() replace the SIGINT/SIGTERM-driven run()/shutdown() — driven by scenePhase in ClientiOSApp (.active → start + disable idle timer; .background/.inactive → stop) since the app is foreground-only by design (see README)
  • ensureValidTokens() → runSignalingLoop(), 5s retry-on-error, 30s heartbeat Task
  • runSignalingLoop() fetches TURN credentials, opens SignalingClient, and dispatches each SignalingMessage case to PeerConnectionCoordinator (offer/candidate) or DevicePreferencesStore (.displayMessage)

Signaling wire protocol (Models/SignalingMessage.swift)

Mirrors APIServer/live_stream/consumers.py exactly — .offer/.answer/.candidate always carry viewerChannel (backend-assigned, echoed verbatim); .status carries it only on viewer disconnect; .displayMessage (viewer → device only) carries no viewerChannel at all. ICE candidate strings are passed through verbatim, including the candidate: prefix — iOS's native RTCIceCandidate.sdp is already prefixed like the browser's, unlike ClientPython's aiortc-specific stripping workaround.

CameraController + PeerConnectionCoordinator (Services/)

One RTCVideoSource/RTCVideoTrack fed by whichever RTCCameraVideoCapturer matches the current DevicePreferences.cameraPosition (front/back). Native WebRTC lets a single RTCVideoTrack be added directly to multiple RTCPeerConnections, so — unlike ClientPython's SharedCameraSource (a manual per-viewer frame-queue fan-out, needed there because aiortc is one-track-per-connection) — one capture session feeds every viewer directly, plus the local preview renderer. Switching camera position swaps only the capturer, keeping the same track alive so existing viewers aren't dropped; the switch takes effect on the next start() call (next viewer connection), not instantaneously. PeerConnectionCoordinator also mirrors ClientPython's connectionState == "failed" handling: replaces the peer connection with a fresh empty one without renegotiating — a no-op reset that avoids errors from a dead PC, ported faithfully for parity rather than "fixed".

Two-way audio: MicrophoneController + WebRTCAudioSession (Services/)

MicrophoneController mirrors CameraController's lazy-track pattern but is simpler — WebRTC's audio pipeline has no capturer-session concept like RTCCameraVideoCapturer; creating an RTCAudioSource/RTCAudioTrack via the factory is enough, since WebRTC's Audio Device Module engages the mic automatically once the track is active on a peer connection. Owned at the DeviceClient level (like cameraController) so the track persists across reconnects. No mute control on the device side by design — mute, if wanted, is a viewer-side concern (see Frontend's ViewDevice.tsx). Added to every peer connection in PeerConnectionCoordinator.handleOffer with the same streamIds as the video track, so the viewer receives both grouped in one MediaStream (one <video> element plays both — no separate audio element needed on the Frontend). Receiving the viewer's incoming audio needs no extra code, unlike video: WebRTC plays it through the configured session/route automatically once negotiated.

WebRTCAudioSession.configureForVoiceChat() (called once in DeviceClient.init) overrides WebRTC's default audio routing — it defaults to the earpiece, not the speaker, which is wrong for a doorbell/kiosk device that needs to be audible without holding it to your ear:

let config = RTCAudioSessionConfiguration.webRTC()   // not `.webRTCConfiguration()` — renamed in
config.categoryOptions = [.defaultToSpeaker, .allowBluetoothHFP]  // Swift (confirmed via real
RTCAudioSessionConfiguration.setWebRTC(config)                     // compiler errors, not guessed)

No signaling/backend protocol changes were needed for two-way audio at all — it negotiates entirely within the existing SDP offer/answer exchange (an extra m=audio line), so SignalingMessage is unchanged.

TokenStore (Services/TokenStore.swift)

Keychain-backed (kSecClassGenericPassword) equivalent of ClientPython's 0600 tokens.json. TokenStatus.compute(from:now:) (in ClientiOSCore, portable/testable) mirrors PiClient.check_token_status.

DevicePreferences vs DeviceSettings — two separate local stores

  • DeviceSettings (SettingsStore, UserDefaults) — server connection info: HTTP/WebSocket base URLs, OAuth client ID. This is the QR-shareable schema (AddDevice.tsx encodes it as JSON with matching field names — httpAPIBaseURL, websocketAPIBaseURL, oauthClientID).
  • DevicePreferences (DevicePreferencesStore, UserDefaults) — local-only, physical-device preferences: cameraPosition (front/back), displayMessageEnabled/displayMessageText, fullScreenModeEnabled. Deliberately NOT part of the QR schema. Custom Codable conformance (decodeIfPresent + defaults for every field) so previously-stored preferences don't reset entirely when a new field is added — worth preserving as a pattern since this struct keeps growing. displayMessageEnabled/displayMessageText are remote-only now — SettingsView no longer has a local toggle/text field for them (removed as unnecessary complexity); the only writer left is DeviceClient's .displayMessage signaling case (a viewer pushing a message to the device).

Full-screen / "kiosk" mode (Views/FullScreenView.swift)

fullScreenModeEnabled is a separate persisted toggle from displayMessageEnabled — entered via a button on StatusView (shown only when cameraPosition == .front), exited via FullScreenView's 'X'. Independent of whether a message happens to be showing, so remote display_message updates switch what's shown within full-screen mode without needing to physically re-enter it.

Unregistering (SettingsView's "Unregister Device")

DeviceClient.unregisterDevice() is a full "start completely from scratch" reset: stops the run loop, clears the Keychain token (TokenStore.clear()), AND clears DeviceSettings (SettingsStore.clear()) — so ContentView's routing falls all the way back to SettingsView needing a fresh QR scan, not just back to PairingView. Confirmed via an alert before running (destructive, and per its message, local-only — doesn't delete the device server-side; use the web dashboard's delete button for that).

QR-based setup (Views/QRScannerView.swift)

AVCaptureMetadataOutput-based scanner (not VisionKit's DataScannerViewController, which needs iOS 16+ and would break this project's iOS 15 minimum). Scanned JSON decodes directly into DeviceSettings via JSONDecoder — no separate parsing format, since the Frontend encodes the exact same field names.

xtool-specific notes

  • xtool.yml's infoPath merges a plist fragment over xtool's auto-generated base Info.plist (your keys win on conflict) — confirmed by inspecting xtool/ClientiOS.app/Info.plist after a build, not guessed.
  • Non-Sendable ObjC delegate protocols (AVCaptureMetadataOutputObjectsDelegate, RTCPeerConnectionDelegate) need explicit Swift 6 concurrency isolation — see QRScannerView.swift's , @MainActor AVCaptureMetadataOutputObjectsDelegate conformance syntax and WebRTCSendable.swift's @unchecked @retroactive Sendable extensions on WebRTC's types.
  • URLSessionWebSocketTask compiles on Linux but crashes at runtime ("WebSockets not supported by libcurl") — confirmed working correctly on a real iPhone (Apple's implementation isn't libcurl-based). Don't trust a Linux build alone to validate signaling.
  • Installing via xtool install under a free/personal Apple Developer account may require revoking an existing "iOS Development" certificate first (personal accounts are limited to one active dev cert) — this invalidates other apps signed with that cert until rebuilt, so confirm with the user before doing it.
  • xtool launch's automatic launch (attaches a debugserver) can fail with DebugserverClient.Error.unknown on older iOS versions (seen on 15.8.3) — likely a Developer Disk Image mismatch; launching manually from the home screen is unaffected.

Testing

# Pure-logic tests (Models, Services with no WebRTC/UIKit dependency) — runs on Linux
swift test

# Real iOS build (requires the Darwin SDK: `xtool sdk install`)
xtool dev build

# Install + run on a connected device
xtool install xtool/ClientiOS.app
xtool launch com.intercom.clientios   # or launch manually from the home screen

Verified vs. unverified

See README.md's "Real-device verification status" section for what's been confirmed on physical hardware vs. only compile-checked.