[Performance/Arch] Cross-Platform Latency Profiling & Optimization Roadmap (Windows / macOS / Linux)

[SentimentalK]

1. Context & Background

Currently, the ASR inference core across all three platforms (Linux, macOS, Windows) is identical and deterministic:

• Model: X-ASR 480ms streaming transducer (model.json locked to provider: "cpu", num_threads: 1, greedy_search)
• Runtime: Sherpa-ONNX v1.13.6 (CPU execution provider)
• Pipeline: Shared App::tick loop and streaming audio channel

Perceived latency differences (e.g. "Linux feels faster than Windows/macOS") are not attributable to ONNX GPU acceleration (inference runs entirely on CPU). Subjective variations are largely driven by host CPU single-core performance, OS audio capture buffer cadence, text injection event dispatching, and core loop polling.

---

2. Latency Bottleneck Breakdown & Investigation Points

#1. num_threads = 1 Sensitivity on Different CPUs (Highest Priority)

• num_threads: 1 in model.json is currently the most prominent performance knob.
• Certain Windows/macOS CPUs may suffer under single-threaded execution compared to high single-core IPC machines.
• Benchmark Goal: Evaluate num_threads = 1 vs 2 vs 4 measuring:
  • Time to First Partial
  • Partial-to-Partial latency
  • Real-Time Factor (RTF)
  • Active CPU usage %

#2. CPAL OS Default Buffer Sizes & Audio Callback Cadence

• AudioInput::start currently invokes device.default_input_config()? without forcing a low-latency buffer size.
• OS audio backends behave differently:
  • Linux (ALSA / Pulse / PipeWire)
  • Windows (WASAPI)
  • macOS (CoreAudio)
• Each platform decides callback chunk intervals independently (e.g., 1020ms vs 3050ms). If Linux delivers PCM frames earlier to Sherpa, it creates the illusion of faster recognition.
• Action: Add diagnostic instrumentation in AudioInput::start logging:
  • device_name, sample_rate, channels, sample_format, callback_frames, and callback_interval.

#3. Platform-Specific Text Injection Latency

• Windows: SendInput batch sends backspaces and UTF-16 characters in a single atomic syscall (fast).
• Linux: Clipboard synchronization + uinput Ctrl+V with slight sleep intervals.
• macOS: Multi-hop dispatch:
  $$\text{ASR Worker} \xrightarrow{\text{channel}} \text{MacUiCommand::InjectDiff} \xrightarrow{\text{CFRunLoop timer (15ms)}} \text{NSPasteboard} \xrightarrow{} \text{Cmd+V}$$
  • The main-thread runloop timer only drains every 15ms. Combined with App::run 15ms polling, macOS can accumulate 10~30ms of extra dispatch lag.
  • Optimization: Transition from 15ms timer polling to explicit runloop wake-up / dispatch to the main queue.

#4. Audio Callback Allocations & Heap Jitter

• extract_primary_channel(data, channels) creates a new Vec<f32> on every audio callback.
• I16/U16 paths perform double allocations (Vec<f32> conversion + channel extraction).
• Optimization: Replace per-callback allocations with pre-allocated ring/scratch buffers or single-pass conversion to minimize GC/allocator jitter.

#5. Event-Driven Wakeup vs sleep(15ms) Polling

• App::run() polls with std::thread::sleep(Duration::from_millis(15)).
• Future architectural evolution: Audio/action arrival directly triggers core loop wakeups instead of periodic polling.

---

3. Proposed Cross-Platform Latency Profiler

Implement structured timestamp profiling for streaming partials:

• T0: Audio callback captures hardware frame
• T1: App::tick receives AudioChunk
• T2: Sherpa decode begins
• T3: Sherpa decode emits partial result
• T4: Text injector starts diff execution
• T5: Injector completes keystrokes / paste

Profiling Matrix:

Stage	Linux (ms)	Windows (ms)	macOS (ms)
Audio Chunk Delivery ($T_0 \to T_1$)	?	?	?
ASR Inference ($T_2 \to T_3$)	?	?	?
Text Injection ($T_4 \to T_5$)	?	?	?
Total End-to-End ($T_0 \to T_5$)	?	?	?

---

4. Action Items & Milestones

• Instrument AudioInput::start with hardware buffer size & callback frequency metrics.
• Add latency profiler logging (T0..T5) under a --profile-latency feature flag or debug log.
• Run reproducible A/B benchmarks across Windows, macOS, and Linux for num_threads = [1, 2, 4].
• Refactor macOS UI command draining to event-driven main thread wakeup.
• Optimize audio.rs mono extraction buffer allocation.