README.md

Hex for iPhone

Status: Internal TestFlight app (HEXMobile), custom keyboard extension (HEXKeyboard), and Live Activity + Control Center widget extension (HEXWidgets). The containing app records a generation-scoped temporary WAV, transcribes on-device with Apple Speech (SpeechTranscriber on iOS 26+) or FluidAudio Parakeet V2 (iOS 18+), and removes each capture's file after inference, cancellation, or failure. The keyboard extension sends commands and inserts completed text over the shared App Group (group.com.kitlangton.hex.mobile); it never accesses the microphone, speech model, or network.

Capabilities

• One-tap keyboard dictation (HEXKeyboard):
  • Setup: The keyboard needs Full Access to reach Hex through the App Group. Without it, the keyboard shows the Settings path and an Open Hex setup button (hex-dictation://keyboard/setup). The app's setup card shows live status for the keyboard, Full Access, the microphone, and the speech model, and opens Hex's page in Settings. Full Access shows as on once the keyboard has run with it (keyboard.fullAccessSeenAt).
  • Opening Hex: SwiftUI Link, openURL, and extensionContext.open do nothing in a keyboard extension, so the keyboard opens Hex through the UIApplication at the end of its responder chain (ContainingAppOpener).
  • Cold start: Tapping Dictate opens the containing app (hex-dictation://keyboard/record?...) with a 10-second handoff window, starts recording, and opens the selected host app's URL scheme (messages://, slack://, claude://, chatgpt://, mobilenotes://, discord://, mail://, x-web-search://, etc.) using public UIApplication.shared.open(_:). Because iOS keyboard extensions cannot inspect the host app's bundle ID without private APIs, you pick your usual return app once on the ↩ chip (or jump bar), and apps without a registered URL scheme fall back cleanly within ~1s to the bottom-edge swipe bar and quick-jump chips. Edit apps… in the app's return menu picks and orders the apps the keyboard tray and jump bar offer.
  • GPU Dot Scope (HEXShared/DotScope.swift, DotScopeShader.metal): Both HEXMobile and HEXKeyboard render Hex's desktop menu-bar braided oscilloscope (src/dot_scope.rs, src/rune_hud.wgsl) at native display refresh via CAMetalLayer + CADisplayLink, pausing at 0% CPU/GPU once settled in idle. Lock Screen / Dynamic Island Live Activities render a matching Canvas dot-matrix badge (HexDotMatrixBadge) because WidgetKit archives static view snapshots rather than hosting live CAMetalLayers. Meter polling stays alive through a temporary non-recording state of the same capture; it resumes app levels and keyboard snapshots when the recorder is ready again. Capture completion, interruption, and Stop end polling.
  • Warm session (Off, 5m, 15m, 1h): Picking a duration on Keep mic warm starts an AVAudioEngine input session immediately (default 15m) so subsequent taps on Dictate record directly inside the keyboard with zero app switches; picking Off or tapping the header timer ends it.
  • Styles: Polished (sentence case + punctuation + filler trim), Casual (lowercase chat without trailing period), Notes (spoken "bullet" / "next line" markers), and Verbatim, with cursor-aware insertion spacing and repeating backspace.
  • Full keyboard Hex key (Experiments › Full keyboard): A tap does nothing, so a stray one while typing is harmless. A still hold of 0.35 s starts dictation in a warm session (or opens Hex without one), with a haptic. Moving more than 10 pt before then opens the style and Return to menu; releasing over a row picks it, and releasing elsewhere leaves the menu open for taps (HexKeyGesture). VoiceOver activation dictates, and the Dictation styles action opens the menu.
• Resilient capture & expiry contract (DictationController):
  • Monotonic jobGeneration, retained transcriptionTask, and unique per-capture WAV filenames ensure an expiring warm session never drops an already-accepted transcription (deliverable for 30s), while a stale winding-down inference can never overwrite a newer capture or delete its audio file.
  • Phone/FaceTime/Siri audio interruptions finalize captures >= 300 ms so spoken words are not lost, cancel shorter taps cleanly out of Listening, and resume warm keepalive when iOS signals .shouldResume. Bluetooth HFP input is opt-in (Use Bluetooth headphones, default off) so AirPods stay in full-fidelity output by default.
• Action Button, Control Center, Back Tap & Live Activity:
  • DictateWithHexIntent (AudioRecordingIntent + LiveActivityIntent) and HexDictationControl (ControlWidget) start recording on first press with a Dynamic Island / Lock Screen Live Activity (Stop and Cancel buttons) and transcribe + copy to the clipboard on second press.

Build & Test

Prerequisites: Xcode 26+ and XcodeGen 2.46.0+.

cd ios
xcodegen generate
xcodebuild test -project HEXMobile.xcodeproj -scheme HEXMobile -destination "id=<SIM_UDID>"

Simulator vs. Physical iPhone Proof Limits

• Verified in Simulator (HEXMobileTests + --snap-state scenes): Compilation across HEXMobile, HEXKeyboard, and HEXWidgets; App Group command/snapshot framing; the DictationController generation/expiry race and 30s post-close transcript delivery; 10s handoff window + URL-scheme auto-return fallback; interruption handling; meter recovery after injected startup and mid-capture recorder unavailability (including keyboard levels and heartbeat); deterministic style formatting; and UI layouts.
• Requires a physical iPhone: Real AVAudioSession background keepalive survival across lock/app switches, live orange indicator behavior, on-device SpeechTranscriber / Neural Engine Parakeet latency, URL-scheme return into installed third-party apps (Messages, Slack, Claude, ChatGPT), Dynamic Island Live Activity interaction, and hardware Action Button invocation. Transient AVAudioRecorder.isRecording changes during warm handoff or Bluetooth route changes have not been reproduced on hardware; the meter recovery tests inject that condition through ActiveAudioCapture.