Superlips

Read my lips. Silent dictation for macOS.
Download for Mac · Apple Silicon · macOS 14+

Hold a key, mouth words at your webcam, and the text is pasted wherever your cursor is. Superlips is a lipreading take on Superwhisper or Handy: it reads your lips instead of listening, so you can dictate in a library, next to a sleeping baby, or on a call without saying a word. While you mouth, a live transcript appears in a small overlay.

It runs entirely on your Mac. The engine and the optional LLM listen only on 127.0.0.1, nothing is uploaded, and the video of each dictation is deleted as soon as it's transcribed.

This is an experiment, not a product. It's an early prototype for Apple Silicon. It works more often than you might expect, but lipreading is hard and it gets plenty wrong (see Accuracy).

It started with The Subvocal Company, which is building a wearable that turns silently mouthed words into text. I wanted to know how far you could get with the webcam you already have: an open lipreading model and a small local LLM to clean up its guesses. It's open source because there's a lot of room to improve it (see Ideas), and pull requests are welcome.

If you just want dictation, start with a voice app instead. They're much more polished than this: Handy (free, open source, local, and what I use every day), Superwhisper (the original), or Wispr Flow (great onboarding).

Caveats so far

• It needs good light. In a dark room it can't see your mouth. Depth cameras like Face ID's, or just better webcams, might change that someday.
• Mouthing words feels weird. Talking to a computer takes getting used to, whispering to one takes longer, and silently mouthing is the strangest of all. Whispering also gets better results.
• Keep your head still. Moving around seems to throw it off, even though it tracks your face.
• Short phrases are guesswork. It needs full sentences for context.

Install

Download the DMG from Releases, open it, and drag Superlips to Applications. It needs an Apple Silicon Mac and macOS 14 or later. Ollama is optional but recommended.

The app itself is small. The first launch walks you through setup, the way Handy does, and downloads the rest:

1. Permissions. Allow Camera, and turn on Accessibility for Superlips. Accessibility is needed for the hotkey and for pasting; without it macOS silently drops the paste, so the text is left on the clipboard instead.
2. The engine and the model. The lipreading engine runs in its own copy of Python with PyTorch (about 1 GB), which Superlips installs with uv. The model is Auto-AVSR's weights (1.2 GB, checksummed). Both go in ~/Library/Application Support/Superlips, and nothing else on your Mac is touched.
3. Cleanup (optional). Pick a model you already have in Ollama, or download the recommended qwen3.5:4b from the same screen, and add custom words.

After that Superlips lives in the menu bar (the icon is a mouth). The engine is ready about 5 seconds after launch.

To uninstall, delete the app and ~/Library/Application Support/Superlips.

From source

Requirements: Xcode command line tools and Python 3.10–3.12.

git clone https://github.com/texjer/superlips && cd superlips
./setup.sh              # Python env in engine/.venv, builds a development app
open app/build/Superlips.app

A development build runs engine/ in place with engine/.venv, so Python edits take effect when you restart the engine (Settings → Advanced). It skips the engine install step.

Releasing

app/release.sh builds the self-contained app (engine code and uv inside), wraps it in app/build/Superlips-<version>.dmg, signs it with your Developer ID, notarizes and staples it. One-time setup: a Developer ID Application certificate (Xcode → Settings → Accounts → Manage Certificates) and xcrun notarytool store-credentials superlips --apple-id … --team-id …. The version is VERSION in app/build.sh; the engine's packages are pinned in engine/requirements.txt, and changing them makes existing installs reinstall on update. The icon is drawn by app/icon.swift.

Use

• Hold Right ⌥ and mouth a sentence; release to paste it.
• Tap Right ⌥ for hands-free mode: each phrase is pasted as soon as you pause (hold your mouth still for about 0.7 s). Tap again to stop.
• Esc cancels. Pressing any other key while ⌥ is held also cancels, so ⌥ shortcuts keep working.
• The overlay shows a mirrored camera view with a green border while your face is tracked. Turn off Show camera preview while recording to get a small red/green dot instead.
• The dictation key can be Right ⌥, Right ⌘, Right ⌃ or fn (Settings → General).

The window

Menu → History & Custom Words… or Settings… (⌘,) opens the main window. It has a Dock icon while it's open; close it and Superlips goes back to the menu bar.

• History: every dictation, newest first, with what the model read when the LLM changed it, timings, and a button that shows the mouth crops the model saw (kept for a day). That's the first thing to check when results are bad. Stored in ~/Library/Application Support/Superlips/history.jsonl.
• Custom words sit in a shelf on the History page, since that's where you notice a missed word; each history row also has a button to add one. Lips can't tell P/B/M or T/D/N apart, so a transcript word that reduces to the same lip shapes as a custom word is replaced with it (DEX → Tex, BOSTIC SIDE → Bostick site, engine/words.py). This works with or without the LLM.
• Models: download or delete the lipreading model; pick, or download, the cleanup LLM.
• Advanced: pause length and beam size, the engine log, restart the engine.

Accuracy

The model is Auto-AVSR: 19.1 % word error rate on LRS3, which is TED talks spoken aloud. Silent mouthing moves the lips less, so in practice:

• Whisper rather than mouth. Barely audible, well-enunciated whispering works noticeably better than pure silent mouthing.
• Face the camera with light on your face, at about an arm's length.
• Speak in full sentences: the language model and the LLM need context, and single words are close to guesswork.
• Lips can't tell P/B/M, F/V or T/D/N apart. The LLM cleanup pass fixes many of those using context.

On a 12 s TED clip with the audio removed, the raw output was "I have an interest in education actually what I find is everybody has an interest in education to do I think it's very interesting". The speaker said "…don't you, I find this very interesting".

How it works

app/     Swift app: menu bar, main window (SwiftUI), hotkey, camera, Vision face tracking,
         overlay, paste, model downloads, history
engine/  Python loopback server (127.0.0.1:5577): mouth crop → VSR → beam search → LLM cleanup

1. Capture (app/Sources/Camera.swift): 640×480 at about 30 fps. The luma plane is the grayscale frame. ClipWriter streams frames and face landmarks to a temp folder as they arrive, so the engine can decode a recording that's still going.
2. Face tracking (FaceTracker.swift): Apple's Vision framework gives the four points Auto-AVSR aligns on (eye centres, nose base, mouth centre), so no MediaPipe is needed.
3. Mouth crop (engine/crop.py): landmarks are smoothed, each frame is warped onto a mean face, and a 96×96 mouth patch is cut. This is Auto-AVSR's own preprocessing.
4. Motion gating (crop.py): mouth motion is measured on the stabilized crops. Stretches where the mouth isn't moving never reach the model, because given a still mouth it confidently invents fluent sentences. Holding still for 0.7 s marks a pause.
5. Recognition (engine/vsr.py, engine/fast_beam.py): a Conformer encoder, then a joint CTC/attention + Transformer-LM beam search over 5000 subword units.
6. Live loop (app/Sources/Dictation.swift): about every 0.3 s the app decodes what has been said since the last pause, for the overlay. At each pause, that phrase is committed and cleaned up.
7. Custom words (engine/words.py): look-alikes of your custom words are swapped in.
8. Cleanup (engine/cleanup.py): a small local LLM (default qwen3.5:4b through Ollama) fixes look-alike words using context and adds punctuation. It is told to change as little as possible.

Speed (M2 Max)

Clip	espnet beam search	fast_beam.py	+ LLM cleanup
3 s	1.8 s	0.4 s	+0.6 s
5 s	3.0 s	0.7 s	+0.6 s
12 s	8.4 s	2.1 s	+0.7 s

The reference espnet decoder spends its time in Python bookkeeping: a per-frame loop in the CTC prefix scorer, per-hypothesis state shuffling, and cross-attention keys and values recomputed for every hypothesis at every step. fast_beam.py does each step in a fixed handful of tensor ops:

• CTC: the recurrences are solved in closed form with logcumsumexp, in float64 on the CPU.
• Attention: cross-attention keys and values are computed once per utterance.
• Stopping: the search ends exactly when no running hypothesis can still win.

Its output is identical to espnet's, and peak memory drops from 37 GB to 2.6 GB on a 12 s clip. The encoder, decoder and LM run on MPS.

Testing without a camera

app/build/Superlips.app/Contents/MacOS/Superlips --prep talking-head.mp4 /tmp/clip  # Vision tracking
engine/.venv/bin/python engine/server.py /tmp/clip                                 # transcribe

Use H.264 or HEVC video: AVFoundation can't decode AV1 without hardware support.

Settings (environment variables for the engine)

The app sets the beam size and pause length from Settings → Advanced, and sends the LLM model and custom words with each request. Running the engine on its own:

Variable	Default
SUPERLIPS_LLM	qwen3.5:4b	Ollama model for cleanup, when the request doesn't name one
SUPERLIPS_BEAM	30	Beam size; below 30 the decoder starts looping phrases
SUPERLIPS_PAUSE	0.7	Seconds of stillness that count as a pause
SUPERLIPS_MODELS	~/Library/Application Support/Superlips/models	Where the weights are
SUPERLIPS_SPEAKING / SUPERLIPS_STILL	8 / 5	Mouth-motion thresholds
SUPERLIPS_DEVICE	auto	mps or cpu

Background: the state of lipreading (late 2026)

Model	LRS3 visual-only WER	Weights	Notes
Auto-AVSR (Ma et al. 2023)	19.1 %	Yes (non-commercial)	Used here
VALLR (ICCV 2025)	18.7 %	Code only	Video → phonemes → LLM; promising for a v2
USR 2.0 + diffusion LLM (2026)	~19.5 %	Adapters only	Needs a 7B LLM
VSP-LLM	~25 %	Yes	AV-HuBERT + LLaMA-2 7B
AV-HuBERT Large	~26–28 %	Yes	Self-supervised baseline

Related projects: Chaplin, the first local real-time lipreading demo and the source of the vendored inference code, and Telepathy.

Ideas

• Personal fine-tuning. Every accepted dictation is a labelled (video, text) pair, and models adapted to one speaker are far more accurate than speaker-independent ones.
• Audio-visual mode. Fuse a whisper-level microphone signal with the video; the same Auto-AVSR family has an audio-visual model at 0.9 % WER.
• Faster long dictations. Cache self-attention keys and values in the decoder and LM, which are currently recomputed over the whole prefix at each step.
• No Python. Port the encoder to Core ML or MLX.

Licence

The Superlips code is MIT (LICENSE). The vendored inference code and the model weights have their own terms, which are non-commercial for the weights; see THIRD_PARTY.md.