A voice-first AI companion for Windows that sees your screen, points at things, teaches you software and can operate your PC for you. Built for people who can't use a mouse or keyboard comfortably, and for anyone learning complex software.
Lumen lives in the tray. Hold a hotkey (or say "hey lumen"), ask for something, and it answers, points at the right button, walks you through a lesson, types for you, or does the whole task while you watch. One AI key is all it needs; speech, wake word, OCR and dwell run locally.
| You say… | Lumen does… |
|---|---|
| "What's the weather in Larnaca and what time is it?" | Splits the question and answers both in one card, spoken if you like. |
| "Where is the compose button?" | Dims the screen and highlights the real control. |
| "What's this?" (pointing at something) | Explains the control under your pointer, on any monitor. |
| "Teach me Blender." | Lists lessons for the app you're in; pick one by number. |
| "Show me how to add a keyframe." | Runs a lesson: points, waits for you, checks the step, gives more help if you stall. |
| "Write an email to my boss that I'm quitting." | Announces the plan, opens Gmail, fills subject and body. Stops before Send. |
| "Find hotels in Nice for next weekend." | Researches the web and shows answer cards with photos, prices, ratings and sources. |
| "The cheapest one" → "book it" | Picks from the cards and books it while you watch. Payment and the final click stay yours. |
| "In the background, find internships at Exness." | Works on it as a background task while you keep using the PC, and tells you when it's done. |
| "Show numbers" → "click 5" | Numbers every clickable thing and clicks it, instantly, without calling the AI. |
| "Read this aloud" / "Summarize this page" | Built-in skills that run on whatever is on screen. |
| "Top news today" → "open the second one" | Short brief from free news feeds, with numbered sources you can open by voice. |
| "Click this… move this there" (while pointing) | Point and say: combines your words with where the pointer or gaze was. |
| "Summarize my inbox" / "Reply to this saying I'm in" | Works in Gmail, the new Outlook and classic Outlook. Drafts only; sending always asks first. |
| "Make a Word doc with a packing list." | Creates Word, Excel, CSV, PDF, Markdown or HTML files in Documents\Lumen, then opens them. |
| "Summarize this file" (pointing at it in Explorer) | Reads the file under your pointer, or one you dropped, and can reformat it into a new file. |
| "Turn on brief mode." | Reply styles change how Lumen talks (brief, teacher, friendly, formal, or your own). |
| "Every weekday at 9, read me my calendar." | Creates a routine that runs on that schedule. |
| "Undo that." | Reverses Lumen's last actions where possible, and says what can't be undone. |
| "What changed?" | Describes what's different on screen since your last command. |
| "What did you just do?" | Reads back the last actions from the local audit log. |
| "Stop" or Escape | Cancels whatever is running, including the model call and pending clicks. |
- Works in any app. No per-app setup: when an app or task is unfamiliar, Lumen looks up how it works (free official help first, your provider's web search if you allow it), finds the named menus and buttons on screen, and remembers what worked for next time.
- Accurate pointing. Windows UI Automation finds real buttons and fields; when an app exposes little, Lumen draws numbered marks, reads the screen with Windows OCR and takes a zoomed second look before it clicks.
- Agent mode. Multi-step tasks announce a plan, give you a few seconds to cancel, then run an observe → act → verify loop. It prefers UI Automation so your real pointer usually stays put (a "ghost cursor"), and it never retypes text it already entered. Hard caps on steps, time and cost.
- Rich answers. Research questions ("best ramen in Larnaca", "compare these three laptops") end in answer cards with photos, prices, ratings and numbered sources. Prices and ratings are only shown when Lumen actually read them on a source page. Follow up by voice: "the cheapest one", "tell me more about the second", "compare them", "save it", "more like this". Cards open full size as a grid or a sortable table.
- Book it for me. "Book the second one" opens the site and fills in the booking as a supervised agent task. Logins, captchas and card checks are handed back to you, Lumen never types into payment fields, and the final Book / Pay / Place order click always asks first, with the price it just read from the page.
- Background tasks. Say "in the background…" or "keep an eye on…" and the work runs in parallel without touching your mouse or keyboard. Results land in the Tasks list on Home with a tray badge.
- Helpers inside a task. A background or agent task can hand focused jobs to small helpers (researcher, reader, writer, checker) that work at the same time, inside the same safety rules, budget and cancel button. Their work shows as one group in the task chat; set how many run at once and which model they use in Settings → Automations → Helpers.
- Buddies. Make your own named helpers ("make a buddy that checks my mail every weekday at 8") by voice or in Settings. Call one by name ("Inbox Buddy, what's new?"), or let it run on its schedule. Each buddy has its own instructions, permissions, model, monthly budget and notebook; while it works on screen the cursor buddy takes its colour and name. Share them as
.lumenfiles. - Task chat. Click any task (background, agent or Claude Code) to watch its conversation live, see every step it took, steer it with a message, answer its questions, pause or stop it.
- Safety first. Every action is rated low / medium / high risk. Risky ones need your OK ("yes", "always" for this site, or "no"), sending, deleting and buying always ask (Send, Delete and recipient checks also work in Dutch, German, French and Spanish Gmail and Outlook), a recipient or personal detail you never said is shown on the confirm card, dangerous shortcuts (Run dialog, terminals) are blocked, passwords and secrets are redacted, and text on screen is treated as data, never as instructions. Every action is written to a local audit log.
- Multi-monitor and any display scaling. Highlights and clicks land in the right place on every screen.
- Files in and out. Drop a file (PDF, Word, Excel, PowerPoint, CSV, text, image) on the bar or point at one in File Explorer and ask about it; Lumen can also create Word, Excel, CSV, PDF, Markdown and HTML files, convert and reformat files into new ones, and attach them to an email. Originals are never changed without asking.
- Integrations: browse a catalog of well-known MCP servers (Notion, Linear, GitHub, Stripe, Supabase, Microsoft Learn, Playwright and more) and add them with one click and a browser sign-in, or add any MCP server by URL or command. Every tool call goes through the same safety gate; tokens are stored encrypted.
- Bring your Claude Code setup: import Claude Code plugins, marketplaces, skills and output styles from a link, a folder or your own
~/.claude, including marketplace plugins that live in other GitHub repos. Command arguments and connector settings come along; hooks are never imported. - Automations: "every weekday at 9…", "in 20 minutes…", "when a PDF lands in Downloads…", "when I open Excel…", at login, when idle or back online. They run in the background, never take your mouse or speak while you're away, can optionally wake Lumen through Windows Task Scheduler, and risky steps need your pre-approval. Manage them in Settings → Automations.
- Claude Code by voice: open a project, give Claude Code a task, hear what it's doing and answer its questions hands-free, using your own
claudelogin. An optional autopilot answers routine questions for you, with a hard deny list. Coding skills (from a docs link, an import, or a description) are suggested from your project's dependencies, e.g. Next.js and better-auth, and loaded into the session without touching your repo.
- Free local speech recognition (sherpa-onnx) by default, offline in English, Spanish, German and French; cloud Whisper is optional and covers more languages.
- Replies in your language: English, Spanish, German, French, Italian, Portuguese and Dutch, or auto-detect.
- Offline wake word ("hey lumen") and voice cancel, with adjustable sensitivity and microphone choice.
- Hold or tap the hotkey; double-tap for conversation mode, where every sentence is a question until you stop.
- Spoken answers with Windows voices, sentence by sentence, with optional barge-in (talk over it to interrupt).
- Dictation anywhere: light cleanup that never drops your words, a personal dictionary that learns from your corrections, and a guard that never presses Enter in terminals.
- Dictation that keeps up: "Tuesday, actually Wednesday" types Wednesday; spoken lists, numbers, dates and addresses are formatted; each app gets its own style; select text and say "make this friendlier"; snippets, whisper mode, coding mode ("camel case user name"), "send it", mouse-button push-to-talk, history, notes ("take a note…") and local stats.
- Local voice commands: "show numbers", "click 5", "show grid", "scroll down" and more run instantly, with no AI call. Lumen steps aside when Windows Voice Access or Dragon is running.
- Point and say: "click this", "move this… there", "what's that?" use where your pointer or gaze was as you spoke.
- Community labels: unnamed buttons get a clear name your screen reader can use, shareable with others.
- Face gestures (opt-in, local): open your mouth to click, raise your eyebrows to scroll, tilt your head to pick, with a calibration wizard. Camera frames never leave the PC.
- Mouse grid, dwell clicking with a click-type palette (left, right, double, drag) and tremor smoothing that suits eye-gaze and head pointers, plus an on-screen keyboard.
- Switch scanning with a scan ring, menu and scan keyboard for one- or two-switch users.
- Screen reader output through NVDA, JAWS or Narrator, focus narration, "describe the screen" and "read this".
- Simple mode with a calmer bar and plain-language answers.
- Follows Windows text size, reduced motion and high contrast; adjustable UI scale, timings, captions and accessibility profiles.
- App packs for Windows, Blender, DaVinci Resolve, Excel, OBS Studio, VS Code, Photoshop, GIMP, Premiere Pro and Figma: glossary, shortcuts, screen regions and 50+ ready-made lessons.
- Lessons with a hint ladder: point, wait, check, explain why, and "do it for me" when you'd rather watch.
- "Show me how" generates a lesson on the spot for anything a pack doesn't cover.
- Record my steps: do a task once and Lumen turns it into a lesson you can replay or share.
- Practice challenges per app, with feedback and a streak.
- App helpers for Blender and OBS read the app's real state, so lesson checks don't have to guess from the screen.
- Progress is saved, so "continue where we left off" works across sessions. A local learning journal answers "what did I learn this week?"
- Lessons from tutorials: paste a transcript, a web page link or a subtitle file and Lumen drafts a lesson for the real app.
- Share packs and lessons as a single
.lumenfile, e.g. a teacher or helper sending one to a learner; install community packs from a file or link. - Reading level: plain, standard or expert explanations; say "explain simpler" or "more detail".
Skills are small, shareable abilities written as a SKILL.md, optionally with fixed steps that run without the AI. Lumen ships starter skills (read this aloud, summarize this page, reply to this email, screenshot and explain, make text bigger here, morning briefing, clean downloads, export for YouTube, fill this form from my profile) and triggers them by phrase or app. Make your own by saying it ("when I say 'morning', open my mail and calendar"), by doing it once while Lumen records, or with "save that as a skill" after a task went well. Or just describe it: "make a skill that…" writes a full skill for you to review, Lumen offers to save tasks you repeat, and "change my morning skill to also open Slack" edits one by voice. Reply styles are skills too ("turn on teacher mode"). Install and share them from Settings, which also shows each skill's run history.
Optional, local, and off by default (Settings → Smart helpers), except undo:
- Focus mode: dims everything in a busy app except what you need right now. "Only show the timeline", "show everything".
- Undo: "undo that" or "undo the last 3 things" reverses Lumen's actions newest first and says honestly what can't be undone (like a sent email).
- Shortcut coach: notices when you keep using a menu for something with a shortcut and suggests it once, or a voice command if keys are hard for you.
- Comfort: notices when you seem tired and offers slower timings or bigger targets. It never changes a setting without asking.
- Error rescue: spots error dialogs and offers to explain them in plain words, and fix them with your OK.
- What changed? describes the difference since your last command, for blind and low-vision users.
- Home flyout with an ask box, suggestions, recent questions, background tasks and quick toggles.
- First-run setup with profiles, one API key, voice files and a practice round.
- Settings for voice, accessibility, buddy and look, models and keys, memory, lessons, skills, connectors, background and routines, smart helpers, app helpers, Claude Code and privacy (eight themes including a custom one); no JSON editing.
- Memory you can see, edit and delete, plus a private mode.
- Usage and cost: Settings → Usage shows where tokens and money go per feature, task, automation, buddy, skill and model, with monthly limits that pause automations and buddies, a CSV export, and voice questions ("how much did I spend this week?").
- Automatic updates from GitHub Releases (installed build; the portable build tells you when one is out).
- Any AI provider: Anthropic, OpenAI, Google Gemini's free tier (opt-in, with its privacy terms shown), OpenRouter, Groq, Mistral, DeepSeek, Together or any OpenAI-compatible service, and local models through Ollama or LM Studio, auto-detected, with a "local only" mode. Pick a provider and model per job.
Requirements: Windows 10 22H2+ or Windows 11 (x64), a microphone, and one AI key (Anthropic, OpenAI, a free Gemini key, or another compatible service) or a local model in Ollama or LM Studio. No Python, no Admin rights.
- Download
Lumen-Setup-<version>.exefrom Releases, orLumen-<version>-portable.exeto run without installing. Optionally check it againstSHA256SUMS.txtfrom the same release:Get-FileHash .\Lumen-Setup-<version>.exe. - Run it. The builds are not code-signed, so SmartScreen says "Windows protected your PC": click More info, then Run anyway. Lumen installs for your user only, in
%LOCALAPPDATA%\Programs\lumen, and opens setup. - Paste your API key when asked. It is encrypted with Windows (DPAPI) in
%USERPROFILE%\.ai-overlay\keys.dat, never in the config file. - Hold Ctrl+Shift+Space, say what you want, let go.
Speech recognition and wake-word models (about 100 MB and 18 MB) download on first use and are checked against pinned SHA-256 hashes.
Settings, keys and models live in %USERPROFILE%\.ai-overlay; logs in %APPDATA%\Lumen\logs (Settings → About → Open logs folder, or Export diagnostics for a zip without keys). Uninstall from Windows Settings → Apps; it asks whether to remove your settings, keys and models too. The portable build never adds start-at-login entries and uses the same settings folder.
You only need the first one. Everything else is optional, and every shortcut can be changed or turned off in Settings → Accessibility → Shortcuts.
Talking to Lumen
| Press | What happens |
|---|---|
| Ctrl+Shift+Space | Hold it, speak, let go. Or tap once to start and Lumen stops listening when you go quiet. |
| Ctrl+Shift+Space twice | Conversation mode: keep talking back and forth until you say "stop" or press Escape. |
| Ctrl+Shift+D | Dictate: whatever you say is typed into the text field you're in. |
| Ctrl+Shift+H | Open Home (ask box, recent questions, background tasks, quick toggles). |
| Escape | Stop whatever Lumen is doing. |
Hands-free helpers (the F-key row, all with Ctrl+Shift)
| Press | What happens |
|---|---|
| Ctrl+Shift+F1 | Show what you can say. |
| Ctrl+Shift+F2 | Jump to the assistant bar (to type instead of talk). |
| Ctrl+Shift+F3 | Repeat the last answer. |
| Ctrl+Shift+F4 | Pin the answer so it stays on screen. |
| Ctrl+Shift+F5 | Close the answer. |
| Ctrl+Shift+F6 | Show numbers on everything clickable (then say "click 5"). |
| Ctrl+Shift+F7 | Show the mouse grid (say 1–9 to zoom into a box, then "click"). |
| Ctrl+Shift+F8 | Pause or resume dwell clicking. |
| Ctrl+Shift+F9 | Cancel (same as Escape, for when Escape is taken). |
| Ctrl+Shift+F10 | Show the on-screen keyboard. |
F3–F5 only work while an answer is showing, and F8 only while dwell clicking is on, so they don't steal keys from your other apps the rest of the time.
During a lesson (only active while a lesson runs)
| Press | What happens |
|---|---|
| Ctrl+Alt+→ / ← | Next step / previous step |
| Ctrl+Alt+H | More help with this step |
| Ctrl+Alt+D | "Do it for me" |
You can also just say "next", "back", "help" or "do it".
┌──────────────────────────────┐ typed IPC ┌──────────────────────────────┐
│ Electron main (src/main) │ ◄───────────► │ Renderers (src/renderer) │
│ router, agent mode, lessons, │ │ assistant bar, screen layer, │
│ skills, memory, safety gate, │ │ Home, settings, onboarding │
│ background tasks, audit log │ └──────────────────────────────┘
└──────────────┬───────────────┘
│ NDJSON over stdio
┌──────────────▼───────────────┐
│ lumen-native (native/, Rust) │ hotkeys, SendInput, DXGI capture, Windows OCR,
│ │ UI Automation, dwell, switch hook, screen reader output
└──────────────────────────────┘
The main process decides, the renderers draw, the Rust helper touches the OS. Speech recognition and the wake word run in a worker thread with sherpa-onnx. Shared types and IPC contracts live in src/shared/; app packs and built-in skills live in skills/. See CLAUDE.md for the full module map.
Default models: Claude Sonnet 5.5 for answers and planning and Claude Haiku 4.5 for quick checks; with only an OpenAI key, gpt-5-mini and gpt-5-nano. Override per role in Settings → Models.
Requirements: Windows 10 22H2+ or 11, Node.js 22, the Rust toolchain (rustup.rs) and one API key.
git clone https://github.com/JanoTheDev/lumen.git
cd lumen
npm install
npm run build:native # builds the Rust helper once
copy .env.example .env # add ANTHROPIC_API_KEY or OPENAI_API_KEY (or paste a key in Settings later)
npm run devnpm run dev # app + Vite dev server
npm run build # typecheck + bundle
npm run build:win # Rust helper + installer + portable exe in dist/, then package checks
npm run lint # ESLint
npm test # Vitest
npm run test:live # tests that call real APIs (needs a key)
npm run validate:skills # check app packs against their schema
npm run eval:router # intent router eval
npm run eval:dictation # dictation word-preservation eval
npm run eval:grounding # offline pointing eval
npm run size:report # bundle size budgets
cargo test --manifest-path native/Cargo.toml
cargo clippy --manifest-path native/Cargo.toml --all-targets -- -D warningsCI runs typecheck, lint, Vitest, the Rust tests and clippy, and the bundle size check on Windows for every push. Pushing a v<version> tag builds the installer and portable exe into a draft GitHub Release.
Last updated: October 2026.
Where things stand: every feature listed above is built, has automated tests (about 430 test files and over 5,240 tests, plus the Rust helper's own tests) and passes CI. Most of it has not yet been tried by hand in the real app on a range of PCs, so expect rough edges until the hand-test pass below is done. There is no tagged release yet.
Core
- Security: sandboxed windows, validated settings and IPC, risk-rated safety gate, local audit log, redaction
- Rust native helper (no Python), one-click per-user installer, portable build, auto-update
- Assistant bar, screen layer, cursor buddy, Home flyout, tray, first-run setup, rebuilt Settings, eight themes
- Any provider: Anthropic, OpenAI, Gemini free tier, OpenAI-compatible services, Ollama and LM Studio, local-only mode, a model per job
- Memory you can see and edit, private mode, usage and cost per day
- Diagnostics page in Settings: how long each step of your last question took
Pointing and doing
- Accurate pointing: UI Automation, numbered marks, OCR, zoom-in second look
- AI intent router, agent mode with plan, countdown, ghost cursor and caps
- Works in any app: how-to lookups, on-screen grounding, learned per-app notes
- Gmail and Outlook (new and classic): read, summarize, reply, write, search, attach; send and delete safety in five UI languages
- Web research that reads result pages as text and answers with sources
- Rich answer cards with images, prices, ratings and sources for research questions, with voice follow-ups
- Book it for me: supervised booking with a checkout guard (price on the confirm, payment fields left to you)
- Background tasks with a live task chat (steer, answer, pause, stop)
- Helpers inside a task: parallel researcher / reader / writer / checker sub-agents
- Buddies: named helpers you make, call by name or run on a schedule, with their own budget and notebook
- Usage tracking per feature, task, automation, buddy, skill and model, with monthly limits
- Automations: time, app, folder, idle, login and network triggers, optional wake when closed
- Undo, what changed, focus mode, shortcut coach, comfort, error rescue
- Files in and out: drop or point at a file; create Word, Excel, CSV, PDF, Markdown and HTML
Voice
- Local speech recognition, offline wake word, Windows voices, barge-in, conversation mode, seven reply languages
- Dictation anywhere: corrections, formatting, per-app styles, voice edits, snippets, coding mode, history and notes
- Read the web with me: summarize pages, news briefs from free feeds, open sources by voice
Accessibility
- Local voice commands, numbers, mouse grid, dwell click types, switch scanning, eye-gaze tuning, on-screen keyboard
- Point and say ("click this", "move this there")
- Screen reader output (NVDA, JAWS, Narrator), community labels, simple mode, text scale, reduced motion, high contrast
- Face-gesture input (webcam, local, opt-in), with a head pointer that moves the mouse as you turn your head
Teaching and skills
- App packs for ten apps, 50+ lessons, show me how, record my steps, practice challenges, saved progress, learning journal
- Blender and OBS app helpers for exact lesson checks
- Offline lesson-check accuracy test set (no false passes on the current cases)
- Record a lesson as a video: "record this lesson" saves the run with a step-by-step transcript in
Videos\Lumen - Lessons from tutorial transcripts, web pages and subtitle files; sharing as
.lumenfiles - Skills:
SKILL.mdformat, starter skills, make by voice, by recording, by description, or "save that as a skill" - Reply styles, Claude Code plugin import (also marketplace plugins from other GitHub repos)
Integrations
- MCP connectors with per-tool permissions, integrations catalog with browser sign-in
- Claude Code by voice with optional autopilot, coding skills per project
- Hand-test pass on real hardware: every surface at 100–200 % scaling, light / dark / high contrast, two monitors, NVDA and keyboard only, motion smoothness
- Pointing accuracy test set from real apps (Gmail, Outlook, Word, Excel, Slack, VS Code, browsers); the capture tool is ready
- Lesson-check accuracy on real screen recordings (the offline test set is done)
- First public release
- Optional paid streaming voices and transcription (OpenAI, Deepgram, ElevenLabs), always opt-in
- Offline spoken-language detection (a ~75 MB download)
- DaVinci Resolve app helper (needs a Studio licence to test)
- Driving the browser directly instead of through the screen
- A central community index: skills, packs and buddies are shared as
.lumenfiles and GitHub links instead - Code signing and the "uiAccess" build that works over admin windows (paid certificate required)
- winget package (waits for signing)
- Requests go straight from your PC to the AI provider you chose. Nothing is proxied.
- Speech recognition, wake word, OCR, UI Automation and dwell run locally.
- Screenshots are sent with a request and never written to disk.
- API keys come from
.env(development) or the encrypted key store (Windows DPAPI), never from the config file or logs. Passwords and secrets are redacted before anything reaches the model. - Memory is off by default, stays on your PC, and can be viewed, edited or wiped in Settings.
- The audit log stays on your PC, stores typed text only as a length and hash (a redacted copy only if you turn that on), and is pruned after 30 days by default.
- The camera is used only while face gestures are on; frames are processed locally and never stored or sent.
- Answer card images come from the page itself or Wikimedia Commons and are cached on your PC for 7 days (in memory only in private mode).
- News and page reading fetch only public https pages, respect robots.txt and never refetch one-time links.
- How-to lookups use free official help first; paid web search only runs if you turned it on, with a daily cap. What worked in each app is remembered locally.
- Google Gemini's free tier is opt-in: outside the EEA, UK and Switzerland, Google may use free-tier prompts (including screenshots) to improve its products. Use a paid key or a local model if that matters to you.
- Task transcripts, dictation history, notes and learned app notes stay on your PC, with secrets redacted.
- No telemetry or analytics. Crash dumps stay local.
Hotkey doesn't fire: another app may own the shortcut; pick a different one in Settings → General. Lumen does not need Administrator rights.
"No API key found": paste a key in Settings → Models (or, from source, add ANTHROPIC_API_KEY / OPENAI_API_KEY to .env and restart).
Wake word silent: turn it on in Settings → Voice; the model downloads once. Raise the sensitivity or pick the right microphone there.
Wrong window got the text: Lumen types into whatever is focused; click the field first, or say which app to use.
Something else: Settings → About → Export diagnostics, and attach the zip (it contains no keys) to an issue.
GNU Affero General Public License v3.0, see LICENSE. You can use, modify and share Lumen; if you offer a modified version to others over a network, you must publish your source under the same license.