diff --git a/.gitignore b/.gitignore index a3fdc22..2e22b3e 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,4 @@ __pycache__/ /.direnv/ /nix-flake-update.txt /result* +/.claude/ diff --git a/dev/README.md b/dev/README.md new file mode 100644 index 0000000..34e5ff1 --- /dev/null +++ b/dev/README.md @@ -0,0 +1,120 @@ +# Development stack + +A second Home Assistant, running in the agent VM, so voice work does not need a +rebuild of the real machine for every experiment. + + dev/run-ha.sh # start it (builds .#hass-dev if needed) + python3 dev/ha-onboard.py # once, on an empty config dir + python3 dev/ha-setup.py # configure it; idempotent + python3 dev/ha-ask.py "is the bed light on?" + dev/run-ha.sh stop + +Voice and model experiments, all driven through `dev/halib.py`. See +`dev/semantic-endpointing.md` for what they found: + + dev/turn-test.py # does it keep listening through a pause? + dev/silence-sweep.py # latency against pause tolerance + dev/stage-breakdown.py # where the milliseconds go + dev/e2e-compare.py # old settings against new + dev/eval.py 3 ... # scenario scores, state reset each run + dev/model-compare.py # models, end to end + dev/toolcall-stress.py # the tool-call path, repeatedly + dev/smart-turn-probe.py # the turn model's opinion, cut by cut + dev/smart-turn-eval.py # its accuracy on real labelled speech + dev/speculation-sweep.py # what speculating is worth, against the wait + dev/speculation-safety.py # a wrong guess must not act + dev/speculation-speech.py # a wrong guess must not be spoken + dev/stt-compare.py : # transcribers, side by side + dev/local-intent-test.py # what an alias is worth + dev/ollama-overhead.py # time spent before inference starts + dev/ollama-tap.py # what Home Assistant really sends ollama + dev/host-e2e.py # the real machine, audio in to audio out + dev/kokoro-stream-test.py # the synthesiser's protocol handling + +Run them with `dev/py`, which supplies websockets, numpy and ffmpeg; which +packages a bare `python3` happens to have is not something to depend on. + +`dev/speculation-speech.py` needs the stand-in synthesiser, added to Home +Assistant as a Wyoming entry on port 10211: + + dev/run-fake-tts.sh + +Anything that streams audio needs a transcriber in this VM, because the host's +is bound to loopback: + + $(nix build --no-link --print-out-paths nixpkgs#wyoming-faster-whisper)/bin/wyoming-faster-whisper \ + --model tiny-int8 --language en --uri tcp://127.0.0.1:10300 \ + --data-dir ~/ha-dev/whisper --download-dir ~/ha-dev/whisper + +`stt.demo_stt` cannot stand in for it: it accepts only stereo and the pipeline +sends mono. + +State lives in `~/ha-dev`, the token in `scratch/hadev/token.txt`. Delete the +directory to start over; the three scripts rebuild everything. + +## What runs where + +Only two things need the GPU, and they stay on the NixOS host: + +| | where | why | +|---|---|---| +| ollama | host | CUDA, and the model is 28 GiB | +| Kokoro TTS | host | CUDA | +| Home Assistant | **here** | no GPU; this is what we iterate on | +| faster-whisper, openWakeWord | either | CPU; ~90 ms to transcribe | + +The dev instance talks to the host's ollama at `192.168.122.1:11434`, which is +open on `virbr0`. The host's speech services are bound to loopback and are not +reachable from here; run local ones if a test needs them. + +## Things that cost an hour to find + +- The package is pinned to **nixpkgs-stable**, matching the host. The Silero + patch does not apply to Home Assistant 2026.8.2 in unstable. A dev instance on + a different version teaches you nothing transferable. +- `extraComponents` does not change the derivation. The NixOS module passes the + component dependencies through `environment.PYTHONPATH = package.pythonPath`, + which is why `.#hass-dev` is a wrapper that exports it. +- The module also always adds `defaultIntegrations`, including **frontend**. + Without it `hass_frontend` is missing, frontend setup fails, and Home + Assistant silently drops into **recovery mode** -- which ignores + `configuration.yaml`, so nothing loads and the failure looks like anything but + a missing frontend. +- Do not use `default_config:`; it pulls dhcp, go2rtc, logbook, my, ssdp and + stream, and taking everything it would have set up down with it when they are + missing. +- `pkill -f hass` matches the shell running it. Use the bracketed pattern in + `run-ha.sh`. + +## More things that cost an hour to find + +- **Websocket ids must increase.** Home Assistant rejects a lower id with + `id_reuse`, so `halib` hands them out centrally rather than letting callers + pick. +- **Do not truncate `hass.log` while Home Assistant holds it open.** The write + offset stays where it was and the file fills with nul bytes, so `grep` finds + nothing and the log looks empty. Restart it instead. +- **Subentry ids are not in the REST entry listing**, which reports only + `num_subentries`. They come from `config_entries/subentries/list` over the + websocket. +- **Turn-detection verdicts log at debug level.** Without the `logger:` block in + `configuration.yaml` the decision is invisible and you can only infer it from + timing. +- **`pkill -f` matches the shell that typed it.** A command containing + `fake-tts.py` is itself a match, so the pattern kills the session. Bracketing + only helps when the name does not appear elsewhere on the line; a pid file and + a script, as in `run-fake-tts.sh`, always works. +- **Home Assistant caches synthesised speech.** A test that plays the same clip + repeatedly is served from the cache and never reaches the synthesiser, which + reads as "nothing was spoken". `tts.clear_cache` between cases. +- **A Wyoming synthesiser must send an audio header even with nothing to say.** + Without an `AudioStart`/`AudioStop` pair Home Assistant waits for audio that + never arrives, and the whole pipeline appears to hang somewhere else entirely. +- **Home Assistant sends the whole message again after the chunks**, as a plain + `Synthesize`, for servers that cannot stream. One that can must ignore it or + it says everything twice. +- **Do not reconstruct a request you can capture.** Estimating Home Assistant's + prompt from its parts gave a wrong answer twice. `dev/ollama-tap.py` proxies + the real one and the numbers stopped moving. +- **ollama's `load_duration` is not loading**, and is not zero for a resident + model: it brackets everything before the runner is handed the request. diff --git a/dev/configuration.yaml b/dev/configuration.yaml new file mode 100644 index 0000000..601c9a6 --- /dev/null +++ b/dev/configuration.yaml @@ -0,0 +1,30 @@ +# Development Home Assistant, run in the agent VM. Copied to the config dir by +# dev/run-ha.sh. +# +# Deliberately NOT default_config: that pulls dhcp, go2rtc, logbook, my, ssdp and +# stream, which are not in the package's extraComponents, and when it fails to +# set up everything it would have pulled in fails with it -- including +# conversation and assist_pipeline. +homeassistant: + name: Dev + time_zone: America/New_York + unit_system: us_customary + country: US + +http: +api: +websocket_api: +config: + +conversation: +assist_pipeline: + +# Fake lights and sensors, so the assistant has something to control. +demo: + +# Turn detection logs its verdict at debug level; without this the decision is +# invisible and a test can only infer it from timing. +logger: + default: info + logs: + homeassistant.components.assist_pipeline: debug diff --git a/dev/e2e-compare.py b/dev/e2e-compare.py new file mode 100644 index 0000000..276a491 --- /dev/null +++ b/dev/e2e-compare.py @@ -0,0 +1,79 @@ +"""End to end, old settings against new: audio in, answer out.""" +import asyncio, json, statistics, subprocess, sys, time, urllib.request +sys.argv = sys.argv[:1] +exec(open("dev/turn-test.py").read().split("async def main")[0]) + +async def one(ws, ident, pid, audio, **settings): + stream = audio + b"\x00" * (16000 * 2 * 4) + await ws.send(json.dumps({"id": ident, "type": "assist_pipeline/run", + "start_stage": "stt", "end_stage": "intent", + "input": {"sample_rate": 16000, **settings}, + "pipeline": pid, "timeout": 60})) + hid = None; task = None; marks = {}; t_audio_end = None; reply = "" + async def pump(): + nonlocal t_audio_end + for i in range(0, len(stream), 3200): + await ws.send(bytes([hid]) + stream[i:i + 3200]) + if i <= len(audio) < i + 3200: + t_audio_end = time.monotonic() + await asyncio.sleep(0.1) + while True: + m = json.loads(await ws.recv()) + if m.get("id") != ident: continue + if m.get("type") == "result" and not m.get("success"): + return None, m.get("error"), "" + if m.get("type") != "event": continue + e = m["event"] + if e["type"] == "run-start": + hid = e["data"]["runner_data"]["stt_binary_handler_id"] + task = asyncio.create_task(pump()) + marks[e["type"]] = time.monotonic() + if e["type"] == "intent-end": + reply = e["data"]["intent_output"]["response"]["speech"]["plain"]["speech"] + if e["type"] in ("run-end", "error"): break + if task: task.cancel() + if "intent-end" not in marks or t_audio_end is None: + return None, "no intent-end", "" + return (marks["intent-end"] - t_audio_end) * 1000, None, reply + +async def main(): + ensure_whisper() + ws = await websockets.connect(URL, max_size=None); await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + states = rest("/api/states") + stt = next(s["entity_id"] for s in states + if s["entity_id"].startswith("stt.") and "demo" not in s["entity_id"]) + conv = next((s["entity_id"] for s in states + if s["entity_id"].startswith("conversation.") and "ollama" in s["entity_id"]), + "conversation.home_assistant") + pls = (await call(ws, 1, type="assist_pipeline/pipeline/list"))["result"]["pipelines"] + p = next((x for x in pls if x.get("name") == "e2e"), None) + if not p: + res = await call(ws, 2, type="assist_pipeline/pipeline/create", name="e2e", + language="en", conversation_engine=conv, conversation_language="en", + stt_engine=stt, stt_language="en", tts_engine=None, + tts_language=None, tts_voice=None, wake_word_entity=None, + wake_word_id=None) + pid = res["result"]["id"] + else: + pid = p["id"] + print(f"conversation agent: {conv}\n") + audio = pcm(sys.argv[1] if len(sys.argv) > 1 else "scratch/hadev/cmd.wav") + ident = 400 + for label, s in (("before: silence 0.7, no turn model", + dict(silence_seconds=0.7, turn_detection=False)), + ("after: silence 0.1, turn model", + dict(silence_seconds=0.1, turn_detection=True, + turn_threshold=0.9, turn_max_seconds=2.0))): + runs = [] + for _ in range(3): + ident += 1 + ms, err, reply = await one(ws, ident, pid, audio, **s) + if err: print(f" {label}: ERROR {err}"); break + runs.append(ms) + await asyncio.sleep(2) + if runs: + print(f" {label:38} {statistics.median(runs):6.0f} ms {reply[:40]!r}") + +asyncio.run(main()) diff --git a/dev/eval.py b/dev/eval.py new file mode 100644 index 0000000..5ceee7e --- /dev/null +++ b/dev/eval.py @@ -0,0 +1,122 @@ +"""Score conversation models on scenarios, against the development instance. + +Latency scripts cannot tell you whether a model does the right thing. Each +scenario sets entity state, says something, and checks either the resulting +state or the words of the reply. State is reset before every single run, so a +light left on by one scenario cannot make the next one look correct. + + dev/eval.py [reps] [model ...] +""" +import asyncio, json, statistics, sys, time +sys.path.insert(0, "dev") +from halib import call, connect, engines, rest + +SCENARIOS = json.load(open("dev/scenarios.json")) + + +def set_state(entity, state): + rest(f"/api/states/{entity}", {"state": state}) + + +def get_state(entity): + try: + return rest(f"/api/states/{entity}")["state"] + except Exception: + return None + + +async def set_model(ws, entry_id, sub_id, model): + flow = rest("/api/config/config_entries/subentries/flow", + {"handler": [entry_id, "conversation"], "subentry_id": sub_id, + "show_advanced_options": True}) + cur = {f["name"]: f.get("description", {}).get("suggested_value") + for f in flow["data_schema"] if "name" in f} + body = {k: v for k, v in cur.items() if v is not None} + body["model"] = model + for k in ("num_ctx", "max_history", "keep_alive"): + if k in body: + body[k] = int(body[k]) + return rest(f"/api/config/config_entries/subentries/flow/{flow['flow_id']}", + body).get("reason") + + +async def say(ws, pipeline, text): + ident_start = time.monotonic() + await ws.send(json.dumps({"id": (ident := _next()), "type": "assist_pipeline/run", + "start_stage": "intent", "end_stage": "intent", + "input": {"text": text}, "pipeline": pipeline, + "timeout": 120})) + reply = "" + while True: + m = json.loads(await ws.recv()) + if m.get("id") != ident: + continue + if m.get("type") == "result" and not m.get("success"): + return "", 0.0 + if m.get("type") != "event": + continue + e = m["event"] + if e["type"] == "intent-end": + reply = e["data"]["intent_output"]["response"]["speech"]["plain"]["speech"] + if e["type"] in ("run-end", "error"): + break + return reply, (time.monotonic() - ident_start) * 1000 + + +_id = 1000 +def _next(): + global _id + _id += 1 + return _id + + +async def main(): + reps = int(sys.argv[1]) if len(sys.argv) > 1 else 3 + models = sys.argv[2:] or ["qwen3.8:27b-mtp-q8_0"] + ws = await connect() + entry = next(e for e in rest("/api/config/config_entries/entry") + if e["domain"] == "ollama") + subs = (await call(ws, type="config_entries/subentries/list", + entry_id=entry["entry_id"]))["result"] + entry_id, sub_id = entry["entry_id"], subs[0]["subentry_id"] + _stt, conv = engines() + pls = (await call(ws, type="assist_pipeline/pipeline/list"))["result"]["pipelines"] + pipe = next((p["id"] for p in pls if p.get("conversation_engine") == conv), None) + if pipe is None: + res = await call(ws, type="assist_pipeline/pipeline/create", name="eval", + language="en", conversation_engine=conv, + conversation_language="en", stt_engine=None, stt_language=None, + tts_engine=None, tts_language=None, tts_voice=None, + wake_word_entity=None, wake_word_id=None) + pipe = res["result"]["id"] + + for model in models: + if await set_model(ws, entry_id, sub_id, model) != "reconfigure_successful": + print(f"{model}: could not select"); continue + await asyncio.sleep(2) + await say(ws, pipe, "hello") # load the model + print(f"\n### {model}") + total = passed = 0 + latencies = [] + for sc in SCENARIOS: + ok = 0 + for _ in range(reps): + for entity, state in (sc.get("setup") or {}).items(): + set_state(entity, state) + reply, ms = await say(ws, pipe, sc["say"]) + latencies.append(ms) + good = True + for entity, want in (sc.get("expect_state") or {}).items(): + if get_state(entity) != want: + good = False + if sc.get("expect_any"): + good = good and any(w.lower() in reply.lower() + for w in sc["expect_any"]) + ok += good + await asyncio.sleep(0.3) + total += reps; passed += ok + flag = "" if ok == reps else f" <- {ok}/{reps}" + print(f" {sc['id']:16} {ok}/{reps}{flag}") + print(f" {'TOTAL':16} {passed}/{total} median {statistics.median(latencies):.0f} ms") + +asyncio.run(main()) diff --git a/dev/fake-tts.py b/dev/fake-tts.py new file mode 100644 index 0000000..6028e91 --- /dev/null +++ b/dev/fake-tts.py @@ -0,0 +1,135 @@ +"""A text-to-speech server that says nothing, and writes down what it was asked. + +Kokoro needs CUDA, so the real synthesiser cannot run in this VM -- but the +question "did the pipeline ask for this to be spoken, and when" does not need a +synthesiser to answer. This advertises streaming synthesis, accepts both the +one-shot and the streamed form, returns silence, and appends a line per sentence +to its journal. + + dev/py dev/fake-tts.py [--uri tcp://127.0.0.1:10211] [--journal scratch/spoken.jsonl] +""" +import argparse, asyncio, json, time +from wyoming.audio import AudioChunk, AudioStart, AudioStop +from wyoming.event import Event +from wyoming.info import Attribution, Describe, Info, TtsProgram, TtsVoice +from wyoming.server import AsyncEventHandler, AsyncServer +from wyoming.tts import ( + Synthesize, SynthesizeChunk, SynthesizeStart, SynthesizeStop, SynthesizeStopped, +) + +RATE, WIDTH, CHANNELS = 22050, 2, 1 +ENDINGS = ".!?" + + +def take_complete_sentences(buffer: str) -> tuple[list[str], str]: + """Split off what is certainly finished, keeping the rest for later text.""" + cut = max((buffer.rfind(c) for c in ENDINGS), default=-1) + if cut < 0: + return [], buffer + done, rest = buffer[: cut + 1], buffer[cut + 1 :] + return [s.strip() for s in done.replace("! ", "!|").replace("? ", "?|") + .replace(". ", ".|").split("|") if s.strip()], rest + + +class Handler(AsyncEventHandler): + def __init__(self, info, journal, *args, **kwargs): + super().__init__(*args, **kwargs) + self.info = info + self.journal = journal + self.buffer = "" + self.started = False + self.streaming = False + + def note(self, what, text): + with open(self.journal, "a") as f: + f.write(json.dumps({"at": time.time(), "event": what, "text": text}) + "\n") + + async def speak(self, text): + self.note("speak", text) + if not self.started: + await self.write_event( + AudioStart(rate=RATE, width=WIDTH, channels=CHANNELS).event()) + self.started = True + # A tenth of a second of silence, so there is something to receive. + await self.write_event(AudioChunk( + rate=RATE, width=WIDTH, channels=CHANNELS, + audio=bytes(RATE * WIDTH // 10)).event()) + + async def finish(self): + if not self.started: + # Even with nothing to say, the client needs a header before the + # stop or it waits for audio that never comes. The real server does + # this too; a stand-in that skipped it would hang Home Assistant + # here and look like a bug in the pipeline. + await self.write_event( + AudioStart(rate=RATE, width=WIDTH, channels=CHANNELS).event()) + self.started = True + await self.write_event(AudioStop().event()) + self.started = False + + async def handle_event(self, event: Event) -> bool: + if Describe.is_type(event.type): + await self.write_event(self.info.event()) + return True + + if Synthesize.is_type(event.type): + if self.streaming: + # Home Assistant sends the whole message again after the chunks, + # for servers that cannot stream. This one can, and has already + # said it. A stand-in that got this wrong would hide the same + # mistake in the real server. + self.note("ignored-trailing-synthesize", "") + return True + text = Synthesize.from_event(event).text + self.note("synthesize", text) + for sentence in take_complete_sentences(text + " ")[0] or [text]: + await self.speak(sentence) + await self.finish() + return True + + if SynthesizeStart.is_type(event.type): + self.note("start", "") + self.buffer = "" + self.streaming = True + return True + + if SynthesizeChunk.is_type(event.type): + self.buffer += SynthesizeChunk.from_event(event).text + sentences, self.buffer = take_complete_sentences(self.buffer) + for sentence in sentences: + await self.speak(sentence) + return True + + if SynthesizeStop.is_type(event.type): + if self.buffer.strip(): + await self.speak(self.buffer.strip()) + self.buffer = "" + await self.finish() + await self.write_event(SynthesizeStopped().event()) + self.streaming = False + self.note("stop", "") + return True + + return True + + +async def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--uri", default="tcp://127.0.0.1:10211") + parser.add_argument("--journal", default="scratch/spoken.jsonl") + args = parser.parse_args() + + info = Info(tts=[TtsProgram( + name="fake", description="says nothing, writes it down", installed=True, + version="1", attribution=Attribution(name="dev", url=""), + supports_synthesize_streaming=True, + voices=[TtsVoice(name="silence", description="silence", installed=True, + version="1", languages=["en"], + attribution=Attribution(name="dev", url=""))])]) + + open(args.journal, "w").close() + print(f"fake tts on {args.uri}, journal {args.journal}", flush=True) + await AsyncServer.from_uri(args.uri).run( + lambda *a, **k: Handler(info, args.journal, *a, **k)) + +asyncio.run(main()) diff --git a/dev/ha-ask.py b/dev/ha-ask.py new file mode 100755 index 0000000..1ebf183 --- /dev/null +++ b/dev/ha-ask.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +"""Ask the development assistant something and time the intent stage. + + python3 dev/ha-ask.py "is the bed light on?" +""" +import asyncio, json, os, sys, time +import websockets + +HA = os.environ.get("HA_DEV_URL", "http://127.0.0.1:8123") +HERE = os.path.dirname(os.path.abspath(__file__)) +TOKEN = open(os.path.join(HERE, "..", "scratch", "hadev", "token.txt")).read().strip() +QUESTIONS = sys.argv[1:] or ["Is the bed light on?", "Turn on the kitchen lights.", + "What time is it?"] + + +async def main(): + ws = await websockets.connect(HA.replace("http", "ws") + "/api/websocket", + max_size=None) + await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + await ws.send(json.dumps({"id": 1, "type": "assist_pipeline/pipeline/list"})) + while True: + m = json.loads(await ws.recv()) + if m.get("id") == 1: + pipe = next(p["id"] for p in m["result"]["pipelines"] if p["name"] == "Dev") + break + i = 100 + for q in QUESTIONS: + i += 1 + await ws.send(json.dumps({"id": i, "type": "assist_pipeline/run", + "start_stage": "intent", "end_stage": "intent", "input": {"text": q}, + "pipeline": pipe, "timeout": 120})) + marks, reply = {}, "" + while True: + m = json.loads(await ws.recv()) + if m.get("id") != i: + continue + if m.get("type") == "result" and not m.get("success"): + print("ERR", m.get("error")); return + if m.get("type") != "event": + continue + e = m["event"]; marks[e["type"]] = time.monotonic() + if e["type"] == "intent-end": + reply = e["data"]["intent_output"]["response"]["speech"]["plain"]["speech"] + if e["type"] in ("run-end", "error"): + break + dur = (marks.get("intent-end", 0) - marks.get("intent-start", 0)) * 1000 + print(f"{dur:7.0f} ms {q}\n -> {reply}") + await asyncio.sleep(1) + +asyncio.run(main()) diff --git a/dev/ha-onboard.py b/dev/ha-onboard.py new file mode 100755 index 0000000..60a75d2 --- /dev/null +++ b/dev/ha-onboard.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Onboard a fresh development Home Assistant and save a long-lived token. + +Run once after dev/run-ha.sh on an empty config dir. Idempotent enough: if +onboarding is already done it just reports that. +""" +import asyncio, json, os, sys, urllib.error, urllib.request + +HA = os.environ.get("HA_DEV_URL", "http://127.0.0.1:8123") +HERE = os.path.dirname(os.path.abspath(__file__)) +TOKEN_FILE = os.path.join(HERE, "..", "scratch", "hadev", "token.txt") + + +def post(path, body, token=None): + headers = {"Content-Type": "application/json"} + if token: + headers["Authorization"] = "Bearer " + token + req = urllib.request.Request(HA + path, json.dumps(body).encode(), headers) + with urllib.request.urlopen(req, timeout=60) as r: + return json.load(r) + + +steps = json.load(urllib.request.urlopen(HA + "/api/onboarding", timeout=30)) +if all(s["done"] for s in steps): + sys.exit("already onboarded; delete the config dir to start over") + +auth = post("/api/onboarding/users", + {"client_id": HA + "/", "name": "Agent", "username": "agent", + "password": "devdevdev", "language": "en"}) +import urllib.parse +req = urllib.request.Request( + HA + "/auth/token", + urllib.parse.urlencode({"grant_type": "authorization_code", + "code": auth["auth_code"], + "client_id": HA + "/"}).encode()) +short = json.load(urllib.request.urlopen(req, timeout=30))["access_token"] + +for step in ("core_config", "analytics"): + try: + post("/api/onboarding/" + step, {"client_id": HA + "/"}, short) + except urllib.error.HTTPError as e: + print(f" {step}: {e.code} (continuing)") + + +async def long_lived(): + import websockets + async with websockets.connect(HA.replace("http", "ws") + "/api/websocket") as ws: + await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": short})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + await ws.send(json.dumps({"id": 1, "type": "auth/long_lived_access_token", + "client_name": "agent", "lifespan": 3650})) + while True: + m = json.loads(await ws.recv()) + if m.get("id") == 1: + if not m.get("success"): + raise SystemExit(m) + return m["result"] + + +tok = asyncio.run(long_lived()) +os.makedirs(os.path.dirname(TOKEN_FILE), exist_ok=True) +with open(TOKEN_FILE, "w") as f: + f.write(tok) +print(f"onboarded; long-lived token written to {os.path.relpath(TOKEN_FILE, os.getcwd())}") diff --git a/dev/ha-setup.py b/dev/ha-setup.py new file mode 100644 index 0000000..777a88b --- /dev/null +++ b/dev/ha-setup.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +"""Configure a fresh development Home Assistant to mirror the real one. + +Reproducible so the dev instance never drifts into a state nobody can rebuild. +Idempotent: re-running reuses whatever already exists. + + nix run .#hass-dev -- -c ~/ha-dev # start it + python3 dev/ha-setup.py # configure it + +Talks to 127.0.0.1:8123 (the dev instance) and points its conversation agent at +the ollama on the NixOS host, which is the one thing here that needs a GPU. +""" +import asyncio, json, os, sys, urllib.error, urllib.request + +HA = os.environ.get("HA_DEV_URL", "http://127.0.0.1:8123") +OLLAMA = os.environ.get("HA_DEV_OLLAMA", "http://192.168.122.1:11434") +MODEL = os.environ.get("HA_DEV_MODEL", "qwen3.8:27b-mtp-q8_0") +TOKEN = open(os.path.join(os.path.dirname(__file__) or ".", + "../scratch/hadev/token.txt")).read().strip() + +# Mirrors the real machine's prompt. Everything after <|fim_pad|> is re-read on +# every request; everything before it is cached. See packages/prefix-cache-findings.md. +PROMPT = """You are a voice assistant for Home Assistant. +Answer in plain text. Keep it simple and to the point: one or two short sentences unless asked for detail. + +Your training data is out of date and your memory of current facts is wrong. +The current time and the state of the devices listed at the very end of this prompt are live; trust them over anything you remember, and answer from them directly without calling a tool. +Call GetLiveContext only for the state of something not listed there. +Never say you lack access to current information. Only say you do not know if a tool returned nothing useful. +<|fim_pad|> +Live state, correct as of now: +Current time: {{ now().strftime('%A, %B %-d, %Y at %-I:%M %p %Z') }} +Bed Light: {{ states('light.bed_light') }} +Ceiling Lights: {{ states('light.ceiling_lights') }} +Kitchen Lights: {{ states('light.kitchen_lights') }}""" + +EXPOSE = ["light.bed_light", "light.ceiling_lights", "light.kitchen_lights", + "weather.forecast_home"] + + +def rest(path, body=None, method=None): + req = urllib.request.Request( + HA + path, + data=json.dumps(body).encode() if body is not None else None, + headers={"Content-Type": "application/json", + "Authorization": "Bearer " + TOKEN}, + method=method) + with urllib.request.urlopen(req, timeout=120) as r: + return json.load(r) + + +def ollama_entry(): + """Create the ollama config entry, or return the existing one.""" + for e in rest("/api/config/config_entries/entry"): + if e["domain"] == "ollama": + print(f" ollama entry exists: {e['entry_id']}") + return e["entry_id"] + flow = rest("/api/config/config_entries/flow", + {"handler": "ollama", "show_advanced_options": True}) + res = rest(f"/api/config/config_entries/flow/{flow['flow_id']}", {"url": OLLAMA}) + if res.get("type") == "create_entry": + entry = res["result"]["entry_id"] + print(f" created ollama entry {entry} -> {OLLAMA}") + return entry + raise SystemExit(f"unexpected flow result: {json.dumps(res)[:400]}") + + +def conversation_subentry(entry_id, existing_id=None): + """Create the conversation agent, or reconfigure the one that exists.""" + body = {"handler": [entry_id, "conversation"], "show_advanced_options": True} + if existing_id: + body["subentry_id"] = existing_id # reconfigure rather than add + flow = rest("/api/config/config_entries/subentries/flow", body) + data = {"model": MODEL, "prompt": PROMPT, "llm_hass_api": ["assist"], + "num_ctx": 8192, "max_history": 20, "keep_alive": -1, "think": False} + res = rest(f"/api/config/config_entries/subentries/flow/{flow['flow_id']}", data) + print(f" conversation agent: {res.get('type')} {res.get('reason', '')} model={MODEL}") + + +async def ws_setup(): + import websockets + async with websockets.connect(HA.replace("http", "ws") + "/api/websocket", + max_size=None) as ws: + await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + n = [0] + + async def call(**kw): + n[0] += 1 + await ws.send(json.dumps({"id": n[0], **kw})) + while True: + m = json.loads(await ws.recv()) + if m.get("id") == n[0] and m.get("type") == "result": + return m + + r = await call(type="homeassistant/expose_entity", + assistants=["conversation"], entity_ids=EXPOSE, should_expose=True) + print(f" exposed {len(EXPOSE)} entities: {r.get('success')}") + + pipelines = await call(type="assist_pipeline/pipeline/list") + if not pipelines.get("success"): + print(f" pipeline/list failed: {pipelines.get('error')}") + return + agents = await call(type="conversation/agent/list") + agent = next((a["id"] for a in agents["result"]["agents"] + if a["id"] != "conversation.home_assistant"), None) + print(f" conversation agents: {[a['id'] for a in agents['result']['agents']]}") + if agent is None: + print(" !! no LLM agent yet; re-run after the entry finishes setting up") + return + existing = next((p for p in pipelines["result"]["pipelines"] + if p["name"] == "Dev"), None) + spec = {"name": "Dev", "language": "en", "conversation_engine": agent, + "conversation_language": "en", "stt_engine": None, "stt_language": None, + "tts_engine": None, "tts_language": None, "tts_voice": None, + "wake_word_entity": None, "wake_word_id": None, + "prefer_local_intents": True} + if existing: + r = await call(type="assist_pipeline/pipeline/update", + pipeline_id=existing["id"], **spec) + print(f" updated pipeline {existing['id']}: {r.get('success')}") + else: + r = await call(type="assist_pipeline/pipeline/create", **spec) + print(f" created pipeline: {r['result']['id'] if r.get('success') else r}") + + +async def existing_subentry(entry_id): + import websockets + async with websockets.connect(HA.replace("http", "ws") + "/api/websocket", + max_size=None) as ws: + await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + await ws.send(json.dumps({"id": 1, "type": "config_entries/subentries/list", + "entry_id": entry_id})) + while True: + m = json.loads(await ws.recv()) + if m.get("id") == 1 and m.get("type") == "result": + subs = [s for s in m.get("result", []) + if s["subentry_type"] == "conversation"] + return subs[0]["subentry_id"] if subs else None + + +print(f"configuring dev Home Assistant at {HA}") +entry = ollama_entry() +conversation_subentry(entry, asyncio.run(existing_subentry(entry))) +asyncio.run(ws_setup()) +print("done") diff --git a/dev/halib.py b/dev/halib.py new file mode 100644 index 0000000..e80889a --- /dev/null +++ b/dev/halib.py @@ -0,0 +1,157 @@ +"""Shared helpers for driving the development Home Assistant. + +The voice scripts all need the same few things: a REST call with the dev token, +a websocket command, audio as 16 kHz mono PCM, and a pipeline run that streams +audio in real time and reports when each stage finished. +""" +import asyncio, json, subprocess, time, urllib.request +import websockets + +URL = "ws://127.0.0.1:8123/api/websocket" +TOKEN = open("scratch/hadev/token.txt").read().strip() +SR = 16000 + + +def rest(path, body=None): + req = urllib.request.Request( + f"http://127.0.0.1:8123{path}", + data=json.dumps(body).encode() if body is not None else None, + headers={"Content-Type": "application/json", "Authorization": f"Bearer {TOKEN}"}) + return json.load(urllib.request.urlopen(req, timeout=120)) + + +def ensure_whisper(): + """Point the dev instance at the faster-whisper running in this VM.""" + for e in rest("/api/config/config_entries/entry"): + if e["domain"] == "wyoming": + return + flow = rest("/api/config/config_entries/flow", + {"handler": "wyoming", "show_advanced_options": True}) + rest(f"/api/config/config_entries/flow/{flow['flow_id']}", + {"host": "127.0.0.1", "port": 10300}) + + +def pcm(path): + return subprocess.run(["ffmpeg", "-v", "error", "-i", path, "-ar", str(SR), + "-ac", "1", "-f", "s16le", "-"], + capture_output=True, check=True).stdout + + +async def connect(): + ws = await websockets.connect(URL, max_size=None) + await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + return ws + + +_next_id = 0 + + +def _ident(): + """Home Assistant requires websocket ids to increase, so hand them out here.""" + global _next_id + _next_id += 1 + return _next_id + + +async def call(ws, **msg): + ident = _ident() + await ws.send(json.dumps({"id": ident, **msg})) + while True: + m = json.loads(await ws.recv()) + if m.get("id") == ident and m.get("type") == "result": + if not m.get("success"): + raise RuntimeError(f"{msg.get('type')} failed: {m.get('error')}") + return m + + +async def pipeline_id(ws, name, **fields): + """Find or create a pipeline by name, keeping its engines up to date.""" + pls = (await call(ws, type="assist_pipeline/pipeline/list"))["result"]["pipelines"] + existing = next((p for p in pls if p.get("name") == name), None) + if existing: + stale = {k: v for k, v in fields.items() if existing.get(k) != v} + if stale: + await call(ws, type="assist_pipeline/pipeline/update", + pipeline_id=existing["id"], + **{k: v for k, v in existing.items() if k != "id"}, **stale) + return existing["id"] + res = await call(ws, type="assist_pipeline/pipeline/create", + **{"name": name, "language": "en", + "conversation_language": "en", "tts_engine": None, + "tts_language": None, "tts_voice": None, + "wake_word_entity": None, "wake_word_id": None, + **fields}) + return res["result"]["id"] + + +def engines(): + """The dev instance's speech-to-text and conversation entities.""" + states = rest("/api/states") + stt = next(s["entity_id"] for s in states + if s["entity_id"].startswith("stt.") and "demo" not in s["entity_id"]) + conv = next((s["entity_id"] for s in states + if s["entity_id"].startswith("conversation.") and "ollama" in s["entity_id"]), + "conversation.home_assistant") + return stt, conv + + +async def run(ws, pid, audio, end_stage="intent", trailing=4, on_event=None, + **settings): + """Stream audio in real time; return (ms from end of speech, text, reply, error). + + on_event, if given, is called with every pipeline event as it arrives -- for + tests that need to act on one, such as fetching the speech as a satellite + would rather than leaving it to be synthesised whenever. + """ + ident = _ident() + stream = audio + b"\x00" * (SR * 2 * trailing) + await ws.send(json.dumps({"id": ident, "type": "assist_pipeline/run", + "start_stage": "stt", "end_stage": end_stage, + "input": {"sample_rate": SR, **settings}, + "pipeline": pid, "timeout": 60})) + hid = None; task = None; marks = {}; audio_end = None; text = None; reply = "" + + async def pump(): + nonlocal audio_end + for i in range(0, len(stream), 3200): + await ws.send(bytes([hid]) + stream[i:i + 3200]) + if i <= len(audio) < i + 3200: + audio_end = time.monotonic() + await asyncio.sleep(0.1) + + while True: + m = json.loads(await ws.recv()) + if m.get("id") != ident: + continue + if m.get("type") == "result" and not m.get("success"): + return None, None, "", m.get("error") + if m.get("type") != "event": + continue + e = m["event"] + marks[e["type"]] = time.monotonic() + if on_event is not None: + on_event(e) + if e["type"] == "run-start": + hid = e["data"]["runner_data"]["stt_binary_handler_id"] + task = asyncio.create_task(pump()) + if e["type"] == "stt-end": + text = e["data"]["stt_output"]["text"] + if e["type"] == "intent-end": + reply = e["data"]["intent_output"]["response"]["speech"]["plain"]["speech"] + if e["type"] == "error": + return None, text, reply, e["data"] + if e["type"] == "run-end": + break + if task: + task.cancel() + last = "intent-end" if end_stage == "intent" else "stt-end" + if last not in marks: + return None, text, reply, f"no {last} event" + if audio_end is None: + # The turn ended before the whole clip had been streamed, so there is no + # "end of speech" to measure from. Not a failure: it is what holding a + # pause and then hitting the cap looks like on a long recording. + return None, text, reply, None + return (marks[last] - audio_end) * 1000, text, reply, None diff --git a/dev/host-e2e.py b/dev/host-e2e.py new file mode 100644 index 0000000..88b47f8 --- /dev/null +++ b/dev/host-e2e.py @@ -0,0 +1,132 @@ +"""The whole thing, on the host: audio in, audio out. + +Streams a clip in real time to the real machine's pipeline and reports when each +stage finished and when the first byte of speech came back. This is the number +that matters -- everything else measured so far is a piece of it. + + dev/py dev/host-e2e.py [repeats] [clip...] + +Set HOST_E2E_SETTINGS to a JSON object to override the pipeline's audio +settings for the run, e.g. to turn speculation off and compare. +""" +import asyncio, json, os, statistics, subprocess, sys, time +import aiohttp, websockets + +HOST = "192.168.122.1:8123" +TOKEN = open("scratch/ha/token.txt").read().strip() +SR = 16000 +CLIPS = ["scratch/hadev/Is_the_kitchen_light_o.wav", + "scratch/hadev/Turn_off_the_ceiling_l.wav"] + + +def pcm(path): + return subprocess.run(["ffmpeg", "-v", "error", "-i", path, "-ar", str(SR), + "-ac", "1", "-f", "s16le", "-"], + capture_output=True, check=True).stdout + + +async def fetch(session, url, marks, key): + async with session.get(f"http://{HOST}{url}", + headers={"Authorization": f"Bearer {TOKEN}"}) as r: + async for _ in r.content.iter_chunked(1024): + if key not in marks: + marks[key] = time.monotonic() + + +async def once(ws, ident, pipeline, audio, session, trailing=4, **settings): + # How finely the audio is fed in. Home Assistant only looks at the stream + # when a chunk arrives, so this sets the granularity of every decision made + # from it -- when speech is seen to stop, and so when the snapshot for + # speculative transcription is taken. A satellite streams continuously; a + # test client that sends 100 ms at a time makes the pipeline look slower + # than it is. + chunk_ms = int(os.environ.get("HOST_E2E_CHUNK_MS", "100")) + chunk = SR * 2 * chunk_ms // 1000 + stream = audio + bytes(SR * 2 * trailing) + await ws.send(json.dumps({"id": ident, "type": "assist_pipeline/run", + "start_stage": "stt", "end_stage": "tts", + "input": {"sample_rate": SR, **settings}, + "pipeline": pipeline, "timeout": 60})) + hid = None; pump = None; marks = {}; fetcher = None; text = None + + async def send_audio(): + for i in range(0, len(stream), chunk): + await ws.send(bytes([hid]) + stream[i:i + chunk]) + if i <= len(audio) < i + chunk: + marks["speech_ends"] = time.monotonic() + await asyncio.sleep(chunk_ms / 1000) + + while True: + m = json.loads(await ws.recv()) + if m.get("id") != ident: + continue + if m.get("type") == "result" and not m.get("success"): + return None, m.get("error") + if m.get("type") != "event": + continue + e = m["event"] + if e["type"] == "run-start": + hid = e["data"]["runner_data"]["stt_binary_handler_id"] + pump = asyncio.create_task(send_audio()) + if (out := e["data"].get("tts_output")): + fetcher = asyncio.create_task( + fetch(session, out["url"], marks, "first_audio")) + if e["type"] == "stt-end": + marks["transcribed"] = time.monotonic() + text = e["data"]["stt_output"]["text"] + if e["type"] == "intent-end": + marks["answered"] = time.monotonic() + if e["type"] in ("run-end", "error"): + break + if pump: + pump.cancel() + if fetcher: + await fetcher + if "speech_ends" not in marks: + return None, "never saw the end of the audio" + t0 = marks["speech_ends"] + return {k: (v - t0) * 1000 for k, v in marks.items() if k != "speech_ends"}, text + + +async def main(): + repeats = int(sys.argv[1]) if len(sys.argv) > 1 else 3 + clips = sys.argv[2:] or CLIPS + ws = await websockets.connect(f"ws://{HOST}/api/websocket", max_size=None) + await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + await ws.send(json.dumps({"id": 1, "type": "assist_pipeline/pipeline/list"})) + pref = json.loads(await ws.recv())["result"]["preferred_pipeline"] + + ident = 100 + async with aiohttp.ClientSession() as session: + for clip in clips: + audio = pcm(clip) + print(f"\n{clip.rsplit('/', 1)[-1]}") + rows = [] + for _ in range(repeats): + ident += 1 + await ws.send(json.dumps({"id": ident, "type": "call_service", + "domain": "tts", "service": "clear_cache"})) + while True: + m = json.loads(await ws.recv()) + if m.get("id") == ident and m.get("type") == "result": + break + ident += 1 + marks, text = await once(ws, ident, pref, audio, session, + **json.loads(os.environ.get("HOST_E2E_SETTINGS", "{}"))) + if marks is None: + print(f" ERROR {text}") + continue + rows.append(marks) + await asyncio.sleep(2) + if not rows: + continue + print(f" heard {text!r}") + print(" ms from the last sample of speech:") + for k in ("transcribed", "answered", "first_audio"): + vals = [r[k] for r in rows if k in r] + if vals: + print(f" {statistics.median(vals):7.0f} {k}") + +asyncio.run(main()) diff --git a/dev/kokoro-stream-test.py b/dev/kokoro-stream-test.py new file mode 100644 index 0000000..59d59df --- /dev/null +++ b/dev/kokoro-stream-test.py @@ -0,0 +1,106 @@ +"""Drive the patched Kokoro server's protocol handling with a stub synthesiser. + +Building the real thing needs onnxruntime with CUDA, which is irrelevant to what +changed: this exercises the event handling, the sentence splitting, and the order +of the audio events Home Assistant expects. + +The exchange is exactly the one Home Assistant performs, including the plain +Synthesize carrying the whole message that it sends after the chunks "for +backwards compatibility". A server that streams has already said all of it, and +must ignore it -- the first version of this test left it out, and the reply was +synthesised twice for a day before anything noticed. + + dev/py dev/kokoro-stream-test.py +""" +import asyncio, importlib.util, sys, types + +import numpy as np + +# main.py imports kokoro_onnx at module scope; stand in for it. +import logging + +fake = types.ModuleType("kokoro_onnx") +fake.__path__ = [] # make it a package, it has submodules +fake.config = types.SimpleNamespace(SAMPLE_RATE=24000, MAX_PHONEME_LENGTH=510) +fake.Kokoro = object +fake.EspeakConfig = object +fake_log = types.ModuleType("kokoro_onnx.log") +fake_log.log = logging.getLogger("stub") +fake_config = types.ModuleType("kokoro_onnx.config") +fake_config.SAMPLE_RATE = 24000 +fake_config.MAX_PHONEME_LENGTH = 510 +sys.modules["kokoro_onnx"] = fake +sys.modules["kokoro_onnx.log"] = fake_log +sys.modules["kokoro_onnx.config"] = fake_config + +spec = importlib.util.spec_from_file_location("kmain", sys.argv[1]) +kmain = importlib.util.module_from_spec(spec) +spec.loader.exec_module(kmain) + +from wyoming.audio import AudioChunk, AudioStart, AudioStop +from wyoming.tts import ( + Synthesize, SynthesizeChunk, SynthesizeStart, SynthesizeStop, SynthesizeStopped, +) + + +class StubKokoro: + def __init__(self): + self.spoken = [] + + def create_stream(self, text, voice, speed, lang): + self.spoken.append(text) + + async def gen(): + yield np.zeros(2400, dtype=np.float32), 24000 + + return gen() + + +class Recorder(kmain.KokoroEventHandler): + def __init__(self, kokoro): + self.kokoro = kokoro + self.default_voice = "af_heart" + self.default_speed = 1.0 + self.wyoming_info_event = None + self._semaphore = asyncio.Semaphore(1) + self._cache = None + self._stream_voice = "af_heart" + self._stream_buffer = "" + self._stream_started = False + self._streaming = False + self._stream_t0 = 0.0 + self.events = [] + + async def write_event(self, event): + self.events.append(event) + + +async def main(): + stub = StubKokoro() + h = Recorder(stub) + await h.handle_event(SynthesizeStart(voice=None).event()) + # A reply arriving the way a language model produces it. + for piece in ["The bed ", "light is off. ", "The kitchen ", "lights are on", + ". Anything else?"]: + await h.handle_event(SynthesizeChunk(text=piece).event()) + n = sum(AudioChunk.is_type(e.type) for e in h.events) + print(f" after {piece!r:22} synthesised={stub.spoken!r:60} audio chunks={n}") + # What Home Assistant sends next: the whole message, again. + whole = "The bed light is off. The kitchen lights are on. Anything else?" + await h.handle_event(Synthesize(text=whole).event()) + print(f" after the trailing Synthesize synthesised={stub.spoken!r}") + await h.handle_event(SynthesizeStop().event()) + + kinds = [e.type for e in h.events] + print(f"\n sentences synthesised: {stub.spoken}") + print(f" event order: {kinds[0]} ... {kinds[-2]} {kinds[-1]}") + ok = ( + AudioStart.is_type(kinds[0]) + and AudioStop.is_type(kinds[-2]) + and SynthesizeStopped.is_type(kinds[-1]) + and stub.spoken == ["The bed light is off.", "The kitchen lights are on.", + "Anything else?"] + ) + print(f"\n {'PASS' if ok else 'FAIL'}") + +asyncio.run(main()) diff --git a/dev/local-intent-test.py b/dev/local-intent-test.py new file mode 100644 index 0000000..867b3e3 --- /dev/null +++ b/dev/local-intent-test.py @@ -0,0 +1,80 @@ +"""How much of the latency is avoidable by never reaching the model at all. + +With prefer_local_intents on, Home Assistant matches commands against its own +sentence templates first and only falls back to the conversation agent when +nothing matches. A match costs a few milliseconds; a miss costs the whole model +round trip. Matching is strict -- exact wording, exact entity name -- so whether +a phrase is fast depends entirely on what the entity happens to be called. + +Aliases are the lever. This measures a phrase before and after adding one. + + dev/py dev/local-intent-test.py +""" +import asyncio, statistics, sys, time +sys.path.insert(0, "dev") +from halib import call, connect, engines, pipeline_id, rest + +ENTITY = "light.ceiling_lights" +EXACT = "Turn off the Ceiling Lights." +LOOSE = "Turn off the ceiling light." +ALIAS = "ceiling light" + + +async def timed(ws, pid, text, repeats=3): + out = [] + for _ in range(repeats): + ident = None + ms, kind, speech = await one(ws, pid, text) + out.append((ms, kind, speech)) + await asyncio.sleep(1) + return statistics.median(m for m, _, _ in out), out[-1][1], out[-1][2] + + +async def one(ws, pid, text): + import json + from halib import _ident + ident = _ident() + await ws.send(json.dumps({"id": ident, "type": "assist_pipeline/run", + "start_stage": "intent", "end_stage": "intent", + "input": {"text": text}, "pipeline": pid, "timeout": 60})) + t0 = time.monotonic() + while True: + m = json.loads(await ws.recv()) + if m.get("id") != ident or m.get("type") != "event": + continue + e = m["event"] + if e["type"] == "intent-end": + r = e["data"]["intent_output"]["response"] + return ((time.monotonic() - t0) * 1000, r["response_type"], + r["speech"]["plain"]["speech"]) + if e["type"] == "error": + return (time.monotonic() - t0) * 1000, "error", str(e["data"]) + + +async def aliases(ws, value): + res = await call(ws, type="config/entity_registry/update", + entity_id=ENTITY, aliases=value) + return res["result"]["entity_entry"]["aliases"] + + +async def main(): + stt, conv = engines() + ws = await connect() + pid = await pipeline_id(ws, "local-intents", stt_engine=stt, stt_language="en", + conversation_engine=conv, prefer_local_intents=True) + + print(f"{'phrase':32} {'alias':8} {'ms':>7} {'result':14} reply") + await aliases(ws, []) + for label, text in (("exact", EXACT), ("loose", LOOSE)): + ms, kind, speech = await timed(ws, pid, text) + print(f" {text:30} {'none':8} {ms:7.0f} {kind:14} {speech[:34]!r}") + + got = await aliases(ws, [ALIAS]) + print(f"\n added alias {got}\n") + for label, text in (("exact", EXACT), ("loose", LOOSE)): + ms, kind, speech = await timed(ws, pid, text) + print(f" {text:30} {ALIAS:8} {ms:7.0f} {kind:14} {speech[:34]!r}") + + await aliases(ws, []) + +asyncio.run(main()) diff --git a/dev/model-compare.py b/dev/model-compare.py new file mode 100644 index 0000000..f25ea2b --- /dev/null +++ b/dev/model-compare.py @@ -0,0 +1,76 @@ +"""Compare conversation models end to end. + +qwen35moe was rejected long ago for a CUDA illegal memory access during +constrained decoding of tool calls with array or enum parameters -- which is +what Home Assistant sends. That was ollama 0.32.3; we run 0.32.13. +""" +import asyncio, statistics, sys +sys.path.insert(0, "dev") +from halib import call, connect, engines, ensure_whisper, pcm, pipeline_id, rest, run + +MODELS = ["qwen3.8:27b-mtp-q8_0", "qwen3.6:35b-a3b-q4_K_M", "ornith:35b-q4_K_M"] + + +async def subentry(ws): + """Subentry ids come from the websocket API; REST only reports the count.""" + entry = next(e for e in rest("/api/config/config_entries/entry") + if e["domain"] == "ollama") + res = await call(ws, type="config_entries/subentries/list", + entry_id=entry["entry_id"]) + subs = res["result"] + return entry["entry_id"], (subs[0]["subentry_id"] if subs else None) + + +def set_model(entry_id, sub_id, model): + flow = rest("/api/config/config_entries/subentries/flow", + {"handler": [entry_id, "conversation"], "subentry_id": sub_id, + "show_advanced_options": True}) + cur = {f["name"]: f.get("description", {}).get("suggested_value") + for f in flow["data_schema"] if "name" in f} + body = {k: v for k, v in cur.items() if v is not None} + body["model"] = model + for k in ("num_ctx", "max_history", "keep_alive"): + if k in body: + body[k] = int(body[k]) + res = rest(f"/api/config/config_entries/subentries/flow/{flow['flow_id']}", body) + return res.get("reason") or res.get("errors") + + +async def main(): + ensure_whisper() + ws = await connect() + entry_id, sub_id = await subentry(ws) + if sub_id is None: + print("no ollama conversation subentry; run dev/ha-setup.py first") + return + stt, conv = engines() + pid = await pipeline_id(ws, "e2e", stt_engine=stt, stt_language="en", + conversation_engine=conv) + audio = pcm("scratch/hadev/cmd.wav") + print(f"{'model':28}{'end of speech -> answer':>26} reply") + for model in MODELS: + problem = set_model(entry_id, sub_id, model) + if problem != "reconfigure_successful": + print(f" {model:26} could not select: {problem}") + continue + await asyncio.sleep(2) + runs, reply, failed = [], "", None + for i in range(4): + ms, _text, rep, err = await run(ws, pid, audio, silence_seconds=0.1, + turn_detection=True, turn_threshold=0.9, + turn_max_seconds=2.0) + if err: + failed = err + break + if i: # first call loads the model + runs.append(ms) + reply = rep + await asyncio.sleep(2) + if failed: + print(f" {model:26} ERROR {failed}") + elif runs: + print(f" {model:26} {statistics.median(runs):21.0f} ms {reply[:32]!r}") + set_model(entry_id, sub_id, MODELS[0]) + print(f"\nrestored {MODELS[0]}") + +asyncio.run(main()) diff --git a/dev/ollama-overhead.py b/dev/ollama-overhead.py new file mode 100644 index 0000000..d7aebbf --- /dev/null +++ b/dev/ollama-overhead.py @@ -0,0 +1,41 @@ +"""How much of an ollama request happens before any inference does. + +ollama reports load_duration for every request. On a model that is already +resident that should be nothing, and it is not: answering "what is this model +capable of" means reading and parsing the GGUF metadata, and the chat path asks +twice per request. Generating a single token makes the rest of the request small +enough that the overhead is the whole measurement. + + dev/py dev/ollama-overhead.py [ ...] +""" +import json, statistics, sys, time, urllib.request + + +def measure(url, model, n=7): + walls, loads, prompts = [], [], [] + for i in range(n): + body = {"model": model, "stream": False, "keep_alive": "-1s", + "think": False, "options": {"num_predict": 1}, + "messages": [{"role": "user", "content": "hi"}]} + req = urllib.request.Request(f"{url}/api/chat", + data=json.dumps(body).encode(), + headers={"Content-Type": "application/json"}) + t0 = time.monotonic() + d = json.load(urllib.request.urlopen(req, timeout=600)) + if i: # the first may still be loading the model + walls.append((time.monotonic() - t0) * 1000) + loads.append(d.get("load_duration", 0) / 1e6) + prompts.append(d.get("prompt_eval_duration", 0) / 1e6) + m = statistics.median + return m(walls), m(loads), m(prompts) + + +args = sys.argv[1:] +if len(args) < 2 or len(args) % 2: + print(__doc__) + raise SystemExit(1) + +print(f"{'target':44} {'wall':>8} {'before inference':>17} {'prompt':>8}") +for url, model in zip(args[::2], args[1::2]): + wall, load, prompt = measure(url, model) + print(f" {url + ' ' + model:42} {wall:8.1f} {load:17.1f} {prompt:8.1f}") diff --git a/dev/ollama-probe.py b/dev/ollama-probe.py new file mode 100755 index 0000000..c71ae12 --- /dev/null +++ b/dev/ollama-probe.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Measure prompt-prefix caching and generation speed on any ollama. + + OLLAMA_URL=http://127.0.0.1:11434 python3 dev/ollama-probe.py qwen3.8:27b-mtp-q8_0 + +Reports three things: + fresh conversation -- a new conversation sharing only the cached prefix, which + is what every voice command looks like + follow-up turn -- appending to the conversation already in the slot + generation -- tokens per second + +Read prompt_eval_duration only. prompt_eval_count always reports the whole +prompt, reused or not. + +Nothing else may talk to this ollama while it runs: one other request replaces +the slot contents and the next measurement then shares nothing with it. +""" +import json, os, statistics, sys, urllib.request + +URL = os.environ.get("OLLAMA_URL", "http://127.0.0.1:11434") + "/api/chat" +MODEL = sys.argv[1] if len(sys.argv) > 1 else "qwen3.8:27b-mtp-q8_0" + +# Shaped like Home Assistant's: a preamble plus a long entity table. The size and +# the fact that it is byte-identical across conversations are what matter. +# +# It ends with the cache boundary marker, because that is what the server is +# told to look for. Without it the delimiter matches nothing, the server falls +# back to checkpointing near the end of the prompt, and the probe measures the +# degraded path -- which looks like a hardware difference if the two machines +# are configured differently. +MARKER = os.environ.get("OLLAMA_CACHE_MARKER", "<|fim_pad|>") +SYSTEM = ( + "You are a voice assistant for a home. Answer in one or two short sentences, " + "in plain spoken language, with no markdown and no lists.\n\n" + "An overview of the areas and the devices in this smart home:\n" +) + "\n".join( + f"- device_{i}: Device {i} ('Device {i}', {'on' if i % 3 else 'off'})" + for i in range(300)) + f"\n{MARKER}\nCurrent time: Monday, 9:00 PM" + +QUESTIONS = ["Turn on device 1.", "What state is device 2 in?", "Is device 3 on?", + "Turn off device 4.", "Check device 5.", "Toggle device 6."] + + +def chat(messages, num_predict=40): + body = json.dumps({"model": MODEL, "messages": messages, "stream": False, + "think": False, "keep_alive": -1, + "options": {"temperature": 0, "num_predict": num_predict}}).encode() + req = urllib.request.Request(URL, body, {"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=900) as r: + return json.load(r) + + +ms = lambda d, k: d[k] / 1e6 +print(f"model {MODEL} at {URL}") +d = chat([{"role": "system", "content": SYSTEM}, {"role": "user", "content": "Hello."}]) +rate = d["prompt_eval_count"] / (d["prompt_eval_duration"] / 1e9) +print(f"first {ms(d, 'prompt_eval_duration'):8.1f} ms / {d['prompt_eval_count']} tok" + f" ({rate:.0f} tok/s)") +if rate > 3000: + print(" ^ the slot was already warm, so this is NOT a cold prefill rate;" + " restart the server to measure that") + +fresh, follow = [], [] +for q in QUESTIONS: + msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": q}] + d = chat(msgs) + fresh.append(ms(d, "prompt_eval_duration")) + d2 = chat(msgs + [d["message"], {"role": "user", "content": "And the one after?"}]) + follow.append(ms(d2, "prompt_eval_duration")) + +# Long enough to measure: short replies make tokens/sec mostly startup noise. +# Warn rather than quietly report a number derived from a handful of tokens. +# Open-ended prose rather than a countable task: the same counting prompt at +# temperature 0 produced 300 tokens on CUDA and 17 on Metal, because the +# backends diverge numerically and one model gave up early. Prose keeps +# generating on both. +g = chat([{"role": "system", "content": SYSTEM}, + {"role": "user", "content": "Describe a kitchen in detail: the counters, " + "the light, the smells, the sounds. Write " + "several paragraphs."}], 300) +if g["eval_count"] < 60: + print(f"\n!! only {g['eval_count']} tokens generated; tokens/sec below is unreliable") +print(f"\nfresh conversation : {statistics.median(fresh):8.1f} ms") +print(f"follow-up turn : {statistics.median(follow):8.1f} ms") +print(f"generation : {g['eval_count'] / (g['eval_duration'] / 1e9):8.1f} tok/s" + f" ({g['eval_count']} tok)") diff --git a/dev/ollama-tap.py b/dev/ollama-tap.py new file mode 100644 index 0000000..02c858c --- /dev/null +++ b/dev/ollama-tap.py @@ -0,0 +1,74 @@ +"""A proxy that records exactly what Home Assistant asks ollama for. + +Reconstructing the prompt from the pieces has burned me before: what Home +Assistant sends is the template, the API preamble, the entity overview, the +tools and the options, assembled by code that has opinions. This forwards +everything to the real server and writes each request body to a file, so the +question "how big is the prompt, and what options are set" has an answer rather +than an estimate. + + dev/py dev/ollama-tap.py [--listen 11435] [--upstream 192.168.122.1:11434] + +Then point the development instance's ollama entry at 127.0.0.1:11435. +""" +import argparse, asyncio, json, time +from aiohttp import ClientSession, web + +JOURNAL = "scratch/ollama-requests.jsonl" + + +async def handle(request): + body = await request.read() + if request.path.endswith("/api/chat") and body: + try: + d = json.loads(body) + with open(JOURNAL, "a") as f: + f.write(json.dumps({"at": time.time(), "body": d}) + "\n") + msgs = d.get("messages", []) + chars = sum(len(m.get("content") or "") for m in msgs) + print(f" chat: {len(msgs)} messages, {chars} chars, " + f"{len(d.get('tools') or [])} tools, options={d.get('options')}, " + f"think={d.get('think')!r}, stream={d.get('stream')}", flush=True) + except (ValueError, OSError): + pass + + up = request.app["upstream"] + async with request.app["session"].request( + request.method, f"http://{up}{request.path_qs}", data=body or None, + headers={k: v for k, v in request.headers.items() if k.lower() != "host"}, + ) as r: + resp = web.StreamResponse(status=r.status, headers={ + k: v for k, v in r.headers.items() + if k.lower() not in ("content-length", "content-encoding", + "transfer-encoding")}) + await resp.prepare(request) + async for chunk in r.content.iter_any(): + await resp.write(chunk) + await resp.write_eof() + return resp + + +async def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--listen", type=int, default=11435) + parser.add_argument("--upstream", default="192.168.122.1:11434") + args = parser.parse_args() + + app = web.Application(client_max_size=64 * 1024 * 1024) + app["upstream"] = args.upstream + app.router.add_route("*", "/{tail:.*}", handle) + app.cleanup_ctx.append(session_ctx) + open(JOURNAL, "w").close() + print(f"tapping {args.upstream} on :{args.listen}, writing {JOURNAL}", flush=True) + runner = web.AppRunner(app) + await runner.setup() + await web.TCPSite(runner, "127.0.0.1", args.listen).start() + await asyncio.Event().wait() + + +async def session_ctx(app): + async with ClientSession(timeout=None) as s: + app["session"] = s + yield + +asyncio.run(main()) diff --git a/dev/py b/dev/py new file mode 100755 index 0000000..6028ec3 --- /dev/null +++ b/dev/py @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Run a dev script with the packages they need. +# +# dev/py dev/speculation-compare.py 3 +# +# The scripts are plain Python, but the system interpreter does not carry +# websockets (or numpy, for the turn-model ones), and which packages a bare +# `python3` has is not something a repository should depend on. +set -euo pipefail +cd "$(dirname "$0")/.." +env=$(nix build --no-link --print-out-paths --impure --expr \ + 'with import {}; buildEnv { + name = "assist-dev"; + paths = [ + ffmpeg # the clips are wav and mp3; the pipeline wants raw 16 kHz mono + (python3.withPackages (ps: with ps; [ + websockets aiohttp numpy onnxruntime wyoming + ])) + ]; + }' 2>/dev/null | tail -1) +PATH="$env/bin:$PATH" exec "$env/bin/python" "$@" diff --git a/dev/run-fake-tts.sh b/dev/run-fake-tts.sh new file mode 100755 index 0000000..5def5c3 --- /dev/null +++ b/dev/run-fake-tts.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Start or stop the stand-in text-to-speech server. +# +# dev/run-fake-tts.sh # start it in the background +# dev/run-fake-tts.sh stop +# +# It lives in a script rather than a shell one-liner because `pkill -f fake-tts` +# also matches the shell that typed it, which kills the wrong process. +set -euo pipefail +cd "$(dirname "$0")/.." +pidfile=scratch/fake-tts.pid + +if [ "${1:-start}" = stop ]; then + [ -f "$pidfile" ] && kill "$(cat "$pidfile")" 2>/dev/null && echo stopped || echo "not running" + rm -f "$pidfile" + exit 0 +fi + +if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then + echo "already running as $(cat "$pidfile")"; exit 0 +fi + +setsid dev/py dev/fake-tts.py > scratch/fake-tts.log 2>&1 < /dev/null & +echo $! > "$pidfile" +sleep 8 +cat scratch/fake-tts.log diff --git a/dev/run-ha.sh b/dev/run-ha.sh new file mode 100755 index 0000000..8357f97 --- /dev/null +++ b/dev/run-ha.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# Start the development Home Assistant in the agent VM. +# +# dev/run-ha.sh # start it in the background +# dev/run-ha.sh stop # stop it +# +# The real machine is untouched: this instance keeps its own state in ~/ha-dev +# and only borrows the host's ollama, which is the one part that needs a GPU. +set -euo pipefail +cd "$(dirname "$0")/.." +DIR="${HA_DEV_DIR:-$HOME/ha-dev}" + +# The pattern is bracketed so it cannot match this script's own command line. +pid() { pgrep -f "bin/[.]hass-wrapped" | head -1; } + +if [ "${1:-start}" = stop ]; then + p=$(pid || true); [ -n "$p" ] && kill "$p" && echo "stopped $p" || echo "not running" + exit 0 +fi + +p=$(pid || true) +if [ -n "$p" ]; then echo "already running as $p"; exit 0; fi + +mkdir -p "$DIR" +cp dev/configuration.yaml "$DIR/configuration.yaml" +wrapper=$(nix build --no-link --print-out-paths .#hass-dev) +setsid "$wrapper/bin/hass-dev" -c "$DIR" > "$DIR/hass.log" 2>&1 < /dev/null & +sleep 3 +echo "started $(pid) -- log: $DIR/hass.log" diff --git a/dev/run-ollama-tap.sh b/dev/run-ollama-tap.sh new file mode 100755 index 0000000..45ac9f7 --- /dev/null +++ b/dev/run-ollama-tap.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +# Start or stop the ollama request tap. A script, not a one-liner, because +# `pkill -f` on the name also matches the shell that typed it. +set -euo pipefail +cd "$(dirname "$0")/.." +pidfile=scratch/ollama-tap.pid +if [ "${1:-start}" = stop ]; then + [ -f "$pidfile" ] && kill "$(cat "$pidfile")" 2>/dev/null && echo stopped || echo "not running" + rm -f "$pidfile"; exit 0 +fi +if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then + echo "already running as $(cat "$pidfile")"; exit 0 +fi +setsid dev/py dev/ollama-tap.py > scratch/ollama-tap.log 2>&1 < /dev/null & +echo $! > "$pidfile" +sleep 6 +cat scratch/ollama-tap.log diff --git a/dev/scenarios.json b/dev/scenarios.json new file mode 100644 index 0000000..1924754 --- /dev/null +++ b/dev/scenarios.json @@ -0,0 +1,201 @@ +[ + { + "id": "ctl_on", + "say": "Turn on the kitchen lights.", + "kind": "control", + "setup": { + "light.kitchen_lights": "off" + }, + "expect_state": { + "light.kitchen_lights": "on" + }, + "path": "local" + }, + { + "id": "ctl_off", + "say": "Turn off the bed light.", + "kind": "control", + "setup": { + "light.bed_light": "on" + }, + "expect_state": { + "light.bed_light": "off" + }, + "path": "local" + }, + { + "id": "ctl_compound", + "say": "Turn on the ceiling lights and the bed light.", + "kind": "control", + "setup": { + "light.ceiling_lights": "off", + "light.bed_light": "off" + }, + "expect_state": { + "light.ceiling_lights": "on", + "light.bed_light": "on" + }, + "path": "llm" + }, + { + "id": "ctl_negated", + "say": "Turn off all the lights except the kitchen.", + "kind": "control", + "setup": { + "light.kitchen_lights": "on", + "light.bed_light": "on", + "light.ceiling_lights": "on" + }, + "expect_state": { + "light.kitchen_lights": "on", + "light.bed_light": "off", + "light.ceiling_lights": "off" + }, + "path": "llm" + }, + { + "id": "ctl_missing", + "say": "Turn on the garage light.", + "kind": "graceful", + "expect_any": [ + "don't", + "not", + "no ", + "cannot", + "can't", + "unable", + "couldn't" + ], + "path": "llm" + }, + { + "id": "state_query", + "say": "Is the kitchen light on?", + "kind": "query", + "setup": { + "light.kitchen_lights": "on" + }, + "expect_any": [ + "yes", + "is on", + "currently on" + ], + "path": "llm" + }, + { + "id": "weather", + "say": "What is the weather right now?", + "kind": "query", + "expect_any": [ + "rain", + "cloud", + "sun", + "clear", + "degree", + "\u00b0", + "fair", + "snow" + ], + "path": "llm" + }, + { + "id": "date", + "say": "What day is it today?", + "kind": "tool_date", + "expect_any": [ + "august", + "2026" + ], + "path": "llm" + }, + { + "id": "search_hit", + "say": "Who is the current president of the United States?", + "kind": "tool_search", + "expect_any": [ + "trump" + ], + "path": "llm" + }, + { + "id": "search_skip", + "say": "What is two plus two?", + "kind": "no_search", + "expect_any": [ + "4", + "four" + ], + "max_seconds": 2.0, + "path": "llm" + }, + { + "id": "chitchat", + "say": "Thanks, that's all.", + "kind": "no_search", + "expect_any": [ + "" + ], + "max_seconds": 2.0, + "path": "llm" + }, + { + "id": "multi", + "say": "Turn off the bed light and tell me the weather.", + "kind": "control", + "setup": { + "light.bed_light": "on" + }, + "expect_state": { + "light.bed_light": "off" + }, + "expect_any": [ + "rain", + "cloud", + "sun", + "clear", + "degree", + "\u00b0", + "fair", + "snow" + ], + "path": "llm" + }, + { + "id": "ctl_implicit", + "say": "It's too dark in the kitchen.", + "kind": "control", + "path": "llm", + "setup": { + "light.kitchen_lights": "off" + }, + "expect_state": { + "light.kitchen_lights": "on" + } + }, + { + "id": "ctl_bedtime", + "say": "I'm heading to bed, shut everything down.", + "kind": "control", + "path": "llm", + "setup": { + "light.kitchen_lights": "on", + "light.ceiling_lights": "on", + "light.bed_light": "on" + }, + "expect_state": { + "light.kitchen_lights": "off", + "light.ceiling_lights": "off", + "light.bed_light": "off" + } + }, + { + "id": "search_skip2", + "say": "What is the capital of France?", + "kind": "no_search", + "path": "llm", + "expect_any": [ + "paris" + ], + "max_seconds": 2.5 + } +] \ No newline at end of file diff --git a/dev/semantic-endpointing.md b/dev/semantic-endpointing.md new file mode 100644 index 0000000..1fd9bc9 --- /dev/null +++ b/dev/semantic-endpointing.md @@ -0,0 +1,703 @@ +# Semantic endpointing + +**Result: 1406 ms -> 600 ms** from the last sample of speech to the assistant's +answer, and it now keeps listening through a mid-sentence pause, which no +setting could do before. + +## What changed + +| | | +|---|---| +| Turn detection | Smart Turn v3 decides whether a pause is final, so `silence_seconds` no longer has to be long enough to forgive one | +| `silence_seconds` | 0.7 -> 0.25 by default, 0.1 available; every 100 ms off it is 100 ms off the answer | +| Speculative transcription | transcribes a snapshot taken when speech stops, during the wait, instead of after it | +| Speculative conversation | runs the model on that transcript too, holding side effects and speech until the turn is confirmed | +| Streaming synthesis | Kokoro is fed text as it is generated, so the first sentence is spoken while the last is still being written | +| Model | `ornith:35b-q4_K_M` scored 45/45 against 38/45 for the incumbent, and is faster | + +## The one thing to understand + +Silence cannot tell "still thinking" from "finished" -- they are acoustically +identical and only the words differ. So with a silence threshold alone, how fast +the assistant answers and how long a pause it forgives are **the same number**, +and every setting is a compromise between them. A turn model separates them: the +threshold governs the common case, the model holds the turn open when the +utterance sounds unfinished. + +## Try it + + dev/run-ha.sh # the development instance + dev/turn-test.py # does it hold a pause? + dev/silence-sweep.py # latency against pause tolerance + dev/stage-breakdown.py # where the time goes + dev/eval.py 3 ... # scenario scores, state reset each run + dev/model-compare.py # models end to end + dev/smart-turn-probe.py # the model's opinion, cut by cut + +Voice tests need a transcriber in the VM: + + $(nix build --no-link --print-out-paths nixpkgs#wyoming-faster-whisper)/bin/wyoming-faster-whisper \ + --model tiny-int8 --language en --uri tcp://127.0.0.1:10300 \ + --data-dir ~/ha-dev/whisper --download-dir ~/ha-dev/whisper + +## The transcriber was the wrong one + +`wyoming-faster-whisper` defaults to `model = "auto"`, which reads as "the best +one available". It is not. For English it prefers Parakeet through sherpa-onnx, +and the test is whether `sherpa_onnx` imports -- a module that lives in the +package's `optional-dependencies` and is not installed. So the preference falls +through to `rhasspy/faster-whisper-base-int8`, silently. + +Same machine, same four clips, `dev/stt-compare.py`: + +| | median | +|---|---| +| whisper-base-int8 -- what "auto" resolves to today | 368 ms | +| tiny-int8 | 193 ms | +| **Parakeet TDT 0.6b v2 int8** -- what "auto" prefers | **65 ms** | + +5.7x faster than the host runs now, from the more accurate model of the two. +Turning the extra on is the whole change. + +It matters twice over. Speculative transcription is only free while it finishes +*inside* the wait for silence: at 368 ms it overran a 250 ms wait outright, so +the conversation could not start until after the turn was already confirmed. At +65 ms it leaves most of the wait for the model. + +## Where the remaining time goes + +Measured against the host, intent stage only, `scratch/host-stages.py`: + +| | first tool call | first word | done | +|---|---|---|---| +| no tool ("say hello") | -- | 276 ms | 329 ms | +| a state question | -- | 271 ms | 378 ms | +| a command, loose wording | 1194 ms | 314 ms | 1521 ms | +| **a command, exact wording** | -- | -- | **4 ms** | + +The last row is not the model being fast. It is Home Assistant never asking it. +`prefer_local_intents` matches the sentence against its own templates first, and +a match costs single-digit milliseconds. The filter is inverted from how it +reads: `_async_local_fallback_intent_filter` returns True to *reject*, so with a +controlling agent it is state *questions* that are pushed to the model and +actions that stay local. + +Matching is strict -- exact wording, exact entity name -- so whether a phrase is +fast depends entirely on what the entity is called. Aliases are the lever, and +`dev/local-intent-test.py` measures one: + + Turn off the ceiling light. no alias 1679 ms (the model) + Turn off the ceiling light. alias added 5 ms (local) + +Three hundred times faster, on the slowest kind of request, from configuration +alone. Worth doing for every phrasing actually used, once there are real devices +to name. + +## The largest remaining block was not inference + +Time to first token on the host is 276 ms, and that is the biggest single item +left. Taking it apart needed the real request rather than a reconstruction of +it, so `dev/ollama-tap.py` proxies the development instance and writes down what +Home Assistant actually sends: 2 messages, 4118 characters, **29 tools**, +`num_ctx` 8192, thinking off. Replayed against ollama directly: + +| | | +|---|---| +| before inference starts (`load_duration`) | **148 ms** | +| prompt processing, 4787 tokens | 104 ms | +| the first token itself | ~14 ms | + +So the prompt cache is working -- 4787 tokens in 104 ms -- and the 29 tools cost +nothing measurable. Over half the wait is `load_duration`, on a model that never +left the GPU. + +It is not loading. `load_duration` brackets everything ollama does before +handing off to the runner, and what dominates it is deciding what the model is +capable of: open the GGUF, parse its metadata. The chat path does this three +times per request -- `GetModel`, the handler's `Capabilities()`, and again +through `CheckCapabilities` inside `scheduleRunner`. + +strace of one request, one token generated: the model blob opened **6 times**, +**35.79 MB** read from it. It reproduces on a 0.6B model on CPU in the VM at +91 ms, so it is ollama, not this hardware or this model. + +`ollama-model-metadata-cache.patch` caches those key-values on the blob path, +which is a content hash. Measured with `dev/ollama-overhead.py`: + + before 205.1 ms wall, 90.9 ms before inference + after 112.6 ms wall, 1.2 ms before inference (0.03 MB read, was 35.79) + +The first attempt cached the *capabilities* on the manifest digest, and ollama's +own test suite rejected it: capabilities also depend on `OLLAMA_GO_TEMPLATE`, so +two models with identical bytes can legitimately disagree. The file read is what +is expensive, not the logic, so only the read is cached. Worth running someone +else's tests before believing your own reasoning. + +## Where that leaves the budget + +Measured, host, question, text in to first byte of audio: first token 276 ms, +generated 383 ms, first audio 474 ms. With the audio front end that is roughly +540 ms from the last sample of speech. Removing the 148 ms should put it near +390 ms, against an architectural floor of + + 250 ms silence, before the turn may be called over + 25 ms the turn model + 91 ms synthesising the first sentence + ------ + 366 ms + +which is 26 ms away. **Speech-to-text on the GPU is no longer worth having**: +at 68 ms it is almost entirely hidden behind the wait, and removing all of it +saves ~26 ms before the silence threshold becomes the binding constraint +instead. Before Parakeet and speculation it would have been worth ~300 ms; the +order things were fixed in changed which of them mattered. + +Below 366 ms means shortening the wait, which is the one number that trades +directly against cutting people off. Everything else is saturated. + +## Measured on the host, with all of it running + +`dev/host-e2e.py` streams a clip to the real machine and times from the last +sample of speech. The model side landed exactly as the ollama work predicted -- +first token 276 -> **128 ms**, first audio (text in) 474 -> **338 ms**. + +End to end, audio in to first byte of audio out: + +| | transcribed | answered | first audio | +|---|---|---|---| +| "Is the kitchen light on right now?" | 379 ms | 517 ms | 619 ms | +| "Turn off the ceiling lights." (matches the entity name) | 379 ms | 379 ms | **379 ms** | + +The second row never reaches the model at all. It is the same local-intent path +as before, now visible end to end. + +From the development instance's log, where the timeline is legible: + + speech ends 0 ms + speculative transcript ready 147 ms <- conversation starts here + turn confirmed, committed 287 ms + answer complete 434 ms + +So the wait is no longer the constraint: the answer is not ready until +147 + 282 ms, and the turn was confirmed at 287. Two things are on the critical +path one-for-one, and nothing else is: + +- **the model, 282 ms** -- the largest remaining item by far +- **147 ms before the speculative transcript exists** -- Silero releasing, + a Wyoming round trip, and Parakeet itself + +### The wait is free up to about 0.35 s -- but only for questions + +Sweeping `silence_seconds` on the host, first audio for a question: + + 0.10 615 ms 0.40 660 ms + 0.25 619 ms 0.70 950 ms + +0.10 and 0.25 are the same number, because the model is still working either +way. The crossover is around 0.35 s. + +It is tempting to raise the default to 0.35 and get more pause tolerance for +nothing. Do not: a command that matches an entity name is answered in 379 ms +without the model, and for those the wait *is* the whole latency. Raising it +would put 100 ms onto every one of them. 0.25 stays. + +## Still open + +- **Recordings of the person who will use it.** Everything here is public data + and text-to-speech. The 6.1% of unfinished utterances the model cuts off at + threshold 0.9 is the number that matters, and it cannot be checked against a + corpus that does not contain your voice, your room or your phrasing. +- **Speculating past transcription** is now done; see below. Tool calls that + would change something are held mid-call rather than abandoned, so a guess + that turns out right keeps the prefill and the tokens that chose the tool. +- **Streaming speech synthesis** is done and verified on the host; see below. +- **Switching the live assistant to ornith.** Left deliberately for a person: + the eval is 15 scenarios against demo entities, not a house. + +--- + +No silence threshold can tell "still thinking" from "finished": they are +acoustically identical and only the words differ. Every setting we can reach +trades response speed against how long a pause is tolerated, one for one. + +A turn-detection model breaks that trade. It reads the utterance and predicts +whether the speaker is done, so a short silence threshold can be used for the +common case while genuine mid-thought pauses are rescued. + +## Smart Turn v3 looks like the right model + +`pipecat-ai/smart-turn-v3` on Hugging Face, BSD-2-Clause. Whisper Tiny encoder +plus a linear head, 8M parameters, 8 MB int8 ONNX. It reads the waveform, not a +transcript, so it needs no extra speech-to-text pass. Published accuracy 92.6% +over 31,527 samples across 23 languages. + +This is the same shape LiveKit and Pipecat both use: a cheap VAD for +speech/silence, and a separate end-of-utterance model on top. + +Interface: `input_features` `[batch, 80, 800]` — a Whisper mel spectrogram of +the **last 8 seconds** — and one output which is already a probability, not a +logit. Preprocessing is `WhisperFeatureExtractor(chunk_length=8)` with +`do_normalize=True`. + +## It works, on real speech + +Sweeping cut points through the JFK sample: + + cut (s) P(complete) + 2.5 0.864 "And so my fellow Americans," clause end + 3.0 0.568 + 4.0 0.035 mid-clause + 7.5 0.028 mid-clause + 10.5 0.904 "...do for your country." sentence end + 11.0 0.625 + +Clause and sentence ends score high, mid-clause scores near zero. Inference is +~30 ms on this VM's CPU, unoptimised. + +## Text-to-speech cannot test this + +Piper clips scored 0.93-0.99 whether the sentence was complete or truncated +mid-phrase. That is not the model failing: asked to say "Turn on the kitchen", +a synthesiser produces the falling intonation of a finished sentence, because it +does not know the text is a fragment. The model reads prosody, so a TTS fragment +is indistinguishable from a TTS sentence. + +**Validation needs real recordings**, ideally of the person who will use it, +pausing naturally mid-request. `ldc.wav`, a single read TIMIT sentence, also +scores 0.96+ at every cut, so read speech may be a poor test too. + +## What it buys + +Today latency and pause tolerance are the same number: 0.25 s means both a +376 ms response and being cut off after a 250 ms pause. With a turn model the +short threshold governs the common case, and the model holds the turn open when +the utterance sounds unfinished -- fast when you are done, patient when you are +not. + + +## Measured on the model's own labelled test set + +`pipecat-ai/smart-turn-data-v3.1-test` carries `endpoint_bool`, plus `synthetic` +and `midfiller` flags. Filtering to **real, non-synthetic English** recordings +(607 of them in one shard, 312 complete and 295 not): + +**93.2% accuracy**, against the 92.63% published across all languages. That also +validates the preprocessing port -- a wrong mel pipeline reads as chance. + +The two errors cost very different amounts, so the threshold matters: + +| threshold | cut off mid-thought | made to wait when done | +|---|---|---| +| 0.3 | 19.7% | 1.0% | +| 0.5 (default) | 14.6% | 1.9% | +| 0.7 | 11.2% | 3.8% | +| 0.8 | 8.8% | 4.8% | +| 0.9 | 6.1% | 8.7% | +| 0.95 | 2.4% | 13.1% | +| 0.98 | 0.7% | 32.1% | + +Being cut off mid-thought is the failure worth avoiding; being made to wait costs +one extra increment. Today *every* pause longer than `silence_seconds` cuts you +off, so even the default threshold is a large improvement, and ~0.9 looks like a +sensible operating point. + +## The verdict does not drift, so the design needs a cap + +Appending silence to unfinished utterances barely moves the score: + + +0 ms median P 0.033 + +500 ms median P 0.044 + +2000 ms median P 0.060 + +So re-checking as silence accumulates will not converge on ending the turn by +itself. Something that sounds unfinished stays unfinished, and an utterance +someone simply trails off from would hold the turn open forever. The design +needs an explicit maximum. + +## Design + +1. Silero detects silence as now, with `silence_seconds` set low (~0.25 s). +2. On expiry, run Smart Turn over the last 8 s of buffered audio (~30 ms). +3. `P(complete) > 0.9` -> end the turn. Total ~280 ms. +4. Otherwise extend by another increment and re-check. +5. Cap the total extension (~2-3 s) and end regardless, because of the drift + result above. + +This is the shape Pipecat and LiveKit both use. It decouples the two numbers +that are currently the same: a short threshold governs the common case, and the +model holds the turn open only when the utterance actually sounds unfinished. + + +## Working end to end + +Streaming a real recording in real time through the dev instance, with no +artificial pause -- the speaker's own pauses are the test: + + silence only, 0.25 s ended 2.5s heard: ' And so my fellow Americans' + silence only, 0.70 s ended 3.0s heard: ' And so my fellow Americans!' + turn detection, 0.25 s ended 5.2s heard: ' And so my fellow Americans, ASK NOT!' + +The model recognised the pause after "Americans," as unfinished and kept +listening, at the *shorter* silence threshold. It still stops at 5.2 s because +`turn_max_seconds` was 2.0 and this speaker pauses for dramatic effect -- which +is the cap doing its job. + +`dev/turn-test.py` runs this. It needs a speech-to-text engine in the VM: + + nix build --no-link --print-out-paths nixpkgs#wyoming-faster-whisper + .../bin/wyoming-faster-whisper --model tiny-int8 --language en \ + --uri tcp://127.0.0.1:10300 --data-dir ~/ha-dev/whisper \ + --download-dir ~/ha-dev/whisper + +`demo_stt` cannot stand in for it: it accepts only stereo, and the pipeline +sends mono. + +## A bug worth remembering + +`VoiceCommandSegmenter.process()` calls `reset()` as it reports the command +finished, which clears `in_command`. Granting another silence window therefore +has to restore that flag as well as the counter -- the silence counter only +decrements *inside* a command, so without it the segmenter sits waiting for +speech that may never come and the turn never ends at all. The symptom was runs +that hung until the 60 s pipeline timeout, with the model logging "keep +listening" correctly each time. + + +## The payoff: latency and pause tolerance finally separate + +Sweeping `silence_seconds` with and without the turn model, measuring both +things that matter -- when a *finished* utterance ends, and whether a +*mid-sentence* pause survives: + +| silence_seconds | turn | finished utterance ends | mid-sentence pause | +|---|---|---|---| +| 0.70 | off | 3.6 s | cut short | +| 0.70 | on | 3.7 s | **held** | +| 0.40 | off | 3.3 s | cut short | +| 0.40 | on | 3.4 s | **held** | +| 0.25 | off | 3.1 s | cut short | +| 0.25 | on | 3.2 s | **held** | +| 0.15 | off | 3.0 s | cut short | +| 0.15 | on | 3.1 s | **held** | +| 0.10 | off | 3.0 s | cut short | +| **0.10** | **on** | **3.1 s** | **held** | + +Without the model, every setting cuts the pause short -- the tolerance *is* the +threshold. With it, the pause is held at every setting, and the threshold is +free to be small. `silence_seconds` 0.7 -> 0.1 takes **600 ms** off a finished +utterance while pauses keep working, and the model itself costs about 100 ms. + +Recommended: `silence_seconds` 0.1-0.15 with `turn_detection` on. That is +roughly 200 ms from end of speech to decision, against 789 ms measured for +silence alone at 0.7, and it is inside the 200 ms band of human turn-taking. + + +## End to end + +Full pipeline, audio in to answer out, timed from the last sample of speech, +through the real conversation agent on the host: + + before: silence 0.7, no turn model 1408 ms 'No, the bed light is off.' + after: silence 0.1, turn model 824 ms 'No, the bed light is off.' + +**584 ms**, same answer. `dev/e2e-compare.py` runs it. + + +## Model choice, re-tested + +End to end from the last sample of speech, with the turn model on and +`silence_seconds` 0.1: + +| model | question answered from the prompt | control command, needs a tool call | +|---|---|---| +| qwen3.8:27b-mtp-q8_0 | 817 ms | 1503 ms | +| qwen3.6:35b-a3b-q4_K_M | 717 ms | 1251 ms | +| ornith:35b-q4_K_M | 713 ms | **1116 ms** | + +**No CUDA fault in 18 tool-call runs across the two MoE models.** That fault -- +an illegal memory access during constrained decoding of tool calls with array or +enum parameters -- is what disqualified qwen35moe, on ollama 0.32.3. We run +0.32.13. It appears resolved, which reopens the faster models. + +Caveats before switching: `qwen3.6:35b-a3b` answered "I am unable to turn on the +kitchen lights" where ornith turned it on, so speed is not the only axis and this +needs the scenario eval rather than a latency script. And the replies above are +not strictly comparable because the light's state carried between runs -- one +model reported "already on". Reset entity state between runs when comparing +behaviour rather than timing. + + +## Quality: ornith is the one to use + +`dev/eval.py` runs 15 scenarios against the development instance, resetting +entity state before *every* run so a light left on by one scenario cannot make +the next look correct. Three repetitions each, 45 runs per model: + +| model | scenarios passed | median intent | +|---|---|---| +| qwen3.8:27b-mtp-q8_0 (current) | 38/45 | 441 ms | +| qwen3.6:35b-a3b-q4_K_M | 39/45 | 466 ms | +| **ornith:35b-q4_K_M** | **45/45** | 678 ms | + +ornith is perfect where the others miss six or seven. Its higher median here is +partly an artefact of being correct: "I am unable to turn on the kitchen lights" +returns faster than actually turning it on. On real audio end to end it is the +*fastest* of the three -- 713 ms against 817 for a question, 1116 against 1503 +for a command. + +`<|fim_pad|>` is a single special token in all three vocabularies, so the cache +boundary carries over unchanged. + +Recommendation: switch the conversation agent to `ornith:35b-q4_K_M`. Left for a +person to decide, because the eval is 15 scenarios against demo entities rather +than a real house, and the assistant's manner of speaking is a matter of taste. + + +## Defaults + +The settings above are no use if a real satellite never passes them, so the +patch changes what the defaults are: + +- `turn_detection` defaults to **on** +- `silence_seconds` defaults to **0.25**, upstream 0.7 +- `VadSensitivity` becomes relaxed 0.7 / default 0.25 / aggressive 0.1, + upstream 1.25 / 0.25 / 0.7 + +Those are only safe *because* the model supplies the pause tolerance. Without it, +`silence_seconds` has to be both how fast the assistant answers and how long a +pause it forgives, which is why upstream's numbers are so long. + +Measured with no per-run settings at all: **940 ms** from end of speech to +answer, against 1408 ms on the old defaults. Selecting "aggressive" takes it to +~824 ms. + + +## Where the time goes, and speculative transcription + +Per stage, from the last sample of speech, with the turn model on: + + VAD + turn model decided 243 ms + speech-to-text finished 454 ms (+211) + answer ready 837 ms (+383) + +The wait for silence and the transcription are both dead time, and they were +consecutive for no reason. `speculative_stt` takes a snapshot the moment speech +stops and transcribes *that* during the wait, so the text is ready when the turn +is confirmed over. If the speaker resumes, the snapshot is cancelled and the +normal path is used. + + speculative stt off 838 ms + speculative stt on 720 ms + +118 ms, bounded by how much wait there is to hide behind. It costs one extra +transcription of audio that is thrown away when the speaker turns out not to +have finished, which is cheap: whisper tiny on CPU. + +Verified that holding a pause still works with it on -- the JFK clip still +reaches "ASK NOT", so the snapshot really is discarded when speech resumes. + +**The snapshot needs padding.** Ending it on the last phoneme, with none of the +silence the full stream would carry, changes what the transcriber hears: across +six clips two came out different, one of them a real mishearing ("set the bad +light" for "set the bed light") and one only capitalisation. Appending 250 ms of +silence to the snapshot makes all six identical. Worth checking again if the +speech-to-text engine changes, because this is a property of the engine, not of +the idea. + + +## Altogether + +From the last sample of speech to the answer, through the real conversation +agent: + +| configuration | question | control command | +|---|---|---| +| before any of this | 1406 ms | 2026 ms | +| turn model + speculative transcription | 717 ms | 774 ms | +| + ornith as the agent | **600 ms** | 1013 ms | + +**1406 -> 600 ms on a question, 2.3x.** And it now holds a mid-sentence pause, +which no setting could do before. + +The command column is not trustworthy: entity state carries between runs, so a +model that says "already on" looks faster than one that actually switches the +light. Compare behaviour with `dev/eval.py`, which resets state before every +run, not with a latency script. + + +## Streaming speech synthesis + +Nothing was spoken until the last token of the reply was generated, because +Kokoro did not advertise `supports_synthesize_streaming` and Home Assistant will +not stream text into an engine that does not. + +It turned out the hard part was already done. `kokoro-wyoming` **already** splits +text into sentences and emits audio for each as it is synthesised: + + sentences = split_into_sentences(text) + for sentence in sentences: + stream = self.kokoro.create_stream(sentence, ...) + if i == 0: + await self.write_event(AudioStart(...).event()) + async for audio, sample_rate in stream: + ...AudioChunk... + +Only the *input* side was missing: it waited for one complete `Synthesize` event. +`kokoro-wyoming-streaming.patch` adds `SynthesizeStart` / `SynthesizeChunk` / +`SynthesizeStop`, buffering text and synthesising each sentence as soon as it is +finished. So the model does not need to stream -- Kokoro synthesises a whole +utterance at once -- it just has to be fed sooner. + +`take_complete_sentences` only hands over text up to the last sentence-ending +punctuation, so a sentence is never synthesised from a fragment that more text +would have changed. + +`dev/kokoro-stream-test.py` drives the handler with a stub synthesiser, since +building the real one needs onnxruntime with CUDA. Feeding it a reply the way a +language model produces it: + + after 'The bed ' synthesised=[] audio chunks=0 + after 'light is off. ' synthesised=['The bed light is off.'] audio chunks=1 + after 'The kitchen ' synthesised=['The bed light is off.'] audio chunks=1 + after 'lights are on' synthesised=['The bed light is off.'] audio chunks=1 + after '. Anything else?' synthesised=[all three] audio chunks=3 + + event order: audio-start ... audio-stop synthesize-stopped + +The first sentence is spoken while the third is still being generated. The +longer the reply, the more this saves, and it is the only change here that +attacks time-to-*first-audio* rather than time-to-answer. + +### Verified on the host + +`run-start` carries `tts_output.stream_response`, which is true only when the +synthesiser takes streamed input *and* the agent produces streamed output, so +the question needs no guessing: + + stream_response = True + +Fetching the audio from the moment the URL exists, and timing the first byte +against the moment generation finished: + +| reply | first audio | generation done | speaking starts | +|---|---|---|---| +| "Is the bed light on?" | 516 ms | 381 ms | 135 ms *after* | +| three sentences | 1054 ms | 2535 ms | **1482 ms before** | +| eight sentences | 748 ms | 5500 ms | **4752 ms before** | + +A one-sentence answer gains nothing -- there is no later sentence to overlap +with, and synthesis still has to happen. Everything longer gains roughly the +whole of its own generation time. Reproduce with `scratch/host-tts-firstbyte.py`. + +### Every reply was being synthesised twice + +Home Assistant sends `SynthesizeStart`, then the chunks, and then the whole +message again as a plain `Synthesize` -- commented in `wyoming/tts.py` as "for +backwards compatibility", for servers that cannot stream. The patched server +fell through to its ordinary `Synthesize` handler for that, so it said +everything a second time and spent twice the GPU on it. + +The stub test had not caught it because it sent only the events the streaming +path cares about. It now performs the exchange Home Assistant actually performs, +trailing `Synthesize` included, which is the version worth keeping: the bug was +not in the logic under test but in the half of the protocol the test omitted. + +## Speculating on the conversation, not just the transcription + +Transcribing early leaves the *model* idle for the rest of the wait, and the +model is the slower of the two. So the conversation now starts on the +speculative transcript as well, and everything it produces is held until the +turn is confirmed: + +| what | how it is held | +|---|---| +| pipeline events | buffered and replayed in order at commit | +| speech | the stream is not handed to the synthesiser until commit | +| tools that read | run immediately -- a wrong one wastes milliseconds | +| tools that act | block inside `llm.APIInstance.async_call_tool` | + +**Held, not abandoned.** The prefill and the tokens that chose the tool are +still valid if the guess was right, which it usually is, so a paused call costs +nothing and resumes on commit. Abandoning would throw away the most expensive +part of the work to save nothing. + +`llm.Tool.reads_only` says which tools may run on a guess. It defaults to +*acts*, because the two mistakes do not cost the same: a needless read wastes a +few milliseconds, a needless action cannot be undone. Only `GetLiveContext`, +`GetDateTime`, `calendar_get_events`, `todo_get_items` and the read-only intents +are marked. + +Discard is nearly free, which is what makes the whole thing safe. +`conversation.async_get_chat_log` builds on a copy of the history and writes it +back only *after* the block it guards finishes -- so cancelling the task leaves +nothing behind. `async_get_chat_session` does the same. Neither needed changing. + +### What it is worth + +Speculation cannot remove the wait itself: the answer still must not arrive +before the speaker is known to have finished. What it removes is the work that +used to happen *after* the wait. So the number to look at is not the saving at +one setting, it is how flat the curve becomes. + + question command + silence off on silence off on + 0.10 785 753 0.10 1477 1379 + 0.25 787 760 0.25 1476 1421 + 0.70 1244 838 0.70 1887 1370 + +Median ms from the last sample of speech to the answer, four runs a cell, +`dev/speculation-sweep.py`. With speculation on, latency barely depends on the +wait at all: 753-838 ms across the whole range for a question, 1370-1421 ms for +a command. Without it, going from 0.25 to 0.7 costs 457 ms and 411 ms. + +**So pause tolerance is close to free now.** The default stays at 0.25 s, since +the turn model already holds mid-sentence pauses and there is no reason to make +a finished utterance wait longer. But 0.7 s costs about 80 ms instead of about +460 ms, which makes it a reasonable thing to reach for if the model turns out to +cut you off -- a pause shorter than the threshold is never submitted to it at +all. + +The saving is smaller here than it will be on the host, because this VM +transcribes in ~211 ms against the host's 94-167 ms, and the transcription has +to finish before the conversation can start on it. The idle left inside a +250 ms wait is whatever the transcriber does not use. + +### Checking it does not act on a guess + +`dev/speculation-safety.py` plays a complete command, a pause, then more speech +-- the shape of someone who was not finished -- with `turn_threshold` at 1.0 so +the pause is held however final the fragment sounds. Counting states is not +enough, because the full utterance contains that command too and the light ends +up off either way. What separates them is how many times the service was called: + + heard: ' Turn off the ceiling lights. Is the kitchen light on right now?' + service calls: ['homeassistant.turn_off', 'light.turn_off'] + light.turn_off called 1 time(s) + +and in the log, the held call is dropped rather than released: + + holding tool call HassTurnOff until the turn is confirmed + abandoning speculative conversation: Turn off the ceiling lights. + speculating on: Turn off the ceiling lights. Is the kitchen light on... + holding tool call HassTurnOff until the turn is confirmed + committing speculative conversation, releasing held HassTurnOff + +`dev/speculation-speech.py` checks the other half, that a guess is never spoken, +against `dev/fake-tts.py` -- a synthesiser that says nothing and writes down what +it was asked to say, since Kokoro needs CUDA and that question does not. + +### Two things that only a real synthesiser in the loop would have found + +**Home Assistant caches speech.** These clips draw the same few replies over and +over, so runs were served from the cache and the synthesiser was never asked +anything -- which looks exactly like "nothing was spoken". The test clears the +cache between cases now. + +**A race between commit and the start of streaming.** Home Assistant only starts +streaming text into the synthesiser once a reply looks long enough to be worth +it. That can happen *after* commit, and the first version stored the stream +whenever it was a speculation -- so a stream created after commit was stored for +a commit that had already happened, and nobody wired it. Home Assistant then +skipped `async_set_message`, believing a stream was set, and the reply was never +spoken. The decision is on the gate now, the same condition the event buffer +uses, not on "is this a speculation". diff --git a/dev/silence-sweep.py b/dev/silence-sweep.py new file mode 100644 index 0000000..5f932ca --- /dev/null +++ b/dev/silence-sweep.py @@ -0,0 +1,38 @@ +"""How low can silence_seconds go once a turn model is catching continuations? + +Two things matter and they pull apart: a finished utterance should end fast, and +a natural mid-sentence pause should still be held open. Measures both. +""" +import asyncio, json, sys +sys.argv = sys.argv[:1] +exec(open("dev/turn-test.py").read().split("async def main")[0]) + +COMPLETE = "scratch/ha/ldc.wav" # one finished sentence +PAUSED = "scratch/ha/jfk.wav" # pauses mid-sentence, dramatically + +async def main(): + ensure_whisper() + ws = await websockets.connect(URL, max_size=None); await ws.recv() + await ws.send(json.dumps({"type": "auth", "access_token": TOKEN})) + assert json.loads(await ws.recv())["type"] == "auth_ok" + pls = (await call(ws, 1, type="assist_pipeline/pipeline/list"))["result"]["pipelines"] + pid = next(p["id"] for p in pls if p.get("name") == "turn-test") + done, paused = pcm(COMPLETE), pcm(PAUSED)[: 16000 * 2 * 6] + ident = 200 + print(f"{'silence':>9} {'turn':>6} {'finished utterance':>28} {'mid-sentence pause':>30}") + for ss in (0.10, 0.15, 0.25, 0.40, 0.70): + for td in (False, True): + ident += 1 + t1, e1, _ = await run(ws, ident, pid, done, silence_seconds=ss, + turn_detection=td, turn_threshold=0.9, + turn_max_seconds=2.0) + ident += 1 + t2, e2, _ = await run(ws, ident, pid, paused, silence_seconds=ss, + turn_detection=td, turn_threshold=0.9, + turn_max_seconds=2.0) + held = "held" if (t2 and "ASK" in (t2 or "").upper()) else "cut short" + print(f"{ss:>9} {str(td):>6} {(e1 or 0):>7.1f}s {str(t1)[:19]:>20} " + f"{(e2 or 0):>7.1f}s {held:>10} {str(t2)[:12]}") + await asyncio.sleep(1) + +asyncio.run(main()) diff --git a/dev/smart-turn-eval.py b/dev/smart-turn-eval.py new file mode 100644 index 0000000..ad99396 --- /dev/null +++ b/dev/smart-turn-eval.py @@ -0,0 +1,48 @@ +"""Evaluate Smart Turn v3 on its own labelled test set, real recordings only. + +Validates the preprocessing (a wrong mel pipeline shows up as chance accuracy) +and, more usefully, reports the subset with a mid-utterance filler -- someone +pausing mid-thought, which is the case a silence threshold cannot handle. +""" +import io, sys +import numpy as np, onnxruntime as ort, pyarrow.parquet as pq, soundfile as sf +from transformers import WhisperFeatureExtractor + +SR = 16000 +fe = WhisperFeatureExtractor(chunk_length=8) +sess = ort.InferenceSession("/home/agent-amd64/models/smart-turn/smart-turn-v3.2-cpu.onnx", + providers=["CPUExecutionProvider"]) + +def predict(a): + if len(a) > 8*SR: a = a[-8*SR:] + inp = fe(a, sampling_rate=SR, return_tensors="np", padding="max_length", + max_length=8*SR, truncation=True, do_normalize=True) + f = np.expand_dims(inp.input_features.squeeze(0).astype(np.float32), 0) + return sess.run(None, {"input_features": f})[0][0].item() + +tbl = pq.read_table(sys.argv[1] if len(sys.argv) > 1 + else "/home/agent-amd64/models/smart-turn/data/t0.parquet") +cols = tbl.column_names +print("columns:", cols) +n = tbl.num_rows +limit = int(sys.argv[2]) if len(sys.argv) > 2 else 400 +rows = tbl.to_pylist() +real = [r for r in rows if not r.get("synthetic") and r.get("language") == "eng"] +print(f"{n} rows, {len(real)} real English; scoring up to {limit}\n") + +groups = {} +for r in real[:limit]: + audio = r["audio"] + data, sr = sf.read(io.BytesIO(audio["bytes"]), dtype="float32") + if data.ndim > 1: data = data.mean(axis=1) + p = predict(data) + correct = (p > 0.5) == bool(r["endpoint_bool"]) + for key in ("all", "midfiller" if r.get("midfiller") else "no filler", + "complete" if r["endpoint_bool"] else "incomplete"): + g = groups.setdefault(key, [0, 0]); g[0] += correct; g[1] += 1 + +print(f"{'subset':>12} {'n':>5} {'accuracy':>9}") +for k in ("all", "complete", "incomplete", "midfiller", "no filler"): + if k in groups: + c, t = groups[k] + print(f"{k:>12} {t:5d} {c/t*100:8.1f}%") diff --git a/dev/smart-turn-probe.py b/dev/smart-turn-probe.py new file mode 100644 index 0000000..05cb7d3 --- /dev/null +++ b/dev/smart-turn-probe.py @@ -0,0 +1,40 @@ +"""Does Smart Turn v3 tell a finished utterance from a mid-sentence one? + +Sweeps cut points through a recording and prints P(turn complete) at each. +High scores should land on clause and sentence ends, low ones mid-clause. + +Needs real speech: text-to-speech gives a *fragment* the falling intonation of +a finished sentence, because the synthesiser does not know it is a fragment, +and this model reads prosody. Piper clips score 0.93-0.99 whether complete or +truncated, which says nothing about the model. + + nix shell --impure --expr 'with import {}; [ (python3.withPackages + (ps: [ ps.onnxruntime ps.transformers ps.numpy ])) ffmpeg ]' \ + --command python3 dev/smart-turn-probe.py +""" +import subprocess +import numpy as np, onnxruntime as ort +from transformers import WhisperFeatureExtractor +SR = 16000 +fe = WhisperFeatureExtractor(chunk_length=8) +sess = ort.InferenceSession("/home/agent-amd64/models/smart-turn/smart-turn-v3.2-cpu.onnx", + providers=["CPUExecutionProvider"]) +def pcm(p): + raw = subprocess.run(["ffmpeg","-v","error","-i",p,"-ar",str(SR),"-ac","1", + "-f","f32le","-"], capture_output=True, check=True).stdout + return np.frombuffer(raw, dtype=np.float32).copy() +def predict(a, tail_ms=300): + a = np.concatenate([a, np.zeros(SR*tail_ms//1000, dtype=np.float32)]) + if len(a) > 8*SR: a = a[-8*SR:] + inp = fe(a, sampling_rate=SR, return_tensors="np", padding="max_length", + max_length=8*SR, truncation=True, do_normalize=True) + f = np.expand_dims(inp.input_features.squeeze(0).astype(np.float32), 0) + return sess.run(None, {"input_features": f})[0][0].item() + +import sys +a = pcm(sys.argv[1] if len(sys.argv) > 1 else "jfk.wav") +print() +print(f"{'cut (s)':>8} {'P(complete)':>11} bar") +for ms in range(1000, int(len(a)/SR*1000)+1, 500): + p = predict(a[:SR*ms//1000]) + print(f"{ms/1000:8.1f} {p:11.3f} {'#'*int(p*40)}") diff --git a/dev/smart-turn-threshold.py b/dev/smart-turn-threshold.py new file mode 100644 index 0000000..6e64a34 --- /dev/null +++ b/dev/smart-turn-threshold.py @@ -0,0 +1,45 @@ +"""Score every real English sample once, then sweep the decision threshold. + +The two errors are not equal. Calling an unfinished utterance complete cuts the +speaker off mid-thought. Calling a finished one incomplete just waits a little +longer. So the threshold should be biased towards waiting. +""" +import io, json, os, sys +import numpy as np, onnxruntime as ort, pyarrow.parquet as pq, soundfile as sf +from transformers import WhisperFeatureExtractor + +SR = 16000 +CACHE = "/home/agent-amd64/models/smart-turn/scores.json" +fe = WhisperFeatureExtractor(chunk_length=8) +sess = ort.InferenceSession("/home/agent-amd64/models/smart-turn/smart-turn-v3.2-cpu.onnx", + providers=["CPUExecutionProvider"]) + +def predict(a): + if len(a) > 8*SR: a = a[-8*SR:] + inp = fe(a, sampling_rate=SR, return_tensors="np", padding="max_length", + max_length=8*SR, truncation=True, do_normalize=True) + f = np.expand_dims(inp.input_features.squeeze(0).astype(np.float32), 0) + return sess.run(None, {"input_features": f})[0][0].item() + +if os.path.exists(CACHE): + scored = json.load(open(CACHE)) +else: + scored = [] + for shard in sys.argv[1:]: + for r in pq.read_table(shard).to_pylist(): + if r.get("synthetic") or r.get("language") != "eng": + continue + d, _ = sf.read(io.BytesIO(r["audio"]["bytes"]), dtype="float32") + if d.ndim > 1: d = d.mean(axis=1) + scored.append({"p": predict(d), "complete": bool(r["endpoint_bool"]), + "dataset": r["dataset"]}) + json.dump(scored, open(CACHE, "w")) + +comp = [s["p"] for s in scored if s["complete"]] +inc = [s["p"] for s in scored if not s["complete"]] +print(f"scored {len(scored)} real English samples: {len(comp)} complete, {len(inc)} incomplete\n") +print(f"{'threshold':>10} {'cut off mid-thought':>21} {'made to wait when done':>24}") +for thr in (0.3, 0.5, 0.7, 0.8, 0.9, 0.95, 0.98, 0.99): + cut = sum(1 for p in inc if p > thr) / len(inc) * 100 # said complete, was not + wait = sum(1 for p in comp if p <= thr) / len(comp) * 100 # said incomplete, was not + print(f"{thr:>10} {cut:>20.1f}% {wait:>23.1f}%") diff --git a/dev/speculation-compare.py b/dev/speculation-compare.py new file mode 100644 index 0000000..85cf9f9 --- /dev/null +++ b/dev/speculation-compare.py @@ -0,0 +1,60 @@ +"""Does starting the conversation on a guess make the answer arrive sooner -- +and is it the same answer? + +Runs each clip with speculative_intent off and on, alternating, and reports the +time from the last sample of speech to intent-end. The replies are compared as +well: speculation is only worth having if it changes nothing but the timing. + + dev/speculation-compare.py [repeats] [clip...] +""" +import asyncio, statistics, sys +sys.path.insert(0, "dev") +from halib import connect, ensure_whisper, engines, pcm, pipeline_id, run + +CLIPS = ["scratch/hadev/Is_the_kitchen_light_o.wav", + "scratch/hadev/Set_the_bed_light_to_f.wav", + "scratch/hadev/Turn_off_the_ceiling_l.wav"] + +BASE = dict(silence_seconds=0.25, turn_detection=True, turn_threshold=0.9, + turn_max_seconds=2.0) + + +async def main(): + repeats = int(sys.argv[1]) if len(sys.argv) > 1 else 3 + clips = sys.argv[2:] or CLIPS + ensure_whisper() + stt, conv = engines() + ws = await connect() + pid = await pipeline_id(ws, "speculation", stt_engine=stt, stt_language="en", + conversation_engine=conv) + print(f"conversation agent: {conv}\n") + + for clip in clips: + audio = pcm(clip) + print(clip.rsplit("/", 1)[-1]) + results = {} + for _ in range(repeats): + for on in (False, True): + ms, text, reply, err = await run( + ws, pid, audio, speculative_intent=on, **BASE) + if err: + print(f" ERROR {err}") + continue + results.setdefault(on, []).append((ms, text, reply)) + await asyncio.sleep(2) + for on in (False, True): + runs = results.get(on) + if not runs: + continue + label = "speculation on " if on else "speculation off" + median = statistics.median(r[0] for r in runs) + replies = {r[2] for r in runs} + print(f" {label} {median:6.0f} ms {len(replies)} distinct reply") + for r in sorted(replies): + print(f" {r[:70]!r}") + if results.get(False) and results.get(True): + off = statistics.median(r[0] for r in results[False]) + onn = statistics.median(r[0] for r in results[True]) + print(f" -> {off - onn:.0f} ms saved\n") + +asyncio.run(main()) diff --git a/dev/speculation-safety.py b/dev/speculation-safety.py new file mode 100644 index 0000000..008045f --- /dev/null +++ b/dev/speculation-safety.py @@ -0,0 +1,69 @@ +"""A speculation that turns out wrong must not have changed anything. + +Plays a complete command, a pause, then more speech -- the shape of someone who +was not finished. The pipeline starts the conversation on the first fragment, +the model calls a tool that would act, and the call is held. Speech resumes, so +the guess is thrown away with the call still held. + +Counting states is not enough: the full utterance contains that command too, so +the light ends up off either way. What distinguishes them is how many times the +service was called -- once if the held call was dropped, twice if it leaked. + +turn_threshold is 1.0 so the pause is held however final the fragment sounds: +this is a test of the discard path, not of the turn model. + + dev/speculation-safety.py +""" +import asyncio, json, sys +sys.path.insert(0, "dev") +from halib import SR, connect, ensure_whisper, engines, pcm, pipeline_id, rest, run + +ENTITY = "light.ceiling_lights" +COMMAND = "scratch/hadev/Turn_off_the_ceiling_l.wav" +MORE = "scratch/hadev/Is_the_kitchen_light_o.wav" + + +async def watch(calls): + """Record every service call Home Assistant makes, on its own connection.""" + ws = await connect() + await ws.send(json.dumps({"id": 1, "type": "subscribe_events", + "event_type": "call_service"})) + while True: + m = json.loads(await ws.recv()) + if m.get("type") == "event": + d = m["event"]["data"] + calls.append(f"{d['domain']}.{d['service']}") + + +async def main(): + ensure_whisper() + stt, conv = engines() + ws = await connect() + pid = await pipeline_id(ws, "speculation", stt_engine=stt, stt_language="en", + conversation_engine=conv) + + rest("/api/services/light/turn_on", {"entity_id": ENTITY}) + await asyncio.sleep(1) + + calls: list[str] = [] + watcher = asyncio.create_task(watch(calls)) + await asyncio.sleep(1) + calls.clear() + + # A command, a pause long enough to speculate in, then more speech. + audio = pcm(COMMAND) + bytes(SR * 2 * 1) + pcm(MORE) + ms, text, reply, err = await run( + ws, pid, audio, silence_seconds=0.25, turn_detection=True, + turn_threshold=1.0, turn_max_seconds=3.0, speculative_intent=True) + await asyncio.sleep(2) + watcher.cancel() + + print(f"heard: {text!r}") + print(f"reply: {reply!r}") + off = [c for c in calls if c == "light.turn_off"] + print(f"service calls: {calls}") + print(f"\nlight.turn_off called {len(off)} time(s)") + print("PASS: the held call was dropped with the guess" if len(off) == 1 + else "FAIL: the abandoned speculation acted as well") + +asyncio.run(main()) diff --git a/dev/speculation-speech.py b/dev/speculation-speech.py new file mode 100644 index 0000000..a15cf7b --- /dev/null +++ b/dev/speculation-speech.py @@ -0,0 +1,117 @@ +"""Nothing is spoken on a guess -- and the real reply still streams. + +Home Assistant hands the synthesiser text as the model produces it, so the first +sentence is spoken while the last is still being written. A speculative +conversation produces that same text, and must not reach the synthesiser until +the turn is confirmed. This checks both halves against dev/fake-tts.py, which +answers "what was I asked to say, and when" without needing a GPU. + +Run the fake synthesiser first, and add it to Home Assistant as a Wyoming entry +on port 10211: + + dev/py dev/fake-tts.py & + dev/py dev/speculation-speech.py +""" +import asyncio, json, sys, urllib.request +sys.path.insert(0, "dev") +from halib import ( + SR, TOKEN, connect, ensure_whisper, engines, pcm, pipeline_id, rest, run, +) + +JOURNAL = "scratch/spoken.jsonl" +QUESTION = "scratch/hadev/Is_the_kitchen_light_o.wav" +COMMAND = "scratch/hadev/Turn_off_the_ceiling_l.wav" + + +def spoken(): + with open(JOURNAL) as f: + return [json.loads(line) for line in f if line.strip()] + + +def clear(): + open(JOURNAL, "w").close() + # Home Assistant caches synthesised speech, and these clips draw the same + # few replies over and over, so without this a run is served from the cache + # and the server is never asked anything. + rest("/api/services/tts/clear_cache", {}) + + +async def say_it(url): + """Pull the audio, the way a satellite does. + + Nothing is synthesised until something asks for it, so a test that only + watches the pipeline events sees synthesis land whenever -- sometimes inside + the next test. Fetching makes it deterministic. + """ + req = urllib.request.Request(f"http://127.0.0.1:8123{url}", + headers={"Authorization": f"Bearer {TOKEN}"}) + return await asyncio.to_thread( + lambda: urllib.request.urlopen(req, timeout=60).read()) + + +def fetcher(pending): + """Start pulling the speech as soon as the run says where it is. + + run-start carries the URL when the reply will be streamed into the + synthesiser, and tts-end when it will not; take whichever comes first. + """ + def on_event(e): + out = (e.get("data") or {}).get("tts_output") or {} + if out.get("url") and not pending: + pending.append(asyncio.create_task(say_it(out["url"]))) + pending.append(e["type"]) + return on_event + + +async def main(): + ensure_whisper() + stt, conv = engines() + tts = next(s["entity_id"] for s in rest("/api/states") + if s["entity_id"] == "tts.fake") + ws = await connect() + pid = await pipeline_id(ws, "speculation-tts", stt_engine=stt, stt_language="en", + conversation_engine=conv, tts_engine=tts, + tts_language="en", tts_voice="silence") + + settings = dict(silence_seconds=0.25, turn_detection=True, turn_threshold=0.9, + turn_max_seconds=2.0, speculative_intent=True) + + print("1. a plain reply is spoken") + clear() + pending = [] + ms, text, reply, err = await run(ws, pid, pcm(QUESTION), end_stage="tts", + on_event=fetcher(pending), **settings) + if pending: + await pending[0] + print(f" url from {pending[1]}") + said = [e["text"] for e in spoken() if e["event"] == "speak"] + print(f" heard {text!r}\n said {said}") + if err: + print(f" error {err}") + ok_plain = bool(said) + print(" PASS" if ok_plain else " FAIL: nothing was spoken") + + print("\n2. an abandoned guess is not spoken") + clear() + audio = pcm(COMMAND) + bytes(SR * 2 * 1) + pcm(QUESTION) + pending = [] + ms, text, reply, err = await run( + ws, pid, audio, end_stage="tts", on_event=fetcher(pending), + **{**settings, "turn_threshold": 1.0, "turn_max_seconds": 3.0}) + if pending: + await pending[0] + print(f" url from {pending[1]}") + events = spoken() + said = [e["text"] for e in events if e["event"] == "speak"] + starts = [e for e in events if e["event"] == "start"] + print(f" heard {text!r}\n said {said}") + if err: + print(f" error {err}") + # One utterance, spoken once: a leaked guess would have opened a second + # stream, or spoken the fragment's reply as well as the real one. + ok_guess = len(starts) <= 1 and len(said) > 0 + print(" PASS" if ok_guess else f" FAIL: {len(starts)} synthesis streams") + + print("\n" + ("PASS" if ok_plain and ok_guess else "FAIL")) + +asyncio.run(main()) diff --git a/dev/speculation-sweep.py b/dev/speculation-sweep.py new file mode 100644 index 0000000..c59aac9 --- /dev/null +++ b/dev/speculation-sweep.py @@ -0,0 +1,53 @@ +"""What speculating on the conversation is worth, against how long the wait is. + +The wait for silence is the one thing speculation cannot remove: the answer +still must not arrive before the speaker is known to have finished. What it +removes is the work that used to happen *after* the wait. So the interesting +number is not the saving at one setting but how flat the curve gets: if the +model finishes inside the wait, a longer and safer wait costs nothing. + + dev/speculation-sweep.py [repeats] +""" +import asyncio, statistics, sys +sys.path.insert(0, "dev") +from halib import connect, ensure_whisper, engines, pcm, pipeline_id, run + +CLIPS = {"question": "scratch/hadev/Is_the_kitchen_light_o.wav", + "command": "scratch/hadev/Turn_off_the_ceiling_l.wav"} +SILENCES = (0.1, 0.25, 0.7) + + +async def main(): + repeats = int(sys.argv[1]) if len(sys.argv) > 1 else 4 + ensure_whisper() + stt, conv = engines() + ws = await connect() + pid = await pipeline_id(ws, "speculation", stt_engine=stt, stt_language="en", + conversation_engine=conv) + print(f"conversation agent: {conv}") + print(f"{repeats} runs per cell, median ms from last sample of speech to answer\n") + + for name, clip in CLIPS.items(): + audio = pcm(clip) + print(f" {name:9} {'silence':>9} {'off':>8} {'on':>8} {'saved':>8}") + for silence in SILENCES: + got = {} + for _ in range(repeats): + for on in (False, True): + ms, text, reply, err = await run( + ws, pid, audio, silence_seconds=silence, + turn_detection=True, turn_threshold=0.9, + turn_max_seconds=2.0, speculative_intent=on) + if err: + print(f" ERROR {err}") + continue + got.setdefault(on, []).append(ms) + await asyncio.sleep(1.5) + if not (got.get(False) and got.get(True)): + continue + off = statistics.median(got[False]) + onn = statistics.median(got[True]) + print(f" {'':9} {silence:9.2f} {off:8.0f} {onn:8.0f} {off - onn:8.0f}") + print() + +asyncio.run(main()) diff --git a/dev/stage-breakdown.py b/dev/stage-breakdown.py new file mode 100644 index 0000000..61d351e --- /dev/null +++ b/dev/stage-breakdown.py @@ -0,0 +1,59 @@ +"""Where does the time go now? Per-stage, timed from the last sample of speech.""" +import asyncio, json, statistics, sys, time +sys.path.insert(0, "dev") +from halib import SR, connect, engines, ensure_whisper, pcm, pipeline_id + +async def one(ws, pid, audio, **settings): + from halib import _ident + ident = _ident() + stream = audio + b"\x00" * (SR * 2 * 4) + await ws.send(json.dumps({"id": ident, "type": "assist_pipeline/run", + "start_stage": "stt", "end_stage": "intent", + "input": {"sample_rate": SR, **settings}, + "pipeline": pid, "timeout": 60})) + hid = None; task = None; marks = {}; audio_end = None + async def pump(): + nonlocal audio_end + for i in range(0, len(stream), 3200): + await ws.send(bytes([hid]) + stream[i:i + 3200]) + if i <= len(audio) < i + 3200: + audio_end = time.monotonic() + await asyncio.sleep(0.1) + while True: + m = json.loads(await ws.recv()) + if m.get("id") != ident: continue + if m.get("type") != "event": continue + e = m["event"]; marks[e["type"]] = time.monotonic() + if e["type"] == "run-start": + hid = e["data"]["runner_data"]["stt_binary_handler_id"] + task = asyncio.create_task(pump()) + if e["type"] in ("run-end", "error"): break + if task: task.cancel() + if audio_end is None or "intent-end" not in marks: return None + return {k: (v - audio_end) * 1000 for k, v in marks.items()} + +async def main(): + ensure_whisper() + ws = await connect(); stt, conv = engines() + pid = await pipeline_id(ws, "e2e", stt_engine=stt, stt_language="en", + conversation_engine=conv) + audio = pcm("scratch/hadev/cmd.wav") + rows = [] + for i in range(5): + r = await one(ws, pid, audio, silence_seconds=0.1, turn_detection=True, + turn_threshold=0.9, turn_max_seconds=2.0) + if r and i: rows.append(r) + await asyncio.sleep(2) + def med(k): + vals = [r[k] for r in rows if k in r] + return statistics.median(vals) if vals else float("nan") + print("milliseconds after the last sample of speech:\n") + prev = 0.0 + for stage, label in (("stt-vad-end", "VAD + turn model decided"), + ("stt-end", "speech-to-text finished"), + ("intent-start", "conversation started"), + ("intent-end", "answer ready")): + v = med(stage) + print(f" {label:26} {v:7.0f} ms (+{v - prev:5.0f})") + prev = v +asyncio.run(main()) diff --git a/dev/stt-compare.py b/dev/stt-compare.py new file mode 100644 index 0000000..b4c4999 --- /dev/null +++ b/dev/stt-compare.py @@ -0,0 +1,80 @@ +"""Which transcriber, and what it costs. + +The host runs wyoming-faster-whisper with `model = "auto"`. That looks like it +picks the best available, but the sherpa-onnx bindings live in an optional +extra that is not installed, so the Parakeet branch is unreachable and it falls +back to whisper-base-int8. This transcribes the same clips through both and +reports the text and the time. + +Start each server first; they are separate processes on separate ports. + + dev/py dev/stt-compare.py :