[drowzeys]

Stacked on #943 (upstream/studio-flux3k), which is stacked on the Motion Previs and Slate Studio PRs; merge after all three.
Commits of this PR only: drowzeys/TensorFold@upstream/studio-flux3k...upstream/studio-voice (one commit).

The Studio now speaks, with TensorFold's own Drama 3 engine on both machines. The Voice tab turns text, a dialogue or a Slate voice sheet into speech with Fish Audio's Drama 3, run locally as a persistent TensorFold server: the native CUDA port on a DGX Spark (the engine in #939) and the MLX port on a Mac. A take plays while it is made, is checked word for word, and can go into a clip as MiniMax H3's reference audio, so the voice you cast is the voice the character speaks with.

Measured through the app, one run each (studio/docs/voice.md):

	DGX Spark (GB10, 128 GB), drama3-cuda 81173e6	Mac Studio M5 Ultra, 256 GB, drama3-mlx 92c09a8
One speaker, 16 words, first take	8.2 s of audio; engine 5.9 s (30.9 frames/s); 16 of 16 words	8.0 s of audio; first sound 0.21 s on the engine's clock; 16 of 16
Cloned voice from a 3.9 s MP3 clip	6.1 s; engine 5.0 s, first sound 0.47 s; 16 of 16	6.6 s; first sound 0.52 s; 16 of 16
Two speakers, three lines	11.6 s; engine 8.4 s (30.2 frames/s); 20 of 20	10.6 / 9.4 / 11.1 s (seeds 1 to 3); first sound 0.23 to 0.30 s; 20 of 20 each
A 120-word read	55.7 s; engine 40.8 s (30.7 frames/s); 118 of 120	46.4 s; 120 of 120

• Words heard are Whisper's transcript checked with the engine's own rules (app/wordcheck.py, copied from TensorFold's verify.py); on a Spark a take with missing words is drawn again, not played, and the best kept.
• The voice carries: a cloned take came out at about 130 Hz against the clip's 125 Hz.
• Memory: a take needs about 50 GB free when it starts on a Spark; 48.5 to 49.9 GB on the Mac (55.7 GB with a redraw). The server holds the Studio's GPU lock while loaded, stands under a memory kill-guard, and is unloaded after 300 idle seconds and before any other job of the queue, so Drama 3 and MiniMax H3 never run together.

Why it matters for TensorFold: speech, pictures and video with sound on one runtime, in one queue: a voice made by TensorFold's Drama 3 is fed back as the reference audio of a clip made by TensorFold's H3 engine.

What it adds

• studio/app/voice.py (the tab's API, takes, the queue's voice jobs) with voice_text.py (speaker tags, parts of about 60 words, dialogue a turn a line, the word match), voice_machine.py (the GPU lock and memory reads), voice_served.py (the persistent-server protocol both engines speak, live playback), voice_engines.py (the MLX and CUDA back ends); app/wordcheck.py.
• The page: static/voice.js, the Voice tab, and Cast voice on Slate's Sound sheets (slate.js, slate.py: a cast take goes into shots that open on cast stills as <Audio N>, within MiniMax's limits of 2 to 15 s a clip and 15 s in all).
• Shared code it touches: server.py (the router, a queue entry for voice jobs), job_run.py (Drama 3 unloaded before any other local job), job_build.py and server.py (a loaded Drama 3's memory counts as free for a clip), api_jobs.py (the tab's state), clear.py (voice takes go to the trash like clips).
• Install: scripts/setup-voice.sh (oneshot-setup.sh --voice): the Mac engine in its own environment; on a Spark it checks the hand-built CUDA engine and prints the steps for what is missing.
• Tests: tests/test_voice.py (11) and tests/test_voice_cuda.py (5) with stand-in engines: sheets to directions, the word check, parts and turns, the lock, live playback and retakes on the MLX server, a cast voice into a shot, the CUDA server holding the lock until unloaded, a long take keeping one voice, an H3 job unloading Drama 3 first, and a take with missing words drawn again.
• Docs and notices: docs/voice.md, NOTICE, CREDITS.md, AGENTS.md; the Studio's title gains built with integrated Fish Audio Drama 3, and the tab carries "Built with Fish Audio".

How it relates to the engine and to app/

The Studio never imports Drama 3: it starts the engine as a separate process and speaks JSON lines with it. On a DGX Spark that is drama3-generate --serve 1 from the CUDA family in #939; on a Mac python -m tensorfold.drama3 --serve from the MLX port (the Metal port follows #939). When those land, setup-voice.sh can build or install them from main instead of the fork branches. In the desktop app the same server could back a voice feature directly, with the Studio's protocol (voice_served.py) as the client.

Tests

	Passed	Failed	Skipped
DGX Spark (GB10, 128 GB)	131	3	0
Mac Studio M5 Ultra, 256 GB	119	15	0

All 16 voice tests pass on the Spark; on the Mac the 11 in test_voice.py pass and the 5 in test_voice_cuda.py fail because they drive the Linux-only CUDA path (as on the Studio's 2.8.0 tree). The other failures are the environment ones listed in #941.

Lean check

python3 tools/zig/lean_check.py: 1941 files, 517 problems, unchanged (zig/ and tools/zig/ only). Over studio/ with the same rules: 21 findings, the same as #943; voice.js adds one false hit ("audio/*" in a file input's accept list).

Not tested

• Takes through the split tree on the real engines (the receipts are from the Studio's 2.7.5 to 2.7.8 code; the split moved functions unchanged, with one edit: Served.stop now removes the lock through voice_machine.LOCK, the single patchable name the module global was before).
• The CUDA engine's setup from nothing: it is built by hand today (setup-voice.sh prints the steps).
• Voices other than the ones listed, other languages, and more than one seed per row.

What comes next

• Install the engines from main once Drama 3 lands there (CUDA in drama3: Fish Audio Drama 3 text to speech on one CUDA GPU (new family, plain decoding; proposal) #939, Metal after it), so --voice needs no hand steps on a Spark.

Licences

• Fish Audio Drama 3 is under the Fish Audio Community License v1.0: free for non-commercial use and for commercial use below US$500,000 annual revenue; a hosted text-to-speech or voice-cloning service needs Fish Audio's own licence; cloning a real person's voice without consent is not allowed. No Drama 3 weights ship: they are downloaded from Fish Audio's release when a user installs them. Built with Fish Audio.
• A take with a voice to clone needs the consent box ("consent": true through the API). Fish Audio's cloud API is never called.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PfjmufXm4L7qsVNFyRw9ZK

[ashhart]

Thanks @drowzeys for the voice workflow and the candid 118/120 versus 120/120 word checks on the long read. The cast-voice handoff, playback and unloading speech before a video job are useful app ideas. As explained in #941, the media workspace is being built in TensorFold's app itself, and we'll credit those ideas as the stages land. #944 closes with its corresponding stage rather than importing a separate application. The app's voice release timing is separate from the 1.1.0 engine release, and the merged Drama engine still needs its own qualification. Nothing further is needed from you on this PR now.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you. Our plan for the whole Studio stack is in #941: we're folding it into the TensorFold app with your authorship kept. We'll keep this open and link the pieces here as they land.

- Synthema, Ash's AI Assistant.