Repository navigation
Conversation
There was a problem hiding this comment.
Host control-flow/ownership review of c6af3262, limited to the new Metal wrapper and its buffer callers. The three earlier #939 fixes are present, including the Metal warmup bound. The shared AudioOut and returned-frame ownership blocks are byte-identical to #939's 3515c27f; the two-line cleanup patch already offered there still applies unchanged.
Two new error-path findings are below. Separate patches and CPU reproducers fix them at 85a5fdba + 5fd8de65. The actual complete Gpu wrapper runs over inert Metal objects, with the unchanged mapShards function body and real allocator-failure injection. On this head, three controls pass and nine assertions fail; all twelve cases pass with the patches in Debug and Fast. The fixture checks direct close and two failed commands, not just a second cleanup call after a failed commit. Source review found no remaining issue in this delta.
Whole-tree format, whitespace and lean pass (517/517, zero new). The complete Metal CLI compiles and links locally with Zig 0.17.0 and Xcode 26.6, without execution. Fork validation #15 has all six current-head checks green on the exact candidate, including all four No GPU commands, formatter and app/mobile workflows. The upstream No GPU run 38917505503 is action_required with zero jobs. These checks do not verify Metal device execution, model/codec fidelity or the performance receipts.
| cb.commit(); | ||
| g.enc = null; | ||
| g.cb = null; | ||
| try g.waitPending(); |
There was a problem hiding this comment.
[P2] Drain the new command before returning the previous command's failure
The new command is already committed and explicitly retained above. If waiting for the previous pending command reports CommandBufferFailed, this try returns before storing the new command or popping its pool. waitPending has cleared the old pending pointer, so a direct close() no longer waits for the new work before releasing buffers. Engine.destroy also relies on its initial flush before unmapping the shards.
The CPU transport reproducer sees resource teardown before the new command's wait. The patch waits/releases the newly submitted command on this error path and always pops the pool, while preserving the original error and the success-path pending reference.
| /// An address for a buffer made elsewhere (a mapped checkpoint shard); `own` releases it at close. | ||
| pub fn register(g: *Gpu, b: mtl.Buffer, own: bool) !u64 { | ||
| try g.bufs.append(g.gpa, b); | ||
| try g.owned.append(g.gpa, own); |
There was a problem hiding this comment.
[P2] Make registration atomic before transferring buffer ownership
If bufs.append grows successfully and this second append fails, the lists have different lengths. close() later iterates them as equal-length slices, so allocation failure leaves the wrapper unable to tear down normally. The creator also needs to release a newly created buffer if registration fails: this applies to alloc and the mapped-shard caller.
The offered second patch reserves both capacities before appending either entry, adds creator-side buffer cleanup on failed handoff, and frees mapShards' temporary address vector on error. The later indirect.append failure keeps ownership with Gpu, avoiding a double release. Real FailingAllocator cases reproduce both list-growth and caller-cleanup failures.
|
Thank you, @cwschroeder. Both findings were right, and they're fixed in
On the M5 Ultra at
🤖 Generated with Claude Code |
|
Thank you, @drowzeys, sharing the CUDA/Metal family structure and FP32 codec gives the merged Drama work a useful model baseline. We checked d88f709 and #939. Both generation paths return an EOS-shortened slice of a larger allocation; we will preserve the full owner or shrink it and add early-EOS cleanup tests. The shared config also needs named refusals for wrong JSON kinds. Metal already refuses unsupported sampled top-k, and that guard stays. We will combine this model with @plotarmordev's protocol/PTH and optimization work, then the common frame scheduler and native speech adapter. Please keep prompt-chunk failures, two-sided reference margins and long/reset/concurrent audio cases visible for the final fidelity gate. Current resource checks are separate from all-kernel numerical qualification. Posted by Ash's AI Assistant Synthena |
common.zig: the request settings, the parity probe, the streaming stats and the clock. shards.zig: the checkpoint's shards by tensor name and lm_head's candidate ids. check.zig and generate.zig no longer touch the CUDA runtime: the engine gives them scratch bytes and its peak allocation, and tts.CodecOnly opens the codec alone for --encode and --codec-check. No change to what the CUDA engine computes.
…rnels, codec, drama3-generate) zig/src/families/drama3: the Qwen3.5-MoE backbone (Gated DeltaNet and gated attention layers, 256 FP8 block experts top 8 with a gated shared expert) and the 13-layer codebook tower, decoding one frame of a main token and ten codes a step; prompt rows in the reference-bracket-v1 layout; the Fish S2 codec in fp32 for voice references (encode) and streamed audio (decode with carried state). zig/kernels/cuda/drama3*.cu: the glue, the samplers, the codec's kernels, and fp8_experts.cu's kernels on FP8 rows with per-row scales (the reference's dynamic activation scheme). cuBLAS (fp32 compute, no TF32) for prompt rows and the codec. zig build drama3-generate: text (and an optional voice reference) to WAV, the codec's encode, and --check / --codec-check against tools/drama3/oracle.py's dumps. tools/drama3: the PyTorch oracle (SGLang-Omni 416a70c0 semantics) and the one-time codec.pth conversion. Receipts are in the pull request.
…codec, drama3-generate on macOS) zig/kernels/metal/drama3: the backbone and tower glue, bf16 projection rows (one fma chain a lane, so a row's bits do not depend on the batch), Gated DeltaNet in fp32, gated attention, the router, FP8 activation rows, the FP8 block-scaled experts read in place from the checkpoint (e4m3 decoded exactly, the reference's group scales), the Philox samplers with RAS, and the codec's fp32 GEMM and glue. drama3.metallib is built ahead of time with safe math and precise fp32 functions. zig/src/families/drama3/metal: the same family over TensorFold's Metal runtime, behind the API drama3-generate and its --check / --codec-check already use, so parity is measured the same way as on CUDA. Every kernel reserves its threads; gate 5 holds all 38 launches and d3_rmsnorm passes gates 1-4. Receipts are in the pull request.
…tes 1-4 registry.zig: every drama3, drama3_fp8 and drama3_codec launch with the records kernel-contract-limits reads from fatbins built for sm_80, sm_86, sm_89, sm_120 and sm_121. contract_drama3.zig: d3_rmsnorm's plain-gamma form against the rows family's float64 RMS norm.
Co-authored-by: Christian Schröder <1188270+cwschroeder@users.noreply.github.com>
…inned Zig formatter The Metal backend's warmup had the same four-frame decode as CUDA's (ashhart#939 review by @cwschroeder). Co-authored-by: Christian Schröder <1188270+cwschroeder@users.noreply.github.com>
…ithout a model - The codec warmup decodes min(max_chunk, 4) frames, inside the audio buffer a short --chunk allocates. - Every WAV and PCM write, flush, header seek and close is checked; a failure ends the request before its RESULT line. - --codec CODEC --encode OUT --ref-audio REF.wav no longer needs --model, as the usage says. Reported in review by @cwschroeder; the warmup bound is his. Co-authored-by: Christian Schröder <1188270+cwschroeder@users.noreply.github.com>
…uffers atomically - commit waits for and releases the command it just queued when waiting for the previous one fails, and always pops the pool, so close() cannot release buffers under queued work. - register grows both slot lists before appending either; alloc and mapShards release a buffer whose registration failed, and mapShards frees its address vector on error. Reported in review by @cwschroeder. Co-authored-by: Christian Schröder <1188270+cwschroeder@users.noreply.github.com>
- AudioOut.open abandons what it opened if the first header write fails. - The frames speak returns are freed if finalizing the output fails. Reported in review by @cwschroeder. Co-authored-by: Christian Schröder <1188270+cwschroeder@users.noreply.github.com>
d88f709 to
b8e01ee
Compare
|
Rebased on main |
…actly their length Engine.generate and Tts.speak returned host[0..made], a prefix of the limit-sized allocation, so a caller freeing the result after an early end of speech freed the wrong length. frames.zig's owned() shrinks the allocation to the frames kept (realloc), and endOfSpeech() is the scan both paths share. Host tests cover an early end, no end, and an end at the first frame, freed through the testing allocator.
Config.parse read .object, .array, .string and .bool straight off the parsed JSON, so a string where a number belongs, or an array where an object belongs, trapped instead of failing. Each field is now matched on its kind and refused with error.ConfigNotObject, ConfigNotInteger, ConfigNotArray, ConfigNotString or ConfigNotBool, naming the field in a warning. Token ids that do not fit int32 are error.BadConfig rather than a trapping cast, and the semantic band's width is computed in i64. Host tests feed the shipped config with one field changed for each kind.
# Conflicts: # zig/src/families/drama3/drama3.zig # zig/src/families/drama3/engine.zig # zig/src/families/drama3/tts.zig
… of exactly their length The Metal Engine.generate and Tts.speak returned host[0..made] of the limit-sized allocation, as the CUDA paths did. They now use frames.zig from the merged CUDA fix: endOfSpeech() for the scan and owned() to shrink the allocation to the frames kept, so drama3-generate frees the length it holds. frames.zig's host tests run in the Metal family's tests too.
|
Thank you. Both points are fixed; the head is now
Metal's guard that refuses unsupported sampled top-k is unchanged. Checks on the M5 Ultra:
These are build and host tests only; I didn't do a device run. The prompt-chunk failure cases, reference margins and long/reset/concurrent audio cases are all kept. |
cwschroeder
left a comment
cwschroeder
left a comment
There was a problem hiding this comment.
Scoped host-ownership re-review of ec6f16dc on main 6ee828c2: the two P2 findings in my earlier review are closed. Astra independently found no material regression in the adopted cleanup or the new EOS/config delta.
The published CPU fixture runs the complete unchanged Gpu source and the actual mapShards body over inert Metal objects. All 12 cases pass in Debug and Fast at this head. Running the same fixture against original c6af3262 still gives 3 pass / 9 fail, covering the missing drain, pool cleanup and allocation ownership. The five actual stdio/frame-cleanup cases also pass in both modes. No device is opened.
The shared frames/config tests pass 5/5 in Debug and Fast, including early EOS and named wrong-kind refusals. Restoring only the old shortened-view return in a temporary copy makes the early-EOS test abort at the wrong-length free. Source review confirms all four CUDA/Metal generate/speak callers use the owned result and retain cleanup on realloc failure.
Whole-tree format and patch whitespace pass; lean is 517 on both pinned main and this head, zero new. Current-head remote checks are absent. This approval covers the host ownership changes reviewed here; it does not qualify device execution, model/audio fidelity, the remaining kernel gates, scheduling, or performance. No deployment or model run was performed.
|
Thanks @drowzeys for carrying the shared config/EOS fixes from #939 and fixing both Metal return paths too. Freeing the exact owned length after early EOS, including zero frames, is the important ownership control. We are keeping CUDA and Metal behind one Drama family and shared frame lifecycle, with both contributors credited. The unsupported sampled-top-k refusal remains explicit. The host checks do not establish prompt/decode fidelity, chunk invariance or long/reset/concurrent audio behavior on the merged backend. Those numerical, resource and served checks remain required, rather than inferred from the generator compiling. Posted by Ash's AI Assistant Synthena |
…tal, with #915-#926 and #939 beneath them #927's stack brings the process-wide cap on device and pinned host bytes, direct reads a file system refuses going through the page cache, Module.holds, the Drama 3 backbone, tower and speech codec on CUDA (codec.pth read without Python), speech requests and audio events in engine_api, POST /v1/audio/speech with streamed PCM, and its decode kernels; #956's (with #939) the family's shared parts, its Metal backend and drama3-generate on macOS. Laid as the maintained series: the bridges between the two, qualification, and owners that keep buffers and modules until their completion is proved. On this line the cap's reservations join the pinned peaks' accounting, the learner's API moves to core/learning_api.zig with engine_api.zig re-exporting it (Example keeps its context row), the CUDA family list holds LFM2 and fish_omni, and both build files keep this line's layout with Drama's steps added. The tests are named for what they check.
#956 registered d3_rmsnorm under gates 1-4 and its 45 CUDA launches under gate 5, and the Metal kernels with their sources; the maintained series kept the adapters (contract_drama3.zig, metal_drama3.zig) but not their wiring. The CUDA runner maps the drama3, drama3_fp8 and drama3_codec images and runs d3_rmsnorm's gates (gate 5 failed on d3_gemv's missing image), and the Metal contract takes Drama's norm, its sources and its launches' limits.
tools/drama3/oracle_prompt.py is a copy of SGLang-Omni's prompt.py, and the oracle, its model half and the codec converter import it; code copied from another project does not land, so the four scripts stay on #927's and #956's branches with the 25 archived #927 diagnostic programs (oracles_927/*.zig.txt, whose originals are in #927's merged history) and the series' provenance note. Serving reads the checkpoint and codec.pth in Zig, so nothing a user runs changes; the notices and the codec's comments no longer name the scripts.
|
Thank you, @plotarmordev, for the speech protocol and infrastructure, and @drowzeys for the family, backends and codec. The selected work from #956 is included through #927 ( Posted by Ash's AI Assistant Synthena |
|
The consolidated Drama contribution shipped in 1.1.0 as Posted by Ash's AI Assistant Synthena |
…on CUDA and Metal, with ashhart#915-ashhart#926 and ashhart#939 beneath them ashhart#927's stack brings the process-wide cap on device and pinned host bytes, direct reads a file system refuses going through the page cache, Module.holds, the Drama 3 backbone, tower and speech codec on CUDA (codec.pth read without Python), speech requests and audio events in engine_api, POST /v1/audio/speech with streamed PCM, and its decode kernels; ashhart#956's (with ashhart#939) the family's shared parts, its Metal backend and drama3-generate on macOS. Laid as the maintained series: the bridges between the two, qualification, and owners that keep buffers and modules until their completion is proved. On this line the cap's reservations join the pinned peaks' accounting, the learner's API moves to core/learning_api.zig with engine_api.zig re-exporting it (Example keeps its context row), the CUDA family list holds LFM2 and fish_omni, and both build files keep this line's layout with Drama's steps added. The tests are named for what they check.
ashhart#956 registered d3_rmsnorm under gates 1-4 and its 45 CUDA launches under gate 5, and the Metal kernels with their sources; the maintained series kept the adapters (contract_drama3.zig, metal_drama3.zig) but not their wiring. The CUDA runner maps the drama3, drama3_fp8 and drama3_codec images and runs d3_rmsnorm's gates (gate 5 failed on d3_gemv's missing image), and the Metal contract takes Drama's norm, its sources and its launches' limits.
tools/drama3/oracle_prompt.py is a copy of SGLang-Omni's prompt.py, and the oracle, its model half and the codec converter import it; code copied from another project does not land, so the four scripts stay on ashhart#927's and ashhart#956's branches with the 25 archived ashhart#927 diagnostic programs (oracles_927/*.zig.txt, whose originals are in ashhart#927's merged history) and the series' provenance note. Serving reads the checkpoint and codec.pth in Zig, so nothing a user runs changes; the notices and the codec's comments no longer name the scripts.
This puts the first speech model on TensorFold's own Metal runtime. Fish Audio's Drama 3 (a 35B-A3B MoE backbone with Gated DeltaNet, a 13-layer codebook tower and a neural codec) now runs end to end on Apple Silicon from Zig and Metal. There is no MLX, no Python and no PyTorch at run time. Text, and optionally a voice to clone, go in; streamed audio comes out. It is the Metal half of #939 and shares that pull request's config, prompt, checkpoint layout, WAV code and check tools. TensorFold could already run text models on both backends; with this, an audio model whose step is a frame of eleven columns runs on both too, on the same runtimes, kernel contract and lean code as the video work.
On one Mac Studio (M5 Ultra, 256 GB):
--check. Every differing pick is listed. The largest margin is 0.25, the same as on CUDA, where the figures are 98.80 % and 97.97 %.d3_rmsnormpasses gates 1–4.zig build testpasses every step on macOS.Precision is the checkpoint's own, as on CUDA:
per_token_group_quant_fp8), and the checkpoint's block scales are applied per 128-input group.drama3.metallibis built with safe math and precise fp32 functions.Next steps:
top_kover 64. Metal has only the fast sampler (32 KiB of threadgroup memory cannot hold the full sort's 64 KiB of keys). Largertop_kis refused with an error; Fish's default is 30.What it adds
On top of #939, in two commits:
drama3: share the backend-independent parts before a Metal backend. No change to what the CUDA engine computes.common.zigholds the request settings, the parity probe, the streaming stats and the clock.shards.zigholds the checkpoint's shards by tensor name and lm_head's candidate ids.check.zigandgenerate.zigno longer touch the CUDA runtime. The engine gives them scratch bytes and its peak allocation, andtts.CodecOnlyopens the codec alone. The same two files now drive both backends.drama3: Fish Audio Drama 3 text to speech on Metal:zig/kernels/metal/drama3/(one metallib, built ahead of time):dense.metalholds the bf16 projection rows, with 4 columns a simdgroup and one fma chain a lane, so a row's bits do not depend on how many rows run. It also holds the zero-centred norms, the embeddings, SwiGLU, the MoE combine and the tower's input mix.gdn.metalholds the causal conv and the gated delta rule in fp32.attn.metalholds the q / k norms with rotary, and gated attention for prompt and decode rows alike.moe.metalholds the router, the FP8 activation rows, the prompt's routing plan, and the FP8 block-scaled experts read through per-layer pointer tables into the mapped shards.sample.metalholds the greedy picks, the Philox race with RAS (drama3: Fish Audio Drama 3 text to speech on one CUDA GPU (new family, plain decoding; proposal) #939's fast path), and the frame's bookkeeping.codec.metalholds the codec's fp32 GEMM (a causal conv's taps in one pass) and its glue.zig/src/families/drama3/metal/:gpu.zig(the queue, the pipelines, and buffers addressed asu64, so the shared tools work unchanged),weights.zig,engine.zig,forward.zig,kernels.zig,codec.zig,codec_encode.zigandtts.zig.drama3_metal.zigis the family's Metal root.zig/build/drama3_metal.zig, plus 4 lines inbuild.zig: the metallib, the family module, andzig build drama3-generateon macOS.registry.zig;zig/tests/contract/metal_drama3.zig(gates 1–4 ford3_rmsnorm, and the drama3 sources for gate 5); a few lines in the runner; and the sources inzig/kernels/metal/sources.zig.Each frame is encoded straight into a serial compute encoder and committed as soon as it is encoded, so the next frame encodes while the GPU runs the last one. Device scalars (position, step, counts) advance on the GPU, as on CUDA, so a burst of frames needs no host round trip.
Receipts
Machine and settings for every run:
-std=metal4.1)zig build drama3-generate(ReleaseFast)e0a9218aa48888270f06ef1ed9ea67622c0e17d2(including itscodec.pth)Another process on the machine: during the timed runs an Ollama server, which is not part of this work, held a 35B model in memory. I did not measure whether it was busy.
The prompts are #939's public ones:
Sampling uses Fish's defaults (temperature 1.0, top-p 0.9, top-k 30) with seed 1.
Fidelity against the PyTorch oracle (
drama3-generate --check, #939's dumps)Teacher-forced, 300 frames each. My greedy pick is compared with the oracle's at every column.
Prompt pass, clone (168 rows):
Greedy: as on CUDA, free-running greedy frames differ from frame 0 on a near tie (frame 0, codebook head 0: a margin of 0.125 on both sides). Token-for-token greedy parity does not hold; the teacher-forced numbers above are the fidelity claim.
Intelligibility: Whisper large-v3-turbo transcribes the plain run's audio as "The birch canoe slid on the smooth planks. Glue the sheet to the dark blue background." It transcribes the clone run's as "Fine. If we miss it, it's on you. Well, maybe that's fair." Whisper does not transcribe the bracketed directions.
Codec (
drama3-generate --codec-check)Performance: this port against the MLX engine on the same machine
The MLX engine is the fork's
drama3-mlxbranch in its own venv. It quantizes the experts to 8 bits, where this port serves the checkpoint's FP8. The two therefore make different frames from the same seed, and each run is timed over its own frames. Each was run with streaming on (--streamfor MLX; this port always streams) and without its own speech check (--no-verifyfor MLX). The runs alternated (Metal, MLX, Metal, MLX, ...), three each.--codecleft out), the plain run makes 91 frames/s.time -l). The experts' 32.2 GB are file pages read in place, so they are not in that figure. The runtime's own count of Metal buffers is 48.5–48.9 GB, because it includes whole mapped shards. The MLX engine's figure is its own peak footprint.Determinism and request state
f5d12873…; clone frames are868b41e6….--max-rowswere not tested.Kernel contract (M5 Ultra)
d3_rmsnorm, plain-gamma form, against the rows family's float64 RMS norm, as on CUDA.max_total_threads_per_threadgroup), so none needs an M1/M2 compile record.Lean check and tests
python3 tools/zig/lean_check.py: 517 problems on drama3: Fish Audio Drama 3 text to speech on one CUDA GPU (new family, plain decoding; proposal) #939's head and 517 on this branch, so there are no new ones. Every new file is at or below 600 lines.zig build test -Dcpu=apple_m1, fresh cache, on #939zig build test -Dcpu=apple_m1, fresh cache, this branchzig build drama3-generate -Dtarget=aarch64-linux-gnucompiles and links.zig build test-cuda-hostpasses.zig build test. The macOS failures drama3: Fish Audio Drama 3 text to speech on one CUDA GPU (new family, plain decoding; proposal) #939 reported came from a machine without the Metal toolchain.Not tested, or not in this pull request
top_kabove 64 (see above).--max-rows(1,024 rows), and contexts near--max-ctx 4096.Licenses
As in #939:
fishaudio/drama-3is under the Fish Audio Community License Agreement v1.0, and users download it and accept that license themselves.codec.metal,metal/codec.zigandmetal/codec_encode.zigare a separate implementation of the codec arithmetic, with no Fish Speech source. drama3: Fish Audio Drama 3 text to speech on one CUDA GPU (new family, plain decoding; proposal) #939's question about that license applies to them too.Reproducing
This branch's own commits: drowzeys/TensorFold@upstream/drama3-cuda...upstream/drama3-metal
🤖 Generated with Claude Code
https://claude.ai/code/session_01PfjmufXm4L7qsVNFyRw9ZK
The review fixes on #939 (warmup bound, checked audio writes, encode without a model) are merged into this branch, and the same warmup bound is applied to the Metal codec. After them, on the M5 Ultra:
zig build test -Dcpu=apple_m1passes 216/216 steps, 3,002 of 3,012 tests (10 skipped), andzig fmt --check build.zig zigis clean.The Studio that runs Drama 3 is #941, and its Voice tab is #944.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PfjmufXm4L7qsVNFyRw9ZK