[plotarmordev]

What this changes

cuda.setAllocationLimit caps the bytes this process holds in device buffers plus pinned host buffers (part of
#914). Each allocation reserves before the driver call, so a refused one never reaches the driver and a failed
driver call refunds its reservation.

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision:
  • Commit 881cccaa4d87, on main c7137bb3. No MLX or PyTorch.
  • An NVIDIA GB10 build host on Ubuntu 24.04.4 LTS aarch64 (Linux 6.17), with Zig 0.17.0 and nvcc 13.0.88.
  • No checkpoint is involved.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: not run.
  The change is limited to allocation accounting and touches no token or output path.
• Decode speed before and after, from tools/bench_openai.py: not run. Allocation accounting now reserves under the
  counts lock before the driver call, and pinned-host accounting also takes that lock. Decode performance was not
  measured.
• Prompt speed before and after: the cold first request's ttft_s from tools/bench_cache.py --require-cold on a long prompt, or "n/a" with the reason when the change cannot reach prompt processing: not run, for the same
  reason as decode.
• Tests run, with pass, fail and skip counts:
  • CUDA_VISIBLE_DEVICES= zig build test-cuda-host --summary all from an empty local cache: 27/27 steps, 467 passed, 0 failed, 0 skipped.
  • Two new tests, both CPU-only. "allocation cap includes pinned and in-flight bytes and refunds failed driver
    calls" covers the accounting. "public allocations refused by the cap skip the driver, and failed driver calls
    are refunded" drives DeviceBuffer.alloc and HostBuffer.alloc through a fake driver table whose allocations
    fail. A refusal by the cap must not call the driver, a driver failure must refund its reservation, and usage
    and the peak must stay unchanged.
  • The same suite with DeviceBuffer.alloc's refund (errdefer finishAllocation(null, len)) removed fails the
    second test: 151 passed, 1 failed (that test).
  • zig build native -Dnvcc=/usr/local/cuda/bin/nvcc --summary all: 42/42 steps.
  • python3 tools/zig/lean_check.py: 527 problems, the same as main's 527 (tools/zig/lean_new.py: 0 new).
  • zig fmt --check build.zig zig: clean.
• Not run, and why:
  • Metal: no macOS host here, and the change is in the CUDA runtime only.
  • A real driver allocation failure on a GPU: the failure is injected through the driver table.
  • Speed: see above.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (No token
  path is touched.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Not
  run; no output path is touched.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No
  arithmetic changes.)
• Prompt processing is no slower than the last release. (Not measured.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (Both need no
  GPU. The allocator test fails with the refund removed. The accounting test calls the new API, so before the
  change it does not build.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thank you, @plotarmordev, the public-wrapper refusal/refund tests give the combined Drama family a useful allocation gate.

We will land #915 with that one native family, combining your stack and @drowzeys' work with both credited. The cap reserves device and pinned bytes before the driver call and refunds failures.

We will retain the strict cap tests, add an overlapping-allocation control and bind configuration/restoration to the combined family owner. Its ordinary unset-cap and memory/performance checks remain part of that landing. This resolves the first-consumer hold; the family still needs its lane, fidelity and audio gates.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thank you, @plotarmordev, for the speech protocol and infrastructure, and @drowzeys for the family, backends and codec. The selected work from #915 is included through #927 (5dcf77d7c), the consolidated Drama 3 landing in 1.1.0. One CUDA/Metal family uses bounded shared frame rounds, request-owned state, native checkpoint/codec loading and streamed PCM speech. HTTP retains its checked default-voice subset. Full scope and audio, interleaving and completion results are on #927: Apple M5 Ultra norm checks passed 1/1 and CUDA resources passed; checkpoint audio fidelity, seeded concurrency and cancellation remain unrun.

Posted by Ash's AI Assistant Synthena

[ashhart]

The consolidated Drama contribution shipped in 1.1.0 as 5dcf77d7c; thanks @plotarmordev and @drowzeys. Experimental attention variants remain separate. We're closing it as landed.

Posted by Ash's AI Assistant Synthena