[jayleaton]

What this changes

cuda: a keyed graph cache and a device arena, off until a family installs them; failed launches name their kernel. Depends on #634. This cumulative PR targets main; the incremental range is e7c523eb..c58d83ca.

#634 → #635 → later DeepSeek-V4.1 layers

Commits in this layer:

• 3688f3ce CUDA keyed graph cache and device arena, both off until a family installs them.
• b7be793d A failed CUDA kernel launch names its kernel and launch configuration; CUresult names fall back to a table when the driver cannot answer.
• faf59014 fix(cuda): correct RDMA VMM capability attribute
• 3cc03974 Graph cache: a capture is freed until the map holds it, the keep floor stays under the max, a max of 0 is eager
• c58d83ca CUDA arena: the address maps are rehashed before tombstones fill them

The combined correction removes personal MIT headers while retaining third-party notices, relocates development Python tools, and preserves stored fixtures with native consumers. Expanded provenance covers the EXL3 kernel stack; no copied text was found by the performed scan, and maintainer copy scanning remains authoritative.

Receipt

Before: public head cee12a2b. After: exact corrected head c58d83ca53086a0f60944ce050cab5167838f40a, gated on 2026-10-12, Linux x86_64 and Zig 0.17.0.

Executed gate output:

G635: commit c58d83ca53086a0f60944ce050cab5167838f40a tree c8cb7982fc6878c96bc1c2e5a08f64fd9ef37854 dirty 0 2026-10-12T04:28:38Z
install rc=0 Build Summary: 8/8 steps succeeded
test rc=0 Build Summary: 39/39 steps succeeded; 475/475 tests passed
zig fmt --check rc=0
git diff --check rc=0
c58d83ca lean: 1172 files, 534 problems | new: 0 | gone: 0
FAILED STEPS 0

Lean uses pinned main e7c523eb: 534 inherited findings, zero new or removed identities. The mode gate covers all 57 introducing pins with zero executable-mode losses. Family suites and CLI receipts run where introduced. Host checks do not qualify checkpoint inference, GPU execution, concurrent/resumed serving parity, prompt/decode speed, or macOS/Metal behavior on this rewritten tree.

The family documentation pins development tools at recipe commit c1cb9184.

Exactness and prompt/decode speed: not run on this host, which has no GPU or checkpoint weights. No new performance claim. MLX/PyTorch/checkpoint revision: not applicable to the host gate.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• Prompt processing is no slower than the last release.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

Merge Danger

Door: two-way. Reverting restores the parent without a data migration.

Blast Radius: engine. The incremental commits define the affected runtime, build and test surfaces. The production branch is unchanged.

[jrb-ai]

I reviewed 5fa8ad0fd9c53131c599c39f8885117d5a582fda and prepared two focused fix commits with regression tests, plus a separate reproducer/receipt commit. The code fixes have passed a bounded, model-free CUDA check on one DGX Spark. Compare branch. Please take or cherry-pick whichever parts fit.

Findings and changes

1. RDMA-with-VMM device attribute. gpu_direct_rdma_with_cuda_vmm_supported is 110, while 126 is MEM_SYNC_DOMAIN_COUNT in the CUDA driver ABI. This Spark returns 0 for 110 and 4 for 126, so the current query sets the RDMA allocation flag incorrectly. The submitted code fails the VMM arena reuse check; changing only 126 to 110 passes the identical probe. The exact failing VMM driver-call return code was not instrumented.
2. Captured executable ownership. A successful capture can be lost if the following Agree callback throws or the map insertion runs out of host memory. The fix reserves map capacity and agrees on success before capture, then keeps scoped ownership until an infallible insertion transfers it to the cache.
3. Rank-local invalidation versus a cache hit. One rank can invalidate its fingerprint and enter the miss-path agreement while a peer replays without participating. Two host threads with blocking agreement reproduce the mismatch. The fix agrees on active state, fingerprint validity and cache hits before branching, invalidates all ranks when needed, and coordinates allocation failure before capture.
4. Configured cache maximum. With max = 2 and the default min_keep = 8, memory pressure can raise the cap above the maximum; max = 0 can later capture too. The fix clamps the floor/cap, keeps zero-max caches eager, and allows a positive-max cache to recover from a zero pressure cap.

Commits

• 2b9d983e: CUDA attribute correction and ABI regression.
• 26c82f71: graph ownership, rank decisions and cache bounds, with regressions.
• 82ecb0f9: portable tests and validation receipt, separate so it can be omitted when taking the fixes.

Test evidence

Exact Zig 0.17.0; DGX Spark / GB10, Ubuntu 24.04.5 LTS aarch64, NVIDIA driver 580.178.04.

Check	Submitted 5fa8ad0f	Code fixes
zig build test-cuda-host -j2 on Linux	293/293 pass	305/305 pass
Same target on macOS	286 pass, 2 Linux-only skips	298 pass, same 2 skips
Six independent regression probes on macOS	0/6 pass	6/6 pass
Independent probes on Linux	Not run	6/6 pass
Same model-free CUDA probe on Spark	VMM reuse assertion fails	Plain arena, graphs and VMM pass
Lean checker	542 inherited findings	542, zero new

The device probe covers capture/replay, eviction, fingerprint invalidation, an injected capture-body failure and eager recovery. The runtime files match the qualified source byte-for-byte. The published probe differs only in formatting, and the independent host probes only in comment formatting; hashes and commands are in the receipt. No checkpoint was loaded, no device reset was performed, and existing services stayed unchanged.

Multi-rank trade-off

With Agree installed, a cache hit now makes three agreement calls instead of zero, and a successful capture miss makes six instead of two. This fixes the demonstrated decision mismatch, but the collective/host-synchronisation cost has not been measured on a GPU pair. With no Agree, the checks stay local. I would treat the rank change as a correctness candidate needing your multi-rank cost review; the attribute fix can be taken independently. Actual multi-GPU/NCCL execution, model serving, performance and quality are outside this validation.

Jonathan Regan-Beasley, contributing in a private capacity. This does not represent the views or positions of any organisation.

[jayleaton]

Thank you, Jonathan. This was a careful review, and all four findings are real. The stack is rebased onto current main, and this layer now has two new commits. Head: 5fa8ad0f → 842c32e4.

Taken

• 2b9d983 (attribute 110), cherry-picked unchanged as your commit (85220b9). cuda.h agrees: GPU_DIRECT_RDMA_WITH_CUDA_VMM_SUPPORTED = 110 and MEM_SYNC_DOMAIN_COUNT = 126.
• 26c82f7, taken in modified form as 842c32e, with you as co-author:
  • Ownership (2): the executable stays owned until an insert that cannot fail, and the map slot is reserved before the capture.
  • Bounds (4): min_keep is clamped to max, and max = 0 is eager. A cap lowered to 0 recovers once every rank has room.
  • Regression tests: your tests are kept in graph_cache_regression_test.zig, adapted to the call counts below. Against the old cache, 6 of the 19 tests fail: the leak, the OOM, the floor above max, max = 0, and the 0-cap recovery. With the fix, 19/19 pass.

Changed: rank-local invalidation (3). You're right that a pointer fingerprint can change on one rank and not on the other. I fixed that through the contract instead of with more agreements:

• With Agree installed, the caller passes a fingerprint that changes at the same step on every rank (a layout generation). fingerprintOf is documented as single-rank only.
• Settings are the same on every rank, and failed is only set after an agreed verdict, so the "active" vote isn't needed either.
• Hits therefore make no collective, and a miss still makes two. A failed host-map reserve is folded into the existing room vote.

The reason is the cost on our decode path. Each agreement there is an allGather plus a host sync on the compute stream. Three of them per hit would stop a graph from being queued while the previous one still runs.

Declined: 82ecb0f. It is a one-PR review harness and receipt, and tools/zig/ holds reusable checkers. Your receipt numbers are quoted here instead. cuda_gate.zig could become a generic device check later if that is wanted.

Gates on the new head, host only (Linux x86_64, Zig 0.17.0): lean_check.py 534 problems, the same as main (0 new); zig fmt --check 0 files worse than main; zig build 8/8 steps; zig build test 39/39 steps, 474/474 tests. test-golden is a macOS-only step, on main too.

[jayleaton]

Added cee12a2b: the arena's address maps are rehashed before tombstones fill them (a fix from production, with a test that fails without it); head cee12a2b, gates in the body.

[cwschroeder]

Independent receipt for cee12a2 on two ThinkStation PGX (GB10), aarch64 Linux, sm_121. Zig 0.17.0, driver 580.178.04, CUDA 13.0.2 (nvcc 13.0.88), NCCL 2.28.9+cuda13.0.

zig build test passes 475/475 per machine; a fresh test-cuda-host run passes 347/347. For the keyed cache and arena, a standalone Zig receipt helper outside this PR imports the head unchanged and passes 15/15 device check groups on each GB10: Plain and VMM arenas, byte round-trips, reuse, large releases, borrowed/pre-hook buffers, and 1,024 small/large allocation/free cycles per backend; real CudaEngine capture/replay, LRU, fingerprint invalidation, eviction and recovery through an injected room hook, max 0 eager execution, and fallback after an injected capture error.

Existing tf-cuda-test smoke, graph, launch-ex, ptx and one-rank nccl checks also pass on both. Both devices report VMM supported and RDMA-with-VMM unsupported. Cross-rank cache agreement remains host-tested only. No PR code changes or model/checkpoint qualification. The helper is available if it would be useful as a device mode here.

[ashhart]

Thank you, @jayleaton. This is a big piece of work. At #695 the stack adds about 85,000 lines in 366 files, and our lean check passes on all of it: no new findings, and the largest new file is 596 lines. We read through the whole stack this evening. Three things need to change before any of it can land. Then there's the order we'd like to take it in.

Python tools. The stack adds 40 Python files under tools/zig/ (dsv41_*, dsv41_bench/, dsv41_triton/). Some import torch, and dsv41_decode_ref.py imports a Python tensorfold.families.deepseek_v41 package. Since 10 October the rule is that no tooling in the tree imports the engine from Python, and the Python engine itself is frozen. The fixtures those tools produce are fine to keep. Could the checks that use them be Zig tests against the stored fixtures? Tools that need torch or Triton could live in your own repository, linked from the family's docs.

License headers. Thirty files carry SPDX-License-Identifier: MIT with a personal copyright line. TensorFold is Apache-2.0, and no file in the tree has a per-file header; contributions come in under the project's license. Please drop the headers. Your authorship stays in the commits and the CHANGELOG credit.

Provenance. experts_grouped.cuh and decode.cuh say "After ExLlamaV3, MIT, Copyright (c) 2025 Turboderp". We read other projects to learn from them, but we don't copy their code, so we need to know what came from ExLlamaV3. Is it only the EXL3 format, which THIRD_PARTY_NOTICES already covers, or code structure too? We'll run our copy scan against ExLlamaV3 either way.

Order. We'd take the shared pieces first, each reviewed and gated on its own:

1. cuda: a keyed graph cache and a device arena, off until a family installs them; failed launches name their kernel (DeepSeek-V4.1 stack 2/14) #635, the keyed graph cache and device arena.
2. fabric: tensor parallelism across CUDA processes (bootstrap, NCCL, mailbox and RoCE collectives, plan link, fail-fast) (DeepSeek-V4.1 stack 3/14) #636, tensor parallelism across CUDA processes. Loading NCCL with dlopen fits our rule that outside libraries are used only at runtime.
3. sessions: session memory for paged-KV families (page pool, prefix trie, RAM residency budget, streamed NVMe tier) (DeepSeek-V4.1 stack 4/14) #637, session memory.
4. lanes: a family's round planner drives the lane host with prompt pieces, and a halt ends the rounds at a drain deadline (DeepSeek-V4.1 stack 6/14) #662's planner hook in the lane host, and server: graceful drain and pre-token keepalives (DeepSeek-V4.1 stack 7/14) #663's server hooks.

On #663, we'd like to understand the compatibility wire (TENSORFOLD_WIRE=compat) before it lands: a second HTTP surface shaped after a Python serving app is a lot to keep stable.

The family layers would follow, gated on CUDA machines that hold the whole model, under the same contract as every family: drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.

None of this will make 1.0.8, which is closing now. If you can make those three changes, we'll start with #635.

[jayleaton]

The combined correction is published across the stack.

On provenance: decode.cuh and experts_grouped.cuh are TensorFold's own 0.3.6 kernels, introduced in b16fb91 and 1bbd2dd and released in 85653c7. The three routines we identified follow ExLlamaV3's operation order, written differently; that order gives bit identity. So the relationship includes structure, not just the format. Our scan found no copied text, and we defer to your copy scan.

We also checked the other EXL3 kernels: exl3_experts.cu, linear.cu, x3ld/x3gm/x3seg/x3pf.cu, dense3.cu, the pfdense family and lin2.cu. Their includes carry the same decode routines. Beyond those, rotation/finish order and split-K completion follow the same operation order, and parts of the load ring, lane-shuffle decode, linear loop and fused epilogue follow ExLlamaV3's designs. No copied text was found in this expanded check either. Line matching cannot establish originality or detect every renamed/reflowed expression; your scan remains the authority. All third-party notices remain, and the lost antirez/ds4 load-geometry credit in dense3.cu is restored.

The personal MIT headers are removed at their introducing layers. Development Python tools are out of the upstream tree; native tests consume the stored fixtures. The tools are in jayleaton/deepseek-v41-tensorfold-spark#30 under tools/tensorfold-dsv41/, and the family docs link the pinned commit c1cb91840fb8ff5db0bd27418fc7cc766a222cbc.

The compatibility wire and its settings/log plumbing/tests are removed from #663. Family hooks, graceful drain and keepalives remain, with default-wire coverage on the native surface. The HCA correction and executable-mode fixes are included. Each rewritten head passed its host/family/receipt gates, formatting, the zero-new lean comparison against 534 inherited findings, and mode checks. Compiler/SASS receipts are separate from GPU/model qualification; no hardware inference or transport run is claimed for these rewritten heads.

[Bizuayeu]

@jayleaton, a data point for the graph cache, and one request.

Before #635 we tried a family-local graph for GLM-5.3-Flash's verify forward (#548): one capture per key, the key being the window's rows, the KDA chain's parity and the chunk and pool counts the position decides; staging, the draw and the commit stayed eager. On one GB10 it replayed as intended (on one slot, 37 captures and 1,740 replays over the lanes cover check). On two GB10s at TP=2, graphs on and off alternating twice in one sitting:

decode	graphs on	graphs off
greedy, no drafts (three prompts, tok/s)	17.2 / 17.1-17.2 / 17.1-17.2	17.1-17.2 / 17.1-17.2 / 17.1-17.2
MTP drafts, 2 to 4-row verify windows (tok/s)	39.9-40.0 / 26.1-26.2 / 32.0-32.3	40.5-40.6 / 26.4 / 32.4-32.5

Level without drafts, and 0.9 to 1.5% slower with them, so we're not taking ours. Our reading, not yet profiled: decode on GB10 is GPU-bound here (about 1.4% of a step is GPU idle), so there's little launch time to save, while MTP windows change their row count and the position-dependent counts move on, so captures keep coming back. We'll use your cache rather than our own once #635 lands, and we can run the same A/B through it.

The request: since a graph can be slower on some workloads, as here, could the cache be turned off by the operator without a rebuild? An environment variable or a serve flag that sets Settings.on = false for any family that installs the cache would do. Then a family can ship graphs on by default and a deployment that measures otherwise can opt out.

[ashhart]

Thank you, @jayleaton, for turning all three around in a day. The development tools now live in your repository and the fixtures stay here, the personal headers are gone, the compatibility wire is out of #663, and the provenance note is the clear answer we needed. As promised, we'll run our copy scan against ExLlamaV3 over the EXL3 kernels you listed before any of them lands.

Here's how we'd like to go from here.

The shared pieces first, gated by us. We'll take #635, #636, #637, #662 and #663 in that order, each as its own increment, on our two DGX Sparks:

• cuda: a keyed graph cache and a device arena, off until a family installs them; failed launches name their kernel (DeepSeek-V4.1 stack 2/14) #635: capture and replay through the keyed cache and the arena on a real model, with the cache on and off giving the same tokens.
• fabric: tensor parallelism across CUDA processes (bootstrap, NCCL, mailbox and RoCE collectives, plan link, fail-fast) (DeepSeek-V4.1 stack 3/14) #636: the transport on hardware: RoCE and NCCL gathers bit-exact, fail-fast from either rank, and the port pick with both HCA variables unset.
• sessions: session memory for paged-KV families (page pool, prefix trie, RAM residency budget, streamed NVMe tier) (DeepSeek-V4.1 stack 4/14) #637, lanes: a family's round planner drives the lane host with prompt pieces, and a halt ends the rounds at a drain deadline (DeepSeek-V4.1 stack 6/14) #662 and server: graceful drain and pre-token keepalives (DeepSeek-V4.1 stack 7/14) #663: the host suites, plus a served check that every family we ship today gives the same output with and without them.

Each one lands when its gate passes, as your commits under your name.

The family layers need one run at the current heads. As you said, the rewritten heads haven't run on a GPU yet. Before we review the family layers (#667 to #727, about 85,000 lines) in depth, could you post one run of #727 (or #703) on two GB10s with:

• drafted output equal to "draft": false by token SHA, a resumed prompt equal to a fresh one, and concurrent equal to solo;
• cold prompt processing from 2k to 32k tokens and decode tok/s, with peak memory;
• the same checkpoint on vLLM or SGLang beside it.

@ecohash-co's production numbers on #559 are encouraging, but they come from their own build, so a run at these heads is what we can review against.

Shared changes as their own pull requests. #702's increment touches 57 files under zig/src/server/. 49 of them are the one-line change that makes json.zig a named module, which is easy to take on its own. #662 changes the lane host and the lane core. Every model goes through that code, so could the family-neutral parts come as small pull requests of their own, each with its tests, while the DeepSeek text, tool and template code stays inside the family? A regression in shared code is then easy to find and to revert.

Thanks again. This is the most complete community port we've had, and we want to bring it in properly.

[ashhart]

Thanks, @Bizuayeu. That A/B is useful: on two GB10s, GLM-5.3-Flash's verify graphs were level without drafts and 0.9 to 1.5% slower with 2- to 4-row MTP windows.

Yes to the switch. An operator should be able to turn the keyed cache off at serve time, without a rebuild, for any family that installs it. We'll make it part of #635's gate: the same tokens with the cache on and off, and speed measured both ways.

@jayleaton, could you add it in #635, as a serve flag or an environment variable that sets Settings.on = false? Or, if you'd rather, we'll add it in our own commit when #635 lands.

[akol1]

@jayleaton — bringing the family-package question here, as ashhart asked on #690: your stack and the Metal draft both write zig/src/families/deepseek_v41/, and the draft would land a second family if we don't agree the split first.

Concretely: #667 adds zig/src/families/deepseek_v41/ with host configuration, EXL3 pack headers and rank weight plans (config.zig, exl3.zig, deepseek_v41.zig); the #690 draft adds the same directory with its own config.zig, resident weights, the Engram tables, history state, and zig/kernels/metal/deepseek_v41/ operators.

Proposed split, following ashhart's note that config, weights, Engram tables and the host side should be one package with Metal and CUDA kernels beside it:

• zig/src/families/deepseek_v41/ — one backend-neutral package: config, weight/pack loading, the Engram tables and the token map, cache/history state, workspace and round planning. No Python steps; the token map is moving into Zig at pull or load.
• zig/kernels/metal/deepseek_v41/ — the Metal operators (the draft's csa2, mhc, engram, kv, rope, index kernels).
• Your CUDA kernels stay where your layers put them, beside the same package.
• Each backend implements the lane-core Backend; prompt caching, drafts and concurrency come from the core, so drafted equals "draft": false, resumed equals fresh, and concurrent equals solo are checked per backend.

To agree the file-level split, three questions:

1. Loaders: your DeepSeek-V4.1: host configuration, EXL3 packs and rank weight plans #667 reads EXL3 packs with rank weight plans for TP. The Metal draft loads the same EXL3 snapshot single-GPU on a Mac, plus the original shards' Engram tables. Can one loader serve both — pack headers plus a plan that degenerates to rank 0 of 1 holding every named tensor — or do the TP rank plans stay CUDA-only over a thin shared header layer?
2. Which pieces of the Experimental DS4.1 resident Metal backend and qualification (draft-only) #690 family does your stack already have in some form — the Engram token-map reader, kv/kv_state history, csa2_index selection state, the hyperconnection and CSA2 host math? Where you have equivalents, we take yours as canonical and port the Metal callers; the draft's files were written without knowing your layout.
3. Where should served-side family state live: inside the family directory (as both drafts do now), or as a dsv41 module beside exlr8 under zig/src/core/ the way GLM-5.3's does?

The Metal draft itself stays draft-only for now — its whole-model numerical gate is still failed at 4.0961% against the unchanged 1% limit, and the next run is being rebuilt on the lane core. Until you confirm the split, we'll keep our work in worktrees and write nothing further into families/deepseek_v41.

[ashhart]

Thank you, @jayleaton, and thank you, @akol1, for raising the split here.

We've opened #847 to bring DeepSeek-V4.1 together as one family with CUDA and Metal backends. It takes @akol1's layout from this thread: one package in zig/src/families/deepseek_v41/ with no backend code in it, the kernels beside it, each backend on the lane core's Backend, and EXL3 through the one module (#832).

Nothing changes for the shared pieces. #635, #636, #637, #662 and #663 still come first, each through the gate described above. The notes on #636, #637 and #663 are what those need.

@jayleaton, two things would help most:

• Your answers to @akol1's three questions, on DeepSeek-V4.1 on the native engine: one family, CUDA and Metal (tracker) #847.
• Where the CUDA kernels marked "Copy of src/tensorfold/families/deepseek_v41/cuda/..." come from. That path isn't in our Python line, and we want to credit whoever wrote them.

The GPU run at the current heads is still the step before the family layers. Thanks again for all of this work.

[jayleaton]

The family-neutral extractions are now separate draft PRs: JSON module #860 and lane host/core #861. Each is independent on main c7137bb31a7dad924fb541a7068422127b9a94b2, includes its tests, and passed Zig 0.17.0 build, host tests, frozen server goldens, Cortheon fuzzing, CUDA-host and server CPU targets, formatting and the identity-based lean comparison (527 inherited, zero new). Five inherited host skips remain in each invocation. DeepSeek text/tool/template code is not in either PR. Neither receipt qualifies a model or device; the served family-equivalence and two-GB10 gates remain unrun.

[jayleaton]

Thanks, @Bizuayeu. You're right: an off rank returns at the top of run and never votes, so a rank left on waits at its first miss. Updated to 864fe253, a fast-forward from d3355206:

• Settings.effective(value) applies TF_GRAPH_CACHE, clamps min_keep under max (0 with hold, where it's unused), and maps every off cache (0, off, OFF, or a max of 0) to one value. Cache.init runs with it. Settings.digest() gives those bytes for ranks to compare at start, before any cache is built. Hits still make no collective.
• RUNBOOK now says what a mismatch does: a rank left on waits at its first miss, only the off rank logs the switch, and ranks sharing a cache should compare Settings.digest at start.
• graph_cache_ranks_test.zig runs two in-process ranks with fake engines over keys 1,2,1,2,1:
  • Both enabled: they capture together and cast 4 votes each (room and capture for each of the two misses), none on the three hits.
  • 0 on one rank and OFF on the other: one digest, every step eager, no votes on either rank.
  • Off against on: the off rank never votes, and the enabled rank is still waiting after its first vote. The test bounds that wait at 50 ms and expects PeerNeverVoted; in production it has no limit. The two digests differ before either cache exists.

The startup refusal itself is in #636's bootstrap agreement, as you suggested; there's a note there.

Host gate at 864fe253 (Linux x86_64, Zig 0.17.0): build 10/10, test 46/46, test-golden 4/4, fuzz-cortheon 3/3, test-cuda-host 27/27, test-server-cpu 3/3. zig fmt and git diff --check are clean; GitHub desktop, mobile, zig and fmt pass at this head; lean_check shows 527 inherited findings and 0 new, and the Mac-host compile emulation shows 0 Zig errors. The GPU gate (real-model cache on/off token parity and speed) is still not run.

[ashhart]

Thank you, @jayleaton, the effective-settings digest and two-rank missing-vote test make the off/on startup problem explicit, with the capture-fix credit retained.

We checked abcad7d. Capture ownership and keep-floor repairs are present. arena.zig:89-90 still pops a reusable range before live.put can fail; we can restore that ownership sequence and add retry/conservation coverage at landing. The global allocation hook also needs its installation/restoration tied to the real family owner.

We will carry these with the common runtime and #636's actual bootstrap agreement, keeping cache-on/off token parity and rank key/invalidation checks with the first consumer. Your host receipts remain useful; the real capture/replay gate is still open.

Posted by Ash's AI Assistant Synthena

[cwschroeder]

GPU receipt for 3aeebb97 on two ThinkStation PGX (GB10, sm_121, driver CUDA 13.0, aarch64 Linux), Zig 0.17.0, built in nvidia/cuda:13.0.2-devel-ubuntu24.04 (nvcc 13.0, NCCL 2.28.3) with -Dnvcc, main 6ee828c2 for reference.

• zig build test 84/84 steps, 1405/1412 tests (7 skipped); test-cuda-host 39/39 steps, 512/512 (test counts from the first host; every step passes on both). CUDA build 57/57 steps.
• tf-cuda-test on each GB10: smoke 7 groups, graph 3, launch-ex 3, ptx 1 (the broken-PTX refusal included), one-rank nccl exact.
• The independent cache/arena helper from my receipt on cee12a2b compiles unchanged against this head and passes 15/15 device check groups on each GB10: Plain and VMM arenas, byte round-trips, reuse, large releases, borrowed and pre-hook buffers, 1,024 allocation/free cycles per backend; real CudaEngine capture/replay, LRU, fingerprint invalidation, memory-floor recovery, max 0 and the injected capture-error fallback.

Not covered: the real-model capture/replay gate (same tokens with the cache on and off, speed both ways) needs a consumer family, and the cross-rank Settings.digest agreement stays host-tested here. No code changes.

[ashhart]

Thanks @cwschroeder for the 15/15 device groups on each GB10 and the explicit capture-error, reuse and memory-floor controls. Those are useful runtime receipts, separate from a model consumer's qualification.

Our integration also reserves ownership before consuming reusable arena ranges and preserves the allocating hook's lifetime. The real-model cache-on/off token comparison, speed in both modes, rank agreement and invalidation checks remain open on the integrated consumer. The supplied helper and host results should stay pinned to their tested head; they do not establish that final model gate.

Posted by Ash's AI Assistant Synthena

[ashhart]

Thanks @jayleaton. 1.1.1 ships the shared keyed graph cache, family-installed arena, rank agreement and named launch failures. The range-reuse safety in 3aeebb9 is already covered by our maintained insert-before-pop path; adbdb8a's NotReady handling is also present. We'll carry the additional allocation-failure and portable-clock regression coverage into 1.1.2 with your credit. Closing for the shipped infrastructure scope.

Posted by Ash's AI Assistant Synthena