Repository navigation
Living Weights on Qwen3.8-27B (CUDA, GB10): the pocket in every pass, the trainer, --slide, and pocket neurons that no longer drive one another - #911
Draft
ZackSample wants to merge 24 commits into
Conversation
The lane host admits every queued request at once: a lone one still goes to the backend's driver, the rest to the core's addStreams (lanes/admit.zig). It runs the backend's optional prefill_many when there is more than one, draws each stream's first token, then asks for every stream's first drafts in one request, as multi.py's _fill_batch admits a batch. addStream is its one-stream case, with the same calls in the same order; a backend without prefill_many prefills each stream as before.
…ilies share it Source needed Nemotron's kernel handles only for the driver, the stream and whether the card is discrete; it now takes those as a Target (Target.of(ops) for Nemotron's own loader). cuda.zig exports the source module and the torch ops for the 27B's CUDA engine.
…y stream drafted A qwen3_5 checkpoint with the 27B's shapes opens zig/src/families/qwen27/cuda in tensorfold-native, beside the family's Metal engine. It reads the family's config.zig and refuses other shapes (cuda/shape.zig). The forward is the Python 0.6.6 engine's: its CUDA kernels (qmm_group, qmm_prefill at prompt tile 9, the GDN tree, replay and prompt chain, prefill attention) built from zig/kernels/cuda, and its Triton glue from a set captured on the chip (TENSORFOLD_CUDA_KERNELS). Verify rounds take every stream's window together, up to 16 streams of 16 rows, as multi_tree_forward lays them out, and commit only each stream's accepted path. DFlash2 (--drafter, else the pulled z-lab/Qwen3.8-27B-DFlash2) drafts every stream in one batch: each stream's kept taps absorbed together, every block in one forward, a best-first tree per stream, or a copied continuation where the context repeats. The lane core verifies the trees and trims them by each node's chance against a cost curve timed at load. Prompts that arrive together share one prefill pass (prefill_rows). Checked against captures of the Python engine (zig/tests/cuda/qwen27), on an RTX PRO 6000 and on one DGX Spark (GB10): serial replies equal token for token (a 4722-token prompt among them); 54 verify windows (chains, trees, shared rounds of 2 to 16 streams and up to 256 rows, every alignment of the packed inputs) equal row for row in their logits' bytes; drafted replies and each stream's rounds equal Python's; served replies of 1, 4 and 16 concurrent requests, greedy and sampled, equal the same request alone and its "draft": false decode.
…ore admission; GLM-5.3-Flash reads them on Metal Server (engine-independent): - `Info.vision` is an engine's offer: a `prepare` hook (an image's bytes to its tower's patches, grid, tokens and a content hash, decoded by the platform's own decoder), the image token, the wrapper's tokens, and a request's visual-token budget and image limit. `--vision`, `--vision-max-images` and `--vision-image-tokens` reach native engines. - User messages keep their image_url parts (data URLs) beside their text for the template. Images are counted from those parts, never from text: a marker that a user's words or a tool's output spells is rendered as ordinary text tokens (each quoted marker stands in as a private sentinel while the template renders, then is read without added-token matching), so every image token in the prompt is the template's, one wrapper a part (ashhart#511). A prompt that quotes no marker gets exactly the ids it got before. - Each image's placeholder is expanded to its tokens before the window, the chunks and the cache see the prompt; `Request.images` carries the prepared patches and positions, and `Request.cache_key` replaces each image's rows by ids derived from its content, so a kept state of one image never resumes another under the same placeholders. - 0.6.6's limits and refusals: 4 images, 10 MB each and 20 MB a request, 8192 px a side and 16 MP, a single frame, a shared visual-token budget, detail low at 256 tokens; PNG and JPEG (images.py's formats), WebP refused by name until a decoder exists; remote URLs refused. GLM-5.3-Flash on Metal: the image tower (24 pre-norm blocks with 2-D rotary attention, the 2x2 merge and the merger, its projections on gemm_nax), its rows written over the placeholders' embeddings in the prompt pass and one position on for the MTP head; preprocessing as 0.6.6's vision/images.py and mlx-vlm's Glm5NextImageProcessor (EXIF orientation, alpha over white, PIL's 8-bit bicubic resampler). tf-glm-vision compares a reference dump's stages. tools/zig/glm_vision_fixtures.py draws the fixed image set and dumps the Python frontend's tensors; tools/zig/vision_served.py checks a running server.
…0.6.6 did media_fetch.zig follows images_http.py: an https URL on port 443 without credentials, a fragment, a backslash or controls; every address the host resolves to public (Python's is_global less 192.0.0.0/24 and 168.63.129.16; IPv6 global unicast only, no 6to4 or Teredo); TLS verified for the host on the address that was checked; GET with identity encoding; up to 3 redirects, each checked again; the declared media type JPEG, PNG or WebP (WebP is then refused by its bytes, as with data URLs); the bytes within what the request has left of 10 MB an image and 20 MB in all; 10 s a download and 30 s for a request's downloads. std's connect has no timeout yet: a nonblocking connect and poll bound it, the socket's send and receive timeouts bound the rest. The HTTP exchange reads from any reader, so the response rules (redirects joined and rechecked, status, types, lengths, chunking, encodings, the deadline) are tested against canned responses; the URL and address rules are tested directly. Without --vision-urls an https part is refused with the flag's name, other schemes as 0.6.6 refused them; --vision-urls needs --vision. tools/zig/vision_urls_served.py checks a served model against public hosts.
Conflicts resolved toward main's admission (open.zig, chunked prompt passes); the PR's shared prefill batching (admit.zig, addStreams) is dropped, prefill_many stays as an unused optional vtable entry. Living Weights setup in native/cuda.zig is guarded by @hasDecl(F, "slide").
… URLs) onto qwen27-cuda-1.0.7
…its check against 0.6.6 Row cosine against 0.6.6's flash-attention tower on nine fixture images: mean 0.9997-0.99998, min 0.981-0.99998; 0.6.6's own math-vs-flash attention gives min 0.9895 on the same shapes image.
…rt_resize, PIL bicubic, patches PNG decode and patches equal 0.6.6's bit for bit on seven fixtures; JPEGs differ in 1-6% of bytes (stb's IDCT against libjpeg-turbo's).
…he prompt pass, M-RoPE positions - Media (vision.zig): the request's images encoded, each prompt row's feature row, and get_rope_index's three positions; rope_delta kept on the stream for decode and verify positions (the drafter's stay plain, as 0.6.6's do) - Forward.chunk: image rows take the tower's rows; a chunk holding them runs attn_prep with three position rows (qwen_vision.cu), others the captured kernel at their shifted 1-D positions - native/cuda.zig: --vision for families that serve it (Info.vision from the family's prepareImage and markers), the tower's call scratch kept out of the streams' room; other CUDA families refuse --vision - JPEGs through libturbojpeg when it loads (PIL's decoder): patches equal 0.6.6's on every fixture, PNG and JPEG - aot: a launch with no exact captured variant takes one that assumes less of its arguments; the 27B's attention plan segments start on 16-byte boundaries, as the Python engine's tensors do Checked on an RTX 5090 against 0.6.6: prompt token counts equal on five image requests, three replies word for word, text exactness unchanged (serial 4/4, windows 54/54, drafted 4/4, shared rounds 6/6), concurrent text replies equal solo beside an image request.
…itted streams --kv-cache fp8 keeps the 27B's attention caches as e4m3 rows with one fp32 scale a (position, key head) row (amax / 448), half the bytes of bf16: 256K tokens with the image tower in one stream on a 32 GB card. bf16 stays the default and its bits are unchanged (0.6.6 exactness: serial 4/4, windows 54/54, drafted 4/4, shared 0 replies differ). - tf_kv_quant (qwen_ops.cu) quantizes commits' and prompts' rows into the caches - pattn8_kernel: prefill attention staging e4m3 rows widened to bf16, a step ahead in registers (within 5% of bf16 at 100K) - _shared8/_tail8 (tests/cuda/qwen27/attention_kv8.py, captured like the other glue): the decode kernels over e4m3; _tail8 quantizes the window's own rows as their commit will, so a key's bits never depend on windows, drafts or batching (drafted equals serial, concurrent equals solo); attention.py's _merge folds the partials unchanged - attention partials leave the scratch arena: sized after admission for the streams admitted (Engine.fit, native fit hook) and charged to each stream, not for 16 streams at full context (1.4 GiB back at 122,880, 2.9 GiB at 262,144) - tf-qwen27-cuda kvcompare: fp8 against bf16 teacher-forced, top-1 agreement and KL
…tate The 27B serves the shared prompt cache (core/prompt_cache.zig) as Nemotron does, its copies in pinned host memory: a long window's caches leave the device no room for them. - reuse.zig: a kept state is a stream's DeltaNet recurrences and conv windows whole and its attention rows (and fp8 scales) through the mark, about 275 MiB at 4,096 tokens under fp8 - with a cache, prompts cut the zero-anchored 4,096-row grid (engine.Chunks), so a pass resumed or kept at a grid point computes what a fresh pass of the same prompt does; without one (--prompt-cache-gib 0, and tf-qwen27-cuda) the even splits stay, bit-exact against 0.6.6 - prefillFrom (engine, lone): a restored stream's pass from its grid point, stopping at the lane host's marks; a resumed drafting stream's context holds only the rows it read (drafts change no token) - image prompts neither resume nor keep (their rows' positions are the tower's) - native/cuda.zig: each family's snapshot vtable, and a host-memory budget (8 GiB or --prompt-cache-gib, at most half of MemAvailable) for a family whose copies live there - zig fmt over native/cuda.zig and qwen27/cuda/triton.zig (the fmt check) The qwen suite's soak check: 60/60 replies equal to the same request alone and 60/60 turns resumed (0/60 resumed before), in 117 s against 244 s; drafts, concurrent and behaviour unchanged.
Conflicts resolved toward both sides: - build/cuda.zig: nativeServer takes main's core module and the branch's qwen27 - engine_api, lanes/stream, lane_host: main's untaught and prompt_logprobs beside the branch's images and cache key; the lookup skips untaught and prompt_logprobs requests (main) and matches by cacheKey (branch) - native/cuda.zig: Nemotron's store as main makes it (keeps at any row, snap_ready's charge); the 27B's on its grid in host memory; prompt_logprobs and vision both in Info - glm_host: main's serves window with the branch's vision; growSnap and prepareImage both kept - metal.zig: the fp8 refusal before main's refusal(), the vision refusal before glm_moe_dsa - lane_host_reuse_test: the event switch handles prompt_logprobs zig build test 274/274; zig fmt clean; tf-qwen27-cuda against 0.6.6 unchanged (serial 4/4, windows 54/54, drafted 4/4, shared rounds 0 replies differ).
attention_kv8.py (which records the fp8 attention launches) and qwen_vision_fixtures.py (which records the vision fixtures) import tensorfold, so they move out of the tree; their recordings stay fixture files, made once outside it.
Conflicts resolved toward both sides: - build/cuda.zig: the CUDA test list keeps qwen27 beside main's tokenizer tests - lane_host: main's spans helper and its warm-pass lookahead rule - tokenizer: the branch's encodePlain beside main's memoized encodeSegment - cuda/kernels.zig: the branch's torch_norm_rotary beside main's norms and embed_rows - server/cli.zig: --cortheon beside --vision and --kv-cache - native/cuda.zig: main's shared lane binder (cuda_bound.zig); the startup line takes main's "passes of up to N rows" with the branch's fp8 note and host-memory cache; Nemotron's cache rules take main's .warm = .waits; warm turns stay off for the 27B, whose cache keeps states on its prompt grid Follow-ups the merge needs: - cuda_bound.zig forwards prefill_many, which the 27B's backend gives - qwen27's LoneRun takes main's resumed argument (the 27B has no lone driver) - qwen27 declares takes_drafter: main's new --drafter refusal would otherwise refuse its DFlash2 drafter Checked on an RTX 5090: zig build test 48/48; bf16 against 0.6.6 serial 4/4, windows 54/54, drafted 4/4, shared rounds no reply differs; fp8 serial, drafted and shared rounds unchanged; the server with the prompt cache on matches 0.6.6 in 20/20 (12 resumed); images, prompt-cache resumes and decode speed unchanged.
- aot.zig takes main's version: ashhart#794 already holds this branch's general pass, and also refuses a launch of 1 a runtime variant - lane_host.zig: main's born lengths and host spans, with this branch's cacheKey so an image prompt is still matched by its images' content - glm.zig: main's affine and wide modules beside vision and image
- docs/api.md: main's note that the Python 0.6.6 engine serves --vision, with the native server's image input for Qwen3.8-27B on CUDA and GLM-5.3-Flash on Metal; other native families refuse --vision
…emption Main's new lean check (94ab8f8) found 666 problems this branch adds: - 548 in zig/vendor/stb/stb_image.h, the third-party decoder kept as its authors wrote it. lean_check.py now skips zig/vendor/, as it skips generator outputs. - 118 in our own code. Each multi-line comment or docstring is now one line of at most 120 columns. Only comment and docstring text changes. - The vision fixture check moves from cli/qwen27_cuda.zig (635 lines) to cli/qwen27_vision_check.zig. The lean check against main now reports 0 new problems.
…unchanged The learner, its whole-lesson and routed paths, the pocket and its fits and helpers move from families/nemotron to a `living` module that both builds import, as 19 renames with no change to their contents. The family-neutral half of learn_shape.zig (block sizes, Adam settings, direction lengths) moves with them; Nemotron's learn_shape.zig keeps where its blocks sit and names the shared constants from the module. Nemotron's files import the moved ones through the module. The learner's tests now also run on Linux, as `zig build test-living` and under test-cuda-host. No behaviour change.
… layout, neuron kind and shard format A family now describes itself once, as data (living/families.zig): where its shared experts sit and their tensor names, the neuron's activation (relu^2, or SwiGLU with an optional per-row gate), the quantization, the config keys and the object they live in, and its MTP head's expert. Its host gives the learner a layout and the pocket hooks (living/family.zig). Nemotron is the one family listed, and its lessons, routed blocks, edits and probes are unchanged. - The pocket reads its layers and width from the family's layout, not from Nemotron's config, and the learner offers routed blocks, edits and probes only to a trainer with `sites` (Nemotron's). - living/shards.zig widens any family's experts at its group and width; Nemotron's writer is now a wrapper over it. shard_edit sets and reads nested config keys (text_config). - A SwiGLU neuron fitted as u is written as gate row 16u and up row u/16: relu(u x)^2 on its fact's rows, at most 8e-5 of it at a 0.5 margin; a gated family's outputs fit gate * act(x). No listed family uses it yet; the tests use a stand-in family shaped as Qwen3.8 Flash Next's shared expert. - The server's view reads layers, pockets and shards through the family its config's model_type names, and native/nemotron_slide.zig is native/slide_adapter.zig. - pocket.zig's quiet refit and its loading move to pocket_quiet.zig and pocket_load.zig, for the 600-line rule. - Tests: the generic writer equals affine4_host at 4 bits in groups of 64, Nemotron's save test over the wrapper, a SwiGLU neuron fitted and stored at 4 bits in groups of 32, and a whole pocket lesson over a stand-in host for each activation (test-living, also under test-cuda-host).
…shhart#726 (Qwen3.8-27B on CUDA, 1.0.11) The stacking merge this branch needs: the 27B's Living Weights hosts the learner in zig/src/living from ashhart#878 on the qwen27 CUDA family from ashhart#608/ashhart#726, and neither contains the other. Conflicts: zig/build/cuda.zig (the qwen27 and living modules both kept, qwen27 imports living) and native/cuda.zig (the slide adapter of ashhart#878 behind ashhart#726's @hasDecl check).
On ashhart#878's learner: the refit and recheck fixes (ashhart#870's two PRs), facts learned and checked in conversation (the context PR), checks scored through serving's own rounds instead of decoded again (slide_known.zig, the learner's score request), the writer's asks side by side, and the drift fix: pocket neurons kept from driving one another (pocket_decouple.zig; the values backward holds the pocket's activations through Trainer.pocket_back), 92 broad keep prompts, and 12 drift checks a round may not change where the open lesson's neurons fire (the learner's quiet request). families.zig lists qwen3_5 (and qwen4_exp, ashhart#896's). pocket.zig, learner.zig and the server's big files are split to stay under 600 lines.
…ernels gdn_train.cu and cuda/gdn_train.zig: a Gated DeltaNet layer's forward for training and its input gradient (out projection, gated RMS norm, the delta-rule chain from checkpoints, q and k norms, gates, conv, in projection), the same bits as serving's chain (--fmad=false). attn_train.cu: gated attention's backward (prep, gate, softmax, RoPE) for any head count. tf-cuda-test gdn-back (synthetic) and gdn-back-27b (hidden 5120, group-64 projections) against float64. Ported from the Flash Next learner (ashhart#895), where these first ran; nothing here names a family.
…the trainer, its backward kernels and --slide - pocket.zig: fp32 SwiGLU neurons a layer, added after each MLP's down projection in every pass (prompt chunks, drafted windows, decode rounds), row by row so drafted equals "draft": false; untaught streams (the lesson writer's judge) read no pocket. A taught folder's widened MLP keeps its own 17408 rows tiled, the pocket's in fp32. - train.zig, train_more.zig, train_load.zig, train_ops.zig, qwen27_train.cu: the learner's passes (keys, loss, target) with 16 sequences side by side on the engine's own batched pass; the loss's gradient from the group-64 head down through the dense SwiGLU MLPs and pocket, gated attention and the DeltaNet (core gdn_train), on MLX copies of layers 12-63; wide 4-bit products (tf_q27_gemm_q4w) and the backward only on rows a nudge needs (same bits); input-weight fits on the GPU with the host's bits; greedy scoring through serving's own decode rounds. - tap.zig, hooks.zig: a prompt in a conversation read through the engine's prompt pass (the context fix's tapRows). - slide.zig, native.zig: --slide for qwen3_5, the pocket written through the core's shard writer. - tf-qwen27-learn: packed passes == alone, the push against finite differences, Q27_GATE (gate taps, with activations) and Q27_SCORE (replies teacher-forced through serial decode); tf-cuda-test q27-back and q27-gemm.
Owner
|
Thank you, @ZackSample, for the 27B trainer, shared backward kernels and rounding the pocket before its held-out checks. We checked 018d2b2. Its large diff includes #726 and #878; the new work is the learner revision, shared backward kernels and 27B port. Please move those three increments onto the current Qwen CUDA and shared learner/image base, using repository-built kernels and independent reference tools. At We'd run trainer/serving and gradient checks, sequential retained facts, unrelated prompts, quantized restart, deletion and rollback, plus drafted/plain, resumed/fresh and concurrent/solo replies with images too. Posted by Ash's AI Assistant Synthena |
Open
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…me lesson gate (#911) The dense Qwen family on CUDA takes --slide for the 27B's affine 4-bit layout; mixed widths, routed experts and an fp8 KV cache are refused for learning. Its lessons go through the shared learner and the gate Flash Next and Nemotron use, saved pockets are kept apart from the original projections with the landed staging and packing helpers, and a row's scales are published only after their copy succeeds, with taught and untaught streams kept as they were. The learning forward keeps the serving launches and their argument sequence exactly, and the shared GDN adapter adds the dense product and a bounded backward span (zero for Flash Next). The trainer's constructor frees what it registered before its list on a failed allocation, drains or exits before releasing an owner, and native close drains the dense trainer as it does Flash Next's. The 34 dense learning launches stay unqualified until their records are measured; nine tower records keep their qualification and two unused media launches go with their wrappers. Co-authored-by: ZackSample <134018356+ZackSample@users.noreply.github.com> Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com>
ashhart
added a commit
that referenced
this pull request
Oct 18, 2026
…896, #911), which move to 1.1.1 This reverts a291b68de and 4e0d7927c with the follow-ups that only served them: ca7a0882d (Nemotron's learning drain), 493cdf5d4 (the save marker), 650fac5d7 (the MoE backward pack), the learning records in cda734f7d, ec141ed97 and d9566f585, and the trainer tests that 2708455af moved into cuda_close_tests.zig. The training images leave the kernel list and the CUDA contract's image list, gradient_pending, qwen_learning_pending and registry_learning go, and the 13.2 set and the live audit keep the serving, prompt, GLM and paging records. The lean moves that later commits build on stay: the kernel list remains in cuda_kernels.zig and the close-test file goes only because every test in it was a trainer's.
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…me lesson gate (ashhart#911) The dense Qwen family on CUDA takes --slide for the 27B's affine 4-bit layout; mixed widths, routed experts and an fp8 KV cache are refused for learning. Its lessons go through the shared learner and the gate Flash Next and Nemotron use, saved pockets are kept apart from the original projections with the landed staging and packing helpers, and a row's scales are published only after their copy succeeds, with taught and untaught streams kept as they were. The learning forward keeps the serving launches and their argument sequence exactly, and the shared GDN adapter adds the dense product and a bounded backward span (zero for Flash Next). The trainer's constructor frees what it registered before its list on a failed allocation, drains or exits before releasing an owner, and native close drains the dense trainer as it does Flash Next's. The 34 dense learning launches stay unqualified until their records are measured; nine tower records keep their qualification and two unused media launches go with their wrappers. Co-authored-by: ZackSample <134018356+ZackSample@users.noreply.github.com> Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com>
BlivionIaG
pushed a commit
to BlivionIaG/TensorFold
that referenced
this pull request
Oct 20, 2026
…rt#895, ashhart#896, ashhart#911), which move to 1.1.1 This reverts a291b68de and 4e0d7927c with the follow-ups that only served them: ca7a0882d (Nemotron's learning drain), 493cdf5d4 (the save marker), 650fac5d7 (the MoE backward pack), the learning records in cda734f7d, ec141ed97 and d9566f585, and the trainer tests that 2708455af moved into cuda_close_tests.zig. The training images leave the kernel list and the CUDA contract's image list, gradient_pending, qwen_learning_pending and registry_learning go, and the 13.2 set and the live audit keep the serving, prompt, GLM and paging records. The lean moves that later commits build on stay: the kernel list remains in cuda_kernels.zig and the close-test file goes only because every test in it was a trainer's.
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
This sits on two open PRs, so it opens as a draft until they land:
cache and a prompt cache). Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608 alone sits on 1.0.5, while the learner below sits on
main; Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) #726 is Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608 on a basethe learner merges into (and the base this port was built and measured on), so this stacks on Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) #726, not Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608.
Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608's commit (99bcc5b) keeps philip's authorship; nothing of it is rewritten.
lw-b-core75978a1).The branch is
pr726(bcd433c) plus one merge commit oflw-b-core(dac84b6), then 3 commits. That merge is the onlyone: the 27B's trainer needs #608/#726's qwen27 CUDA family and the learner in
zig/src/livingfrom #878, and neithercontains the other. Its conflicts were
zig/build/cuda.zig(the qwen27 and living modules both kept, qwen27 importsliving) and
native/cuda.zig(#878's slide adapter behind #726's@hasDecl(F, "slide")). Once #726 and #878 land Iwill rebase onto
mainand the merge goes away; the diff to review isdac84b63..lw-27b-pr.The first of the 3 commits carries learner work that is also open as its own PRs (the refit and recheck fixes for #870,
facts learned in conversation), as this port runs it; it shrinks to the 27B-specific parts when those land.
What this changes
zig/src/livingand the server's lesson code:(
slide_known.zig, the learner'sscorerequest), and the lesson writer's asks run side by side;pocket_decouple.zig,Trainer.pocket_back, broad keep prompts and drift checks;families.ziglistsqwen3_5(a dense SwiGLU MLP at every layer, ungated, 4-bit groups of 64);pocket.zig,learner.zigand the server's big files split to stay under 600 lines.(
gdn_train.cu,cuda/gdn_train.zig; the same bits as serving's chain at--fmad=false) and gated attention'sbackward (
attn_train.cu), family-neutral.tf-cuda-test gdn-back,gdn-back-27b.qwen27/cuda/pocket.zig: fp32 SwiGLU pocket neurons beside every MLP, added after the down projection in everypass (prompt chunks, drafted windows, decode rounds), row by row, so drafted equals
"draft": false; an untaughtstream (the lesson writer's judge) reads no pocket. A taught folder's widened MLP keeps its own 17408 rows tiled.
train.zig,train_more.zig,train_load.zig,train_ops.zig,qwen27_train.cu): keys, loss andtarget passes with 16 sequences side by side on the engine's own batched pass; the loss gradient from the
group-64 head through the dense MLPs and pocket, attention and the DeltaNet on MLX copies of layers 12-63
(18.9 GiB); wide 4-bit products and a backward over only the rows a nudge needs (same bits); pocket input fits on
the GPU with the host's bits; greedy scoring through decode rounds (a prompt pass over prompt + reply is not
decode's arithmetic on this model: 0/576 rows bit-equal, 4/576 argmaxes differ at near ties).
tap.zig: a prompt in a conversation read through the engine's prompt pass (the context fix'stapRows).--slideforqwen3_5(slide.zig,native.zig); the pocket written through the core's shard writer.tf-qwen27-learn: packed passes against one at a time, the push against finite differences,Q27_GATE(gatetaps with activations),
Q27_SCORE(replies teacher-forced through serial decode);tf-cuda-test q27-back,q27-gemm.The drift fix
On a 21-fact scale test (below) the taught model's unrelated replies drifted as facts accumulated, and some grew
garbage tokens (
$x->$xdera,Waves->WavesToggleButton). Gate taps on every unrelated reply row found why:zero, at most +0.56). With the pocket live: 113 of 160 neurons reach above 0.25 somewhere, up to u.x = +16.5 (the fit
targets about 1). At each garbage row about 100 of the 160 fire at once, layer by layer: one fact's neuron leaks
+0.43 on a digit in LaTeX, its own next neurons read +0.8, +5.3, +16.2, then other facts' neurons follow.
neurons a median +0.94 and no other fact's neuron above 1 (0/6,400); live, 662/6,400 other facts' neurons read above
1 (one probe: 77 of them, up to +30). The lesson logs show it growing: earlier facts' neurons firing on a new fact's
questions read at most 3-7 by lessons 6-8 and 10-33 from lesson 13, and their refits fail. This is why later facts
got harder to learn, not a neuron budget (at most 9 of a layer's 64 neurons were used).
(s * u) . v. The values fit backpropagated through the higher neurons' activations ((u x)^2, unbounded) and learned
outputs that drive them; later lessons stacked on that.
Fix: every output is kept off what the neurons above its layer read (s * u, s from the trainer's optional
readScale),every input and refit off what the neurons below write there (s * v), and the values backward holds the pocket's
activations (
pocket_back). Projecting a finished folder's outputs this way removes the cascade at once (other facts'neurons above 1: 662 -> 25 of 6,400) but loses recall, since its outputs were fitted with the chains; fitted under the
constraint from the first lesson, recall holds (receipt). Two cheaper guards come with it: 92 broad keep prompts
(other languages and scripts, math and LaTeX, code, verse, lists) read as rows a fact stays silent on, and 12 drift
checks whose replies a round may not change where the open lesson's own neurons read above zero (a reply tipped at a
near tie by the neurons' tails below zero is logged, not held against the lesson). None of these prompts is in the
51-prompt drift set the receipt measures with.
TF_SLIDE_POCKET_COUPLE=on,TF_SLIDE_BROAD=offandTF_SLIDE_DRIFT=offturn each off.Receipt
checkpoint
TensorFold/Qwen3.8-27B-MLX-4bitrevision 22d8d538 (its weights are Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608's 70ae7fac), drafterz-lab/Qwen3.8-27B-DFlash250307d4c. Servers:--parallel 16 --context 8192 --no-thinking --temperature 0,drafts on. All replies greedy.
zig build -Dnvcc=... -Dsm=121 -Doptimize=ReleaseFast install native; host tests (test-living,test-server-cpu,test-cuda-host) pass;zig fmt --checkclean;lean_check527 problems, 527 on the base(dac84b6), 0 new.
#608 on this GB10 (before any of this)
#608's own build against Python 0.6.6 (its environment): serial 4/4 replies match (one 4,722-token prompt), 54 window
cases with no row differing, drafted 4/4 match, multi-stream 6/6. Served at 1/4/16 streams, every reply equals alone
and
"draft": false(12/12 cells); code T0 70.3 / 202.3 / 380.9 tok/s, cold prefill 2k 1.47 s ... 64k 58.13 s, allwithin a few % of #608's own receipts. The merged learner build (#608 + #726 + the learner) passes the same checks.
Kernels
tf-cuda-test q27-back25/25 within float64 tolerance (MLP + pocket + post norm, attention with 4 kv heads, thegroup-64 head; 64 and 256 rows);
gdn-backandgdn-back-27bpass;q27-gemm181 cases byte-equal to the 64x64products.
tf-qwen27-learn: packed passes equal one at a time exactly (0 difference); the push against finitedifferences is bf16-limited (the residual stream is bf16), 2-36% on direct rows; learning itself is the test.
Lessons and restarts
User facts taught by chat (
/v1/slide/learn,about: user). After three facts: held-out questions 3/3, 3/3 and 2/3;unrelated replies 7/8 unchanged (one word late in one reply); a restart with and without
--slidegives the same replies (17/17); the base shards'sha256 unchanged. A fact that borrowed another's neurons was found and fixed on the way (the judge's untaught stream
read the pocket; now it does not).
Context (8 user facts, one retry each for a miss)
The same probe matrix as the Flash Next context PR: questions asked cold, under a system prompt, after 3 turns of small
talk, after an 8-turn ~4k-token task conversation, chained, and with thinking on. With this branch's learner, all 8
facts kept: cold 16/18, system 16/18, small talk 17/18, after the ~4k task chat 17/18, chained 16/16, thinking cold
8/8; unrelated 10/11 replies identical to the untaught model. Restarts plain and with
--slide: 32/32 probes and11/11 unrelated identical; a take-out puts the pocket back bit for bit.
Speed (same lessons, same bits)
Two facts taught on a fresh copy before and after the speed work: the same lesson events, graph, conversations, log
lines and shard sha256. Fitting 821.1 -> 429.9 s and 315.5 -> 173.1 s; a lesson 1,551 -> 683 s and 609 -> 260 s
(the first build ran
--parallel 1). A target pass over 16 sequences / 984 rows 3.88 -> 1.68 s. A later fact thatneeds repairs: fitting 1,071.5 -> 525.9 s. The 8 context facts: 6,732 -> 3,936 s for all lessons.
Scale: 21 facts, before and after the drift fix
The same 21 facts about a fictional person (a name, a spouse, a pet, two towns, a job, a birthday, a car, numbers, a
favourite food and game...), taught in order on a fresh copy, one retry for each miss, then the probe matrix (two
probes a fact) and the 51-prompt drift set (the 11 unrelated prompts plus 40 more: math with LaTeX, code, verse, lists,
five other languages, JSON and tables; garbage = a reply token the untaught model gives logprob below -9 after the same
prefix, scored through serial decode). "Before" is the learner without the drift fix (lw-27b 72103fc2's build).
One fact (the spouse's name) missed after its retry in the run after; it was marginal before too (kept on its round
3-4 confirmations only).
This branch's build
The pushed branch is the code above ported onto
dac84b63(no Flash Next files; the DeltaNet kernels renamed to core).On the GPU with its build:
q27-back25/25,gdn-backandgdn-back-27bpass,q27-gemm181/181 byte-equal;tf-qwen27-learngives the same losses-and-pushes hash as the build the receipts came from (d61e74b8e32ea396, targetpass 1.43 s); untaught, all 51 drift-set replies are identical to that build's. The receipts' servers used #608's
captured kernel set (
TENSORFOLD_CUDA_KERNELS).Known limits
through the model's own layers (seen as other facts' neurons rising from -0.5 to +1.5 behind a leak), so a leaking
neuron on a row no lesson read can still change a reply. The broad keep prompts and drift checks narrow that.
binds.
Draft while #608/#726 and the learner PRs it repeats (#876, #877, #878, #891) land; I'll rebase it onto them then. Some long comments were cut to one line for the lean check and will get a read-through before it leaves draft.