Repository navigation
GLM-5.3-Flash on two CUDA ranks: video input - #976
BobClawblaw wants to merge 46 commits into
Conversation
….cu) and the ranks' control link (tp_link.zig): small gathers as RDMA writes over both halves of the port, after b12x's exchange; NCCL carries the rest
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
…glm_cuda), its kernels (zig/kernels/cuda/glm: the dense matmul at 4/6/8/16 bits, the EXL3 trellis experts with the fused prompt kernel, latent attention, the indexer, KDA, the hyper-connections, the vision tower's helpers, DFlash2), tf-cuda-test glm-batch/glm-prof/glm-tf, -Dglm (cherry picked from commit 611959f15964cda97ecd12d0e459ea4ac0e84f78)
…h), no include across families
…prefill, sample) and no Flash Next test commands
…meration, kernel parameter info, virtual memory)
…ardown hung in NCCL's service threads under the library's unload, reported on PR 855); the startup line says a family keeps its own prompt states instead of 'no prompt cache'
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…ders in docs/glm-cuda-notes.md Splits move code unchanged (only imports, aliases and pub change): kern.zig into kern_attn, kern_kda and kern_moe; weights.zig into ckpt, experts_load and qpack; dflash.zig into dflash_load and dflash_select; native.zig into native_draft, native_prompt, native_follow and native_env; exl3.cu's transposed prompt design into exl3_transposed.cuh; tf-cuda-test glm-tf and glm-prof into glm_forward_run.zig.
…cs/roce-gather.md, zig fmt
…rozen words (the errors/content-image golden)
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
…; fits tests the smaller on both), its host test in the cuda tests
… reads a gather (sampled draws, the full row, DFlash2), the ranks' settings agreed before the first collective, an image frame bound to its prefill by request id, a kept state allocated before rank 1 hears the keep
…eed only at its last unpin; a job reports resumed before it unpins
# Conflicts: # zig/src/cuda/root.zig
…CCL communicator (no TCP link: NCCL_COMM_ID and tp_watch do that now), Group.openRoce brings it up on two ranks, allGather takes it by size alone, Group.check after a sync that reads a gather; tf-cuda-test roce-gather (every size against NCCL's bytes, graph replays, time a call)
…) into pr/glm-flash-cuda: the glm_cuda family leaves; DFlash2, history-copy drafts, the vision tower and the EXL3 routed experts move into families/glm commit by commit, its kept states are main's
…he main stream's all-gathers (each layer's partials, the MTP head's, the greedy pick padded to 16 bytes) take its RoCE writes, side-stream prompt exchanges stay on NCCL; Group.check after every draw that read gathered rows
…lis experts, the rest bf16): admitted beside ModelOpt NVFP4, loaded by the rank's 128-column cut, run by glm_exl3 (grouped decode, fused prompt) on an 8-slot plan into the MoE block's fp32 slot rows; bf16 dense layers through b16; the SwiGLU limits keep a NaN; tf-cuda-test glm-exl3 against float64 from core's EXL3 decoder
…/121 usage (tf-cuda-test kernel-contract-limits)
… dies stops the others instead of leaving them waiting, a failed follower leaves with _exit; every peer's send/recv and NCCL's rings connect before the weights load, so a full GB10 never registers them (ibv_reg_mr refused the first prompt pass's split exchange)
…ore's copy proposals reach 7 tokens; head chains stay 3 deep and a first copy as wide as a head round (EXL3 on two GB10s: a rewrite-this-code reply 34.2 to 41.1 tok/s, others unchanged)
…glm_vision's steps, the processor's patches through image_core over core's decoder (vision_config and image_core split from the Metal files, which re-export them); tf-cuda-test glm-vision: patches as vision_ref.py's within 4e-7, features 1.003x the torch tower's own bf16 distance from float32
…ompt's images, an image step sends their spans and rows to the other ranks, every rank splices them over the placeholders after the embedding (the MTP head's one place on) on that sequence only; the ranks agree on --vision before loading; the server's CUDA image offer takes any family's tower (a template that wraps images itself gets no server wrap)
…reads (float32, and --bf16 for its bound), from processor_config.json
…: weights tiled in place at load, decode windows in K slices with dependent launches, prompts in one launch of 128-row blocks sharing weight tiles in L2 (TF_GLM_MM=0 keeps the glue's b16); TF_GLM_PROF=1 times a verify's sections; tf-cuda-test glm-mm-bench (EXL3 on two GB10s: a 2-row round 67.4 to 63.3 ms, one stream 4.5-9% faster)
…rward, draw, the MTP head's pass, commit), by size
…ilies/glm on main's transport; tree unchanged
…ffered through the tower); needs libtfvideo
|
Thank you, @BobClawblaw, reusing the shared video planner and GLM tower keeps this on the one CUDA family. We checked 999ae3b. cuda_images.zig:70-73 reserves the maximum of independently advertised image/video budgets, not their sum; cuda_follow.zig:164-171 caps spans at 64 although 256 frames can make 128 pairs. Startup agreement also carries image tokens without the buffer-defining video limit. We can repair those capacity/protocol checks at landing after #852/#855, with mixed-media, 128-pair and mismatched-rank regressions. Please keep video drafted/plain, resumed/fresh and concurrent/solo cases with per-rank memory on that combined candidate. The existing media examples and text concurrency receipt do not exercise every one of those boundaries. Posted by Ash's AI Assistant Synthena |
… images and video, with #855 beneath it The merged commits keep their authors' names, dates and messages with two changes: a login-only noreply address becomes its author's id+login noreply, and no tree carries the stb_image decoder an early commit vendored and a later one removed. The #853 commits beneath them are the ones the #853 merge landed without that decoder. Both sides keep their kernel lists, launch records, test commands and GLM forward fields. The GLM host keeps this tree's open and openRank split, with #976's image tower inside openRank and its _exit for a failed follower. The CUDA family options take #976's video limits in cuda_options.zig, and the vision offer keeps Qwen's case and the wrapper rule beside #976's video offer.
… for all its picks, with full-K accumulation The common EXL3 expert module now runs prompt passes for verified uniform 4-bit MCG layers with contiguous gate and up projections, model widths divisible by 256 and expert widths by 128; other books, widths and layouts keep the path they had. When gate and up share their input scales, one fp32 Hadamard rotation per input row, rounded to fp16, serves every pick of that row; otherwise each pick keeps its own rotations. Transposed gate/up and down kernels accumulate the whole K in fp32 over bf16 activations, with no split-K partials, so GLM's prompt rows run on the common module with no family kernels of their own. Decode keeps its split-K launches. The three new launches stay pending in the kernel contract until their records are measured. The design follows #976's shared-row rotations and fused prompt path, and the transposed prompt kernels it credits. Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com> Co-authored-by: MiaAI-Lab <83042094+MiaAI-Lab@users.noreply.github.com>
|
Thank you, @BobClawblaw, for GLM-5.3-Flash on two CUDA ranks and its media support. TensorFold 1.1.0 carries #976 with #855 through On two GB10s with EXL3, drafted equals plain 17/17, concurrent replies at two and four streams equal their runs alone, and cached chats equal cache-off chats 12/12. A B B A matched the previous build's throughput and tokens. The ring check passed at 12 streams and 13,312 tokens; 16 streams was refused by the memory budget. The ModelOpt NVFP4 pair regression remains unrun. Two NVIDIA GB10s: EXL3 text 17/17, concurrency 2/4 and chat 12/12; NVFP4 serving was budget-refused, and full media-matrix validation is unrecorded. Posted by Ash's AI Assistant Synthena |
|
Shipped in 1.1.0 as Posted by Ash's AI Assistant Synthena |
|
@ashhart @BobClawblaw results for the 1.1.0 tag ( Setup. 2× DGX Spark (GB10), one direct ConnectX-7 cable with two HCAs. Kernel 7.0.0-1019-nvidia, driver 580.178.04, CUDA 13.0, NCCL 2.30.7 in the container. Built with Server,
Within 1.1.0, the token and output hashes matched across the three modes and repetitions. They differ from With DFlash2 and Cold-prompt TTFT with MTP,
At 32k that is 31.1 s against 37.0 s on Images,
These ran on a separate start with the template override described below. All three answers were right. Sending the three pictures and a text request at once gave the same replies as alone (4/4 by content hash), and asking each picture again did too (3/3). An invalid data URL and a 20,000 × 30 picture were both refused with 400 and their reasons. The same request retried afterwards matched its earlier reply. Finding: this checkpoint's chat template has no image path. With the checkpoint's own
The checkpoint's template could gain the media macros too, but that is for its maintainers. Not tested: video, sampled decoding, contexts above 40,960, and NVFP4. The chat level 4 aggregate was 14.4% below |
|
Thanks @SvangenStudios for the GB10 run and the template finding. Both suggestions are in #1076. With On our pair, with the checkpoint's own template, the warning appears and the image request returns the new message while text requests still work. With |
…xperts, images and video, with ashhart#855 beneath it The merged commits keep their authors' names, dates and messages with two changes: a login-only noreply address becomes its author's id+login noreply, and no tree carries the stb_image decoder an early commit vendored and a later one removed. The ashhart#853 commits beneath them are the ones the ashhart#853 merge landed without that decoder. Both sides keep their kernel lists, launch records, test commands and GLM forward fields. The GLM host keeps this tree's open and openRank split, with ashhart#976's image tower inside openRank and its _exit for a failed follower. The CUDA family options take ashhart#976's video limits in cuda_options.zig, and the vision offer keeps Qwen's case and the wrapper rule beside ashhart#976's video offer.
… for all its picks, with full-K accumulation The common EXL3 expert module now runs prompt passes for verified uniform 4-bit MCG layers with contiguous gate and up projections, model widths divisible by 256 and expert widths by 128; other books, widths and layouts keep the path they had. When gate and up share their input scales, one fp32 Hadamard rotation per input row, rounded to fp16, serves every pick of that row; otherwise each pick keeps its own rotations. Transposed gate/up and down kernels accumulate the whole K in fp32 over bf16 activations, with no split-K partials, so GLM's prompt rows run on the common module with no family kernels of their own. Decode keeps its split-K launches. The three new launches stay pending in the kernel contract until their records are measured. The design follows ashhart#976's shared-row rotations and fused prompt path, and the transposed prompt kernels it credits. Co-authored-by: BobClawblaw <288991054+BobClawblaw@users.noreply.github.com> Co-authored-by: MiaAI-Lab <83042094+MiaAI-Lab@users.noreply.github.com>
…h --vision, a clear request error) With --vision, a template that turns media parts into text (GLM-5.3-Flash EXL3's, reported on #976) made every image request fail with "the chat template wrote 0 image placeholders for 1 image parts", which reads as an engine failure. At startup the server now renders one user turn holding an image part, as a request's is rendered, and warns when no image placeholder comes out, naming --chat-template. A request whose template writes no placeholder at all gets a message saying so; a miscount keeps its counts.
GLM-5.3-Flash on two CUDA ranks: video input
On top of #855 (the branch contains it). The CUDA family now takes
video_urlparts, the same way the Metal family does:families/glm/video_core.zig: main's video planner (Glm5NextVideoProcessor's sampling,smart_resizecanvas, frame-pair patches) asFor(picture), so the Metal build (relativecore/picture) and the CUDA build (thecoremodule's) share one copy.video.zigre-exports it for Metal, and its tests stay where they were.cuda_images.zig:Images.videoandprepareVideo. Each frame pair goes through the tower as an image, and the rows buffer holds a request's images or a video's (--vision-video-tokens), whichever is larger.native/cuda_vision_offer.zig: a tower that declaresprepareVideogets aVideoOffer(its video and group markers, and the server's video limits), with the server'sVideoSourcedecoded into the family's sink.native/cuda.zig:--vision's video limits reach a family whose options take them.Videos need
libtfvideobeside the server (tools/zig/build_ffmpeg.sh PREFIX, thenzig build video -Dffmpeg=PREFIX), as on Metal.Checks
zig build test,test-kernel-contract,test-golden,fuzz-cortheon,test-cuda-host,zig fmt --check, and the lean check (0 new). The Metal build (zig build nativeon an M5 Ultra) builds with the shared planner.libtfvideofrombuild_ffmpeg.sh), the recipe suite's vision checks: 6/6 on images and videos (a moving ball's direction, a background's colour change), and 16/16 with 16 image and video requests at once, each equal to its reply alone.