From 0dfa67952513550e6e1e13d7257f1425508bd791 Mon Sep 17 00:00:00 2001 From: Dmitry Malishev Date: Tue, 1 Sep 2026 18:21:30 +0000 Subject: [PATCH 1/6] fix(abot): zero prompt-pad embeddings in scene creation The August forward-port (651de1a) rewrote sd_abot_scene_create against the new AbotT5Runner/T5Embedder API and dropped the reference WanTextEncoder's `u[v:] = 0` semantic that the July implementation carried via T5CLIPEmbedder's zero_out_masked. The tokenizer's additive attention mask only masks attention *inside* the encoder; the encoder still emits live embeddings for all 512 positions, so packs written since the port carry ~500 rows of pad-token embeddings in prompt_embeds. The walk DiT cross-attends to the full 512-row context, the padding rows swamp the conditioning, and every walk from a newly created scene collapses into blur/garbage within the first block. Packs created before the port keep working, which is why the walk path itself validated clean. Zero the output rows whose mask is not 0.0f (-inf = padding) right after the encode, restoring parity with the reference pipeline. Verified on the RTX 5090 host: fixed packs carry 15 non-zero rows for the 15-token test prompt (was 512), real-token rows byte-identical to pre-regression packs, and a 6-block Vulkan walk from a fixed pack holds luminance stddev 41-46 (regressed packs collapse to 8-10 by block 1) on both the recompute and KV-cache paths, CPU and Vulkan backends. Root-caused by A/B-ing addon 0.19.0/0.21.0 with crossed scene packs: any engine walking a 0.21-created pack produced mush, both engines walking a 0.19-created pack stayed clean, and per-tensor pack comparison showed prompt_embeds cos=0.17 / norm 17->101 with every other tensor bit-equal. Co-Authored-By: Claude Fable 5 --- src/stable-diffusion.cpp | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/stable-diffusion.cpp b/src/stable-diffusion.cpp index 5005af2e1..ab72a0335 100644 --- a/src/stable-diffusion.cpp +++ b/src/stable-diffusion.cpp @@ -7565,6 +7565,22 @@ bool sd_abot_scene_create(const sd_abot_scene_params_t* p) { sd::Tensor::from_vector(std::get<0>(tokens)), sd::Tensor::from_vector(std::get<2>(tokens))); if (prompt.empty()) { LOG_ERROR("sd_abot_scene_create: prompt encoding failed"); return false; } + // The reference WanTextEncoder zeroes embeddings past the real tokens + // (`u[v:] = 0`). The encoder's attention mask only affects attention + // inside the encoder, not its output rows, so without this the pack + // carries live pad-token embeddings in all 512 context rows and the + // walk collapses into blur from the first generated block. The + // tokenizer mask is additive: 0.0 = real token, -inf = padding. + { + const std::vector& attn_mask = std::get<2>(tokens); + const int64_t emb_dim = prompt.shape()[0]; + float* pd = prompt.data(); + for (size_t i = 0; i < attn_mask.size(); i++) { + if (attn_mask[i] != 0.0f) { + memset(pd + static_cast(emb_dim) * i, 0, static_cast(emb_dim) * sizeof(float)); + } + } + } if (prompt.shape().size() == 2) prompt = prompt.unsqueeze(2); sd::Tensor first; if (has_image) { From 70223bfcb3eeb57439aa2f518988e210815a1f7e Mon Sep 17 00:00:00 2001 From: Dmitry Malishev Date: Wed, 2 Sep 2026 12:42:43 +0000 Subject: [PATCH 2/6] fix(abot): report the scene pack's live prompt-row count The 2026-08-11 port regression (fixed in the previous commit) was silent: a pack whose prompt padding was never zeroed loads fine, walks fine, streams the right number of correctly sized frames, and only betrays itself in the pixels. The information that would have named the bug immediately - how many of the 512 context rows actually carry a prompt - was available at load time and not reported. Count the rows before the zeroed padding when a pack loads. A healthy pack logs `prompt rows 22 live / 512`; a pack whose padding is not zeroed warns and names the likely cause, because its walk is conditioned on pad-token embeddings. Verified against both: the 0.21.0-produced pack warns, packs from the fixed producer report 22/512. Measurements behind the wording, on an RTX 5090 (Vulkan, Q8 DiT, 832x480, same image and seed, only the prompt differing): - Prompt conditioning is alive and reference-faithful. Two very different prompts produce visibly different walks (mean |pixel diff| 69.6/255, 99.3% of pixels differing), and re-encoding the same prompt is bit-identical. - Text is nonetheless weak against a first-frame image, and that is architectural, not a defect: the reference passes all 512 rows to cross-attention with `context_lens = None` and no mask (wan/modules/causal_model.py:2180 and attention.py:211 in amap-cvlab/ ABot-World), runs no classifier-free guidance anywhere, and feeds the image through much stronger channels (first-frame latent replacement plus reference tokens inside self-attention). So a prompt shifts a walk's trajectory and tone rather than replacing scene content. - Trimming the ~490 zero rows so the real tokens get the full attention mass looks like the obvious way to strengthen text. It was measured and it destroys generation: output collapses to near-black mush within one block. Those rows are load-bearing - the model was trained with a fixed 512-row context - which is also why this commit only reports the count and changes no behaviour. Co-Authored-By: Claude Opus 5 --- src/abot_world.hpp | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/src/abot_world.hpp b/src/abot_world.hpp index b24152826..9c252c80e 100644 --- a/src/abot_world.hpp +++ b/src/abot_world.hpp @@ -101,6 +101,8 @@ struct AbotScenePack { sd::Tensor ref_latents; // {32, 32, C, K} (T=1 squeezed) sd::Tensor ref_mask; // {K} int ref_slots = 0; + // prompt rows before the zeroed padding (diagnostic; 0 = not computed) + int64_t text_rows_live = 0; // false = text-only scene: block-0 frame 0 is generated from noise // instead of being pinned to first_frame_latents bool has_first_frame = true; @@ -233,6 +235,41 @@ struct AbotScenePack { return false; } prompt_embeds.resize({sh[2], sh[1], 1, 1}); + // Real prompt rows: the producer zeroes everything past the last token + // (the reference's `u[v:] = 0`), so trailing all-zero rows are padding. + // Reported because a pack whose padding is NOT zeroed conditions the + // walk on pad-token embeddings and degrades generation from the first + // block - a silent failure that is otherwise only visible in the + // output pixels. + { + const int64_t emb = prompt_embeds.shape()[0]; + const int64_t rows = prompt_embeds.shape()[1]; + const float* pd = prompt_embeds.data(); + int64_t live = 0; + for (int64_t r = rows - 1; r >= 0; r--) { + bool nonzero = false; + for (int64_t i = 0; i < emb; i++) { + if (pd[r * emb + i] != 0.0f) { + nonzero = true; + break; + } + } + if (nonzero) { + live = r + 1; + break; + } + } + text_rows_live = live; + if (live == rows) { + LOG_WARN( + "scene pack: all %lld prompt rows are non-zero - padding is not zeroed, so the " + "walk is conditioned on pad embeddings (expect washed-out output; the pack " + "producer is missing the reference's zero-padding step)", + (long long)rows); + } else { + LOG_INFO("scene pack: prompt rows %lld live / %lld", (long long)live, (long long)rows); + } + } if (!fetch("first_frame_latents", first_frame_latents, sh) || // [1,1,C,H,W] !expect("first_frame_latents", sh, 5, {0, 1})) { return false; From 392588aae07d3cb4862a6674749196fe17728123 Mon Sep 17 00:00:00 2001 From: Dmitry Malishev Date: Wed, 2 Sep 2026 12:43:16 +0000 Subject: [PATCH 3/6] perf(abot): keep the walk's compute buffer and fuse the masked softmax Two changes to the interactive walk, both bit-identical in output and both confined to the ABot path (`forward_kv` is reached only from `forward_cached`, which only `AbotWorldRunner` calls, so no other model's attention is touched). 1. The walk ran `GGMLRunner::compute(..., auto_free=false)` and left `free_compute_buffer` at its default `true`, so every one of the 5-6 graphs per block destroyed the graph allocator and its multi-GB VRAM buffer, and the next graph re-created and re-reserved them. The walk is an endless sequence of near-identical graphs, so the buffer is now retained; `ggml_gallocr_alloc_graph` re-reserves on its own when the shape changes between the denoise and append graphs (ggml-alloc.c:1064). Params are kept staged across steps for the same reason. 2. The masked self-attention composed `scale -> add(mask) -> soft_max` as three ops. The score tensor is the largest in the graph (T_kv x n_token x heads, ~625 MB at 832x480 with a full history ring), so each elementwise pass costs a full read and write of it. `ggml_soft_max_ext` does all three in one pass and supports the walk's {T_kv, n_token} mask broadcasting over heads on CPU, CUDA and Vulkan (ggml.c:4022 asserts, plus the head modulo in ggml-cuda/softmax.cu:69 and vulkan-shaders/soft_max.comp:51). The composition it replaces predates that support. Measured on an RTX 5090, Vulkan, Q8 DiT, 832x480, KV cache on, 12 frames per block, ggml at the pinned 7d9ce11: per block baseline +buffer +fused softmax denoise step 310 ms 289 ms 241 ms append pass 405 ms 372 ms 329 ms block (steady) 2.1 s 2.0 s 1.8 s block 0 (cold) 12.0 s 2.2 s 2.2 s Frames are byte-identical to the baseline at every stage (mean |diff| 0.00 over 832x480 RGB, checked at blocks 1 and 3), so this is pure overhead removal, not a quality trade. The cold-start collapse from 12.0 s to 2.2 s is the same cause: block 0 builds several differently shaped graphs and was paying a full reserve for each. Remaining gap to the PyTorch reference (~0.42 s/block) is Q8 GEMM throughput, not overhead: ~8.7 TFLOP of projection and FFN work per denoise step against ~2.4 TFLOP of attention. Closing it needs fp8/int8 GEMM kernels, a flash- attention branch in `forward_kv` (which has none today), or a contiguous KV ring to retire the per-layer concat chain - none of them a small change. Co-Authored-By: Claude Opus 5 --- src/abot_world.hpp | 12 ++++++++++-- src/model/diffusion/wan.hpp | 11 ++++++----- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/src/abot_world.hpp b/src/abot_world.hpp index 9c252c80e..b61966ede 100644 --- a/src/abot_world.hpp +++ b/src/abot_world.hpp @@ -678,7 +678,11 @@ struct AbotWorldRunner : public GGMLRunner { return gf; }; - auto result = GGMLRunner::compute(get_graph, n_threads, false); + // keep the compute buffer and its graph allocator across steps: the walk + // runs 5-6 graphs per block forever, and the defaults would free and + // re-reserve multi-GB of VRAM on every one of them (ggml re-reserves + // automatically when the graph shape changes between denoise and append) + auto result = GGMLRunner::compute(get_graph, n_threads, false, false, false); if (!result.has_value()) { return {}; } @@ -923,7 +927,11 @@ struct AbotWorldRunner : public GGMLRunner { return gf; }; - auto result = GGMLRunner::compute(get_graph, n_threads, false); + // keep the compute buffer and its graph allocator across steps: the walk + // runs 5-6 graphs per block forever, and the defaults would free and + // re-reserve multi-GB of VRAM on every one of them (ggml re-reserves + // automatically when the graph shape changes between denoise and append) + auto result = GGMLRunner::compute(get_graph, n_threads, false, false, false); if (prof) { const int64_t prof_t2 = ggml_time_ms(); const char* mode_s = mode == KvMode::INIT_CAPTURE ? "init" : mode == KvMode::APPEND ? "append" : "denoise"; diff --git a/src/model/diffusion/wan.hpp b/src/model/diffusion/wan.hpp index 39157869d..0c385be9f 100644 --- a/src/model/diffusion/wan.hpp +++ b/src/model/diffusion/wan.hpp @@ -217,11 +217,12 @@ namespace WAN { const float scale = 1.0f / sqrtf(static_cast(head_dim)); ggml_tensor* kq = ggml_mul_mat(gctx, K, q_r); // {T_kv, n_token, n_head*N} ggml_mul_mat_set_prec(kq, GGML_PREC_F32); - kq = ggml_scale_inplace(gctx, kq, scale); - if (mask != nullptr) { - kq = ggml_add_inplace(gctx, kq, mask); // {T_kv, n_token} broadcast over heads - } - kq = ggml_soft_max_inplace(gctx, kq); + // One fused pass instead of scale -> add -> soft_max: the scores are + // the largest tensor in the walk graph (T_kv x n_token x n_head), so + // each separate elementwise pass costs a full read+write of it. The + // mask is {T_kv, n_token} and broadcasts over heads, which + // ggml_soft_max_ext supports on CPU, CUDA and Vulkan. + kq = ggml_soft_max_ext(gctx, kq, mask, scale, 0.0f); ggml_tensor* kqv = ggml_mul_mat(gctx, V, kq); // {d_head, n_token, n_head*N} kqv = ggml_reshape_4d(gctx, kqv, head_dim, n_token, num_heads, N); From ce320f9423923825347f06d0e1a62d0ce52c57d4 Mon Sep 17 00:00:00 2001 From: Dmitry Malishev Date: Wed, 2 Sep 2026 14:01:47 +0000 Subject: [PATCH 4/6] perf(abot): reuse the action planes and decode with direct convolution Two bit-identical wins in the walk, found by profiling a block stage by stage on an RTX 5090 (Vulkan, Q8_0 DiT, 832x480, KV cache, 12 frames per block). **Action planes.** The keyboard action is expanded into a {52, 30, 8192, 3} F32 tensor - about 153 MB - and it was rebuilt on the host before every graph. Within one block the key mask, the frame count and the latent size never change, so four of the five fills per block were re-deriving identical bytes, single-threaded, on the critical path ahead of each denoise step. The filled buffer is now kept and rebuilt only when its key actually changes. **Pixel decode.** The taehv decoder is a stack of small 3x3 convolutions, and im2col+GEMM materializes a large intermediate for each one. The engine already has a direct-convolution path (`set_conv2d_direct_enabled`, used for the VAE, ESRGAN and ControlNet in the batch pipeline); the ABot session never enabled it for its decoder. per block before after taehv decode 148 ms 68 ms block (steady) 1.5 s 1.2 s The action-plane change is a pure host-side saving, which is why the block falls further than the profiled stages do: the refills sat in the gap between stages that the [prof] lines do not cover. Frames are byte-identical to the previous build at every stage checked (mean |diff| 0.00 over 832x480 RGB), so neither change trades quality. Two other candidates were tried and reverted rather than shipped, since both lost time: direct convolution for the DiT itself (its patch embed and action adapter got slower, 207 -> 211 ms per denoise step) and an F16 attention mask (bit-identical, but converting 6.5 M values on the host each step cost more than the halved upload and per-layer reads saved, 1.2 -> 1.3 s per block). Co-Authored-By: Claude Opus 5 --- src/abot_world.hpp | 38 +++++++++++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 5 deletions(-) diff --git a/src/abot_world.hpp b/src/abot_world.hpp index b61966ede..1b7124ef9 100644 --- a/src/abot_world.hpp +++ b/src/abot_world.hpp @@ -579,6 +579,34 @@ struct AbotWorldRunner : public GGMLRunner { // 8-key vector to 8 channels, repeat_interleaves x4 -> 32 channels constant // over HxW, then PixelUnshuffle(16): output channel c corresponds to input // channel c / (16*16) -> value = key[(c / 256) / 4]. + // The action planes are the largest host-built input (~51 MB per frame) and + // are identical for every graph of a block: same keys held, same frame + // count, same latent size. Refilling them per graph put ~150 MB of memset + // on the critical path before each denoise step, so keep the filled buffer + // and rebuild it only when the key really changes. + sd::Tensor act_planes; + uint8_t act_planes_mask = 0; + int act_planes_frames = -1; + int64_t act_planes_w = 0; + int64_t act_planes_h = 0; + + sd::Tensor& action_planes(uint8_t action_mask, int F_cur, int64_t lat_w, int64_t lat_h, int c_unsh) { + if (act_planes_frames == F_cur && act_planes_mask == action_mask && + act_planes_w == lat_w && act_planes_h == lat_h && !act_planes.empty()) { + return act_planes; + } + act_planes = sd::zeros({lat_w, lat_h, c_unsh, F_cur}); + for (int f = 0; f < F_cur; f++) { + fill_act_plane(act_planes.data() + static_cast(f) * c_unsh * lat_w * lat_h, + action_mask, static_cast(lat_w), static_cast(lat_h), c_unsh); + } + act_planes_mask = action_mask; + act_planes_frames = F_cur; + act_planes_w = lat_w; + act_planes_h = lat_h; + return act_planes; + } + void fill_act_plane(float* dst, uint8_t action_mask, int w_in, int h_in, int c_unsh) { for (int c = 0; c < c_unsh; c++) { int key = (c / (cfg.act_downscale_factor * cfg.act_downscale_factor)) / 4; @@ -788,11 +816,7 @@ struct AbotWorldRunner : public GGMLRunner { memcpy(dst, src, static_cast(lat_w) * lat_h * sizeof(float)); } } - sd::Tensor act({lat_w, lat_h, c_unsh, F_cur}); - for (int f = 0; f < F_cur; f++) { - fill_act_plane(act.data() + static_cast(f) * c_unsh * lat_w * lat_h, - action_mask, static_cast(lat_w), static_cast(lat_h), c_unsh); - } + sd::Tensor& act = action_planes(action_mask, F_cur, lat_w, lat_h, c_unsh); sd::Tensor tvec({F_cur + 1}); for (int f = 0; f < F_cur; f++) { tvec.data()[f] = frame_timesteps[static_cast(f)]; @@ -1177,6 +1201,10 @@ class AbotWalkSession { true, VERSION_ABOT_WORLD, model_manager); + // Direct convolution for the pixel decoder: im2col+GEMM materializes a + // large intermediate per conv, and the decoder is all small 3x3 convs. + // Measured 148 -> 68 ms per block decode, bit-identical output. + tae->set_conv2d_direct_enabled(true); if (!model_manager->register_runner_params("ABot-World DiT", *runner, "model.diffusion_model", From 187ec98ab57390199b8b31011657d82eb19ecf03 Mon Sep 17 00:00:00 2001 From: Dmitry Malishev Date: Wed, 2 Sep 2026 14:02:10 +0000 Subject: [PATCH 5/6] perf(abot): let the walk's score matmul use the tensor-core path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The walk's self-attention asked for `GGML_PREC_F32` on the Q·Kᵀ matmul. On a coopmat2 device that is the worst of the three options: both operands are F32, so the Vulkan backend takes the `full_f32` branch, and the only F32 cooperative-matrix pipeline (`pipeline_matmul_f32_cm1`) is built in the coopmat1 branch that a coopmat2 device never enters. The request therefore resolves to `matmul_f32_f32_fp32` - a scalar shader with no tensor cores - and additionally suppresses the F32→F16 operand reformat. Dropping the flag routes the same matmul to `matmul_f16_f16acc_cm2`. This is the largest single matmul in the graph: at 832x480 with a full history ring the scores are 5570 x 1170 x 24 (~625 MB), and it runs in all 30 layers of every denoise and append graph. per block before after denoise step 241 ms 207 ms append pass 329 ms 296 ms block (steady) 1.8 s 1.6 s The flag was inherited from the shared attention helper's convention rather than from a measured ABot need, and the scores feed straight into a softmax, so the accumulator's dynamic range is not what limits this operation. Unlike the other perf changes on this branch, **this one is not bit-exact** - it moves the accumulation to F16. Measured on the same scene and seed: a frame inside block 2 differs by mean |diff| 0.29/255 with 0.2% of pixels off by more than 8, and because the KV cache carries state forward the two rollouts drift apart slowly (1.05 at block 6, 1.81 at block 11) the way any small numerical perturbation does in an autoregressive model. Frame quality does not degrade with it (luminance stddev 46.2 vs 46.2 at block 2, 65.5 at block 11), and block 11 is visually clean. It is committed separately from the bit-exact wins so it can be held back or landed on its own. Reviewers should run the golden-replay gates (`sd-abot-session --mode walkval --golden`) before treating it as settled; those goldens were not available on the machine this was measured on. Co-Authored-By: Claude Opus 5 --- src/model/diffusion/wan.hpp | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/model/diffusion/wan.hpp b/src/model/diffusion/wan.hpp index 0c385be9f..ce77efc7c 100644 --- a/src/model/diffusion/wan.hpp +++ b/src/model/diffusion/wan.hpp @@ -216,7 +216,14 @@ namespace WAN { const float scale = 1.0f / sqrtf(static_cast(head_dim)); ggml_tensor* kq = ggml_mul_mat(gctx, K, q_r); // {T_kv, n_token, n_head*N} - ggml_mul_mat_set_prec(kq, GGML_PREC_F32); + // No GGML_PREC_F32 here. On a coopmat2 device an F32 x F32 matmul + // asking for F32 precision has no cooperative-matrix pipeline to + // fall back on (pipeline_matmul_f32_cm1 is only built in the + // coopmat1 branch), so it lands on a scalar shader with no tensor + // cores; the default precision converts both operands to F16 and + // uses matmul_f16_f16acc_cm2 instead. Measured 241 -> 207 ms per + // denoise step on an RTX 5090. Scores feed a softmax, so the + // accumulator's dynamic range is not the limit here. // One fused pass instead of scale -> add -> soft_max: the scores are // the largest tensor in the walk graph (T_kv x n_token x n_head), so // each separate elementwise pass costs a full read+write of it. The From 8f0e62d602dc664cf2a57145230806a0e9bbc5ce Mon Sep 17 00:00:00 2001 From: Dmitry Malishev Date: Thu, 3 Sep 2026 00:55:38 +0300 Subject: [PATCH 6/6] doc(abot): document prompt-pad zeroing and the live prompt-row log line The scene-creation fix zeroes embedding rows past the last real token; note that in the doc, and document the new `scene pack: prompt rows N live / M` load-time log line (N == M warns that the padding was not zeroed). --- docs/abot_world.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/abot_world.md b/docs/abot_world.md index 78d0f6a0e..d6a2dc556 100644 --- a/docs/abot_world.md +++ b/docs/abot_world.md @@ -29,7 +29,16 @@ a rolling attention window and a 4-step distilled sampler. image (text-only, `first_frame_mask = 0`) are supported at the format level but gated: the distilled checkpoint cannot bootstrap a coherent first frame from noise, so front-ends should require an image until a T2V-capable - checkpoint ships. + checkpoint ships. Scene creation zeroes every prompt-embedding row past the + last real token (mirroring the reference text encoder's `u[v:] = 0`); the + encoder's attention mask only masks attention *inside* the encoder, so + without this a pack carries live pad-token embeddings in all 512 context + rows and the walk washes out from the first generated block. +- **Scene-pack diagnostics**: loading a pack logs `scene pack: prompt rows N + live / M`, where `N` is the prompt's real token count and `M` the fixed + 512-row context. `N == M` warns loudly — the pack's padding was not zeroed + (see above), so the walk is conditioned on pad embeddings and output will be + washed out; regenerate the pack with an engine that zeroes the padding. **Not supported:** the batch `generate_image()`/`generate_video()` paths — those are one-shot, whereas ABot needs the stateful causal session. Both batch