From 22007a7aeda2e6103723ea5d4625db284c3f66cc Mon Sep 17 00:00:00 2001 From: lichang Date: Tue, 10 Nov 2026 15:41:06 -0700 Subject: [PATCH 1/4] mtmd: batch Qwen2.5-VL windows and avoid extra copies Group equal-sized vision windows, size graph metadata for the resulting groups, and skip the unused dense mask. Convert K/V before splitting and fuse eligible attention output writes into the joined buffer. Include runtime controls, focused backend tests, and fresh Qwen2.5-VL-3B validation against gfx11. Assisted-by: Codex --- .../qwen25vl-window-batch-gfx11.md | 64 ++++++++++++++++ ggml/src/ggml-cuda/ggml-cuda.cu | 37 +++++++++ tests/test-backend-ops.cpp | 25 +++++++ tools/mtmd/clip.cpp | 47 ++++++++++-- tools/mtmd/models/qwen2vl.cpp | 75 ++++++++++++++++++- 5 files changed, 238 insertions(+), 10 deletions(-) create mode 100644 docs/development/qwen25vl-window-batch-gfx11.md diff --git a/docs/development/qwen25vl-window-batch-gfx11.md b/docs/development/qwen25vl-window-batch-gfx11.md new file mode 100644 index 000000000000..8a8aaf4688d2 --- /dev/null +++ b/docs/development/qwen25vl-window-batch-gfx11.md @@ -0,0 +1,64 @@ +# Qwen2.5-VL window batching on gfx11 + +This focused branch starts at ROCm llama.cpp `gfx11` revision `f70aba1cb60547b13a6d2f1ef70c158dd59ca51e`. It contains the window-attention graph transformation and its two input/output follow-ups. Measurements below were collected for this branch on November 10, 2026. + +## Scope + +Qwen2.5-VL local-attention layers become batches of independent windows instead of a single large masked attention operation. Consecutive windows of equal token count share a batch entry group; partial edge windows form separate groups. Existing patch order, position IDs, global-attention layers, image preprocessing, and the output projection are preserved. + +The core change also skips allocation/upload of the unused dense window mask and grows graph/scheduler metadata capacity with the number of window groups. Scheduler evaluation callbacks are preserved when capacity grows. The first direct port exposed the fixed metadata-pool limit during model warmup; capacity growth is required support for batching, not an additional measured speedup. + +Two companion changes convert complete K/V tensors before taking window views, and let eligible RDNA3.5 attention operations write directly into the joined output buffer. The latter retains checks for consumers, aliasing, alignment, output bounds, and temporary-buffer requirements. + +No MMDQ changes, new attention kernels, WMMA dispatch changes, or language-model fusions are included. + +## Controls + +Each switch defaults to enabled. Set it to `0` in a new process to select the fallback. + +| Switch | Function | Prerequisite | +| --- | --- | --- | +| `MTMD_QWEN25VL_WINDOW_BATCH` | Window batching, mask suppression, and graph-capacity support | Qwen2.5-VL window attention with Flash Attention enabled | +| `MTMD_QWEN25VL_KV_CAST_EARLY` | Convert K/V once before window views | Window batching | +| `GGML_CUDA_FUSE_ATTN_SET` | Write attention output directly into its joined buffer | Eligible attention -> reshape -> in-place SET graph on RDNA3.5 | + +The three switches define one tested package. This experiment does not attribute an independent gain to either companion switch. + +## One-model verification + +- Board: `xconucstrhalo06`, Strix Halo / gfx1151. +- Model: Qwen2.5-VL-3B-Instruct, Q4_K_M language weights and FP16 projector. +- Input: the saved 1024 x 800 JPEG and original text request; **1,606 prompt positions** on both builds. +- Tool: `llama-server`, streamed HTTP TTFT to the first nonempty content token, measured identically for both builds. +- Settings: context 4096, batch and microbatch 2048, 8 CPU threads, one slot, full GPU offload, Flash Attention enabled, FP16 K/V cache, checkpoints 0, prompt caching disabled, temperature 0, output cap 32. +- Timing order: baseline, candidate, candidate, baseline. Each process has two untimed warmups and five retained requests; the table uses the median of ten retained samples per build. Loading and model staging are outside the timed interval. + +| Configuration | Median TTFT ms | Relative to gfx11 | +| --- | --- | --- | +| Unchanged gfx11 | 2158.59 | 1.00x | +| Window package enabled | 974.06 | 2.216x faster; 54.88% less time | +| Candidate with all three switches disabled | 2163.24 | Fallback check; three retained samples | + +One candidate request took 2,059.42 ms and remains included in the ten retained samples. Candidate process medians were 972.41 and 974.41 ms; baseline process medians were 2,156.07 and 2,161.65 ms. + +The comparison uses matching model hashes, request hashes, prompt positions, and effective workload settings. No retained sample has a detected foreign-job overlap or cached prompt tokens. The initial Halo41 run was excluded after that board became occupied; both sides of the reported comparison were measured on Halo06. + +## Correctness and build validation + +- Release HIP build completed with the same ROCm SDK and compiler settings as the baseline. Target gfx1151, VMM disabled, unsafe math disabled. +- `test-backend-ops test -b ROCm0 -o FLASH_ATTN_EXT_SET`: 18/18 cases passed with fusion enabled, and 18/18 with it disabled, against the CPU reference. Cases include small/full/partial attention lengths, multiple batch entries, an additional consumer, and out-of-place SET fallbacks. +- Separate model verification generated identical text: **true**. Common generated-token prefix: 32 of 32 baseline tokens; maximum absolute log-probability difference over that prefix: 0.015620947. +- Final vision embedding shape: 1073 tokens x 2048 channels. Relative L2 difference: 0.0078391009; normalized mean square error: 6.1451503e-05; cosine similarity: 0.9999693038. +- These checks establish numerical agreement for the tested operators and request. They are not a model-wide accuracy benchmark or validation of other GPU backends. + +## Reproduction and raw evidence + +The [experiment directory](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110) contains the frozen request, source patch, binary hashes, exact commands, per-request timings, model output, and cleanup records. See [analysis.json](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/analysis.json), [source_state.json](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/source_state.json), and [build manifest](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/builds/candidate/manifest.json). + +Build: + +```bash +python3 /proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/build.py candidate /proj/gdba/lichang/llama.cpp-qwen25vl-window-batch-gfx11 +``` + +The exact benchmark commands and runtime settings are saved in each run's `metadata.json`. The driver is [run_verification.py](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/run_verification.py); use a new artifact directory or labels when collecting another run. diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 929353e00ec9..d289e43042e2 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -4087,6 +4087,43 @@ static int ggml_cuda_try_fuse(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph ggml_tensor * node = cgraph->nodes[i]; + static const bool fuse_attn_set = ggml_cuda_env_int("GGML_CUDA_FUSE_ATTN_SET", 1) != 0; + if (fuse_attn_set && node->op == GGML_OP_FLASH_ATTN_EXT && GGML_CUDA_CC_IS_RDNA3_5(ggml_cuda_info().devices[cuda_ctx->device].cc)) { + const ggml_op ops[] = { GGML_OP_FLASH_ATTN_EXT, GGML_OP_RESHAPE, GGML_OP_SET }; + const int output_idx = i + 2; + if (ggml_can_fuse_subgraph(cgraph, i, 3, ops, &output_idx, 1)) { + ggml_tensor * view = cgraph->nodes[i + 1]; + ggml_tensor * set = cgraph->nodes[output_idx]; + const int32_t * params = (const int32_t *) set->op_params; + const size_t offset = (size_t) params[3]; + if (view->src[0] == node && set->src[1] == view && params[4] == 1 && + node->type == GGML_TYPE_F32 && set->type == GGML_TYPE_F32 && + ggml_is_contiguous(node) && ggml_is_contiguous(set) && + view->ne[2] == 1 && view->ne[3] == 1 && params[0] == view->nb[1] && + offset <= ggml_nbytes(set) && ggml_nbytes(node) <= ggml_nbytes(set) - offset && + ggml_cuda_flash_attn_ext_get_alloc_size(cuda_ctx->device, node) == ggml_nbytes(node)) { + ggml_tensor dst = *node; + dst.data = (char *) set->data + offset; + dst.buffer = set->buffer; + bool overlap = false; + const uintptr_t begin = (uintptr_t) dst.data; + const uintptr_t end = begin + ggml_nbytes(&dst); + for (int j = 0; j < GGML_MAX_SRC; ++j) { + const ggml_tensor * src = node->src[j]; + if (src) { + const uintptr_t src_begin = (uintptr_t) src->data; + overlap = overlap || (begin < src_begin + ggml_nbytes(src) && src_begin < end); + } + } + if (!overlap && begin % 16 == 0) { + ggml_cuda_flash_attn_ext(*cuda_ctx, &dst); + return 2; + } + } + } + } + + if (node->op == GGML_OP_MUL_MAT_ID && cuda_ctx->stream_context().concurrent_events.empty() && ggml_cuda_match_shared_expert(cgraph, i, i + 3) && ggml_cuda_should_fuse_mul_mat_vec_q(cgraph->nodes[i + 2]->src[1])) { diff --git a/tests/test-backend-ops.cpp b/tests/test-backend-ops.cpp index 3e52e4abe10c..a44ee04f3baa 100644 --- a/tests/test-backend-ops.cpp +++ b/tests/test-backend-ops.cpp @@ -8926,6 +8926,23 @@ struct test_flash_attn_ext_kv_rows : public test_case { } }; +struct test_flash_attn_ext_set : public test_flash_attn_ext { + const bool extra_consumer, inplace; + test_flash_attn_ext_set(int64_t hs, int64_t tokens, bool extra_consumer = false, bool inplace = true) + : test_flash_attn_ext(hs, hs, 4, {1, 2}, tokens, tokens, false), extra_consumer(extra_consumer), inplace(inplace) {} + bool run_whole_graph() override { return true; } + std::string op_desc(ggml_tensor *) override { return "FLASH_ATTN_EXT_SET"; } + std::string vars() override { return test_flash_attn_ext::vars() + "," + VARS_TO_STR2(extra_consumer, inplace); } + ggml_tensor * build_graph(ggml_context * ctx) override { + ggml_tensor * attn = test_flash_attn_ext::build_graph(ctx); + ggml_tensor * flat = ggml_reshape_2d(ctx, attn, attn->ne[0] * attn->ne[1], attn->ne[2] * attn->ne[3]); + ggml_tensor * buffer = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, flat->ne[0], flat->ne[1] + 3); + ggml_tensor * out = inplace ? ggml_set_2d_inplace(ctx, buffer, flat, buffer->nb[1], buffer->nb[1]) : + ggml_set_2d(ctx, buffer, flat, buffer->nb[1], buffer->nb[1]); + return extra_consumer ? ggml_add(ctx, out, ggml_sum(ctx, flat)) : out; + } +}; + // GGML_OP_CROSS_ENTROPY_LOSS struct test_cross_entropy_loss : public test_case { const ggml_type type; @@ -12554,6 +12571,14 @@ static std::vector> make_test_cases_eval() { } // asymmetric head_dim (hsk != hsv) with one or both sides not 64-aligned + for (int64_t hs : {80, 128}) { + for (int64_t tokens : {16, 64, 65}) { + test_cases.emplace_back(new test_flash_attn_ext_set(hs, tokens)); + test_cases.emplace_back(new test_flash_attn_ext_set(hs, tokens, true)); + test_cases.emplace_back(new test_flash_attn_ext_set(hs, tokens, false, false)); + } + } + test_cases.emplace_back(new test_flash_attn_ext(72, 64, 4, {1, 1}, 256, 2, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16)); test_cases.emplace_back(new test_flash_attn_ext(64, 72, 4, {1, 1}, 256, 2, true, false, 0, 0, GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16)); test_cases.emplace_back(new test_flash_attn_ext(65, 67, 4, {1, 1}, 113, 75, true, true, 8.0f, 0, GGML_PREC_F32, GGML_TYPE_F16, GGML_TYPE_F16)); diff --git a/tools/mtmd/clip.cpp b/tools/mtmd/clip.cpp index 9a043d7f9ca3..f785a3ad249a 100644 --- a/tools/mtmd/clip.cpp +++ b/tools/mtmd/clip.cpp @@ -162,6 +162,8 @@ struct clip_ctx { int max_nodes = 8192; ggml_backend_sched_ptr sched; + ggml_backend_sched_eval_callback cb_eval; + void * cb_eval_user_data; clip_flash_attn_type flash_attn_type = CLIP_FLASH_ATTN_TYPE_AUTO; bool is_allocated = false; @@ -174,7 +176,8 @@ struct clip_ctx { bool support_batch = false; - clip_ctx(clip_context_params & ctx_params) { + clip_ctx(clip_context_params & ctx_params) : + cb_eval(ctx_params.cb_eval), cb_eval_user_data(ctx_params.cb_eval_user_data) { flash_attn_type = ctx_params.flash_attn_type; no_alloc = ctx_params.no_alloc; backend_cpu = ggml_backend_init_by_type(GGML_BACKEND_DEVICE_TYPE_CPU, nullptr); @@ -259,6 +262,31 @@ clip_graph::clip_graph(clip_ctx * ctx, const clip_image_f32 & img) : eps(hparams.eps), kq_scale(d_head > 0 ? 1.0f / sqrtf((float)d_head) : 0.0f), flash_attn_type(ctx->flash_attn_type) { + static const bool window_batch = [] { + const char * value = std::getenv("MTMD_QWEN25VL_WINDOW_BATCH"); + return value == nullptr || std::atoi(value) != 0; + }(); + if (window_batch && proj_type == PROJECTOR_TYPE_QWEN25VL && hparams.n_wa_pattern > 0 && + flash_attn_type != CLIP_FLASH_ATTN_TYPE_DISABLED) { + const int window_size = hparams.attn_window_size > 0 ? hparams.attn_window_size : 112; + const int grid_window = window_size / patch_size / 2; + GGML_ASSERT(grid_window > 0); + // Each row of windows has at most two runs of equal-length windows. + const int64_t max_runs = 2 * ((int64_t(n_patches_y) / 2 + grid_window - 1) / grid_window); + const int64_t max_nodes = 8192 + int64_t(n_layer) * max_runs * 32; + if (max_nodes > std::numeric_limits::max()) { + throw std::runtime_error("Qwen2.5-VL window graph is too large"); + } + if (max_nodes > ctx->max_nodes) { + ctx->max_nodes = int(max_nodes); + ctx->sched.reset(ggml_backend_sched_new(ctx->backend_ptrs.data(), ctx->backend_buft.data(), + ctx->backend_ptrs.size(), ctx->max_nodes, false, true)); + if (ctx->cb_eval != nullptr) { + ggml_backend_sched_set_eval_callback(ctx->sched.get(), ctx->cb_eval, ctx->cb_eval_user_data); + } + ctx->buf_compute_meta.resize(ctx->max_nodes * ggml_tensor_overhead() + ggml_graph_overhead_custom(ctx->max_nodes, false)); + } + } struct ggml_init_params params = { /*.mem_size =*/ ctx->buf_compute_meta.size(), /*.mem_buffer =*/ ctx->buf_compute_meta.data(), @@ -813,8 +841,12 @@ ggml_tensor * clip_graph::build_attn( if (flash_attn_type == CLIP_FLASH_ATTN_TYPE_ENABLED) { ggml_tensor * v = ggml_permute(ctx0, v_cur, 0, 2, 1, 3); - k = ggml_cast(ctx0, k, GGML_TYPE_F16); - v = ggml_cast(ctx0, v, GGML_TYPE_F16); + if (k->type != GGML_TYPE_F16) { + k = ggml_cast(ctx0, k, GGML_TYPE_F16); + } + if (v->type != GGML_TYPE_F16) { + v = ggml_cast(ctx0, v, GGML_TYPE_F16); + } // mask must be f16 here, use build_inp_attn_mask() GGML_ASSERT(!kq_mask || kq_mask->type == GGML_TYPE_F16); @@ -5839,7 +5871,8 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) { const int grid_window = attn_window_size / patch_size / merge_ratio; int dst = 0; // [num_vision_tokens, num_vision_tokens] attention mask tensor - std::vector mask(pow(ipw * iph, 2), std::numeric_limits::lowest()); + const bool has_mask = ggml_graph_get_tensor(gf, get_attn_mask_name(0).c_str()) != nullptr; + std::vector mask(has_mask ? size_t(ipw * iph) * (ipw * iph) : 0, std::numeric_limits::lowest()); int mask_row = 0; for (int y = 0; y < ph; y += grid_window) { @@ -5859,7 +5892,7 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) { } } - for (int r=0; r < win_h * win_w * merge_ratio * merge_ratio; r++) { + for (int r=0; has_mask && r < win_h * win_w * merge_ratio * merge_ratio; r++) { int row_offset = mask_row * (ipw * iph); std::fill( mask.begin() + row_offset + (dst_0 * merge_ratio * merge_ratio), @@ -5872,7 +5905,9 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) { set_input_i32("window_idx", idx); set_input_i32("inv_window_idx", inv_idx); - set_input_attn_mask(mask); + if (has_mask) { + set_input_attn_mask(mask); + } } else { for (int i = 0; i < ph * pw; i++) { idx[i] = i; diff --git a/tools/mtmd/models/qwen2vl.cpp b/tools/mtmd/models/qwen2vl.cpp index 7516aa23a986..126827ff6287 100644 --- a/tools/mtmd/models/qwen2vl.cpp +++ b/tools/mtmd/models/qwen2vl.cpp @@ -1,15 +1,47 @@ #include "models.h" +#include + ggml_cgraph * clip_graph_qwen2vl::build() { GGML_ASSERT(model.patch_bias == nullptr); GGML_ASSERT(model.class_embedding == nullptr); const int batch_size = 1; const bool use_window_attn = hparams.n_wa_pattern > 0; + static const bool window_batch = [] { + const char * value = std::getenv("MTMD_QWEN25VL_WINDOW_BATCH"); + return value == nullptr || std::atoi(value) != 0; + }(); + static const bool window_cast = [] { + const char * value = std::getenv("MTMD_QWEN25VL_KV_CAST_EARLY"); + return value == nullptr || std::atoi(value) != 0; + }(); + const bool batch_windows = window_batch && use_window_attn && flash_attn_type == CLIP_FLASH_ATTN_TYPE_ENABLED; const int n_wa_pattern = hparams.n_wa_pattern; const int n_pos = n_patches; const int num_position_ids = n_pos * 4; // m-rope requires 4 dim per position + // Keep the input window order and batch adjacent windows of equal size. + std::vector> window_runs; + if (batch_windows) { + const int merge_ratio = 2; + const int window_size = hparams.attn_window_size > 0 ? hparams.attn_window_size : 112; + const int grid_window = window_size / patch_size / merge_ratio; + GGML_ASSERT(grid_window > 0); + const int pw = n_patches_x / merge_ratio; + const int ph = n_patches_y / merge_ratio; + for (int y = 0; y < ph; y += grid_window) { + for (int x = 0; x < pw; x += grid_window) { + const int len = std::min(grid_window, ph - y) * std::min(grid_window, pw - x) * merge_ratio * merge_ratio; + if (!window_runs.empty() && window_runs.back().first == len) { + window_runs.back().second++; + } else { + window_runs.emplace_back(len, 1); + } + } + } + } + norm_type norm_t = proj_type == PROJECTOR_TYPE_QWEN25VL ? NORM_TYPE_RMS // qwen 2.5 vl : NORM_TYPE_NORMAL; // qwen 2 vl @@ -52,8 +84,9 @@ ggml_cgraph * clip_graph_qwen2vl::build() { inv_window_idx = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_pos / 4); ggml_set_name(inv_window_idx, "inv_window_idx"); ggml_set_input(inv_window_idx); - // mask for window attention - window_mask = build_inp_attn_mask(n_pos, n_pos); + if (!batch_windows) { + window_mask = build_inp_attn_mask(n_pos, n_pos); + } // inpL shape: [n_embd, n_patches_x * n_patches_y, batch_size] GGML_ASSERT(batch_size == 1); @@ -103,8 +136,42 @@ ggml_cgraph * clip_graph_qwen2vl::build() { ggml_tensor * attn_mask = full_attn ? nullptr : window_mask; - cur = build_attn(layer.o_w, layer.o_b, - Qcur, Kcur, Vcur, attn_mask, kq_scale, il); + if (batch_windows && !full_attn) { + // Convert each full K/V tensor once before taking window views. + if (window_cast) { + Kcur = ggml_cast(ctx0, Kcur, GGML_TYPE_F16); + Vcur = ggml_cast(ctx0, Vcur, GGML_TYPE_F16); + } + std::vector outputs; + int offset = 0; + for (const auto & run : window_runs) { + auto view = [&](ggml_tensor * t) { + return ggml_view_4d(ctx0, t, d_head, n_head, run.first, run.second, + t->nb[1], t->nb[2], t->nb[2] * run.first, t->nb[2] * offset); + }; + outputs.push_back(build_attn(nullptr, nullptr, + view(Qcur), view(Kcur), view(Vcur), nullptr, kq_scale, il)); + offset += run.first * run.second; + } + GGML_ASSERT(offset == n_patches); + ggml_tensor * joined = outputs.front(); + if (outputs.size() > 1) { + joined = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_embd, n_patches); + size_t offset_bytes = 0; + for (ggml_tensor * output : outputs) { + joined = ggml_set_2d_inplace(ctx0, joined, output, joined->nb[1], offset_bytes); + offset_bytes += ggml_nbytes(output); + } + GGML_ASSERT(offset_bytes == ggml_nbytes(joined)); + } + cur = build_mm(layer.o_w, joined); + if (layer.o_b) { + cur = ggml_add(ctx0, cur, layer.o_b); + } + } else { + cur = build_attn(layer.o_w, layer.o_b, + Qcur, Kcur, Vcur, attn_mask, kq_scale, il); + } cb(cur, "attn_out", il); } From ee8ce9ae9d6887dd81c031b61dd7afe8cc184a09 Mon Sep 17 00:00:00 2001 From: lichang Date: Tue, 10 Nov 2026 18:02:00 -0700 Subject: [PATCH 2/4] docs: move Qwen2.5-VL validation to PR #192 Assisted-by: Codex --- .../qwen25vl-window-batch-gfx11.md | 64 ------------------- 1 file changed, 64 deletions(-) delete mode 100644 docs/development/qwen25vl-window-batch-gfx11.md diff --git a/docs/development/qwen25vl-window-batch-gfx11.md b/docs/development/qwen25vl-window-batch-gfx11.md deleted file mode 100644 index 8a8aaf4688d2..000000000000 --- a/docs/development/qwen25vl-window-batch-gfx11.md +++ /dev/null @@ -1,64 +0,0 @@ -# Qwen2.5-VL window batching on gfx11 - -This focused branch starts at ROCm llama.cpp `gfx11` revision `f70aba1cb60547b13a6d2f1ef70c158dd59ca51e`. It contains the window-attention graph transformation and its two input/output follow-ups. Measurements below were collected for this branch on November 10, 2026. - -## Scope - -Qwen2.5-VL local-attention layers become batches of independent windows instead of a single large masked attention operation. Consecutive windows of equal token count share a batch entry group; partial edge windows form separate groups. Existing patch order, position IDs, global-attention layers, image preprocessing, and the output projection are preserved. - -The core change also skips allocation/upload of the unused dense window mask and grows graph/scheduler metadata capacity with the number of window groups. Scheduler evaluation callbacks are preserved when capacity grows. The first direct port exposed the fixed metadata-pool limit during model warmup; capacity growth is required support for batching, not an additional measured speedup. - -Two companion changes convert complete K/V tensors before taking window views, and let eligible RDNA3.5 attention operations write directly into the joined output buffer. The latter retains checks for consumers, aliasing, alignment, output bounds, and temporary-buffer requirements. - -No MMDQ changes, new attention kernels, WMMA dispatch changes, or language-model fusions are included. - -## Controls - -Each switch defaults to enabled. Set it to `0` in a new process to select the fallback. - -| Switch | Function | Prerequisite | -| --- | --- | --- | -| `MTMD_QWEN25VL_WINDOW_BATCH` | Window batching, mask suppression, and graph-capacity support | Qwen2.5-VL window attention with Flash Attention enabled | -| `MTMD_QWEN25VL_KV_CAST_EARLY` | Convert K/V once before window views | Window batching | -| `GGML_CUDA_FUSE_ATTN_SET` | Write attention output directly into its joined buffer | Eligible attention -> reshape -> in-place SET graph on RDNA3.5 | - -The three switches define one tested package. This experiment does not attribute an independent gain to either companion switch. - -## One-model verification - -- Board: `xconucstrhalo06`, Strix Halo / gfx1151. -- Model: Qwen2.5-VL-3B-Instruct, Q4_K_M language weights and FP16 projector. -- Input: the saved 1024 x 800 JPEG and original text request; **1,606 prompt positions** on both builds. -- Tool: `llama-server`, streamed HTTP TTFT to the first nonempty content token, measured identically for both builds. -- Settings: context 4096, batch and microbatch 2048, 8 CPU threads, one slot, full GPU offload, Flash Attention enabled, FP16 K/V cache, checkpoints 0, prompt caching disabled, temperature 0, output cap 32. -- Timing order: baseline, candidate, candidate, baseline. Each process has two untimed warmups and five retained requests; the table uses the median of ten retained samples per build. Loading and model staging are outside the timed interval. - -| Configuration | Median TTFT ms | Relative to gfx11 | -| --- | --- | --- | -| Unchanged gfx11 | 2158.59 | 1.00x | -| Window package enabled | 974.06 | 2.216x faster; 54.88% less time | -| Candidate with all three switches disabled | 2163.24 | Fallback check; three retained samples | - -One candidate request took 2,059.42 ms and remains included in the ten retained samples. Candidate process medians were 972.41 and 974.41 ms; baseline process medians were 2,156.07 and 2,161.65 ms. - -The comparison uses matching model hashes, request hashes, prompt positions, and effective workload settings. No retained sample has a detected foreign-job overlap or cached prompt tokens. The initial Halo41 run was excluded after that board became occupied; both sides of the reported comparison were measured on Halo06. - -## Correctness and build validation - -- Release HIP build completed with the same ROCm SDK and compiler settings as the baseline. Target gfx1151, VMM disabled, unsafe math disabled. -- `test-backend-ops test -b ROCm0 -o FLASH_ATTN_EXT_SET`: 18/18 cases passed with fusion enabled, and 18/18 with it disabled, against the CPU reference. Cases include small/full/partial attention lengths, multiple batch entries, an additional consumer, and out-of-place SET fallbacks. -- Separate model verification generated identical text: **true**. Common generated-token prefix: 32 of 32 baseline tokens; maximum absolute log-probability difference over that prefix: 0.015620947. -- Final vision embedding shape: 1073 tokens x 2048 channels. Relative L2 difference: 0.0078391009; normalized mean square error: 6.1451503e-05; cosine similarity: 0.9999693038. -- These checks establish numerical agreement for the tested operators and request. They are not a model-wide accuracy benchmark or validation of other GPU backends. - -## Reproduction and raw evidence - -The [experiment directory](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110) contains the frozen request, source patch, binary hashes, exact commands, per-request timings, model output, and cleanup records. See [analysis.json](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/analysis.json), [source_state.json](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/source_state.json), and [build manifest](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/builds/candidate/manifest.json). - -Build: - -```bash -python3 /proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/build.py candidate /proj/gdba/lichang/llama.cpp-qwen25vl-window-batch-gfx11 -``` - -The exact benchmark commands and runtime settings are saved in each run's `metadata.json`. The driver is [run_verification.py](/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/run_verification.py); use a new artifact directory or labels when collecting another run. From 66358db15ff599c93106a1ef0cf80a7ae439252b Mon Sep 17 00:00:00 2001 From: lichang Date: Tue, 10 Nov 2026 18:02:00 -0700 Subject: [PATCH 3/4] ci: refresh HIP attention spill exception symbols Account for the KV_rows_ptr argument added in 1b6935b22. Update only the 20 existing kernel specializations reported by the gfx908 check; keep the spill threshold unchanged. Assisted-by: Codex --- scripts/hip/gcn-cdna-vgpr-check.py | 41 +++++++++++++++--------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/scripts/hip/gcn-cdna-vgpr-check.py b/scripts/hip/gcn-cdna-vgpr-check.py index 2d371c691986..04a06ef76b0b 100644 --- a/scripts/hip/gcn-cdna-vgpr-check.py +++ b/scripts/hip/gcn-cdna-vgpr-check.py @@ -156,27 +156,28 @@ def main(): '_ZL9mul_mat_qIL9ggml_type10ELi64ELb1EEvPKcPKiS4_S4_PfS5_PKf15HIP_vector_typeIjLj3EEiiiiiS9_S9_iiiS9_S9_iiiS9_S4_S4_i', '_ZL9mul_mat_qIL9ggml_type10ELi64ELb1EEvPKcPKiS4_S4_PfS5_PKf15HIP_vector_typeIjLj3EEiiiiiS9_S9_iiiS9_S9_iiiS9_', '_ZL9mul_mat_qIL9ggml_type42ELi128ELb1EEvPKcPKiS4_S4_PfS5_PKf15HIP_vector_typeIjLj3EEiiiiiS9_S9_iiiS9_S9_iiiS9_', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi2ELi32ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi4ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi4ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi32ELi2ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi32ELi2ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi8ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi8ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi16ELi4ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi4ELi16ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi2ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi2ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi4ELi8ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi4ELi8ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi4ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi4ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi1ELi32ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi2ELi16ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi8ELi4ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_vecILi256ELi1EL9ggml_type2ELS0_2ELb0EEvPKcS2_S2_S2_S2_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS6_IjLj3EEiiiiiiiiiiiliiliiiiil', + # KV_rows_ptr added in 1b6935b22 changes the HIP names of these existing exceptions. + '_ZL18flash_attn_ext_f16ILi576ELi512ELi2ELi32ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi4ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi4ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi32ELi2ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi32ELi2ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi8ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi8ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi576ELi512ELi16ELi4ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi576ELi512ELi4ELi16ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi2ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi16ELi2ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi4ELi8ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi4ELi8ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi4ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi512ELi512ELi8ELi4ELb1ELb0ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi576ELi512ELi1ELi32ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi576ELi512ELi2ELi16ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi576ELi512ELi8ELi4ELb0ELb1ELb0EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_vecILi256ELi1EL9ggml_type2ELS0_2ELb0EEvPKcS2_S2_S2_S2_PKiS4_PfP15HIP_vector_typeIfLj2EEffffjfiS6_IjLj3EEiiiiiiiiiiiliiliiiiil', '_ZL18flash_attn_ext_f16ILi320ELi256ELi2ELi32ELb0ELb0ELb0EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', - '_ZL18flash_attn_ext_f16ILi576ELi512ELi1ELi16ELb0ELb1ELb1EEvPKcS1_S1_S1_S1_PKiPfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', + '_ZL18flash_attn_ext_f16ILi576ELi512ELi1ELi16ELb0ELb1ELb1EEvPKcS1_S1_S1_S1_PKiS3_PfP15HIP_vector_typeIfLj2EEffffjfiS5_IjLj3EEiiiiiiiiiiiliiliiiiil', } functions = parse_log_file(log_file) From 78fadfe184cb418b10821513be1109bc624a3565 Mon Sep 17 00:00:00 2001 From: lichang Date: Fri, 13 Nov 2026 12:51:07 -0700 Subject: [PATCH 4/4] mtmd: reserve replacement Qwen2.5-VL schedulers Prepare graph capacity before the warmup check and clear allocation state when replacing the scheduler, so a larger image reserves the new scheduler before graph allocation. Assisted-by: Codex --- tools/mtmd/clip.cpp | 61 ++++++++++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 25 deletions(-) diff --git a/tools/mtmd/clip.cpp b/tools/mtmd/clip.cpp index f785a3ad249a..ca70568b4ccb 100644 --- a/tools/mtmd/clip.cpp +++ b/tools/mtmd/clip.cpp @@ -240,40 +240,22 @@ struct clip_ctx { } }; -// -// clip_graph -// - -clip_graph::clip_graph(clip_ctx * ctx, const clip_image_f32 & img) : - model(ctx->model), - hparams(model.hparams), - proj_type(ctx->proj_type()), - img(img), - patch_size(hparams.patch_size), - n_patches_x(img.nx() / patch_size), - n_patches_y(img.ny() / patch_size), - n_patches(n_patches_x * n_patches_y), - n_embd(hparams.n_embd), - n_head(hparams.n_head), - n_head_kv(hparams.n_head_kv), - d_head(hparams.n_embd_head > 0 ? hparams.n_embd_head : (n_head > 0 ? n_embd / n_head : 0)), - n_layer(hparams.n_layer), - n_mmproj_embd(clip_n_mmproj_embd(ctx)), - eps(hparams.eps), - kq_scale(d_head > 0 ? 1.0f / sqrtf((float)d_head) : 0.0f), - flash_attn_type(ctx->flash_attn_type) { +static void clip_ensure_graph_capacity(clip_ctx * ctx, const clip_image_f32 & img) { static const bool window_batch = [] { const char * value = std::getenv("MTMD_QWEN25VL_WINDOW_BATCH"); return value == nullptr || std::atoi(value) != 0; }(); - if (window_batch && proj_type == PROJECTOR_TYPE_QWEN25VL && hparams.n_wa_pattern > 0 && - flash_attn_type != CLIP_FLASH_ATTN_TYPE_DISABLED) { + const auto & hparams = ctx->model.hparams; + if (window_batch && ctx->proj_type() == PROJECTOR_TYPE_QWEN25VL && hparams.n_wa_pattern > 0 && + ctx->flash_attn_type != CLIP_FLASH_ATTN_TYPE_DISABLED) { + const int patch_size = hparams.patch_size; + const int n_patches_y = img.ny() / patch_size; const int window_size = hparams.attn_window_size > 0 ? hparams.attn_window_size : 112; const int grid_window = window_size / patch_size / 2; GGML_ASSERT(grid_window > 0); // Each row of windows has at most two runs of equal-length windows. const int64_t max_runs = 2 * ((int64_t(n_patches_y) / 2 + grid_window - 1) / grid_window); - const int64_t max_nodes = 8192 + int64_t(n_layer) * max_runs * 32; + const int64_t max_nodes = 8192 + int64_t(hparams.n_layer) * max_runs * 32; if (max_nodes > std::numeric_limits::max()) { throw std::runtime_error("Qwen2.5-VL window graph is too large"); } @@ -281,12 +263,38 @@ clip_graph::clip_graph(clip_ctx * ctx, const clip_image_f32 & img) : ctx->max_nodes = int(max_nodes); ctx->sched.reset(ggml_backend_sched_new(ctx->backend_ptrs.data(), ctx->backend_buft.data(), ctx->backend_ptrs.size(), ctx->max_nodes, false, true)); + ctx->is_allocated = false; if (ctx->cb_eval != nullptr) { ggml_backend_sched_set_eval_callback(ctx->sched.get(), ctx->cb_eval, ctx->cb_eval_user_data); } ctx->buf_compute_meta.resize(ctx->max_nodes * ggml_tensor_overhead() + ggml_graph_overhead_custom(ctx->max_nodes, false)); } } +} + +// +// clip_graph +// + +clip_graph::clip_graph(clip_ctx * ctx, const clip_image_f32 & img) : + model(ctx->model), + hparams(model.hparams), + proj_type(ctx->proj_type()), + img(img), + patch_size(hparams.patch_size), + n_patches_x(img.nx() / patch_size), + n_patches_y(img.ny() / patch_size), + n_patches(n_patches_x * n_patches_y), + n_embd(hparams.n_embd), + n_head(hparams.n_head), + n_head_kv(hparams.n_head_kv), + d_head(hparams.n_embd_head > 0 ? hparams.n_embd_head : (n_head > 0 ? n_embd / n_head : 0)), + n_layer(hparams.n_layer), + n_mmproj_embd(clip_n_mmproj_embd(ctx)), + eps(hparams.eps), + kq_scale(d_head > 0 ? 1.0f / sqrtf((float)d_head) : 0.0f), + flash_attn_type(ctx->flash_attn_type) { + clip_ensure_graph_capacity(ctx, img); struct ggml_init_params params = { /*.mem_size =*/ ctx->buf_compute_meta.size(), /*.mem_buffer =*/ ctx->buf_compute_meta.data(), @@ -5286,6 +5294,9 @@ bool clip_encode(struct clip_ctx * ctx, struct clip_encode_params * params) { return false; } + // Grow before the warmup check so a replacement scheduler gets its buffers reserved. + clip_ensure_graph_capacity(ctx, imgs.entries[0]); + // if buffers are not allocated, we need to do a warmup run to allocate them if (!ctx->is_allocated) { clip_model_loader::warmup(*ctx, *params->imgs);