Skip to content

mtmd: batch Qwen2.5-VL windows and avoid extra copies - #192

Merged
jimw567 merged 4 commits into
gfx11from
lichang.qwen25vl-window-batch-gfx11
Nov 13, 2026
Merged

jimw567 merged 4 commits into
gfx11from
lichang.qwen25vl-window-batch-gfx11

Conversation

@liangliangchang

@liangliangchang liangliangchang commented Nov 10, 2026 •

Copy link
Copy Markdown

Qwen2.5-VL local-attention layers currently process all image patches with a dense window mask. This change batches consecutive equal-sized windows as independent attention problems, handles partial edge windows, and avoids redundant K/V and output copies. On Qwen2.5-VL-3B-Instruct on Strix Halo, median end-to-end TTFT drops from 2158.59 ms to 974.06 ms: 54.88% lower, or 2.216x faster.

Implementation

  • Batch consecutive windows with the same token count. Preserve patch order, position IDs, global-attention layers, image preprocessing, and the combined output projection.
  • Skip allocation, preparation, and transfer of the dense window mask when the graph no longer needs it.
  • Grow graph and scheduler metadata capacity with the number of window groups; preserve the evaluation callback when rebuilding the scheduler and guard against node-count overflow.
  • Convert complete K/V tensors to FP16 before taking window views, and skip redundant casts of tensors already in FP16.
  • On RDNA3.5, let eligible FLASH_ATTN_EXT -> RESHAPE -> SET operations write directly into the joined output buffer. Retain checks for consumers, aliasing, alignment, destination bounds, and extra output-buffer storage.

The implementation reuses existing attention kernels. It does not change language GEMM, MMDQ, WMMA dispatch, or language-model fusions.

Performance

Fresh measurements collected on November 10, 2026, comparing gfx11 at f70aba1cb60547b13a6d2f1ef70c158dd59ca51e with the implementation in 22007a7aeda2e6103723ea5d4625db284c3f66cc.

Configuration Median TTFT Change from gfx11
Unchanged gfx11 2158.59 ms Baseline
Window package enabled 974.06 ms 1184.53 ms saved; 54.88% lower; 2.216x faster
Candidate with all three switches disabled 2163.24 ms Fallback verification

Conditions were identical on both builds:

  • Hardware: Strix Halo / gfx1151, board xconucstrhalo06.
  • Model: Qwen2.5-VL-3B-Instruct, Q4_K_M language weights and FP16 projector.
  • Input: the same 1024 x 800 JPEG and text request, with 1606 prompt positions on both builds.
  • Measurement: llama-server, streamed HTTP TTFT to the first nonempty content token. Loading and model staging are outside the timed interval.
  • Settings: context 4096, batch/microbatch 2048, 8 CPU threads, one slot, full GPU offload, Flash Attention enabled, FP16 K/V cache, checkpoints 0, prompt caching disabled, temperature 0, output cap 32.
  • Sampling: baseline/candidate/candidate/baseline order; two process starts per build, two warmups and five retained requests per start. Each reported baseline/candidate median uses ten retained requests; the fallback uses three.

One candidate request took 2059.42 ms and remains included. Candidate process medians were 972.41 and 974.41 ms; baseline process medians were 2156.07 and 2161.65 ms. The retained comparison uses matching model/request hashes, prompt positions, and effective workload settings, with no detected foreign-job overlap or cached prompt tokens.

These numbers measure the complete package. They do not establish an independent gain for either companion optimization.

Runtime controls

Each switch defaults to enabled. Set it to 0 before starting a new process to select its fallback.

Switch Function Prerequisite
MTMD_QWEN25VL_WINDOW_BATCH Window batching, mask suppression, and graph-capacity support Qwen2.5-VL window attention with Flash Attention enabled
MTMD_QWEN25VL_KV_CAST_EARLY Convert K/V once before window views Window batching
GGML_CUDA_FUSE_ATTN_SET Write attention output directly into its joined buffer Eligible attention -> reshape -> in-place SET graph on RDNA3.5

Validation

  • Release HIP build completed with the same ROCm SDK and compiler settings as the baseline: gfx1151, VMM disabled, unsafe math disabled.
  • test-backend-ops test -b ROCm0 -o FLASH_ATTN_EXT_SET: 18/18 passed with fusion enabled and 18/18 disabled, against the CPU reference. Cases cover small/full/partial attention lengths, multiple batch entries, an additional consumer, and out-of-place SET fallbacks.
  • Separate model verification produced the same 32 generated tokens. Maximum absolute log-probability difference over that prefix: 0.015620947.
  • Vision embeddings: 1073 tokens x 2048 channels; relative L2 difference 0.0078391009, normalized mean square error 6.1451503e-05, cosine similarity 0.9999693038.

These checks establish numerical agreement for the tested operators and request; they are not a model-wide accuracy benchmark or validation of other GPU backends.

HIP quality check maintenance

The gfx942 build with warnings treated as errors passed. The gfx908 spill check flagged 20 kernels whose exact names were already in the exception list before the KV_rows_ptr parameter was added in 1b6935b22. Refresh those 20 symbol names to match the existing signatures. The affected kernel sources are unchanged from the gfx11 base; the spill threshold and the set of excepted kernel specializations remain unchanged.

The HIP quality-check rerun passed on 66358db15, including the gfx942 build with warnings treated as errors and the gfx908 spill check. Local checker validation also confirms that unlisted spilling kernels and unlisted attention shapes still fail.

Reproduction and raw evidence

Raw artifacts are retained on the local filesystem at:

/proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110

This directory contains the frozen request, source patch, binary/model hashes, exact commands, per-request timings, model outputs, and cleanup records. analysis.json, source_state.json, and builds/candidate/manifest.json summarize the run and build evidence. Exact benchmark commands and effective settings are recorded in each run's metadata.json; run_verification.py is the driver.

python3 /proj/gdba/lichang/regression_results/qwen25vl_window_gfx11_20261110/build.py candidate /proj/gdba/lichang/llama.cpp-qwen25vl-window-batch-gfx11

The standalone validation document has been moved into this PR description.

Requirements

  • I have read and agree with the contributing guidelines.
  • AI usage disclosure: YES. Codex assisted with implementation, testing, analysis, and preparation of this description.

Group equal-sized vision windows, size graph metadata for the resulting groups, and skip the unused dense mask. Convert K/V before splitting and fuse eligible attention output writes into the joined buffer.

Include runtime controls, focused backend tests, and fresh Qwen2.5-VL-3B validation against gfx11.

Assisted-by: Codex
Account for the KV_rows_ptr argument added in 1b6935b. Update only the 20 existing kernel specializations reported by the gfx908 check; keep the spill threshold unchanged.

Assisted-by: Codex
Comment thread tools/mtmd/clip.cpp Resolved
Prepare graph capacity before the warmup check and clear allocation state when replacing the scheduler, so a larger image reserves the new scheduler before graph allocation.

Assisted-by: Codex

@jimw567 jimw567 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Please follow up with additional test after the merge.

@jimw567
jimw567 merged commit 740e717 into gfx11 Nov 13, 2026
13 of 21 checks passed
@jimw567
jimw567 deleted the lichang.qwen25vl-window-batch-gfx11 branch November 13, 2026 21:21
@liangliangchang

Copy link
Copy Markdown
Author

PR #192: long-lived server small-to-large validation

The requested small -> large -> same-large sequence completes in one llama-server process. The two large requests return identical text and identical vision embeddings. Scheduler replacement adds a small, measurable first-use cost, approximately 14-17 ms (2-3% of HTTP TTFT). The rebuilt scheduler is reserved before graph allocation and its compute-buffer reservation stays constant on subsequent requests.

There is a memory qualification: total process RSS is not flat during the first requests. A longer run identifies most of that increase as freed memory retained in allocator arenas as the HTTP worker pool is exercised. RSS then plateaus. The scheduler itself does not accumulate compute buffers.

Model and request conditions

  • Model: Qwen2.5-VL-3B-Instruct, Q4_K_M language weights and FP16 projector.
  • One long-lived llama-server process per validation session, one inference slot, full GPU offload, 8 CPU threads, Flash Attention enabled, FP16 KV cache, context 4096, batch/microbatch 2048.
  • Startup warmup enabled. Image-token budget 1024; loader dummy image 896 x 896.
  • Checkpoints disabled; prompt caching, prompt-cache RAM, and idle-slot caching disabled. Every request reports cache_n = 0 and runs the image encoder.
  • Actual PNG uploads through /v1/chat/completions, followed by the same text prompt: Describe the colors and shapes in this image.
  • Temperature 0, seed 1234, 32 generated tokens, streaming response, log probabilities recorded.
  • Default HTTP worker count: 31 on this 32-thread board.
  • All GPU work was under /scratch/gpu-locks/gpu0.lock, held through server shutdown. No foreign GPU overlap or foreign CPU contention was detected by the guards.

The preprocessed image dimensions match the uploaded dimensions. The tall image is within the image budget but requires more rows of vision windows, growing graph capacity from 24,576 to 32,768 nodes. The server PID stays unchanged throughout each session.

Requested sequence

Primary session: PID 3168880, runs/xconucstrhalo06/.

Request Image Vision tokens Prompt positions HTTP result Vision encode ms HTTP TTFT ms
1 224 x 224 64 94 200, 32 tokens completed 1580.81 1835.81
2 476 x 1344 816 846 200, 32 tokens completed 384.54 656.75
3 Same 476 x 1344 image 816 846 200, 32 tokens completed 371.65 642.93

The first small request includes cold execution initialization. Its timing is not a steady-state small-image benchmark.

Requests 2 and 3 have:

  • Identical generated text and all 32 generated token strings.
  • Bitwise-identical vision embeddings: 1,671,168 FP32 elements per image.
  • Embedding SHA-256: 7a8793bef9cbdb474a13117710ee39325a7d05c85e8bab6d6d031caf97a6b670.
  • Identical prompt length and no cached tokens.

Language token log probabilities are not bitwise identical; the maximum absolute difference is 0.0467324. Some additional repeats vary wording, such as Here's versus Here is, while retaining identical vision embeddings. This validation establishes repeatable vision output and equivalent requested responses; it does not establish bitwise determinism for the language backend.

First-large-request cost

The first large request is 13.82 ms slower than the following identical request in HTTP TTFT, or 2.15%. Its vision encode is 12.89 ms slower.

Operation during first large request Time ms
Create larger scheduler 0.249
Free previous scheduler and its buffers 12.566
Explicitly reserve replacement scheduler 1.851
Allocate inference graph using reserved buffers 1.064

Creation, teardown, and reservation sum to 14.67 ms. These measured components are not an exact subtraction of complete request times, which include ordinary execution variation.

The replacement scheduler already has 650,776,448 compute-buffer bytes (620.63 MiB across GPU and CPU) before ggml_backend_sched_alloc_graph(); that call leaves the byte count unchanged. Request 3 and every subsequent identical large request use the same scheduler and reservation, with no further scheduler replacement or explicit reservation.

A second server session reproduces the sequence: first large TTFT 660.81 ms, repeated large 644.21 ms, a 16.61 ms / 2.58% difference. Scheduler creation, teardown, and reservation total 14.46 ms in that session. This is a small one-time cost, not zero allocation overhead.

Memory investigation

The initial session includes three extra large requests. Vision compute-buffer bytes remain exactly constant, but host RSS initially increases by roughly 22 MiB per request. To distinguish retained allocator memory from accumulating live allocations, a second session runs one small image followed by 35 large requests and records malloc_info() along with process RSS and per-process KFD GPU allocation counters.

Second session: PID 3173552, runs/xconucstrhalo06_default_memory/. All 36 requests complete, and all 35 large-image embedding files are bitwise identical. The first two large responses again have identical text and 32 token strings.

Request Host RSS MiB Allocator arena space MiB Free arena space MiB Estimated used arena space MiB Process KFD GPU allocation MiB
2, first large 1136.12 275.93 47.60 228.32 4800.83
3, repeated large 1159.84 298.29 51.54 246.75 4803.51
16 1450.73 597.07 350.58 246.49 4805.21
32 1799.00 950.32 702.84 247.48 4807.77
36 1799.81 950.32 702.81 247.51 4808.27

Allocator arena statistics are sampled at the end of vision encoding; RSS and GPU counters are sampled after response completion. The arena estimate excludes separately mapped allocations; the allocator's mmap total remains constant at 735.45 MiB from the first large request onward. RSS and GPU counters are separate accounting views on this UMA machine and must not be added together.

The evidence identifies the large RSS increase as allocator retention: from request 3 onward, used arena space stays near 247 MiB while free arena space grows. Total arena space stops growing by request 32, consistent with exercising the default 31 HTTP workers. Between requests 32 and 36, RSS increases only 0.81 MiB and used arena space about 0.03 MiB. This is a bounded plateau in the tested run, not evidence of a continuing 22 MiB live-allocation leak per request.

The vision scheduler reservation remains exactly 620.63 MiB throughout the large-image repeats. Total process GPU allocation rises 7.43 MiB (about 0.15%) from the first large request to the last. That small residual is outside the unchanged vision compute-buffer reservation; this validation does not attribute every byte of runtime/backend allocation or prove zero memory growth over an indefinitely long run.

Validation method and artifacts

The production code is unchanged from commit 78fadfe18. The existing PR server and HIP backend libraries are reused with the rebuilt mtmd library from that commit. Loaded-library mappings and the mtmd hash verify the intended library was loaded.

A separate diagnostic shared library forwards scheduler and clip_encode() calls unchanged. It records scheduler event order and durations, and saves embeddings already present in the normal host output buffer. It adds no extra GPU-to-host embedding transfer. Local embedding-file writes cost approximately 1.7-2.0 ms for the large image and are included in HTTP TTFT. The extended memory run also records allocator XML. These are model-backed server integration checks, not backend-op unit tests.

liangliangchang added a commit that referenced this pull request Nov 16, 2026
liangliangchang added a commit that referenced this pull request Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants