Repository navigation
mtmd: batch Qwen2.5-VL windows and avoid extra copies - #192
Conversation
Group equal-sized vision windows, size graph metadata for the resulting groups, and skip the unused dense mask. Convert K/V before splitting and fuse eligible attention output writes into the joined buffer. Include runtime controls, focused backend tests, and fresh Qwen2.5-VL-3B validation against gfx11. Assisted-by: Codex
Assisted-by: Codex
Account for the KV_rows_ptr argument added in 1b6935b. Update only the 20 existing kernel specializations reported by the gfx908 check; keep the spill threshold unchanged. Assisted-by: Codex
Prepare graph capacity before the warmup check and clear allocation state when replacing the scheduler, so a larger image reserves the new scheduler before graph allocation. Assisted-by: Codex
jimw567
left a comment
There was a problem hiding this comment.
Approved. Please follow up with additional test after the merge.
PR #192: long-lived server small-to-large validationThe requested small -> large -> same-large sequence completes in one llama-server process. The two large requests return identical text and identical vision embeddings. Scheduler replacement adds a small, measurable first-use cost, approximately 14-17 ms (2-3% of HTTP TTFT). The rebuilt scheduler is reserved before graph allocation and its compute-buffer reservation stays constant on subsequent requests. There is a memory qualification: total process RSS is not flat during the first requests. A longer run identifies most of that increase as freed memory retained in allocator arenas as the HTTP worker pool is exercised. RSS then plateaus. The scheduler itself does not accumulate compute buffers. Model and request conditions
The preprocessed image dimensions match the uploaded dimensions. The tall image is within the image budget but requires more rows of vision windows, growing graph capacity from 24,576 to 32,768 nodes. The server PID stays unchanged throughout each session. Requested sequencePrimary session: PID 3168880,
The first small request includes cold execution initialization. Its timing is not a steady-state small-image benchmark. Requests 2 and 3 have:
Language token log probabilities are not bitwise identical; the maximum absolute difference is 0.0467324. Some additional repeats vary wording, such as First-large-request costThe first large request is 13.82 ms slower than the following identical request in HTTP TTFT, or 2.15%. Its vision encode is 12.89 ms slower.
Creation, teardown, and reservation sum to 14.67 ms. These measured components are not an exact subtraction of complete request times, which include ordinary execution variation. The replacement scheduler already has 650,776,448 compute-buffer bytes (620.63 MiB across GPU and CPU) before A second server session reproduces the sequence: first large TTFT 660.81 ms, repeated large 644.21 ms, a 16.61 ms / 2.58% difference. Scheduler creation, teardown, and reservation total 14.46 ms in that session. This is a small one-time cost, not zero allocation overhead. Memory investigationThe initial session includes three extra large requests. Vision compute-buffer bytes remain exactly constant, but host RSS initially increases by roughly 22 MiB per request. To distinguish retained allocator memory from accumulating live allocations, a second session runs one small image followed by 35 large requests and records Second session: PID 3173552,
Allocator arena statistics are sampled at the end of vision encoding; RSS and GPU counters are sampled after response completion. The arena estimate excludes separately mapped allocations; the allocator's mmap total remains constant at 735.45 MiB from the first large request onward. RSS and GPU counters are separate accounting views on this UMA machine and must not be added together. The evidence identifies the large RSS increase as allocator retention: from request 3 onward, used arena space stays near 247 MiB while free arena space grows. Total arena space stops growing by request 32, consistent with exercising the default 31 HTTP workers. Between requests 32 and 36, RSS increases only 0.81 MiB and used arena space about 0.03 MiB. This is a bounded plateau in the tested run, not evidence of a continuing 22 MiB live-allocation leak per request. The vision scheduler reservation remains exactly 620.63 MiB throughout the large-image repeats. Total process GPU allocation rises 7.43 MiB (about 0.15%) from the first large request to the last. That small residual is outside the unchanged vision compute-buffer reservation; this validation does not attribute every byte of runtime/backend allocation or prove zero memory growth over an indefinitely long run. Validation method and artifactsThe production code is unchanged from commit A separate diagnostic shared library forwards scheduler and |
Qwen2.5-VL local-attention layers currently process all image patches with a dense window mask. This change batches consecutive equal-sized windows as independent attention problems, handles partial edge windows, and avoids redundant K/V and output copies. On Qwen2.5-VL-3B-Instruct on Strix Halo, median end-to-end TTFT drops from 2158.59 ms to 974.06 ms: 54.88% lower, or 2.216x faster.
Implementation
FLASH_ATTN_EXT -> RESHAPE -> SEToperations write directly into the joined output buffer. Retain checks for consumers, aliasing, alignment, destination bounds, and extra output-buffer storage.The implementation reuses existing attention kernels. It does not change language GEMM, MMDQ, WMMA dispatch, or language-model fusions.
Performance
Fresh measurements collected on November 10, 2026, comparing
gfx11atf70aba1cb60547b13a6d2f1ef70c158dd59ca51ewith the implementation in22007a7aeda2e6103723ea5d4625db284c3f66cc.Conditions were identical on both builds:
xconucstrhalo06.llama-server, streamed HTTP TTFT to the first nonempty content token. Loading and model staging are outside the timed interval.One candidate request took 2059.42 ms and remains included. Candidate process medians were 972.41 and 974.41 ms; baseline process medians were 2156.07 and 2161.65 ms. The retained comparison uses matching model/request hashes, prompt positions, and effective workload settings, with no detected foreign-job overlap or cached prompt tokens.
These numbers measure the complete package. They do not establish an independent gain for either companion optimization.
Runtime controls
Each switch defaults to enabled. Set it to
0before starting a new process to select its fallback.MTMD_QWEN25VL_WINDOW_BATCHMTMD_QWEN25VL_KV_CAST_EARLYGGML_CUDA_FUSE_ATTN_SETValidation
test-backend-ops test -b ROCm0 -o FLASH_ATTN_EXT_SET: 18/18 passed with fusion enabled and 18/18 disabled, against the CPU reference. Cases cover small/full/partial attention lengths, multiple batch entries, an additional consumer, and out-of-place SET fallbacks.These checks establish numerical agreement for the tested operators and request; they are not a model-wide accuracy benchmark or validation of other GPU backends.
HIP quality check maintenance
The gfx942 build with warnings treated as errors passed. The gfx908 spill check flagged 20 kernels whose exact names were already in the exception list before the
KV_rows_ptrparameter was added in1b6935b22. Refresh those 20 symbol names to match the existing signatures. The affected kernel sources are unchanged from the gfx11 base; the spill threshold and the set of excepted kernel specializations remain unchanged.The HIP quality-check rerun passed on
66358db15, including the gfx942 build with warnings treated as errors and the gfx908 spill check. Local checker validation also confirms that unlisted spilling kernels and unlisted attention shapes still fail.Reproduction and raw evidence
Raw artifacts are retained on the local filesystem at:
This directory contains the frozen request, source patch, binary/model hashes, exact commands, per-request timings, model outputs, and cleanup records.
analysis.json,source_state.json, andbuilds/candidate/manifest.jsonsummarize the run and build evidence. Exact benchmark commands and effective settings are recorded in each run'smetadata.json;run_verification.pyis the driver.The standalone validation document has been moved into this PR description.
Requirements