Repository navigation
Sync master with upstream (2026-11-12) - #202
Merged
Merged
Conversation
* hexagon: initial support for LIGHTNING_INDEXER op * hex-lght-idxr: hmx for prefill * hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute * hex-lght-idxr: pipelined hmx path for decode and prefill * hex-lid: add support for K (f32) * hex-lid: add support for K (f32) * hex-lid: add support for q4_0 * hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills * hex-lid: host/kernel kparams alighnment and remove sw devices in kernel * hex-lit: update dma pipeline to use canonical dbl buffering in all paths * hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths * hex-lit: add support for dims collapse * hex-lit: make kparams constistent with other ops --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* common : fix -hf tag matching inside longer quant names ":F16" picked a BF16 file when the repo had both, since "BF16" contains "F16". The same applied to ":Q4_XS" vs "IQ4_XS". The tag now has to start at a token boundary in the path. Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB) * common : accept underscore as -hf tag boundary Some repos separate the quant with an underscore, e.g. stepfun-ai/Step-3.5-Flash-GGUF-Q4_K_S ships step3p5_flash_Q4_K_S-00001-of-00012.gguf, which :Q4_K_S no longer matched. Add _ to the token boundary and pin the tag and default lookup cases. --------- Co-authored-by: Pascal <admin@serveurperso.com>
…g#31144) The gguf writer staged each tensor into a temporary vector before writing it. For host-buffer tensors the data pointer is directly writable, so skip the staging copy and write it straight to the file. This avoids the per-tensor allocation, zero-init and memcpy. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* ggml: bound views by the bytes their strides reach A view was checked against its source with the size of a contiguous tensor of the same shape, computed before its strides were set. Views whose rows overlap failed that check although they stay inside their source, and views with wider strides passed it while reaching past the end. The view functions now pass their strides to the tensor constructor, which checks the bytes they actually reach, like ggml_nbytes. The conv windows of delta_net are created with all their rows at once instead of being widened after the check. * ggml: assert that a strided view starts from contiguous elements The view functions set nb[0] to the type size, so the source must have contiguous elements along dim 0 for the view to read them.
* opencl: improve q2_0 decoding and small f32 matmuls Assisted-by: Codex * opencl: optimize small f32 matmuls on Adreno 8xx Assisted-by: Codex
Co-authored-by: Your Name <you@example.com>
The RISC-V vector intrinsic specification requires __riscv_v_intrinsic to be defined when the compiler supports the API, even when the vector extension is not selected. Clang therefore defines it for rv64gc, causing ggml to enter RVV branches where vector types are rejected because the required ISA extensions are absent. Use __riscv_v for optional RVV implementations; this macro represents the selected target ISA. Retain __riscv_v_intrinsic checks in SpaceMIT sources, where they validate mandatory compiler API support rather than select optional vector code. Closes: ggml-org/ggml#1535
…ant (ggml-org#31635) * llama : follow ctx_other devices and flash attention in Gemma4 assistant * llama : include <algorithm> for std::any_of
* ggml : bump version to 0.29.0 * authors : update
* llama.cpp : bump version to 0.9.0 * authors : update
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…ml-org#31631) * gguf-py : validate every tokenizer merge pair, not only the first The transformers 4.45 pair format was detected by checking merges[0], then every element was converted unchecked: a non-pair element raised a bare TypeError, and pairs with the wrong arity or non-string parts were silently mangled into invalid merges. Validate each element (2-string list) up front and raise the existing "Unknown tokenizer merges format" error for anything else. Assisted-by: Devin * gguf-py : remove test_vocab.py per review
* add oneDNN in build image * Update .devops/intel.Dockerfile Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
… bindings in memset/clear (ggml-org#31348) * Fix get_max_size and improve the memset shader * Fix buffer_memset to handle the buffer size more than max binding size.
…onversion retries (ggml-org#31468) * fix(ui): show extracted text in PDF attachment preview Assisted-by: Claude (Opus 5.5 Medium) * feat(ui): show page thumbnails for PDF attachments sent as images Assisted-by: Claude (Opus 5.5 Medium) * ui: stop retrying PDF preview conversion after it fails A failed PDF-to-image conversion or text extraction reset its loading flag, which re-ran the effect and retried forever. Remember the failed item for each view and only retry when the item changes, so the error state stays on screen. Add a PdfConversionError story that feeds a broken PDF. Assisted-by: Claude (Opus 5.5 Medium) * ui: keep the text of PDFs sent as images and tidy the PDF preview Store the extracted text of a PDF sent as images in its content, so the Text view reads it instead of parsing the PDF again; only messages stored with the old placeholder extract it. The model still receives the images only. Guard retries on the error state, as the item remounts per id, skip an empty base64 payload, and share the status markup between views. Assisted-by: Claude (Opus 5.5 Medium)
* model, mtmd: drama-3 tts support * clean up * mtmd: drama3 helper stops at the codebook tower limit --------- Co-authored-by: Pascal <admin@serveurperso.com>
…gml-org#31545) * ui : share directory resolution and the cwd row Move the lookup of a browser-picked folder to ToolsService.resolveDirectoryPath, let the working directory picker run without its chip, and split the 'Set working directory to' row into its own component, so other views can pick and show a folder the same way. Assisted by: Claude (Opus 5.5 Medium) * ui : open an item's actions on right-click Sidebar conversations and models manager rows open the same actions as their "more actions" menu on right-click. Both menus render one shared list of actions, so they cannot drift apart. Phones keep the browser's long-press; the dropdown's drawer already holds the actions there. An action can render a flyout submenu in place of its row, through the kit of the menu it is in, and close the menu after a pick. Adds the shadcn-svelte context-menu component, mirroring the dropdown-menu one. Assisted by: Claude (Opus 5.5 Medium) * ui : close conversation tabs from a right-click menu A tab's right-click menu closes it, the other tabs or the tabs to its right, and stops a response the tab is generating. Closing tabs that include the active one keeps the tab the menu was opened on. Assisted by: Claude (Opus 5.5 Medium) * ui : open a message's actions on right-click User and assistant messages open the same actions as their action icons on right-click. Selected text, links, media, code and fields keep the browser's own menu, so copying them works as before. The fork dialog opens from the actions context, like the delete one, so both the icon and the menu can open it. Assisted by: Claude (Opus 5.5 Medium) * ui : scope right-click menu shortcuts and selection to their item Show the conversation shortcuts only on the active row, as they act on the active conversation, and keep the browser menu only for a selection inside the message. Name the close-tab shortcut, and drop the unused chip toggle, menu submenus and shared cwd row. Assisted-by: Claude (Opus 5.5 Medium)
* opencl: use more workgroups for q4_K gate up + GLU Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> * opencl: handle FA for GQA 2 and 6 with head size 256 Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> * opencl: handle FA for GQA 16 with head size of 128, 512 Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> * opencl: handle FA for GQA 2 with head size 512 Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
…ons (ggml-org#31632) * server : realtime: send an error event instead of crashing on exceptions * server : route websocket handlers through ex_wrapper, remove the realtime test * server : realtime: keep the previous tools when the chat template rejects new ones * server : realtime: reject tool schemas the grammar cannot convert, drop ws_ex_wrapper * server : realtime: clarify the tool schema check comment
…org#30541) * llama : fix on-device state copy sizes for quantized KV caches The on-device sequence state copies (LLAMA_STATE_SEQ_FLAGS_ON_DEVICE) computed their element count as size / ggml_element_size(). For a quantized type such as q8_0 or q4_0 that is the number of blocks, not elements, so only part of each range was copied: the restored state was wrong, or the copy crashed. Compute the count as size / ggml_type_size() * ggml_blck_size() for the save, the restore and the copy between differently chunked ranges, and assert that each range covers whole blocks. * llama : drop the one-line comments * llama : drop the redundant cast in the state copy size
* cuda: fuse NVFP4 gate/up and GLU in MMQ * cuda: honor the requested src1 precision in fused gate/up MMQ * cuda: share NVFP4 scale checks, skip unused fused FP4 kernels * cuda: test NVFP4 fused MMQ at W4A8 without scales and with batches, clean up naming * cuda: check gate_scale before loading the fused NVFP4 gate scale
* opencl: add DSV4_HC_PRE and DSV4_HC_POST Co-authored-by: Shafei Qi <shaoqi@qti.qualcomm.com> * opencl: improve dsv4_hc_post Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Li He <lih@qti.qualcomm.com>
ggml-org#31694) * opencl: use the small-N f32 GEMV by default on X2E Assisted-by: Claude Opus 5.5 Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> * opencl: use the q5_K multi-column verify GEMV by default on X2E Assisted-by: Claude Opus 5.5 Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com> --------- Co-authored-by: Li He <lih@qti.qualcomm.com>
This commit adds nvfp4 scales to qwen2's build_ffn call which are currently passed as NULL. The motivation for this is that a model converted to NVFP4 will currently crash because the scales are loaded but never applied in the FFN: ```console llama-completion: llama.cpp/src/llama-sampler.cpp:1218: void llama_sampler_dist_apply(llama_sampler*, llama_token_data_array*): Assertion `found' failed. ``` The attention already applies its scales but ffn is missing them.
…1688) The first command buffer takes the first 10% of the graph nodes and the rest is split evenly between the extra command buffers. A fusion cannot span two command buffers, so a boundary that falls inside a fusable group runs that group unfused, and the output then depends on the node count of the graph. Move each boundary to the end of the fusable group it falls into, so the encoders fuse the same nodes as with a single command buffer.
* tests: fix CI flakes of TOPK_MOE caused by near ties With uniform random logits, two experts sometimes land within one f32 ulp of each other at a routing boundary, mostly with sqrt_softplus gating whose output range is narrow. A backend whose gating differs by a few ulps from the CPU then picks the other expert and the test fails. Draw each row of logits again until the group selection and the final top-k both have a relative margin of 1e-4, computed on the host in double precision. * tests: build the TOPK_MOE logits by inverting the gating function Draw the selection scores (probs plus bias) of each row as a random permutation of a grid, move the experts of the best groups up by half a step, and set the logits to the inverse of the gating function at these scores. Every expert and group decision is then one grid step apart on all backends, and the host no longer recomputes the routing. The softmax probs are built to sum to one so that the softmax keeps the scores, and the views of the logits are not initialized again after the logits are set.
…-org#31915) * chat : build a schema-shaped PEG parser for Gemma 4 tool arguments * chat : enforce only the key structure of Gemma 4 tool arguments * chat : remove old gemma4 workarounds * cont : add node.as<>() to simplify static casts * cont : add depth_guard and use numerical rule names * cont : clean up * cont : extract gemma 4 parsers * cont : clean up * cont : clean up * cont : remove dynamic cast
…g#32112) npm 11.20 refuses git dependencies by default (EALLOWGIT), so npm ci failed on fresh installs. Pin llama-brand to the same commit through an HTTPS tarball, which also adds an integrity hash to the lockfile. Assisted-by: Claude (Opus 5.5 Medium)
* ui: add search_chats tool * add from_timestamp / to_timestamp * ui: search_chats review fixes - keep search_chats disabled on import, copy tool policy on fork - stream messages without attachment payloads, one transaction for time ranges - keep the output within budget, cut whole messages from the older end - reject impossible date-times, quote titles and attachment names - typed readOnly tables, move localIsoString to formatters as formatLocalIso Assisted-by: Claude (Opus 5.5 Medium) --------- Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
…31929) The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0. Assisted-by: Codex Signed-off-by: gitGurugu <2049447146@qq.com> Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
…org#32100) * ggml-backend : prefetch only inputs computed by earlier splits The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write. * llama : skip the MoE cache when its device cannot run the cached mul_mat_id Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
… checkpoints (ggml-org#26048) * ggml-cuda: Apply NVFP4 weight scale in MMQ write-back * ggml-cuda: Fuse mul_mat(_id) + scale into MMQ * ggml-cuda: Tests for the fused scale * ggml-cuda: reuse existing graph to test fusion in mmq epiloque * ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency * ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions * ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
Assisted-by: Claude Fable 5.2
* vulkan : Intel customized FA shader for prefill usage * vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path * vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader * vulkan : Intel FA shader and host refinement, with accuracy issues fixes * vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader * vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases * vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing. * vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support * vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support. * vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops * vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants. * vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling. * vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite feature check.
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel. Assisted-by: Codex Signed-off-by: gitGurugu <2049447146@qq.com> Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
…g#32130) The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their input with a temporal merge that assumes a 2x2 patch grid, and the Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the GGUF reaches a GGML_ASSERT at the first image request and aborts the server. Reject a spatial_merge_size other than 2 when loading the hparams.
…#32129) * mtmd: require the MLP projector and patch embedding tensors The loader marks the patch embedding and the LLaVA projector tensors as optional, while the graphs and n_mmproj_embd() dereference them unconditionally. A GGUF without them crashes on a null tensor during the warmup. Load the patch embedding as required for the vision encoders that start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1, mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3. The graph no longer tests mm.2 and the yasa2 assert on the patch embedding is covered by the loader. * mtmd: keep the Yi-type llava TODO The Yi-type variant is still detected by the presence of mm.3, the TODO on that detection stays.
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in real models, the projection is square. A non square projection passes the text model n_embd check and only aborts at the encode, on the output buffer size check. Return ne[1], so the width matches the graph output.
HunyuanVL resamples its position embedding on the host into a float buffer sized for F32, then reads ggml_nbytes() of the tensor into it. The graph never uses the tensor, so any type loads, and an F64 tensor overflows the buffer at the first image request. Require the position embedding to be present and F32 when loading the HunyuanVL tensors.
…g#32133) With --sleep-idle-seconds, a handler passes wait_until_no_sleep() while the server is awake, then keeps using the server state, for example to parse the body and tokenize. If the idle timer expires in that window, the main loop enters sleep and the callbacks destroy the model: the handler reads freed vocab memory, and a task posted after that stays in the queue with nobody awake to process it. Count the requests that passed wait_until_no_sleep() and are not done yet, and do not enter sleep while the count is positive. The response generator releases its count in its destructor, so every return path of a handler gives it back. Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
…port (ggml-org#32113) * server: explicit opt-in tools via non-loopback addr, check host port * fix ci
* CUDA: few-row MMQ for 2-24 src1 columns New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes. Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on. * nit fixes * CUDA-only guard for few-row MMQ, weights flag in test_mul_mat * few-row quantize: DS4 sum from the quantized values, as in ggml-org#31247 * few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build * few-row MMQ: dispatch order MMVQ, few-row, MMQ
* qwen3.5: merge tensors on convert also * Remove conversion test for qwen3.5
…org#32025) * server : cancel MCP tool calls on client disconnect or timeout * server : shorten MCP cancellation tests and fix a type-check error
* hexagon: add q5_0 and q5_1 support * hex-mm: dedicated q5_0 kernels
* ggml : prune deprecated API * cont : bump RPC version * cont : minor * ggml : remove outdated documentation from ggml.h The 172-line comment block at the top of the header has not been updated in a substantive way since the backend v2 split (Nov 2023) and now documents APIs that no longer exist: - `ggml_set_f32()`, `ggml_get_f32_1d()`, `ggml_set_f32_1d()` were removed - `ggml_conv_1d_1s()`, `ggml_conv_1d_2s()` ops no longer exist - `ggml_set_param()` no longer takes a context argument - `ggml_graph_compute_with_ctx()` moved to the CPU backend header - the optimization API now lives in `ggml-opt.h` The block also ended with 5 empty `TODO` sections. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * ggml : remove DIAG_MASK_INF and DIAG_MASK_ZERO (ggml-org#31946) * rpc : bump RPC protocol major version The tensor op enum is sent over the wire, so removing GGML_OP_ADD1, GGML_OP_DIAG_MASK_INF and GGML_OP_DIAG_MASK_ZERO shifts the value of every op that follows them and peers built before and after this change would misinterpret each other's ops. The handshake compares the major version for exact equality and ignores the patch version, so bumping the major version is what rejects incompatible peers instead of silently misbehaving. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) --------- Co-authored-by: losewayy <1134285612@qq.com>
* llama : remove deprecated `llama_set_warmup` Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) * llama : remove remaining deprecated API Drop the APIs marked DEPRECATED in llama.h, the deprecated llama_context_params::defrag_thold field and the --defrag-thold option. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) * fit-params : remove stale `llama-ext.h` include Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) * llama : document `llama_model_dflash_selector_top_k` Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) * llama : promote the NextN / speculative API from `llama-ext.h` Move the NextN, layer input, embd row and state row APIs to `llama.h` and rename `llama_set_embd_state_src` to `llama_set_ctx_state_src`, with the internal `llama_context::set_state_src` following the same name. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB) * llama : polish the NextN API promotion Move the NextN draft model queries next to the Adapters section, rename the `src` parameters of `llama_set_ctx_state_src` and `llama_context::set_ctx_state_src` to `ctx_src`, and drop the now-redundant `llama-ext.h` includes from `common/speculative.cpp` and `tools/imatrix`. Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
Upstream: 8ca5a80 Preserve the fork README and GGML_HIP_UNSAFE_MATH default OFF. Assisted-by: Codex
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge upstream
ggml-org/llama.cpp:masterthrough8ca5a8008b57(llama : prune (#30565)), bringing in 285 commits since the current fork master.Preserve the fork README and
GGML_HIP_UNSAFE_MATHdefault OFF. The README was the only merge conflict; it was resolved by retaining the fork version. The merged tree otherwise matches upstream except for the existing HIP unsafe-math option in two CMake files.Validation:
llama-cli,llama-server, and selected test targets passed.Pending before merge: HIP build, GPU smoke tests, and performance regression testing.
AI assistance: Codex prepared the merge, resolved the README conflict, ran validation, and created this draft PR.