Repository navigation
sanity - #1
Closed
figkcoci-debug wants to merge 120 commits into
Closed
sanity#1figkcoci-debug wants to merge 120 commits into
figkcoci-debug wants to merge 120 commits into
Conversation
…g#30416) The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels for head size 512 require a multiple of FATTN_KQ_STRIDE, so the op falls back to the CPU for most lengths, and the -fa auto probe (1 token) disables flash attention for such models. Pad the mask like llama_kv_cache::get_n_kv and pad K and V with zeros up to the KV dimension of the mask. clef builds its own causal mask and pads it the same way.
* model: fix vllm d3 multi seq case * add TODO
…org#30245) * tests: cover weighted scalar broadcasts Assisted-by: OpenCode * ggml : support extended scalar DMA binary ops Assisted-by: OpenCode * hex-binary: remove all non-dma paths * hex-binary: add support for non-contig strides and cleanup dma preable a bit * hex-binary: cleanup the gating and simplify dma retry * hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op * tests: cover weighted repeat broadcast Assisted-by: OpenCode * hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time) * hex-add-id: add dma support for src2 (idx) --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
…kernels (ggml-org#30198) * models: remove redundant ggml_cont calls now supported by CUDA unary kernels * models: restore cont_2d after sigmoid for broadcast mul, clean dflash cont * Update src/models/qwen3next.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * models: restore cont_2d before sigmoid in qwen3next --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* ggml-zdnn: initial prototype of q8_0 matmul Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update zdnn docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
* convert : fix automatic LoRA output type * convert : remove LoRA test and CI changes
* cuda : warnings cleanup Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Update ggml/src/ggml-cuda/common.cuh Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* cuda : fix warnings from dead breaks Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Remove useless braces Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>
cont of ggml-org#29622, where we forgot to remove token assert
…upCount (ggml-org#29533) * vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount * vulkan : fix narrowing in chunked matmul dispatch * vulkan : zero fill the A side of mul_mmq for k steps past the end The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32. Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side. Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape. Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com> * vulkan : remove redundant matmul tile adjustment --------- Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG * ggml-cuda: cache env var lookups in static const instead of re-reading per call * ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int * ggml-cuda: consolidate env flag reads into single static accessors * ggml-cuda : drop env helper comments Assisted-by: pi:llama.cpp/Qwen3.8-27B * cont : remove NDEBUG gates for CUDA graph logs
* ggml-cpu : add AVX2 dot product for Q2_0 * fix comment [no ci] --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
…l-org#29741) * vulkan: allow multiple devices to access gpu-visible host memory * clean up erroneous newlines * add Windows support * handle nullptr * fix enable condition
…-org#30023) * tests : consolidate the context state tests into test-llama-context test-save-load-state, test-recurrent-state-rollback and test-state-restore-fragmented check the same object from three places: the KV cache and the recurrent state of a llama_context, saved, loaded, copied, rolled back. Each had its own main, its own loop over the models, its own copy of the token and the comparison helpers, and a run of all three loaded the same models several times. They are one binary now, and the case is the unit of the test: tests/llama-context/test-case.h the interface a case implements tests/llama-context/cases-kv.cpp the cases of the KV cache tests/llama-context/cases-rs.cpp the cases of the recurrent state tests/llama-context/test-util.cpp tokens, generation, comparison tests/llama-context/test-runner.cpp the models of a case, on threads tests/llama-context/test-log.cpp the log file of a model A run is a table: a row per model, a column per case, a cell holds the verdict of that pair. The models load on several threads (-j, --threads, min(cores, 4) by default), the table is printed when it is complete and the log of a model that failed is printed after it, in the order of the rows. A case returns a verdict and the reason for it, and the runner is the only one who reports - the test does not use tests/testing.h, a run over 130 models is a matrix of verdicts and not a tree of subtests. Staging no longer travels through the environment: llama_model_set_attn_rot replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one thread used to change for the models the other threads were loading, and llama_memory_get_buffers is added for the cases that copy the buffers of a memory. common_log_set_stdo keeps the console quiet while the table is being built. ggml_aligned_malloc allocates with posix_memalign when the thread sanitizer is on. On macOS it allocates with the Mach VM, which the sanitizer does not intercept, so it never sees a page range change hands and reports the first write into recycled memory as a race - 30 of them in this test, and every one of them false. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : run single-threaded by default * cont : try to build the tests on all platforms * cont : trim down some body fat * cont : fix * cont : fix the fix
* graph: skip the LM head for pooled embeddings A context that extracts pooled embeddings outputs one vector per sequence, but every model still expanded its LM head, so the graph computed n_vocab logits per token and the context kept a host buffer for them. The models now only declare t_logits, and set_outputs() expands it unless the context extracts embeddings with a pooling type other than NONE. Since the toggle changes the graph shape for a pooled context, set_embeddings() schedules a re-reserve in that case. * graph: drop the redundant t_embd expand Address review from @CISC: with embeddings on, build_pooling() already expands its output for every pooling type, and a decoder graph without embeddings does not read t_embd, so set_outputs() only expands t_logits. * graph: add build_logits() for the LM head output Address review from @ggerganov: the LM head is expanded by a dedicated build_logits(), called from llama_model::build_graph() like build_pooling(), and set_outputs() only marks the outputs again. * graph: call build_logits() before build_sampling() Address review from @ggerganov: build_sampling() reads t_logits, so the LM head is built first. A pooled context then skips the sampling layers too, since it has no logits to sample from. * nit
Assisted-by: Codex Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
* llama : do not map foreign ranges into a backend buffer A buffer created from a host pointer pins everything that it wraps. When several contexts share the same file (lazy tensors, MoE tensors kept on the CPU with -cmoe), the single buffer that spans the first and the last tensor of a context also wraps the ranges owned by the other contexts, pinning data that the backend never reads. For models with per-layer token embeddings this pins the whole lazy tensor into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the point of reading it lazily. Instead, compute the file ranges of each context and subtract the ranges owned by the other contexts, then create one buffer per range. Weights that are mapped in more than one context (e.g. token_embd.weight) keep being part of the range of every context that uses them. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * llama : enable prefetch on macOS The prefetch hint of the tensors marked as prefetchable is a no-op on macOS, so their pages are faulted in one by one while the graph runs. madvise (MADV_WILLNEED) behaves as expected here, so add the platform to the guard. With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to 735 / 749 / 788 t/s for pp512 / pp2048 / pp4096. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * llama : use a plain buffer list per context Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
The no-cache attention pads K, V and the mask to 256, but T5 also adds a relative position bias of [n_tokens, n_tokens], so the add with KQ fails unless n_tokens divides the padded width. Pad the bias with zeros up to the KV dimension of the mask, like K and V.
* hexagon: add configurable HTP DCVS/HMX power mode * hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup) * hexagon: add --hex-power-mode flag to run.py Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following the existing --hex-nhvx pattern. The env var must be set before the process touches the Hexagon backend registry (see comment in ggml_hexagon_init), so this is exposed as a runner flag rather than a common/arg.cpp CLI option. * hexagon: keep HTP awake for DCVS power profiles * Apply suggestion from @max-krasnyansky --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
…30453) * meta: recreate all subgraphs when the arena is reallocated * chore: revert added test cases
…0119) When the window is larger than a subgroup, use a whole subgroup to compute each output.
Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27 Change-Id: I406a6606c3fea51f8cd32331c1ed71466845d363 Change-Id: Ib4aa4c9814c010f9cf084783cc80612e364f9a27 Signed-off-by: Charles Xu <charles.xu@arm.com>
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
Assisted-by: Codex
* pplx-decider-text-only convert : support text-only checkpoints in pplx-decider * Apply suggestions from code review * convert : remove unused import
…e x in registers (ggml-org#29720) * CUDA: HIP: rms_norm: load float4 at a time and store x in registers * Add template paramter comments * use ggml_cuda_is_aligned for src * nits * also use ggml_cuda_is_aligned for src in fused variants * fix nits
…0630) * gguf-py : add numpy Q1_0 and Q2_0 de/quantization support * cont: added q1_0 / q2_0 to the test list removing the small test Assisted-by: Claude Opus 5.5 <noreply@anthropic.com> --------- Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test