[alenizy361]

Overview

• I ran an experiment to run a 27B model (from the hybrid Qwen3.5 series) on a laptop that has no GPU and uses single-channel RAM.
• While running concurrent sequences, perf showed that about 27% of the step time is wasted in get_rows and cpy operations for the GDN layer states, even though the state is already in place.
• I added an in-place state update inside the cache, applied only when all sequences read their own cells; otherwise the previous path is used, with an environment variable to go back to the old behavior for comparison. I also made the zeroing of the state memory lazy via mmap instead of memset, so inactive cells do not consume RAM.
• Results on my machine (using the AVX2 kernels from the PrismML fork): for 16 sequences the speed went from 9.5 to 13.5 tokens/s, for 8 sequences from 10.1 to 13.3 tokens/s, and RAM usage for 16 slots with speculative decoding dropped from 12.8 to 8.4 GB.

Additional information

• This change resolves Feature Request: skip redundant recurrent-state gather when n_rs == 1 && n_seqs == 1 #28841 (the single-sequence case) and handles any number of sequences.
• Verification: a byte-for-byte unit test passes for 48 cases; perplexity is exactly identical with or without the change (even at ub=32); and the md5 of greedy generation matches exactly.
• Fork version: llama: in-place recurrent (GDN) state update for multi-sequence decode + lazily zeroed RS buffer PrismML-Eng/llama.cpp#207. The new path is for CPU only; the old path is kept when n_rs_seq > 0.
• Note: I wrote this description in Arabic and translated it to English.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — the code was written by Claude Code under my supervision; I defined the problem and the goal, ran the tests and measurements on my computer, and reviewed the results.

[ggml-gh-bot]

Hi @alenizy361, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.