Repository navigation
Description
Is your feature request related to a problem? Please describe.
rocm diagnose --model (#407) tells you whether a curated model fits this GPU before you download it. It does that by comparing the recipe's fixed minimum GPU memory with the card's. Most of what people ask about local LLMs on Radeon cards falls outside that:
- The model isn't in the catalog. People pick a model and quant from Hugging Face, and for those
diagnose --modelanswers UNDETERMINED. - The question is about context size, not whether it loads. How much context fits, whether a q8_0 KV cache would help, how to make room for more. Nothing in
diagnose --modelorrocm servetakes context size into account, androcm servehas no option to set it.
Recent threads asking exactly this:
- Double R9700 alternative to Qwen 3.8 27B: which larger models fit without buying more RAM
- Can I reduce vram usage on 7900 xtx?
- Low VRAM models on an AMD Radeon 5700 8GB
- Anyone running Qwen 3.8 27B Q3/Q4 on an RX 9060 XT 16GB?
- llama.cpp use of R9700 for document querying: context limits for document work
- Where to get quants for common models on vllm?: FP8 27B model out of memory on a single R9700
- ollama/ollama#18865: a default context of 16384 fails to load with OOM on a Radeon 890M; the workaround was a smaller
num_ctx
Describe the solution you'd like
Extend rocm diagnose --model rather than adding a new command:
- For a GGUF model outside the catalog, read the size and the tensor metadata (layers, KV heads, head dimension) from the file header. That's enough to estimate memory without downloading the weights.
- Estimate weights + KV cache at a given context size and KV-cache type + compute buffer, against the dedicated VRAM feat(diagnose): answer whether a model will run before downloading it #407 already reads. On APUs that check answers UNDETERMINED today because the GTT pool isn't readable, so APUs would need that first.
- Report the largest context size that fits. When the model doesn't fit at the requested size, say what would make it fit: a q8_0/q4_0 KV cache, a smaller quant, or partial CPU offload.
- Give
rocm servea way to set the context size, so the answer from step 3 can be acted on directly.
Scope: one model on one GPU, as rocm-cli serves today. vLLM can follow; GGUF comes first because the file carries everything the estimate needs.
Describe alternatives you've considered
- Online VRAM calculators: they don't know this machine's free VRAM, APU GTT, or the engine's overhead.
- Adding more models to the curated catalog: it helps for the models listed, but doesn't answer the context-size question.
- Reading the engine's OOM afterwards (the vLLM OOM guidance in feat(vllm): Tackle out of memory errors (EAI-8058) #251/feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) #290): useful, but it comes after a failed load.
Additional context
#407 deliberately reports UNDETERMINED rather than guess for models outside the catalog. An estimate from the GGUF header is still an estimate, so it could be shown under its own label (e.g. "estimated: fits up to 32k context") instead of READY/BLOCKED.
Activity
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model
feat(diagnose): estimate context size and memory for a GGUF model (RO…
tomastola commentedon Nov 9, 2026
#826 delivered most of this, and two parts are still open.
Done in #826:
rocm diagnose --modelnow estimates a GGUF model outside the catalog from the file header alone, without downloading the weights. It works for a local.ggufpath orowner/repo:QUANTon Hugging Face. It sizes weights, KV cache at--context-sizeand--kv-cache-type(f16, q8_0 or q4_0), and a compute buffer against the card's dedicated VRAM. The result is labelled ESTIMATED. It reports the largest context that fits, and when the model does not fit it suggests a smaller context, a quantized KV cache, a smaller quant, or partial CPU offload.Still open:
rocm servehas no option to set the context size, so the answer fromrocm diagnosecannot yet be acted on directly.vLLM models are still not estimated, which was already out of scope for the first step. I'm leaving this open for the two items above and have retitled it to match.
🤖 by agenthub on AMD AgentHub
[-]diagnose --model: take context size into account, and answer for models outside the catalog[/-][+]diagnose/serve: set the context size in rocm serve, and give an APU verdict in diagnose --model[/+]