Skip to content

diagnose/serve: set the context size in rocm serve, and give an APU verdict in diagnose --model #781

Open

Description

@tomastola

Is your feature request related to a problem? Please describe.

rocm diagnose --model (#407) tells you whether a curated model fits this GPU before you download it. It does that by comparing the recipe's fixed minimum GPU memory with the card's. Most of what people ask about local LLMs on Radeon cards falls outside that:

  • The model isn't in the catalog. People pick a model and quant from Hugging Face, and for those diagnose --model answers UNDETERMINED.
  • The question is about context size, not whether it loads. How much context fits, whether a q8_0 KV cache would help, how to make room for more. Nothing in diagnose --model or rocm serve takes context size into account, and rocm serve has no option to set it.

Recent threads asking exactly this:

Describe the solution you'd like

Extend rocm diagnose --model rather than adding a new command:

  1. For a GGUF model outside the catalog, read the size and the tensor metadata (layers, KV heads, head dimension) from the file header. That's enough to estimate memory without downloading the weights.
  2. Estimate weights + KV cache at a given context size and KV-cache type + compute buffer, against the dedicated VRAM feat(diagnose): answer whether a model will run before downloading it #407 already reads. On APUs that check answers UNDETERMINED today because the GTT pool isn't readable, so APUs would need that first.
  3. Report the largest context size that fits. When the model doesn't fit at the requested size, say what would make it fit: a q8_0/q4_0 KV cache, a smaller quant, or partial CPU offload.
  4. Give rocm serve a way to set the context size, so the answer from step 3 can be acted on directly.

Scope: one model on one GPU, as rocm-cli serves today. vLLM can follow; GGUF comes first because the file carries everything the estimate needs.

Describe alternatives you've considered

Additional context

#407 deliberately reports UNDETERMINED rather than guess for models outside the catalog. An estimate from the GGUF header is still an estimate, so it could be shown under its own label (e.g. "estimated: fits up to 32k context") instead of READY/BLOCKED.

Activity

  1. added 13 commits that reference this issue on Nov 3, 2026
    1872eb7
    5b33875
    4b6124c
    578985a
    e016fca
    f15cb47
    631d280
    cf4457c
    18a616d
    f86e2a0
    fd50501
    94c0b67
    06fd7e4
  2. tomastola commented on Nov 9, 2026

    @tomastola
    CollaboratorAuthor

    #826 delivered most of this, and two parts are still open.

    Done in #826: rocm diagnose --model now estimates a GGUF model outside the catalog from the file header alone, without downloading the weights. It works for a local .gguf path or owner/repo:QUANT on Hugging Face. It sizes weights, KV cache at --context-size and --kv-cache-type (f16, q8_0 or q4_0), and a compute buffer against the card's dedicated VRAM. The result is labelled ESTIMATED. It reports the largest context that fits, and when the model does not fit it suggests a smaller context, a quantized KV cache, a smaller quant, or partial CPU offload.

    Still open:

    1. rocm serve has no option to set the context size, so the answer from rocm diagnose cannot yet be acted on directly.
    2. On APUs the estimate gives figures but no fits / does-not-fit verdict, because the memory pool the engine allocates from (GTT) cannot be read yet.

    vLLM models are still not estimated, which was already out of scope for the first step. I'm leaving this open for the two items above and have retitled it to match.


    🤖 by agenthub on AMD AgentHub

  3. changed the title [-]diagnose --model: take context size into account, and answer for models outside the catalog[/-] [+]diagnose/serve: set the context size in rocm serve, and give an APU verdict in diagnose --model[/+] on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions