Follow-up from vllm-project#41834 — filing here for a cleaner trace, as you suggested.
Context: testing your SM12x sparse-MLA selector probe (commit 7224e68) on GLM-5.1 NVFP4. Your probe works — FLASHMLA_SPARSE selects on all 4 GB10 (sm_121). The next blocker is upstream of the sparse runtime: DSA indexer weight loading under NVFP4.
Environment
- 4× DGX Spark (GB10 / sm_121), TP=4, ray executor, verified RoCE.
- Base image
vllm/vllm-openai:nightly-aarch64 @ 42fcb680 (commit 43914dd), with your selector commit 7224e68 cherry-picked (CC gate supports_compute_capability [9,10]→[9,10,12]); overlay engagement verified in-log. NOT the full branch (139 commits ahead) — say the word if the full branch loads weights differently.
- Checkpoint:
0xSero/GLM-5.1-555B-NVFP4 (REAP-pruned 555B, modelopt-style NVFP4).
What happens
Selector clears:
INFO [cuda.py:378] Using FLASHMLA_SPARSE attention backend out of potential backends: ['FLASHMLA_SPARSE']
INFO [selector.py:163] Using FLASH_ATTN MLA prefill backend.
Then weight loading fails before the sparse runtime is reached:
ray.exceptions.RayTaskError(KeyError): ray::RayWorkerProc.initialize_worker()
File ".../vllm/model_executor/models/deepseek_v2.py", line 1739, in load_weights
File ".../vllm/model_executor/models/utils.py", line 368/301/274, in load_weights/_load_module
File ".../vllm/model_executor/models/deepseek_v2.py", line 1448, in load_weights
KeyError: 'layers.0.self_attn.indexer.wk_weights_proj.input_global_scale'
Diagnosis
This NVFP4 checkpoint quantizes the lightning-indexer projections — input_global_scale tensors are present in the checkpoint — while the GLM/DSA model path builds the indexer unquantized, so the scale key has no destination parameter at params_dict[name].
Looks like the indexer modules need quant-config awareness before any NVFP4 GLM checkpoint can exercise the sparse path — either honored exclude/ignore handling for this checkpoint family, or NVFP4-capable indexer linears.
Blocker sequence on SM121
selector gate (your probe) → indexer NVFP4 loading (this issue) → sparse runtime (your anticipated Triton-fallback question — still untested, blocked behind this).
Offer
Full head log (690 lines, selector/backend lines + complete trace) available — can attach or gist. We'll run any follow-up branch same-day on this 4× GB10 setup (single-node not possible — REAP checkpoint is 320 GB vs 128 GB unified/node, so TP=4 only). Can also test a BF16 / unquantized-indexer checkpoint variant if you point us at the expected layout — that would isolate whether the gap is purely the quant-config plumbing.
Follow-up from vllm-project#41834 — filing here for a cleaner trace, as you suggested.
Context: testing your SM12x sparse-MLA selector probe (commit 7224e68) on GLM-5.1 NVFP4. Your probe works — FLASHMLA_SPARSE selects on all 4 GB10 (sm_121). The next blocker is upstream of the sparse runtime: DSA indexer weight loading under NVFP4.
Environment
vllm/vllm-openai:nightly-aarch64@ 42fcb680 (commit 43914dd), with your selector commit 7224e68 cherry-picked (CC gatesupports_compute_capability[9,10]→[9,10,12]); overlay engagement verified in-log. NOT the full branch (139 commits ahead) — say the word if the full branch loads weights differently.0xSero/GLM-5.1-555B-NVFP4(REAP-pruned 555B, modelopt-style NVFP4).What happens
Selector clears:
Then weight loading fails before the sparse runtime is reached:
Diagnosis
This NVFP4 checkpoint quantizes the lightning-indexer projections —
input_global_scaletensors are present in the checkpoint — while the GLM/DSA model path builds the indexer unquantized, so the scale key has no destination parameter atparams_dict[name].Looks like the indexer modules need quant-config awareness before any NVFP4 GLM checkpoint can exercise the sparse path — either honored exclude/ignore handling for this checkpoint family, or NVFP4-capable indexer linears.
Blocker sequence on SM121
selector gate(your probe) → indexer NVFP4 loading (this issue) → sparse runtime (your anticipated Triton-fallback question — still untested, blocked behind this).Offer
Full head log (690 lines, selector/backend lines + complete trace) available — can attach or gist. We'll run any follow-up branch same-day on this 4× GB10 setup (single-node not possible — REAP checkpoint is 320 GB vs 128 GB unified/node, so TP=4 only). Can also test a BF16 / unquantized-indexer checkpoint variant if you point us at the expected layout — that would isolate whether the gap is purely the quant-config plumbing.