Launch models with their own engine image and flags (Nemotron 3.5 Lightning, Qwen3.8-27B)

[webdevtodayjason]

Problem

Two models we validated on the GB10 fleet could not be launched by AInode at all, only by hand-rolled docker run commands. That meant the two fastest and most capable models on the cluster were invisible to the dashboard, missing from /v1/models, and unreachable through the master node's federated endpoint. If a node rebooted they were simply gone.

Three things blocked them:

• _build_vllm_serve_args emitted a fixed flag set with no passthrough, so recipes needing speculative decoding, MoE/mamba backends, or reasoning and tool-call parsers could not be expressed.
• The engine image was fleet-global via $NVIDIA_VLLM_IMAGE. Both models need vLLM 0.27.1 while the fleet default is a 0.17 build.
• --enforce-eager and the NVFP4 MARLIN env were applied unconditionally. They are workarounds for a FlashInfer bug in the 0.17 build, and on 0.27.1 they only disable CUDA graphs and cost throughput.

Two related failure modes made every launch problem undebuggable: engines ran with docker run --rm -d, so one that died during startup deleted itself and left no logs and no corpse (about 18 self-erased containers were found on spark-4), and start_solo() returned success as soon as the docker CLI forked, so a crashed engine still registered as a live instance.

Fix

NodeConfig gains extra_vllm_args and engine_image, threaded through _OVERRIDE_KEYS so they persist and survive restart-replay like every other per-load override. A flag supplied by the caller suppresses the matching built-in rather than appearing twice, since vLLM errors on duplicates. The legacy GB10 workarounds now apply only when the instance runs the pinned default image.

Curated catalog entries for both models carry their proven recipe (image, flags, recommended memory fraction), applied as defaults so a bare {"model": "..."} load, which is what the dashboard sends, launches correctly.

Dropped --rm, and start_solo() now confirms the container reached Running and surfaces the engine's last output when it did not.

Two bugs turned up while verifying on hardware and are fixed here too:

• vllm/vllm-openai bakes ENTRYPOINT ["vllm","serve"] while the default image uses NVIDIA's passthrough shim, so our args produced vllm serve vllm serve <model>. The launch path now inspects the image entrypoint and emits only the missing tokens. It deliberately does not override the entrypoint, which would bypass CUDA setup, and an unreadable image falls back to the legacy prefix.
• The automatic fp8 to auto KV downgrade for multimodal models only fires for models on local disk because it reads config.json. Qwen3.8 serves from the HF cache, so it was getting fp8 KV, which corrupts vision output on GB10. The recipe states auto explicitly.

Tests

696 pass, ruff clean. 26 new tests cover recipe passthrough, flag dedup in both --flag value and --flag=value forms, image gating of the legacy workarounds, entrypoint normalization, and launch confirmation. Two existing nvidia-backend tests were updated to the stronger launch contract, since "the docker CLI forked" no longer counts as a successful start.

Verified end to end on spark-3. A bare POST /api/models/load {"model":"unsloth/Qwen3.8-27B-NVFP4"} produced this command:

IMAGE: vllm/vllm-openai:v0.27.1
ARGS : serve unsloth/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8000
       --gpu-memory-utilization 0.6 --enable-prefix-caching --kv-cache-dtype auto
       --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice
       --speculative_config {"method":"qwen3_5_mtp","num_speculative_tokens":2}

It served chat, emitted correct tool calls (proving the parser applied), answered a vision prompt correctly, and ran at 18.1 tok/s with MTP speculative decoding. The model now appears on the master node's /v1/models and a request sent there routes to spark-3 and comes back. That was not possible before this change.

Nemotron 3.5 Lightning uses the identical mechanism and its recipe is unit-tested, but it has not yet been launched through the API on hardware. That is tracked in FOLLOWUPS.md along with pre-staging its DSpark drafter.

Default behavior is unchanged for every model that does not carry a recipe. The generated command for the pinned default image is byte for byte what it was before.