Skip to content

refuse local media jobs when the vLLM container is holding the GPU - #4808

Merged
atomantic merged 1 commit into
mainfrom
claim/issue-4766
Aug 22, 2026
Merged

refuse local media jobs when the vLLM container is holding the GPU#4808
atomantic merged 1 commit into
mainfrom
claim/issue-4766

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

  • A serving vLLM Qwen3.8-27B container holds ~23 GB of a 24 GB RTX 3090, and unloadResidentModels() cannot see it — it evicts loopback Ollama / LM Studio only. Every GPU-heavy local job proceeded, allocated, and died inside its model load with an OOM naming neither vLLM nor a remedy.
  • New detectGpuBlockers() in server/lib/localMemory.js probes an enabled vllmBacked provider's local endpoint and surfaces the result as an additive blockers array on the prepareLocalMemory() report.
  • All four GPU-heavy callers (local image, video, image-to-3D, LoRA training) now fail fast on a non-empty blockers, with prose naming the container, its project directory, and the docker compose --profile single stop that frees the card.
  • Detect and refuse, never auto-stop — nothing would restart the container, an attached agent session dies with it, and a cold start is ~5–7 minutes. This mirrors PortOS's existing refusal to start it.
  • No latency on the common path: the probe is skipped entirely with no enabled vllmBacked provider and on any non-CUDA host. Per the sentinel rule, a probe that fails, times out, or throws means "not serving" → the job proceeds; only an endpoint that actually answers blocks (including a 401/403, which is a container up behind VLLM_API_KEY).
  • Nothing is unloaded while blocked — evicting the user's resident LLM for a run that is about to be refused helps nobody.
  • Rolled in: the image-generation pre-wiring failure path now unwinds its jobs entry and stepwise temp dir the way the busy branch already does, instead of only releasing the claim.

Test plan

  • cd server && npm test — 1565 files, 32,773 tests green.
  • New server/lib/localMemory.test.js (17 cases): no probe with no enabled vllmBacked provider (and no CUDA probe either), no probe when it is disabled, no probe on a non-CUDA host, still probes on an unknown CUDA verdict, blocker when the endpoint answers, blocker on a 401, no blocker when the probe fails / times out / throws, no blocker for a vLLM on another machine, one probe per endpoint preferring the credentialed record, no blocker when the provider store cannot be read, and the message naming the container / project dir / stop command.
  • Caller-level: server/services/videoGen/local.test.js asserts a blocked render rejects with the stop-command prose, never reaches spawnDetached, and releases the heavy claim.
  • Existing prepareLocalMemory mocks updated to carry blockers, so a schema drift fails rather than silently passing.

Closes #4766

…4766)

A serving vLLM Qwen3.8-27B container holds ~23 GB of a 24 GB RTX 3090, and
unloadResidentModels() cannot see it — it evicts loopback Ollama / LM Studio
only. Every GPU-heavy local job therefore proceeded, allocated, and died inside
its model load with an out-of-memory error naming neither vLLM nor a remedy,
sending the operator to debug the wrong subsystem.

prepareLocalMemory() now carries an additive blockers array from a new
detectGpuBlockers(): image, video, image-to-3D and LoRA-training jobs fail fast
with prose that names the container, its project directory and the
\`docker compose --profile single stop\` that frees the card. PortOS detects and
refuses; it never stops the container, mirroring its existing refusal to start
one — nothing would restart it, an attached agent session dies with it, and a
cold start is ~5-7 minutes.

The probe is skipped entirely when no vllmBacked provider is enabled and on any
non-CUDA host, so the common path gains no latency. A probe that fails, times
out, or throws means the container is not serving and the job proceeds:
\"couldn't check\" never collapses into \"is blocking\".
@atomantic
atomantic merged commit 4381623 into main Aug 22, 2026
7 checks passed
@atomantic
atomantic deleted the claim/issue-4766 branch August 22, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local media jobs don't notice the vLLM container holding the GPU

1 participant