Skip to content

Self-hosted model: a deploy Modal never schedules waits silently; Claude Code settings keep the caller's session variables #33

Open

Description

@flujo-app

Found on Nov 11 while Greenswarm started a run on a self-hosted Qwen through FACTORY 0.7.1 (Greenswarm issues flujo-app/greenswarm#318 and #324).

1. A deploy that Modal never schedules waits silently

On a Modal "starter" plan a function with a request time limit of 45 minutes or more is never given an H100. The qwen profile asks for 60 (request_timeout_minutes), nemotron too. Modal says so in the app log ("... is waiting to be scheduled on a GPU_H100 worker. Your workspace's concurrency limits are reached (H100 unavailable on starter plan)"), but ModalSelfHostedModel.deploy() only sees a held request and waits up to 45 minutes per replica, then reports an uncertain outcome. Measured with test functions of the same size: 10 to 30 minutes start in 3 to 4 seconds, 45 and 60 minutes never start.

Wanted: while verifying, read the app's log for the scheduler's "waiting to be scheduled" line and stop with that text; and a default request limit in the profiles that the common plans accept (30 minutes worked).

2. Claude Code settings for a self-hosted provider leave the caller's Claude session variables in place

claudeProviderEnvironment() sets the base address and the key and clears ANTHROPIC_API_KEY and CLAUDE_CODE_OAUTH_TOKEN. When the caller itself runs inside a Claude session (a desktop or SDK session), the child also inherits that session's variables (CLAUDE_CODE_SDK_HAS_HOST_AUTH_REFRESH, CLAUDE_CODE_ENTRYPOINT, CLAUDECODE, USE_STAGING_OAUTH, ...). Claude Code 2.1.321 then offers the session's own login to the endpoint instead of the endpoint key: HTTP 401, and a login is sent to a server it was not meant for. An older Claude Code (2.1.289) hangs instead.

Greenswarm now clears every inherited variable whose name starts with CLAUDE or ANTHROPIC_ or contains OAUTH before applying FACTORY's settings (swarm/lib/endpoint.mjs, sessionVariablesCleared). Wanted here: the same in claudeProviderEnvironment() (null for those names), so every caller gets it.

3. Small: a stand-in endpoint for tests has to speak https

validateModelProvider() refuses http://127.0.0.1:<port>. That is right for real endpoints; for tests a loopback address could be allowed. Greenswarm's stand-in makes its own certificate meanwhile.

Activity

  1. flujo-app commented on Nov 12, 2026

    @flujo-app
    OwnerAuthor

    Released in 0.7.2 (PR #34): the request limit defaults to 30 minutes (the cause of the silent wait on the starter plan), and Claude Code settings clear the caller's Claude session variables. Still open here: the deploy cannot yet tell a scheduler that never starts the GPU from a slow cold start (reading Modal's waiting message). Point 3 (loopback http for test stand-ins) is also still open.

  2. flujo-app commented on Nov 12, 2026

    @flujo-app
    OwnerAuthor

    Also found on the Nemotron deploy of Nov 12 and fixed in 0.7.3 (PR #35): the model's own code and tokenizer files came from the Hub's default branch, not the pinned revision; vLLM now gets --code-revision and --tokenizer-revision.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions