Repository navigation
Add an opt-in request-shape warmup to the DeepSeek launchers - #394
Merged
Merged
Conversation
The DeepSeek-V4 pair and cycle launchers began serving without exercising request shapes the engine compiles on first use, so a cold compiler cache compiled known shapes during client traffic while ranks waited in a collective. Rank 0 now runs scripts/deepseek_v4_shape_warmup.py after /health and before the launcher returns: a concurrency ladder, medium and multi-chunk prefills, a thinking-off arm, and an exact token ladder verified through /tokenize, each request tagged with a leading nonce. MAX_NUM_SEQS, NUM_SPECULATIVE_TOKENS, BLOCK_SIZE, MAX_NUM_BATCHED_TOKENS and MAX_MODEL_LEN come from the rank env file, and SHAPE_WARMUP_API_KEY_FILE reuses the cycle controller's key-file rule for authenticated clusters. The bucket rule comes from measurements on a different serving image, so SHAPE_WARMUP defaults to 0 and the quickstart documents the cold-cache calibration procedure. A warmup failure logs a warning and never fails the launch. Validation: pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py runtime/host scripts/test_sparkring_cli.py -q (2189 passed, 86 skipped); ruff --select E,F,W --ignore E501; check_repository_layout.py; check_markdown_links.py. No hardware run: the sweep and its bucket mapping remain unverified on the served image.
The release-safety scan matches an identifier named bearer immediately before '=' followed by a long dotted expression, so bearer=config.bearer in the warmup module and the test-local bearer assignment were reported as credential assignments. The value an Authorization: Bearer header carries is now named token in WarmupConfig, read_token, the request helper and the tests. The header text, the --api-key-file flag, the key-file reading rule and every request remain unchanged. Validation: scripts/check_release_safety.py . reports 0 findings; pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py -q (135 passed); ruff --select E,F,W --ignore E501 scripts clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #188. The calibration the issue requires on a real image remains open.
What and why
The DeepSeek pair and cycle launchers start serving with no request-shape warmup. On a cold compiler cache, the first traffic at a new shape compiles while every rank waits in a collective, and the distributed watchdog can terminate the waiting rank.
With
SHAPE_WARMUP=1in rank 0's env file,--runstarts the container and then runsscripts/deepseek_v4_shape_warmup.pyin the launcher's foreground. The phase waits for/health, then fires a deterministic sweep: a concurrency ladder up toMAX_NUM_SEQS, a medium prefill, a prefill longer than the recorded 8192-token boundary, a thinking-off chat arm, and one completion prompt per compile bucket at the exact token count the provisional rule names./tokenize; a target the nonce alone exceeds is listed asskipped, never approximated.python3warns and the container keeps serving. An unreadableSHAPE_WARMUP_API_KEY_FILEstops the launch before the container starts.BLOCK_SIZE,MAX_NUM_SEQS,NUM_SPECULATIVE_TOKENS,MAX_NUM_BATCHED_TOKENSandMAX_MODEL_LENcome from the rank env file the container receives; the module has no numeric defaults for them.SHAPE_WARMUP_API_KEY_FILEsends the file's first non-whitespace line asAuthorization: Bearer, the same ruledeepseek_v4_cycle_ctl.py --api-key-fileuses;/healthis probed without it.Uncalibrated. The bucket rule was measured on a different serving image than this repository pins.
docs/operations/deepseek-0731.md#calibrating-the-sweepdefines the cold-cache +--jit-monitor-verboseprocedure that must pass on an image and topology before the phase is enabled by default. Nothing in this change claims that procedure or the sweep was run on hardware.Validation
scripts/test_deepseek_v4_shape_warmup.py: 21 CPU tests over a loopback fake API covering bucket arithmetic, arm order, exact/tokenizelengths, nonce uniqueness, skip and failure accounting, bearer headers, readiness timeout and exit codes.scripts/test_deepseek_v4_pair_launcher.py: the off-by-default gate, invalid gate/budget values and key file,BLOCK_SIZEpropagation, no sweep when disabled, non-fatal failure with exit 0, and the sweep arguments passed.pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py runtime/host scripts/test_sparkring_cli.py -q: 2219 passed, 86 skipped on the merged head.ruff check --select E,F,W,scripts/check_repository_layout.py(1201 sources),scripts/check_markdown_links.py .(4228 links): clean.Compatibility
SHAPE_WARMUP(default 0),SHAPE_WARMUP_WAIT_SECONDS(default 900),SHAPE_WARMUP_TIMEOUT_SECONDS(default 600),SHAPE_WARMUP_API_KEY_FILE.profiles/environment-exports.jsonbaselines are not rewritten.--block-size "$BLOCK_SIZE"(default 256, the recipe value) explicitly, so the engine flag and the sweep cannot disagree.SHAPE_WARMUPunset, the launch is unchanged.Known limits
cycle_ctl startlaunches rank 0 in the background, so its API-ready line can precede the sweep; the docs tell operators to wait for theshape_warmupsummary inctl-start-rank0.logbefore sending traffic.