Skip to content

Add an opt-in request-shape warmup to the DeepSeek launchers - #394

Merged
FujitsuPolycom merged 3 commits into
mainfrom
issue-188-deepseek-warmup
Nov 1, 2026
Merged

FujitsuPolycom merged 3 commits into
mainfrom
issue-188-deepseek-warmup

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Refs #188. The calibration the issue requires on a real image remains open.

What and why

The DeepSeek pair and cycle launchers start serving with no request-shape warmup. On a cold compiler cache, the first traffic at a new shape compiles while every rank waits in a collective, and the distributed watchdog can terminate the waiting rank.

With SHAPE_WARMUP=1 in rank 0's env file, --run starts the container and then runs scripts/deepseek_v4_shape_warmup.py in the launcher's foreground. The phase waits for /health, then fires a deterministic sweep: a concurrency ladder up to MAX_NUM_SEQS, a medium prefill, a prefill longer than the recorded 8192-token boundary, a thinking-off chat arm, and one completion prompt per compile bucket at the exact token count the provisional rule names.

  • Every request starts with a unique nonce, so prefix caching cannot satisfy the prefill.
  • Completion targets are searched through /tokenize; a target the nonce alone exceeds is listed as skipped, never approximated.
  • The phase is non-fatal: a failed request, an exhausted budget or a missing python3 warns and the container keeps serving. An unreadable SHAPE_WARMUP_API_KEY_FILE stops the launch before the container starts.
  • BLOCK_SIZE, MAX_NUM_SEQS, NUM_SPECULATIVE_TOKENS, MAX_NUM_BATCHED_TOKENS and MAX_MODEL_LEN come from the rank env file the container receives; the module has no numeric defaults for them.
  • SHAPE_WARMUP_API_KEY_FILE sends the file's first non-whitespace line as Authorization: Bearer, the same rule deepseek_v4_cycle_ctl.py --api-key-file uses; /health is probed without it.

Uncalibrated. The bucket rule was measured on a different serving image than this repository pins. docs/operations/deepseek-0731.md#calibrating-the-sweep defines the cold-cache + --jit-monitor-verbose procedure that must pass on an image and topology before the phase is enabled by default. Nothing in this change claims that procedure or the sweep was run on hardware.

Validation

  • scripts/test_deepseek_v4_shape_warmup.py: 21 CPU tests over a loopback fake API covering bucket arithmetic, arm order, exact /tokenize lengths, nonce uniqueness, skip and failure accounting, bearer headers, readiness timeout and exit codes.
  • scripts/test_deepseek_v4_pair_launcher.py: the off-by-default gate, invalid gate/budget values and key file, BLOCK_SIZE propagation, no sweep when disabled, non-fatal failure with exit 0, and the sweep arguments passed.
  • pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py runtime/host scripts/test_sparkring_cli.py -q: 2219 passed, 86 skipped on the merged head.
  • ruff check --select E,F,W, scripts/check_repository_layout.py (1201 sources), scripts/check_markdown_links.py . (4228 links): clean.
  • Not run: any Spark. The sweep has never executed against a serving image, so the bucket rule itself, cold-cache compile coverage and the A/B soak named in the issue are unverified.

Compatibility

  • New optional env-file keys: SHAPE_WARMUP (default 0), SHAPE_WARMUP_WAIT_SECONDS (default 900), SHAPE_WARMUP_TIMEOUT_SECONDS (default 600), SHAPE_WARMUP_API_KEY_FILE.
  • Templates and generated examples change by comments only; profiles/environment-exports.json baselines are not rewritten.
  • Both launchers now pass --block-size "$BLOCK_SIZE" (default 256, the recipe value) explicitly, so the engine flag and the sweep cannot disagree.
  • With SHAPE_WARMUP unset, the launch is unchanged.

Known limits

  • cycle_ctl start launches rank 0 in the background, so its API-ready line can precede the sweep; the docs tell operators to wait for the shape_warmup summary in ctl-start-rank0.log before sending traffic.
  • The multi-chunk arm's 8192-token boundary comes from the pair's serving record; prompt sizes are recomputed from the rank env, but the boundary constant is not measured per image.

The DeepSeek-V4 pair and cycle launchers began serving without exercising request shapes the engine compiles on first use, so a cold compiler cache compiled known shapes during client traffic while ranks waited in a collective. Rank 0 now runs scripts/deepseek_v4_shape_warmup.py after /health and before the launcher returns: a concurrency ladder, medium and multi-chunk prefills, a thinking-off arm, and an exact token ladder verified through /tokenize, each request tagged with a leading nonce. MAX_NUM_SEQS, NUM_SPECULATIVE_TOKENS, BLOCK_SIZE, MAX_NUM_BATCHED_TOKENS and MAX_MODEL_LEN come from the rank env file, and SHAPE_WARMUP_API_KEY_FILE reuses the cycle controller's key-file rule for authenticated clusters.

The bucket rule comes from measurements on a different serving image, so SHAPE_WARMUP defaults to 0 and the quickstart documents the cold-cache calibration procedure. A warmup failure logs a warning and never fails the launch.

Validation: pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py runtime/host scripts/test_sparkring_cli.py -q (2189 passed, 86 skipped); ruff --select E,F,W --ignore E501; check_repository_layout.py; check_markdown_links.py. No hardware run: the sweep and its bucket mapping remain unverified on the served image.
The release-safety scan matches an identifier named bearer immediately before '=' followed by a long dotted expression, so bearer=config.bearer in the warmup module and the test-local bearer assignment were reported as credential assignments. The value an Authorization: Bearer header carries is now named token in WarmupConfig, read_token, the request helper and the tests. The header text, the --api-key-file flag, the key-file reading rule and every request remain unchanged.

Validation: scripts/check_release_safety.py . reports 0 findings; pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py -q (135 passed); ruff --select E,F,W --ignore E501 scripts clean.
@FujitsuPolycom
FujitsuPolycom merged commit ab89391 into main Nov 1, 2026
10 checks passed
@FujitsuPolycom
FujitsuPolycom deleted the issue-188-deepseek-warmup branch November 1, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant