From 6ce473c12042f91b76712c6f10154b758723323e Mon Sep 17 00:00:00 2001 From: Fixture Date: Sun, 1 Nov 2026 19:06:49 -0600 Subject: [PATCH] Align the cancel and warmup prose with the writing standard The warmup section claimed an unreadable `SHAPE_WARMUP_API_KEY_FILE` only warns. Both launchers exit 20 at their key-file check before `docker run -d`, so the section now states that it stops the launch. The installation reference links to the cancel section instead of repeating its mechanism, the cancel section records the `failed` state an interrupted installation writes when its restore did not complete, and duplicated mechanism sentences leave the quickstart and the script docstring. The env templates state that the warmup stays off until calibration, and `scripts/config/*.env.example` is regenerated from them. The operator view appends `rollout.SWITCH_NEXT`, so a group member and `sudo sparkring status` name the same action for an unsettled switch. That is the only behavior change; `profiles/environment-exports.json` is unchanged. Validation: `pytest runtime/host` (2078 passed, 86 skipped), `pytest scripts/test_deepseek_v4_shape_warmup.py scripts/test_deepseek_v4_pair_launcher.py` (135 passed), `ruff check --select E,F,W --ignore E501 runtime scripts`, `scripts/check_repository_layout.py` (1201 sources) and `scripts/check_markdown_links.py .` (4231 links). --- docs/operations/commands.md | 28 +++++------ docs/operations/deepseek-0731.md | 49 +++++++------------ docs/operations/install-reference.md | 4 +- .../runtime.env.template | 6 +-- .../runtime.env.template | 6 +-- runtime/host/cancel.py | 7 +-- runtime/host/controller.py | 2 +- runtime/host/install_workflow.py | 9 ++-- runtime/host/operator_view.py | 3 +- runtime/host/rollout.py | 10 ++-- .../deepseek-v4-flash-0731-pair.env.example | 6 +-- .../config/deepseek-v4-flash-0731.env.example | 6 +-- scripts/deepseek_v4_cycle_serve.sh | 2 +- scripts/deepseek_v4_pair_serve.sh | 2 +- scripts/deepseek_v4_shape_warmup.py | 8 +-- 15 files changed, 65 insertions(+), 83 deletions(-) diff --git a/docs/operations/commands.md b/docs/operations/commands.md index 631d9683..7d0a19a7 100644 --- a/docs/operations/commands.md +++ b/docs/operations/commands.md @@ -139,31 +139,25 @@ switch, stops there, stops the candidate deployment through that deployment's own stop, starts the model the switch replaced again and records the switch as `cancelled` with the time and the operator. The active pointer never leaves the model that served. The interrupted run prints the reason and exits 2 with -state `cancelled`; `--cancel` itself exits 0 when it asks, stops or settles a -switch, and 2 when another command holds the cluster or a switch it tried to -settle still needs attention. +state `cancelled`, or `failed` when the restore did not complete. `--cancel` +itself exits 0 when it asks, stops or settles a switch, and 2 when another +command holds the cluster or a switch it tried to settle still needs attention. ```bash sudo sparkring install --cancel ``` -- A step that cannot be abandoned, such as a checkpoint download, is the - boundary: the installation stops when that step ends. The request outlives - the command, and `sparkring status` reports the switch until it settles. +- A step that cannot be abandoned, such as a checkpoint download, runs to its + end; the installation stops at the next boundary. The request outlives the + command, and `sparkring status` reports the switch until it settles. - With no installation running, the same command settles a model switch an earlier installation left unfinished, for example after `SIGTERM`, `Ctrl-C` or a Spark restart. It stops that switch's candidate on every Spark and - starts the model the switch replaced, so the candidate's own `sparkring down - PROFILE --instance i` is no longer needed for this. -- `--cancel` takes no other installation option and no lock: it ends the - operation whose admission already holds them. An operator's [hold](#hold) - does not refuse it, because a hold reserves the cluster for work rather than - for keeping an unwanted installation. While another command changes the - cluster (`up`, `down` or `restart`), it names that command and changes - nothing: repeat it when the operation finishes. -- A switch in progress, or one that stopped before it finished, is reported by - [status](#status) as `Model switch: ... | next: ...`, and by the - group-readable operator view a member prints without sudo. + starts the model the switch replaced. +- `--cancel` takes no other installation option and no installation lock: the + operation it ends already holds that lock. A hold does not refuse it. While + another command changes the cluster (`up`, `down` or `restart`), it names + that command and changes nothing: repeat it when the operation finishes. ## install.sh diff --git a/docs/operations/deepseek-0731.md b/docs/operations/deepseek-0731.md index eb45255e..8df0cd57 100644 --- a/docs/operations/deepseek-0731.md +++ b/docs/operations/deepseek-0731.md @@ -374,11 +374,12 @@ launcher and does not change the launch contract. ## 2.6 Request-shape warmup -Status: **uncalibrated, disabled by default.** The sweep walks request shapes a -DeepSeek-V4-Flash deployment can schedule, so a cold compiler cache compiles -them during boot rather than during client traffic that holds ranks in a -collective. It does not establish that the covered shapes match the kernels -this image compiles; [Calibrating the sweep](#calibrating-the-sweep) does. +Status: **Development**; off by default and uncalibrated on every image. The +sweep walks request shapes a DeepSeek-V4-Flash deployment can schedule, so a +cold compiler cache compiles them during boot rather than during client traffic +that holds ranks in a collective. It does not establish that the covered shapes +match the kernels the image the launcher pins compiles; +[Calibrating the sweep](#calibrating-the-sweep) does. With `SHAPE_WARMUP=1` in rank 0's env file, `--run` starts the container and then runs [`scripts/deepseek_v4_shape_warmup.py`](../../scripts/deepseek_v4_shape_warmup.py) @@ -389,27 +390,20 @@ arms in order: |---|---:|---| | `concurrency-1` … `concurrency-MAX_NUM_SEQS` | 1, 2, 4, … then the sequence limit | Chat requests fired together | | `prefill-medium` | 1 | Completion prompt of half `MAX_NUM_BATCHED_TOKENS` tokens | -| `prefill-multi-chunk` | 1 | Completion prompt of 8192 + `MAX_NUM_BATCHED_TOKENS` tokens, longer than the 8192-token boundary | +| `prefill-multi-chunk` | 1 | Completion prompt of 8192 + `MAX_NUM_BATCHED_TOKENS` tokens, past the 8,193-token longest request the pair's serving record completed | | `thinking-off` | 1 | Chat request with `chat_template_kwargs {"enable_thinking": false}` | | `tokens-…` | 1 per bucket | Completion prompts sized so `min(BLOCK_SIZE, next_pow2(tokens + NUM_SPECULATIVE_TOKENS + 1))` reaches each power-of-two bucket through `BLOCK_SIZE` | -Every request starts with a unique nonce, so prefix caching cannot satisfy the -prefill from an earlier request. Chat arms report the rendered prompt length -that `/tokenize` returns. Completion arms search filler length through -`/tokenize` until the prompt has exactly the bucket's token count; a target the -nonce alone exceeds is listed under `skipped` in the summary instead of being -approximated. `MAX_NUM_SEQS`, `BLOCK_SIZE`, `NUM_SPECULATIVE_TOKENS`, +`MAX_NUM_SEQS`, `BLOCK_SIZE`, `NUM_SPECULATIVE_TOKENS`, `MAX_NUM_BATCHED_TOKENS` and `MAX_MODEL_LEN` come from the same rank env file the container receives, and the module has no defaults for them. -The phase is non-fatal. A failed request, an exhausted budget, a missing -`python3` or an unreadable key file logs a warning and the container keeps -serving; only the container command itself fails the launch. The launcher's -standard output carries one `shape_warmup` JSON summary with each arm's -completed and failed request counts, measured prompt lengths, compile buckets -and every skip. The pinned server serves `/v1/completions` for the -completion arms; a server that rejects it reports those arms as failures rather -than silently skipping them. +The phase is non-fatal. A failed request, an exhausted budget or a missing +`python3` logs a warning and the container keeps serving. An unreadable +`SHAPE_WARMUP_API_KEY_FILE` stops the launch before the container starts. The +launcher's standard output carries one `shape_warmup` JSON summary with each +arm's completed and failed request counts, measured prompt lengths, compile +buckets and every skip. On a cluster that enforces bearer authentication, set `SHAPE_WARMUP_API_KEY_FILE` to the rank-0 file the @@ -423,11 +417,10 @@ traffic, wait for the `shape_warmup` summary in ### Calibrating the sweep Run this procedure on each image and topology before setting `SHAPE_WARMUP=1` -in a deployment. Measurements on an earlier DeepSeek-V4-Flash serving image -found the DSpark input-preparation kernel keyed by the token-count rule above -rather than by request concurrency; those measurements used a different -serving image than this repository pins, so the rule and the arm list both -need confirmation. +in a deployment. A measurement on a serving image this repository does not pin +(its digest was not recorded) found the DSpark input-preparation kernel keyed +by the token-count rule above rather than by request concurrency. The rule and +the arm list both need confirmation on the pinned image. 1. Stop the deployment with the recovery procedure. Keep the model and flight-recorder data, and move `CACHE_HOST_PATH/jit` aside so the next @@ -453,11 +446,7 @@ need confirmation. 5. Replay the same client mix. Passing evidence is that none of the kernels recorded in step 2 compiles after the sweep. A kernel that still compiles names a bucket the sweep does not cover; record the arm-to-kernel mapping - and extend the sweep before claiming coverage. Kernels this profile has - compiled during serving include `_prepare_dflash_inputs_kernel`, - `_compute_global_topk_indices_and_lens_kernel` and the B12X CuTeDSL MoE - kernels. The summary's `skipped` entries state which arms the nonce or - request limit excluded. + and extend the sweep before claiming coverage. 6. Compare the same request mix with the phase disabled and enabled: TP rank disconnects and restarts, and first-request time to first token. Record conditions, measurements and the conclusion with the deployment record. diff --git a/docs/operations/install-reference.md b/docs/operations/install-reference.md index 4922aa01..403b6ab6 100644 --- a/docs/operations/install-reference.md +++ b/docs/operations/install-reference.md @@ -248,9 +248,7 @@ are host settings that SparkRing neither applies nor has measured resumes the switch. Another profile or checkpoint replaces it instead: after the GPU check, the installer stops the unfinished model through its own deployment and switches from the last model that served. - `sudo sparkring install --cancel` settles it without installing anything: it - stops the unfinished candidate on every Spark, starts the model the switch - replaced again and records the switch as `cancelled` + `sudo sparkring install --cancel` settles it without installing anything ([cancel an installation](commands.md#cancel-an-installation)). - **Selected model's last start did not complete**, for example when a first installation failed its readiness check or a restart of the installed model diff --git a/profiles/deepseek-v4-flash-0731-pair/runtime.env.template b/profiles/deepseek-v4-flash-0731-pair/runtime.env.template index 7760eae4..677e10f6 100644 --- a/profiles/deepseek-v4-flash-0731-pair/runtime.env.template +++ b/profiles/deepseek-v4-flash-0731-pair/runtime.env.template @@ -63,9 +63,9 @@ MAX_MODEL_LEN={{max_model_len}} MAX_NUM_SEQS={{max_num_seqs}} MAX_NUM_BATCHED_TOKENS={{max_num_batched_tokens}} -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/profiles/deepseek-v4-flash-0731/runtime.env.template b/profiles/deepseek-v4-flash-0731/runtime.env.template index 5e4ba951..a7975441 100644 --- a/profiles/deepseek-v4-flash-0731/runtime.env.template +++ b/profiles/deepseek-v4-flash-0731/runtime.env.template @@ -54,9 +54,9 @@ MAX_MODEL_LEN={{max_model_len}} MAX_NUM_SEQS={{max_num_seqs}} MAX_NUM_BATCHED_TOKENS={{max_num_batched_tokens}} -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/runtime/host/cancel.py b/runtime/host/cancel.py index 39f93c87..ab790d07 100644 --- a/runtime/host/cancel.py +++ b/runtime/host/cancel.py @@ -76,7 +76,7 @@ def entries(state_root): def request(state_root, operations, *, now=None, who=None): - """Record that the installations ``operations`` must stop at their next step boundary.""" + """Record that the installations named in ``operations`` must stop at their next step boundary.""" ids = sorted({str(item) for item in operations if item}) if not ids: raise ValueError("No installation is running to cancel") @@ -157,8 +157,9 @@ def run(state_root, *, now=None, who=None, say=print, sleep=time.sleep, active=N ``switches`` the model switches this command settled, ``busy`` the slots another operation kept, and ``message`` the one line an operator reads. ``state`` is ``requested`` while a running installation finishes a step it - cannot abandon, ``failed`` when a switch this command settled still needs - attention, and otherwise ``cancelled`` or ``idle``. + cannot abandon, ``busy`` when another command holds the cluster, ``failed`` + when a switch this command settled still needs attention, and otherwise + ``cancelled`` or ``idle``. """ from runtime.host import placement as placements, slices state_root = Path(state_root) diff --git a/runtime/host/controller.py b/runtime/host/controller.py index d53d8d70..78be259e 100644 --- a/runtime/host/controller.py +++ b/runtime/host/controller.py @@ -769,7 +769,7 @@ def _lifecycle(argv, *, output, locks): # An operator's hold (runtime.host.hold) comes first: other operators' changes refuse while it stands. result["hold"] = hold.current(STATE) from runtime.host import rollout - # A model switch that stopped before it finished, and the step that settles it. + # A model switch that has not settled. switches = rollout.unfinished(STATE) if switches: result["switches"] = [{"placement": list(slot) if slot else None, diff --git a/runtime/host/install_workflow.py b/runtime/host/install_workflow.py index 9ddd6e30..4f53b0e1 100644 --- a/runtime/host/install_workflow.py +++ b/runtime/host/install_workflow.py @@ -1732,8 +1732,8 @@ def prepare(path): unfinished=installer.unfinished, placement=placement, displaced=displaced, cancelled=lambda boundary: cancel.check(state_root, operation_id, boundary)) finally: - # Every step a cancel names is behind this run either way, so its - # request must not stand for a later installation. + # Every step a cancel can name belongs to this run, so the request + # must not stand for a later installation. cancel.clear(state_root, operation_id) # Records the deployment's choice (on unless --no-auto-recover, also # after sparkring recover off), resets failures and restarts, records @@ -1864,9 +1864,8 @@ def main(argv=None): args.serving_text = {name: value for name, value in vars(sizes).items() if value is not None} colour.set_mode(args.color) if args.cancel: - # Ending an installation changes nothing an operator holds: the request - # takes no installation lock and no hold override, and the command - # carries no other option. + # The request takes no installation lock and no hold override, and the + # command accepts no other option. used = sorted(name for name, value in vars(args).items() if name not in ("cancel", "json", "color") and value not in (None, False, {}, [])) if used: diff --git a/runtime/host/operator_view.py b/runtime/host/operator_view.py index 26ecaeac..7de81936 100644 --- a/runtime/host/operator_view.py +++ b/runtime/host/operator_view.py @@ -207,10 +207,11 @@ def lines(document: dict, *, now=time.time) -> list[str]: + (f", {recovery_record['failures']} failed attempt(s)" if recovery_record.get("failures") else "")) if not document.get("deployments"): result.append("No model deployment is recorded.") + from runtime.host import rollout for switch in document.get("switches") or []: where = ("Sparks " + ",".join(map(str, switch["placement"])) if switch.get("placement") else "every Spark") name = f"{switch['previous']} to {switch['candidate']}" if switch.get("previous") else switch.get("candidate") result.append(f"Model switch: {name} on {where}, recorded {switch['state']}" - " | next: an operator's sudo sparkring install --cancel") + " | next: " + rollout.SWITCH_NEXT) return result diff --git a/runtime/host/rollout.py b/runtime/host/rollout.py index ad64b3fd..353eef90 100644 --- a/runtime/host/rollout.py +++ b/runtime/host/rollout.py @@ -38,9 +38,9 @@ def execute(directory, previous, *, state_root, prepare, apply, verify, supersed ``cancelled(boundary)`` is asked before each step the switch records, and raises ``cancel.Cancelled`` to stop it there. A switch that has not yet - stopped anything records ``cancelled``; one that has brings back every - deployment it stopped first, and one whose restore fails records - ``needs-attention`` with the restore's error. + stopped anything records ``cancelled``; one that has stopped something + brings back every deployment it stopped first, and one whose restore fails + records ``needs-attention`` with the restore's error. """ directory = Path(directory).resolve() previous = Path(previous).resolve() if previous else None @@ -237,7 +237,7 @@ def unfinished(state_root): def switch_line(slot, record, size=None): - """The first half of one ``sparkring status`` line for a model switch in progress or stopped.""" + """``Model switch: OLD to NEW on Sparks ...`` and the record's state, for a switch in progress or stopped.""" candidate = Path(record.get("candidate") or "").name previous = Path(record["previous"]).name if record.get("previous") else None switch = f"{previous} to {candidate}" if previous else candidate @@ -245,7 +245,7 @@ def switch_line(slot, record, size=None): f"recorded {record.get('state')}") -# What settles a switch that stopped before it finished, appended to switch_line's line. +# The action that settles a switch which stopped before it finished: cancel it, or install that model again. SWITCH_NEXT = "sudo sparkring install --cancel stops it and records it cancelled, or install that model again to resume it" diff --git a/scripts/config/deepseek-v4-flash-0731-pair.env.example b/scripts/config/deepseek-v4-flash-0731-pair.env.example index b8a90f58..076c2b44 100644 --- a/scripts/config/deepseek-v4-flash-0731-pair.env.example +++ b/scripts/config/deepseek-v4-flash-0731-pair.env.example @@ -63,9 +63,9 @@ MAX_MODEL_LEN=1048576 MAX_NUM_SEQS=32 MAX_NUM_BATCHED_TOKENS=4096 -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/scripts/config/deepseek-v4-flash-0731.env.example b/scripts/config/deepseek-v4-flash-0731.env.example index 76cbf69e..8a8e56b4 100644 --- a/scripts/config/deepseek-v4-flash-0731.env.example +++ b/scripts/config/deepseek-v4-flash-0731.env.example @@ -54,9 +54,9 @@ MAX_MODEL_LEN=1048576 MAX_NUM_SEQS=32 MAX_NUM_BATCHED_TOKENS=4096 -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/scripts/deepseek_v4_cycle_serve.sh b/scripts/deepseek_v4_cycle_serve.sh index 1b09888a..f006a562 100755 --- a/scripts/deepseek_v4_cycle_serve.sh +++ b/scripts/deepseek_v4_cycle_serve.sh @@ -311,7 +311,7 @@ if [ "$NODE_RANK" = 0 ] && [ "$SHAPE_WARMUP" = 1 ]; then shape_warmup_args+=(--api-key-file "$SHAPE_WARMUP_API_KEY_FILE") fi printf 'Shape warmup: waiting for rank 0 health, then sweeping request shapes.\n' - # The phase is non-fatal: a failure keeps the boot and only logs a warning. + # Non-fatal: a sweep failure warns and the container keeps serving. if ! command -v python3 >/dev/null 2>&1; then echo "deepseek cycle launcher: warning: python3 is unavailable; request-shape warmup skipped" >&2 elif ! python3 "$script_dir/deepseek_v4_shape_warmup.py" "${shape_warmup_args[@]}"; then diff --git a/scripts/deepseek_v4_pair_serve.sh b/scripts/deepseek_v4_pair_serve.sh index 2a4acb23..e53abda9 100755 --- a/scripts/deepseek_v4_pair_serve.sh +++ b/scripts/deepseek_v4_pair_serve.sh @@ -267,7 +267,7 @@ if [ "$NODE_RANK" = 0 ] && [ "$SHAPE_WARMUP" = 1 ]; then shape_warmup_args+=(--api-key-file "$SHAPE_WARMUP_API_KEY_FILE") fi printf 'Shape warmup: waiting for rank 0 health, then sweeping request shapes.\n' - # The phase is non-fatal: a failure keeps the boot and only logs a warning. + # Non-fatal: a sweep failure warns and the container keeps serving. if ! command -v python3 >/dev/null 2>&1; then echo "deepseek pair launcher: warning: python3 is unavailable; request-shape warmup skipped" >&2 elif ! python3 "$script_dir/deepseek_v4_shape_warmup.py" "${shape_warmup_args[@]}"; then diff --git a/scripts/deepseek_v4_shape_warmup.py b/scripts/deepseek_v4_shape_warmup.py index 2dc6a634..73977905 100644 --- a/scripts/deepseek_v4_shape_warmup.py +++ b/scripts/deepseek_v4_shape_warmup.py @@ -7,11 +7,11 @@ after its API reports ready and before the launcher hands the endpoint to clients. -The bucket rule is provisional. Measurements on an earlier DeepSeek-V4-Flash -serving image found the DSpark input-preparation kernel keyed by +The bucket rule is unconfirmed on any pinned image. A measurement on a serving +image this repository does not pin (its digest was not recorded) found the +DSpark input-preparation kernel keyed by ``min(block_size, next_pow2(scheduled_tokens + num_speculative_tokens + 1))``, -not by request concurrency; those measurements used a different serving image -than this repository pins. The sweep covers that rule, and the calibration +not by request concurrency. The sweep covers that rule, and the calibration procedure in ``docs/operations/deepseek-0731.md`` establishes the actual keys on a deployment before it enables the phase by default.