diff --git a/docs/operations/commands.md b/docs/operations/commands.md index 631d9683..7d0a19a7 100644 --- a/docs/operations/commands.md +++ b/docs/operations/commands.md @@ -139,31 +139,25 @@ switch, stops there, stops the candidate deployment through that deployment's own stop, starts the model the switch replaced again and records the switch as `cancelled` with the time and the operator. The active pointer never leaves the model that served. The interrupted run prints the reason and exits 2 with -state `cancelled`; `--cancel` itself exits 0 when it asks, stops or settles a -switch, and 2 when another command holds the cluster or a switch it tried to -settle still needs attention. +state `cancelled`, or `failed` when the restore did not complete. `--cancel` +itself exits 0 when it asks, stops or settles a switch, and 2 when another +command holds the cluster or a switch it tried to settle still needs attention. ```bash sudo sparkring install --cancel ``` -- A step that cannot be abandoned, such as a checkpoint download, is the - boundary: the installation stops when that step ends. The request outlives - the command, and `sparkring status` reports the switch until it settles. +- A step that cannot be abandoned, such as a checkpoint download, runs to its + end; the installation stops at the next boundary. The request outlives the + command, and `sparkring status` reports the switch until it settles. - With no installation running, the same command settles a model switch an earlier installation left unfinished, for example after `SIGTERM`, `Ctrl-C` or a Spark restart. It stops that switch's candidate on every Spark and - starts the model the switch replaced, so the candidate's own `sparkring down - PROFILE --instance i` is no longer needed for this. -- `--cancel` takes no other installation option and no lock: it ends the - operation whose admission already holds them. An operator's [hold](#hold) - does not refuse it, because a hold reserves the cluster for work rather than - for keeping an unwanted installation. While another command changes the - cluster (`up`, `down` or `restart`), it names that command and changes - nothing: repeat it when the operation finishes. -- A switch in progress, or one that stopped before it finished, is reported by - [status](#status) as `Model switch: ... | next: ...`, and by the - group-readable operator view a member prints without sudo. + starts the model the switch replaced. +- `--cancel` takes no other installation option and no installation lock: the + operation it ends already holds that lock. A hold does not refuse it. While + another command changes the cluster (`up`, `down` or `restart`), it names + that command and changes nothing: repeat it when the operation finishes. ## install.sh diff --git a/docs/operations/deepseek-0731.md b/docs/operations/deepseek-0731.md index eb45255e..8df0cd57 100644 --- a/docs/operations/deepseek-0731.md +++ b/docs/operations/deepseek-0731.md @@ -374,11 +374,12 @@ launcher and does not change the launch contract. ## 2.6 Request-shape warmup -Status: **uncalibrated, disabled by default.** The sweep walks request shapes a -DeepSeek-V4-Flash deployment can schedule, so a cold compiler cache compiles -them during boot rather than during client traffic that holds ranks in a -collective. It does not establish that the covered shapes match the kernels -this image compiles; [Calibrating the sweep](#calibrating-the-sweep) does. +Status: **Development**; off by default and uncalibrated on every image. The +sweep walks request shapes a DeepSeek-V4-Flash deployment can schedule, so a +cold compiler cache compiles them during boot rather than during client traffic +that holds ranks in a collective. It does not establish that the covered shapes +match the kernels the image the launcher pins compiles; +[Calibrating the sweep](#calibrating-the-sweep) does. With `SHAPE_WARMUP=1` in rank 0's env file, `--run` starts the container and then runs [`scripts/deepseek_v4_shape_warmup.py`](../../scripts/deepseek_v4_shape_warmup.py) @@ -389,27 +390,20 @@ arms in order: |---|---:|---| | `concurrency-1` … `concurrency-MAX_NUM_SEQS` | 1, 2, 4, … then the sequence limit | Chat requests fired together | | `prefill-medium` | 1 | Completion prompt of half `MAX_NUM_BATCHED_TOKENS` tokens | -| `prefill-multi-chunk` | 1 | Completion prompt of 8192 + `MAX_NUM_BATCHED_TOKENS` tokens, longer than the 8192-token boundary | +| `prefill-multi-chunk` | 1 | Completion prompt of 8192 + `MAX_NUM_BATCHED_TOKENS` tokens, past the 8,193-token longest request the pair's serving record completed | | `thinking-off` | 1 | Chat request with `chat_template_kwargs {"enable_thinking": false}` | | `tokens-…` | 1 per bucket | Completion prompts sized so `min(BLOCK_SIZE, next_pow2(tokens + NUM_SPECULATIVE_TOKENS + 1))` reaches each power-of-two bucket through `BLOCK_SIZE` | -Every request starts with a unique nonce, so prefix caching cannot satisfy the -prefill from an earlier request. Chat arms report the rendered prompt length -that `/tokenize` returns. Completion arms search filler length through -`/tokenize` until the prompt has exactly the bucket's token count; a target the -nonce alone exceeds is listed under `skipped` in the summary instead of being -approximated. `MAX_NUM_SEQS`, `BLOCK_SIZE`, `NUM_SPECULATIVE_TOKENS`, +`MAX_NUM_SEQS`, `BLOCK_SIZE`, `NUM_SPECULATIVE_TOKENS`, `MAX_NUM_BATCHED_TOKENS` and `MAX_MODEL_LEN` come from the same rank env file the container receives, and the module has no defaults for them. -The phase is non-fatal. A failed request, an exhausted budget, a missing -`python3` or an unreadable key file logs a warning and the container keeps -serving; only the container command itself fails the launch. The launcher's -standard output carries one `shape_warmup` JSON summary with each arm's -completed and failed request counts, measured prompt lengths, compile buckets -and every skip. The pinned server serves `/v1/completions` for the -completion arms; a server that rejects it reports those arms as failures rather -than silently skipping them. +The phase is non-fatal. A failed request, an exhausted budget or a missing +`python3` logs a warning and the container keeps serving. An unreadable +`SHAPE_WARMUP_API_KEY_FILE` stops the launch before the container starts. The +launcher's standard output carries one `shape_warmup` JSON summary with each +arm's completed and failed request counts, measured prompt lengths, compile +buckets and every skip. On a cluster that enforces bearer authentication, set `SHAPE_WARMUP_API_KEY_FILE` to the rank-0 file the @@ -423,11 +417,10 @@ traffic, wait for the `shape_warmup` summary in ### Calibrating the sweep Run this procedure on each image and topology before setting `SHAPE_WARMUP=1` -in a deployment. Measurements on an earlier DeepSeek-V4-Flash serving image -found the DSpark input-preparation kernel keyed by the token-count rule above -rather than by request concurrency; those measurements used a different -serving image than this repository pins, so the rule and the arm list both -need confirmation. +in a deployment. A measurement on a serving image this repository does not pin +(its digest was not recorded) found the DSpark input-preparation kernel keyed +by the token-count rule above rather than by request concurrency. The rule and +the arm list both need confirmation on the pinned image. 1. Stop the deployment with the recovery procedure. Keep the model and flight-recorder data, and move `CACHE_HOST_PATH/jit` aside so the next @@ -453,11 +446,7 @@ need confirmation. 5. Replay the same client mix. Passing evidence is that none of the kernels recorded in step 2 compiles after the sweep. A kernel that still compiles names a bucket the sweep does not cover; record the arm-to-kernel mapping - and extend the sweep before claiming coverage. Kernels this profile has - compiled during serving include `_prepare_dflash_inputs_kernel`, - `_compute_global_topk_indices_and_lens_kernel` and the B12X CuTeDSL MoE - kernels. The summary's `skipped` entries state which arms the nonce or - request limit excluded. + and extend the sweep before claiming coverage. 6. Compare the same request mix with the phase disabled and enabled: TP rank disconnects and restarts, and first-request time to first token. Record conditions, measurements and the conclusion with the deployment record. diff --git a/docs/operations/install-reference.md b/docs/operations/install-reference.md index 4922aa01..403b6ab6 100644 --- a/docs/operations/install-reference.md +++ b/docs/operations/install-reference.md @@ -248,9 +248,7 @@ are host settings that SparkRing neither applies nor has measured resumes the switch. Another profile or checkpoint replaces it instead: after the GPU check, the installer stops the unfinished model through its own deployment and switches from the last model that served. - `sudo sparkring install --cancel` settles it without installing anything: it - stops the unfinished candidate on every Spark, starts the model the switch - replaced again and records the switch as `cancelled` + `sudo sparkring install --cancel` settles it without installing anything ([cancel an installation](commands.md#cancel-an-installation)). - **Selected model's last start did not complete**, for example when a first installation failed its readiness check or a restart of the installed model diff --git a/profiles/deepseek-v4-flash-0731-pair/runtime.env.template b/profiles/deepseek-v4-flash-0731-pair/runtime.env.template index 7760eae4..677e10f6 100644 --- a/profiles/deepseek-v4-flash-0731-pair/runtime.env.template +++ b/profiles/deepseek-v4-flash-0731-pair/runtime.env.template @@ -63,9 +63,9 @@ MAX_MODEL_LEN={{max_model_len}} MAX_NUM_SEQS={{max_num_seqs}} MAX_NUM_BATCHED_TOKENS={{max_num_batched_tokens}} -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/profiles/deepseek-v4-flash-0731/runtime.env.template b/profiles/deepseek-v4-flash-0731/runtime.env.template index 5e4ba951..a7975441 100644 --- a/profiles/deepseek-v4-flash-0731/runtime.env.template +++ b/profiles/deepseek-v4-flash-0731/runtime.env.template @@ -54,9 +54,9 @@ MAX_MODEL_LEN={{max_model_len}} MAX_NUM_SEQS={{max_num_seqs}} MAX_NUM_BATCHED_TOKENS={{max_num_batched_tokens}} -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/runtime/host/cancel.py b/runtime/host/cancel.py index 39f93c87..ab790d07 100644 --- a/runtime/host/cancel.py +++ b/runtime/host/cancel.py @@ -76,7 +76,7 @@ def entries(state_root): def request(state_root, operations, *, now=None, who=None): - """Record that the installations ``operations`` must stop at their next step boundary.""" + """Record that the installations named in ``operations`` must stop at their next step boundary.""" ids = sorted({str(item) for item in operations if item}) if not ids: raise ValueError("No installation is running to cancel") @@ -157,8 +157,9 @@ def run(state_root, *, now=None, who=None, say=print, sleep=time.sleep, active=N ``switches`` the model switches this command settled, ``busy`` the slots another operation kept, and ``message`` the one line an operator reads. ``state`` is ``requested`` while a running installation finishes a step it - cannot abandon, ``failed`` when a switch this command settled still needs - attention, and otherwise ``cancelled`` or ``idle``. + cannot abandon, ``busy`` when another command holds the cluster, ``failed`` + when a switch this command settled still needs attention, and otherwise + ``cancelled`` or ``idle``. """ from runtime.host import placement as placements, slices state_root = Path(state_root) diff --git a/runtime/host/controller.py b/runtime/host/controller.py index d53d8d70..78be259e 100644 --- a/runtime/host/controller.py +++ b/runtime/host/controller.py @@ -769,7 +769,7 @@ def _lifecycle(argv, *, output, locks): # An operator's hold (runtime.host.hold) comes first: other operators' changes refuse while it stands. result["hold"] = hold.current(STATE) from runtime.host import rollout - # A model switch that stopped before it finished, and the step that settles it. + # A model switch that has not settled. switches = rollout.unfinished(STATE) if switches: result["switches"] = [{"placement": list(slot) if slot else None, diff --git a/runtime/host/install_workflow.py b/runtime/host/install_workflow.py index 9ddd6e30..4f53b0e1 100644 --- a/runtime/host/install_workflow.py +++ b/runtime/host/install_workflow.py @@ -1732,8 +1732,8 @@ def prepare(path): unfinished=installer.unfinished, placement=placement, displaced=displaced, cancelled=lambda boundary: cancel.check(state_root, operation_id, boundary)) finally: - # Every step a cancel names is behind this run either way, so its - # request must not stand for a later installation. + # Every step a cancel can name belongs to this run, so the request + # must not stand for a later installation. cancel.clear(state_root, operation_id) # Records the deployment's choice (on unless --no-auto-recover, also # after sparkring recover off), resets failures and restarts, records @@ -1864,9 +1864,8 @@ def main(argv=None): args.serving_text = {name: value for name, value in vars(sizes).items() if value is not None} colour.set_mode(args.color) if args.cancel: - # Ending an installation changes nothing an operator holds: the request - # takes no installation lock and no hold override, and the command - # carries no other option. + # The request takes no installation lock and no hold override, and the + # command accepts no other option. used = sorted(name for name, value in vars(args).items() if name not in ("cancel", "json", "color") and value not in (None, False, {}, [])) if used: diff --git a/runtime/host/operator_view.py b/runtime/host/operator_view.py index 26ecaeac..7de81936 100644 --- a/runtime/host/operator_view.py +++ b/runtime/host/operator_view.py @@ -207,10 +207,11 @@ def lines(document: dict, *, now=time.time) -> list[str]: + (f", {recovery_record['failures']} failed attempt(s)" if recovery_record.get("failures") else "")) if not document.get("deployments"): result.append("No model deployment is recorded.") + from runtime.host import rollout for switch in document.get("switches") or []: where = ("Sparks " + ",".join(map(str, switch["placement"])) if switch.get("placement") else "every Spark") name = f"{switch['previous']} to {switch['candidate']}" if switch.get("previous") else switch.get("candidate") result.append(f"Model switch: {name} on {where}, recorded {switch['state']}" - " | next: an operator's sudo sparkring install --cancel") + " | next: " + rollout.SWITCH_NEXT) return result diff --git a/runtime/host/rollout.py b/runtime/host/rollout.py index ad64b3fd..353eef90 100644 --- a/runtime/host/rollout.py +++ b/runtime/host/rollout.py @@ -38,9 +38,9 @@ def execute(directory, previous, *, state_root, prepare, apply, verify, supersed ``cancelled(boundary)`` is asked before each step the switch records, and raises ``cancel.Cancelled`` to stop it there. A switch that has not yet - stopped anything records ``cancelled``; one that has brings back every - deployment it stopped first, and one whose restore fails records - ``needs-attention`` with the restore's error. + stopped anything records ``cancelled``; one that has stopped something + brings back every deployment it stopped first, and one whose restore fails + records ``needs-attention`` with the restore's error. """ directory = Path(directory).resolve() previous = Path(previous).resolve() if previous else None @@ -237,7 +237,7 @@ def unfinished(state_root): def switch_line(slot, record, size=None): - """The first half of one ``sparkring status`` line for a model switch in progress or stopped.""" + """``Model switch: OLD to NEW on Sparks ...`` and the record's state, for a switch in progress or stopped.""" candidate = Path(record.get("candidate") or "").name previous = Path(record["previous"]).name if record.get("previous") else None switch = f"{previous} to {candidate}" if previous else candidate @@ -245,7 +245,7 @@ def switch_line(slot, record, size=None): f"recorded {record.get('state')}") -# What settles a switch that stopped before it finished, appended to switch_line's line. +# The action that settles a switch which stopped before it finished: cancel it, or install that model again. SWITCH_NEXT = "sudo sparkring install --cancel stops it and records it cancelled, or install that model again to resume it" diff --git a/scripts/config/deepseek-v4-flash-0731-pair.env.example b/scripts/config/deepseek-v4-flash-0731-pair.env.example index b8a90f58..076c2b44 100644 --- a/scripts/config/deepseek-v4-flash-0731-pair.env.example +++ b/scripts/config/deepseek-v4-flash-0731-pair.env.example @@ -63,9 +63,9 @@ MAX_MODEL_LEN=1048576 MAX_NUM_SEQS=32 MAX_NUM_BATCHED_TOKENS=4096 -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/scripts/config/deepseek-v4-flash-0731.env.example b/scripts/config/deepseek-v4-flash-0731.env.example index 76cbf69e..8a8e56b4 100644 --- a/scripts/config/deepseek-v4-flash-0731.env.example +++ b/scripts/config/deepseek-v4-flash-0731.env.example @@ -54,9 +54,9 @@ MAX_MODEL_LEN=1048576 MAX_NUM_SEQS=32 MAX_NUM_BATCHED_TOKENS=4096 -# Request-shape warmup runs on rank 0 after the API reports ready when -# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is -# calibrated on the deployed image; read +# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after +# the API reports ready. Leave it at 0 until the sweep is calibrated on the +# deployed image; read # docs/operations/deepseek-0731.md#26-request-shape-warmup first. # SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound # readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer diff --git a/scripts/deepseek_v4_cycle_serve.sh b/scripts/deepseek_v4_cycle_serve.sh index 1b09888a..f006a562 100755 --- a/scripts/deepseek_v4_cycle_serve.sh +++ b/scripts/deepseek_v4_cycle_serve.sh @@ -311,7 +311,7 @@ if [ "$NODE_RANK" = 0 ] && [ "$SHAPE_WARMUP" = 1 ]; then shape_warmup_args+=(--api-key-file "$SHAPE_WARMUP_API_KEY_FILE") fi printf 'Shape warmup: waiting for rank 0 health, then sweeping request shapes.\n' - # The phase is non-fatal: a failure keeps the boot and only logs a warning. + # Non-fatal: a sweep failure warns and the container keeps serving. if ! command -v python3 >/dev/null 2>&1; then echo "deepseek cycle launcher: warning: python3 is unavailable; request-shape warmup skipped" >&2 elif ! python3 "$script_dir/deepseek_v4_shape_warmup.py" "${shape_warmup_args[@]}"; then diff --git a/scripts/deepseek_v4_pair_serve.sh b/scripts/deepseek_v4_pair_serve.sh index 2a4acb23..e53abda9 100755 --- a/scripts/deepseek_v4_pair_serve.sh +++ b/scripts/deepseek_v4_pair_serve.sh @@ -267,7 +267,7 @@ if [ "$NODE_RANK" = 0 ] && [ "$SHAPE_WARMUP" = 1 ]; then shape_warmup_args+=(--api-key-file "$SHAPE_WARMUP_API_KEY_FILE") fi printf 'Shape warmup: waiting for rank 0 health, then sweeping request shapes.\n' - # The phase is non-fatal: a failure keeps the boot and only logs a warning. + # Non-fatal: a sweep failure warns and the container keeps serving. if ! command -v python3 >/dev/null 2>&1; then echo "deepseek pair launcher: warning: python3 is unavailable; request-shape warmup skipped" >&2 elif ! python3 "$script_dir/deepseek_v4_shape_warmup.py" "${shape_warmup_args[@]}"; then diff --git a/scripts/deepseek_v4_shape_warmup.py b/scripts/deepseek_v4_shape_warmup.py index 2dc6a634..73977905 100644 --- a/scripts/deepseek_v4_shape_warmup.py +++ b/scripts/deepseek_v4_shape_warmup.py @@ -7,11 +7,11 @@ after its API reports ready and before the launcher hands the endpoint to clients. -The bucket rule is provisional. Measurements on an earlier DeepSeek-V4-Flash -serving image found the DSpark input-preparation kernel keyed by +The bucket rule is unconfirmed on any pinned image. A measurement on a serving +image this repository does not pin (its digest was not recorded) found the +DSpark input-preparation kernel keyed by ``min(block_size, next_pow2(scheduled_tokens + num_speculative_tokens + 1))``, -not by request concurrency; those measurements used a different serving image -than this repository pins. The sweep covers that rule, and the calibration +not by request concurrency. The sweep covers that rule, and the calibration procedure in ``docs/operations/deepseek-0731.md`` establishes the actual keys on a deployment before it enables the phase by default.