Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 11 additions & 17 deletions docs/operations/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,31 +139,25 @@ switch, stops there, stops the candidate deployment through that deployment's
own stop, starts the model the switch replaced again and records the switch as
`cancelled` with the time and the operator. The active pointer never leaves
the model that served. The interrupted run prints the reason and exits 2 with
state `cancelled`; `--cancel` itself exits 0 when it asks, stops or settles a
switch, and 2 when another command holds the cluster or a switch it tried to
settle still needs attention.
state `cancelled`, or `failed` when the restore did not complete. `--cancel`
itself exits 0 when it asks, stops or settles a switch, and 2 when another
command holds the cluster or a switch it tried to settle still needs attention.

```bash
sudo sparkring install --cancel
```

- A step that cannot be abandoned, such as a checkpoint download, is the
boundary: the installation stops when that step ends. The request outlives
the command, and `sparkring status` reports the switch until it settles.
- A step that cannot be abandoned, such as a checkpoint download, runs to its
end; the installation stops at the next boundary. The request outlives the
command, and `sparkring status` reports the switch until it settles.
- With no installation running, the same command settles a model switch an
earlier installation left unfinished, for example after `SIGTERM`, `Ctrl-C`
or a Spark restart. It stops that switch's candidate on every Spark and
starts the model the switch replaced, so the candidate's own `sparkring down
PROFILE --instance i<hash>` is no longer needed for this.
- `--cancel` takes no other installation option and no lock: it ends the
operation whose admission already holds them. An operator's [hold](#hold)
does not refuse it, because a hold reserves the cluster for work rather than
for keeping an unwanted installation. While another command changes the
cluster (`up`, `down` or `restart`), it names that command and changes
nothing: repeat it when the operation finishes.
- A switch in progress, or one that stopped before it finished, is reported by
[status](#status) as `Model switch: ... | next: ...`, and by the
group-readable operator view a member prints without sudo.
starts the model the switch replaced.
- `--cancel` takes no other installation option and no installation lock: the
operation it ends already holds that lock. A hold does not refuse it. While
another command changes the cluster (`up`, `down` or `restart`), it names
that command and changes nothing: repeat it when the operation finishes.

## install.sh

Expand Down
49 changes: 19 additions & 30 deletions docs/operations/deepseek-0731.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,11 +374,12 @@ launcher and does not change the launch contract.

## 2.6 Request-shape warmup

Status: **uncalibrated, disabled by default.** The sweep walks request shapes a
DeepSeek-V4-Flash deployment can schedule, so a cold compiler cache compiles
them during boot rather than during client traffic that holds ranks in a
collective. It does not establish that the covered shapes match the kernels
this image compiles; [Calibrating the sweep](#calibrating-the-sweep) does.
Status: **Development**; off by default and uncalibrated on every image. The
sweep walks request shapes a DeepSeek-V4-Flash deployment can schedule, so a
cold compiler cache compiles them during boot rather than during client traffic
that holds ranks in a collective. It does not establish that the covered shapes
match the kernels the image the launcher pins compiles;
[Calibrating the sweep](#calibrating-the-sweep) does.

With `SHAPE_WARMUP=1` in rank 0's env file, `--run` starts the container and then
runs [`scripts/deepseek_v4_shape_warmup.py`](../../scripts/deepseek_v4_shape_warmup.py)
Expand All @@ -389,27 +390,20 @@ arms in order:
|---|---:|---|
| `concurrency-1` … `concurrency-MAX_NUM_SEQS` | 1, 2, 4, … then the sequence limit | Chat requests fired together |
| `prefill-medium` | 1 | Completion prompt of half `MAX_NUM_BATCHED_TOKENS` tokens |
| `prefill-multi-chunk` | 1 | Completion prompt of 8192 + `MAX_NUM_BATCHED_TOKENS` tokens, longer than the 8192-token boundary |
| `prefill-multi-chunk` | 1 | Completion prompt of 8192 + `MAX_NUM_BATCHED_TOKENS` tokens, past the 8,193-token longest request the pair's serving record completed |
| `thinking-off` | 1 | Chat request with `chat_template_kwargs {"enable_thinking": false}` |
| `tokens-…` | 1 per bucket | Completion prompts sized so `min(BLOCK_SIZE, next_pow2(tokens + NUM_SPECULATIVE_TOKENS + 1))` reaches each power-of-two bucket through `BLOCK_SIZE` |

Every request starts with a unique nonce, so prefix caching cannot satisfy the
prefill from an earlier request. Chat arms report the rendered prompt length
that `/tokenize` returns. Completion arms search filler length through
`/tokenize` until the prompt has exactly the bucket's token count; a target the
nonce alone exceeds is listed under `skipped` in the summary instead of being
approximated. `MAX_NUM_SEQS`, `BLOCK_SIZE`, `NUM_SPECULATIVE_TOKENS`,
`MAX_NUM_SEQS`, `BLOCK_SIZE`, `NUM_SPECULATIVE_TOKENS`,
`MAX_NUM_BATCHED_TOKENS` and `MAX_MODEL_LEN` come from the same rank env file
the container receives, and the module has no defaults for them.

The phase is non-fatal. A failed request, an exhausted budget, a missing
`python3` or an unreadable key file logs a warning and the container keeps
serving; only the container command itself fails the launch. The launcher's
standard output carries one `shape_warmup` JSON summary with each arm's
completed and failed request counts, measured prompt lengths, compile buckets
and every skip. The pinned server serves `/v1/completions` for the
completion arms; a server that rejects it reports those arms as failures rather
than silently skipping them.
The phase is non-fatal. A failed request, an exhausted budget or a missing
`python3` logs a warning and the container keeps serving. An unreadable
`SHAPE_WARMUP_API_KEY_FILE` stops the launch before the container starts. The
launcher's standard output carries one `shape_warmup` JSON summary with each
arm's completed and failed request counts, measured prompt lengths, compile
buckets and every skip.

On a cluster that enforces bearer authentication, set
`SHAPE_WARMUP_API_KEY_FILE` to the rank-0 file the
Expand All @@ -423,11 +417,10 @@ traffic, wait for the `shape_warmup` summary in
### Calibrating the sweep

Run this procedure on each image and topology before setting `SHAPE_WARMUP=1`
in a deployment. Measurements on an earlier DeepSeek-V4-Flash serving image
found the DSpark input-preparation kernel keyed by the token-count rule above
rather than by request concurrency; those measurements used a different
serving image than this repository pins, so the rule and the arm list both
need confirmation.
in a deployment. A measurement on a serving image this repository does not pin
(its digest was not recorded) found the DSpark input-preparation kernel keyed
by the token-count rule above rather than by request concurrency. The rule and
the arm list both need confirmation on the pinned image.

1. Stop the deployment with the recovery procedure. Keep the model and
flight-recorder data, and move `CACHE_HOST_PATH/jit` aside so the next
Expand All @@ -453,11 +446,7 @@ need confirmation.
5. Replay the same client mix. Passing evidence is that none of the kernels
recorded in step 2 compiles after the sweep. A kernel that still compiles
names a bucket the sweep does not cover; record the arm-to-kernel mapping
and extend the sweep before claiming coverage. Kernels this profile has
compiled during serving include `_prepare_dflash_inputs_kernel`,
`_compute_global_topk_indices_and_lens_kernel` and the B12X CuTeDSL MoE
kernels. The summary's `skipped` entries state which arms the nonce or
request limit excluded.
and extend the sweep before claiming coverage.
6. Compare the same request mix with the phase disabled and enabled: TP rank
disconnects and restarts, and first-request time to first token. Record
conditions, measurements and the conclusion with the deployment record.
Expand Down
4 changes: 1 addition & 3 deletions docs/operations/install-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,9 +248,7 @@ are host settings that SparkRing neither applies nor has measured
resumes the switch. Another profile or checkpoint replaces it instead: after
the GPU check, the installer stops the unfinished model through its own
deployment and switches from the last model that served.
`sudo sparkring install --cancel` settles it without installing anything: it
stops the unfinished candidate on every Spark, starts the model the switch
replaced again and records the switch as `cancelled`
`sudo sparkring install --cancel` settles it without installing anything
([cancel an installation](commands.md#cancel-an-installation)).
- **Selected model's last start did not complete**, for example when a first
installation failed its readiness check or a restart of the installed model
Expand Down
6 changes: 3 additions & 3 deletions profiles/deepseek-v4-flash-0731-pair/runtime.env.template
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ MAX_MODEL_LEN={{max_model_len}}
MAX_NUM_SEQS={{max_num_seqs}}
MAX_NUM_BATCHED_TOKENS={{max_num_batched_tokens}}

# Request-shape warmup runs on rank 0 after the API reports ready when
# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is
# calibrated on the deployed image; read
# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after
# the API reports ready. Leave it at 0 until the sweep is calibrated on the
# deployed image; read
# docs/operations/deepseek-0731.md#26-request-shape-warmup first.
# SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound
# readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer
Expand Down
6 changes: 3 additions & 3 deletions profiles/deepseek-v4-flash-0731/runtime.env.template
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ MAX_MODEL_LEN={{max_model_len}}
MAX_NUM_SEQS={{max_num_seqs}}
MAX_NUM_BATCHED_TOKENS={{max_num_batched_tokens}}

# Request-shape warmup runs on rank 0 after the API reports ready when
# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is
# calibrated on the deployed image; read
# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after
# the API reports ready. Leave it at 0 until the sweep is calibrated on the
# deployed image; read
# docs/operations/deepseek-0731.md#26-request-shape-warmup first.
# SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound
# readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer
Expand Down
7 changes: 4 additions & 3 deletions runtime/host/cancel.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ def entries(state_root):


def request(state_root, operations, *, now=None, who=None):
"""Record that the installations ``operations`` must stop at their next step boundary."""
"""Record that the installations named in ``operations`` must stop at their next step boundary."""
ids = sorted({str(item) for item in operations if item})
if not ids:
raise ValueError("No installation is running to cancel")
Expand Down Expand Up @@ -157,8 +157,9 @@ def run(state_root, *, now=None, who=None, say=print, sleep=time.sleep, active=N
``switches`` the model switches this command settled, ``busy`` the slots
another operation kept, and ``message`` the one line an operator reads.
``state`` is ``requested`` while a running installation finishes a step it
cannot abandon, ``failed`` when a switch this command settled still needs
attention, and otherwise ``cancelled`` or ``idle``.
cannot abandon, ``busy`` when another command holds the cluster, ``failed``
when a switch this command settled still needs attention, and otherwise
``cancelled`` or ``idle``.
"""
from runtime.host import placement as placements, slices
state_root = Path(state_root)
Expand Down
2 changes: 1 addition & 1 deletion runtime/host/controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -769,7 +769,7 @@ def _lifecycle(argv, *, output, locks):
# An operator's hold (runtime.host.hold) comes first: other operators' changes refuse while it stands.
result["hold"] = hold.current(STATE)
from runtime.host import rollout
# A model switch that stopped before it finished, and the step that settles it.
# A model switch that has not settled.
switches = rollout.unfinished(STATE)
if switches:
result["switches"] = [{"placement": list(slot) if slot else None,
Expand Down
9 changes: 4 additions & 5 deletions runtime/host/install_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -1732,8 +1732,8 @@ def prepare(path):
unfinished=installer.unfinished, placement=placement, displaced=displaced,
cancelled=lambda boundary: cancel.check(state_root, operation_id, boundary))
finally:
# Every step a cancel names is behind this run either way, so its
# request must not stand for a later installation.
# Every step a cancel can name belongs to this run, so the request
# must not stand for a later installation.
cancel.clear(state_root, operation_id)
# Records the deployment's choice (on unless --no-auto-recover, also
# after sparkring recover off), resets failures and restarts, records
Expand Down Expand Up @@ -1864,9 +1864,8 @@ def main(argv=None):
args.serving_text = {name: value for name, value in vars(sizes).items() if value is not None}
colour.set_mode(args.color)
if args.cancel:
# Ending an installation changes nothing an operator holds: the request
# takes no installation lock and no hold override, and the command
# carries no other option.
# The request takes no installation lock and no hold override, and the
# command accepts no other option.
used = sorted(name for name, value in vars(args).items()
if name not in ("cancel", "json", "color") and value not in (None, False, {}, []))
if used:
Expand Down
3 changes: 2 additions & 1 deletion runtime/host/operator_view.py
Original file line number Diff line number Diff line change
Expand Up @@ -207,10 +207,11 @@ def lines(document: dict, *, now=time.time) -> list[str]:
+ (f", {recovery_record['failures']} failed attempt(s)" if recovery_record.get("failures") else ""))
if not document.get("deployments"):
result.append("No model deployment is recorded.")
from runtime.host import rollout
for switch in document.get("switches") or []:
where = ("Sparks " + ",".join(map(str, switch["placement"]))
if switch.get("placement") else "every Spark")
name = f"{switch['previous']} to {switch['candidate']}" if switch.get("previous") else switch.get("candidate")
result.append(f"Model switch: {name} on {where}, recorded {switch['state']}"
" | next: an operator's sudo sparkring install --cancel")
" | next: " + rollout.SWITCH_NEXT)
return result
10 changes: 5 additions & 5 deletions runtime/host/rollout.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,9 @@ def execute(directory, previous, *, state_root, prepare, apply, verify, supersed

``cancelled(boundary)`` is asked before each step the switch records, and
raises ``cancel.Cancelled`` to stop it there. A switch that has not yet
stopped anything records ``cancelled``; one that has brings back every
deployment it stopped first, and one whose restore fails records
``needs-attention`` with the restore's error.
stopped anything records ``cancelled``; one that has stopped something
brings back every deployment it stopped first, and one whose restore fails
records ``needs-attention`` with the restore's error.
"""
directory = Path(directory).resolve()
previous = Path(previous).resolve() if previous else None
Expand Down Expand Up @@ -237,15 +237,15 @@ def unfinished(state_root):


def switch_line(slot, record, size=None):
"""The first half of one ``sparkring status`` line for a model switch in progress or stopped."""
"""``Model switch: OLD to NEW on Sparks ...`` and the record's state, for a switch in progress or stopped."""
candidate = Path(record.get("candidate") or "").name
previous = Path(record["previous"]).name if record.get("previous") else None
switch = f"{previous} to {candidate}" if previous else candidate
return (f"Model switch: {switch} on {placements.text(slot, size)}, "
f"recorded {record.get('state')}")


# What settles a switch that stopped before it finished, appended to switch_line's line.
# The action that settles a switch which stopped before it finished: cancel it, or install that model again.
SWITCH_NEXT = "sudo sparkring install --cancel stops it and records it cancelled, or install that model again to resume it"


Expand Down
6 changes: 3 additions & 3 deletions scripts/config/deepseek-v4-flash-0731-pair.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ MAX_MODEL_LEN=1048576
MAX_NUM_SEQS=32
MAX_NUM_BATCHED_TOKENS=4096

# Request-shape warmup runs on rank 0 after the API reports ready when
# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is
# calibrated on the deployed image; read
# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after
# the API reports ready. Leave it at 0 until the sweep is calibrated on the
# deployed image; read
# docs/operations/deepseek-0731.md#26-request-shape-warmup first.
# SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound
# readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer
Expand Down
6 changes: 3 additions & 3 deletions scripts/config/deepseek-v4-flash-0731.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ MAX_MODEL_LEN=1048576
MAX_NUM_SEQS=32
MAX_NUM_BATCHED_TOKENS=4096

# Request-shape warmup runs on rank 0 after the API reports ready when
# SHAPE_WARMUP=1 is present in this file. It is disabled until the sweep is
# calibrated on the deployed image; read
# With SHAPE_WARMUP=1 in this file, rank 0 runs a request-shape warmup after
# the API reports ready. Leave it at 0 until the sweep is calibrated on the
# deployed image; read
# docs/operations/deepseek-0731.md#26-request-shape-warmup first.
# SHAPE_WARMUP_WAIT_SECONDS (900) and SHAPE_WARMUP_TIMEOUT_SECONDS (600) bound
# readiness and the sweep, and SHAPE_WARMUP_API_KEY_FILE names a rank-0 bearer
Expand Down
2 changes: 1 addition & 1 deletion scripts/deepseek_v4_cycle_serve.sh
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ if [ "$NODE_RANK" = 0 ] && [ "$SHAPE_WARMUP" = 1 ]; then
shape_warmup_args+=(--api-key-file "$SHAPE_WARMUP_API_KEY_FILE")
fi
printf 'Shape warmup: waiting for rank 0 health, then sweeping request shapes.\n'
# The phase is non-fatal: a failure keeps the boot and only logs a warning.
# Non-fatal: a sweep failure warns and the container keeps serving.
if ! command -v python3 >/dev/null 2>&1; then
echo "deepseek cycle launcher: warning: python3 is unavailable; request-shape warmup skipped" >&2
elif ! python3 "$script_dir/deepseek_v4_shape_warmup.py" "${shape_warmup_args[@]}"; then
Expand Down
2 changes: 1 addition & 1 deletion scripts/deepseek_v4_pair_serve.sh
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,7 @@ if [ "$NODE_RANK" = 0 ] && [ "$SHAPE_WARMUP" = 1 ]; then
shape_warmup_args+=(--api-key-file "$SHAPE_WARMUP_API_KEY_FILE")
fi
printf 'Shape warmup: waiting for rank 0 health, then sweeping request shapes.\n'
# The phase is non-fatal: a failure keeps the boot and only logs a warning.
# Non-fatal: a sweep failure warns and the container keeps serving.
if ! command -v python3 >/dev/null 2>&1; then
echo "deepseek pair launcher: warning: python3 is unavailable; request-shape warmup skipped" >&2
elif ! python3 "$script_dir/deepseek_v4_shape_warmup.py" "${shape_warmup_args[@]}"; then
Expand Down
8 changes: 4 additions & 4 deletions scripts/deepseek_v4_shape_warmup.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
after its API reports ready and before the launcher hands the endpoint to
clients.

The bucket rule is provisional. Measurements on an earlier DeepSeek-V4-Flash
serving image found the DSpark input-preparation kernel keyed by
The bucket rule is unconfirmed on any pinned image. A measurement on a serving
image this repository does not pin (its digest was not recorded) found the
DSpark input-preparation kernel keyed by
``min(block_size, next_pow2(scheduled_tokens + num_speculative_tokens + 1))``,
not by request concurrency; those measurements used a different serving image
than this repository pins. The sweep covers that rule, and the calibration
not by request concurrency. The sweep covers that rule, and the calibration
procedure in ``docs/operations/deepseek-0731.md`` establishes the actual keys
on a deployment before it enables the phase by default.

Expand Down
Loading