Skip to content

Let an operator cancel an installation, or settle the switch it left - #393

Merged
FujitsuPolycom merged 2 commits into
mainfrom
issue-329-install-cancel
Nov 1, 2026
Merged

FujitsuPolycom merged 2 commits into
mainfrom
issue-329-install-cancel

Conversation

@FujitsuPolycom

@FujitsuPolycom FujitsuPolycom commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Closes #329.

What and why

sudo sparkring install --cancel stops the installation that is running. It records the live operation ids, the installation reads its own entry before each model-switch step, stops the candidate through that deployment's own stop, starts the model the switch replaced, and records the switch as cancelled with the time and the operator. The active pointer never leaves the model that served.

With no installation running, the same command settles a switch that SIGTERM, Ctrl-C or a Spark restart left unfinished, holding that slot's admission while it stops the candidate and starts the replaced model. That replaces finding the candidate's directory and running sudo sparkring down PROFILE --instance i<hash> --execute by hand.

sparkring status and the group-readable operator view print Model switch: ... | next: ... while a switch is in progress or after it stopped, naming install --cancel or installing the model again.

  • A failed restore records needs-attention, which another installation already refuses to replace without approval. The commit step itself is not cancelable.
  • A cancelled switch reserves no Spark (slices.affected), is not a switching rollback target (checkpoints.roles), and never blocks a later install (placements.PENDING).
  • --cancel takes no other option and no hold override, and refuses while up, down or restart holds the cluster.

Validation

  • pytest runtime/host scripts/test_sparkring_cli.py: 2084 passed, 86 skipped.
  • ruff check --select E,F,W, scripts/check_repository_layout.py, scripts/check_markdown_links.py ., scripts/check_release_safety.py .: clean.
  • runtime/host/test_cancel.py covers each boundary (preparing, stopping the replaced model, starting, verifying, committing), a same-model restart, a resumed switch, a failed restore, a settle refused by another operation's admission, a settle that loses its journal to another cancel, a committed switch's active record, a settle whose restore fails (exit 2, state failed, the restore error in switches[].error), and the --cancel JSON and exit contract.
  • Not run: a cancel against live Sparks, and a cancel during a real checkpoint download. The stack was busy; the deployment adapters are exercised with recorded adapters, not dockerd and real deployments.

Compatibility

  • New records: install-cancel.json (sparkring-install-cancel/v1, mode 0600) and the --cancel result (sparkring-install-cancel-result/v1).
  • The install transaction gains state cancelled. An older package reading it finds a state outside its pending tuple, so it replaces the record rather than resuming it; its checkpoint and retention roles keep the candidate as a rollback target, which is conservative and unchanged for interrupted switches.
  • Exit codes: the interrupted installation exits 2 with state cancelled; --cancel exits 0 when it asks, stops or settles a switch, and 2 when another command holds the cluster or a switch it tried to settle still needs attention (enforced by runtime/host/test_cancel.py).
  • Docs: docs/operations/commands.md (flag, cancel section, status) and docs/operations/install-reference.md (interrupted switch).

Known limits

  • Preparation is a single step: a cancel during the image sync or the checkpoint download takes effect when that step ends, not mid-transfer.
  • A switch killed between writing the candidate's active record and recording completion is settled by --cancel, but the slots it displaced are not re-recorded as active; sudo sparkring up --on ARC records one again. sparkring status shows the affected slot without an active deployment meanwhile.

`sudo sparkring install --cancel` stops the installation that is running
instead of leaving its candidate's containers and an abandoned switch record.
The installation reads the request before each model-switch step, stops the
candidate through that deployment's own stop, starts the model the switch
replaced, and records the switch as `cancelled` with the time and the
operator; the active pointer stays on the model that served. A restore that
fails records `needs-attention`, which another installation already refuses to
replace without approval. The commit itself is not a cancel boundary.

With no installation running, the command settles a switch left unfinished by
SIGTERM, Ctrl-C or a Spark restart, holding that slot's admission while it
stops the candidate and starts the replaced model again. `sparkring status`
and the group-readable operator view report a switch in progress or one that
stopped. A `cancelled` switch reserves no Spark and keeps no deployment as a
rollback target, and each request keeps the operator and time that asked.

Validation: pytest runtime/host (2082 passed, 86 skipped),
scripts/test_sparkring_cli.py, ruff E,F,W, layout and link checks. Not run: a
cancel against live Sparks; the deployment adapters are exercised with
recorded adapters rather than dockerd and real deployments.
`install --cancel` treats every settled switch as cancelled, so a restore that
failed leaves the switch recorded `needs-attention` while the command exits 0,
contrary to its documented exit contract. A settled switch now carries the
restore error, and the command exits 2 with state `failed` and a message naming
the switch and the error.

Validation: pytest runtime/host/test_cancel.py runtime/host/test_rollout.py
runtime/host/test_install_workflow.py (162 passed); ruff check --select E,F,W
--ignore E501 runtime scripts; scripts/check_repository_layout.py;
scripts/check_markdown_links.py .
@FujitsuPolycom
FujitsuPolycom merged commit 9288f6b into main Nov 1, 2026
10 checks passed
@FujitsuPolycom
FujitsuPolycom deleted the issue-329-install-cancel branch November 1, 2026 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide a supported way to cancel an in-flight sparkring install

1 participant