Repository navigation
Let an operator cancel an installation, or settle the switch it left - #393
Merged
Merged
Conversation
`sudo sparkring install --cancel` stops the installation that is running instead of leaving its candidate's containers and an abandoned switch record. The installation reads the request before each model-switch step, stops the candidate through that deployment's own stop, starts the model the switch replaced, and records the switch as `cancelled` with the time and the operator; the active pointer stays on the model that served. A restore that fails records `needs-attention`, which another installation already refuses to replace without approval. The commit itself is not a cancel boundary. With no installation running, the command settles a switch left unfinished by SIGTERM, Ctrl-C or a Spark restart, holding that slot's admission while it stops the candidate and starts the replaced model again. `sparkring status` and the group-readable operator view report a switch in progress or one that stopped. A `cancelled` switch reserves no Spark and keeps no deployment as a rollback target, and each request keeps the operator and time that asked. Validation: pytest runtime/host (2082 passed, 86 skipped), scripts/test_sparkring_cli.py, ruff E,F,W, layout and link checks. Not run: a cancel against live Sparks; the deployment adapters are exercised with recorded adapters rather than dockerd and real deployments.
`install --cancel` treats every settled switch as cancelled, so a restore that failed leaves the switch recorded `needs-attention` while the command exits 0, contrary to its documented exit contract. A settled switch now carries the restore error, and the command exits 2 with state `failed` and a message naming the switch and the error. Validation: pytest runtime/host/test_cancel.py runtime/host/test_rollout.py runtime/host/test_install_workflow.py (162 passed); ruff check --select E,F,W --ignore E501 runtime scripts; scripts/check_repository_layout.py; scripts/check_markdown_links.py .
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #329.
What and why
sudo sparkring install --cancelstops the installation that is running. It records the live operation ids, the installation reads its own entry before each model-switch step, stops the candidate through that deployment's own stop, starts the model the switch replaced, and records the switch ascancelledwith the time and the operator. The active pointer never leaves the model that served.With no installation running, the same command settles a switch that
SIGTERM,Ctrl-Cor a Spark restart left unfinished, holding that slot's admission while it stops the candidate and starts the replaced model. That replaces finding the candidate's directory and runningsudo sparkring down PROFILE --instance i<hash> --executeby hand.sparkring statusand the group-readable operator view printModel switch: ... | next: ...while a switch is in progress or after it stopped, naminginstall --cancelor installing the model again.needs-attention, which another installation already refuses to replace without approval. The commit step itself is not cancelable.cancelledswitch reserves no Spark (slices.affected), is not aswitchingrollback target (checkpoints.roles), and never blocks a later install (placements.PENDING).--canceltakes no other option and no hold override, and refuses whileup,downorrestartholds the cluster.Validation
pytest runtime/host scripts/test_sparkring_cli.py: 2084 passed, 86 skipped.ruff check --select E,F,W,scripts/check_repository_layout.py,scripts/check_markdown_links.py .,scripts/check_release_safety.py .: clean.runtime/host/test_cancel.pycovers each boundary (preparing, stopping the replaced model, starting, verifying, committing), a same-model restart, a resumed switch, a failed restore, a settle refused by another operation's admission, a settle that loses its journal to another cancel, a committed switch's active record, a settle whose restore fails (exit 2, statefailed, the restore error inswitches[].error), and the--cancelJSON and exit contract.Compatibility
install-cancel.json(sparkring-install-cancel/v1, mode 0600) and the--cancelresult (sparkring-install-cancel-result/v1).cancelled. An older package reading it finds a state outside its pending tuple, so it replaces the record rather than resuming it; its checkpoint and retention roles keep the candidate as a rollback target, which is conservative and unchanged for interrupted switches.cancelled;--cancelexits 0 when it asks, stops or settles a switch, and 2 when another command holds the cluster or a switch it tried to settle still needs attention (enforced byruntime/host/test_cancel.py).docs/operations/commands.md(flag, cancel section, status) anddocs/operations/install-reference.md(interrupted switch).Known limits
--cancel, but the slots it displaced are not re-recorded as active;sudo sparkring up --on ARCrecords one again.sparkring statusshows the affected slot without an active deployment meanwhile.