Skip to content

fix(engine-manager): fail closed on unpinned downloads and escalate engine stop - #107

Closed
woodsonl wants to merge 1 commit into
NVIDIA:developfrom
woodsonl:fix/engine-manager-install-and-stop
Closed

woodsonl wants to merge 1 commit into
NVIDIA:developfrom
woodsonl:fix/engine-manager-install-and-stop

Conversation

@woodsonl

@woodsonl woodsonl commented Sep 22, 2026 •

Copy link
Copy Markdown

Description

Unpinned downloads fail closed. download() executed any HTTPS artifact
whose manifest carried no sha256. A manifest without a digest means any host
that serves that URL yields code execution on the machine. It now returns an
error naming the computed digest, so the manifest author can pin it, and the
checksum-mismatch error names the same escape hatch so a stale pin is not a
dead end. Operators who accept unverified downloads can opt in with
NVPAIR_ALLOW_UNPINNED_DOWNLOADS=1. The bundled Ollama and LM Studio fetch
URLs are rolling, so their manifests carry no pin and install through that
opt-in; the check exists so a manifest can add one.

Engine stop escalates. stop() sent one graceful signal and then waited
forever, so an engine that ignored SIGTERM was never stopped. It now waits
the manifest's stop grace (default 5s) and escalates to a forced kill of the
whole process group or tree, which reaches engines that forked helper processes
such as model runners. Both waits are bounded, so a forced kill that itself
fails cannot hang shutdown either.

Release intent

Changelog title

Engine installs require a pinned digest, and stop escalates

Changelog body

Engine downloads without a pinned sha256 in their manifest are now refused by
default; an operator can opt in to unverified downloads explicitly. Stopping an
engine that ignores the graceful signal now escalates to a forced kill after
the manifest's grace period instead of waiting indefinitely.

Bumps

  • services: patch
  • nvpair-cluster-manager: none
  • nvpair-engine-manager: patch
  • nvpair-errors: none
  • nvpair-job-scheduler: none
  • nvpair-manual-nodes: none
  • nvpair-node-info: none
  • nvpair-node-scanner: none
  • nvpair-node-settings: none
  • nvpair-proxy: none
  • nvpair-tui: none
  • nvpair-ui-broker: none
  • nvpair-workload-manager: none

Scope

Included: the download pin check and its opt-out, the stop escalation, the
manifest pins, and the tests for both.

Excluded: the install and stop commands themselves are unchanged apart from the
pin check and the added escalation step.

Validation

  • go build ./... and go test ./... in services/nvpair-engine-manager
  • services/nvpair-engine-manager/remediation_test.go and launch_test.go
    cover the changed install and stop paths.

Risk

Behavior change for operators with unpinned manifests: a download that used to
succeed now fails until the manifest is pinned or the opt-out is set. The
grace-period escalation force-kills an engine that has not exited; that is the
intended behavior and matches what Windows already did.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.

Two engine-manager changes:

- download() executed any HTTPS artifact whose manifest had no sha256 pin.
  A manifest without a digest means any host serving that URL yields code
  execution, so fail closed instead: return an error naming the computed
  digest so the author can pin it. Operators who accept unverified
  downloads can opt in with NVPAIR_ALLOW_UNPINNED_DOWNLOADS=1. The
  checksum-mismatch error names the same escape hatch, so a stale pin is
  not a dead end. The bundled Ollama and LM Studio fetch URLs are rolling,
  so their manifests stay unpinned and install through the escape hatch;
  the pin exists for manifests that add one.

- stop() sent one graceful signal and then waited forever, so an engine
  that ignored SIGTERM was never stopped. It waits the manifest's stop
  grace (default 5s) and escalates to a forced kill of the whole process
  group/tree, which reaches engines that forked helper processes. Both
  waits are bounded, so a forced kill that itself fails cannot hang
  shutdown either. A test drives a process that ignores SIGTERM and
  asserts stop ends it after the grace.

Signed-off-by: woodsonl <65194841+woodsonl@users.noreply.github.com>
@woodsonl woodsonl closed this Sep 24, 2026
@woodsonl
woodsonl deleted the fix/engine-manager-install-and-stop branch September 24, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant