Skip to content

Add managed llama.cpp engine with Windows CUDA support - #116

Open
nv-pgoode wants to merge 13 commits into
NVIDIA:developfrom
nv-pgoode:codex/llamacpp-core-minimal
Open

nv-pgoode wants to merge 13 commits into
NVIDIA:developfrom
nv-pgoode:codex/llamacpp-core-minimal

Conversation

@nv-pgoode

@nv-pgoode nv-pgoode commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Changelog title

Managed llama.cpp engine

Changelog body

  • Install and manage llama.cpp from PAIR, with models retained across runtime uninstall.
  • Route llama.cpp requests through the unified proxy to any downloaded model; the engine loads it on demand.
  • Prefer the official CUDA build on compatible Windows x64 NVIDIA systems, with a recorded fallback when CUDA cannot be used.

Bumps

  • services: minor
  • nvpair-cluster-manager: none
  • nvpair-engine-manager: minor
  • nvpair-errors: none
  • nvpair-job-scheduler: minor
  • nvpair-manual-nodes: minor
  • nvpair-node-info: minor
  • nvpair-node-scanner: minor
  • nvpair-node-settings: none
  • nvpair-proxy: minor
  • nvpair-tui: minor
  • nvpair-ui-broker: minor
  • nvpair-workload-manager: none

Summary

Add managed llama.cpp using the existing Engine Manager, unified proxy and
engine-settings UI. No new service process, no parallel settings implementation,
and no llama.cpp-specific routing rule.

  • Install the official llama app into owned runtime storage; start/stop it;
    download/import/load/unload/delete models; cancel model downloads.
  • On Windows x64, stage the pinned official CUDA app and runtime archives when
    every detected NVIDIA GPU has compute capability 7.5 or newer, and require a
    working CUDA device probe before promotion. Otherwise retain vendor selection
    and record why CUDA was not used; no local CUDA Toolkit installation is required.
  • Keep model files when uninstalling the runtime. Treat external runtimes as
    read-only rather than taking ownership of another installation.
  • Use the existing port/settings flow: proxy 8080, engine 8081 by default.
  • Load models on demand: the managed router runs serve --models-autoload as a
    fixed argument, so a request for a downloaded model loads it first (the first
    response waits for the load; the router keeps up to four models resident and
    evicts idle ones). Routing eligibility follows the advertised catalog, exactly
    as it does for Ollama and LM Studio; the loaded-model watcher remains for the
    UI's residency indicator and Load/Eject controls.
  • Preserve client-disconnect and shutdown cleanup, retry/CORS behavior and
    exactly-once workload terminal delivery, with correct workload attribution.
  • Add desktop and TUI engine/model controls, including producer-shaped tests
    for live proxy events and observed model residency.

Excluded: a per-engine switch to turn on-demand loading off (with
loaded-only routing), engine updating, GPU/OS telemetry or chart redesign,
expanded workload browsing, and explicit inference-workload cancellation
controls/API. Those are independent follow-ups, not prerequisites for this
integration.

Non-goals and known limits

  • Existing installed runtimes keep their current backend. To move an existing
    Vulkan installation to the CUDA selection path, uninstall and reinstall the
    managed runtime; model files are retained.
  • Engine Manager's manifest-only onboarding requirement is not met for three
    llama.cpp capabilities (staged official-app installer, on-disk model cache
    operations, router identity probe); they are engine-specific Go accepted only
    for the bundled manifest and are recorded in spec.md as a current gap.
  • The broker's managed-facade coordinator for llama.cpp is a copy of LM
    Studio's with two policy differences; sharing it is left for the next
    managed engine.
  • A cold load longer than the proxy's two-minute first-response budget fails
    that request while the router finishes loading; retry or load explicitly.

Review First

Start with services/nvpair-engine-manager/manifests/llamacpp.json (fixed
serve --models-autoload, declared --host/--port controls), then
services/nvpair-proxy/proxy.go subscribedToNode (one eligibility rule for
every facade), services/nvpair-manual-nodes/manager.go probeLlamaCpp, and
the settings regressions in settings_llama_test.go and
services/nvpair-ui-broker/enginesettings_test.go.

Validation

  • All 13 GitHub checks pass on 81624e8: desktop checks, SPDX, release intent,
    build-script checks on three operating systems, the full Linux Go component,
    cross-process and tooling suites, and six platform installer builds.
  • Local Windows checks on 26271a0: all 15 Go modules build and vet. Full Engine Manager,
    broker, TUI, manual-node, cluster-manager, scheduler, workload-manager, errors,
    node-settings, shared and EAP suites pass; focused scanner/node-info and the
    cross-process external-listener refusal check pass. Network-test limits below.
  • Desktop on 26271a0: 260 tests pass, 2 existing skips; typecheck, lint,
    dead-code, service contracts, build-script and SPDX checks pass. Go binaries
    and Electron production bundles build.
  • Windows x64 CUDA on 26271a0: fresh managed CUDA installation passed.
    Four consecutive full GPU journeys passed after one earlier streaming-reload
    timeout. The passes cover cold/warm and streamed inference, unload/reload,
    unknown-model rejection, restart persistence, workload attribution and cleanup.
    The initial timeout remains recorded; its cause is not established.
  • Windows CPU Desktop/TUI on predecessor f305c3b: actual Load/Unload,
    Start/Stop, proxy inference, completed-workload display and clean exit passed.
    Desktop code is unchanged by 26271a0; this is not a CUDA desktop-UI test.
  • Linux x64 NVIDIA on 26271a0: fresh managed install, model pull and first
    proxied inference passed, followed by warm inference, a completed long stream,
    streamed reload after unload, unknown-model rejection, restart persistence and
    five locally attributed completed workloads. Owned compute ran on the selected
    GPU; memory returned to baseline, both brokers exited normally, test ports and
    owned processes closed, and temporary state was removed. Unrelated GPU process
    identities were preserved. The earlier interrupted harness run is retained.
    This establishes the Linux GPU lifecycle, but retained diagnostics do not yet
    establish the exact CUDA backend or offloaded-layer count.
  • GitHub Actions built unsigned installers for Windows x64/arm64, Linux
    x64/arm64 and macOS x64/arm64 on 81624e8; these are build results, not native
    hardware qualification for every platform.
  • The separately committed desktop startup repair prevents late initial
    snapshots from replacing newer running/stopped events; regression checks pass.
  • Test-only follow-up 81624e8 corrects an assertion that assumed Windows
    environment-name case folding on Unix. Canonical cache overrides remain
    rejected on every platform; lowercase aliases are additionally rejected on
    Windows. Production code is unchanged from the hardware-tested 26271a0.

Remaining Qualification

  • Linux exact CUDA-backend/offloaded-layer evidence remains unqualified; the
    managed GPU lifecycle and owned-process/data cleanup passed separately.
  • The full proxy, scanner, node-info and cross-process suites passed on the
    isolated Linux CI runner. The local Windows gate used the focused network
    selection described above; this does not claim full Windows network-suite coverage.
  • CUDA desktop-UI validation, Windows arm64, Linux arm64 and macOS hardware qualification remain.
  • The first Windows CUDA streaming-reload timeout was not reproduced in four
    later full runs or a targeted reload probe, but is not explained or erased.

Terve adapted the earlier integration onto the unified proxy. The earlier
llama.cpp work includes the proxy adaptation from #41; #18 and #36 provided
additional design/review context. This does not represent those PRs as merged
unchanged or claim authorship of their underlying shared PAIR infrastructure.

Checklist

  • Contribution rules read; every commit has a matching DCO sign-off.
  • Required tests and documentation updated; no new dependency added.
  • Changed source/filenames/messages checked for private data and internal links.
  • Release intent declared; bot-owned versions and changelog unchanged.
  • Windows retained-CPU desktop/TUI functional journeys completed.
  • Remaining platform/GPU qualification above completed.

pgoode41 and others added 13 commits September 22, 2026 03:44
Register llama.cpp in the shared engine table and give it a facade on
the unified proxy at 8080, with the PAIR-managed engine on 8081. Engine
Manager installs the official llama app into an owned directory from a
checksum-pinned installer or archive recipe (Windows ARM64 tries the
current upstream build first and falls back to pinned CUDA archives;
confirmed non-NVIDIA ARM and Intel Macs use the pinned CPU recipes),
starts and stops it, keeps a private model cache, and exposes download,
import, load, unload, delete and download cancellation. Uninstall
removes only the runtime slots and keeps models.

Routing eligibility for llama.cpp is loaded models only, because the
engine runs with --no-models-autoload. The proxy keeps per-facade
cancellation isolated and delivers exactly one terminal workload event
even when a committed upstream stream is truncated; the classification
of that case follows the foundation's existing mid-stream test. The
broker advertises the engine, probes manual peers for it, and applies
the same engine-settings port path it uses for Ollama and LM Studio,
made profile-generic so a third engine does not fall through two-engine
branches. llama.cpp declares reviewed launch controls (--port, --host)
with serve --no-models-autoload fixed; its owned cache environment is
injected on every launch rather than edited, and the settings preview
refuses attempts to set it.

This is the focused integration replayed onto the public develop
foundation from the internal review branch. Runtime updating is not
offered for llama.cpp (the desktop bridge refuses update rather than
substituting uninstall and reinstall), and the GPU/OS telemetry work is
kept as a separate patch candidate.

Co-authored-by: Terve <ntervalon@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
validateLlamaPath on Unix resolved the whole path with EvalSymlinks and
returned nil as soon as the leaf did not exist, so a symlinked ancestor
of a not-yet-created runtime slot, model file or archive entry was never
inspected, and a dangling link at the leaf passed. The Windows guard
already walks every component up to the volume root and refuses any
reparse point regardless of whether the leaf exists.

Make the Unix guard do the same with Lstat per component: skip missing
components, refuse a symlink anywhere in the chain, return other errors.
Existing real paths keep the strictness they had.

The fixture covers an ordinary tree, a plain missing leaf, a models slot
redirected outside the managed root, missing children beneath it, and a
dangling leaf, and checks the outside marker is untouched. It carries the
!windows build tag; on this Windows host it was compile-checked with
GOOS=linux and GOOS=darwin vet and test builds, not executed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
Register the engine type, display name and links; map the llamacpp-proxy
relay source; carry the engine manager's install-support, install-reason
and managed facts through the bridge so an external llama.cpp runtime is
shown read-only and the install decision follows the manager rather than
the displayed OS; keep cached models distinct from loaded ones; and route
model import, load, delete and download cancellation through the owning
engine manager, locally and on peers. The settings editor develop
introduced is reused for llama.cpp and hidden for an external runtime
PAIR does not own.

The bridge refuses the update command for llama.cpp instead of letting
it fall into the generic uninstall-then-install pair; remote update is
refused for every engine as before. Workload identity carries the
engine and proxy run so two engines' equal request ids stay distinct.
The service contract inventory is regenerated for the new methods.

Co-authored-by: Terve <ntervalon@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
Describe the managed llama.cpp engine in the product docs: install,
lifecycle, model actions, loaded-only routing, launch settings, data
retention on uninstall, and the terminal interface keys. The inference
dispatcher recognises the engine, the app-data wipe scripts preserve an
unmigrated llama model library instead of deleting it, and shell scripts
are checked out with LF endings so Windows-built packages ship
executable Unix scripts.

Co-authored-by: Terve <ntervalon@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
Defer optional desktop/TUI inference-cancel controls and expanded workload browsing while retaining full workload identity and engine/model controls. Restore the public-base live workload table.

Consume llama proxy relay events, nested Engine Manager residency events, and LM Studio model keys. Cover the real consumer boundaries with socket-free regressions for all three engines.

Signed-off-by: pgoode41 <pgoode41@gmail.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
The desktop's service-binary cache keys on a content hash of the Go sources
and module files, but nvpair-engine-manager compiles manifests/*.json into
the binary (`//go:embed manifests/*.json`). A manifest-only change, such as a
pin bump or a launch default, therefore produced a different binary while
the cache still reported cli-bin as current, so a developer or packaging run
could ship stale engine behaviour without noticing.

Hash the embedded manifests with the sources. Verified by building twice
(rebuild, then skip), appending one byte to llamacpp.json (rebuild), and
restoring it (rebuild back to the earlier fingerprint).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
(cherry picked from commit 2dae370af14c1662d0f05c6f3ba63b4e030396bd)
Each fix was verified against source and carries a regression test.

- The paired-node settings relay dropped every llama.cpp snapshot, so a
  peer's llama editor never received live updates; relay every engine in the
  shared table.
- The llama.cpp advertiser polled without the node configuration lock its
  siblings hold, so a tick could re-register lc between a settings apply's
  withdraw and restart.
- The copied engine:set-port reservation guard was unreachable because the
  settings path intercepts the method before the relay; removed with its
  parser.
- The registry pinned one vendor build's literals (archive root name, archive
  count, platform keys); validate the recipe shape and leave the exact pins to
  the tests so a pin bump is a manifest change.
- Install and uninstall discarded the Detect result after promoting or
  removing the runtime; both now fail like the generic paths.
- The fake engine gained the vendor's `download` shape so pull_model, preset
  rejection and cancel_pull are exercised; public Uninstall for llama.cpp is
  covered for runtime-only removal, model retention, saved Off and refusal
  while another process serves the port.
- The broker engine-table tests pin llama.cpp's ownership, probe path,
  occupied-facade outcome and the third restore gate.
- The node list offered Start/Stop for an observe-only external runtime and
  Install where the node reported no recipe; one shared predicate now gates
  the engine row, model manager and node list, and an unreachable Cancel
  button is gone.
- Documentation drift: manual-probe port, engine default port, removed
  workload cancellation, executable count, facade relay list, profile and
  advertiser comments. spec.md and MANIFEST.md now name the bundled driver,
  builtins, identity probe, archive recipe fields, {model_dir} and dotted
  result fields as the fail-closed exception to manifest-only onboarding, and
  EngineStatus's install and managed fields.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
(cherry picked from commit 9845b82963d827e3abddb3668edf673ab5c770ce)
Managed llama.cpp launched with --no-models-autoload, so a downloaded model
was not served until someone loaded it explicitly, and the stack grew a
second routing rule to match: the proxy consulted a loaded-only inventory for
this one engine, the residency watcher published "unknown" whenever llama
missed an observation, the manual-node probe filtered the router's list down
to loaded ids, and the shared node record carried a loaded-set accessor used
by the proxy alone.

Run the vendor default instead: `serve --models-autoload` is now a fixed
argument (stated explicitly so a LLAMA_ARG_MODELS_AUTOLOAD in the user's
environment cannot flip it). A cached model loads on the first request that
names it, which is how Ollama and LM Studio already behave through PAIR, so
the per-engine eligibility switch, the loaded-set accessor and both filters
go away and every facade routes on the advertised catalog. The loaded-model
watcher and `loadedByEngine` stay for the UI's residency dot and its
Load/Eject controls.

Documentation follows: the explicit-load guidance becomes the on-demand
behaviour (the first response waits for the load; the router keeps up to four
models resident and evicts idle ones), the stale claim that a llama request
can be cancelled from the workload card is dropped because that cancellation
was deferred in dd958ba, and the managed engine port is 8081, not 8082.

A persisted settings switch for turning on-demand loading off is deliberately
not part of this change; it follows separately.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
Two sentences still described llama.cpp routing on the loaded set after
931647a made every facade route on the advertised catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
Ignore per-engine hydration results when a newer state notification arrived while the request was in flight. This prevents an already-running engine from displaying Start and then stopping when toggled.

Cover both running and stopped updates for llama.cpp, Ollama, and LM Studio, plus a later uncontested refresh. Native UI reproduction retained privately; fixed-build UI retest remains pending.

Signed-off-by: pgoode41 <pgoode41@gmail.com>
The official Windows installer selects its CUDA build only when the CUDA
Toolkit is installed, so NVIDIA PCs with just the driver received the
Vulkan build. When every NVIDIA GPU reports compute capability 7.5 or
newer, Install now stages the checksum-pinned b10826 CUDA 13.3 app and
CUDA runtime archives and requires a CUDA device before promotion. Any
other outcome runs the pinned vendor installer as before and records the
reason in the install receipt.

Signed-off-by: pgoode41 <pgoode41@gmail.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants