Conversation
Register llama.cpp in the shared engine table and give it a facade on the unified proxy at 8080, with the PAIR-managed engine on 8081. Engine Manager installs the official llama app into an owned directory from a checksum-pinned installer or archive recipe (Windows ARM64 tries the current upstream build first and falls back to pinned CUDA archives; confirmed non-NVIDIA ARM and Intel Macs use the pinned CPU recipes), starts and stops it, keeps a private model cache, and exposes download, import, load, unload, delete and download cancellation. Uninstall removes only the runtime slots and keeps models. Routing eligibility for llama.cpp is loaded models only, because the engine runs with --no-models-autoload. The proxy keeps per-facade cancellation isolated and delivers exactly one terminal workload event even when a committed upstream stream is truncated; the classification of that case follows the foundation's existing mid-stream test. The broker advertises the engine, probes manual peers for it, and applies the same engine-settings port path it uses for Ollama and LM Studio, made profile-generic so a third engine does not fall through two-engine branches. llama.cpp declares reviewed launch controls (--port, --host) with serve --no-models-autoload fixed; its owned cache environment is injected on every launch rather than edited, and the settings preview refuses attempts to set it. This is the focused integration replayed onto the public develop foundation from the internal review branch. Runtime updating is not offered for llama.cpp (the desktop bridge refuses update rather than substituting uninstall and reinstall), and the GPU/OS telemetry work is kept as a separate patch candidate. Co-authored-by: Terve <ntervalon@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
validateLlamaPath on Unix resolved the whole path with EvalSymlinks and returned nil as soon as the leaf did not exist, so a symlinked ancestor of a not-yet-created runtime slot, model file or archive entry was never inspected, and a dangling link at the leaf passed. The Windows guard already walks every component up to the volume root and refuses any reparse point regardless of whether the leaf exists. Make the Unix guard do the same with Lstat per component: skip missing components, refuse a symlink anywhere in the chain, return other errors. Existing real paths keep the strictness they had. The fixture covers an ordinary tree, a plain missing leaf, a models slot redirected outside the managed root, missing children beneath it, and a dangling leaf, and checks the outside marker is untouched. It carries the !windows build tag; on this Windows host it was compile-checked with GOOS=linux and GOOS=darwin vet and test builds, not executed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
Register the engine type, display name and links; map the llamacpp-proxy relay source; carry the engine manager's install-support, install-reason and managed facts through the bridge so an external llama.cpp runtime is shown read-only and the install decision follows the manager rather than the displayed OS; keep cached models distinct from loaded ones; and route model import, load, delete and download cancellation through the owning engine manager, locally and on peers. The settings editor develop introduced is reused for llama.cpp and hidden for an external runtime PAIR does not own. The bridge refuses the update command for llama.cpp instead of letting it fall into the generic uninstall-then-install pair; remote update is refused for every engine as before. Workload identity carries the engine and proxy run so two engines' equal request ids stay distinct. The service contract inventory is regenerated for the new methods. Co-authored-by: Terve <ntervalon@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
Describe the managed llama.cpp engine in the product docs: install, lifecycle, model actions, loaded-only routing, launch settings, data retention on uninstall, and the terminal interface keys. The inference dispatcher recognises the engine, the app-data wipe scripts preserve an unmigrated llama model library instead of deleting it, and shell scripts are checked out with LF endings so Windows-built packages ship executable Unix scripts. Co-authored-by: Terve <ntervalon@nvidia.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
Defer optional desktop/TUI inference-cancel controls and expanded workload browsing while retaining full workload identity and engine/model controls. Restore the public-base live workload table. Consume llama proxy relay events, nested Engine Manager residency events, and LM Studio model keys. Cover the real consumer boundaries with socket-free regressions for all three engines. Signed-off-by: pgoode41 <pgoode41@gmail.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
The desktop's service-binary cache keys on a content hash of the Go sources and module files, but nvpair-engine-manager compiles manifests/*.json into the binary (`//go:embed manifests/*.json`). A manifest-only change, such as a pin bump or a launch default, therefore produced a different binary while the cache still reported cli-bin as current, so a developer or packaging run could ship stale engine behaviour without noticing. Hash the embedded manifests with the sources. Verified by building twice (rebuild, then skip), appending one byte to llamacpp.json (rebuild), and restoring it (rebuild back to the earlier fingerprint). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com> (cherry picked from commit 2dae370af14c1662d0f05c6f3ba63b4e030396bd)
Each fix was verified against source and carries a regression test.
- The paired-node settings relay dropped every llama.cpp snapshot, so a
peer's llama editor never received live updates; relay every engine in the
shared table.
- The llama.cpp advertiser polled without the node configuration lock its
siblings hold, so a tick could re-register lc between a settings apply's
withdraw and restart.
- The copied engine:set-port reservation guard was unreachable because the
settings path intercepts the method before the relay; removed with its
parser.
- The registry pinned one vendor build's literals (archive root name, archive
count, platform keys); validate the recipe shape and leave the exact pins to
the tests so a pin bump is a manifest change.
- Install and uninstall discarded the Detect result after promoting or
removing the runtime; both now fail like the generic paths.
- The fake engine gained the vendor's `download` shape so pull_model, preset
rejection and cancel_pull are exercised; public Uninstall for llama.cpp is
covered for runtime-only removal, model retention, saved Off and refusal
while another process serves the port.
- The broker engine-table tests pin llama.cpp's ownership, probe path,
occupied-facade outcome and the third restore gate.
- The node list offered Start/Stop for an observe-only external runtime and
Install where the node reported no recipe; one shared predicate now gates
the engine row, model manager and node list, and an unreachable Cancel
button is gone.
- Documentation drift: manual-probe port, engine default port, removed
workload cancellation, executable count, facade relay list, profile and
advertiser comments. spec.md and MANIFEST.md now name the bundled driver,
builtins, identity probe, archive recipe fields, {model_dir} and dotted
result fields as the fail-closed exception to manifest-only onboarding, and
EngineStatus's install and managed fields.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
(cherry picked from commit 9845b82963d827e3abddb3668edf673ab5c770ce)
Managed llama.cpp launched with --no-models-autoload, so a downloaded model was not served until someone loaded it explicitly, and the stack grew a second routing rule to match: the proxy consulted a loaded-only inventory for this one engine, the residency watcher published "unknown" whenever llama missed an observation, the manual-node probe filtered the router's list down to loaded ids, and the shared node record carried a loaded-set accessor used by the proxy alone. Run the vendor default instead: `serve --models-autoload` is now a fixed argument (stated explicitly so a LLAMA_ARG_MODELS_AUTOLOAD in the user's environment cannot flip it). A cached model loads on the first request that names it, which is how Ollama and LM Studio already behave through PAIR, so the per-engine eligibility switch, the loaded-set accessor and both filters go away and every facade routes on the advertised catalog. The loaded-model watcher and `loadedByEngine` stay for the UI's residency dot and its Load/Eject controls. Documentation follows: the explicit-load guidance becomes the on-demand behaviour (the first response waits for the load; the router keeps up to four models resident and evicts idle ones), the stale claim that a llama request can be cancelled from the workload card is dropped because that cancellation was deferred in dd958ba, and the managed engine port is 8081, not 8082. A persisted settings switch for turning on-demand loading off is deliberately not part of this change; it follows separately. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
Two sentences still described llama.cpp routing on the loaded set after 931647a made every facade route on the advertised catalog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: pgoode41 <pgoode41@gmail.com>
Ignore per-engine hydration results when a newer state notification arrived while the request was in flight. This prevents an already-running engine from displaying Start and then stopping when toggled. Cover both running and stopped updates for llama.cpp, Ollama, and LM Studio, plus a later uncontested refresh. Native UI reproduction retained privately; fixed-build UI retest remains pending. Signed-off-by: pgoode41 <pgoode41@gmail.com>
The official Windows installer selects its CUDA build only when the CUDA Toolkit is installed, so NVIDIA PCs with just the driver received the Vulkan build. When every NVIDIA GPU reports compute capability 7.5 or newer, Install now stages the checksum-pinned b10826 CUDA 13.3 app and CUDA runtime archives and requires a CUDA device before promotion. Any other outcome runs the pinned vendor installer as before and records the reason in the install receipt. Signed-off-by: pgoode41 <pgoode41@gmail.com>
Signed-off-by: pgoode41 <pgoode41@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog title
Managed llama.cpp engine
Changelog body
Bumps
Summary
Add managed llama.cpp using the existing Engine Manager, unified proxy and
engine-settings UI. No new service process, no parallel settings implementation,
and no llama.cpp-specific routing rule.
download/import/load/unload/delete models; cancel model downloads.
every detected NVIDIA GPU has compute capability 7.5 or newer, and require a
working CUDA device probe before promotion. Otherwise retain vendor selection
and record why CUDA was not used; no local CUDA Toolkit installation is required.
read-only rather than taking ownership of another installation.
serve --models-autoloadas afixed argument, so a request for a downloaded model loads it first (the first
response waits for the load; the router keeps up to four models resident and
evicts idle ones). Routing eligibility follows the advertised catalog, exactly
as it does for Ollama and LM Studio; the loaded-model watcher remains for the
UI's residency indicator and Load/Eject controls.
exactly-once workload terminal delivery, with correct workload attribution.
for live proxy events and observed model residency.
Excluded: a per-engine switch to turn on-demand loading off (with
loaded-only routing), engine updating, GPU/OS telemetry or chart redesign,
expanded workload browsing, and explicit inference-workload cancellation
controls/API. Those are independent follow-ups, not prerequisites for this
integration.
Non-goals and known limits
Vulkan installation to the CUDA selection path, uninstall and reinstall the
managed runtime; model files are retained.
llama.cpp capabilities (staged official-app installer, on-disk model cache
operations, router identity probe); they are engine-specific Go accepted only
for the bundled manifest and are recorded in
spec.mdas a current gap.Studio's with two policy differences; sharing it is left for the next
managed engine.
that request while the router finishes loading; retry or load explicitly.
Review First
Start with
services/nvpair-engine-manager/manifests/llamacpp.json(fixedserve --models-autoload, declared--host/--portcontrols), thenservices/nvpair-proxy/proxy.gosubscribedToNode(one eligibility rule forevery facade),
services/nvpair-manual-nodes/manager.goprobeLlamaCpp, andthe settings regressions in
settings_llama_test.goandservices/nvpair-ui-broker/enginesettings_test.go.Validation
81624e8: desktop checks, SPDX, release intent,build-script checks on three operating systems, the full Linux Go component,
cross-process and tooling suites, and six platform installer builds.
26271a0: all 15 Go modules build and vet. Full Engine Manager,broker, TUI, manual-node, cluster-manager, scheduler, workload-manager, errors,
node-settings, shared and EAP suites pass; focused scanner/node-info and the
cross-process external-listener refusal check pass. Network-test limits below.
26271a0: 260 tests pass, 2 existing skips; typecheck, lint,dead-code, service contracts, build-script and SPDX checks pass. Go binaries
and Electron production bundles build.
26271a0: fresh managed CUDA installation passed.Four consecutive full GPU journeys passed after one earlier streaming-reload
timeout. The passes cover cold/warm and streamed inference, unload/reload,
unknown-model rejection, restart persistence, workload attribution and cleanup.
The initial timeout remains recorded; its cause is not established.
f305c3b: actual Load/Unload,Start/Stop, proxy inference, completed-workload display and clean exit passed.
Desktop code is unchanged by
26271a0; this is not a CUDA desktop-UI test.26271a0: fresh managed install, model pull and firstproxied inference passed, followed by warm inference, a completed long stream,
streamed reload after unload, unknown-model rejection, restart persistence and
five locally attributed completed workloads. Owned compute ran on the selected
GPU; memory returned to baseline, both brokers exited normally, test ports and
owned processes closed, and temporary state was removed. Unrelated GPU process
identities were preserved. The earlier interrupted harness run is retained.
This establishes the Linux GPU lifecycle, but retained diagnostics do not yet
establish the exact CUDA backend or offloaded-layer count.
x64/arm64 and macOS x64/arm64 on
81624e8; these are build results, not nativehardware qualification for every platform.
snapshots from replacing newer running/stopped events; regression checks pass.
81624e8corrects an assertion that assumed Windowsenvironment-name case folding on Unix. Canonical cache overrides remain
rejected on every platform; lowercase aliases are additionally rejected on
Windows. Production code is unchanged from the hardware-tested
26271a0.Remaining Qualification
managed GPU lifecycle and owned-process/data cleanup passed separately.
isolated Linux CI runner. The local Windows gate used the focused network
selection described above; this does not claim full Windows network-suite coverage.
later full runs or a targeted reload probe, but is not explained or erased.
Terve adapted the earlier integration onto the unified proxy. The earlier
llama.cpp work includes the proxy adaptation from #41; #18 and #36 provided
additional design/review context. This does not represent those PRs as merged
unchanged or claim authorship of their underlying shared PAIR infrastructure.
Checklist