diff --git a/.cursor/rules/proxy-inference-routing.mdc b/.cursor/rules/proxy-inference-routing.mdc index 9dabf40b..87d9697e 100644 --- a/.cursor/rules/proxy-inference-routing.mdc +++ b/.cursor/rules/proxy-inference-routing.mdc @@ -10,7 +10,7 @@ SPDX-License-Identifier: Apache-2.0 # Proxy and Inference Routing Routing is owned by `nvpair-proxy` (one process hosting a facade per engine, -addressed by clients as `ollama-proxy:` / `lmstudio-proxy:`), +addressed by clients as `ollama-proxy:` / `lmstudio-proxy:` / `llamacpp-proxy:`), `nvpair-job-scheduler`, and `nvpair-ui-broker`. For model-bearing inference, each facade first filters a request-local discovery diff --git a/.cursor/rules/system-architecture.mdc b/.cursor/rules/system-architecture.mdc index 7b3b0f3b..45e4ab0b 100644 --- a/.cursor/rules/system-architecture.mdc +++ b/.cursor/rules/system-architecture.mdc @@ -28,7 +28,7 @@ Broker-owned workers: - `nvpair-proxy`, one process hosting a facade per enabled engine. It starts with no engine and no listener; the broker sends a `facade/enable` per engine carrying that engine's port. Clients still address each facade as - `ollama-proxy:` / `lmstudio-proxy:`, and one supervisor covers them all, so a + `ollama-proxy:` / `lmstudio-proxy:` / `llamacpp-proxy:`, and one supervisor covers them all, so a crash is reported against `nvpair-proxy` and restarts every facade together; - `nvpair-node-scanner`; - `nvpair-node-info`; diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000..a727c7f1 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,5 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Windows-built packages must ship executable Unix shell scripts. +*.sh text eol=lf diff --git a/README.md b/README.md index 757c3f39..0e2930ad 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ one, and both report live GPU and memory use throughout. | **Architectures** | x64 and arm64 on all three. Windows on ARM is experimental. | | **Installers** | Windows `.exe`; Linux `.deb`; macOS `.dmg`. On other Linux distributions, [build from source](docs/building.mdx). | | **Mixing nodes** | Windows, Linux, and macOS nodes can all be paired with each other | -| **Inference engines** | Ollama and LM Studio | +| **Inference engines** | Ollama, LM Studio and managed llama.cpp | **PAIR running on a machine does not mean an engine will.** PAIR itself runs on any supported Windows, Linux, or macOS machine. Each engine sets its own requirements @@ -49,6 +49,11 @@ before assuming a node can serve a model. A node only becomes a candidate for a request once it is actually running a compatible engine, and PAIR prefers the nodes it already knows hold the model. +Managed llama.cpp provides an official CPU app for Intel Macs and confirmed +non-NVIDIA Windows ARM hardware. NVIDIA Windows ARM remains CUDA-required; +a failed driver or hardware query does not silently select CPU. Apple Silicon +uses the official Metal app. See the [managed engine requirements and policies](services/nvpair-engine-manager/README.md#managed-llama-app). + ## Quick start Download a released build and use the desktop application. That is the path we diff --git a/desktop/docs/architecture.md b/desktop/docs/architecture.md index 0874b5ff..5b41875d 100644 --- a/desktop/docs/architecture.md +++ b/desktop/docs/architecture.md @@ -109,7 +109,7 @@ subscribes to broker relays after `app:ready`, and converts backend responses into stable UI contracts. Electron reports the service connected after broker `app:ready`. The -broker-owned Ollama and LM Studio proxies remain asynchronous capabilities; a +broker-owned Ollama, LM Studio, and llama.cpp proxies remain asynchronous capabilities; a late or failed proxy does not misreport the broker startup as failed. If `app:ready` does not arrive within the startup deadline, Overview opens Settings @@ -227,16 +227,18 @@ ordinary environment assignments can be edited locally or by a pinned peer. authoritative settings operation rather than forwarding to the engine manager, so both entry points validate, restart, and persist identically. -The Ollama and LM Studio proxies are cluster-aware. For model-bearing inference, -each proxy first keeps only nodes whose per-engine discovery inventory advertises -the requested model. Empty and non-matching inventories are excluded; an empty -owner set returns a local `502`. Routing precedence within the eligible set is: +The Ollama, LM Studio, and llama.cpp proxies are cluster-aware. For +model-bearing inference, each proxy first keeps only nodes whose per-engine +discovery inventory advertises the requested model. Empty and non-matching +inventories are excluded; an +empty owner set returns a local `502`. Routing precedence within the eligible set +is: 1. a user-selected manual node; 2. the priority list emitted by `nvpair-job-scheduler`; 3. the proxy's deterministic default ordering. -The scheduler combines total pending (queued and running) workload across both +The scheduler combines total pending (queued and running) workload across all engines with a smoothed 0–3 pressure derived from the busiest GPU. Missing, invalid, or older-than-10-second telemetry has neutral pressure. It emits the order, pending count, and pressure, reranking on meaningful workload, discovery, @@ -298,6 +300,12 @@ cannot yet be reported are centralized in `src/shared/constants/modular-runtime.ts`. - Ollama-compatible clients use the proxy port reported by the broker. +- llama.cpp clients use the OpenAI-compatible proxy at `http://127.0.0.1:8080/v1` + by default. Engine Manager installs the official `llama` app, serves its + managed router on the separately reported engine port (default `8081`), and + owns model download, load/unload and removal. The proxy routes to the models + the engine advertises; a cold model loads on its first request. Existing external listeners remain externally owned; + their presence does not authorize PAIR to mutate them. - Cluster pairing currently uses port `14321`. - Node telemetry is read from `/v1/node-info` at each discovered node's advertised port. diff --git a/desktop/docs/service-contract-exceptions.json b/desktop/docs/service-contract-exceptions.json index 90ae4eec..523b9040 100644 --- a/desktop/docs/service-contract-exceptions.json +++ b/desktop/docs/service-contract-exceptions.json @@ -9,6 +9,7 @@ "engine:restore-enabled": "Broker-internal startup restoration. nvpair-ui-broker emits engine:restore-enabled directly to its supervised engine-manager after the managed Ollama port gate and on manager respawn; it is not a renderer/UI notification.", "ollama-proxy:ready": "Consumed, not missing: the broker relays it and normalizeBrokerProxy (modular-supervisor.ts) strips the `ollama-proxy:` prefix, so the bridge handles the de-prefixed `ready` (sets proxyPort). The literal `ollama-proxy:ready` is intentionally absent from our TS — extractor limitation, not a gap.", "lmstudio-proxy:ready": "Consumed, not missing: the LM Studio counterpart of ollama-proxy:ready, de-prefixed by the same normalizeBrokerProxy loop. It only became visible to the checker when METHOD_RE started accepting hyphens in a namespace segment; before that the whole lmstudio-proxy:* surface was silently unmatched.", + "llamacpp-proxy:ready": "Consumed, not missing: the llama.cpp counterpart of the two above, de-prefixed by the same normalizeBrokerProxy loop, which iterates PROXY_NODE_SOURCES and so covers every engine without naming one. llama.cpp ships no proxy binary — one nvpair-proxy process hosts a facade per engine — but `llamacpp-proxy` remains the facade's relay namespace, which is what the broker emits and this entry names.", "node/selection-changed": "Automatic routing has no selected-node UI, so PAIR deliberately does not consume proxy selection changes.", "proxy/request": "Per-request proxy telemetry is not rendered; workload lifecycle uses the broker workloads stream.", "proxy/request-started": "Per-request proxy telemetry is not rendered; see proxy/request.", diff --git a/desktop/docs/services-api.md b/desktop/docs/services-api.md index 7aea40f9..4c2e914f 100644 --- a/desktop/docs/services-api.md +++ b/desktop/docs/services-api.md @@ -47,6 +47,9 @@ - ⚠️ nvpair-ui-broker → engine:set-reserved-port - ⚠️ nvpair-ui-broker → engine:unsubscribe - ⚠️ nvpair-ui-broker → internal:set-reserved-port +- ⚠️ nvpair-ui-broker → llamacpp-proxy:get-status +- ⚠️ nvpair-ui-broker → llamacpp-proxy:set-port +- ⚠️ nvpair-ui-broker → llamacpp-proxy:unsubscribe - ⚠️ nvpair-ui-broker → lmstudio-proxy:get-status - ⚠️ nvpair-ui-broker → lmstudio-proxy:set-port - ⚠️ nvpair-ui-broker → lmstudio-proxy:unsubscribe @@ -109,6 +112,7 @@ | `engine:prepare-shutdown` | request (we call) | ✅ yes | | `engine:preview-launch` | request (we call) | ⚠️ not called | | `engine:remote-apply-settings` | request (we call) | ⚠️ not called | +| `engine:remote-cancel-pull` | request (we call) | ✅ yes | | `engine:remote-delete-model` | request (we call) | ✅ yes | | `engine:remote-get-installed` | request (we call) | ✅ yes | | `engine:remote-get-settings` | request (we call) | ⚠️ not called | @@ -233,6 +237,7 @@ | `cluster:identity-changed` | request (we call) | ✅ yes | | `cluster:invite-received` | request (we call) | ✅ yes | | `engine:install-progress` | request (we call) | ✅ yes | +| `engine:models-changed` | request (we call) | ✅ yes | | `engine:pull-progress` | request (we call) | ✅ yes | | `engine:state-changed` | request (we call) | ✅ yes | | `error` | request (we call) | ✅ yes | @@ -253,6 +258,7 @@ | `errors:clear` | notification (we consume) | ✅ yes | | `errors:report` | notification (we consume) | ✅ yes | | `errors:update` | notification (we consume) | ✅ yes | +| `llamacpp-proxy:ready` | notification (we consume) | ➖ ignored | | `lmstudio-proxy:ready` | notification (we consume) | ➖ ignored | | `ollama-proxy:ready` | notification (we consume) | ➖ ignored | | `workloads:upsert` | notification (we consume) | ✅ yes | @@ -273,6 +279,10 @@ | `engine:unsubscribe` | request (we call) | ⚠️ not called | | `errors:get-initial` | request (we call) | ✅ yes | | `internal:set-reserved-port` | request (we call) | ⚠️ not called | +| `llamacpp-proxy:get-status` | request (we call) | ⚠️ not called | +| `llamacpp-proxy:set-port` | request (we call) | ⚠️ not called | +| `llamacpp-proxy:subscribe` | request (we call) | ✅ yes | +| `llamacpp-proxy:unsubscribe` | request (we call) | ⚠️ not called | | `lmstudio-proxy:get-status` | request (we call) | ⚠️ not called | | `lmstudio-proxy:set-port` | request (we call) | ⚠️ not called | | `lmstudio-proxy:subscribe` | request (we call) | ✅ yes | diff --git a/desktop/docs/services-backend.md b/desktop/docs/services-backend.md index 25d8328c..b7ab5607 100644 --- a/desktop/docs/services-backend.md +++ b/desktop/docs/services-backend.md @@ -89,8 +89,8 @@ engine, workload, cluster, and error relays. The bridge then emits renderer push events from backend notifications. Connector readiness follows the broker contract: `app:ready` establishes the -service connection, while Ollama and LM Studio proxy readiness remains an -asynchronous capability signal. Personal AI Router waits up to the canonical +service connection, while Ollama, LM Studio, and llama.cpp proxy readiness +remains an asynchronous capability signal. Personal AI Router waits up to the canonical startup deadline in `src/shared/constants/modular-runtime.ts` for `app:ready`; an outright failure or stalled broker startup is surfaced in Settings > Service with retry and log access. If a stalled broker reports ready @@ -116,7 +116,7 @@ reserved for inference clients. | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | `app:ready` | Complete broker startup and refresh snapshots | `state:request-refresh` | | `discovery:nodes-changed` | Replace discovery snapshot and diff nodes | `discovery:nodes-changed`, `nodes:upsert`, `nodes:remove` | -| `ollama-proxy:ready` / `lmstudio-proxy:ready` | Record engine proxy port | `engines:state-changed` | +| `ollama-proxy:ready` / `lmstudio-proxy:ready` / `llamacpp-proxy:ready` | Record engine proxy port | `engines:state-changed` | | proxy `node/*` | Update per-engine node presence; the advertised port is the peer's promoted proxy port (not the engine's private loopback port) | node and engine pushes | | `engine:ready` / `engine:state-changed` | Update engine facts and models | `engines:state-changed` | | `engine:settings-changed` | Validate and republish the owning node's settings snapshot | `engines:settings-changed` | @@ -128,7 +128,7 @@ reserved for inference clients. | `nodes:changed` | Replace membership snapshot | `nodes:changed` | | `workloads:upsert` / `workloads:remove` | Update workload catalog | workload pushes | -`nvpair-job-scheduler` combines queued and running work across both engines with +`nvpair-job-scheduler` combines queued and running work across all engines with a smoothed 0–3 pressure from the busiest GPU. Invalid, missing, or older-than-10-second telemetry receives neutral pressure. It emits `schedule:priority` with order, pending count, and pressure; the broker applies @@ -150,8 +150,14 @@ waiting for authoritative state. Pending state clears on matching engine state, progress, or error pushes. Local engine operations include install, start, stop, uninstall, update, port -changes, and model actions. Remote cluster operations use the engine manager's -remote control surface where supported. +changes, and model actions. Engine Manager owns the official llama app install, +router process, managed cache, downloads, load/unload and delete. llama.cpp has +no managed update: the bridge refuses `update` for it rather than substituting +an uninstall and reinstall. The desktop uses reported install support and +ownership; external runtimes remain read-only. Downloaded cache entries remain +distinct from runtime residency. The app endpoint is `http://127.0.0.1:8080/v1`; +a downloaded model loads on the first request that names it. Remote cluster +operations use the engine manager's remote control surface where supported. ### Engine settings diff --git a/desktop/docs/services-parity.md b/desktop/docs/services-parity.md index c0b8cbc5..8702ca5b 100644 --- a/desktop/docs/services-parity.md +++ b/desktop/docs/services-parity.md @@ -24,6 +24,7 @@ history. | Manual nodes | Complete with local persistence | Broker owns probing and proxy registration; Electron persists entries for replay | | Ollama routing | Complete | Broker relay and backend scheduler drive proxy routing | | LM Studio routing | Complete | Parallel broker relay and scheduler path | +| llama.cpp routing | Source integration; native validation pending | `nvpair-proxy` facade on llama.cpp's own `8080`, relayed as `llamacpp-proxy:` | | Local engine lifecycle | Complete | Install, start, stop, uninstall, update, and port configuration | | Remote engine lifecycle | Partial | Remote install, start, stop, status, and model pull are supported | | Engine models | Partial | Core list, pull, load, unload, and supported delete actions are wired | @@ -97,23 +98,50 @@ Manual nodes use the broker's `node/add`, `node/remove`, and `nodes/list` surface. Electron persists user entries and replays them after broker startup so they survive worker restarts. +### Workload display + +The desktop displays workload snapshots and live updates, retaining origin, +engine, proxy run and request identity. Execution labels and connection lines +use the reported destination, not the request origin. Workload cancellation is +not exposed by the desktop API or UI. Engine lifecycle and model-download +cancellation are separate controls and remain supported. + +### Multi-node UI acceptance + +Engine integration must preserve each participating desktop's view of the +cluster, not only the request origin's view. During the same bounded inference +run, verify every available participating desktop independently: + +- Cluster and member UUIDs, engine availability and loaded-model ownership agree + after discovery converges; offline members do not look live or routable. +- New workloads agree by their full `(originatedFrom, engine, runId, id)` + identity on model, destination and terminal state. Record propagation delay; + do not require identical historical catalog totals or instantaneous equality + during a state transition. +- Capture each desktop's original Performance view during actual work. A remote + backend response or the origin's aggregate UI does not prove another native UI. + +Record unavailable or untested desktop cells explicitly. This checklist states +the acceptance requirement; it does not assert that every platform has passed. + ## Routing and inference -Both text-engine facades are broker-owned and cluster-aware. They live in one +Every text-engine facade is broker-owned and cluster-aware. They live in one `nvpair-proxy` process, each enabled after spawn on its own port, and each serves its engine's dialect: - the Ollama facade serves the Ollama-compatible surface; -- the LM Studio facade serves the LM Studio/OpenAI-compatible surface. +- the LM Studio facade serves the LM Studio/OpenAI-compatible surface; +- the llama.cpp facade serves its OpenAI-compatible surface on `8080`. Sharing a process is what lets them share the burst reservations the scheduler -depends on: two facades bursting at once compete for the same node's GPU, so a -dispatch through either has to be visible to the other. +depends on: facades bursting at once compete for the same node's GPU, so a +dispatch through any of them has to be visible to the others. Routing precedence is manual selection, scheduler priority, then deterministic proxy ordering. Personal AI Router leaves proxies in automatic mode. -`nvpair-job-scheduler` combines total queued and running workload across both +`nvpair-job-scheduler` combines total queued and running workload across all engines with a smoothed 0–3 GPU-pressure signal. The backend scanner and manual node worker provide maximum-GPU utilization, while invalid, missing, or older-than-10-second samples receive neutral pressure. The scheduler emits order, @@ -159,6 +187,10 @@ Personal AI Router supports local: - desired-state restoration across app restarts; - engine and model progress. +Managed update covers Ollama and LM Studio. llama.cpp has no managed update; +the bridge refuses `update` for it instead of substituting an uninstall and +reinstall. + Before shutdown, Personal AI Router calls `engine:prepare-shutdown`. This stops managed engine processes without changing the persisted desired state; the broker restores enabled engines on the next launch. The broker also self-initiates @@ -209,9 +241,12 @@ Personal AI Router uses: - `list_models`; - `pull_model`; - Ollama `run_model`, `unload_model` (`keep_alive: 0`), and `delete_model`; -- LM Studio `load_model`, `unload_model`, and `delete_model` (`remove_path`). +- LM Studio `load_model`, `unload_model`, and `delete_model` (`remove_path`); +- llama.cpp `load_model`, `unload_model` (router `/models/load` and + `/models/unload`, settled on observed residency), `delete_model`, + `pull_model` and `import_model` (managed cache builtin), and `cancel_pull`. -Both engines expose Load, Eject, and Delete in the model manager when the +All three engines expose Load, Eject, and Delete in the model manager when the backend action exists. Keep-alive / expiry controls remain unsupported. LM Studio's `delete_model` declares `restart_after`, so the engine manager diff --git a/desktop/electron-builder.config.ts b/desktop/electron-builder.config.ts index 1f713d47..957ea06e 100644 --- a/desktop/electron-builder.config.ts +++ b/desktop/electron-builder.config.ts @@ -389,6 +389,21 @@ const config: Configuration = { } }, deb: { + // Explicit depends replace electron-builder defaults. Preserve its current + // runtime set, adding GBM and the ALSA SONAME on both pre/post-t64 distros. + depends: [ + 'libgtk-3-0', + 'libnotify4', + 'libnss3', + 'libxss1', + 'libxtst6', + 'xdg-utils', + 'libatspi2.0-0', + 'libuuid1', + 'libsecret-1-0', + 'libgbm1', + 'libasound2t64 | libasound2' + ], afterInstall: 'scripts/build/linux/after-install.sh', afterRemove: 'scripts/build/linux/after-remove.sh' }, diff --git a/desktop/scripts/build-modular-binaries.ts b/desktop/scripts/build-modular-binaries.ts index e84b4995..ed3167c0 100644 --- a/desktop/scripts/build-modular-binaries.ts +++ b/desktop/scripts/build-modular-binaries.ts @@ -211,6 +211,11 @@ function listFingerprintFiles(repo: string): string[] { out.push(full) } else if (entry === 'go.mod' || entry === 'go.sum') { out.push(full) + } else if (entry.endsWith('.json') && path.basename(dir) === 'manifests') { + // Engine manifests are compiled into nvpair-engine-manager + // (`//go:embed manifests/*.json`), so a manifest-only change + // produces a different binary and must miss the cache. + out.push(full) } } } @@ -218,7 +223,7 @@ function listFingerprintFiles(repo: string): string[] { return out.sort() } -/** Content hash of services Go sources + module files — not monorepo git HEAD. */ +/** Content hash of services Go sources, module files and embedded engine manifests — not monorepo git HEAD. */ function servicesSourceFingerprint(repo: string): string { const hash = createHash('sha256') for (const file of listFingerprintFiles(repo)) { diff --git a/desktop/scripts/build/installer.nsh b/desktop/scripts/build/installer.nsh index 1517393e..1914da07 100644 --- a/desktop/scripts/build/installer.nsh +++ b/desktop/scripts/build/installer.nsh @@ -71,8 +71,20 @@ DetailPrint "Removing Personal AI Router user data..." ClearErrors ReadEnvStr $0 LOCALAPPDATA - RMDir /r "$0\Nvidia Corporation\Personal AI Router" - RMDir /r "$0\NVIDIA Corporation\PAIR" + ; Engine Manager migrates old caches outside app data. An uninstall before + ; first launch must preserve unmigrated weights, not recursively delete them. + IfFileExists "$0\Nvidia Corporation\Personal AI Router\engine-bin\llamacpp\models\*.*" pairKeepCurrentModels + RMDir /r "$0\Nvidia Corporation\Personal AI Router" + Goto pairCurrentDataDone + pairKeepCurrentModels: + DetailPrint "Preserving app data containing unmigrated llama models. Reinstall and open PAIR to migrate the library." + pairCurrentDataDone: + IfFileExists "$0\NVIDIA Corporation\PAIR\engine-bin\llamacpp\models\*.*" pairKeepLegacyModels + RMDir /r "$0\NVIDIA Corporation\PAIR" + Goto pairLegacyDataDone + pairKeepLegacyModels: + DetailPrint "Preserving legacy app data containing unmigrated llama models." + pairLegacyDataDone: RMDir "$0\NVIDIA Corporation" RMDir /r "$0\nvpair-updater" ClearErrors @@ -108,7 +120,7 @@ IfFileExists "$0\nvpair-updater\*.*" 0 +2 StrCpy $9 "$9$\n$0\nvpair-updater" StrCmp $9 "" pairNoLeftover - MessageBox MB_OK|MB_ICONEXCLAMATION "Some Personal AI Router data could not be removed because files were still in use (for example, a running engine such as Ollama).$\n$\nClose those programs, then delete these folders manually:$9" + MessageBox MB_OK|MB_ICONEXCLAMATION "Some Personal AI Router data was retained because it contains unmigrated llama models or files still in use.$\n$\nDo not delete model folders manually. Reinstall and open the updated app to migrate the library before removing app data.$\n$9" pairNoLeftover: !macroend @@ -152,6 +164,7 @@ ; leaves rules pointing at binaries this version no longer ships. nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Ollama Proxy"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router LM Studio Proxy"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router llama.cpp Proxy"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Node Info"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Node Scanner"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router Workload Manager"' @@ -161,6 +174,7 @@ nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS (UDP 5353)"' ; Pre-unification; see above. nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS LM Studio Proxy (UDP 5353)"' + nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS llama.cpp Proxy (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Node Info (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Node Scanner (UDP 5353)"' nsExec::ExecToLog 'netsh advfirewall firewall delete rule name="Personal AI Router mDNS Workload Manager (UDP 5353)"' diff --git a/desktop/scripts/build/linux/after-remove.sh b/desktop/scripts/build/linux/after-remove.sh index f506a05d..d06449ae 100644 --- a/desktop/scripts/build/linux/after-remove.sh +++ b/desktop/scripts/build/linux/after-remove.sh @@ -69,8 +69,13 @@ case "${1:-}" in # AI Router" (backend base = $XDG_CONFIG_HOME or ~/.config). The living, # append-only inventory is scripts/wipe-app-data.sh — do not silently diverge. # Every delete is best-effort so a locked or missing path never aborts removal. - rm -rf "$user_home/.config/Nvidia Corporation/Personal AI Router" 2>/dev/null || true - rm -rf "$user_home/.config/NVIDIA Corporation/PAIR" 2>/dev/null || true + for data_root in "$user_home/.config/Nvidia Corporation/Personal AI Router" "$user_home/.config/NVIDIA Corporation/PAIR"; do + if [ -e "$data_root/engine-bin/llamacpp/models" ] || [ -L "$data_root/engine-bin/llamacpp/models" ]; then + echo "Preserving unmigrated llama models in $data_root; reinstall and open PAIR before purging app data." >&2 + else + rm -rf "$data_root" 2>/dev/null || true + fi + done # Remove the current and previous parents only when empty so other NVIDIA # applications survive. rmdir "$user_home/.config/Nvidia Corporation" 2>/dev/null || true diff --git a/desktop/scripts/build/macos/uninstall.sh b/desktop/scripts/build/macos/uninstall.sh index ec948cc8..1bc6148e 100644 --- a/desktop/scripts/build/macos/uninstall.sh +++ b/desktop/scripts/build/macos/uninstall.sh @@ -49,6 +49,15 @@ fi APP_SUPPORT="$target_home/Library/Application Support" +if [ "$PURGE_DATA" = "1" ]; then + for root in "$APP_SUPPORT/Nvidia Corporation/Personal AI Router" "$APP_SUPPORT/NVIDIA Corporation/PAIR"; do + if [ -e "$root/engine-bin/llamacpp/models" ] || [ -L "$root/engine-bin/llamacpp/models" ]; then + echo "Llama models remain under app data. Open the updated app to migrate them before purging: $root" >&2 + exit 1 + fi + done +fi + echo "Stopping Personal AI Router processes..." # Keep this list in sync with MODULAR_RUNTIME_BINARIES and # MODULAR_BUNDLED_BINARIES in src/shared/constants/modular-binaries.ts, plus the diff --git a/desktop/scripts/generate-licenses.ts b/desktop/scripts/generate-licenses.ts index eb1ab9c1..1b61ef69 100644 --- a/desktop/scripts/generate-licenses.ts +++ b/desktop/scripts/generate-licenses.ts @@ -167,7 +167,10 @@ function renderMarkdown(entries: Entry[]): string { parts.push('```text', e.licenseText, '```', '') } - return `${parts.join('\n').trimEnd()}\n` + return `${parts + .join('\n') + .replace(/[\t ]+$/gm, '') + .trimEnd()}\n` } async function serviceComponentNames(): Promise { diff --git a/desktop/src/electron/model-hub/index.ts b/desktop/src/electron/model-hub/index.ts index d1fda727..570923bf 100644 --- a/desktop/src/electron/model-hub/index.ts +++ b/desktop/src/electron/model-hub/index.ts @@ -52,7 +52,7 @@ export async function getEngineHubModels(engineType: EngineType): Promise): ) break case 'update': + if (engine === 'llamacpp') { + // llama.cpp has no managed update, and the generic + // uninstall-then-install pair below must not stand in for one. + // Refuse without touching the runtime or beginning a pending op; + // the engineType context lets the renderer drop its optimistic + // lifecycle entry. + supervisor.reportError( + 'llama.cpp has no managed update; uninstall and reinstall the managed runtime instead.', + 'warning', + `engine-cmd:${payload.command}:${engine}`, + { engineType: payload.engineType, operation: 'update' } + ) + break + } // The engine-manager serializes per-engine ops via its lifecycle // lock, so the queued install waits for the uninstall to finish. The // uninstall's engine:state-changed briefly clears this, then the @@ -255,6 +269,29 @@ function routeEngineManagerCommand(payload: WsInvokeRequest<'engine:command'>): void supervisor.deleteModel(engine, payload.engineType, payload.model) } break + case 'cancelPull': + if (payload.model) { + supervisor.sendProcess( + 'broker', + 'engine:action', + { + engine, + action: 'cancel_pull', + params: { model: payload.model } + }, + failAction('cancel model download'), + // Observe the response like every other long-running engine + // action here: without it a backend refusal is indistinguishable + // from a cancel that worked. + true + ) + } + break + case 'importModel': + if (payload.model && engine === 'llamacpp') { + void supervisor.importLlamaModel(payload.model) + } + break case 'loadModel': // "Load" warms a model into the engine's memory/VRAM. Ollama has no // first-class load action, so we POST its `run_model` HTTP action @@ -282,6 +319,23 @@ function routeEngineManagerCommand(payload: WsInvokeRequest<'engine:command'>): }), true ) + } else if (payload.engineType === 'llamacpp') { + supervisor.sendProcess( + 'broker', + 'engine:action', + { + engine, + action: 'load_model', + params: { model: payload.model } + }, + failAction('load model', { + nodeId: payload.nodeId, + engineType: payload.engineType, + operation: 'load', + modelName: payload.model + }), + true + ) } else { supervisor.sendProcess( 'broker', @@ -371,6 +425,21 @@ function routeRemoteEngineCommand(payload: WsInvokeRequest<'engine:command'>): v `${payload.command} is only available on the local node — remote uninstall/update is not supported yet.` ) break + case 'cancelPull': + if (payload.model && engine === 'llamacpp') { + supervisor.sendProcess( + 'broker', + 'engine:remote-cancel-pull', + { + node: nodeId, + engine, + model: payload.model + }, + detail => refuseRemote(`Cancel download failed: ${detail}`), + true + ) + } + break case 'deleteModel': case 'loadModel': case 'unloadModel': diff --git a/desktop/src/electron/service-bridge/modular-state.ts b/desktop/src/electron/service-bridge/modular-state.ts index 151f1745..918a9019 100644 --- a/desktop/src/electron/service-bridge/modular-state.ts +++ b/desktop/src/electron/service-bridge/modular-state.ts @@ -35,10 +35,10 @@ import { emitBridgePush } from './broadcaster' import { mergePullProgressPercent } from './pull-error-handling' import type { JsonObject, JsonRpcNotification, JsonValue } from './json-rpc-subprocess' import { serviceLogLevel } from './service-log-level' -// Live node sources are the two reverse proxies, relayed through the broker, +// Live node sources are the reverse proxies, relayed through the broker, // and the broker's consolidated discovery snapshot. Electron does not consume // worker discovery protocols directly. -type ProxyNodeSource = 'ollama-proxy' | 'lmstudio-proxy' +type ProxyNodeSource = 'ollama-proxy' | 'lmstudio-proxy' | 'llamacpp-proxy' type BrokerNodeSource = ProxyNodeSource | 'broker' /** @@ -47,19 +47,30 @@ type BrokerNodeSource = ProxyNodeSource | 'broker' * recorded here. That is `ComponentName` in `services/shared/engines`, always * `-proxy`. */ -export const PROXY_NODE_SOURCES: readonly ProxyNodeSource[] = ['ollama-proxy', 'lmstudio-proxy'] +export const PROXY_NODE_SOURCES: readonly ProxyNodeSource[] = [ + 'ollama-proxy', + 'lmstudio-proxy', + 'llamacpp-proxy' +] /** * Engines surfaced by the broker's proxy plane. Other engine-manager engines * are not currently routed across nodes. */ -export type ProxyEngine = Extract -export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio'] +export type ProxyEngine = Extract +export const PROXY_ENGINES: readonly ProxyEngine[] = ['ollama', 'lm-studio', 'llamacpp'] /** Map a proxy node source onto the engine it describes. */ const PROXY_SOURCE_ENGINE: Record = { 'ollama-proxy': 'ollama', - 'lmstudio-proxy': 'lm-studio' + 'lmstudio-proxy': 'lm-studio', + 'llamacpp-proxy': 'llamacpp' +} + +function proxySourceForEngine(engine: EngineType): ProxyNodeSource { + if (engine === 'ollama') return 'ollama-proxy' + if (engine === 'lm-studio') return 'lmstudio-proxy' + return 'llamacpp-proxy' } /** Per-engine presence on a node — each proxy reports its own engine. */ @@ -68,7 +79,7 @@ interface EnginePresence { /** * The node's promoted inference **proxy** port for this engine, as * advertised in discovery. Under secure inference the broker registers the - * `ol`/`lm` service at the proxy's port — never the engine's own port, which + * `ol`/`lm`/`lc` service at the proxy's port — never the engine's own port, which * is loopback-private and reachable by peers only through that proxy's * cluster-mTLS ingress. The engine's real server port is not in discovery; * it comes from `engine:remote-get-installed` facts (a peer) or @@ -92,6 +103,9 @@ interface RemoteEngineFacts { running: boolean healthy: boolean port: number + installSupported?: boolean + installReason?: string + managed?: boolean } interface ModularNode { @@ -177,19 +191,16 @@ function emptyPresence(): EnginePresence { } function emptyEngines(): Record { - return { ollama: emptyPresence(), 'lm-studio': emptyPresence() } + return { ollama: emptyPresence(), 'lm-studio': emptyPresence(), llamacpp: emptyPresence() } } -/** Immutably set one engine's presence, preserving the other. */ +/** Immutably set one engine's presence, preserving the others. */ function setEngine( engines: Record, engine: ProxyEngine, presence: EnginePresence ): Record { - return { - ollama: engine === 'ollama' ? presence : engines.ollama, - 'lm-studio': engine === 'lm-studio' ? presence : engines['lm-studio'] - } + return { ...engines, [engine]: presence } } /** @@ -369,6 +380,7 @@ function parseWorkload(value: JsonValue | undefined): Workload | null { const workload: Workload = { id, + runId: stringValue(obj.runId) || undefined, model: stringValue(obj.model), engine, state: stateValue, @@ -403,7 +415,7 @@ export function parseWorkloadsInitial(value: JsonValue | undefined): Workload[] /** True for an engine fronted by a broker-supervised reverse proxy. */ export function isProxyEngine(engine: EngineType): engine is ProxyEngine { - return engine === 'ollama' || engine === 'lm-studio' + return engine === 'ollama' || engine === 'lm-studio' || engine === 'llamacpp' } const PENDING_OP_IDLE_TIMEOUT_MS = 90_000 @@ -582,11 +594,11 @@ function sameTelemetry( ) } -function modelItem(name: string, loaded = false): ModelItem { +function modelItem(name: string, loaded = false, downloaded = true): ModelItem { return { name, size: 0, - downloaded: true, + downloaded, status: loaded ? 'loaded' : 'idle', parameterSize: '', quantization: '', @@ -721,7 +733,7 @@ function parseProxyNode(params: JsonValue | undefined, engine: ProxyEngine): Mod } return { id, - sources: [engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy'], + sources: [proxySourceForEngine(engine)], // `Node.Host` is the hostname; empty for the self-bridge manual node, // in which case the broker discovery entry supplies the display name on // merge (see mergeNode). Never fall back to the UUID id here. @@ -892,8 +904,9 @@ class ModularBridgeState { private logs: LogEntry[] = [] // Per-engine bound proxy port reported by the broker. 0 = not reported yet; // we never fabricate a default — an unknown port surfaces as null, not a - // guess. `ollama` is the `ollama-proxy`, `lm-studio` is the `lmstudio-proxy`. - private proxyPorts: Record = { ollama: 0, 'lm-studio': 0 } + // guess. `ollama` is the `ollama-proxy`, `lm-studio` is the `lmstudio-proxy`, + // `llamacpp` is the `llamacpp-proxy`. + private proxyPorts: Record = { ollama: 0, 'lm-studio': 0, llamacpp: 0 } private selfId: string | null = null /** * Authoritative local-engine facts from `nvpair-engine-manager`, keyed by @@ -903,7 +916,14 @@ class ModularBridgeState { */ private engineManagerFacts = new Map< EngineType, - { installed: boolean; running: boolean; port: number } + { + installed: boolean + running: boolean + port: number + installSupported?: boolean + installReason?: string + managed?: boolean + } >() /** * Local model lists pulled from `nvpair-engine-manager`'s `list_models` action by @@ -1287,7 +1307,12 @@ class ModularBridgeState { */ seedWorkloads(workloads: Workload[]): WsInvokeResponse<'workloads:get-initial'> { for (const workload of workloads) { - const key = workloadKey(workload.originatedFrom, workload.id) + const key = workloadKey( + workload.originatedFrom, + workload.id, + workload.engine, + workload.runId + ) if (!this.workloads.has(key)) this.workloads.set(key, workload) } return this.getWorkloads() @@ -1298,7 +1323,10 @@ class ModularBridgeState { const obj = objectValue(params) const workload = parseWorkload(obj?.workloadInfo) if (!workload) return - this.workloads.set(workloadKey(workload.originatedFrom, workload.id), workload) + this.workloads.set( + workloadKey(workload.originatedFrom, workload.id, workload.engine, workload.runId), + workload + ) emitBridgePush('workloads:upsert', workload) } @@ -1308,8 +1336,18 @@ class ModularBridgeState { const workloadId = stringValue(obj?.workloadId) if (!workloadId) return const originatedFrom = nullableStringValue(obj?.originatedFrom) - this.workloads.delete(workloadKey(originatedFrom, workloadId)) - emitBridgePush('workloads:remove', { workloadId, originatedFrom }) + const engine = engineTypeFromManagerName(stringValue(obj?.engine)) ?? undefined + const runId = stringValue(obj?.runId) || undefined + for (const [key, workload] of this.workloads) { + if ( + workload.originatedFrom === originatedFrom && + workload.id === workloadId && + (!engine || workload.engine === engine) && + (!runId || workload.runId === runId) + ) + this.workloads.delete(key) + } + emitBridgePush('workloads:remove', { workloadId, originatedFrom, engine, runId }) } /** Push a `workloads:remove` for an entry and drop it from the catalog. */ @@ -1317,7 +1355,9 @@ class ModularBridgeState { this.workloads.delete(key) emitBridgePush('workloads:remove', { workloadId: workload.id, - originatedFrom: workload.originatedFrom + originatedFrom: workload.originatedFrom, + engine: workload.engine, + runId: workload.runId }) } @@ -1368,7 +1408,11 @@ class ModularBridgeState { this.engineManagerFacts.set(engineType, { installed: booleanValue(obj.installed), running: booleanValue(obj.running), - port: numberValue(obj.port) + port: numberValue(obj.port), + installSupported: + typeof obj.install_supported === 'boolean' ? obj.install_supported : undefined, + installReason: stringValue(obj.install_reason), + managed: typeof obj.managed === 'boolean' ? obj.managed : undefined }) // A fresh authoritative state is the resolution of whatever op was in // flight (start/stop done, install `done`+installed, uninstall removed). @@ -1705,7 +1749,7 @@ class ModularBridgeState { * the renderer renders the peer engine as unavailable rather than as an * installed-but-off toggle. * - * The peer's promoted **proxy** port IS carried in discovery (the `ol`/`lm` + * The peer's promoted **proxy** port IS carried in discovery (the `ol`/`lm`/`lc` * advertisement points at the proxy), so it is surfaced as `proxyPort` from * that per-engine presence regardless of facts. * The peer's engine port stays private (loopback) and comes only from facts; @@ -1725,6 +1769,9 @@ class ModularBridgeState { base = { engineType: engine, nodeId, + installSupported: facts.installSupported, + installReason: facts.installReason, + managed: facts.managed, processStatus: facts.running ? 'running' : facts.installed @@ -1782,7 +1829,13 @@ class ModularBridgeState { installed: booleanValue(engineObj.installed), running: booleanValue(engineObj.running), healthy: booleanValue(engineObj.healthy), - port: numberValue(engineObj.port) + port: numberValue(engineObj.port), + installSupported: + typeof engineObj.install_supported === 'boolean' + ? engineObj.install_supported + : undefined, + installReason: stringValue(engineObj.install_reason), + managed: typeof engineObj.managed === 'boolean' ? engineObj.managed : undefined }) seen.add(engineType) } @@ -1811,7 +1864,11 @@ class ModularBridgeState { installed: booleanValue(obj.installed), running: booleanValue(obj.running), healthy: booleanValue(obj.healthy), - port: numberValue(obj.port) + port: numberValue(obj.port), + installSupported: + typeof obj.install_supported === 'boolean' ? obj.install_supported : undefined, + installReason: stringValue(obj.install_reason), + managed: typeof obj.managed === 'boolean' ? obj.managed : undefined }) this.emitRemoteEngineStatus(nodeId, engineType) } @@ -1854,14 +1911,17 @@ class ModularBridgeState { * Called by the supervisor after a `list_models` pull (or with an empty list * when the engine is stopped, since its HTTP `list_models` is unreachable). */ - setLocalEngineModels(engineType: EngineType, modelNames: string[]): void { + setLocalEngineModels(engineType: EngineType, modelNames: string[], downloaded = true): void { const nodeId = this.selfId // Stamp `'loaded'` from the self node's loaded set so a `list_models` // refresh (pull/lifecycle) preserves residency instead of resetting every // row to idle. The loaded set is seeded by discovery self-enrichment and // kept fresh by {@link applyLocalLoadedModels}. - const loaded = loadedNamesForEngine(nodeId ? this.nodes.get(nodeId) : undefined, engineType) - const items = modelNames.map(name => modelItem(name, loaded.has(name))) + const loaded = + this.engineManagerFacts.get(engineType)?.running === false + ? new Set() + : loadedNamesForEngine(nodeId ? this.nodes.get(nodeId) : undefined, engineType) + const items = modelNames.map(name => modelItem(name, loaded.has(name), downloaded)) this.localManagerModels.set(engineType, items) if (!nodeId) return emitBridgePush('engines:state-changed', { @@ -1989,11 +2049,21 @@ class ModularBridgeState { private toEngineModels(node: ModularNode, engine: ProxyEngine): EngineModels { const loaded = loadedNamesForEngine(node, engine) + // Authenticated managed-peer facts and its attributed inventory prove + // downloaded weights; an external router catalogue alone does not. + const managedPeerInventory = + node.id !== this.selfId && + this.remoteEngineFacts.get(this.remoteOpKey(node.id, engine))?.managed === true && + Object.hasOwn(node.modelsByEngine, engineManagerName(engine)) return { engineType: engine, nodeId: node.id, models: this.modelsForEngine(node, engine).map(name => - modelItem(name, loaded.has(name)) + modelItem( + name, + loaded.has(name), + engine !== 'llamacpp' || managedPeerInventory || loaded.has(name) + ) ) } } @@ -2024,7 +2094,10 @@ class ModularBridgeState { // The cached list is the authoritative `list_models` set (names // only); stamp `'loaded'` from the self node's discovery/push // loaded set so the local card reflects in-memory residency too. - const loaded = loadedNamesForEngine(node, engine) + const loaded = + this.engineManagerFacts.get(engine)?.running === false + ? new Set() + : loadedNamesForEngine(node, engine) return { engineType: engine, nodeId, @@ -2066,6 +2139,9 @@ class ModularBridgeState { return { engineType, nodeId, + installSupported: facts?.installSupported, + installReason: facts?.installReason, + managed: facts?.managed, processStatus: pending, enginePort: facts && facts.running && facts.port > 0 ? facts.port : null, proxyPort: isProxyEngine(engineType) ? this.getProxyPort(engineType) : null @@ -2076,6 +2152,9 @@ class ModularBridgeState { return { engineType, nodeId, + installSupported: facts.installSupported, + installReason: facts.installReason, + managed: facts.managed, processStatus: facts.running ? 'running' : facts.installed @@ -2282,6 +2361,10 @@ class ModularBridgeState { this.handleProxyNotification(notification, 'lm-studio') return } + if (notification.source === 'llamacpp-proxy') { + this.handleProxyNotification(notification, 'llamacpp') + return + } if (notification.source === 'broker') { this.handleBrokerNotification(notification) } @@ -2327,7 +2410,7 @@ class ModularBridgeState { if (notification.method === 'node/discovered' || notification.method === 'node/updated') { const node = parseProxyNode(notification.params, engine) if (!node) return - this.upsertNode(node, engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy') + this.upsertNode(node, proxySourceForEngine(engine)) } } @@ -2339,7 +2422,7 @@ class ModularBridgeState { private clearNodeEngine(nodeId: string, engine: ProxyEngine): void { const existing = this.nodes.get(nodeId) if (!existing) return - const source: BrokerNodeSource = engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' + const source: BrokerNodeSource = proxySourceForEngine(engine) const sources = removeSource(existing.sources, source) if (sources.length === 0 && !existing.nodeInfoUp) { this.removeNodeEntry(nodeId) @@ -2532,7 +2615,7 @@ class ModularBridgeState { // install/running state; discovery only fills in models. A remote node // has no local engine-manager, so its status comes from authoritative // peer facts or its advertisement, and is omitted when neither is known. - // Push per proxy-engine (Ollama + LM Studio) so both light up per node. + // Push per proxy-engine (Ollama, LM Studio, llama.cpp) so each lights up per node. const isSelf = merged.id === this.selfId for (const engine of PROXY_ENGINES) { if (!isSelf) { @@ -2587,7 +2670,7 @@ class ModularBridgeState { } } - // A proxy source (ollama-proxy / lmstudio-proxy): refresh only that + // A proxy source (ollama-proxy / lmstudio-proxy / llamacpp-proxy): refresh only that // engine's presence; keep the other engine, telemetry, and node-info. const engine = PROXY_SOURCE_ENGINE[source] return { diff --git a/desktop/src/electron/service-bridge/modular-supervisor.ts b/desktop/src/electron/service-bridge/modular-supervisor.ts index 8cd1012b..d92146a6 100644 --- a/desktop/src/electron/service-bridge/modular-supervisor.ts +++ b/desktop/src/electron/service-bridge/modular-supervisor.ts @@ -163,13 +163,14 @@ export function parseListModelNames(result: JsonValue | undefined): string[] { const obj = objectValue(result) if (!obj) throw new Error('list_models returned a non-object response') const names: string[] = [] - if (Array.isArray(obj.models)) { - for (const entry of obj.models) { + const rows = Array.isArray(obj.models) ? obj.models : Array.isArray(obj.data) ? obj.data : null + if (rows) { + for (const entry of rows) { const row = objectValue(entry) - const name = stringValue(row?.name) || stringValue(row?.key) + const name = stringValue(row?.name) || stringValue(row?.key) || stringValue(row?.id) if (name) names.push(name) } - if (obj.models.length > 0 && names.length === 0) { + if (rows.length > 0 && names.length === 0) { throw new Error('list_models returned no usable model names') } return names @@ -300,7 +301,16 @@ function proxyEngineFromManagerId(id: string): ProxyEngine | null { /** The broker relay namespace fronting an engine's reverse proxy. */ function proxyRelayPrefix(engine: ProxyEngine): string { - return engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy' + if (engine === 'ollama') return 'ollama-proxy' + if (engine === 'lm-studio') return 'lmstudio-proxy' + return 'llamacpp-proxy' +} + +function proxyEngineFromRelaySource(source: string): ProxyEngine | null { + if (source === 'ollama-proxy') return 'ollama' + if (source === 'lmstudio-proxy') return 'lm-studio' + if (source === 'llamacpp-proxy') return 'llamacpp' + return null } /** @@ -310,12 +320,13 @@ function proxyRelayPrefix(engine: ProxyEngine): string { * `docs/services-backend.md`): * * - The `nvpair-ui-broker` is the **only** Electron-spawned binary and is itself the - * parent of every broker-owned worker (`ollama-proxy`, `lmstudio-proxy`, + * parent of every broker-owned worker (`nvpair-proxy`, which hosts a facade + * per engine rather than shipping one binary each, * `nvpair-node-scanner`, `nvpair-node-info`, `nvpair-workload-manager`, * `nvpair-cluster-manager`, `nvpair-node-settings`, `nvpair-manual-nodes`, * `nvpair-engine-manager`, `nvpair-errors`, `nvpair-job-scheduler`). Electron passes their resolved paths to * the broker (see `brokerStartupArgs`) and reaches each through a broker relay: - * `ollama-proxy:` / `lmstudio-proxy:` for the two engine proxies, `engine:` for the + * `ollama-proxy:` / `lmstudio-proxy:` / `llamacpp-proxy:` for the engine facades, `engine:` for the * engine-manager, `errors:` for the error pipeline, `node/*` for manual nodes, * `settings/*` and `cluster:` for the rest. Local inference jobs arrive on the * broker's `workloads:subscribe` stream. @@ -324,6 +335,7 @@ function proxyRelayPrefix(engine: ProxyEngine): string { */ class ModularSupervisor { private processes = new Map() + private engineEventRevisions = new Map() private readinessWaiters = new Map() private nextReadinessWaiterId = 0 private readinessReported = false @@ -338,6 +350,7 @@ class ModularSupervisor { // last-known-good result (including an authoritative empty list). private stoppedModelSentinels = new Set() private stoppedModelEngines = new Set() + private managedLlama = false private successfulModelInventories = new Set() private modelRefreshGenerations = new Map() private discoveryModelRetryTimers = new Map>() @@ -875,6 +888,7 @@ class ModularSupervisor { await subscribe('discovery:subscribe', 'subscribe to broker discovery') await subscribe('ollama-proxy:subscribe', 'subscribe to broker ollama-proxy relay') await subscribe('lmstudio-proxy:subscribe', 'subscribe to broker lmstudio-proxy relay') + await subscribe('llamacpp-proxy:subscribe', 'subscribe to broker llamacpp-proxy relay') // Engine events are opt-in and replay no baseline — subscribe then hydrate. await subscribe('engine:subscribe', 'subscribe to broker engine relay') await subscribe('workloads:subscribe', 'subscribe to broker workloads stream') @@ -1076,7 +1090,7 @@ class ModularSupervisor { const obj = objectValue(result) if (obj && booleanValue(obj.ready)) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy', + source: proxyRelayPrefix(engine), method: 'ready', params: { port: numberValue(obj.port) } }) @@ -1097,7 +1111,7 @@ class ModularSupervisor { if (!obj || !Array.isArray(obj.nodes)) return for (const node of obj.nodes) { getModularBridgeState().handleNotification({ - source: engine === 'ollama' ? 'ollama-proxy' : 'lmstudio-proxy', + source: proxyRelayPrefix(engine), method: 'node/discovered', params: node }) @@ -1113,10 +1127,14 @@ class ModularSupervisor { private async hydrateEngineManager(): Promise { if (!this.processes.has('broker')) return try { + // A push received while this snapshot is in flight wins for its engine. + const atStart = new Map(this.engineEventRevisions) const result = await this.callProcess('broker', 'engine:get-installed') const obj = objectValue(result) if (!obj || !Array.isArray(obj.engines)) return for (const engine of obj.engines) { + const type = engineTypeFromManagerName(stringValue(objectValue(engine)?.engine)) + if (!type || this.engineEventRevisions.get(type) !== atStart.get(type)) continue getModularBridgeState().applyEngineManagerStatus(engine) this.refreshManagedEngineModels(engine) } @@ -1263,12 +1281,7 @@ class ModularSupervisor { this.scheduleRemoteEngineStatusRefresh() } - const proxyEngine: ProxyEngine | null = - event.source === 'ollama-proxy' - ? 'ollama' - : event.source === 'lmstudio-proxy' - ? 'lm-studio' - : null + const proxyEngine = proxyEngineFromRelaySource(event.source) if (proxyEngine && event.method === 'ready') { // A (re)bound proxy starts with an empty manual-node set, so forget // what we think we bridged and re-push the local node if applicable. @@ -1629,6 +1642,12 @@ class ModularSupervisor { return } if (notification.method === 'engine:state-changed') { + const type = engineTypeFromManagerName( + stringValue(objectValue(notification.params)?.engine) + ) + if (type) { + this.engineEventRevisions.set(type, (this.engineEventRevisions.get(type) ?? 0) + 1) + } getModularBridgeState().applyEngineManagerStatus(notification.params) this.refreshManagedEngineModels(notification.params) this.updateLocalNodeBridgeFromEngineState(notification.params) @@ -1906,6 +1925,24 @@ class ModularSupervisor { } } + async importLlamaModel(path: string): Promise { + try { + await this.callProcess( + 'broker', + 'engine:action', + { engine: 'llamacpp', action: 'import_model', params: { path } }, + PULL_TIMEOUT_MS + ) + await this.refreshEngineModels('llamacpp', 'llamacpp') + } catch (err) { + this.reportError( + `Model import failed: ${getErrorString(err)}`, + 'error', + 'llamacpp-import' + ) + } + } + /** * Load, unload (eject), or delete a model on a remote peer via the ec surface. * @@ -1965,7 +2002,10 @@ class ModularSupervisor { try { const result = await this.callProcess('broker', 'engine:action', { engine, - action: 'list_models' + action: + engineType === 'llamacpp' && this.managedLlama + ? 'list_downloaded' + : 'list_models' }) const models = parseListModelNames(result) this.commitModelInventory(engineType, models, generation) @@ -1994,8 +2034,9 @@ class ModularSupervisor { const engine = stringValue(obj.engine) const engineType = getModularBridgeState().modelPullTarget(engine) if (!engineType) return + if (engineType === 'llamacpp') this.managedLlama = booleanValue(obj.managed) - if (!booleanValue(obj.running)) { + if (!booleanValue(obj.running) && !(engineType === 'llamacpp' && this.managedLlama)) { this.beginModelRefresh(engineType) this.stoppedModelEngines.add(engineType) this.stoppedModelSentinels.add(engineType) @@ -2016,7 +2057,8 @@ class ModularSupervisor { const generation = this.beginModelRefresh(engineType) this.callProcess('broker', 'engine:action', { engine, - action: 'list_models' + action: + engineType === 'llamacpp' && this.managedLlama ? 'list_downloaded' : 'list_models' }) .then(result => { const models = parseListModelNames(result) @@ -2050,7 +2092,7 @@ class ModularSupervisor { const generation = this.beginModelRefresh(engine) this.callProcess('broker', 'engine:action', { engine: engineManagerName(engine), - action: 'list_models' + action: engine === 'llamacpp' && this.managedLlama ? 'list_downloaded' : 'list_models' }) .then(result => { const models = parseListModelNames(result) @@ -2077,7 +2119,11 @@ class ModularSupervisor { if (isProxyEngine(engine)) { this.cancelDiscoveryModelRefreshRetry(engine) } - getModularBridgeState().setLocalEngineModels(engine, models) + getModularBridgeState().setLocalEngineModels( + engine, + models, + engine !== 'llamacpp' || this.managedLlama + ) } private scheduleDiscoveryModelRefreshRetry(engine: ProxyEngine): void { diff --git a/desktop/src/shared/constants/engines.ts b/desktop/src/shared/constants/engines.ts index 5904ae61..d924c17e 100644 --- a/desktop/src/shared/constants/engines.ts +++ b/desktop/src/shared/constants/engines.ts @@ -4,16 +4,16 @@ import { EngineType, ModelExpiry } from '@/shared/types/engines' // The engines `nvpair-engine-manager` ships a manifest for, and therefore the -// only ones PAIR can install, run or route to. llama-cpp, whisper-cpp, -// piper-tts, sherpa-onnx-tts and stable-diffusion-cpp were carried here as -// never-enabled placeholders; they were removed with the chat window, which was -// their only in-app consumer. Adding an engine back means shipping its manifest -// first -- an engine row without one renders commands that fail with `-32000`. -export const EngineTypes = ['ollama', 'lm-studio'] as const +// only ones PAIR can install, run or route to. whisper-cpp, piper-tts, +// sherpa-onnx-tts and stable-diffusion-cpp were carried here as never-enabled +// placeholders; they were removed with the chat window, which was their only +// in-app consumer. Adding an engine back means shipping its manifest first -- +// an engine row without one renders commands that fail with `-32000`. +export const EngineTypes = ['ollama', 'lm-studio', 'llamacpp'] as const // Kept as a distinct export so a future engine can ship behind it rather than // appearing the moment its type exists. -export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio'] as const +export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio', 'llamacpp'] as const /** * How `nvpair-engine-manager` spells each engine on the wire. Only LM Studio @@ -23,20 +23,26 @@ export const EnabledEngineTypes: EngineType[] = ['ollama', 'lm-studio'] as const */ export const EngineManagerNames = { ollama: 'ollama', - 'lm-studio': 'lmstudio' + 'lm-studio': 'lmstudio', + llamacpp: 'llamacpp' } as const satisfies Record export const EngineSources = ['bundled', 'detected', 'installed'] as const export const EngineDisplayNames: Record = { ollama: 'Ollama', - 'lm-studio': 'LM Studio' + 'lm-studio': 'LM Studio', + llamacpp: 'llama.cpp' } as const /** Default docs/install URLs for built-in backends. Single source of truth for UI and adapter buildInfo(). */ export const EngineDefaultLinks: Record = { ollama: { docsUrl: 'https://docs.ollama.com/', installUrl: 'https://ollama.com/download' }, - 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' } + 'lm-studio': { docsUrl: 'https://lmstudio.ai/docs', installUrl: 'https://lmstudio.ai/' }, + llamacpp: { + docsUrl: 'https://github.com/ggml-org/llama.cpp', + installUrl: 'https://github.com/ggml-org/llama.cpp' + } } as const export const ModelItemStatuses = ['idle', 'loading', 'loaded', 'ejecting', 'pulling'] as const diff --git a/desktop/src/shared/types/engine-api.ts b/desktop/src/shared/types/engine-api.ts index 9187e658..58467cff 100644 --- a/desktop/src/shared/types/engine-api.ts +++ b/desktop/src/shared/types/engine-api.ts @@ -45,6 +45,8 @@ export type EngineCommandType = | 'uninstall' | 'update' | 'pullModel' + | 'cancelPull' + | 'importModel' | 'loadModel' | 'unloadModel' | 'deleteModel' diff --git a/desktop/src/shared/types/engines.ts b/desktop/src/shared/types/engines.ts index 863c415c..de934f61 100644 --- a/desktop/src/shared/types/engines.ts +++ b/desktop/src/shared/types/engines.ts @@ -49,6 +49,9 @@ export interface EngineStatusData { * reported version data or engines that are not installed. */ installedVersion?: string + installSupported?: boolean + installReason?: string + managed?: boolean } export type ModelItemStatus = (typeof ModelItemStatuses)[number] diff --git a/desktop/src/shared/types/inference-demo.ts b/desktop/src/shared/types/inference-demo.ts index 4af2f35a..adccd928 100644 --- a/desktop/src/shared/types/inference-demo.ts +++ b/desktop/src/shared/types/inference-demo.ts @@ -54,10 +54,11 @@ export const DEMO_REQUEST_TIMEOUT_SECONDS = 120 export const DEMO_ENGINE_PROBES: readonly { backend: DispatcherBackend /** Engine key used by the broker's proxy port registry. */ - proxyEngine: 'ollama' | 'lm-studio' + proxyEngine: 'ollama' | 'lm-studio' | 'llamacpp' }[] = [ { backend: 'ollama', proxyEngine: 'ollama' }, - { backend: 'lmstudio', proxyEngine: 'lm-studio' } + { backend: 'lmstudio', proxyEngine: 'lm-studio' }, + { backend: 'llamacpp', proxyEngine: 'llamacpp' } ] /** diff --git a/desktop/src/shared/types/inference-dispatcher.ts b/desktop/src/shared/types/inference-dispatcher.ts index 2abe35ae..9c6d7a45 100644 --- a/desktop/src/shared/types/inference-dispatcher.ts +++ b/desktop/src/shared/types/inference-dispatcher.ts @@ -10,7 +10,7 @@ * reads its `--list-models` inventory. */ -export type DispatcherBackend = 'ollama' | 'lmstudio' +export type DispatcherBackend = 'ollama' | 'lmstudio' | 'llamacpp' /** One entry from the binary's `--list-models` JSON inventory. */ export interface DispatcherModel { diff --git a/desktop/src/shared/types/workloads.ts b/desktop/src/shared/types/workloads.ts index 71340812..effc2584 100644 --- a/desktop/src/shared/types/workloads.ts +++ b/desktop/src/shared/types/workloads.ts @@ -6,16 +6,24 @@ import { EngineType } from '@/shared/types/engines' export type WorkloadState = (typeof WorkloadStates)[number] +export interface WorkloadRemoval { + workloadId: string + originatedFrom: string | null + engine?: EngineType + runId?: string +} + export interface Workload { id: string + runId?: string model: string engine: EngineType state: WorkloadState /** * Owner/origin node of the workload — the node whose proxy received the - * request. This is the identity half of the backend's `(originatedFrom, id)` - * global catalog key (workload ids are a per-node proxy counter, so they - * collide across nodes; `originatedFrom` disambiguates). + * request. Together with engine, runId and id, this identifies one request + * in the global catalog; proxy counters can repeat across nodes, engines + * and proxy runs. */ originatedFrom: string | null /** diff --git a/desktop/src/shared/types/ws-channels.ts b/desktop/src/shared/types/ws-channels.ts index 15923765..8484982e 100644 --- a/desktop/src/shared/types/ws-channels.ts +++ b/desktop/src/shared/types/ws-channels.ts @@ -41,7 +41,7 @@ import type { AppInitialSnapshot, ClusterInitialSnapshot } from '@/shared/types/ import type { ServiceError } from '@/shared/types/errors' import type { NodeItem } from '@/shared/types/nodes' import type { NodeItemMetrics } from '@/shared/types/metrics' -import type { Workload } from '@/shared/types/workloads' +import type { Workload, WorkloadRemoval } from '@/shared/types/workloads' import type { AvailableNode, ClusterIdentityPayload, @@ -141,7 +141,7 @@ export interface WsPushChannelMap { // workload ids are a per-node proxy counter (the catalog is keyed by the // (originatedFrom, id) pair). 'workloads:upsert': Workload - 'workloads:remove': { workloadId: string; originatedFrom: string | null } + 'workloads:remove': WorkloadRemoval // Errors 'errors:update': ServiceError[] diff --git a/desktop/src/shared/utils/engine-progress.ts b/desktop/src/shared/utils/engine-progress.ts index 123919a8..3e7cea2b 100644 --- a/desktop/src/shared/utils/engine-progress.ts +++ b/desktop/src/shared/utils/engine-progress.ts @@ -9,6 +9,13 @@ import { EngineOperationType, EngineProgress, EngineType } from '@/shared/types/engines' +/** Unknown vendor progress (including -1) must not become a numeric percentage. */ +export function roundedProgressPercent(percent: number | undefined): number | null { + return percent !== undefined && Number.isFinite(percent) && percent >= 0 && percent <= 100 + ? Math.round(percent) + : null +} + /** Build the map key for an EngineProgress entry. */ export function engineProgressKey(p: { nodeId: string diff --git a/desktop/src/shared/utils/workloads.ts b/desktop/src/shared/utils/workloads.ts index 0cee3b6d..9ed23b46 100644 --- a/desktop/src/shared/utils/workloads.ts +++ b/desktop/src/shared/utils/workloads.ts @@ -7,17 +7,19 @@ import type { Workload } from '@/shared/types/workloads' * Stable catalog key for a workload. * * The backend's catalog is keyed by `(originatedFrom, engine, runId, id)`, but - * the `workloads:remove` push carries only `(workloadId, originatedFrom)` — and - * the broker's own `Store.Remove` drops every record matching that pair — so - * `(originatedFrom, id)` is the only key a subscribe client can maintain - * consistently across upsert and remove. Each node's proxy assigns workload ids - * from its own monotonic counter, so ids collide across nodes; `originatedFrom` - * (the origin node) disambiguates. Mirror that here so a remote node's job never - * overwrites a local one that happens to share an id. The `\u0000` separator - * cannot appear in a host id or proxy counter, so the key is unambiguous. + * clients retain all four fields so equal counters from different engines or + * proxy runs never overwrite one another. A legacy removal without engine/run + * identity removes the matching origin/id prefix, as the broker does. Targeted + * removals retain exact identity. */ -export function workloadKey(originatedFrom: string | null, id: string): string { - return `${originatedFrom ?? ''}\u0000${id}` +export function workloadKey( + originatedFrom: string | null, + id: string, + engine?: string, + runId?: string +): string { + const prefix = `${originatedFrom ?? ''}\u0000${id}` + return engine === undefined ? prefix : `${prefix}\u0000${engine}\u0000${runId ?? ''}` } /** diff --git a/desktop/src/ui/api/engine-api.ts b/desktop/src/ui/api/engine-api.ts index 71d8f163..20202016 100644 --- a/desktop/src/ui/api/engine-api.ts +++ b/desktop/src/ui/api/engine-api.ts @@ -54,6 +54,8 @@ export interface IEngineApi { uninstall(engineType: EngineType, nodeId: string): void /** Pull (download) a model on a node. */ pullModel(engineType: EngineType, nodeId: string, model: string): void + cancelPull(engineType: EngineType, nodeId: string, model: string): void + importModel(engineType: EngineType, nodeId: string, path: string): void /** Load a model into memory on a node. */ loadModel(engineType: EngineType, nodeId: string, model: string): void /** Unload a model from memory on a node. */ @@ -100,6 +102,10 @@ export function createEngineApi(transport: ServiceTransport): IEngineApi { fireCommand(transport, { command: 'uninstall', engineType, nodeId }), pullModel: (engineType, nodeId, model) => fireCommand(transport, { command: 'pullModel', engineType, nodeId, model }), + cancelPull: (engineType, nodeId, model) => + fireCommand(transport, { command: 'cancelPull', engineType, nodeId, model }), + importModel: (engineType, nodeId, path) => + fireCommand(transport, { command: 'importModel', engineType, nodeId, model: path }), loadModel: (engineType, nodeId, model) => fireCommand(transport, { command: 'loadModel', engineType, nodeId, model }), unloadModel: (engineType, nodeId, model) => diff --git a/desktop/src/ui/api/pair-api.ts b/desktop/src/ui/api/pair-api.ts index a0a34155..a01e0180 100644 --- a/desktop/src/ui/api/pair-api.ts +++ b/desktop/src/ui/api/pair-api.ts @@ -12,7 +12,7 @@ import type { import type { NodeItem } from '@/shared/types/nodes' import type { ServiceError } from '@/shared/types/errors' import type { NodeItemMetrics } from '@/shared/types/metrics' -import type { Workload } from '@/shared/types/workloads' +import type { Workload, WorkloadRemoval } from '@/shared/types/workloads' import type { AppInitialSnapshot, ClusterInitialSnapshot } from '@/shared/types/bootstrap' // --------------------------------------------------------------------------- @@ -85,9 +85,7 @@ export interface IWorkloadsApi { /** A workload was created or updated. */ onUpsert(callback: (workload: Workload) => void): () => void /** A workload was completed and removed. */ - onRemove( - callback: (removal: { workloadId: string; originatedFrom: string | null }) => void - ): () => void + onRemove(callback: (removal: WorkloadRemoval) => void): () => void } export interface IErrorsApi { diff --git a/desktop/src/ui/components/BackendRow/BackendFooter.tsx b/desktop/src/ui/components/BackendRow/BackendFooter.tsx index fc7ece39..8aa56d5b 100644 --- a/desktop/src/ui/components/BackendRow/BackendFooter.tsx +++ b/desktop/src/ui/components/BackendRow/BackendFooter.tsx @@ -33,7 +33,10 @@ export function BackendFooter({ disabled: boolean onUninstall: () => void }) { - const autoInstall = canAutoInstallBackendForOs(backend.type, targetOs) + const autoInstall = + backend.type === 'llamacpp' + ? backend.installSupported === true + : canAutoInstallBackendForOs(backend.type, targetOs) const isTransitioning = backend.processStatus === 'installing' || backend.processStatus === 'uninstalling' const isNotInstalled = backend.processStatus === 'not-installed' diff --git a/desktop/src/ui/components/BackendRow/BackendHeader.tsx b/desktop/src/ui/components/BackendRow/BackendHeader.tsx index d54bd629..05473127 100644 --- a/desktop/src/ui/components/BackendRow/BackendHeader.tsx +++ b/desktop/src/ui/components/BackendRow/BackendHeader.tsx @@ -11,6 +11,7 @@ import { DismissibleTooltip } from '@/ui/components/DismissibleTooltip/Dismissib import { gatewayEndpointDisplayUrl } from '@/ui/utils/gateway-inference-paths' import { EngineCapabilities } from '@/ui/constants/engine-capabilities' import { statusLabel } from '@/ui/utils/status' +import { roundedProgressPercent } from '@/shared/utils/engine-progress' /** Install/uninstall lines include asset names + percentages — allow more room than generic status. */ const INSTALL_STATUS_MAX_LEN = 52 @@ -91,7 +92,17 @@ export function BackendHeader({ className={`${isUnavailable ? 'cursor-default' : 'cursor-pointer'} p-4 -m-4`} > - + {isLocalNode && @@ -114,7 +125,7 @@ export function BackendHeader({ e.stopPropagation() handleCopy() }} - title={`Copy ${backend.displayName} API http://127.0.0.1:${backend.proxyPort}`} + title={`Copy ${backend.displayName} API ${proxyUrl}`} style={{ padding: '2px 6px', minWidth: 'auto' }} aria-label={`Copy ${backend.displayName} API URL`} > @@ -160,7 +171,7 @@ export function BackendHeader({ const baseStatus = backend.installProgress?.status ?? statusLabel[backend.processStatus] const pct = backend.installProgress?.percent - const pctRounded = pct != null && Number.isFinite(pct) ? Math.round(pct) : null + const pctRounded = roundedProgressPercent(pct) const pctSuffix = pctRounded != null ? ` · ${pctRounded}%` : '' const baseWithoutDuplicatePercent = pctRounded != null diff --git a/desktop/src/ui/components/BackendRow/BackendRow.tsx b/desktop/src/ui/components/BackendRow/BackendRow.tsx index f3c16238..db5dcf92 100644 --- a/desktop/src/ui/components/BackendRow/BackendRow.tsx +++ b/desktop/src/ui/components/BackendRow/BackendRow.tsx @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { useCallback, useEffect, useMemo, useState } from 'react' -import { Divider, Stack } from '@nvidia/foundations-react-core' +import { Divider, Stack, Text } from '@nvidia/foundations-react-core' import type { BackendInfo } from '@/ui/types/engine-info' import type { EngineProcessStatus } from '@/shared/types/engines' @@ -23,6 +23,7 @@ import { BackendFooter } from './BackendFooter' import { BackendUpdateBanner } from './BackendUpdateBanner' import { EngineSettingsSection } from './EngineSettingsSection' +import { isExternalRuntime } from '@/ui/utils/engine-ownership' /** * The transitional status to display while an optimistic lifecycle command is @@ -173,11 +174,20 @@ export function BackendRow({ }, [isUnavailable]) // Install/start/stop, model pull, and the settings editor work on clustered - // peers; uninstall, update, and model load/delete remain local-only. - const controlsDisabled = isTransitioning + // peers; uninstall, update, and model load/delete remain local-only. A + // llama.cpp runtime PAIR detected but does not manage is observe-only: its + // owner keeps lifecycle, settings, and model changes. + const externalLlama = isExternalRuntime(backend.type, backend.processStatus, backend.managed) + const controlsDisabled = isTransitioning || externalLlama const content = expanded ? ( + {externalLlama && ( + + External llama.cpp runtime. PAIR observes it; lifecycle and model changes remain + with its owner. + + )} )} - {canShowAccordions && ( + {canShowAccordions && !externalLlama && ( diff --git a/desktop/src/ui/components/BackendRow/InstallButton.tsx b/desktop/src/ui/components/BackendRow/InstallButton.tsx index 88127754..cadca10b 100644 --- a/desktop/src/ui/components/BackendRow/InstallButton.tsx +++ b/desktop/src/ui/components/BackendRow/InstallButton.tsx @@ -23,7 +23,10 @@ export function InstallButton({ disabled: boolean onInstall: () => void }) { - const autoInstall = canAutoInstallBackendForOs(backend.type, targetOs) + const autoInstall = + backend.type === 'llamacpp' + ? backend.installSupported === true + : canAutoInstallBackendForOs(backend.type, targetOs) const isNotInstalled = backend.processStatus === 'not-installed' const missingPrereqs = (backend.prerequisites ?? []).filter(p => !p.installed) const prereqsMet = missingPrereqs.length === 0 @@ -36,6 +39,14 @@ export function InstallButton({ const onInstallAllClick = useDismissibleTooltipTrigger(onInstall) + if (backend.type === 'llamacpp' && backend.installSupported !== true && isNotInstalled) { + return ( + + {backend.installReason || 'Install support not reported by this node.'} + + ) + } + return ( <> {!autoInstall && isLocalNode && isNotInstalled && backend.installUrl && ( diff --git a/desktop/src/ui/components/EngineIcon.tsx b/desktop/src/ui/components/EngineIcon.tsx index 4ee81367..a5f71755 100644 --- a/desktop/src/ui/components/EngineIcon.tsx +++ b/desktop/src/ui/components/EngineIcon.tsx @@ -21,23 +21,39 @@ export default function EngineIcon({ type, size = 32 }: { type: EngineType; size overflow: 'hidden' } - if (type === 'ollama') { - return ( -
- Ollama -
- ) + switch (type) { + case 'ollama': + return ( +
+ Ollama +
+ ) + case 'lm-studio': + imgStyle.objectFit = 'cover' + return ( +
+ LM Studio +
+ ) + case 'llamacpp': + return ( +
+ + cpp + +
+ ) } - - if (type === 'lm-studio') { - imgStyle.objectFit = 'cover' - - return ( -
- LM Studio -
- ) - } - - return null } diff --git a/desktop/src/ui/components/ModelManager/ModelManager.tsx b/desktop/src/ui/components/ModelManager/ModelManager.tsx index e63dacf2..ee45d5c1 100644 --- a/desktop/src/ui/components/ModelManager/ModelManager.tsx +++ b/desktop/src/ui/components/ModelManager/ModelManager.tsx @@ -2,7 +2,7 @@ // SPDX-License-Identifier: Apache-2.0 import { useCallback, useMemo, useState } from 'react' -import { Button, Flex, Stack, Text } from '@nvidia/foundations-react-core' +import { Button, Flex, FormField, Stack, Text, TextInput } from '@nvidia/foundations-react-core' import type { BackendInfo } from '@/ui/types/engine-info' import { EngineCapabilities } from '@/ui/constants/engine-capabilities' import { formatModelDisplayName } from '@/ui/utils/format-model-display-name' @@ -13,14 +13,19 @@ import { ModelHubModal } from '@/ui/components/ModelHub/ModelHubModal' import { useEngineProgressStore } from '@/ui/stores/engine-progress.store' import { usePendingActionsStore } from '@/ui/stores/pending-actions.store' import { isEnginePullInProgress } from '@/shared/utils/engine-progress' +import { useConnectionStore } from '@/ui/stores/connection.store' import ModelRow from './ModelRow' import { IncomingSyncPullRow } from './IncomingSyncPullRow' import { TransientModelStatusRow } from './TransientModelStatusRow' import type { IncomingSyncRow } from '@/ui/types/model-manager' import type { ModelEntry } from '@/ui/types/model-hub' +import { isExternalRuntime } from '@/ui/utils/engine-ownership' export function ModelManager({ backend, nodeId }: { backend: BackendInfo; nodeId: string }) { + const selfId = useConnectionStore(state => state.selfId) + const [importPath, setImportPath] = useState('') + const [llamaRepo, setLlamaRepo] = useState('') const [openModelHubModal, setOpenModelHubModal] = useState(false) const [modelPendingDelete, setModelPendingDelete] = useState(null) const models = (backend.models ?? []).sort((a, b) => @@ -28,7 +33,10 @@ export function ModelManager({ backend, nodeId }: { backend: BackendInfo; nodeId formatModelDisplayName(b.name, backend.type) ) ) - const caps = EngineCapabilities[backend.type] + const externalLlama = isExternalRuntime(backend.type, backend.processStatus, backend.managed) + const caps = externalLlama + ? { ...EngineCapabilities[backend.type], hasEject: false, hasDeleteModel: false } + : EngineCapabilities[backend.type] const backendType = backend.type const getProgress = useEngineProgressStore(state => state.getProgress) @@ -195,7 +203,20 @@ export function ModelManager({ backend, nodeId }: { backend: BackendInfo; nodeId )} {incomingSyncs.map(p => ( - + + + {backendType === 'llamacpp' && !externalLlama && ( + + )} + ))} {!isBusy && ( @@ -222,7 +243,7 @@ export function ModelManager({ backend, nodeId }: { backend: BackendInfo; nodeId - {supportsSearch && ( + {/* + llama.cpp has no browsable catalog, so where the other engines + offer "Add model" and a searchable list, it takes the model's + identifier directly. These are the same brand-coloured download + action the hub uses once a row is picked, so the two routes to a + model read as the same operation. + */} + {backendType === 'llamacpp' && !externalLlama && ( + + + { + if (e.key === 'Enter' && llamaRepo.trim()) { + window.pairApi.engines.pullModel( + backendType, + nodeId, + llamaRepo.trim() + ) + } + }} + className="min-w-0" + /> + + + + )} + {/* + Importing is deliberately the quieter of the two: it only works + on this machine, because the path is resolved where the engine + runs rather than where the window is. + */} + {backendType === 'llamacpp' && !externalLlama && nodeId === selfId && ( + + + { + if (e.key === 'Enter' && importPath.trim()) { + window.pairApi.engines.importModel( + backendType, + nodeId, + importPath.trim() + ) + } + }} + className="min-w-0" + /> + + + + )} + {supportsSearch && backendType !== 'llamacpp' && (