Skip to content

About

DGX Spark + OpenHands setup

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

124 Commits

Folders and files

Repository files navigation

Local LLM Agent

Run a Qwen model on a DGX Spark and connect to it from macOS with the native (no-Docker) OpenCode client (opencode_native/, a terminal coding agent with a web UI and HTTP API). It drives the vLLM endpoint over an SSH tunnel, so nothing on the DGX side changes. Push notifications of agent activity to your phone via ntfy are included (ntfy/ + the stack's README → Notifications).

The DGX side currently offers one serving option, on port 8000:

See dgx_spark_host/ → The stack.

dgx_spark_host/

Host a Qwen model via vLLM on the DGX Spark machine — one self-contained stack per model, in its own folder (own image, compose, entrypoint and defaults; stacks share no code or parameters). All bind port 8000; run only ONE at a time.

The stack

Stack Serves How it's run Served alias
qwen38-flash-next-fp4/ nvidia/Qwen3.8-Flash-Next-FP4-Local (QAD NVFP4 of the 125B MoE, ~96 GiB) cd dgx_spark_host/qwen38-flash-next-fp4 && docker compose up --build qwen-local

Run it:

cd dgx_spark_host/qwen38-flash-next-fp4
docker compose -f compose.yml up --build

The API is available at http://localhost:8000/v1. For shared networks, bind to 127.0.0.1 in the stack's compose.yml.

Each stack's README.md documents its own environment variables (defaults live in its Dockerfile; override via .env or compose.yml) and tuning notes (DGX Spark, GB10, 128 GB unified memory, LLM-only box). The checkpoint is a native vision-language model and ships an MTP head, so MTP speculative decoding is on by default. The stack follows NVIDIA's own serve command, which leaves the --limit-mm-per-prompt image cap out.

qwen38-flash-next-fp4/ — the Qwen3.8 Flash Next option

  • What it is: nvidia/Qwen3.8-Flash-Next-FP4-Local — NVIDIA Model Optimizer's QAD (quantization-aware distillation) NVFP4/MXFP8 build of Qwen/Qwen3.8-Flash-Next, a 125B multimodal MoE with 6B active parameters plus a 51B n-gram (PLE) embedding table and a 4B MTP module. ~96 GiB on disk; NVIDIA validated it on the DGX Spark.
  • Why fully isolated: it needs a different vLLM image (the community B12X container NVIDIA prescribes, not vllm/vllm-openai:nightly), a different loader/quantization path (--load-format b12x --quantization modelopt_mixed) and its own VLLM_*/B12X_* environment — so it is a self-contained folder with its own defaults rather than a shared stack with presets. It is served on port 8000 under the alias qwen-local — clients need no changes.
  • Requirements: accept the NVIDIA Open Model Agreement on the model page and put HF_TOKEN in this stack's .env; ~96 GiB of free disk for the weights. The first start also spends a long while on weight load + compile.
  • Run: cd dgx_spark_host/qwen38-flash-next-fp4 && docker compose -f compose.yml up --build.

Full details, provenance, and the overridable parameters are in dgx_spark_host/qwen38-flash-next-fp4/README.md.

opencode_native/

Run OpenCode natively as the macOS-side client: a terminal coding agent that ships as a single self-contained binary (no Docker, and no Node.js needed), with a TUI, a bundled web UI, and a headless HTTP API. Agents run as your user on the local filesystem — there is no container sandbox, so the edit/shell approval prompt stays on by default.

Run

./opencode_native/install.sh   # one-time (upgrades on re-run)
./opencode_native/run.sh                     # TUI in the current directory
./opencode_native/run.sh /path/to/repo       # TUI on a project
./opencode_native/run.sh server              # headless API + web UI
./opencode_native/run.sh web                 # + published on the tailnet
./opencode_native/run.sh attach [dir]        # TUI attached to that server
  • Model: OpenCode has no UI for adding a provider, so install.sh generates opencode-state/config/opencode.json from opencode.example.jsonc — provider dgx-vllm against http://localhost:8000/v1, model dgx-vllm/qwen-local. The endpoint and key are substituted from VLLM_BASE_URL / VLLM_API_KEY at launch (no secret in the file), which is why you always start through run.sh.
  • Server mode listens on http://localhost:4096 (loopback only; basic auth opencode / OPENCODE_SERVER_PASSWORD, generated by install.sh — run.sh refuses to bind a shell-capable port without it) and is what attach, the web UI, and the notifier use. Reach it remotely with an SSH tunnel (ssh -L 4096:localhost:4096 host) rather than a wider bind.
  • All state — config copy, sessions (opencode.db), snapshots, logs — stays inside opencode_native/opencode-state/ via XDG_* isolation, so this stack never touches ~/.config/opencode or ~/.local/share/opencode and can't collide with a personal OpenCode install.
  • Optional ntfy push notifications: with NTFY_ENABLED=true in opencode_native/.env, run.sh server also starts ntfy_notifier.py (and the per-PC ntfy server from ntfy/) that pings you when a session finishes, waits for your approval, or errors — with tap-to-open deep links into the web UI. See opencode_native/README.md → Notifications (ntfy).
Variable Default Description
VLLM_BASE_URL http://localhost:8000/v1 Endpoint run.sh preflights and the config substitutes
VLLM_API_KEY local-dgx-key API key for the preflight check and the provider
VLLM_MODEL_ID qwen-local Model id vLLM serves (the alias the DGX stack serves)
OPENCODE_SERVER_PORT 4096 Server-mode port (API + web UI)
OPENCODE_SERVER_PASSWORD (generated) Basic-auth password for server/attach
OPENCODE_AUTO no yes = auto-approve edit/shell (equivalent of --auto)
OPENCODE_STATE_DIR ./opencode-state Isolated config/XDG state, sessions, logs

Needs curl + git, the opencode binary (install.sh installs it), and — only for notifications — any python3. Node/npm are needed only if you add OpenCode plugins. Ports, permissions, deep links, and troubleshooting are documented in opencode_native/README.md.

ntfy/

Per-PC ntfy push server (single Docker container, port 2020). Pairs with the client's notifier (opencode_native/ntfy_notifier.py) to send push notifications to your phone when an agent finishes a turn, needs input, or hits an error. Designed for multiple PCs on Tailscale, each running its own server + notifier: the notifier publishes to 127.0.0.1:2020, the phone subscribes over Tailscale at http://<pc>.tail:2020/<topic>.

cd ntfy
cp example.env .env     # set NTFY_BASE_URL=http://<this-pc>.tail:2020
docker compose up -d    # healthcheck via docker compose ps

Security model (unguessable topic = credential on Tailscale), optional account auth, phone setup (Android instant delivery / iOS relay) and the Firebase/custom-APK caveat are documented in ntfy/README.md. Enable end-to-end notifications via NTFY_ENABLED=true in opencode_native/.env (see opencode_native/README.md → Notifications (ntfy)).

About

DGX Spark + OpenHands setup

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Used by

Contributors

Languages