Skip to content

About

Declarative configuration for self-contained launchers of local language models. Includes download manifests, validation scripts, documentation, and .cmd launchers with all inference parameters materialized.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

115 Commits

Folders and files

Repository files navigation

Local Language Model Inference Setup

Declarative configuration for self-contained launchers of local language models. Includes download manifests, validation scripts, documentation, and .cmd launchers with all inference parameters materialized.

This repository does not contain binary artifacts (GGUF weights, runtimes, downloaded packages). These are obtained by running the included installation scripts.

The installed artifacts in D:\LLM\models and the effective arguments in D:\LLM\scripts\models are the deployment source of truth. This repository mirrors their manifests and launchers; client catalogs do not select server profiles. See the Windows sync record.

Hardware Configuration

The following specifications are relevant for language model inference:

Component Specification
CPU AMD Ryzen 7 9800X3D
GPU RTX 5080 16 GB GDDR7
RAM 64 GB DDR5-6000 (2 x 32 GB)
Storage 2 TB NVMe (primary) + 2 TB NVMe PCIe 4.0 (secondary)

Available Models

All profiles are automatic: every launcher declares --gpu-layers auto --fit on and lets the runtime resolve the device placement. The hand-tuned profiles that pinned --gpu-layers 999 with --n-cpu-moe were retired on 2026-09-30. Their reference copy and the b10502-cuda13.3 runtime were deleted on 2026-10-01. Every launcher pins b12359-cuda13.5 (the build behind llama.cpp v1.0.0) since 2026-11-15, when it replaced b11981-cuda13.4 after measuring faster without MTP on Qwen3.6-35B-A3B; b11981 was removed the same day (changelog). b11981 had replaced b11551-cuda13.4 on 2026-10-30, and b11551 was removed that day (changelog). b11669-cuda13.4 (llama.cpp v0.7.0) was used on 2026-10-27 and removed on 2026-10-28 because it performed worse for the user. b11269-cuda13.4 was removed on 2026-10-22 and b11827-cuda13.4, which flint used until its retirement, on 2026-10-26; all five can be reinstalled.

The LiquidAI LFM3 models (lfm3-1.7b, lfm3-audio-2b, lfm3-omni-2b) and the llama.cpp runtime built from source for them were retired on 2026-10-28 (changelog), as were lfm2.5-24b-a2b, nemotron-3.5-bolt-e4b, ornith-1.5-35b-a3b and the base qwen3.6-35b-a3b (changelog). The MTP profiles were retired on 2026-10-28 and every model moved to its UD-Q4_K_XL build without an MTP head; Tiel Coder and Cyber Tiel dropped the -mtp suffix from their IDs (changelog).

From 2026-10-29 lfm3-1.7b ran from experimental/lfm3 with its own copy of b11981-cuda13.4 (changelog); the VL, Audio and Omni models installed with it were removed the same day (changelog). On 2026-11-01 the four LFM3 models joined this catalog, since it runs the same build: Audio and Omni with the newer projectors that quantize the codec decoder. Every LFM3 alias except Audio's is registered in Pi, OpenCode and DSH (changelog).

On 2026-10-29 nemotron-3.5-lightning-30b-a3b, retired on 2026-10-26, came back with the AD-IQ4_NL build from AtomicChat instead of the earlier NVFP4: K-quants do not fit its tensor shapes, so it is the only model not on UD-Q4_K_XL (changelog).

On 2026-10-30 flint, retired on 2026-10-26, came back with the same UD-Q4_K_XL revision on the official runtime b11981-cuda13.4, because b11551 cannot load its architecture. Its reasoning cannot be turned off per request, only capped with a positive budget (changelog).

On 2026-11-01 apex-35b-v1.0 (Apex Intelligence, built on Qwen3.6-35B-A3B) joined with its imatrix Q4_K_S build. It decides per turn whether and how long to think, so its launchers leave the reasoning budget unlimited; its GGUF carries an MTP head that the profiles do not use (changelog).

On 2026-11-02 gemma-4-26b-a4b-uncensored-balanced (HauhauCS, an uncensored build of Gemma 4 26B A4B from the official QAT weights) joined with its Q4_K_M build and BF16 projector, in text and vision profiles at 131K and 262K. Its MTP drafter ships as a separate file that is not installed (changelog).

On 2026-11-02 experimental/ came back as a separate root, outside this catalog, with a CPU-only 8K profile of gemma-4-26b-a4b-uncensored-balanced (sharing this catalog's GGUF) on loopback port 8081 and Start-Assistant.cmd, a terminal utility that starts the model, answers prompts with an instructions file and stops the model on exit. lfm3-1.7b-uncensored (poor prompt quality) and occult-nail-1.0-35b-a3b were tried first the same day. It is not registered in any client (changelog).

On 2026-11-11 qwen3.6-35b-a3b, retired on 2026-10-28, came back as the official Unsloth UD-Q4_K_XL build (the GGUF with the MTP layer, kept from a Strata comparison; the profiles do not use the layer) with the projector shared with Nail, in text and vision profiles at 131K and 262K. Strata was evaluated and removed the same day (changelog).

On 2026-11-16 ornith-1.5-9b (the official Q4_K_M from ornith-ai) and gemma-4-12b-it-qat (the Unsloth UD-Q4_K_XL QAT build with a BF16 projector and the external MTP drafter) came back. Ornith has text profiles at 131K and 262K, and the same day gained its BF16 projector with vision profiles at both contexts that have not been run yet (changelog); Gemma has text, MTP and vision profiles at 131K and 262K, and the MTP profiles are the only ones using speculative decoding (changelog).

On 2026-11-17 gemma-4-26b-a4b-it-qat (the Unsloth UD-Q4_K_XL QAT build, Q4_0 in its tensors, with a BF16 projector) replaced gemma-4-26b-a4b-uncensored-balanced in this catalog, in text and vision profiles at 131K and 262K without MTP; its profiles were not run, at the user's request. The HauhauCS weights stay in models\ because the unused Gemma launcher and manifest of experimental/ still reference them. The same day gemma-4-e4b-it-qat (Google's official Q4_0 QAT GGUF, with a projector for image and audio) joined with text and vision profiles at its native 131K (changelog).

The publisher is the Hugging Face organization whose GGUF files are installed; each manifest pins the exact repository and revision.

Identifier Publisher Profiles Profile aliases
tiel-coder-35b-a3b peculiar-ragdoll agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 tiel-coder-35b-a3b-131k, tiel-coder-35b-a3b-262k, tiel-coder-35b-a3b-vision-131k, tiel-coder-35b-a3b-vision-262k
nail-qwen3.6-35b-a3b peculiar-ragdoll agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 nail-qwen3.6-35b-a3b-131k, nail-qwen3.6-35b-a3b-262k, nail-qwen3.6-35b-a3b-vision-131k, nail-qwen3.6-35b-a3b-vision-262k
occult-nail-1.0-35b-a3b peculiar-ragdoll agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 occult-nail-1.0-35b-a3b-131k, occult-nail-1.0-35b-a3b-262k, occult-nail-1.0-35b-a3b-vision-131k, occult-nail-1.0-35b-a3b-vision-262k
cyber-tiel-coder-35b-a3b peculiar-ragdoll agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 cyber-tiel-coder-35b-a3b-131k, cyber-tiel-coder-35b-a3b-262k, cyber-tiel-coder-35b-a3b-vision-131k, cyber-tiel-coder-35b-a3b-vision-262k
nemotron-3.5-lightning-30b-a3b AtomicChat agentic-auto-131k-1024, agentic-auto-262k-1024 nemotron-3.5-lightning-30b-a3b-131k, nemotron-3.5-lightning-30b-a3b-262k
flint unsloth agentic-auto-131k-1024, agentic-auto-262k-1024 flint-131k, flint-262k
apex-35b-v1.0 Apexintelligence-AI agentic-auto-131k-1024, agentic-auto-262k-1024 apex-35b-v1.0-131k, apex-35b-v1.0-262k
gemma-4-26b-a4b-it-qat unsloth agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 gemma-4-26b-a4b-it-qat-131k, gemma-4-26b-a4b-it-qat-262k, gemma-4-26b-a4b-it-qat-vision-131k, gemma-4-26b-a4b-it-qat-vision-262k
lfm3-1.7b LiquidAI agentic-auto-262k-1024 lfm3-1.7b-262k
lfm3-vl-2b LiquidAI agentic-auto-vision-128k-1024 lfm3-vl-2b-vision-128k
lfm3-audio-2b LiquidAI agentic-auto-audio-32k-1024 lfm3-audio-2b-audio-32k
lfm3-omni-2b LiquidAI agentic-auto-omni-32k-1024 lfm3-omni-2b-omni-32k
qwen3.6-35b-a3b unsloth agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 qwen3.6-35b-a3b-131k, qwen3.6-35b-a3b-262k, qwen3.6-35b-a3b-vision-131k, qwen3.6-35b-a3b-vision-262k
ornith-1.5-9b ornith-ai agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 ornith-1.5-9b-131k, ornith-1.5-9b-262k, ornith-1.5-9b-vision-131k, ornith-1.5-9b-vision-262k
gemma-4-12b-it-qat unsloth agentic-auto-131k-1024, agentic-auto-262k-1024, agentic-auto-mtp-131k-1024, agentic-auto-mtp-262k-1024, agentic-auto-vision-131k-1024, agentic-auto-vision-262k-1024 gemma-4-12b-it-qat-131k, gemma-4-12b-it-qat-262k, gemma-4-12b-it-qat-mtp-131k, gemma-4-12b-it-qat-mtp-262k, gemma-4-12b-it-qat-vision-131k, gemma-4-12b-it-qat-vision-262k
gemma-4-e4b-it-qat google agentic-auto-131k-1024, agentic-auto-vision-131k-1024 gemma-4-e4b-it-qat-131k, gemma-4-e4b-it-qat-vision-131k

Repository Structure

├── config/
│   ├── catalog.psd1              # Model catalog
│   └── models/                   # Per-model manifests
├── scripts/
│   ├── common/                   # Launcher and integrity validation
│   ├── models/<model-id>/        # start-<profile>-auto.cmd launchers
│   └── setup/                    # Model download and runtime installation
├── docs/
│   ├── adr/                      # Architecture decision records
│   ├── overview/                 # General documentation and model cards
│   └── changelog/                # Change history
├── experimental/                 # Separate root outside the catalog (see its README)
├── speech/                       # Speech models on NeMo-Speech.cpp, outside the catalog and its clients (see its README)

Installation

  1. Install the runtime:
powershell -NoProfile -ExecutionPolicy RemoteSigned -File scripts/setup/Install-LlamaRuntime.ps1

The default is b12359, the build every launcher uses; -Version b11981 reinstalls the previous pinned build if a rollback is needed.

  1. Download a model:
powershell -NoProfile -ExecutionPolicy RemoteSigned -File scripts/setup/Download-Model.ps1 -Model nail-qwen3.6-35b-a3b
  1. Validate integrity:
powershell -NoProfile -ExecutionPolicy RemoteSigned -File scripts/common/Test-ModelIntegrity.ps1 -Model nail-qwen3.6-35b-a3b

Launchers

The launchers are grouped by model (representative commands below):

scripts\models\tiel-coder-35b-a3b\start-agentic-auto-131k-1024.cmd
scripts\models\tiel-coder-35b-a3b\start-agentic-auto-vision-131k-1024.cmd
scripts\models\nail-qwen3.6-35b-a3b\start-agentic-auto-131k-1024.cmd
scripts\models\occult-nail-1.0-35b-a3b\start-agentic-auto-131k-1024.cmd
scripts\models\cyber-tiel-coder-35b-a3b\start-agentic-auto-131k-1024.cmd
scripts\models\nemotron-3.5-lightning-30b-a3b\start-agentic-auto-131k-1024.cmd
scripts\models\flint\start-agentic-auto-131k-1024.cmd
scripts\models\apex-35b-v1.0\start-agentic-auto-131k-1024.cmd
scripts\models\lfm3-1.7b\start-agentic-auto-262k-1024.cmd
scripts\models\lfm3-vl-2b\start-agentic-auto-vision-128k-1024.cmd
scripts\models\lfm3-audio-2b\start-agentic-auto-audio-32k-1024.cmd
scripts\models\lfm3-omni-2b\start-agentic-auto-omni-32k-1024.cmd

Each .cmd contains the runtime path, model path, and all effective parameters for its profile. The launchers materialize --gpu-layers auto, --fit on and --fit-target 1024 —2048 in the vision profiles, because --fit does not account for the projector— and declare the context explicitly, since --fit only adjusts unset arguments. They omit --n-cpu-moe: --fit places the weights with a per-layer -ot override instead of that lever. To customize context, port, sampling, or other values, edit the corresponding launcher directly. The scripts do not accept hidden additional arguments.

Every launcher declares its --alias as <model-id>,<profile-id>, listed in the table above. The bare model id stays first, so clients that already name it keep working, and /v1/models reports both names. The second alias is a label: each client lists one entry per profile alias so the user can see which model, context and vision mode they picked. It does not route or choose a profile: every entry points at the same endpoint, the running launcher answers a request whatever alias it names (on 2026-11-11 the vision 131K launcher of qwen3.6-35b-a3b answered qwen3.6-35b-a3b-262k), and it still decides context, sampling, placement, reasoning budget and whether images are accepted. Only one profile can hold port 8080 at a time, so pick the client entry that matches the launcher you started.

Test-Llm.ps1 requires the first alias to be the model id, requires each profile alias to be unique across launchers, and refuses a profile alias that shadows a model id.

The terminal remains linked to llama-server and displays loading, prompt processing, speed, and errors in real time. Stop the server with Ctrl+C in that same terminal.

API

  • Web interface: http://localhost:8080
  • OpenAI API: http://localhost:8080/v1
  • Health: http://localhost:8080/health
  • Models: http://localhost:8080/v1/models

/v1/models reports the bare model id and the profile id of the running launcher; a request may name either one. /health answers only while a launcher is running, so it doubles as the check for whether port 8080 is free.

Profiles listen on 0.0.0.0:8080 without authentication. Use a private trusted network only. The Pi/OpenCode WSL clients use http://192.168.1.100:8080/v1; Windows-only loopback did not work from this WSL setup. Restart an existing server after changing a launcher. No firewall rules are changed. Only one profile can use port 8080 at a time; check /health and /v1/models from WSL after startup.

Validation

Validate all launchers declaratively without loading weights:

powershell -NoProfile -ExecutionPolicy RemoteSigned -File scripts/common/Test-Llm.ps1 -ConfigurationOnly

Verify model hashes:

powershell -NoProfile -ExecutionPolicy RemoteSigned -File scripts/common/Test-ModelIntegrity.ps1 -Model nail-qwen3.6-35b-a3b

Run a brief functional test:

powershell -NoProfile -ExecutionPolicy RemoteSigned -File scripts/common/Test-Llm.ps1 -Model nail-qwen3.6-35b-a3b -Profile agentic-auto-vision-131k-1024

Functional tests control and close only the process tree they initiate.

Documentation

About

Declarative configuration for self-contained launchers of local language models. Includes download manifests, validation scripts, documentation, and .cmd launchers with all inference parameters materialized.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages