How I ran Ternary-Bonsai-2-27B-PTQ1_0.gguf on macOS

[simonw]

This worked for me:

cd /tmp

# Get the Prism macOS runtime
curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
tar -xzf bonsai-runtime.tar.gz

# Get the ~5.95 GB GGUF model:
curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf

# Run the server, I used port 8331
./llama-prism-b10685-7dffb15/llama-server \
  -m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
  --port 8331 -ngl 99 -fa on -c 32768
Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:
uvx llm openai endpoint http://127.0.0.1:8331/v1 \
  --model bonsai-2-27b --responses hi

See also https://news.ycombinator.com/item?id=49746618#49747390

I used it to "Generate an SVG of a pelican riding a bicycle" and got this https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2Fba4cf3a88f4e7dc32994f2672150f770 after 18 minutes 20 seconds.

[khosravipasha]

🤖 Answering on behalf of the maintainers

Sorry we missed this! We hadn’t realized Discussions was enabled on this fork and weren’t receiving notifications.

Thanks for sharing the macOS walkthrough and the pelican SVG! It’s great to see the example and the reasoning trace together.

Your GPU offload, flash-attention, and 32K context settings look reasonable. For anyone trying this now, we recommend the latest runtime release. The command leaves sampling at the runtime defaults; our current 27B thinking settings are --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 (API clients can override these).

Thanks again for testing it and making the result easy for others to explore!