PrismML-Eng/Bonsai-demoPublic

Bonsai Demo

AI summary: A comprehensive demo repository for running the highly efficient 1-bit Bonsai and Ternary-Bonsai models.

Stars
3.5K
+16 today
Forks
409
Watchers
31
Open issues
13
Open PRs
7
Contributors
~62
Commits
209
Branches
7
Views (30d)
4

ShellApache-2.0Created Mar 25, 2026Last push 16d ago+27 stars this week+145 this month

Quick answers

What is Bonsai-demo?
A comprehensive demo repository for running the highly efficient 1-bit Bonsai and Ternary-Bonsai models.
What does Bonsai-demo do?
Bonsai Demo provides the tools, interfaces, and instructions needed to run the highly compressed Bonsai language models on consumer hardware. It supports the latest Bonsai 27B vision-language models, as well as 1-bit and ternary variants. The repository offers implementations for Mac via Metal, and Linux/Windows via CUDA, Vulkan, or CPU, ensuring broad hardware compatibility. It showcases advanced features like native OpenAI-style agentic tool calling and direct integration with Model Context Protocol (MCP) servers. The demo serves as the primary entry point for developers looking to deploy extremely resource-efficient LLMs locally.
Who is Bonsai-demo for?
AI researchers, local LLM enthusiasts, and developers looking to run highly efficient, quantized models on consumer hardware. It is ideal for those wanting to explore the cutting edge of 1-bit and ternary model performance.
How do I get started with Bonsai-demo?
https://github.com/PrismML-Eng/Bonsai-demo
How popular is Bonsai-demo on GitHub?
PrismML-Eng/Bonsai-demo has 3,529 stars and 409 forks on GitHub, and gained 27 stars in the last 7 days.
What license does Bonsai-demo use?
PrismML-Eng/Bonsai-demo is released under the Apache-2.0 license.

Star history

since Jul 28, 2026
01K2K3KJul 2026Sep 2026Oct 2026Nov 2026
3.5K stars as of Nov 17, 2026. Measured daily since Jul 28, 2026; GitHub no longer exposes earlier star timestamps.

Contribution activity

commits per day, last 52 weeks
NovDecJanFebMarAprMayJunJulAugSepOctNovMonWedFri2025-11-23: 0 commits2025-11-24: 0 commits2025-11-25: 0 commits2025-11-26: 0 commits2025-11-27: 0 commits2025-11-28: 0 commits2025-11-29: 0 commits2025-11-30: 0 commits2025-12-01: 0 commits2025-12-02: 0 commits2025-12-03: 0 commits2025-12-04: 0 commits2025-12-05: 0 commits2025-12-06: 0 commits2025-12-07: 0 commits2025-12-08: 0 commits2025-12-09: 0 commits2025-12-10: 0 commits2025-12-11: 0 commits2025-12-12: 0 commits2025-12-13: 0 commits2025-12-14: 0 commits2025-12-15: 0 commits2025-12-16: 0 commits2025-12-17: 0 commits2025-12-18: 0 commits2025-12-19: 0 commits2025-12-20: 0 commits2025-12-21: 0 commits2025-12-22: 0 commits2025-12-23: 0 commits2025-12-24: 0 commits2025-12-25: 0 commits2025-12-26: 0 commits2025-12-27: 0 commits2025-12-28: 0 commits2025-12-29: 0 commits2025-12-30: 0 commits2025-12-31: 0 commits2026-01-01: 0 commits2026-01-02: 0 commits2026-01-03: 0 commits2026-01-04: 0 commits2026-01-05: 0 commits2026-01-06: 0 commits2026-01-07: 0 commits2026-01-08: 0 commits2026-01-09: 0 commits2026-01-10: 0 commits2026-01-11: 0 commits2026-01-12: 0 commits2026-01-13: 0 commits2026-01-14: 0 commits2026-01-15: 0 commits2026-01-16: 0 commits2026-01-17: 0 commits2026-01-18: 0 commits2026-01-19: 0 commits2026-01-20: 0 commits2026-01-21: 0 commits2026-01-22: 0 commits2026-01-23: 0 commits2026-01-24: 0 commits2026-01-25: 0 commits2026-01-26: 0 commits2026-01-27: 0 commits2026-01-28: 0 commits2026-01-29: 0 commits2026-01-30: 0 commits2026-01-31: 0 commits2026-02-01: 0 commits2026-02-02: 0 commits2026-02-03: 0 commits2026-02-04: 0 commits2026-02-05: 0 commits2026-02-06: 0 commits2026-02-07: 0 commits2026-02-08: 0 commits2026-02-09: 0 commits2026-02-10: 0 commits2026-02-11: 0 commits2026-02-12: 0 commits2026-02-13: 0 commits2026-02-14: 0 commits2026-02-15: 0 commits2026-02-16: 0 commits2026-02-17: 0 commits2026-02-18: 0 commits2026-02-19: 0 commits2026-02-20: 0 commits2026-02-21: 0 commits2026-02-22: 0 commits2026-02-23: 0 commits2026-02-24: 0 commits2026-02-25: 0 commits2026-02-26: 0 commits2026-02-27: 0 commits2026-02-28: 0 commits2026-03-01: 0 commits2026-03-02: 0 commits2026-03-03: 0 commits2026-03-04: 0 commits2026-03-05: 0 commits2026-03-06: 0 commits2026-03-07: 0 commits2026-03-08: 0 commits2026-03-09: 0 commits2026-03-10: 0 commits2026-03-11: 0 commits2026-03-12: 0 commits2026-03-13: 0 commits2026-03-14: 0 commits2026-03-15: 0 commits2026-03-16: 0 commits2026-03-17: 0 commits2026-03-18: 0 commits2026-03-19: 0 commits2026-03-20: 0 commits2026-03-21: 0 commits2026-03-22: 0 commits2026-03-23: 0 commits2026-03-24: 0 commits2026-03-25: 1 commit2026-03-26: 0 commits2026-03-27: 0 commits2026-03-28: 0 commits2026-03-29: 0 commits2026-03-30: 2 commits2026-03-31: 3 commits2026-04-01: 0 commits2026-04-02: 0 commits2026-04-03: 0 commits2026-04-04: 1 commit2026-04-05: 0 commits2026-04-06: 5 commits2026-04-07: 2 commits2026-04-08: 2 commits2026-04-09: 1 commit2026-04-10: 0 commits2026-04-11: 0 commits2026-04-12: 0 commits2026-04-13: 9 commits2026-04-14: 0 commits2026-04-15: 23 commits2026-04-16: 7 commits2026-04-17: 1 commit2026-04-18: 5 commits2026-04-19: 0 commits2026-04-20: 4 commits2026-04-21: 0 commits2026-04-22: 1 commit2026-04-23: 3 commits2026-04-24: 0 commits2026-04-25: 0 commits2026-04-26: 0 commits2026-04-27: 0 commits2026-04-28: 0 commits2026-04-29: 0 commits2026-04-30: 0 commits2026-05-01: 0 commits2026-05-02: 0 commits2026-05-03: 0 commits2026-05-04: 0 commits2026-05-05: 0 commits2026-05-06: 1 commit2026-05-07: 0 commits2026-05-08: 0 commits2026-05-09: 0 commits2026-05-10: 0 commits2026-05-11: 0 commits2026-05-12: 0 commits2026-05-13: 0 commits2026-05-14: 0 commits2026-05-15: 0 commits2026-05-16: 0 commits2026-05-17: 0 commits2026-05-18: 0 commits2026-05-19: 0 commits2026-05-20: 0 commits2026-05-21: 0 commits2026-05-22: 0 commits2026-05-23: 0 commits2026-05-24: 0 commits2026-05-25: 0 commits2026-05-26: 0 commits2026-05-27: 0 commits2026-05-28: 0 commits2026-05-29: 0 commits2026-05-30: 0 commits2026-05-31: 1 commit2026-06-01: 0 commits2026-06-02: 0 commits2026-06-03: 0 commits2026-06-04: 0 commits2026-06-05: 0 commits2026-06-06: 0 commits2026-06-07: 0 commits2026-06-08: 0 commits2026-06-09: 0 commits2026-06-10: 0 commits2026-06-11: 0 commits2026-06-12: 0 commits2026-06-13: 0 commits2026-06-14: 0 commits2026-06-15: 0 commits2026-06-16: 0 commits2026-06-17: 0 commits2026-06-18: 0 commits2026-06-19: 0 commits2026-06-20: 0 commits2026-06-21: 0 commits2026-06-22: 0 commits2026-06-23: 0 commits2026-06-24: 0 commits2026-06-25: 0 commits2026-06-26: 0 commits2026-06-27: 0 commits2026-06-28: 0 commits2026-06-29: 0 commits2026-06-30: 0 commits2026-07-01: 0 commits2026-07-02: 0 commits2026-07-03: 0 commits2026-07-04: 0 commits2026-07-05: 0 commits2026-07-06: 0 commits2026-07-07: 0 commits2026-07-08: 0 commits2026-07-09: 0 commits2026-07-10: 0 commits2026-07-11: 0 commits2026-07-12: 0 commits2026-07-13: 3 commits2026-07-14: 2 commits2026-07-15: 1 commit2026-07-16: 0 commits2026-07-17: 9 commits2026-07-18: 1 commit2026-07-19: 3 commits2026-07-20: 4 commits2026-07-21: 0 commits2026-07-22: 4 commits2026-07-23: 1 commit2026-07-24: 0 commits2026-07-25: 3 commits2026-07-26: 0 commits2026-07-27: 0 commits2026-07-28: 0 commits2026-07-29: 1 commit2026-07-30: 1 commit2026-07-31: 0 commits2026-08-01: 0 commits2026-08-02: 0 commits2026-08-03: 0 commits2026-08-04: 0 commits2026-08-05: 2 commits2026-08-06: 0 commits2026-08-07: 0 commits2026-08-08: 0 commits2026-08-09: 0 commits2026-08-10: 0 commits2026-08-11: 0 commits2026-08-12: 0 commits2026-08-13: 0 commits2026-08-14: 0 commits2026-08-15: 0 commits2026-08-16: 0 commits2026-08-17: 0 commits2026-08-18: 0 commits2026-08-19: 0 commits2026-08-20: 0 commits2026-08-21: 0 commits2026-08-22: 0 commits2026-08-23: 0 commits2026-08-24: 0 commits2026-08-25: 1 commit2026-08-26: 0 commits2026-08-27: 2 commits2026-08-28: 9 commits2026-08-29: 0 commits2026-08-30: 0 commits2026-08-31: 0 commits2026-09-01: 0 commits2026-09-02: 0 commits2026-09-03: 1 commit2026-09-04: 0 commits2026-09-05: 0 commits2026-09-06: 0 commits2026-09-07: 0 commits2026-09-08: 0 commits2026-09-09: 0 commits2026-09-10: 1 commit2026-09-11: 0 commits2026-09-12: 0 commits2026-09-13: 0 commits2026-09-14: 1 commit2026-09-15: 0 commits2026-09-16: 0 commits2026-09-17: 2 commits2026-09-18: 2 commits2026-09-19: 0 commits2026-09-20: 7 commits2026-09-21: 10 commits2026-09-22: 4 commits2026-09-23: 9 commits2026-09-24: 12 commits2026-09-25: 3 commits2026-09-26: 0 commits2026-09-27: 0 commits2026-09-28: 1 commit2026-09-29: 2 commits2026-09-30: 0 commits2026-10-01: 0 commits2026-10-02: 0 commits2026-10-03: 0 commits2026-10-04: 0 commits2026-10-05: 2 commits2026-10-06: 1 commit2026-10-07: 0 commits2026-10-08: 0 commits2026-10-09: 1 commit2026-10-10: 1 commit2026-10-11: 1 commit2026-10-12: 0 commits2026-10-13: 0 commits2026-10-14: 0 commits2026-10-15: 3 commits2026-10-16: 0 commits2026-10-17: 0 commits2026-10-18: 0 commits2026-10-19: 1 commit2026-10-20: 0 commits2026-10-21: 1 commit2026-10-22: 0 commits2026-10-23: 0 commits2026-10-24: 1 commit2026-10-25: 4 commits2026-10-26: 3 commits2026-10-27: 0 commits2026-10-28: 0 commits2026-10-29: 1 commit2026-10-30: 0 commits2026-10-31: 0 commits2026-11-01: 0 commits2026-11-02: 2 commits2026-11-03: 1 commit2026-11-04: 1 commit2026-11-05: 0 commits2026-11-06: 0 commits2026-11-07: 0 commits2026-11-08: 0 commits2026-11-09: 0 commits2026-11-10: 0 commits2026-11-11: 0 commits2026-11-12: 0 commits2026-11-13: 0 commits2026-11-14: 0 commits2026-11-15: 0 commits2026-11-16: 0 commits2026-11-17: 0 commits2026-11-18: 0 commits2026-11-19: 0 commits2026-11-20: 0 commits2026-11-21: 0 commits
198 commits in the last yearLessMore

Signals and awards

derived from tracked data
  • Permissive license

    Apache-2.0

  • Continuous integration

    Automated checks passing

  • Repeat trending

    5 trending appearances

What Bonsai-demo does

Bonsai Demo provides the tools, interfaces, and instructions needed to run the highly compressed Bonsai language models on consumer hardware. It supports the latest Bonsai 27B vision-language models, as well as 1-bit and ternary variants. The repository offers implementations for Mac via Metal, and Linux/Windows via CUDA, Vulkan, or CPU, ensuring broad hardware compatibility. It showcases advanced features like native OpenAI-style agentic tool calling and direct integration with Model Context Protocol (MCP) servers. The demo serves as the primary entry point for developers looking to deploy extremely resource-efficient LLMs locally.

AI researchers, local LLM enthusiasts, and developers looking to run highly efficient, quantized models on consumer hardware. It is ideal for those wanting to explore the cutting edge of 1-bit and ternary model performance.

  • Broad hardware support: Runs efficiently across Apple Silicon (Metal), NVIDIA (CUDA), AMD (ROCm), Vulkan, and standard CPUs.
  • Vision-language capabilities: Supports the new Bonsai 27B models for analyzing photos, screenshots, and PDFs natively.
  • Agentic tool calling: Implements native, OpenAI-compatible tool calling with full round-trip execution for autonomous tasks.
  • Model Context Protocol support: Integrates seamlessly with MCP servers, allowing the models to access external data sources.
  • Extreme quantization: Showcases 1-bit and ternary quantized models that drastically reduce memory usage without destroying performance.
  • Multiple demo interfaces: Includes both terminal-based and graphical UIs for interacting with the local models immediately.

Where teams use it

Low-resource local inference

Run capable, multi-billion parameter language models on older laptops or devices with limited unified memory.

Local vision-language analysis

Use the Bonsai 27B model to privately analyze sensitive PDF documents or screenshots entirely on local hardware.

Agentic workflow testing

Deploy the models to act as local, fast-executing agents that use standard tool calls to interact with MCP servers.

Quantization research

Experiment with the provided 1-bit and ternary models to understand the performance trade-offs of extreme LLM compression.

Getting started: https://github.com/PrismML-Eng/Bonsai-demo

README

main branch

Bonsai Demo

Backend and model format compatibility: BACKEND-SUPPORT.md.

Bonsai

Website  |  GitHub  |  Discord

Models: Bonsai 2 27B GGUF · Bonsai 2 27B MLX


Run Bonsai 2 2B, 4B, 9B, and 27B locally on Mac (Metal), Linux/Windows (CUDA, Vulkan, ROCm), or CPU.

For the earlier Bonsai 1-bit and Ternary-Bonsai models (27B, 8B, 4B, 1.7B), see Bonsai1_README.md. For troubleshooting, see FAQ.md.

🌱 Bonsai 2: 2B, 4B, 9B, and 27B

Bonsai 2 now includes 2B, 4B, and 9B models alongside 27B. Pick a smaller model for a lighter local setup; 27B remains the demo's default. GGUF and MLX checkpoints are available for each size.

The following benchmark and memory highlights describe 27B; see each smaller model's card for its specifications:

  • 98.2% of FP16 intelligence retained at roughly a ninth of the size, with the reasoning core intact: math within half a point of full precision, coding level with the baseline.
  • Vision: send photos, screenshots and PDFs and ask about them, on both llama.cpp and MLX (see VISION.md).
  • Agentic tool calling: native OpenAI-style tool_calls with full round-trips, plus MCP servers in both demo UIs (see TOOLS.md).
  • Thinking: a reasoning model; pick the reasoning effort per chat in the UI or budget it per request.
  • 262K-token context, kept practical on-device by the hybrid-attention backbone.
  • 1.75 bits per weight in the PTQ1_0 packing. A second packing, PQ2_0, trades 1.3 GB for faster prompt processing and is what this demo downloads by default. See MODEL-FORMATS.md.

Official Q2_0 works with upstream llama.cpp b11556 or newer. PQ2_0 and PTQ1_0 still require the PrismML fork. This demo downloads PQ2_0 and the matching fork binaries by default. All three packings contain the same model weights, with different memory/speed trade-offs.

Quick Start below gets you there in two commands: ./setup.sh downloads Bonsai 2 27B, then ./scripts/start_llama_server.sh gives you chat, vision and tools at http://localhost:8080.

Choose a smaller model

export BONSAI_MODEL=4B    # 2B, 4B, 9B, or 27B
./setup.sh
./scripts/start_llama_server.sh

On Windows, set $env:BONSAI_MODEL = "4B" before setup and launch. For Apple Silicon, use ./scripts/run_mlx.sh or ./scripts/start_mlx_server.sh with the same model selection. See small-model sampling and API examples: the 2B/4B/9B profiles use min-p 0 and a 64-token presence-penalty window. The 27B settings below are separate.

Best Practices (Bonsai 2 27B)

These match the model card. The start scripts already apply the thinking-mode sampling, so you only need these when calling the model from your own client.

Generation Parameters

Thinking mode (default) Instruct / non-thinking
temperature 1.0 0.7
top_p 0.95 0.80
top_k 20 20
min_p 0.05 0.0
presence_penalty 0.0 1.5
repetition_penalty 1.0 1.0

min_p=0.05 drops tokens far less likely than the top choice; in our tests it scored at least as well as min_p=0.0 and followed instructions more reliably. It is also llama.cpp's default. Give the model a generous output limit (-n 16384 or more, max_tokens for the API): it reasons before it answers, and a small cap ends generation mid-thought.

The model uses xhigh reasoning effort by default; use medium for shorter responses and a balance of speed and accuracy. low reasoning effort is not supported and when selected the model will behave close to xhigh.

System Prompt

A simple system prompt works well:

You are a helpful assistant

Quick Start

For repeated conversations and prompt-cache troubleshooting, see Prompt reuse and context checkpoints.

Setting things up with an AI coding agent? Point it at AGENTS.md, a guide written for agents (hardware-specific knobs, defaults, and what to ask the user).

macOS / Linux

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
./setup.sh

That installs and downloads only. To chat, start the server yourself, then open http://localhost:8080:

./scripts/start_llama_server.sh

Or ask one question from the terminal without a server:

./scripts/run_llama.sh -p "What is the capital of France?"

setup.sh fetches the llama.cpp binaries for your machine and the Bonsai 2 27B weights, 7.8 GB in the PQ2_0 packing this demo defaults to plus its vision projector. It also sets up Open WebUI and the code interpreter, which add a few GB more and most of the wait. Skip those with BONSAI_OPENWEBUI=0 and BONSAI_CODE_INTERPRETER=0.

Windows (PowerShell)

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\setup.ps1

Then start the server and open http://localhost:8080:

.\scripts\start_llama_server.ps1

Speed Benchmarks

See community-benchmarks/ for results on different hardware and templates to submit your own.

Models

Bonsai 2 27B is the default: plain ./setup.sh downloads it. It supports images as well as text.

Bonsai 2 (ternary, default)

Available in GGUF and MLX 2-bit formats. Use official llama.cpp b11556 or newer for Q2_0, or our fork for PQ2_0 / PTQ1_0. Setup installs the fork for its default PQ2_0 download.

Model GGUF MLX (2-bit)
Bonsai 2 2B GGUF MLX
Bonsai 2 4B GGUF MLX
Bonsai 2 9B GGUF MLX
Bonsai 2 27B GGUF MLX

On Apple Silicon, Bonsai 2 text, images, and tool calls are supported by native mlx-vlm==0.7.8 in .venv-vlm, with mlx-lm==0.32.0 pinned alongside it. The packs load through mlx_vlm.load; the bundled model runtime/ loader is not needed. Re-run ./setup.sh to upgrade an older environment, then use ./scripts/start_mlx_server.sh. The optional BONSAI_BACKEND=mlx ./scripts/start_openwebui.sh integration remains for 27B; use the standalone launchers for the new sizes. The server keeps thinking enabled; use ./scripts/start_mlx_server.sh --thinking-budget 8192 --max-tokens 32768 for an example bounded server run (these are server flags, not run_mlx.sh flags). API clients use thinking_budget (MLX), not llama-server's thinking_budget_tokens. Use the model-card sampling settings in API requests: temperature: 1.0, top_p: 0.95, top_k: 20, min_p: 0.05 for 27B. For 2B/4B/9B, use the small-model profiles, including the explicit presence penalty and window. Open WebUI refuses to reuse an existing server on its MLX port for Bonsai 2 because its loader cannot be verified; stop it first so the launcher can start the tested runtime. This does not update LM Studio's separately bundled MLX runtime.

Environment variables

No model-selection variables are needed for Bonsai 2 27B: that's what plain ./setup.sh downloads and runs.

Every launcher is configured through environment variables. The most common ones:

Variable Default Values Purpose
BONSAI_MODEL 27B 2B, 4B, 9B, 27B for Bonsai 2 Select model size.
BONSAI_THINKING 1 0, 1 Small-model GGUF launchers and MLX one-shot profile; MLX server clients set the mode and sampling per request.
BONSAI_FAMILY bonsai2 bonsai2 Earlier families: Bonsai 1 guide.
BONSAI_NGL auto-detect int; 0 = CPU-only GPU layer offload.
BONSAI_CTX auto (RAM-tiered) 0, or ≤ 262144 Context length (0/unset = automatic safe size).
BONSAI_HOST 127.0.0.1 any bind address Server bind address. A non-loopback value exposes the server — see the security note in the full reference.
BONSAI_KV4 0 1 4-bit KV cache for long contexts (KV-CACHE.md).

Full reference — all variables (model/setup, server, MLX, Open WebUI, tools, and platform coverage): environment_variables.md.

Upstream support

  • llama.cpp: Bonsai 2 (2B, 4B, 9B and 27B) works in official Q2_0 format on upstream llama.cpp from release b11556 onward, which merged the model/Hadamard integration. Older engines may recognize the format but miss the required transforms.
  • PrismML fork: still required for PQ2_0 and PTQ1_0, and for the MTP and DFlash2 drafters on the model cards. Other experimental features may also need our fork; follow the instructions for the specific checkpoint and runtime.
  • MLX: the 1-bit support and Hadamard changes have merged upstream. Support should reach mlx-lm and other MLX engines as they integrate the model support and ship updated runtimes. A core MLX merge alone does not make every engine able to load every checkpoint. This demo currently uses the tested native mlx-vlm loader for Bonsai 2; our current prism_hadamard_qwen35 checkpoints are not directly loadable by the tested mlx-lm version.

Apps such as LM Studio and Locally AI bundle their own engines. Compatibility follows the version and model loader they ship, not just the model file extension. See backend support for our fork's platform matrix.

Earlier model formats and upstream status have moved to the Bonsai 1 guide.

What setup.sh Does

The setup script handles everything for you, even on a fresh machine:

  1. Checks/installs system deps: Xcode CLT on macOS, build-essential on Linux
  2. Installs uv: fast Python package manager (user-local, not global)
  3. Creates a Python venv and runs uv sync — installs cmake, ninja, huggingface-cli from pyproject.toml
  4. Downloads models from Hugging Face (public checkpoints download without a token)
  5. Downloads pre-built binaries from the pinned GitHub Release (or builds from source if you prefer)
  6. Sets up MLX (Apple Silicon): installs the native mlx-vlm environment for Bonsai 2; earlier-family runtime details are in the Bonsai 1 guide
  7. Installs Open WebUI into the venv for the agentic demo (skip with BONSAI_OPENWEBUI=0)
  8. Builds the code-interpreter venv (.venv-jupyter): Jupyter + matplotlib / pandas / numpy / scipy / sympy / yfinance for the Open WebUI code interpreter (skip with BONSAI_CODE_INTERPRETER=0)

Re-running setup.sh is safe — it skips already-completed steps.


Running the Model

Every script runs Bonsai 2 27B unless you set BONSAI_FAMILY and BONSAI_MODEL to pick another one (Environment variables).

llama.cpp (Mac / Linux — auto-detects platform)

./scripts/run_llama.sh -p "What is the capital of France?"

These scripts run the llama.cpp backend and need GGUF weights. On an MLX-only setup (e.g. you used BONSAI_SKIP_GGUF=1), they stop with an error that points at both options — running the MLX script directly (run_mlx.sh / start_mlx_server.sh), or downloading the GGUF weights.

llama.cpp (Windows PowerShell)

.\scripts\run_llama.ps1 -p "What is the capital of France?"

MLX — Mac (Apple Silicon)

./scripts/run_mlx.sh -p "What is the capital of France?"

Bonsai 2 uses the native mlx-vlm environment in .venv-vlm; the launcher selects it automatically. See MLX support and settings above.

Chat Server

Start llama-server with its built-in chat UI:

./scripts/start_llama_server.sh    # http://localhost:8080

For Windows PowerShell:

.\scripts\start_llama_server.ps1

The scripts auto-detect your GPU (Metal, CUDA, ROCm, Vulkan) and offload all layers. If the detection picks a GPU you do not want, for example Vulkan on a machine whose only GPU is a weak integrated one, set BONSAI_NGL=0 for CPU-only inference, or any layer count for partial offload (PowerShell: $env:BONSAI_NGL = "0").

Thinking

Bonsai 2 serves with thinking enabled by default. For small models, switch the sampling profile too when disabling thinking; see SMALL-MODELS.md. To adjust it per conversation in the chat UI (no restart): click the lightbulb in the message box and pick a Reasoning effort: Off, Low (512 tokens), Medium (2,048), High (8,192), or Max (unlimited). The pick persists per browser and is sent with every request.

On slower hardware, thinking is usually the bulk of the wait; pick a lower effort in the UI. For API clients that don't specify a reasoning effort, you can cap the server-wide default by passing llama-server flags straight through the start script:

./scripts/start_llama_server.sh --reasoning-budget 2048

For API clients, the model's own reasoning_effort: "medium" is usually the better way to shorten thinking. At moderate output limits it thinks noticeably less than the default xhigh and is about as accurate, and it rarely runs out of budget mid-reasoning. Send it per request, or make it the server-wide default (a request can still ask for xhigh):

./scripts/start_llama_server.sh --chat-template-kwargs '{"reasoning_effort":"medium"}'

The chat UI's Reasoning effort levels are different: they are fixed thinking budgets (Medium is 2,048 tokens), which cut thinking off at that length rather than asking the model to think less.

Tool calling & MCP

The 27B does native OpenAI-style tool calling over the API, and the chat UI has an MCP client with Hugging Face + DeepWiki preconfigured (per-chat opt-in from the MCP selector in the message box, no prompt cost until you turn one on). Details, costs, and how to add your own servers: TOOLS.md.

Agentic demo

Bonsai 2 driving the Hermes agent end to end: from a two-line brief to a playable 3D skateboard game it verified in its own browser, then a round of plain-English feedback, all run with a fixed seed. Clip, pages, prompts, settings and the two scripts that run it: AGENT-DEMO.md.

Vision

Upload images in the chat UI (+ in the message box) or send image_url parts over the API; the scripts load the vision projector automatically and downscale very large images on slower backends. Costs, the image-token cap, and OCR tips: VISION.md.

Optional extras

Optional features for the llama.cpp chat server:

  • 4-bit KV cache: BONSAI_KV4=1 cuts KV-cache memory roughly 3.5x for very long contexts, with an optional calibration bias for better quality (./scripts/make_kv_bias.sh). Details: KV-CACHE.md.
  • Vision projector in RAM: BONSAI_MMPROJ_CPU=1 keeps the 27B's vision projector in system RAM instead of VRAM (--no-mmproj-offload), freeing VRAM for KV/context on tight cards. The cost is a slower image prompt (the projector runs on CPU); text-only chat is unaffected.

Context Size

Bonsai 2 27B supports up to 262,144 tokens of context. The FP16 KV cache costs 64 KiB per token (~6.3 GiB at 100K), so 100K context fits on many consumer devices even without KV-cache quantization. The model's hybrid attention keeps the cache small for its size.

The launch scripts pick a default context sized to your machine's RAM, from 8K on small machines up to 131K for the 27B on machines with more than 71 GB (roughly 0.5 to 8 GiB of KV cache), so memory use stays predictable. Override with the BONSAI_CTX environment variable: pass any number up to 262144, or 0 (the same as leaving it unset) for the automatic RAM-tiered size. To force the model's full training context, pass the explicit number (e.g. BONSAI_CTX=262144) — only recommended on machines with plenty of headroom, since the scripts will not silently do this for you.

With the optional 4-bit KV cache (BONSAI_KV4=1) the cache drops to roughly 18 KiB per token, about 1.8 GiB at 100K, saving roughly 4.5 GiB of KV memory. Total memory also depends on the model packing, runtime buffers, and vision projector.

Extra arguments pass straight through to llama.cpp, so ./scripts/run_llama.sh -c 8192 -p "Your prompt" also works for a one-off context override.

For earlier-model memory tables, see Bonsai 1 context and memory.


Open WebUI (Optional): the full agentic demo

Open WebUI gives you a ChatGPT-like interface on top of the local 27B: chat with images, tool calling against live tools, a server-side code interpreter (plots + market data), and a hidden-story sales database to investigate. Everything is configured automatically, no clicking through settings:

./scripts/start_openwebui.sh

setup.sh installs it for you; the script starts the backend, seeds the demo (tools, model settings, demo database), and opens http://localhost:9090. Backends, what to try, and customizing: OPENWEBUI.md.


Building from Source

If you prefer to build llama.cpp from source instead of using pre-built binaries:

Mac (Apple Silicon — Metal)

./scripts/build_mac.sh

Clones PrismML-Eng/llama.cpp, builds with Metal, outputs to bin/mac/.

Mac (Intel — CPU only)

./scripts/build_mac.sh

The script auto-detects Intel vs Apple Silicon. On Intel Macs, it builds with -DGGML_METAL=OFF (CPU only). MLX is also skipped automatically since it requires Apple Silicon.

Linux (CPU only)

./scripts/build_cpu_linux.sh

Builds a CPU-only binary with no GPU dependencies. Works on both x64 and arm64. Outputs to bin/cpu/.

Linux (CUDA)

./scripts/build_cuda_linux.sh

Auto-detects CUDA version. Pass --cuda-path /usr/local/cuda-12.8 to use a specific toolkit.

Linux (Vulkan)

# Install Vulkan SDK first (e.g. sudo apt install libvulkan-dev glslc)
git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_VULKAN=ON
cmake --build build -j$(nproc)
# Binaries in build/bin/

Linux (ROCm / AMD GPU)

# Requires ROCm toolkit (hipcc)
git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIP=ON
cmake --build build -j$(nproc)
# Binaries in build/bin/

Windows (CUDA)

.\scripts\build_cuda_windows.ps1

Auto-detects CUDA toolkit. Pass -CudaPath "C:\path\to\cuda" to use a specific version. Requires Visual Studio Build Tools (or full Visual Studio) and CUDA toolkit.

Windows (CPU only)

git clone -b prism https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release
# Binaries in build\bin\Release\

Requires Visual Studio Build Tools or full Visual Studio with C++ workload.


llama.cpp Pre-built Binary Downloads

All binaries are available from the pinned GitHub Release, also used by both setup scripts. The latest release may still be missing platform binaries while builds finish.

Platform
macOS Apple Silicon (arm64)
macOS Apple Silicon (KleidiAI)
macOS Intel (x64)
Linux x64 (CPU)
Linux arm64 (CPU)
Linux x64 (CUDA 12.4)
Linux x64 (CUDA 12.8)
Linux x64 (Vulkan)
Linux arm64 (Vulkan)
Linux x64 (ROCm 7.2)
Windows x64 (CPU)
Windows arm64 (CPU)
Windows x64 (CUDA 12.4)
Windows x64 (Vulkan)
Windows x64 (HIP/ROCm)
iOS (XCFramework)

Folder Structure

After setup, the directory looks like this:

Bonsai-demo/
├── README.md                      # Bonsai 2 setup and usage
├── Bonsai1_README.md               # Earlier binary and ternary models
├── FAQ.md                          # Troubleshooting
├── TOOLS.md                        # Tool calling & MCP guide
├── OPENWEBUI.md                    # Open WebUI agentic demo guide
├── VISION.md                       # Image input: costs, caps, OCR tips
├── SPECULATIVE.md                  # Speculative decoding (experimental)
├── KV-CACHE.md                     # 4-bit KV cache (experimental)
├── AGENTS.md                       # Agent guide (hardware tuning knobs)
├── setup.sh                        # macOS/Linux setup
├── setup.ps1                       # Windows setup
├── pyproject.toml                  # Python dependencies
├── scripts/
│   ├── common.sh                   # Shared helpers + BONSAI_MODEL
│   ├── download_models.sh          # HuggingFace download
│   ├── download_binaries.sh        # GitHub release download
│   ├── run_llama.sh                # llama.cpp (auto-detects Mac/Linux)
│   ├── run_llama.ps1               # llama.cpp (Windows PowerShell)
│   ├── run_mlx.sh                  # MLX inference
│   ├── mlx_generate.py             # MLX Python script
│   ├── start_llama_server.sh       # llama.cpp server (port 8080)
│   ├── start_llama_server.ps1      # llama.cpp server (Windows PowerShell)
│   ├── start_mlx_server.sh         # MLX server (port 8081)
│   ├── start_openwebui.sh          # Open WebUI + auto-starts backends
│   ├── openwebui/                  # Open WebUI demo tools + seeding
│   ├── build_mac.sh                # Build llama.cpp for Mac
│   ├── build_cpu_linux.sh          # Build llama.cpp for Linux (CPU only)
│   ├── build_cuda_linux.sh         # Build llama.cpp for Linux CUDA
│   └── build_cuda_windows.ps1      # Build llama.cpp for Windows CUDA
├── models/                         # ← downloaded by setup
│   ├── bonsai2-gguf/
│   │   └── 27B/                    # Bonsai 2 GGUF + vision projector
│   └── Ternary-Bonsai-2-27B-mlx-2bit/ # Bonsai 2 MLX (Apple Silicon)
├── bin/                            # ← downloaded or built by setup
│   ├── mac/                        # macOS binaries (Metal or CPU)
│   ├── cuda/                       # CUDA binaries (Linux/Windows)
│   ├── cpu/                        # CPU-only binaries (Linux/Windows)
│   ├── vulkan/                     # Vulkan binaries
│   ├── rocm/                       # ROCm binaries (AMD Linux)
│   └── hip/                        # HIP binaries (AMD Windows)
├── mlx/                            # ← cloned by setup (macOS)
├── .venv/                          # ← created by setup
└── .venv-vlm/                      # ← native Bonsai 2 MLX environment

Items marked with ← are created at setup time and excluded from git.


FAQ and troubleshooting

For memory issues, GPU detection, Metal errors, and CUDA build troubleshooting, see FAQ.md.

View on GitHub

Recent activity

commits and pull requests

Code frequency

additions and deletions
+30K-30KWeek of 2026-03-22: +208 linesWeek of 2026-03-22: -0 linesWeek of 2026-03-29: +9,361 linesWeek of 2026-03-29: -96 linesWeek of 2026-04-05: +225 linesWeek of 2026-04-05: -102 linesWeek of 2026-04-12: +4,030 linesWeek of 2026-04-12: -1,489 linesWeek of 2026-04-19: +782 linesWeek of 2026-04-19: -281 linesWeek of 2026-04-26: +0 linesWeek of 2026-04-26: -0 linesWeek of 2026-05-03: +1 linesWeek of 2026-05-03: -9 linesWeek of 2026-05-10: +0 linesWeek of 2026-05-10: -0 linesWeek of 2026-05-17: +0 linesWeek of 2026-05-17: -0 linesWeek of 2026-05-24: +0 linesWeek of 2026-05-24: -0 linesWeek of 2026-05-31: +0 linesWeek of 2026-05-31: -10 linesWeek of 2026-06-07: +0 linesWeek of 2026-06-07: -0 linesWeek of 2026-06-14: +0 linesWeek of 2026-06-14: -0 linesWeek of 2026-06-21: +0 linesWeek of 2026-06-21: -0 linesWeek of 2026-06-28: +0 linesWeek of 2026-06-28: -0 linesWeek of 2026-07-05: +0 linesWeek of 2026-07-05: -0 linesWeek of 2026-07-12: +5,308 linesWeek of 2026-07-12: -1,643 linesWeek of 2026-07-19: +623 linesWeek of 2026-07-19: -184 linesWeek of 2026-07-26: +402 linesWeek of 2026-07-26: -0 linesWeek of 2026-08-02: +68 linesWeek of 2026-08-02: -16 linesWeek of 2026-08-09: +0 linesWeek of 2026-08-09: -0 linesWeek of 2026-08-16: +0 linesWeek of 2026-08-16: -0 linesWeek of 2026-08-23: +1,030 linesWeek of 2026-08-23: -248 linesWeek of 2026-08-30: +160 linesWeek of 2026-08-30: -0 linesWeek of 2026-09-06: +4 linesWeek of 2026-09-06: -4 linesWeek of 2026-09-13: +823 linesWeek of 2026-09-13: -177 linesWeek of 2026-09-20: +30,036 linesWeek of 2026-09-20: -15,482 linesWeek of 2026-09-27: +359 linesWeek of 2026-09-27: -103 linesWeek of 2026-10-04: +594 linesWeek of 2026-10-04: -23 linesWeek of 2026-10-11: +753 linesWeek of 2026-10-11: -184 linesWeek of 2026-10-18: +344 linesWeek of 2026-10-18: -3 linesWeek of 2026-10-25: +529 linesWeek of 2026-10-25: -1,857 linesWeek of 2026-11-01: +267 linesWeek of 2026-11-01: -33 linesWeek of 2026-11-08: +0 linesWeek of 2026-11-08: -0 linesWeek of 2026-11-15: +0 linesWeek of 2026-11-15: -0 linesMar 22, 2026Nov 15, 2026
+55.9K lines added, -21.9K removed over the last year.

Commits per week

last 52 weeks
450Week of 2025-11-23: 0 commitsWeek of 2025-11-30: 0 commitsWeek of 2025-12-07: 0 commitsWeek of 2025-12-14: 0 commitsWeek of 2025-12-21: 0 commitsWeek of 2025-12-28: 0 commitsWeek of 2026-01-04: 0 commitsWeek of 2026-01-11: 0 commitsWeek of 2026-01-18: 0 commitsWeek of 2026-01-25: 0 commitsWeek of 2026-02-01: 0 commitsWeek of 2026-02-08: 0 commitsWeek of 2026-02-15: 0 commitsWeek of 2026-02-22: 0 commitsWeek of 2026-03-01: 0 commitsWeek of 2026-03-08: 0 commitsWeek of 2026-03-15: 0 commitsWeek of 2026-03-22: 1 commitsWeek of 2026-03-29: 6 commitsWeek of 2026-04-05: 10 commitsWeek of 2026-04-12: 45 commitsWeek of 2026-04-19: 8 commitsWeek of 2026-04-26: 0 commitsWeek of 2026-05-03: 1 commitsWeek of 2026-05-10: 0 commitsWeek of 2026-05-17: 0 commitsWeek of 2026-05-24: 0 commitsWeek of 2026-05-31: 1 commitsWeek of 2026-06-07: 0 commitsWeek of 2026-06-14: 0 commitsWeek of 2026-06-21: 0 commitsWeek of 2026-06-28: 0 commitsWeek of 2026-07-05: 0 commitsWeek of 2026-07-12: 16 commitsWeek of 2026-07-19: 15 commitsWeek of 2026-07-26: 2 commitsWeek of 2026-08-02: 2 commitsWeek of 2026-08-09: 0 commitsWeek of 2026-08-16: 0 commitsWeek of 2026-08-23: 12 commitsWeek of 2026-08-30: 1 commitsWeek of 2026-09-06: 1 commitsWeek of 2026-09-13: 5 commitsWeek of 2026-09-20: 45 commitsWeek of 2026-09-27: 3 commitsWeek of 2026-10-04: 5 commitsWeek of 2026-10-11: 4 commitsWeek of 2026-10-18: 3 commitsWeek of 2026-10-25: 8 commitsWeek of 2026-11-01: 4 commitsWeek of 2026-11-08: 0 commitsWeek of 2026-11-15: 0 commitsNov 23, 2025Nov 15, 2026
198 commits in the last 52 weeks.

When work happens

weekday and hour
SunMonTueWedThuFriSat036912151821Sun 0:00 — 0 commitsSun 1:00 — 0 commitsSun 2:00 — 1 commitsSun 3:00 — 0 commitsSun 4:00 — 0 commitsSun 5:00 — 0 commitsSun 6:00 — 1 commitsSun 7:00 — 0 commitsSun 8:00 — 1 commitsSun 9:00 — 0 commitsSun 10:00 — 0 commitsSun 11:00 — 1 commitsSun 12:00 — 0 commitsSun 13:00 — 0 commitsSun 14:00 — 3 commitsSun 15:00 — 0 commitsSun 16:00 — 0 commitsSun 17:00 — 0 commitsSun 18:00 — 3 commitsSun 19:00 — 1 commitsSun 20:00 — 0 commitsSun 21:00 — 2 commitsSun 22:00 — 0 commitsSun 23:00 — 3 commitsMon 0:00 — 0 commitsMon 1:00 — 0 commitsMon 2:00 — 1 commitsMon 3:00 — 1 commitsMon 4:00 — 1 commitsMon 5:00 — 1 commitsMon 6:00 — 1 commitsMon 7:00 — 2 commitsMon 8:00 — 2 commitsMon 9:00 — 1 commitsMon 10:00 — 3 commitsMon 11:00 — 0 commitsMon 12:00 — 1 commitsMon 13:00 — 1 commitsMon 14:00 — 1 commitsMon 15:00 — 1 commitsMon 16:00 — 2 commitsMon 17:00 — 2 commitsMon 18:00 — 3 commitsMon 19:00 — 5 commitsMon 20:00 — 9 commitsMon 21:00 — 3 commitsMon 22:00 — 1 commitsMon 23:00 — 6 commitsTue 0:00 — 0 commitsTue 1:00 — 0 commitsTue 2:00 — 0 commitsTue 3:00 — 0 commitsTue 4:00 — 1 commitsTue 5:00 — 2 commitsTue 6:00 — 0 commitsTue 7:00 — 0 commitsTue 8:00 — 0 commitsTue 9:00 — 1 commitsTue 10:00 — 0 commitsTue 11:00 — 1 commitsTue 12:00 — 1 commitsTue 13:00 — 1 commitsTue 14:00 — 1 commitsTue 15:00 — 2 commitsTue 16:00 — 1 commitsTue 17:00 — 1 commitsTue 18:00 — 0 commitsTue 19:00 — 0 commitsTue 20:00 — 1 commitsTue 21:00 — 0 commitsTue 22:00 — 3 commitsTue 23:00 — 0 commitsWed 0:00 — 1 commitsWed 1:00 — 3 commitsWed 2:00 — 1 commitsWed 3:00 — 1 commitsWed 4:00 — 1 commitsWed 5:00 — 0 commitsWed 6:00 — 0 commitsWed 7:00 — 0 commitsWed 8:00 — 10 commitsWed 9:00 — 4 commitsWed 10:00 — 2 commitsWed 11:00 — 2 commitsWed 12:00 — 3 commitsWed 13:00 — 1 commitsWed 14:00 — 2 commitsWed 15:00 — 2 commitsWed 16:00 — 0 commitsWed 17:00 — 3 commitsWed 18:00 — 1 commitsWed 19:00 — 0 commitsWed 20:00 — 4 commitsWed 21:00 — 3 commitsWed 22:00 — 2 commitsWed 23:00 — 1 commitsThu 0:00 — 3 commitsThu 1:00 — 2 commitsThu 2:00 — 1 commitsThu 3:00 — 0 commitsThu 4:00 — 1 commitsThu 5:00 — 1 commitsThu 6:00 — 1 commitsThu 7:00 — 0 commitsThu 8:00 — 0 commitsThu 9:00 — 1 commitsThu 10:00 — 1 commitsThu 11:00 — 5 commitsThu 12:00 — 1 commitsThu 13:00 — 1 commitsThu 14:00 — 2 commitsThu 15:00 — 2 commitsThu 16:00 — 2 commitsThu 17:00 — 4 commitsThu 18:00 — 3 commitsThu 19:00 — 2 commitsThu 20:00 — 0 commitsThu 21:00 — 0 commitsThu 22:00 — 1 commitsThu 23:00 — 1 commitsFri 0:00 — 2 commitsFri 1:00 — 0 commitsFri 2:00 — 0 commitsFri 3:00 — 1 commitsFri 4:00 — 0 commitsFri 5:00 — 0 commitsFri 6:00 — 0 commitsFri 7:00 — 1 commitsFri 8:00 — 0 commitsFri 9:00 — 1 commitsFri 10:00 — 1 commitsFri 11:00 — 2 commitsFri 12:00 — 2 commitsFri 13:00 — 6 commitsFri 14:00 — 0 commitsFri 15:00 — 0 commitsFri 16:00 — 0 commitsFri 17:00 — 2 commitsFri 18:00 — 0 commitsFri 19:00 — 1 commitsFri 20:00 — 1 commitsFri 21:00 — 3 commitsFri 22:00 — 0 commitsFri 23:00 — 2 commitsSat 0:00 — 0 commitsSat 1:00 — 0 commitsSat 2:00 — 0 commitsSat 3:00 — 0 commitsSat 4:00 — 0 commitsSat 5:00 — 0 commitsSat 6:00 — 0 commitsSat 7:00 — 1 commitsSat 8:00 — 0 commitsSat 9:00 — 0 commitsSat 10:00 — 0 commitsSat 11:00 — 1 commitsSat 12:00 — 1 commitsSat 13:00 — 2 commitsSat 14:00 — 0 commitsSat 15:00 — 2 commitsSat 16:00 — 0 commitsSat 17:00 — 3 commitsSat 18:00 — 1 commitsSat 19:00 — 1 commitsSat 20:00 — 0 commitsSat 21:00 — 0 commitsSat 22:00 — 0 commitsSat 23:00 — 0 commits
Commit volume by weekday and hour (UTC). Larger dots mean more commits.
DateListRankStars gained
Sep 30, 2026weekly#13+243
Sep 29, 2026weekly#13+243
Aug 15, 2026monthly#15+1,323
Aug 14, 2026monthly#15+1,323
Jul 15, 2026daily#15+7
  • mattpocock/skills

    Skills for Real Engineers. Straight from my .agents directory.

    329K stars · Shell

  • obra/superpowers

    An agentic skills framework & software development methodology that works.

    322.9K stars · Shell

  • affaan-m/ECC

    The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

    307.9K stars · JavaScript

  • NousResearch/hermes-agent

    The agent that grows with you

    265K stars · Python

  • firecrawl/firecrawl

    Knowledge infrastructure to make your agents smarter. Search, scrape and access 550+ data providers through one API.

    205.3K stars · TypeScript

  • DietrichGebert/ponytail

    Makes your AI agent think like the laziest senior dev in the room. The best code is the code you never wrote.

    195.7K stars · JavaScript