Mesh-LLM/mesh-llmPublic

Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.

Install the latest release executable:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

On Windows, use PowerShell:

irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

Install the Apple Silicon Homebrew formula with brew install Mesh-LLM/tap/mesh-llm. Versioned formulas, Ubuntu and Arch packages, checksums, SBOMs, and OCI images are produced by the public Mesh-LLM/mesh-packaging repository. See the platform install guides for the supported package matrix and install commands.

Finish setup:

mesh-llm setup

On Windows PowerShell, use mesh-llm.exe setup. (For native Windows notes, and for the optional WSL2 setup and multi-node LAN clustering, see the Windows & WSL2 Troubleshooting Guide.)

To remove an executable install later, preview the cleanup first:

mesh-llm uninstall --dry-run
mesh-llm uninstall --yes

Uninstall preserves ~/.mesh-llm configuration and identity data unless you explicitly pass --purge-config.

Join the public mesh and start serving:

mesh-llm serve --auto

That command chooses a backend flavor, downloads a suitable model if needed, joins the best discovered public mesh, starts the local API on port 9337, and starts the web console on port 3131.

Check available models:

curl -s http://localhost:9337/v1/models | jq '.data[].id'

Send an OpenAI-compatible request:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

For server deployments, add --headless to hide the web UI while keeping the management API on the --console port:

mesh-llm serve --auto --headless

• Single-machine fit first. If one node can host the full model, it serves the model locally without stage traffic.
• Mesh routing. Every node exposes the same /v1 API. Requests are routed by the model field to the peer that can serve that model.
• Encrypted peer transport. QUIC end-to-end encrypts traffic between Mesh nodes, including inference requests, responses, and split-model activations. Iroh relays forward encrypted packets without reading their payload.
• Owner-control plane. Operator config and inventory actions use an additive mesh-llm-control/1 lane with explicit endpoint bootstrap, while public mesh join, gossip, routing, and inference stay on the public mesh plane for mixed-version compatibility.
• Skippy stage splits. Large dense models can load as package-backed layer stages. The coordinator plans contiguous layer ranges, starts downstream stages first, waits for readiness, then publishes the stage-0 route.
• Layer packages. Package repositories contain model-package.json plus GGUF fragments so peers fetch only the pieces needed for their assigned stage.
• Public discovery. Published meshes advertise through Nostr discovery; private meshes stay invite-token based.

For a deeper operator guide, see docs/USAGE.md. For every CLI command and switch, see docs/CLI.md.

Debugging and development endpoint. --local-model-only wires the OpenAI frontend directly to one local Skippy instance and bypasses mesh networking entirely. The supported path for OpenAI clients in production — including tool calls — is the regular mesh inference port, which includes the host-runtime normalizer. Use --local-model-only when you want a lightweight single-process setup without joining a mesh, not as a drop-in for the full serving path.

Use the direct topology when a process should expose one complete local model through the OpenAI API without becoming a mesh node:

mesh-llm serve \
  --local-model-only \
  --gguf /models/model.gguf \
  --port 9337

Use --gguf <absolute-path> to serve a specific GGUF file directly. --model <absolute-path> also works. Note that --model <catalog-name> does not work here — catalog and HF-ref resolution is a mesh function and is not available in the direct topology; the resulting error is not obvious. See the path requirements below.

This mode starts the OpenAI frontend and one local Skippy model runtime. It does not start QUIC, discovery, peer maintenance, split planning, plugins, release lookup, the web console, or the management API. Add --listen-all only when the OpenAI endpoint must bind beyond loopback. Startup fails if the complete model does not fit within detected local capacity (or --max-vram); it never falls back to distributed serving.

Values passed to --model, --gguf, and --mmproj must be absolute paths and must not be symlinks. If you downloaded a model through Hugging Face, resolve the cache symlink before passing the path: realpath <path-from-hf-cache>.

⚠️ Experimental. The MoA gateway is new in this release. Behavior, routing heuristics, error shapes, and tuning knobs may change between versions while we tune it. Treat model: "mesh" as a preview feature rather than a stable production path; use a specific model id when you need stable semantics.

Send a request with "model": "mesh" and the proxy fans it out to every model available in the mesh in parallel, arbitrates their responses with deterministic logic, and returns one OpenAI-compatible reply. The arbiter runs in code (not as another model call) and only escalates to a reducer LLM on genuine conflict. Tool calls flow through the full pipeline.

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mesh","messages":[{"role":"user","content":"What is the capital of Japan?"}]}'

Requires at least two distinct models in the mesh. See docs/design/MOA_GATEWAY.md for the architecture, arbitration rules, and tuning knobs.

Mesh LLM's Skippy runtime tracks llama.cpp family parity with reviewed GGUF representatives. The current reviewed support set covers 72 P0/P1 family rows, with 89 certified rows in the full parity inventory, including Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, RWKV, and many others.

Split multimodal serving is certified for Qwen2-VL, Qwen3-VL, Qwen3-VL-MoE, HunyuanOCR/Hunyuan-VL, and DeepSeek-OCR using real GGUF plus projector fixtures. DeepSeek3 and EXAONE-MoE use package-backed stages because the full GGUFs are too large for the cheap local baseline.

See docs/skippy/FAMILY_STATUS.md for the full artifact, split, wire dtype, cache policy, and exception matrix. See docs/skippy/LLAMA_PARITY.md for the remaining llama.cpp parity queue.

Tagged releases publish macOS bundles plus Linux CPU, Linux ARM64 CPU, Linux ARM64 CUDA, Linux CUDA, Linux CUDA Blackwell, Linux ROCm, Linux Vulkan, Windows CPU, Windows CUDA, Windows ROCm, and Windows Vulkan bundles. Metal is macOS-only. Every flavor is composed from the same backend-neutral host for its OS/architecture plus one versioned native runtime. The Linux ARM64 CPU artifact is mesh-llm-aarch64-unknown-linux-gnu.tar.gz; the Linux ARM64 CUDA artifact is mesh-llm-aarch64-unknown-linux-gnu-cuda.tar.gz. In install and release contexts, arm64 and aarch64 mean the same 64-bit ARM target. Portable archives work offline: the host discovers the adjacent native-runtimes/<runtime-id> tree before consulting the user cache.

Build from source with just:

git clone https://github.com/Mesh-LLM/mesh-llm
cd mesh-llm
just build

Source builds require just, cmake, Rust, sccache, the platform fast linker (mold on Linux and lld on macOS/Windows), and Node.js 24 + npm. Cargo uses these defaults from the checked-in configuration, including plain cargo build. See CONTRIBUTING.md for installation commands and linker fallback behavior. To exercise the release boundary locally, build the neutral host and one runtime:

just release-host-build
just release-runtime-build metal # or cpu, cuda, rocm, vulkan
MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" \
MESH_LLM_NATIVE_RUNTIME_CACHE_DIR="$(mktemp -d)" \
  ./target/release/mesh-llm runtime list

CUDA runtimes need nvcc, ROCm runtimes need ROCm/HIP, and Vulkan runtimes need Vulkan development files plus glslc. See docs/design/NATIVE_RUNTIMES.md for the manifest, discovery, and compatibility contract.

The shipped mesh-llm executable uses embedded release attestation for provenance and admission hardening only. It does not apply to SDK, XCFramework, or other native artifacts, and it is not a runtime integrity proof. Verify a stamped packaged executable with cargo run -p xtask -- release-attestation inspect --binary <path-to-packaged-mesh-llm> --public-key-file <release-signing-public-key.json>. A packaged release binary reports valid, an unstamped local or dev build reports missing, and a binary that changed after packaging reports invalid. Bare inspect --binary ... is only enough to classify an unstamped binary as missing; stamped binaries require --public-key-file and otherwise report invalid with an explicit error. Post-download mutation can flip a stamped binary to invalid, but default startup still allows it.

Native Windows CUDA works, including on CUDA 13.x drivers: GPU detection (mesh-llm gpus) and full-speed CUDA inference have been verified on driver 610.74 (CUDA UMD 13.3) with an RTX 4070 Ti. The earlier advice to switch to WSL2 when mesh-llm reported 0 GPUs on CUDA 13 drivers predates the cuda12-runtime compatibility fix (#1127) and no longer applies to current releases.

As of v0.76.0-rc8, three distribution/loading bugs still block the out-of-the-box native path. Until the fixes ship, this sequence works end to end:

1. Install the prerelease — the stable v0.75.1 Windows bundles fail install.ps1 verification (#1510):

   irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 -OutFile install.ps1
   .\install.ps1 -PreRelease

2. Install the CUDA runtime from the product bundle — mesh-llm runtime install cuda finds no windows/x86_64 runtimes in the release manifest (#1511). Download mesh-llm-x86_64-pc-windows-msvc-cuda.zip for your installed version from the releases page, extract it, then:

   mesh-llm runtime install --bundle-dir "<extracted>\mesh-bundle" cuda

3. Put the runtime's lib directory on PATH before serving — runtime DLLs currently fail to load with LoadLibraryExW error 126 (#1512):

   $env:PATH = "$env:LOCALAPPDATA\mesh-llm\native-runtimes\<version>\meshllm-native-runtime-windows-x86_64-cuda12\lib;" + $env:PATH
   mesh-llm serve --local-model-only --model "C:\path\to\model.gguf"

Good to know on native Windows:

• --local-model-only requires an absolute path to a local .gguf file; catalog and Hugging Face refs are rejected in this mode.
• Models download to %LOCALAPPDATA%\huggingface\hub (the Rust hf-hub convention) — a separate cache from the Python tools' ~\.cache\huggingface.
• mesh-llm.exe is not code-signed yet, so SmartScreen may prompt when launching it manually.

WSL2 remains a solid alternative if you prefer the Linux CUDA runtimes or want the multi-node LAN setups described below.

Inside your Ubuntu WSL2 terminal, install cuda-toolkit-13-0 to supply libcudart.so.13:

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda-repo-wsl-ubuntu-13-0-local_13.0.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-13-0-local_13.0.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-13-0-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update && sudo apt-get -y install cuda-toolkit-13-0

echo 'export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

If you use WSL2 networkingMode=mirrored in %UserProfile%\.wslconfig, Windows 11 manages a separate Hyper-V VM Firewall that defaults to Block for inbound network traffic when third-party security software (e.g. Norton, McAfee) is present.

Create or edit C:\Users\<username>\.wslconfig on the Windows host:

notepad $env:USERPROFILE\.wslconfig

[wsl2]
networkingMode=mirrored
autoProxy=true

Restart WSL in PowerShell:

wsl --shutdown

To allow incoming LAN connections to the Web UI (3131) and P2P QUIC mesh transport (9337), run PowerShell as Administrator on the Windows host:

# 1. Allow Inbound Traffic through the WSL Hyper-V VM Firewall Container
$vmCreatorId = '{40E0AC32-46A5-438A-A0B2-2B479E8F2E90}'
Set-NetFirewallHyperVVMSetting -Name $vmCreatorId -DefaultInboundAction Allow

# 2. Allow MeshLLM Ports in Windows Defender Firewall
New-NetFirewallRule -DisplayName "MeshLLM TCP In" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3131,9337 -Profile Any
New-NetFirewallRule -DisplayName "MeshLLM UDP In" -Direction Inbound -Action Allow -Protocol UDP -LocalPort 9337,5353 -Profile Any

To ensure worker nodes load GGUF model shards directly off local NVMe/SSD storage without streaming tens of gigabytes over the network, ensure the --gguf file path string is identical across all nodes (or use symlinks/bind mounts):

# Example: Mount or symlink model path on worker nodes
sudo mkdir -p /mnt/d/models/
sudo mount --bind /path/to/local/fast/nvme/ /mnt/d/models/

# Launch Master and Worker with matching path strings
mesh-llm --llama-flavor cuda serve \
  --console 3131 \
  --gguf "/mnt/d/models/model.gguf" \
  --mesh-name "MainMesh" \
  --listen-all \
  --auto

Trusted main Linux jobs may use Depot's managed GitHub Actions runners through the checked-in runner policy. Pull requests remain on GitHub-hosted runners until the cache isolation and protected-workflow gates in ci/DEPOT_MIGRATION.md are proven. Hardware-qualified GPU tests remain on dedicated runners.

Mesh LLM is experimental distributed-systems software. When you report bugs, include the command you ran, platform/backend flavor, /api/status output if available, and whether the node was private, published, or joined with --auto.

The Mesh LLM project thanks spiral.xyz for their support.