TL;DR: Mesh LLM is an Apache 2.0 Rust binary that pools GPUs across your machines into a single OpenAI-compatible API using iroh peer-to-peer networking — no central server, no cloud middleman. Setup on two nodes takes about 20 minutes and one join token. It is the best option today for mixed-GPU Linux home labs, but it is experimental distributed-systems software, not a production inference stack.

	Mesh LLM	exo	Single-node Ollama
Best for	Mixed NVIDIA/AMD Linux + Windows home labs pooling capacity	Apple Silicon clusters (MLX)	One machine, zero fuss
License	Apache 2.0	Apache 2.0	MIT
GPU backends	CUDA, ROCm, Vulkan, Metal, CPU	macOS GPU (MLX); Linux is CPU-only without the community CUDA fork	CUDA, ROCm, Metal
The catch	Experimental; no SLA, no health monitoring to lean on	NVIDIA-on-Linux needs the unofficial exo-cuda fork	Can’t pool VRAM across machines at all

Honest take: If your spare hardware is a couple of NVIDIA or AMD boxes on a LAN, Mesh LLM is the tool exo never became — install it, pool them, and treat the result as a capable lab toy. If your hardware is a stack of Macs, stay with exo.

Every self-hoster eventually hits the same wall: the model you want needs 40GB of VRAM and your biggest card has 24GB — but there is another 16GB card sitting idle in the machine across the room. Mesh LLM, which hit the front page of Hacker News in July 2026 (179 points), exists for exactly that situation. It distributes model compute across a mesh of iroh endpoints and serves the result from one local URL.

This guide covers the license check, a two-node home lab setup, how the routing actually behaves, and where Mesh LLM stands against exo and LocalAI.

What Mesh LLM is

Mesh LLM is a single Rust binary (the install is roughly 18MB) that turns every machine it runs on into a node in a peer-to-peer inference mesh. Each node exposes the same OpenAI-compatible API at http://localhost:9337/v1, with the standard /v1/models and /v1/chat/completions endpoints. Point Open WebUI, Continue.dev, or any OpenAI-compatible client at that URL and requests get answered by whichever node in your mesh can serve them.

The networking layer is iroh, the dual MIT/Apache 2.0 Rust library (12.2k GitHub stars) that adds QUIC transport and NAT traversal keyed to cryptographic node IDs instead of IP addresses. That is what lets two machines behind different home routers find each other without port forwarding or a relay server you have to run yourself.

The model catalog is broad for a young project: the README documents 72 priority model family rows (89 certified in the full parity inventory) spanning Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, and RWKV. You can also load your own weights via HuggingFace references (hf://meshllm/<repo>@<rev>) or a plain local GGUF path.

At the time of writing, the latest stable release is v0.75.1 (August 10, 2026), with a v0.76 release candidate already testing a durable KV prefix cache that survives node restarts — a genuinely useful feature for agent workloads that reuse long system prompts.

License check: clean Apache 2.0

The GitHub repo at Mesh-LLM/mesh-llm (~3.3k stars) shows Apache 2.0 in the sidebar. That means commercial use, modification, and redistribution are all fine, with patent grant included. The iroh dependency is dual-licensed MIT/Apache 2.0. No open-core split, no revenue caps, no “community license” fine print. For a project this new, a boring license is the best possible news.

Two-node setup guide

You need two machines with something to contribute — a GPU, or failing that, RAM and CPU cores. The test case here assumes a Linux desktop with an RTX 3090 and a second box with an RTX 4060 Ti 16GB. Builds exist for macOS, Linux (CPU, ARM64, CUDA, CUDA Blackwell, ROCm, Vulkan), and Windows (CPU, CUDA, ROCm, Vulkan). One documented trap: CUDA 13+ on Windows requires running under WSL2.

Step 1 — install on both machines:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash
mesh-llm setup

On Apple Silicon you can use brew install Mesh-LLM/tap/mesh-llm instead. Windows uses an equivalent PowerShell one-liner (install.ps1).

Step 2 — start a private mesh on node A:

mesh-llm serve --model Qwen3-8B-Q4_K_M

This starts a node serving a model and prints a join token. That token is the entire trust model of your private mesh — treat it like an SSH key, and share it only with machines you control.

Step 3 — join from node B:

mesh-llm serve --join <token>

Node B contributes its own GPU and can load different (or additional) models. A machine with nothing to contribute can still consume the mesh with mesh-llm client --auto, and servers should add --headless.

Step 4 — verify and connect a client:

curl http://localhost:9337/v1/models

You should see the models available across the whole mesh, not just the local node. In Open WebUI: Settings → Connections → add an OpenAI-compatible endpoint at http://localhost:9337/v1. Every node exposes the same API, so each machine talks to its own localhost — no single point of failure.

Two flags worth knowing before you deviate from the happy path: --local-model-only pins a node to serving its own weights (no mesh fetch), and --auto joins the public mesh — shared compute with strangers. Keep private workloads on private tokens.

How routing actually behaves

A request hitting any node’s /v1/chat/completions is resolved in three tiers:

1. Single-machine fit first. If one node can host the whole model, it serves the request locally. On your own machine that means no network hop at all; on a LAN peer it adds roughly 1–5ms of round-trip — noise compared to token generation time.
2. Mesh routing by model name. Every node exposes the same API, so requests are routed by the model field to a peer that has that model loaded. This is the everyday win: your laptop asks for a 32B model and the desktop with the 3090 answers.
3. Stage splits for oversized models. Models too large for any single box are loaded as layer stages across several nodes and run as a pipeline. This is the headline feature and also the one to be most skeptical of: each stage boundary adds per-token latency, and activations cross the network every token. Over a LAN it is usable; over the internet, where iroh’s NAT traversal happily connects peers 100ms apart, sharding makes sense for capacity (running a model you otherwise couldn’t) — never for speed.

There is also an experimental Mixture-of-Agents mode (model: "mesh") that fans a request out across at least two distinct models in the mesh and synthesizes the answers. The README marks it experimental, and its behavior may change between releases.

Mesh LLM vs exo vs LocalAI

We covered exo’s setup and its honest limitations previously: despite the “pool all your devices” marketing, exo’s GPU acceleration is macOS/MLX — on Linux it runs on CPU unless you adopt the community exo-cuda fork. That makes the comparison straightforward:

• Mesh LLM wins on Linux and mixed hardware. First-party CUDA, ROCm, and Vulkan builds mean an NVIDIA desktop, an AMD box, and a Windows gaming rig can pool without forks. It also wins on internet-scale flexibility, since iroh handles NAT traversal natively.
• exo wins on Apple Silicon. Its MLX path is mature and fast for Mac clusters, and it auto-discovers devices on a LAN with zero configuration. A 4× Mac Mini cluster is still exo territory.
• LocalAI is a different layer. It is a single-node, multi-backend OpenAI-compatible router — see our LocalAI vs Ollama comparison. Mesh LLM adds the P2P routing layer that LocalAI doesn’t have; they solve different problems and could in principle coexist.

Against renting cloud compute, the trade is privacy versus convenience: a RunPod pod gets you a big GPU by the hour with no mesh to maintain, but your prompts leave your hardware. A private Mesh LLM mesh among machines you (or friends you trust) own keeps inference entirely on hardware you control — the same reason we keep recommending explicit local tags over Ollama’s :cloud routing.

When NOT to use Mesh LLM

• You have one machine. Ollama or llama.cpp is simpler, better documented, and has a vastly larger ecosystem. A mesh of one node is overhead with no benefit.
• You need reliability. If peers go offline you fall back to whatever runs locally. There is no SLA, no built-in health monitoring you’d want to depend on, and the project is explicitly experimental — version 0.75, not 1.0.
• You were planning to join the public mesh with private data. --auto means strangers’ hardware may serve your prompts. For anything sensitive, private tokens only.
• Your cluster is all Macs. exo’s MLX path will outperform, and its zero-config LAN discovery is more pleasant.

For a deeper look at speccing the individual nodes — which GPU per box, RAM floors for CPU-offload peers — see the hardware-focused companion at runaihome.com.

FAQ

Does Mesh LLM work over the internet, or only on a LAN? Both. iroh’s QUIC + NAT traversal connects peers across networks without port forwarding. Just set expectations: WAN peers add 50–500ms per hop, which is fine for routing whole requests to a remote GPU, and painful for layer-sharded pipelines where traffic crosses the link every token.

Is my data private on a Mesh LLM mesh? On a private mesh (join tokens you issued), inference runs only on member machines, so privacy reduces to whether you trust those machines. On the public mesh (--auto), unknown peers can process your prompts — treat it like any third-party API, minus the contract.

Can I keep using Ollama alongside Mesh LLM? Yes. They bind different ports (Ollama on 11434, Mesh LLM on 9337) and can share GGUF files on disk. A sensible migration is to keep Ollama for daily single-node work and stand up Mesh LLM only when a job needs pooled capacity.

Sources

• Mesh-LLM/mesh-llm on GitHub — README, Apache 2.0 license, install and serve commands, model parity inventory (accessed Aug 19, 2026)
• Mesh LLM releases — v0.75.1 (Aug 10, 2026), v0.76.0-rc3 durable KV prefix cache
• n0-computer/iroh on GitHub — dual MIT/Apache 2.0, QUIC + NAT traversal library
• Mesh LLM: distributed AI computing on iroh — announcement post (Hacker News, 179 points, July 11, 2026)

Recommended Gear

• RTX 3090 — the used-market 24GB workhorse for a primary mesh node
• RTX 4060 Ti 16GB — a low-power second node that adds 16GB to the pool

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe