TL;DR: vLLM’s tensor parallelism turns two 24GB cards into one 48GB inference server, and on current builds (tested against v0.26.0, released July 25, 2026) the setup is finally reasonable for home labs. The trade-off is configuration complexity — three flags decide whether you get a stable server or an OOM loop. Worth it if you serve concurrent requests; skip it if you’re the only user.

	vLLM tensor parallel (TP=2)	Two separate Ollama instances	vLLM pipeline parallel
Best for	One large model (30B–70B quantized), concurrent users	Two smaller models, single user, zero fuss	Mismatched GPUs (e.g., 3090 + 4060 Ti)
Setup effort	Medium — flags matter, NCCL can bite	Trivial	Medium, slower than TP
The catch	Both GPUs must be identical class; VRAM headroom math is on you	Can’t run a model bigger than one card	Higher latency; only worth it when TP isn’t possible

Honest take: If you built a dual-GPU box to run 30B+ models with real concurrency, vLLM tensor parallel is the point of that hardware. If you just want two models loaded at once, keep running two Ollama instances and save yourself an afternoon.

A note on versions before anything else: several guides circulating on Reddit and dev.to this summer reference “vLLM v0.6” scheduler improvements. vLLM 0.6 shipped in September 2024 — two years ago. As of this writing the current release is v0.26.0 (July 25, 2026), preceded by v0.25.0 on July 11. Everything below was checked against the current engine arguments; the scheduler and memory flags here exist in all recent releases.

What tensor parallelism actually does

With --tensor-parallel-size 2, vLLM splits every weight matrix across both GPUs. Each layer’s computation happens on both cards simultaneously, with results synchronized over PCIe (or NVLink if you have it). The practical effect: a model that needs ~40GB of weights and KV cache runs on two 24GB cards, and both GPUs contribute compute to every single token.

This is different from what Ollama does when it splits a model across GPUs — Ollama’s llama.cpp backend assigns whole layers to each card, so GPUs take turns rather than working together. Tensor parallel keeps both cards busy at once, which is why vLLM’s throughput advantage shows up hardest under concurrent load. One widely shared June 2026 benchmark report measured a 19× throughput gap between vLLM and Ollama on identical hardware under batch load; treat the exact number with suspicion (batching setups vary wildly), but the direction is real and repeatable.

The reference build for this guide is two RTX 3090 cards — still the price-per-VRAM king in 2026, and the last consumer GeForce card with NVLink support. A dual RTX 4090 build works identically (no NVLink, but PCIe 4.0 x8/x8 is fine for TP=2). For the full hardware side — PSU sizing, slot spacing, thermals — see the multi-GPU build guide on runaihome.com.

Step 1: Install and launch with TP=2

Fresh virtual environment, then:

pip install vllm

Launch a 32B model quantized to AWQ, split across both cards:

vllm serve Qwen/Qwen3-32B-AWQ \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.85 \
  --max-model-len 16384 \
  --max-num-seqs 16

For single-node dual-GPU, that’s all the parallelism config you need — vLLM handles worker orchestration itself and no Ray cluster setup is required. If startup hangs at NCCL initialization (common on consumer boards where PCIe peer-to-peer is flaky), relaunch with NCCL_P2P_DISABLE=1 in the environment. You lose a little synchronization speed and gain a server that actually starts.

Step 2: Verify both GPUs are actually working

Don’t trust the startup logs — check utilization during inference. Send a request, then in another terminal:

watch -n 1 nvidia-smi

You want to see both cards at roughly equal memory consumption and both showing compute utilization while a request is in flight. If GPU 1 sits at 0% utilization with memory allocated, your requests are running single-GPU and something in the config regressed. Equal memory but idle compute on one card usually points at the NCCL issue above.

Step 3: The three flags that prevent OOM crashes

Multi-GPU vLLM crashes are almost never mysterious — they come from one of three defaults being wrong for 24GB cards.

--gpu-memory-utilization — vLLM pre-allocates this fraction of each GPU for weights plus KV cache. The default 0.9 works on datacenter cards with nothing else running; on a desktop where your display server holds 500MB–1GB on GPU 0, it’s a crash waiting for boot. Start at 0.85. If you run headless, push to 0.92 and stop when startup fails.

--max-model-len — context length directly sizes the KV cache reservation. Model cards advertising 128K context will happily try to reserve KV cache for it and fail. 16384 is a sane ceiling for a dual-24GB box running a 32B model; drop to 8192 if you want more concurrent sequences instead of longer ones.

--max-num-seqs — caps concurrent sequences in a batch. The default (256 in recent releases) is a datacenter number. At 16, memory spikes under load stay bounded and per-request latency stays predictable on consumer cards.

The interaction between these three is the whole game: total VRAM budget = weights + (KV cache per token × max-model-len × max-num-seqs, roughly). If you tighten one you can loosen another. Our GPTQ vs AWQ vs GGUF comparison covers picking a quant that leaves enough KV cache headroom in the first place.

Step 4: Scheduler configuration

vLLM’s scheduler decides which queued requests join the running batch each step. Two policies exist, set via --scheduling-policy:

• fcfs (default) — first come, first served. Correct for a single user or a handful of interactive users.
• priority — requests carry a priority value (lower runs earlier), with arrival time breaking ties. Useful when your box serves mixed traffic: an interactive chat UI plus a background batch job (nightly embedding runs, log summarization) that shouldn’t starve the humans.

Client-side, priority is passed per-request through the API. If everything hitting your server is the same kind of traffic, leave fcfs alone — priority scheduling only earns its keep when workloads genuinely compete. What you should not expect from either policy is an OOM fix; the scheduler works within the memory budget set by the three flags above, and no scheduling policy rescues an over-committed KV cache.

When two Ollama instances are still the right answer

Dual-GPU vLLM is the wrong tool in three common situations:

1. You’re the only user. Ollama with layer-split across both cards gives you a 70B Q4 at interactive speeds with zero configuration. Batching throughput — vLLM’s core strength — buys you nothing at concurrency of one. Our Ollama vs vLLM comparison covers this in depth.
2. You want two different models resident. One Ollama instance per GPU (set CUDA_VISIBLE_DEVICES per instance) is simpler and doesn’t tie the cards together.
3. Your GPUs are mismatched. Tensor parallel wants identical cards; splitting a matrix between a 24GB and a 16GB card wastes the difference. Pipeline parallelism (--pipeline-parallel-size 2) tolerates asymmetry but adds latency — it’s the fallback, not the goal.

If your workload only needs big-GPU power occasionally, renting beats building: an A100 80GB on RunPod runs a 70B without any parallelism gymnastics, and you pay only for hours used. For always-on serving with authentication and monitoring on top of this setup, continue with the vLLM production setup guide.

One workload where dual-GPU vLLM genuinely earns its complexity: serving a local model to AI coding tools. Editors like Cline and Continue fire multiple completion and chat requests in parallel, which is exactly the concurrent traffic tensor parallel batching is built for. If that’s your use case, the Cline local LLM privacy-first setup at aicoderscope.com covers the editor side of the pipe.

The limitations nobody puts in the quickstart

Tensor parallel on consumer hardware works, but four constraints don’t show up until you’re living with the setup:

Power draw is a dual-card problem, not a sum. Each RTX 3090 is rated at 350W TDP, and transient spikes can approach double that for milliseconds. Two cards spiking together will trip the overcurrent protection on a marginal PSU even when average draw looks fine — this is the classic “server dies only under full batch load” failure. A quality 1000W+ unit is the realistic floor for a dual-3090 inference box, and it’s why the OOM flags above aren’t the only stability knobs that matter.

One model, resident, period. vLLM pre-allocates its memory budget at boot and holds it. There’s no Ollama-style hot-swapping between models on demand — changing models means restarting the server and waiting through weight loading again. If your household of tools expects three different models at different hours, that’s an orchestration problem vLLM doesn’t solve for you.

Both cards are welded together operationally. With TP=2, a driver hiccup, thermal shutdown, or Xid error on either card takes the whole server down. Two independent Ollama instances degrade to one working GPU; a tensor-parallel pair degrades to zero.

Startup time compounds iteration cost. A 32B AWQ model loading plus CUDA graph capture takes minutes, not seconds, on a consumer platform. That’s irrelevant for an always-on server and painful if you’re restarting to tune flags — which is exactly what the first afternoon of this setup involves. Tune --gpu-memory-utilization last, after the other flags are settled, to minimize restart cycles.

FAQ

Do I need NVLink for tensor parallel on two RTX 3090s? No. TP=2 synchronization traffic fits comfortably in PCIe 4.0 x8/x8 bandwidth for inference. NVLink helps most for training and for TP across 4+ cards. If you already own the bridge, use it; don’t buy one just for this.

Can I mix an RTX 3090 with an RTX 4090? Tensor parallel will run but allocates symmetrically, so the 4090 behaves like a second 3090 and its extra speed is mostly wasted waiting at sync points. Mismatched pairs are better served by pipeline parallelism or by running separate instances per card.

Why does vLLM crash on startup when Ollama runs fine on the same machine? vLLM pre-allocates its memory budget up front instead of growing on demand, so misconfiguration fails at boot rather than mid-request. Check that Ollama (or anything else) isn’t holding VRAM when vLLM starts, and lower --gpu-memory-utilization to 0.85.

What power supply do I need for dual RTX 3090s running vLLM? Plan around transients, not averages. Each card is rated 350W TDP but can spike well past that for milliseconds, and tensor parallel keeps both cards loaded simultaneously — so spikes coincide. A quality 1000W+ PSU is the safe floor; an 850W unit from a top-tier vendor can work but leaves little margin. Undervolting or power-limiting the cards (nvidia-smi -pl 280) costs single-digit percent inference speed and removes most of the risk.

Does tensor parallel work with AWQ and GPTQ quantized models? Yes — the launch example in this guide serves a 32B AWQ model at TP=2. Quantized weights split across GPUs the same way full-precision weights do, and quantization is what makes 30B+ models fit two 24GB cards with KV cache headroom at all. See the GPTQ vs AWQ vs GGUF breakdown for picking the format.

Sources

• vLLM releases on GitHub — v0.26.0, July 25, 2026
• vLLM engine arguments documentation — --scheduling-policy, --gpu-memory-utilization, --max-num-seqs
• vLLM priority scheduling RFC (#6077)
• Setting up vLLM with dual RTX 3090s — independent walkthrough of the same hardware class
• RTX 3090 power consumption and PSU requirements — 350W TDP, transient spike behavior, dual-card PSU sizing
• RTX 3090 transient draw analysis — Notebookcheck — measured spikes well above rated TDP

Recommended Gear

• RTX 3090 — 24GB VRAM, NVLink-capable, the used-market value pick for dual-GPU inference builds
• RTX 4090 — faster per card if you’re buying new; pair two for TP=2 over PCIe

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe