TL;DR: Embedding models are the one part of a RAG stack where going local costs you almost nothing — nomic-embed-text-v1.5 is 137M parameters, Apache 2.0, and runs on any GPU or a bare CPU while matching OpenAI’s text-embedding-3-small on English retrieval. The catch is lock-in, not quality: every vector in your database is tied to the model that produced it, so switching later means re-embedding everything. Anyone running Open WebUI, AnythingLLM, or a pgvector pipeline should make this swap before their document collection grows.

	nomic-embed-text-v1.5	mxbai-embed-large-v1	BGE-M3
Best for	Default local pick, long documents	Max English precision on short chunks	Multilingual corpora
License	Apache 2.0	Apache 2.0	MIT
Parameters / dims	137M / 768 (down to 64)	335M / 1024	570M / 1024
Max input	8,192 tokens	512 tokens	8,192 tokens
The catch	768 dims trail the big models on nuanced ranking	512-token limit forces small chunks	Heaviest of the three; overkill for English-only

Honest take: Pull nomic-embed-text into Ollama and stop paying per-token for embeddings — you will not notice a quality difference on ordinary document Q&A. Add BGE’s reranker only when you have real retrieval-precision problems, not before.

Why embeddings are the easy half of local AI

The r/LocalLLaMA consensus that surfaced this summer is worth repeating: running local embedding models is practical for almost everyone, even people who happily keep paying for a frontier LLM API. The reasoning holds up. A useful local LLM needs 8–24GB of VRAM. A state-of-the-art embedding model needs a few hundred megabytes. You can run one next to your existing stack on a CPU-only mini PC — an RTX 3060 is already overkill.

An embedding model converts text into a fixed-length vector — 768 floats for nomic-embed, 1,536 for OpenAI’s text-embedding-3-small — positioned so semantically similar text lands close together. Your vector database (pgvector, Chroma, or Qdrant) then finds the stored chunks nearest to your query vector. That’s retrieval. Every RAG pipeline does this on every query and on every document you ingest, which is exactly why it’s the component you want off metered billing: it’s the highest-call-volume, lowest-compute piece of the whole stack.

The three models worth running in 2026

nomic-embed-text-v1.5 — the default

Nomic’s 137M-parameter model is the most-pulled embedding model on Ollama for good reason. Apache 2.0 with open training data and code, an 8,192-token context window that handles whole documents without aggressive chunking, and Matryoshka representation learning — you can truncate its 768-dimension output down to 512, 256, or even 64 dims and lose surprisingly little accuracy, which matters when your vector DB’s index size starts hurting. Nomic’s published benchmarks show it beating both OpenAI’s ada-002 and text-embedding-3-small on English retrieval tasks. The full-precision weights are a ~270MB download.

mxbai-embed-large-v1 — precision on short chunks

Mixedbread’s 335M-parameter BERT-large model is the quality pick for English. On the classic 56-dataset MTEB retrieval suite, Mixedbread reports 64.68 — a hair above OpenAI’s text-embedding-3-large at 64.58, from a model you can run on a laptop. Also Apache 2.0.

One widely-miscited spec needs correcting: you’ll see claims floating around that mxbai has a 64-token input limit. It doesn’t — the model’s maximum sequence length is 512 tokens. That’s still the real constraint to design around: 512 tokens is roughly 350–400 words, so mxbai only makes sense in pipelines that chunk documents small. If your chunking strategy produces 1,000-token chunks (see our RAG architecture deep dive for why you might want that), mxbai will silently truncate them and retrieval quality will quietly rot.

BGE-M3 — the multilingual one

BAAI’s BGE-M3 (MIT-licensed, 570M parameters) is the pick the moment your documents aren’t all English. It covers 100+ languages, takes 8,192 tokens of input, and is unusual in supporting three retrieval modes from one model: dense vectors, sparse (lexical) weights, and multi-vector ColBERT-style scoring. Most local stacks only use the dense mode, but the sparse output is genuinely useful for hybrid search in Qdrant or Weaviate. At 1.2GB in FP16 it’s the heaviest model here — still trivial next to any LLM.

Setup: two commands with Ollama

Tested against Ollama v0.32 (released July 2026). Both major models are in the official library:

ollama pull nomic-embed-text
ollama pull mxbai-embed-large

Verify with a raw API call:

curl http://localhost:11434/api/embed \
  -d '{"model": "nomic-embed-text", "input": "self-hosted RAG beats cloud RAG"}'

Expected output (truncated — the real response is 768 floats):

{"model":"nomic-embed-text","embeddings":[[0.0104,-0.0271,0.0813,...]],"total_duration":14570000}

Note the endpoint: /api/embed is the current batch-capable endpoint, and Ollama also serves the OpenAI-compatible /v1/embeddings route — which means anything built for the OpenAI SDK can point at http://localhost:11434/v1 with the model name swapped, and just work.

Wiring it into the tools you already run:

• Open WebUI: Admin Settings → Documents → set Embedding Model Engine to Ollama and the model to nomic-embed-text. Re-index your documents afterward — more on why below.
• AnythingLLM: Settings → Embedding Engine → Ollama → pick the model. Our AnythingLLM RAG setup guide covers the rest of that pipeline.
• LangChain / custom Python: OllamaEmbeddings(model="nomic-embed-text"), or skip Ollama entirely with pip install sentence-transformers for models not in the library (BGE-M3’s sparse mode needs the FlagEmbedding package instead).

Rerankers: the upgrade nobody tells you about

Embedding retrieval is fast but blunt: it compares your query against chunks that were embedded without knowing the question. A reranker is a cross-encoder — it reads the query and a candidate chunk together and scores actual relevance. The standard local pattern is two-stage: pull the top 20–50 candidates by vector similarity, then let the reranker reorder them and keep the top 3–5 for the LLM’s context.

The FOSS options are small enough to be a rounding error on your hardware:

• BAAI/bge-reranker-v2-m3 — Apache 2.0 (the base BGE-M3 embedder is MIT, but the reranker ships under Apache 2.0 — check the actual license file, not a blog post), 568M parameters, multilingual, built on BGE-M3. The default choice.
• jina-reranker-v2-base-multilingual — 278M parameters and fast, but CC-BY-NC licensed for the weights — fine for homelab use, a problem for anything commercial.

Open WebUI has this built in: Admin Settings → Documents → enable Hybrid Search and set BAAI/bge-reranker-v2-m3 as the reranking model. In LangChain it’s a few lines:

from langchain_community.cross_encoders import HuggingFaceCrossEncoder
from langchain.retrievers.document_compressors import CrossEncoderReranker

model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-v2-m3")
compressor = CrossEncoderReranker(model=model, top_n=4)

When does it pay off? When your corpus has many near-duplicate or topically-adjacent documents — internal wikis, changelogs, support tickets — and the top vector hits are “about the right topic” but not “the actual answer.” If your RAG answers are already fine, skip it; a reranker adds 50–300ms per query and another model to babysit.

The real problem you’ll hit: mixed-model vector soup

This one cost me an afternoon. I switched an Open WebUI instance from the default SentenceTransformers model to nomic-embed-text, asked a test question, and got confidently irrelevant chunks back — worse than before the “upgrade.” The cause: documents ingested before the switch were still stored as vectors from the old model. Vectors from two different embedding models live in incompatible coordinate systems; cosine similarity between them is noise, even when the dimensions happen to match.

The fix is unglamorous: delete and re-ingest every document collection after changing embedding models. In pgvector you’ll at least get a loud failure if dimensions differ (expected 1536 dimensions, not 768 on insert — the column is typed) — but if the dims match, nothing errors and retrieval just silently degrades. Two habits prevent this: store the embedding model name in your collection metadata, and treat “change embedding model” as “rebuild the index,” always. This is also the strongest argument for picking a local model early, before you’ve embedded 40GB of documents against a paid API you’ll later want to leave.

The cost math, honestly

The brutal truth: the per-token savings argument is weak. OpenAI’s text-embedding-3-small costs $0.02 per million tokens in August 2026 (text-embedding-3-large is $0.13/M). Embedding ten million tokens a month — a heavy personal workload — costs twenty cents. Nobody self-hosts embeddings to save $2.40 a year.

The actual reasons to go local:

1. Privacy — embedding is the step where your entire document corpus leaves the machine, not just queries. For the self-hosters running a sovereign RAG stack, a cloud embedding API defeats the point.
2. Lock-in — every embedded chunk chains you to that API (see above). Re-embedding a large corpus against a metered endpoint is where costs stop being twenty cents.
3. Rate limits and latency — bulk-ingesting a document archive hits API rate tiers fast; a local model on even modest hardware embeds thousands of chunks per minute with no 429s.
4. Offline operation — your RAG pipeline keeps working when the API, or your internet, doesn’t.

On quality: OpenAI’s small model scores around 62.3 on the MTEB average — below mxbai-embed-large and roughly at nomic-embed’s level. For ordinary document Q&A, the 768-dimension nomic output is indistinguishable from the 1,536-dimension OpenAI output in blind testing. Dimension count is a storage cost, not a quality score.

When NOT to go local

Skip the swap if any of these apply. Embedding a corpus in dozens of languages with strict quality needs — the top proprietary multilingual embedders (Cohere embed-v4, Gemini embedding) still beat BGE-M3 on low-resource languages. Building a product where retrieval quality is the product — eval against your own data first; MTEB averages hide task-specific gaps. Running serverless where a 300MB model download per cold start is a dealbreaker. And if you already have millions of vectors embedded with text-embedding-3 and no pain, re-embedding for ideology alone is a bad trade — wait until you have another reason to rebuild the index.

For hardware sizing the rest of your stack (the LLM is what actually needs VRAM — embeddings will run on whatever’s left), see runaihome.com’s guides at runaihome.com.

FAQ

Can I run these embedding models on CPU only? Yes, and it’s genuinely fine. nomic-embed-text at 137M parameters embeds single queries in tens of milliseconds on a modern CPU. Bulk ingestion of a large archive is where a GPU helps, but even there a CPU-only box just means minutes instead of seconds.

Do my embedding model and LLM need to match? No. They’re completely independent — the embedding model finds relevant chunks, the LLM reads them as plain text. Pairing local nomic-embed with a cloud frontier LLM is exactly the hybrid the r/LocalLLaMA crowd landed on, and it’s a sensible endgame.

What happens if my documents are longer than the model’s input limit? The model truncates silently — everything past 8,192 tokens (nomic, BGE-M3) or 512 tokens (mxbai) never makes it into the vector, so it can never be retrieved. Your chunking step must produce chunks under the embedder’s limit; that’s a pipeline setting, not something the model warns you about.

Sources

• nomic-ai/nomic-embed-text-v1.5 — Hugging Face model card
• mxbai-embed-large-v1 — Mixedbread documentation
• BAAI/bge-m3 — Hugging Face model card
• BAAI/bge-reranker-v2-m3 — Hugging Face model card
• OpenAI — new embedding models and API pricing

What self-hosting actually costs

Real cost breakdowns for self-hosted AI: hardware floors, power, maintenance hours, and the honest comparison against paying for it. No spam, unsubscribe anytime.

Email address Subscribe