How We Pick an Embedding Model for a New RAG Project

The embedding model is the layer that decides how well your RAG system finds anything. Change it and every retrieval quality metric moves. It's also the piece most teams don't think hard enough about — they pick whatever the tutorial used.

Here's how we approach choosing an embedding model on real projects. Not what the MTEB leaderboard says. What actually plays out in production.

Why leaderboards aren't the answer

Benchmark leaderboards rank embedding models on public datasets. Your corpus isn't on those datasets. The model that tops MTEB on academic tasks isn't necessarily the one that retrieves your customer's support tickets or your legal contracts best.

Leaderboards are a useful shortlist. They aren't a decision. The decision comes from testing two or three candidates on your own data.

The dimensions that actually matter

Quality on your domain

The single most important axis. A model trained heavily on English web content may lag on technical documentation, legal text, medical records, or non-English content. Test on the domain you actually have.

Cost per million tokens

Embedding is cheap compared to LLM inference but not free. If you're re-embedding a million documents to try a new model, you'll notice. Costs range from around $0.02 per million tokens for the cheapest hosted models to $0.13 per million for the most expensive at time of writing. Free if you self-host.

Vector dimensions

Dimensions decide storage and search speed. 1536 dims is heavier to store and slower to search than 384. On large corpora, that's a real infra cost. Some hosted models support "Matryoshka" truncation — the same embedding can be shortened at query time — which is useful when you want quality now and speed later.

Context window

How long a chunk can be before the model refuses. Older models cap at 512 tokens. Newer ones handle 8k or more. Longer context makes chunking decisions easier but doesn't automatically improve retrieval.

Multilingual support

If your users write in Spanish and your docs are in English, an English-only embedding model retrieves badly. Multilingual models like the Cohere multilingual embed or open-source E5-multilingual solve this cleanly.

The candidates worth testing today

OpenAI text-embedding-3-small / 3-large

The default many teams pick. Small is cheap and fast. Large is more expensive and often noticeably better on ambiguous queries. Both support dimension truncation. Reliable, well-supported, and boring in a good way.

Voyage AI

Consistently strong on domain-specific retrieval, especially with the domain-tuned models (voyage-code, voyage-law, voyage-finance). If you have a specialized corpus, worth a serious look.

Cohere embed v3

Strong multilingual support and reranking pipeline. Also produces "compressed" embeddings with dimension-reduction options built in. The reranker in the same family pairs well.

Open-source options

bge-large, gte-large, and E5 are the usual contenders. Self-hosted removes per-call cost, adds infrastructure. Good if you're embedding at massive scale or need to keep data fully on-prem. Not the right first choice for a small team.

How to test three models in a day

Build an eval set — 50 to 100 queries with known-correct source chunks. Embed the same corpus with each candidate model. Run retrieval. Measure recall at top 3 and top 10 for each. Whichever wins by a clear margin, ship.

This process takes an afternoon and settles arguments faster than any leaderboard. Our post on debugging RAG covers how to build the eval set if you don't have one.

Where teams get it wrong

Picking based on the leaderboard

Already covered. Your corpus disagrees.

Never testing alternatives

Teams pick an embedding model on day one and never revisit it. New models are released monthly. Every six months is a reasonable cadence to retest against the current best.

Mixing embedding models mid-corpus

Every chunk in a single index must be embedded with the same model. Cosine distance across models is meaningless. If you switch models, re-embed everything. Version the corpus so you can compare.

Ignoring the reranker

Embedding quality matters less if you rerank aggressively. If reranking closes the gap between a cheap embedding and an expensive one, ship the cheap one. This is often the case.

Cost math worth doing

Assume 10 million tokens of corpus, 1 million queries per month at 20 tokens each. That's 30 million tokens embedded initially plus ongoing embedding at query time.

At $0.02 per million: about $0.60 setup and $0.40 per month. At $0.13 per million: $3.90 setup and $2.60 per month. The gap is real but small at this scale.

The math changes fast at hundreds of millions of tokens. That's when self-hosting starts making sense. Below that, hosted usually wins on operational simplicity.

Domain-tuned vs general

Voyage's domain models (code, law, finance) are trained on specialized corpora. On the matching domain they consistently outperform general-purpose embeddings. On unrelated domains they usually don't beat a good general model.

The heuristic: if your entire corpus fits neatly into one specialized domain, test the domain model. If your corpus is mixed (docs, code, tickets, marketing), stick with general and rely on hybrid retrieval to handle jargon.

Trade-offs

Bigger embeddings mean better recall on ambiguous queries, more storage, slower search. Multilingual support means slightly weaker English performance in exchange for coverage. Self-hosted means no per-call cost but real infrastructure responsibility. Domain-tuned models excel on their domain and lag elsewhere.

None of these are one-size answers. They're trade-offs to make consciously.

Common misconceptions

"Bigger dimensions always mean better quality." Not linear. Beyond a point, extra dimensions add cost without accuracy.

"I have to pick one model forever." You don't. Re-embedding a corpus is annoying but not catastrophic. Version and revisit.

"Open-source is always cheaper." True at massive scale, false at startup scale after operational overhead.

Frequently asked questions

What's a safe default embedding model?

text-embedding-3-large from OpenAI covers most use cases well and integrates everywhere.

Do I need a domain-specific model?

Only if your corpus is heavily specialized in one domain. For mixed corpora, general models with reranking usually beat naive domain models.

Should I self-host embeddings?

Only at scale or under strict data-residency needs. Below tens of millions of documents, hosted models are simpler.

How often should I re-evaluate?

Every 6–12 months. New models regularly change the picture.

Do embedding models affect hallucination rate?

Indirectly. Better embeddings mean better retrieval, which means the model sees the right context, which means it hallucinates less.

Conclusion

Choosing an embedding model for RAG is a decision worth making deliberately. The leaderboard is a starting shortlist, not the answer. Build a small eval set, test two or three candidates on your own data, ship the winner, and revisit every six months.

Get this piece right and the rest of the RAG stack becomes tunable instead of frustrating. Get it wrong and you'll spend months blaming the model.

If your RAG pipeline feels stuck and you can't tell which layer is the bottleneck, the AI Audit is the fastest way to get a diagnostic.