FromZero2AI
CoursesPlaygroundBlog
Support on Ko-fi
Embeddings & Vector Databases • Module A: Understanding EmbeddingsLesson 3: Choosing the Right Embedding Model
PreviousNext

Lesson 3: Choosing the Right Embedding Model

Compare embedding model families by dimension count, context length, cost, and benchmark performance. Know when to use a hosted API vs. a local model like nomic-embed.

Built with AI for beginners. Free forever.

Support on Ko-fi•About•Blog•Privacy Policy•Terms of Service

Picking the wrong embedding model costs real money and degrades retrieval quality across your entire system. This lesson gives you a clear framework for choosing — and swapping — embedding models without rebuilding everything from scratch.

The Four Axes That Matter

Dimensions

How long is the vector? More dimensions can capture more nuance but cost more storage and search time.

Context Length

How many tokens can the model embed in one call? Longer context = fewer chunks for the same document.

Cost

Charged per million tokens. For a 10M-token corpus, a 5× price difference is significant.

Benchmark Performance

MTEB (Massive Text Embedding Benchmark) is the industry standard. Higher is better, but domain matters.

Model Comparison Table (2025)

ModelDimsContext$/M tokensMTEB Avg
text-embedding-3-small15368,191$0.0262.3
text-embedding-3-large30728,191$0.1364.6
voyage-3 (Anthropic)★ recommended102432,000$0.0668.1
voyage-3-lite51232,000$0.0265.2
nomic-embed-text (local)7688,192Free62.4
mxbai-embed-large (local)1024512Free64.7

Prices as of 2025. MTEB scores vary by task type — always benchmark on your own data.

The Matryoshka Trick: Truncating Dimensions

OpenAI's text-embedding-3-* models support a Matryoshka representation — they are trained so that the first N dimensions of a 3072-dimension vector are still meaningful on their own. This lets you trade retrieval quality for storage cost:

# Full 3072-dimension vector (highest quality)
response = client.embeddings.create(
    model="text-embedding-3-large",
    input="My document text here",
)
full_vec = response.data[0].embedding  # 3072 dims

# Truncated to 256 dimensions (10x storage savings, ~5% quality loss)
response_small = client.embeddings.create(
    model="text-embedding-3-large",
    input="My document text here",
    dimensions=256,  # API truncates for you
)
small_vec = response_small.data[0].embedding  # 256 dims

Local Models: When to Stop Paying Per Token

Once you cross roughly 500M tokens/month, local embedding models become cheaper than APIs.nomic-embed-text via Ollama is the easiest entry point:

# 1. Pull the model (one-time, ~300MB)
# $ ollama pull nomic-embed-text

from ollama import Client

client = Client()

response = client.embeddings(
    model="nomic-embed-text",
    prompt="My document text here",
)

vector = response["embedding"]  # 768 dimensions
print(len(vector))  # → 768

The trade-off: you manage GPU infrastructure, and quality is slightly lower than the best hosted models. For internal enterprise use cases where data cannot leave your network, local models are often required.

Domain-Specific vs. General-Purpose Models

General-purpose MTEB benchmarks measure average quality across many tasks. For specific domains, a purpose-built model often wins:

  • Legal / medical: voyage-law-2, voyage-finance-2 (Anthropic/Voyage)
  • Code: voyage-code-3 — understands function names, APIs, and code structure
  • Multilingual: multilingual-e5-large — handles 100+ languages in one model
  • Long documents: Any model with >8K context; Voyage-3 at 32K is the current leader

Decision Framework — Pick Your Scenario

Select your situation to see the recommended model and configuration:

Recommendation: nomic-embed-text via Ollama — zero API cost, 768 dims, 8K context

# Run locally: ollama pull nomic-embed-text
from ollama import Client

client = Client()

def embed(text: str) -> list[float]:
    response = client.embeddings(
        model="nomic-embed-text",
        prompt=text,
    )
    return response["embedding"]  # 768 dimensions

# Swap to an API model later by changing this one function
vector = embed("My document text here")
print(len(vector))  # → 768

The Model-Agnostic Design Rule

Always wrap your embedding call behind a single function like embed(text: str) -> list[float]. Swapping models then requires changing one line — not refactoring your entire codebase. You will rebuild the index when you switch models (vectors from different models are incompatible), but your application logic stays unchanged.

You now know what embeddings are, how to generate them, and how to pick the right model. In Lesson 4, we move to the storage side: vector databases — what they are, how they differ from Postgres, and how to use ChromaDB to store and search your first set of vectors.