Embeddings & Vector Databases • Module A: Understanding EmbeddingsLesson 1: What Are Embeddings? Text as Vectors
Every AI system that understands language — from RAG pipelines to semantic search to recommendation engines — runs on a single core idea: text can be represented as a point in space. That representation is called an embedding, and it is the foundational primitive of modern AI engineering.
Neural networks operate entirely on numbers. When you give a model the sentence "The cat sat on the mat", it never actually processes the letters C-A-T. Instead, every token (word or sub-word piece) is first converted into a long list of floating-point numbers — a vector. An embedding model's entire job is to produce vectors that capture meaning, not just identity.
A naive approach would assign each word a unique integer: cat=42, dog=43. The problem is that these numbers carry no semantic information. The model has no way to know that cat and dog are both animals, or that "automobile" and "car" mean the same thing.
Embedding models are trained to place semantically similar content close together in vector space. After training on billions of examples, the model learns that:
This is what makes semantic search possible: instead of matching keywords, you find the documents whose vectors are geometrically closest to the query's vector.
A real embedding for the word "cat" using OpenAI's text-embedding-3-small model is a list of 1536 floating-point numbers:
# The embedding for "cat" — 1536 dimensions [ -0.0231, 0.0418, -0.1052, 0.0776, 0.0334, 0.0891, -0.0562, 0.1204, -0.0788, 0.0445, ... (1526 more numbers) ... 0.0312, -0.0891, 0.0554 ]
Each dimension captures some learned feature of meaning. No single dimension has a human-readable label — the geometry emerges from training.
Imagine a map where every document, sentence, or word in existence has a unique GPS coordinate. Documents that discuss similar topics cluster together on this map. When a user types a search query, you convert it to GPS coordinates and find the nearest documents — regardless of whether they share any exact words with the query.
This is why a search for "how do I train a neural net" can surface a document titled"Backpropagation: A Practical Guide" — they live in the same neighborhood on the meaning map, even though not one word overlaps.
| Method | Era | Key Limitation |
|---|---|---|
| TF-IDF | 1970s–2010s | Keyword matching only — "car" and "automobile" are unrelated |
| Word2Vec / GloVe | 2013–2018 | One vector per word — no context, "bank" is always the same |
| BERT embeddings | 2018–2021 | Contextual but expensive; not optimized for retrieval |
| Modern embedding APIs | 2022–now | High cost at scale; context window limits |
Select a use case to see the code pattern:
Find documents by meaning, not keywords — even when zero words overlap
import chromadb
chroma = chromadb.Client()
collection = chroma.create_collection("docs")
collection.add(
documents=[
"Backpropagation: A Practical Guide",
"Training Neural Networks with Gradient Descent",
"How to bake a chocolate cake",
],
ids=["0", "1", "2"],
)
# Semantic search — no keyword overlap needed!
results = collection.query(
query_texts=["how do I train a neural net"],
n_results=2,
)
# Returns: doc 0 + doc 1 (geometrically close)
# "chocolate cake" is far away — excluded automaticallyInput text (any length)
│
▼
Tokenizer splits text into sub-word tokens
│
▼
Transformer encoder processes tokens in context
│
▼
Pooling layer collapses token vectors → single vector
│
▼
Optional: L2 normalize so all vectors have unit length
│
▼
Output: a dense vector of fixed dimension (e.g. 1536)The pooling step is where different models make different choices. Some take the[CLS] token, some average all tokens, and some use a learned pooling head. This is one reason two models with the same architecture can produce very different embeddings.
In the next lesson, you will generate your first real embeddings by calling the OpenAI and Anthropic APIs, inspect the raw vectors, and compute cosine similarity between sentences in Python.