distiliaa Learn
Module 4 18 min

What Are Embeddings?

The latent vectors you just learned about, applied to words and sentences.

Tokenization turned text into IDs — but an ID is just a row number. Token 9906 is not "closer" to token 9907 in meaning; the numbers are arbitrary labels. For a model to actually reason about language, each token needs to become something richer: a list of numbers that captures what it means. That list is an embedding, and it is arguably the single most important idea in modern AI.

Good news: you already know what an embedding is. In the last module you watched an autoencoder squeeze a 784-pixel digit into a small latent vector, and you saw that latent space organize itself — similar digits landing near each other, directions meaning something. An embedding is exactly that latent vector, applied to language. Same idea, new domain.

Why does this exist?

Neural networks can only do math on numbers, and useful math requires that similar inputs produce similar numbers. Token IDs fail this: cat=5432 and kitten=8901 look unrelated. Embeddings fix it by mapping each token (or sentence, or image) to a point in a high-dimensional space where distance means semantic difference — so "similar meaning" becomes literally "nearby numbers" that math can operate on.

The problem: IDs carry no meaning

Suppose cat is token 5432, kitten is 8901, and carburetor is 5433. Judging by the numbers, cat is nearly identical to carburetor and unrelated to kitten — exactly backwards. IDs are like street addresses assigned alphabetically: house 41 and house 42 might be in different cities.

First attempt: one-hot vectors

A classic fix is the one-hot vector: represent each token as a giant list of zeros with a single 1 in its own slot. With a 50,000-token vocabulary, cat is 50,000 numbers, all zero except position 5432.

cat    = [0, 0, ..., 1, ..., 0]  # 1 at position 5432
kitten = [0, 0, ..., 1, ..., 0]  # 1 at position 8901

This removes the fake ordering — but every pair of words is now equally different. The dot product of any two distinct one-hot vectors is exactly 0. cat vs kitten: zero similarity. cat vs carburetor: zero similarity. We have replaced wrong relationships with no relationships, and burned 50,000 dimensions doing it.

The fix: dense vectors, learned from data

An embedding keeps the "word = list of numbers" idea but makes two changes: the list is short (hundreds to a few thousand numbers, not vocabulary-sized), and the numbers are learned so that words used in similar contexts end up with similar lists.

cat    = [ 0.21, -0.94,  0.13,  0.77, ...]   # 768 numbers
kitten = [ 0.19, -0.88,  0.20,  0.71, ...]   # nearly the same numbers!
lawyer = [-0.61,  0.32, -0.45,  0.02, ...]   # very different numbers

Where do the numbers come from? Training. A model repeatedly sees phrases like the cat sat, the kitten sat, feed the cat, feed the kitten — and to predict text well, it is forced to give cat and kitten similar vectors, because they are interchangeable in so many contexts. Meaning is inferred from company kept: you shall know a word by the words around it.

Dimensions as directions of meaning

Think of each of the hundreds of numbers as a coordinate along some axis. In a cartoon version:

  • Axis 1 could be "how alive is this?" — cat scores high, rock low.
  • Axis 2 could be "how big?" — whale high, ant low.
  • Axis 3 could be "how formal?" — purchase higher than buy.

Real learned dimensions are rarely this clean — meaning gets smeared across combinations of axes — but the intuition holds: each dimension is a direction, and a word's meaning is its position along all of them at once. With 768 dimensions you can encode an enormous number of independent shades of meaning.

  1. Token IDcat is token 5432 — an arbitrary label with no meaning baked in.
  2. One-hot
  3. Dense embedding
  4. Geometry emerges

The most famous demonstration: after training, vector arithmetic works. Take the vector for king, subtract man, add woman — the nearest word to the result is queen. The direction from man to woman encodes something like gender, and it applies consistently across the space.

See the space

Real embeddings have hundreds of dimensions, which we obviously can't draw — so the graph below projects them down to 2D while trying to keep nearby points nearby. Look for neighborhoods: animals near animals, foods near foods. Distance on this map approximates distance in meaning.

Embedding space explorer

Words with similar meanings sit close together. Click a word to see its 3 nearest neighbors.

dogcathorseliontigerbirdfishrabbitpizzaburgerpastasaladbreadcheeseapplericecomputerphonekeyboardsoftwareinternetrobotserveralgorithmhappysadangrycalmexcitedafraidproudlonelyparislondonbeachmountaincityvillagedesertforest
Nothing selected — click any point.

Get real embeddings, for free

You don't have to take our word for any of this. Using the Gemini API key you created in the Setup lesson, you can embed real text right now — in a Colab cell or any Python environment:

# pip install google-genai numpy
import os
import numpy as np
from google import genai

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

words = ["cat", "kitten", "carburetor"]
result = client.models.embed_content(
    model="gemini-embedding-001",
    contents=words,
)
vectors = [np.array(e.values) for e in result.embeddings]
print(len(vectors[0]), "dimensions per word")

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print("cat vs kitten:    ", round(cosine(vectors[0], vectors[1]), 3))
print("cat vs carburetor:", round(cosine(vectors[0], vectors[2]), 3))

Run it: cat vs kitten scores far higher than cat vs carburetor. The similarity math is the topic of the next lesson — for now, the point is that these vectors are real, free to generate, and yours to play with.

Save embeddings for later

Add np.save("word_vectors.npy", np.stack(vectors)) and you have a reusable embedding file — we'll use exactly this pattern in the RAG module, where embedding documents once and saving them is the whole game.

Not just words

Everything here scales up: models embed whole sentences, paragraphs, images, and audio into the same kind of space. A sentence embedding is one vector summarizing the entire sentence — this is the foundation of semantic search, recommendations, and RAG, coming later in this module.

Build it yourself

  1. In the graph above, pick a word and predict its three nearest neighbors before looking. How good is your inner embedding model?
  2. Draw a 2D space on paper with axes "size" and "ferocity". Place: kitten, tiger, hamster, wolf, elephant. You just hand-built a 2-dimensional embedding.
  3. Using your paper space, compute which pair is closest by eyeballing distances. Does it match your intuition of similarity?

Summary

  • Token IDs are arbitrary labels; models need representations where similar meaning means similar numbers.
  • One-hot vectors remove false ordering but make everything equally dissimilar — and are absurdly wasteful.
  • An embedding is a short, dense, learned vector; words appearing in similar contexts get similar vectors.
  • Each dimension is a direction of meaning; relationships (like man→woman) become consistent directions.
  • The same trick embeds sentences, images, and audio — the foundation for search and RAG ahead.