Measuring Similarity

Embeddings put meaning into geometry: similar texts become nearby points. But "nearby" is doing a lot of work in that sentence. Nearby how, exactly? There are several reasonable ways to measure how close two vectors are, they can disagree, and picking the right one matters every time you build search, deduplication, or RAG. This lesson builds the three classic measures from scratch.

The problem: define "close" for lists of numbers

Take three tiny 2D embeddings:

cat    = [2.0, 3.0]
kitten = [4.0, 6.0]     # same direction as cat, twice as far out
car    = [3.0, -1.0]    # different direction

Intuitively kitten should be most similar to cat. Let's see which measures agree.

Measure 1: Euclidean distance

The straight-line, ruler distance — square the differences per dimension, sum, square-root:

def euclidean(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b)) ** 0.5

euclidean(cat, kitten)  # 3.61
euclidean(cat, car)     # 4.12

Smaller = more similar. It works here, but notice something uncomfortable: kitten points in exactly the same direction as cat, yet racks up distance 3.61 purely for being farther from the origin. Euclidean distance punishes magnitude differences even when the direction — the meaning — is identical.

Measure 2: dot product

Multiply matching dimensions and sum:

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

dot(cat, kitten)  # 2*4 + 3*6 = 26
dot(cat, car)     # 2*3 + 3*(-1) = 3

Bigger = more similar. The dot product rewards agreement: dimensions where both vectors are strongly positive (or both strongly negative) push the score up; disagreement pushes it down. But it also rewards sheer length — doubling kitten doubles its score with everything. A long, rambling document could beat a perfectly matching short one just by being big.

Measure 3: cosine similarity — direction only

The fix: take the dot product, then divide out both lengths. What remains is the cosine of the angle between the vectors:

def norm(a):
    return sum(x * x for x in a) ** 0.5

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))

cosine(cat, kitten)  # 1.00  — identical direction
cosine(cat, car)     # 0.26  — pointing rather differently

The score lands between -1 and 1: 1 means same direction (same meaning), 0 means perpendicular (unrelated), -1 means opposite. kitten now scores a perfect 1.0 with cat, because cosine sees only the angle — magnitude is irrelevant.

Why cosine wins for text

Text embeddings often have magnitudes that vary for boring reasons — document length, word frequency, quirks of the model — while direction is where topical meaning lives. A one-line summary and a three-page report on the same subject should count as similar; cosine says yes, Euclidean says no.

Feel the geometry

Play with the explorer below. Drag the vectors around and watch all three measures update. Two experiments to run deliberately: (1) keep the angle fixed and stretch one vector — cosine holds still while the others swing; (2) keep lengths fixed and rotate — now all three move together.

A realistic snippet

import numpy as np

def cosine_sim(a, b):
    a, b = np.asarray(a), np.asarray(b)
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

query = embed("how do I reset my password")      # imagine an embed() API
docs = {name: embed(text) for name, text in knowledge_base.items()}

ranked = sorted(docs, key=lambda d: cosine_sim(query, docs[d]), reverse=True)
print(ranked[:3])   # the three most relevant documents

That's semantic search in six lines — the heart of every RAG pipeline you will build later.

Build it yourself

1. By hand (no code), compute the cosine similarity of [1, 0] and [1, 1]. Check that it equals about 0.707 — the cosine of 45 degrees.
2. Construct two vectors whose dot product is large but whose cosine similarity is small-ish. What did you exploit?
3. Implement cosine_sim yourself, then verify on [3, 4] vs [6, 8] that scaling a vector never changes the result.

Summary

• Euclidean distance measures straight-line separation but punishes magnitude differences that often carry no meaning.
• Dot product rewards dimension-wise agreement but is inflated by vector length.
• Cosine similarity = dot product with lengths divided out; it isolates direction, which is where topical meaning lives — hence the default for text.
• On unit-normalized vectors all three measures rank identically, which is why vector databases use fast dot products internally.
• Never compare absolute similarity scores across different embedding models; calibrate per model.