The Embedding Playground

You know what embeddings are and how to measure similarity between them. Time to stop reading and start poking. This lesson is a guided tour through a live embedding space — a series of experiments designed to build the spatial intuition you will lean on constantly when building semantic search and RAG systems.

The problem: you can't debug what you can't see

When a semantic search returns a bizarre result, the answer is almost always geometric: the query landed in an unexpected neighborhood, or two concepts you consider different sit right on top of each other in the space. Engineers who have explored an embedding map diagnose these issues in minutes; those who haven't stare at similarity scores in a log file. So let's explore.

Keep the graph open — every section below is an experiment to run in it. Remember the caveat from earlier: this is a 2D projection of a space with hundreds of dimensions, so it preserves neighborhoods well but distorts long distances.

Experiment 1: find the clusters

Zoom out and just look. You should see the space is lumpy, not uniform: animals huddle together, food words form their own island, emotions cluster elsewhere. Nobody labeled these groups — they emerged purely from words appearing in similar contexts during training.

Now look closer at one cluster. Clusters have internal structure too: within animals, pets may sit nearer each other than to wild animals. Embedding spaces are organized at every scale.

Experiment 2: nearest neighbors

Pick a word and identify its nearest neighbors. This single operation — given a vector, find the closest vectors — is the workhorse of the entire embedding economy:

• Semantic search = embed the query, return nearest documents.
• Recommendations = embed what the user liked, return nearest items.
• Deduplication = flag pairs that are suspiciously near.
• RAG = embed the question, retrieve nearest chunks, hand them to an LLM as context.

Try a word with multiple meanings, like bank or apple, if the vocabulary has one. A single point must average all its senses — river bank pulls one way, money pulls the other, and the word settles somewhere in between, near neither neighborhood perfectly. This is a genuine limitation of one-vector-per-word embeddings, and it is why modern systems embed words in context (whole sentences) instead.

Experiment 3: add your own words

Add new words to the graph and watch where they land.

1. Add a word that obviously belongs to an existing cluster — puppy near the animals. Did it land where you predicted?
2. Add a word that sits between clusters — try hotdog (food? animal-adjacent pun?) or jaguar (animal? car?). Boundary words reveal how the space negotiates ambiguity.
3. Add an emotionally loaded word and a neutral synonym — terminate vs end. Are they as close as a thesaurus would suggest? Embeddings capture usage, and usage includes tone.

Experiment 4: out-of-vocabulary words

Now try to break it. Add a made-up word like flibbertigig, a fresh slang term, or a niche product name.

What happens depends on the system. Old-school word embeddings (word2vec-style) simply have no vector for unseen words — the lookup fails. Modern embedding models tokenize first, so flibbertigig becomes subword pieces (fl, ibber, tig...) and gets a vector — but one assembled from fragments, often landing somewhere generic and unhelpful. There is no magic: the model can only reflect what its training data taught it.

From playground to production

Everything you just did manually is what a vector database does at scale:

# The pattern behind every semantic search / RAG system
chunks = split_documents(docs)                     # 1. chunk your corpus
vectors = [embed(c) for c in chunks]               # 2. embed every chunk
index = VectorIndex(vectors)                       # 3. index for fast lookup

def search(query, k=5):
    q = embed(query)                               # 4. embed the query
    return index.nearest(q, k)                     # 5. nearest neighbors

results = search("what is our refund policy?")
# 6. For RAG: stuff results into an LLM prompt and ask it to answer

Steps 4-5 are exactly your Experiment 2, just executed over millions of vectors with clever indexing (approximate nearest neighbor algorithms) instead of eyeballs. When the RAG lessons arrive later in this course, you will already have the geometric picture in your head.

Build it yourself

1. Cluster census. List every cluster you can find in the graph and give each a name. Then find the two clusters that sit closest together — why might the model consider them related?
2. Neighbor prediction game. Before checking, write down the predicted top-3 neighbors for five words. Score yourself. Where your intuition disagreed with the model, form a hypothesis about the training data that could explain it.
3. Search engine on paper. You have 6 documents about cooking, 6 about cars. A user searches how to fix a flat. Sketch where the query and documents sit in a 2D space and which documents get retrieved. Now add a document titled fixing a flat-bottomed pan — where does it sit, and does your search get confused?

Summary

• Embedding spaces are lumpy: semantic clusters emerge from training with no human labeling, and structure exists at every scale.
• Nearest-neighbor lookup is the one primitive behind semantic search, recommendations, deduplication, and RAG retrieval.
• Single-vector word embeddings blur multiple word senses together; context-aware sentence embeddings are the modern fix.
• Out-of-vocabulary terms (jargon, codenames, new slang) embed poorly — expect it, and mitigate with fine-tuning or hybrid keyword search.
• A vector database is this playground at scale: chunk, embed, index, then nearest-neighbor your way to relevant context.