Learning to Compress

So far you've seen how text becomes numbers: characters become bytes, and text becomes tokens. But those are mechanical translations — they carry no understanding. A photo of a cat and a photo of a dog are just two grids of pixel numbers that look nothing alike. How does a machine learn that they are both "pet photos"? The cleanest way to understand the answer is the autoencoder — and we'll use images, because you can see compression happen.

The problem: nobody labels the world

Supervised learning needs labels: this photo is a cat, that email is spam. Labels are expensive. But raw, unlabeled data is everywhere. The question that motivated autoencoders was: can a network learn something useful from data alone?

The answer is a beautifully weird trick: ask the network to output exactly what it was given as input. That sounds pointless — copying is easy. So we make copying impossible by squeezing the data through a bottleneck.

The hourglass

An autoencoder has three parts:

• Encoder — layers that shrink the input down (say 8 numbers → 4 → 2).
• Latent space — the tiny middle layer, the bottleneck. This is the compressed representation, often called z.
• Decoder — layers that expand back up (2 → 4 → 8), trying to rebuild the original.

Because the middle layer is smaller than the input, the network cannot just copy. It must figure out what matters and throw the rest away. Training pushes it to minimize reconstruction error — how different the output is from the input.

Watch it compress

Press Compress to see a data pulse flow through the hourglass, then drag the latent-size slider. Watch the smiley face: with 4 latent neurons the reconstruction is nearly perfect; with 1 it becomes a blurry ghost. That is the fundamental trade-off — smaller latent, more information discarded.

Notice that the degradation isn't random noise. The reconstruction keeps the big structure (a round blob with dark regions) and loses the fine detail (individual pixels). The network spends its tiny budget on what matters most — exactly what a good summary does.

A tiny autoencoder in code

This is real, runnable code — open a free notebook at colab.research.google.com, paste it into a cell, and run it. Colab already has PyTorch and the MNIST handwritten-digit images installed; no GPU needed (it takes about a minute on CPU).

import torch, torch.nn as nn
from torchvision import datasets, transforms

data = datasets.MNIST(root=".", download=True, transform=transforms.ToTensor())
loader = torch.utils.data.DataLoader(data, batch_size=256, shuffle=True)

encoder = nn.Sequential(nn.Flatten(), nn.Linear(784, 64), nn.ReLU(), nn.Linear(64, 2))
decoder = nn.Sequential(nn.Linear(2, 64), nn.ReLU(), nn.Linear(64, 784), nn.Sigmoid())
opt = torch.optim.Adam(list(encoder.parameters()) + list(decoder.parameters()), lr=1e-3)

for epoch in range(3):
    for x, _ in loader:              # note: we ignore the labels entirely!
        z = encoder(x)               # compress: 784 pixels -> 2 numbers
        x_hat = decoder(z)           # reconstruct: 2 -> 784
        loss = ((x.flatten(1) - x_hat) ** 2).mean()   # reconstruction error
        opt.zero_grad(); loss.backward(); opt.step()
    print(f"epoch {epoch}: loss {loss.item():.4f}")

Then look at what it learned — show an original digit next to its reconstruction:

import matplotlib.pyplot as plt

x, _ = data[0]
with torch.no_grad():
    x_hat = decoder(encoder(x.unsqueeze(0))).reshape(28, 28)

fig, ax = plt.subplots(1, 2)
ax[0].imshow(x.squeeze(), cmap="gray"); ax[0].set_title("original")
ax[1].imshow(x_hat, cmap="gray");       ax[1].set_title("rebuilt from 2 numbers")
plt.show()

A whole handwritten digit — 784 pixels — squeezed through just 2 numbers and rebuilt. Blurry, but recognizable. The target is the input itself; that's why this is called self-supervised learning — the data supervises itself.

Why the latent space is the interesting part

After training, most people throw away the decoder and keep the encoder. Why? Because the latent vector z is a learned, dense, meaningful summary of the input — an embedding! Two similar inputs get squeezed to nearby latent points, because the decoder has to reconstruct similar outputs from them. Compression forces semantic organization. We'll explore this space in the next lesson.

xs = torch.stack([data[i][0] for i in range(2000)])
ys = [data[i][1] for i in range(2000)]
with torch.no_grad():
    zs = encoder(xs)
plt.scatter(zs[:, 0], zs[:, 1], c=ys, cmap="tab10", s=6)
plt.colorbar(); plt.show()

Summary

• Autoencoders learn by reconstructing their own input — no labels needed (self-supervised learning).
• The architecture is an hourglass: encoder → latent bottleneck → decoder.
• The bottleneck makes copying impossible, forcing the network to keep only essential structure.
• Smaller latent spaces mean more compression and worse reconstruction — a dial you played with above.
• The latent vector is a learned embedding: compression forces similar inputs to land near each other.