Autoencoder / VAE

An Autoencoder is a neural network designed to learn a compressed representation of data. It consists of two parts: an encoder that compresses the input into a smaller latent vector (the bottleneck), and a decoder that attempts to reconstruct the original input from this compressed representation.

By forcing the data through a bottleneck, the network cannot simply copy the input to the output. Instead, it must learn the most important structural features (like edges, shapes, or correlations) that allow it to approximate the original data. If the bottleneck is too small, the reconstruction becomes blurry or averages out distinct patterns because it lacks the capacity to store all necessary information.

In a Variational Autoencoder (VAE), this latent space is further regularized to be continuous and structured, allowing us to sample from it to generate entirely new data points that look like the training data.

1. The Encoder and Decoder

The encoder maps a high-dimensional input $x$ to a lower-dimensional latent vector $z = f(x)$. The decoder maps $z$ back to a reconstruction $\hat{x} = g(z)$. The network is trained by minimizing the reconstruction error, typically the Mean Squared Error between $x$ and $\hat{x}$.

2. The Bottleneck Effect

The size of the bottleneck determines the network's capacity.

• A wide bottleneck might learn an exact mapping without discovering any meaningful patterns.
• A narrow bottleneck forces extreme compression, capturing only the most dominant features (principal components) of the dataset.

In this simulation, you can see how reducing the bottleneck size forces the model to drop less important combinations, increasing the reconstruction error and causing patterns to blur.