NeuralDocs

Types of Neural Networks / Autoencoders & VAEs

Autoencoders & VAEs

Intermediate

Networks that learn to compress input into a latent code and reconstruct it.

Architecture

An encoder maps x to a latent code z, and a decoder reconstructs x̂ from z. Undercomplete autoencoders use a lower-dimensional bottleneck to encourage compressed representations; other variants may use larger latent spaces with regularization.

xzx̂EncoderDecoder
The encoder compresses input x into a small latent code z (the bottleneck); the decoder reconstructs x̂. Training minimises the reconstruction error between x and x̂.
Lrecon=1n∑i=1n∥xi−gθ(fϕ(xi))∥22\boldsymbol{\mathcal{L}_{\mathrm{recon}}=\frac{1}{n}\sum_{i=1}^{n}\left\lVert\mathbf{x}_i-g_\theta(f_\phi(\mathbf{x}_i))\right\rVert_2^2}

fφ is the encoder and gθ is the decoder. This squared-error reconstruction objective averages over n examples and sums over each example’s features.

Variants

ELBO(x)=Eqϕ(z∣x)[log⁡pθ(x∣z)]−DKL ⁣(qϕ(z∣x) ∥ p(z))LVAE(x)=−ELBO(x)\boldsymbol{\begin{aligned}\mathrm{ELBO}(x)&=\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)]\\&\quad-D_{\mathrm{KL}}\!\left(q_\phi(z\mid x)\,\|\,p(z)\right)\\\mathcal{L}_{\mathrm{VAE}}(x)&=-\mathrm{ELBO}(x)\end{aligned}}

Maximize the evidence lower bound (ELBO), or equivalently minimize its negative as the VAE loss. qφ(z|x) is the approximate posterior, p(z) the prior, and pθ(x|z) the decoder likelihood.

  • Denoising AE — reconstructs clean input from corrupted input.
  • Sparse AE — enforces sparse activations.
  • Variational AE — learns a probabilistic latent space and can generate new samples.

Uses

  • Dimensionality reduction
  • Anomaly detection
  • Denoising
  • Generative modelling

Interview questions & answers

  1. Q1. Why might an autoencoder need a bottleneck or regularization?

    Answer: Without constraints, an expressive autoencoder can learn an identity mapping instead of useful representations. A lower-dimensional bottleneck, sparsity constraints or a denoising objective encourages it to capture meaningful structure rather than simply copy inputs.

  2. Q2. How does a variational autoencoder differ from a standard autoencoder?

    Answer: A standard autoencoder typically maps an input to a deterministic code. A VAE learns an approximate posterior distribution over latent variables and balances reconstruction likelihood with a KL penalty toward a prior. Sampling from the prior and decoding supports generation.

  3. Q3. What is the reparameterization trick?

    Answer: For a Gaussian latent posterior, sample standard Gaussian noise and construct the latent value using the learned mean and standard deviation. This separates randomness from the learned parameters, allowing gradients to flow through the sampled latent representation during training.

← Previous

LSTM & GRU

Next →

Generative Adversarial Networks