Paper: https://arxiv.org/pdf/1312.6114.pdf

Code: https://github.com/bhpfelix/Variational-Autoencoder-PyTorch (Unofficial PyTorch implementation)

Problem Definition

To estimate a lower dimensional representation of a given dataset via an encoder-decoder like neural network.

Architecture

architecture

The encoder compresses the data into a lower dimensional latent space (z). The decoder reconstructs the data given the hidden representation.

Key Points

  • The encoder and decoders are neural networks and z is a sampled vector from a distribution.
  • As we cannot back propagate gradients through a sampling vector, a reparameterization trick was introduced.
  • The representation of z is changed as z = mean + (standard deviation * epsilon). Here epsilon is sampled from a standard gaussian distribution with mean = 0 and standard deviation = 1. So the mean and standard deviation of z are learned and random part of z is handled by epsilon which is always sampled from standard gaussian.
    reparam
  • The loss function here consists of two parts a) the construction loss and regularization term. a) The reconstruction loss ensures the latent vector learn useful representation of the data. b) The regularization term ensures that the latent vector is close to standard normal distribution.
    loss_func