Deep learning is finding applications in a wide variety of fields and is also known to perform better than humans in some of the tasks such as image classification. Although the predictions of a neural network are highly accurate, in most scenarios, we do not have a clue as to why and how exactly the neural network is making a certain prediction. More importantly, we do not even know how confident is the neural network about a particular prediction it makes. Lack of this understanding restricts us from using deep learning theory to take critical decisions in very sensitive applications such as health care and military. Hence recently there has been lot of study on understanding how to measure the confidence of a neural network about its predictions. Recent literature has addressed this problem by introducing a metric called "Uncertainty", which is a measure of how uncertain the neural network is about its predictions. This blog post surveys some of the recent advances on this topic.

Basic idea

Roughly speaking, the key idea to obtain uncertainty estimates can be explained as follows. Instead of training a single neural network and taking its predictions for granted, train an ensemble of neural networks, all of which you think can model the given data reasonably well.  Now for a given input, get predictions from each of these neural networks, and infer the uncertainty as follows:

  1. If all the neural networks, or at least most of them are predicting the same output, then you can treat that the ensemble of networks is certain about the prediction.
  2. If the neural networks significantly differ in their predictions, then you infer the prediction to be uncertain.

Formalising the idea

The following are some approaches in the literature that formalise the above idea to obtain an ensemble of neural networks, and thereby compute uncertainty:

  1. Modelling the weights of neural networks as random variables
  2. Monte Carlo dropout technique that obtains different realisations of the networks by dropping different neurons
  3. Train a neural network that directly outputs a distribution on the class distributions, which can be used to measure uncertainty

1. Weights as random variables

Treat each weight of the neural network as a random variable instead of treating it as a fixed number. Once we learn the probability distributions of the weights, we can obtain as many realisations of the neural network as we want by sampling multiple times from those weight distributions. In particular, we get an ensemble of neural networks, and use them to obtain a measure of uncertainty.

Bayes by Backprop: Models each weight of the network as an independent Gaussian random variable and introduces a back propagation method to learn the parameters of the Gaussian distributions. Hence, for each weight there will be a mean parameter and a variance parameter to model the Gaussian distribution, i.e., the numbers of parameters will be double compared to usual fixed weight neural network, where a weight has only parameter associated with it.

To learn the parameters of the Gaussian distributions, the authors try to fit a Gaussian distribution that best approximates the posterior distribution of the weights given the training data. In more technical terms, the authors find a Gaussian distribution which has the minimum KL divergence (this is the cost function being used) with respect to the posterior distribution of weights. Further, inspired by the usual back propagation algorithm, the authors propose a training algorithm that minimises the above cost function.

The following reference post contains some more specific details of this paper, and this github repo has a Pytorch implementation.

2. Monte Carlo dropout based ensembles

A single neural network is trained from the data. During the test time, an ensemble of networks is obtained by randomly dropping some of the neurons. In particular, each neuron is either dropped or retained independently using a Bernoulli distribution.

The following reference post contains some more specific details of this paper

3. Directly output a distribution on class distributions

A conventional neural network outputs a categorical distribution on the set of classes. For example, if there are four possible classes in our classification framework, the network outputs four probabilities corresponding to our four classes. Different from the conventional neural network, this paper proposes a neural network called Dirichlet Prior Network (DPN) which outputs a Dirichlet distribution on the set of categorical distributions, instead of outputting one specific categorical distribution.

The Dirichlet distribution can be used to measure uncertainty, and more importantly, we can understand the type of uncertainty too. In particular, we can understand whether it is an in-domain uncertainty which occurs due to presence of two or more classes which are very similar, or it is an out-of-domain uncertainty, which is caused when an input image is given from a distribution which is entirely different from the training distribution. For example, if we train a network on Fashion MNIST data set and input a image from MNIST dataset, the neural network cannot make certain predictions, because it is not trained for the MNIST dataset. Such uncertainty is referred to as out-of-domain uncertainty.  

Different types of Uncertainties and their illustration of their of corresponding Dirichlet distributions

The above figure illustrates Dirichlet distributions on a simplex, where each point in the simplex corresponds to a categorical distribution of 3 classes. We can understand these figures using the analogy of ensemble of neural networks.

(a) Confident Prediction: Figure 2 (a) corresponds to a confident prediction, where the Dirichlet distribution is sharp around a particular class. If we look at the analogy of having an ensemble of neural networks, this can be interpreted as, "All the neural networks in the ensemble predict the same class with a high probability".

(b) In-domain uncertainty: Figure 2 (b) corresponds to in-domain uncertainty or data uncertainty. Here Dirichlet distribution is sharp around a point on the simplex which assigns equal probabilities to all the three classes. If we look at the analogy of having an ensemble of neural networks, this can be interpreted as, "All the neural networks in the ensemble predict a categorical distribution which is uniform across all the possible classes". Note that in general, the categorical distribution need not be uniform across all the possible classes. As long as two or more classes have significantly large probabilities, it will result in in-domain uncertainty.

(c) Out-of-domain uncertainty: Figure 2 (c) corresponds to out-of-domain uncertainty. Here Dirichlet distribution is flat across all the possible categorical distributions. If we look at the analogy of having an ensemble of neural networks, this can be interpreted as, "Different neural networks in the ensemble predict significantly different categorical distributions". Since the neural network is not trained on this particular sample, each neural network comes up with its own prediction which may not agree with each other, and hence result in a flat Dirichlet distribution.

Training of Dirichlet Prior Network

The DPN is explicitly trained with in-domain data and some out-of-domain data. The loss function of the DPN is designed in such a way that the network learns to predict a sharp Dirichlet distributions for in-domain data and a flat Dirichlet distribution for an out-of-domain data. More specific details can be found in this paper.