How Good is the Bayes Posterior in Deep Neural Networks Really?
Florian Wenzel,u00a0Kevin Roth,u00a0Bastiaan Veeling,u00a0Jakub Swiatkowski,u00a0Linh Tran,u00a0Stephan Mandt,u00a0Jasper Snoek,u00a0Tim Salimans,u00a0Rodolphe Jenatton,u00a0Sebastian Nowozin
During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there areu2014as of early 2020u2014no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions when compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a u201ccold posterioru201d that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of cold posteriors.
