Simone Rossi
← All publications

Handbook of Bayesian Deep Learning

  • Bayesian Deep Learning
  • Uncertainty Quantification
  • Variational Inference
  • Gaussian Processes

A collective reference on Bayesian deep learning, covering posterior inference, priors, scalable computation, uncertainty quantification, and applications.

Book Zenodo 2026

106 authors, including Simone Rossi

Claudio Agostinelli1, Laurence Aitchison2, Emanuel Aldea3, Richard Allmendinger4, Sebastian Ament5, Ben Anson2, Julyan Arbel6, Eytan Bakshy5, Max Balandat5, Nicola Bariletto7, Freddie Bickford Smith8, Battista Biggio9, Michele Caprio10, Neil Chada11, Wenlin Chen12, Wenlong Chen13, Nathaël Da Costa14, Nico Daheim15, Andreas Damianou16, Michael Deistler14, David Dunson17, Nikita Durasov18, Maurizio Filippone19, Lucia Filippozzi1, Gergely Flamich13, Vincent Fortuin20, Gianni Franchi21, Giorgio Fumera9, Yarin Gal8, Wenbo Gong22, Yuqi Gu23, Cheng Guo4, Nick Hauptvogel24, Jiajun He12, Philipp Hennig14, José Miguel Hernández-Lobato12, Nhat Ho7, Clara Hoffmann25, Aliaksandr Hubin26, Christian Igel24, Ajay Jasra27, Matt Jones28, Nadja Klein25, Olivier Laurent3, Kody J. H. Law4, Emanuele Ledda9, Bolian Li29, Yingzhen Li13, Xinzhu Liang4, Jakob H Macke30, Stephan Mandt31, Luckeciano Carvalho Melo8, Benjamin Kurt Miller32, Edward Milsom2, Bruno Mlodozeniec12, Bálint Mucsányi14, Samuel Müller5, Kevin Murphy33, Eric Nalisnick34, Marcos Negre Saura4, Christopher Nemeth35, Huy Nguyen7, Khai Nguyen36, Shreyas Padhy12, Konstantina Palla16, Wei Pan4, Theodore Papamarkou37, Dhruvesh Patel38, Alina Peluso39, Simon Pepin Lehalleur40, Konstantinos Pitas6, Patrick Pynadath29, Tom Rainforth8, Maxime Robeyns2, Fabio Roli41, Breeshey Roskams-Hieter42, Simone Rossi43, Tim G. J. Rudner44, David Rügamer45, Dino Sejdinovic46, Yevgeny Seldin24, Torben Sell42, Louis Sharrock47, Sumeetpal Singh48, Joanna Sliwa14, Emanuel Sommer45, Qifan Song29, Ba-Hien Tran43, Richard E. Turner12, Tycho F. A. van der Ouderaa13, Mark van der Wilk8, Mariia Vladimirova49, Sara Wade42, Xi Wang34, Tobias Weber14, Susan Wei50, Christopher K. Wikle51, Andrew Gordon Wilson52, Lisa Wimmer53, Adam Yang2, Yibo Yang54, Myungsoo Yoo55, Cheng Zhang56, Likun Zhang51, Ruqi Zhang29, Yuren Zhou17

  1. University of Trento
  2. University of Bristol
  3. Université Paris-Saclay
  4. The University of Manchester
  5. Meta
  6. Centre Inria de l'Université Grenoble Alpes
  7. University of Texas at Austin
  8. University of Oxford
  9. University of Cagliari
  10. University of Warwick
  11. City University of Hong Kong
  12. University of Cambridge
  13. Imperial College London
  14. University of Tübingen, Tübingen AI Center
  15. TU Darmstadt
  16. Spotify
  17. Duke University
  18. École polytechnique fédérale de Lausanne
  19. King Abdullah University of Science and Technology
  20. University of Technology Nuremberg
  21. AMIAD
  22. Microsoft
  23. Columbia University
  24. University of Copenhagen
  25. Karlsruhe Institute of Technology
  26. NMBU and University of Oslo
  27. Chinese University of Hong Kong, Shenzhen
  28. U. Colorado, Boulder
  29. Purdue University
  30. ELLIS Institute Tübingen, University of Tübingen, Tübingen AI Center
  31. University of California, Irvine
  32. University of Amsterdam
  33. Google DeepMind
  34. Johns Hopkins University
  35. Lancaster University
  36. Texas A&M University
  37. PolyShape
  38. UMass Amherst
  39. Oak Ridge National Laboratory
  40. University of Duisburg-Essen
  41. University of Genova
  42. University of Edinburgh
  43. EURECOM
  44. University of Toronto
  45. LMU Munich, Munich Center for Machine Learning
  46. Adelaide University
  47. University College London
  48. University of Wollongong
  49. Criteo AI Lab
  50. Monash University
  51. University of Missouri
  52. New York University
  53. LMU Munich
  54. Chan Zuckerberg Initiative
  55. University of Texas-Austin
  56. European Laboratory for Learning and Intelligent Systems (ELLIS)
PublicationPDF

Bayesian deep learning represents uncertainty through a distribution over neural networks and propagates it into predictions. The posterior is usually computationally inaccessible, so practical methods approximate the distribution, its predictive average, or both. Sampling, variational inference, Laplace approximations, ensembles, and kernel methods make different computational choices. Their predictive behavior depends on posterior geometry, function priors, and the available computation.

The posterior predictive

Let θ∈Θ\theta\in\Theta denote network parameters in parameter space Θ\Theta, D\mathcal D the observed training data, and (x∗,y∗)(x^*,y^*) a new input and possible output. The predictive distribution averages over parameter uncertainty:

p^(y∗∣x∗,D)=∫Θp(y∗∣x∗,θ) p(θ∣D) dθ.\hat p(y^*\mid x^*,\mathcal D) =\int_\Theta p(y^*\mid x^*,\theta)\,p(\theta\mid\mathcal D)\,d\theta.

The likelihood describes outputs conditional on a model; the posterior weights plausible parameters after observing data. Averaging predictions propagates parameter uncertainty into the output. This integral is usually intractable for neural networks, creating a common computational problem for approximate inference.

Monte Carlo methods estimate such expectations using samples. Markov chain Monte Carlo (MCMC) uses an ergodic chain with the posterior as its invariant distribution; mixing and effective sample size determine how much information a finite run provides. Kernel methods offer a complementary function-space formulation, with Gaussian-process regression supplying an analytically tractable reference under Gaussian observation noise.

Tractable posterior approximations

Variational inference (VI) chooses a distribution q(θ;λ)q(\theta;\lambda) with parameters λ\lambda. We fit it by maximizing the evidence lower bound:

L(λ)=Eq(θ;λ)[log⁡p(D∣θ)]−DKL ⁣(q(θ;λ)∥p(θ)).\mathcal L(\lambda) =\mathbb E_{q(\theta;\lambda)}[\log p(\mathcal D\mid\theta)] -D_{\mathrm{KL}}\!\left(q(\theta;\lambda)\Vert p(\theta)\right).

The expected log-likelihood rewards data fit; the KL term penalizes departure from the prior. When the terms are finite, maximizing this objective minimizes the reverse KL divergence to the posterior within the selected family. The family therefore determines which dependencies and modes can be represented.

For example, Walsh-Hadamard variational inference uses structured transforms to represent correlated weight distributions.

Laplace approximation instead expands the negative log-posterior energy locally around a trained mode θ∗\theta^*. The construction uses its Hessian Ψ(θ∗)\Psi(\theta^*) at a stationary local minimum of a twice-differentiable energy. For positive-definite curvature,

p(θ∣D)≈N ⁣(θ;θ∗,Ψ−1(θ∗)).p(\theta\mid\mathcal D) \approx\mathcal N\!\left(\theta;\theta^*,\Psi^{-1}(\theta^*)\right).

This turns a fitted network into a local Gaussian approximation through differentiation and linear algebra. Structured curvature and matrix-free solvers control its cost, while its local construction limits the posterior geometry it captures.

Energy contours and Gaussian uncertainty ellipses around the fitted parameters of a two-parameter ReLU network
The Gaussian follows local curvature around the black optimum but differs from the global energy contours. The example makes the strength and limitation of a local approximation visible.

Computation across distinct solutions

Deep ensembles average predictions from independently trained networks. Their members represent optimized solutions, with equal predictive weights. Coverage of functionally distinct basins can complement within-basin uncertainty approximations such as variational inference or Laplace.

Multimodal posterior, parameter-dependent predictions, and approximation discrepancy for deep ensembles, VI, and MultiSWAG
Conceptual comparison adapted from Wilson and Izmailov. Here predictions vary more between basins than within them. Representing an unexplored basin can therefore improve a finite predictive approximation.

The empirical comparisons use high-fidelity Hamiltonian Monte Carlo (HMC) predictions as their reference. They distinguish predictive fidelity from the locations of parameter samples and show why evaluation must specify both the target and the available computation.

One-dimensional regression predictions from an HMC reference, deep ensembles, and factorized VI, with Wasserstein discrepancy curves
Regression comparison adapted from Wilson and Izmailov. On this regression example, 50 independently trained networks approach the ten-chain HMC reference more closely than factorized VI. The discrepancy curves expose VI's overconfidence between data clusters.
Agreement with HMC predictions on CIFAR-10 and CIFAR-10-C for SGD, VI, SGLD, deep ensembles, SGHMC-CLR, and HMC
Classification comparison adapted from Izmailov and colleagues. With ResNet-20-FRN, ensembles agree with the HMC reference more often than VI; SGHMC-CLR is closer in both panels. The metric measures agreement with HMC's top-one predictions.

Function priors and parameter redundancy

Independent Gaussian weight priors induce structured distributions over functions. Their behavior depends jointly on variance, architecture, and activation, so prior-predictive checks belong before posterior fitting.

Our functional-prior study fits neural-network weight priors to specified Gaussian-process function distributions.

Gaussian weight priors with three variances and the resulting prior-predictive functions in a small sigmoid network
Changing variance from 0.02 to 1 to 50 changes the output distribution of a network with two hidden layers of two sigmoid units. A familiar weight prior still encodes consequential assumptions about functions.

Function-preserving transformations also generate equivalent parameterizations. Hidden-unit permutations preserve the posterior under exchangeable priors. ReLU rescaling preserves the likelihood; posterior symmetry additionally requires invariance of the prior measure, including its Jacobian factor. This distinction determines which equivalent parameterizations can be aligned or traversed while preserving the posterior.

Our permutation-symmetry study aligns entire Gaussian variational posteriors through hidden-unit permutations.

Parameter-space symmetry orbit and the additional prior-measure condition required for posterior invariance
The function and likelihood remain fixed along a symmetry orbit. Checking the prior measure identifies which transformations are posterior symmetries, clarifying how redundant parameterizations affect approximate inference.