Walsh-Hadamard Variational Inference for Bayesian Deep Learning

Large neural networks create a difficult variational inference problem: storing a distribution over every weight is expensive, and the accumulated KL penalty can dominate the data-fitting objective. Walsh-Hadamard Variational Inference (WHVI) addresses these costs through a structured distribution over weight matrices.

A structured stochastic weight matrix

For a square layer of width $D$, WHVI represents a sampled weight matrix as

$$W=S_1H\operatorname{diag}(g)HS_2, \qquad g\sim q(g),$$

where $H$ is a Walsh-Hadamard matrix, $S_1$ and $S_2$ are learned diagonal matrices, and $g$ contains the stochastic coordinates. The construction adapts structured random-matrix factorizations used in kernel approximations to posterior inference.

With a diagonal Gaussian distribution over $g$, the stochastic representation uses $O(D)$ parameters. Multiplication by the dense matrix $W$ is implemented through diagonal operations and fast Walsh-Hadamard transforms, giving $O(D\log D)$ cost. An unrestricted dense layer instead requires $O(D^2)$ storage and multiplication.

Correlations from shared coordinates

The entries of $W$ depend on the same random vector $g$. Consequently, even independent coordinates in $g$ induce correlations between network weights. For Gaussian $q(g)$, the induced weight distribution is Gaussian with a structured covariance.

The representation restricts the weight distribution to a structured family. Its reduced parameter count changes the variational approximation and the regularization associated with its stochastic coordinates. The learned diagonal factors control how those coordinates are expressed across the weight matrix.

Training with fast transforms

We derive weight and local reparameterizations for stochastic-gradient optimization of the evidence lower bound. Local reparameterization samples layer outputs rather than a complete weight realization, providing another way to estimate the training objective.

Padding, concatenation, and reshaping extend the factorization to rectangular matrices and parameter vectors. Normalizing flows over $g$ provide a richer, non-Gaussian family while retaining the structured matrix multiplication.

Predictive quality and computational cost

We evaluate regression, image classification, and convolutional architectures, comparing predictive error and log-likelihood, parameter counts, and computational costs with mean-field and other approximate Bayesian methods. Comparisons with alternative diagonal-transform factorizations and tensor factorizations isolate the effect of the chosen weight representation.

In these comparisons, the structured family offers competitive predictive performance with reduced inference costs. Its computational advantage follows from the shared stochastic coordinates and fast transforms. The same factorization determines which posterior means and dependencies the approximation can express.