← All publications

Scaling Laws for Uncertainty in Deep Learning

  • Uncertainty Quantification
  • Bayesian Deep Learning
  • Deep Learning Theory

Predictive uncertainty follows empirical scaling trends as training data grows, with rates that depend on the model, inference method, and optimization setup.

Paper arXiv preprint 2025

Mattia Rosso1, Simone Rossi2, Giulio Franzese2, Markus Heinonen3, Maurizio Filippone1

  1. KAUST
  2. EURECOM
  3. Aalto University

Predictive uncertainty often follows a power law as training sets grow. We measure this behavior across vision and language experiments to ask how much more data would make an ensemble's predictions agree. The observed exponents depend on inference and training. Several observed contraction rates are slower than the classical 1/N1/N rate of identifiable Bayesian models. Ensemble disagreement can therefore remain substantial over the tested data range.

Measuring predictive disagreement

For KK ensemble members or stochastic parameter draws θ(k)\theta^{(k)}, we compute total uncertainty (TU), aleatoric uncertainty (AU), and epistemic uncertainty (EU):

TU(x)=H ⁣[1K∑k=1Kp(y∣x,θ(k))],AU(x)=1K∑k=1KH[p(y∣x,θ(k))],EU(x)=TU(x)−AU(x).\begin{aligned} \mathrm{TU}(x)&=\mathbb H\!\left[\frac1K\sum_{k=1}^{K}p(y\mid x,\theta^{(k)})\right],\\ \mathrm{AU}(x)&=\frac1K\sum_{k=1}^{K}\mathbb H[p(y\mid x,\theta^{(k)})],\\ \mathrm{EU}(x)&=\mathrm{TU}(x)-\mathrm{AU}(x). \end{aligned}

Here H\mathbb H is predictive entropy. AU averages the individual predictive entropies, while EU measures disagreement between predictive distributions. We obtain these distributions through Monte Carlo dropout, Gaussian posterior approximations, Markov chain Monte Carlo (MCMC), or independently trained deep ensembles. Partially stochastic networks restrict inference to selected layers.

We average each metric over a fixed test set and fit straight lines in logarithmic coordinates:

U‾(N)=1Ntest∑n=1NtestU(xn)∝NγU,U∈{TU,AU,EU}.\overline{\mathrm U}(N)=\frac1{N_{\mathrm{test}}}\sum_{n=1}^{N_{\mathrm{test}}}\mathrm U(x_n) \propto N^{\gamma_{\mathrm U}}, \qquad \mathrm U\in\{\mathrm{TU},\mathrm{AU},\mathrm{EU}\}.

A negative exponent describes contraction; its magnitude determines the reduction obtained from multiplying the training size NN. The fit characterizes the complete training and uncertainty-estimation procedure.

Epistemic uncertainty of ResNet-18 on CIFAR-10 under MCMC, Monte Carlo dropout, and deep ensembles
On CIFAR-10, ResNet-18 EU has fitted exponents of −0.44-0.44, −0.36-0.36, and −0.80-0.80 for MCMC, dropout, and deep ensembles. The shared architecture produces different contraction rates under different inference procedures.

More data reduces uncertainty at different rates

For ResNet-18 and ResNet-50 on CIFAR-10 and CIFAR-100, we train with SGD for 400 epochs and dropout probabilities 0.20.2 and 0.50.5. Each estimate averages ten independent combinations of data subsampling and initialization, using 25%, 50%, 75%, or 100% of the training set. TU, AU, and EU decrease, with slower EU contraction on CIFAR-100 in these configurations.

Total, aleatoric, and epistemic uncertainty for two ResNets and dropout rates on CIFAR-10 and CIFAR-100
The two rows compare CIFAR-10 and CIFAR-100, with fitted exponents and repeated-run curves. AU also decreases as data grows, showing that finite-data conditional entropy estimates evolve alongside ensemble disagreement.

For ResNets trained on CIFAR-10 and evaluated on CIFAR-10-C corruptions, the fitted EU trends are much flatter. These measurements average all corruption types and severity levels 1–5, locating the result in a specific distribution-shift benchmark.

Uncertainty of CIFAR-10-trained ResNets evaluated on CIFAR-10-C at dropout probabilities 0.2 and 0.5
Panels (a) and (b) use dropout probabilities 0.20.2 and 0.50.5. The nearly flat EU curves show that ensemble disagreement persists as clean training data grows.

Optimization and capacity change the scaling law

Training ResNets with sharpness-aware minimization (SAM) and dropout reverses the EU trend on CIFAR-10: disagreement increases while TU and AU decrease. We hypothesize that flatter solutions preserve functional diversity. This mechanism remains an interpretation of the experiment.

ResNet-18, ResNet-34, and ResNet-50 uncertainty under SGD with SAM and dropout
With dropout p=0.5p=0.5, EU exponents are positive for all three ResNets. The comparison exposes how the optimizer changes uncertainty scaling even as the other entropy measures fall.

Model-size experiments also depend on inference. Across ResNets with 11.7–60.2 million parameters trained on full CIFAR-10, EU increases with Improved Variational Online Newton (IVON) variational inference and decreases slightly with dropout. The two procedures use their respective training configurations, including 200 and 400 epochs, so the comparison concerns complete procedures.

Uncertainty versus ResNet parameter count for IVON and Monte Carlo dropout
IVON gives γEU=0.33\gamma_{\mathrm{EU}}=0.33; dropout gives −0.11-0.11. Parameter count alone therefore leaves important variation in predictive diversity unexplained.

Language tasks and uncertainty extrapolation

For a two-layer GPT-2 learning addition modulo 97, we train on 5–90% of possible examples with dropout 0.10.1. Clearer scaling emerges after extensive training: the displayed results use 10,000 epochs and three seeds. Pretrained Phi-2 with a Laplace approximation over low-rank adaptation (LoRA) parameters instead yields approximately flat uncertainty across downstream fine-tuning subsets.

We also examine uncertainty around delayed generalization in modular-arithmetic in-context learning.

GPT-2 predictive uncertainty versus the number of modular-addition examples
The modular-addition experiment gives EU exponent −2.66-2.66 after extensive training. The final subset departs from the fitted decay, illustrating the sensitivity to convergence.

For a two-hidden-layer Bayesian MLP on two moons, Hamiltonian Monte Carlo (HMC) supplies posterior samples. Fitting EU on smaller training sets predicts the larger-size observations up to one million examples. AU plateaus, and TU approaches this floor, making component-wise extrapolation useful.

Measured and extrapolated uncertainty for a Bayesian MLP on two moons
Circles mark the fitting range, squares the larger-size observations, and crosses the extrapolated EU. The fitted EU exponent is −0.79-0.79 (R2=0.90R^2=0.90); TU and AU flatten as irreducible uncertainty dominates.

Inverse-data contraction in Bayesian linear models

With fixed features ϕ(x)\phi(x), Gaussian observation noise of variance σ2\sigma^2, and a Gaussian weight prior of covariance S0S_0, Bayesian linear regression gives the posterior covariance and predictive differential entropy

SN=(σ−2Φ⊤Φ+S0−1)−1,TU(x)=12log⁡ ⁣(2πe[σ2+ϕ(x)⊤SNϕ(x)]).\begin{aligned} S_N&=(\sigma^{-2}\Phi^\top\Phi+S_0^{-1})^{-1},\\ \mathrm{TU}(x)&=\tfrac12\log\!\left(2\pi e[\sigma^2+\phi(x)^\top S_N\phi(x)]\right). \end{aligned}

Here Φ\Phi is the training design matrix. For independent samples with a nonsingular feature covariance, posterior variance contracts at order 1/N1/N and predictive entropy approaches the Gaussian noise entropy. Our linearized Laplace study recovers this behavior when accumulated data curvature dominates prior precision.

Hessian growth and logit-space and probability-space uncertainty under linearized Laplace inference
Growing data curvature accompanies approximately inverse-data uncertainty decay for several prior precisions. The linearized setting supplies a tractable reference for interpreting slower deep-network trends.

Bayesian linear models provide a theoretical connection between uncertainty contraction and generalization error. In the deep-network experiments, the measured exponents characterize a specified optimizer, inference approximation, and training regime. Entropy-based AU and EU also depend on the quality of the predictive approximation, so interpreting their rates requires accounting for the full estimation procedure.