Scaling Laws for Uncertainty in Deep Learning
- Uncertainty Quantification
- Bayesian Deep Learning
- Deep Learning Theory
Predictive uncertainty follows empirical scaling trends as training data grows, with rates that depend on the model, inference method, and optimization setup.
Paper arXiv preprint 2025
- KAUST
- EURECOM
- Aalto University
Predictive uncertainty often follows a power law as training sets grow. We measure this behavior across vision and language experiments to ask how much more data would make an ensemble's predictions agree. The observed exponents depend on inference and training. Several observed contraction rates are slower than the classical rate of identifiable Bayesian models. Ensemble disagreement can therefore remain substantial over the tested data range.
Measuring predictive disagreement
For ensemble members or stochastic parameter draws , we compute total uncertainty (TU), aleatoric uncertainty (AU), and epistemic uncertainty (EU):
Here is predictive entropy. AU averages the individual predictive entropies, while EU measures disagreement between predictive distributions. We obtain these distributions through Monte Carlo dropout, Gaussian posterior approximations, Markov chain Monte Carlo (MCMC), or independently trained deep ensembles. Partially stochastic networks restrict inference to selected layers.
We average each metric over a fixed test set and fit straight lines in logarithmic coordinates:
A negative exponent describes contraction; its magnitude determines the reduction obtained from multiplying the training size . The fit characterizes the complete training and uncertainty-estimation procedure.

More data reduces uncertainty at different rates
For ResNet-18 and ResNet-50 on CIFAR-10 and CIFAR-100, we train with SGD for 400 epochs and dropout probabilities and . Each estimate averages ten independent combinations of data subsampling and initialization, using 25%, 50%, 75%, or 100% of the training set. TU, AU, and EU decrease, with slower EU contraction on CIFAR-100 in these configurations.

For ResNets trained on CIFAR-10 and evaluated on CIFAR-10-C corruptions, the fitted EU trends are much flatter. These measurements average all corruption types and severity levels 1–5, locating the result in a specific distribution-shift benchmark.

Optimization and capacity change the scaling law
Training ResNets with sharpness-aware minimization (SAM) and dropout reverses the EU trend on CIFAR-10: disagreement increases while TU and AU decrease. We hypothesize that flatter solutions preserve functional diversity. This mechanism remains an interpretation of the experiment.

Model-size experiments also depend on inference. Across ResNets with 11.7–60.2 million parameters trained on full CIFAR-10, EU increases with Improved Variational Online Newton (IVON) variational inference and decreases slightly with dropout. The two procedures use their respective training configurations, including 200 and 400 epochs, so the comparison concerns complete procedures.

Language tasks and uncertainty extrapolation
For a two-layer GPT-2 learning addition modulo 97, we train on 5–90% of possible examples with dropout . Clearer scaling emerges after extensive training: the displayed results use 10,000 epochs and three seeds. Pretrained Phi-2 with a Laplace approximation over low-rank adaptation (LoRA) parameters instead yields approximately flat uncertainty across downstream fine-tuning subsets.
We also examine uncertainty around delayed generalization in modular-arithmetic in-context learning.

For a two-hidden-layer Bayesian MLP on two moons, Hamiltonian Monte Carlo (HMC) supplies posterior samples. Fitting EU on smaller training sets predicts the larger-size observations up to one million examples. AU plateaus, and TU approaches this floor, making component-wise extrapolation useful.

Inverse-data contraction in Bayesian linear models
With fixed features , Gaussian observation noise of variance , and a Gaussian weight prior of covariance , Bayesian linear regression gives the posterior covariance and predictive differential entropy
Here is the training design matrix. For independent samples with a nonsingular feature covariance, posterior variance contracts at order and predictive entropy approaches the Gaussian noise entropy. Our linearized Laplace study recovers this behavior when accumulated data curvature dominates prior precision.

Bayesian linear models provide a theoretical connection between uncertainty contraction and generalization error. In the deep-network experiments, the measured exponents characterize a specified optimizer, inference approximation, and training regime. Entropy-based AU and EU also depend on the quality of the predictive approximation, so interpreting their rates requires accounting for the full estimation procedure.