Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations

Sparse Gaussian processes summarize a dataset through function values at a smaller set of inducing inputs. Their locations determine which parts of the input space the approximation represents. We study what changes when these locations have a posterior distribution, alongside the inducing values and kernel hyperparameters.

Inducing locations as random variables

Let $Z=\{z_m\}_{m=1}^M$ denote the inducing locations, $u_m=f(z_m)$ the corresponding function values, and $\theta$ the kernel hyperparameters. A common variational treatment approximates the posterior over $u$ while optimizing $Z$ and $\theta$. This leaves the predictive distribution conditional on a single choice of locations and kernel settings.

We place priors on $Z$ and $\theta$ and infer them jointly with $u$. For training inputs $X$ and observations $y$, the model has joint distribution

$$p(\theta,Z,u,f,y\mid X) = p(\theta)\,p(Z)\,p(u\mid Z,\theta)\,p(f\mid u,X,Z,\theta)\,p(y\mid f).$$

Here $f$ collects the latent function values at the training inputs. Integrating over $Z$ and $\theta$ lets predictions reflect uncertainty in the representation and the covariance function.

Sampling the sparse posterior

The likelihood approximation matters when carrying out this inference. We revisit the fully independent training conditional (FITC), which factorizes the training conditionals given the inducing variables. This produces an objective whose data terms can be estimated with minibatches.

We use stochastic gradient Hamiltonian Monte Carlo (SGHMC) to sample the inducing values, locations, and kernel hyperparameters. Predictions average over the retained posterior samples. The construction also extends to deep Gaussian processes, where several GP layers are composed.

This approach retains an inducing-variable approximation to the original model. Its computational cost and posterior accuracy depend on the number of inducing variables and the sampling procedure.

Regression and classification

We evaluate the method on regression and classification tasks, with ablations that separate the effects of the inference objective, the inducing-location prior, and the variables included in sampling.

Jointly sampling $u$, $\theta$, and $Z$ improves predictive density in several of the regression comparisons. Experiments with deep GPs examine the same choice across multiple depths. Averaging over inducing locations gives the approximation a way to represent uncertainty about where its finite set of function values should be placed.