← All publications

Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation

  • Diffusion Models
  • Computer Vision

Optimal Gaussian Diffusion shortens trajectory sampling, while Estimated Clean Manifold guidance applies control gradients directly to estimated clean trajectories.

Paper ECCV 2024

Yixiao Wang1, Chen Tang1, Lingfeng Sun1, Simone Rossi2, Yichen Xie1, Chensheng Peng1, Thomas Hannagan3, Stefano Sabatini3, Nicola Poerio3, Masayoshi Tomizuka1, Wei Zhan1

  1. UC Berkeley
  2. EURECOM
  3. Stellantis

Joint trajectory prediction must represent several road users and the different ways they could interact. Diffusion models can generate these joint futures and steer them toward requested routes or speeds, but repeated denoising and guidance make sampling expensive. We introduce Optimal Gaussian Diffusion (OGD), which starts sampling from an informative Gaussian at an earlier diffusion time, and Estimated Clean Manifold (ECM) guidance, which applies control gradients directly to estimated clean trajectories. On Argoverse 2, the combination improves goal satisfaction while reducing sampling steps and guidance overhead.

Starting closer to the trajectory distribution

Predicting each vehicle independently can combine individually plausible motions into an inconsistent scene. We therefore model the joint future x0=[x0,1,…,x0,n]\mathbf{x}_0=[\mathbf{x}_{0,1},\ldots,\mathbf{x}_{0,n}], conditioned on observed motion and map context. Throughout the equations, we omit this conditioning for clarity.

Vanilla diffusion begins reverse sampling from a non-informative Gaussian after enough forward diffusion to erase the data structure. OGD chooses an informative Gaussian for a smaller starting time TT:

qϕ(T)=N(μ,Σ),min⁡μ,Σ,Σp  KL⁡[pT∥qϕ(T)]subject to ∣Σp∣=1.\begin{aligned} q_\phi(T)&=\mathcal{N}(\boldsymbol\mu,\boldsymbol\Sigma),\\ \min_{\boldsymbol\mu,\boldsymbol\Sigma,\boldsymbol\Sigma_p}\;& \operatorname{KL}[p_T\Vert q_\phi(T)] \quad\text{subject to }|\boldsymbol\Sigma_p|=1. \end{aligned}

Here pTp_T is the noisy trajectory distribution, Σp\boldsymbol\Sigma_p is the perturbation-kernel covariance, and ∣⋅∣|\cdot| denotes the determinant. For a variance-preserving Gaussian perturbation kernel, the minimizing prior and kernel parameters can be expressed in terms of the data mean and covariance. For joint prediction, the practical construction restricts covariance to blocks for individual agents and estimates these statistics with a pretrained marginal predictor. The denoiser learns the remaining joint dependence, and TT can be tuned during inference.

Our study of diffusion times examines the broader trade-off between score approximation and reverse-process initialization.

OGD constructs an informative Gaussian from trajectory statistics and starts reverse diffusion at a smaller time
Marginal means and variances determine an informative starting distribution. The shorter red reverse path illustrates how OGD reduces the diffusion horizon while retaining scene information.

Our implementation uses a frozen QCNet context encoder, attention over agents, maps and marginal predictions, and a learned linear mapping from 120 trajectory coordinates to ten latent dimensions. This representation also regularizes the scale of latent components.

Learned linear trajectory modes, PCA modes, and their component standard deviations
The learned linear mapping (LM) produces more balanced component scales than principal component analysis (PCA). These balanced scales are intended to stabilize latent diffusion.

Applying control gradients to clean estimates

Control costs naturally describe clean trajectories, such as the distance to a requested position. Conventional score-function guidance differentiates that cost through the denoising network. ECM instead treats realism and control as a hierarchical optimization:

min⁡x0[−log⁡qθ(x0),  J(x0)]T.\min_{\mathbf{x}_0} \left[-\log q_\theta(\mathbf{x}_0),\;\mathcal{J}(\mathbf{x}_0)\right]^T.

The learned density qθq_\theta supplies the primary realism objective; J\mathcal{J} encodes the requested behavior. Each iteration first injects noise and estimates a clean trajectory, then reduces the control cost:

x^0(k)=xtk(k)−1−αˉtk ϵθ(xtk(k),tk)αˉtk,x0(k−1)=x^0(k)−ζ∇x^0(k)J(x^0(k)).\begin{aligned} \hat{\mathbf{x}}_0(k)&= \frac{\mathbf{x}_{t_k}(k)-\sqrt{1-\bar\alpha_{t_k}}\, \boldsymbol\epsilon_\theta(\mathbf{x}_{t_k}(k),t_k)} {\sqrt{\bar\alpha_{t_k}}},\\ \mathbf{x}_0(k-1)&=\hat{\mathbf{x}}_0(k) -\zeta\nabla_{\hat{\mathbf{x}}_0(k)}\mathcal{J}(\hat{\mathbf{x}}_0(k)). \end{aligned}

The noise predictor is ϵθ\boldsymbol\epsilon_\theta, αˉtk\bar\alpha_{t_k} specifies the noise level, and ζ\zeta is the control step size. The update assumes a sufficiently small displacement to limit deterioration of the realism objective. Its control gradient requires differentiation only through the cost. We decrease tkt_k over iterations and use deterministic noise reuse in the evaluated implementation.

Conventional guidance backpropagates through the denoiser while ECM updates the estimated clean trajectory directly
The left path transports a clean-trajectory cost gradient back to noisy data through the network. ECM applies that gradient at the clean estimate, reducing the differentiation cost.

Road topology creates separated trajectory modes. ECM with Reference Joint Trajectories (ECMR) evaluates combinations of marginal predictions and the current clean estimate, chooses a low-cost reference, and then applies guidance. This gives the optimizer a starting point near another candidate mode.

Two multimodal optimization examples showing reference points, guidance directions, and low-likelihood regions
Local updates can remain near one likelihood peak or cross uncertain regions between peaks. Reference candidates let ECMR search around a useful alternative mode before taking a gradient step.

Prediction, control, and measured cost

We evaluate on Argoverse 2 with five seconds of observations and six seconds of prediction. Against an architecture-matched vanilla diffusion model starting at T=500T=500, OGD at T=40T=40 matches or improves all four evaluated prediction metrics. With DDIM stride ten, these horizons correspond to four and fifty sampling steps.

Controllable generation uses 128 samples and goals on ground-truth routes (GT) or combinations of marginal routes (U), with normal speed, deceleration, and acceleration settings. Joint Final Displacement Error (JFDE) measures goal satisfaction; Joint Route Deviation Error (JRDE) measures route adherence. We compare against Next Noisy Mean (NNM) guidance and Score Function (SF) guidance, with tuned control step sizes.

Best-sample and mean goal errors and route deviations for four guidance methods across six tasks
ECM and ECMR reduce goal errors across the six tasks; ECMR particularly improves mean errors. N, D and A denote normal speed, deceleration and acceleration. Route-deviation panels show the corresponding realism proxy, including settings where baseline methods remain competitive.

For U with deceleration, OGD+ECMR uses ten DDIM steps and attains minJFDE 0.053, versus 0.538 for vanilla diffusion with score-function guidance and fifty steps. The corresponding minJRDE values are 0.110 and 0.158.

Four guidance methods generate trajectories toward fork-lane and straight-lane goals
Magenta diamonds specify goals for U with deceleration. ECMR concentrates samples near the requested route in both examples, illustrating the role of reference candidates in changing modes.
Runtime table comparing guidance error, route deviation, per-step time, and additional GPU memory
On one RTX A6000, batch size 16 and 128 samples, ECMR takes 116 ms per step with 3.22 GB incremental memory, versus 247 ms and 7.96 GB for score-function guidance. These are per-step measurements.

Selecting the control step size

We tune the guidance step size by grid search to minimize goal error.

Goal error against guidance step size for the U deceleration task, with logarithmic axes
Small steps provide weak control, while large ECM and ECMR steps become unstable in U with deceleration. The intermediate optimum makes step-size selection part of the method's practical use.

These experiments measure realism through route adherence in recorded scenes. The marginal predictor supplies both the statistics used to initialize OGD and the candidate routes used by ECMR, so its predictions influence generation quality and the available control choices.