← All publications

The Initialization Determines Whether In-Context Learning Is Gradient Descent

  • Large Language Models
  • Deep Learning Theory

Query initialization controls the correspondence between linear self-attention and gradient descent; a trainable initial guess recovers this correspondence in the studied regression setting.

Paper TMLR 2025

Shifeng Xie1, Rui Yuan2, Simone Rossi3, Thomas Hannagan4

  1. Telecom Paris, France
  2. Lexsi Labs, France
  3. EURECOM, France
  4. Stellantis, France

Linear self-attention (LSA) can learn to predict a regression task from examples in its context. A familiar explanation interprets its forward pass as one gradient descent (GD) step. We examine how this correspondence changes when regression tasks share a non-zero mean. The crucial choice is the query's initial prediction: a zero guess is misaligned with a non-zero shared prior mean. We introduce yqy_q-LSA, which learns this guess through one additional vector. In controlled linear regression experiments, it reaches the one-step GD reference loss and closely matches the reference predictor's input sensitivities.

Our in-context learning overview explains and reproduces the zero-initialized linear-attention construction.

Why the initial prediction matters

We consider noiseless regression tasks whose weights vary around a shared mean w⋆\mathbf{w}_\star. With dd input features and CC context examples, the data and attention input are

w^∼N(w⋆,Id),x1,…,xC,xq∼iidN(0,Id),yi=w^⊤xi,y=w^⊤xq,E=[X⊤xqy⊤yq].\begin{gathered} \widehat{\mathbf{w}}\sim\mathcal{N}(\mathbf{w}_\star,I_d), \qquad \mathbf{x}_1,\ldots,\mathbf{x}_C,\mathbf{x}_q \overset{\mathrm{iid}}{\sim}\mathcal{N}(0,I_d),\\ y_i=\widehat{\mathbf{w}}^\top\mathbf{x}_i, \qquad y=\widehat{\mathbf{w}}^\top\mathbf{x}_q,\\ E=\begin{bmatrix}X^\top&\mathbf{x}_q\\\mathbf{y}^\top&y_q\end{bmatrix}. \end{gathered}

Here XX contains the context inputs as rows, y\mathbf{y} contains their labels, and yqy_q is the query's initial guess. Attention refines this bottom-right entry, with the query masked from the context update. The conventional choice yq=0y_q=0 aligns naturally with w⋆=0\mathbf{w}_\star=0. A non-zero prior mean gives a useful starting prediction w⋆⊤xq\mathbf{w}_\star^\top\mathbf{x}_q that varies with the query.

Training and evaluation loss remain above the one-step gradient descent baseline
Training and evaluation losses plateau above the dashed one-step GD reference under a non-zero prior mean. This gap motivates examining the starting prediction supplied to attention.

More heads reach a capacity ceiling

Could additional attention heads recover this missing information? For a single linear attention layer with unrestricted trainable projection matrices, we prove that its function class saturates by d+1d+1 heads. Writing FH-LSA\mathcal{F}_{H\text{-LSA}} for the class with HH heads and R(f)=E[(f(E)−y)2]R(f)=\mathbb{E}[(f(E)-y)^2] for query prediction risk,

inf⁡f∈FH-LSAR(f)=inf⁡f∈F(d+1)-LSAR(f),H≥d+1.\inf_{f\in\mathcal{F}_{H\text{-LSA}}}R(f) = \inf_{f\in\mathcal{F}_{(d+1)\text{-LSA}}}R(f), \qquad H\geq d+1.

The proof expresses each head's contribution through rank-one matrices. Choosing d+1d+1 basis vectors lets these heads realize every matrix collection needed to specify the predictor. Further heads therefore preserve the same attainable risk. This is a capacity result for the linear architecture; optimization still determines which predictor training finds.

Training curves and final losses as the number of attention heads increases
With d=C=10d=C=10, w⋆=[2,…,2]⊤\mathbf{w}_\star=[2,\ldots,2]^\top, and yq=0y_q=0, more heads lower the loss before gains flatten near the capacity bound. The remaining gap to GD separates representational saturation from the initialization problem.
Loss gap increases with the squared norm of the regression prior mean
With 11 heads and yq=0y_q=0, training matches the GD reference at zero prior mean. Across the tested non-zero means, the final loss grows approximately linearly with ∥w⋆∥2\|\mathbf{w}_\star\|^2, linking the discrepancy to prior misalignment.

Learning the starting point restores the GD construction

We introduce a trainable vector w∈Rd\mathbf{w}\in\mathbb{R}^d and set yq=w⊤xqy_q=\mathbf{w}^\top\mathbf{x}_q. At w=w⋆\mathbf{w}=\mathbf{w}_\star, a single attention head can implement

fyq-LSA(X,y,xq)=yq−ηC(w⋆⊤X⊤−y⊤)Xxq=(w⋆−ηCX⊤(Xw⋆−y))⊤xq.\begin{aligned} f_{y_q\text{-LSA}}(X,\mathbf{y},\mathbf{x}_q) &=y_q-\frac{\eta}{C} (\mathbf{w}_\star^\top X^\top-\mathbf{y}^\top)X\mathbf{x}_q\\ &=\left(\mathbf{w}_\star-\frac{\eta}{C} X^\top(X\mathbf{w}_\star-\mathbf{y})\right)^\top\mathbf{x}_q. \end{aligned}

The term multiplied by the step size η\eta is the gradient of the context least-squares objective ∥Xw−y∥2/(2C)\|X\mathbf{w}-\mathbf{y}\|^2/(2C). The residual connection retains the starting prediction while attention applies the correction. This identity holds algebraically for the specified weights and finite context; learning those weights is a separate optimization problem.

Linear and constant perturbations to the initial query guess increase final prediction loss
With zero prior mean and 11 heads, we vary the linear guess component and constant bias separately. Both create a persistent loss gap, approximately proportional to their squared magnitudes in the tested ranges. The ablation isolates initialization while holding attention capacity fixed.

Testing predictions, sensitivities, and sentence similarity

We train the linear models on freshly sampled tasks using 5,000 Adam steps, batch size 2,048, and learning rate 5×10−45\times10^{-4}. Each task has d=C=10d=C=10; the layer omits softmax and LayerNorm. Evaluation uses a forward pass with fixed parameters. Across the non-zero means shown below, yqy_q-LSA approaches the GD loss. Its predictions and gradients with respect to xq\mathbf{x}_q also approach the reference, with gradient cosine similarity near one.

Trainable initialization matches the gradient descent loss and approaches its predictions and query gradients
Panel (a) compares training at three non-zero prior means with GD initialized at the corresponding mean. Panel (b) tracks prediction differences, query-input gradient differences, and gradient cosine similarity. These diagnostics assess the predictor's local behavior alongside its risk.

For English STS-Benchmark, we supply ten labelled sentence pairs and initial similarity guesses to LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct. A two-layer predictor learns guesses from concatenated all-MiniLM-L6-v2 sentence embeddings. Under deterministic decoding, both language models achieve lower mean squared error with guesses than their corresponding prompting baselines across the displayed guess-model training epochs. The evaluation averages over 100 experimental runs.

Sentence similarity mean squared error with and without an initial guess for LLaMA and Qwen
Both language models refine the supplied guesses and improve over their baselines without guesses. The horizontal axis varies training of the guess model; it exposes how prediction quality changes with the source of prior information.