The Initialization Determines Whether In-Context Learning Is Gradient Descent
- Large Language Models
- Deep Learning Theory
Query initialization controls the correspondence between linear self-attention and gradient descent; a trainable initial guess recovers this correspondence in the studied regression setting.
Paper TMLR 2025
- Telecom Paris, France
- Lexsi Labs, France
- EURECOM, France
- Stellantis, France
Linear self-attention (LSA) can learn to predict a regression task from examples in its context. A familiar explanation interprets its forward pass as one gradient descent (GD) step. We examine how this correspondence changes when regression tasks share a non-zero mean. The crucial choice is the query's initial prediction: a zero guess is misaligned with a non-zero shared prior mean. We introduce -LSA, which learns this guess through one additional vector. In controlled linear regression experiments, it reaches the one-step GD reference loss and closely matches the reference predictor's input sensitivities.
Our in-context learning overview explains and reproduces the zero-initialized linear-attention construction.
Why the initial prediction matters
We consider noiseless regression tasks whose weights vary around a shared mean . With input features and context examples, the data and attention input are
Here contains the context inputs as rows, contains their labels, and is the query's initial guess. Attention refines this bottom-right entry, with the query masked from the context update. The conventional choice aligns naturally with . A non-zero prior mean gives a useful starting prediction that varies with the query.

More heads reach a capacity ceiling
Could additional attention heads recover this missing information? For a single linear attention layer with unrestricted trainable projection matrices, we prove that its function class saturates by heads. Writing for the class with heads and for query prediction risk,
The proof expresses each head's contribution through rank-one matrices. Choosing basis vectors lets these heads realize every matrix collection needed to specify the predictor. Further heads therefore preserve the same attainable risk. This is a capacity result for the linear architecture; optimization still determines which predictor training finds.


Learning the starting point restores the GD construction
We introduce a trainable vector and set . At , a single attention head can implement
The term multiplied by the step size is the gradient of the context least-squares objective . The residual connection retains the starting prediction while attention applies the correction. This identity holds algebraically for the specified weights and finite context; learning those weights is a separate optimization problem.

Testing predictions, sensitivities, and sentence similarity
We train the linear models on freshly sampled tasks using 5,000 Adam steps, batch size 2,048, and learning rate . Each task has ; the layer omits softmax and LayerNorm. Evaluation uses a forward pass with fixed parameters. Across the non-zero means shown below, -LSA approaches the GD loss. Its predictions and gradients with respect to also approach the reference, with gradient cosine similarity near one.

For English STS-Benchmark, we supply ten labelled sentence pairs and initial similarity guesses to LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct. A two-layer predictor learns guesses from concatenated all-MiniLM-L6-v2 sentence embeddings. Under deterministic decoding, both language models achieve lower mean squared error with guesses than their corresponding prompting baselines across the displayed guess-model training epochs. The evaluation averages over 100 experimental runs.
