paper:doi-10-48550-arxiv-2212-07677Transformers learn in-context by gradient descent
Original abstract (expand)
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- How Transformers Learn Causal Structure with Gradient DescentAlex Damian, Jason D. Lee Eshaan Nichani2024≈ 80%
- Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-ContextSiddhant Gadkari Faris Chaudhry2026≈ 76%
- ≈ 76%
- ≈ 75%
- ≈ 75%
- How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic InterpretabilityChangdae Oh, Zhen Fang, Sharon Li Shawn Im2026≈ 75%
- ≈ 75%
- How Transformers Get Rich: Approximation and Dynamics AnalysisRuoxi Yu, Weinan E, Lei Wu Mingze Wang2025≈ 75%
- Understanding the Emergence of Seemingly Useless Features in Next-Token PredictorsJalal Naghiyev, Michael Hahn Mark Rofin2026≈ 74%
- Birth of a Transformer: A Memory ViewpointVivien Cabannes, Diane Bouchacourt, Herve Jegou, Leon Bottou Alberto Bietti2023≈ 74%
- Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern TransformersKarsten Roth, Quentin Bouniot, Wenjia Xu, Zeynep Akata Yiran Huang2026≈ 74%
- Can Transformers Learn to Solve Problems Recursively?Curt Tigges, Stella Biderman, Maxim Raginsky, Talia Ringer Shizhuo Dylan Zhang2023≈ 74%
- ≈ 74%
- ≈ 73%
- Transformers Struggle to Learn to SearchSrushti Pawar, Shreyas Pimpalgaonkar, Nitish Joshi, Richard Yuanzhe Pang, Vishakh Padmakumar, Seyed Mehran Kazemi, Najoung Kim, He He Abulhair Saparov2025≈ 72%
- Relating transformers to models and neural representations of the hippocampal formationin corpus2021≈ 70%
- Learning without neurons in physical systemsin corpus2022≈ 69%
- A Mathematical Framework for Transformer Circuitsin corpus2021≈ 67%
- ≈ 66%
- Simulators — LessWrongin corpus≈ 64%
- ≈ 63%
- ≈ 63%
- Model Alignment Searchin corpus2025≈ 63%
- ≈ 63%
- ≈ 63%
- ≈ 63%
- ≈ 62%
- Open Problems in Mechanistic Interpretabilityin corpus2025≈ 62%
- The World Inside Neural Networksin corpus2026≈ 62%
Similar preprints — Semantic Scholar
Cited by (2)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
In-context learning and activation steering are the same operation on a latent-concept belief: context adds log-likelihood evidence while a steering vector adds a constant shift to the log prior odds,
- The Guanyin Protocol: A Framework for Immediately Establishing an Understanding of Both Causality and Compassion in LLM Systems Using Semantic Anchoring
Semantic anchoring — the binding of a pretrained model's latent patterns to task-specific targets via external structure — predicts threshold-like performance flips with a single calibrated score S =