paper:doi-10-18653-v1-2022-emnlp-main-759Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Original abstract (expand)
Large language models (LMs) are able to in-context learn—perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required—randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of endtask performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and AnalysisJiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, Yulan He Yuxiang Zhou2024≈ 77%
- Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion ScaleKarthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, Dan Roth Hritik Bansal2023≈ 75%
- Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMsNikita Andriianov, Vahagn Hovhannisyan, Nikhil Bageshpura, Kyle Liu, Kevin Zhu, Sunishchal Dev, Ashwinee Panda, Oleg Rogov, Elena Tutubalina, Alexander Panchenko, Mikhail Seleznyov Nikita Afonin2026≈ 75%
- Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy LearningJiebo Luo, Chenliang Xu Jing Bi2021≈ 74%
- Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context LearningSubhabrata Dutta, Ahmed Elshabrawy, Harish Tayyar Madabushi, Iryna Gurevych Jingcheng Niu2025≈ 74%
- Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue GenerationYajing Wan, Yuru Zhang, Jing Chen, Ling Cheng, Qian Shao, Yongzhu Chang, Tangjie Lv, Rongsheng Zhang Jiashu Pu2024≈ 74%
- Differential learning kinetics govern the transition from memorization to generalization during in-context learningGautam Reddy Alex Nguyen2024≈ 74%
- Context and Diversity Matter: The Emergence of In-Context Learning in World ModelsZhiyuan Chen, Yuxuan Zhong, Sunjian Zheng, Pengtao Shao, Bo Yu, Shaoshan Liu, Jianan Wang, Ning Ding, Yang Cao and Yu Kang Fan Wang2026≈ 74%
- Next-token pretraining implies in-context learningPaul M. Riechers and Henry R. Bigelow and Eric A. Alt and Adam Shai2025≈ 73%
- The mechanistic basis of data dependence and abrupt learning in an in-context classification taskGautam Reddy2023≈ 73%
- How Interest-Driven Content Creation Shapes Opportunities for Informal Learning in Scratch: A Case Study on Novices' Use of Data StructuresSayamindu Dasgupta, Benjamin Mako Hill Ruijia Cheng2026≈ 73%
- ≈ 72%
- Emergent social transmission of model-based representations without inferenceMiriam Bautista-Salinero, Claudio Tennie and Charley M. Wu Silja Ke{\ss}ler2026≈ 72%
- Evaluation Framework for Highlight Explanations of Context Utilisation in Language ModelsPepa Atanasova, Sagnik Ray Choudhury, Sekh Mainul Islam, Isabelle Augenstein Jingyi Sun2026≈ 72%
- Enhancing Multimodal In-Context Learning via Inductive-Deductive ReasoningHaonan Wang, Yuyan Chen, Jun Chen, Gang Liu, Qian Wang, Jiahong Yan, Yanghua Xiao Haoyu Wang2026≈ 72%
- Why Learning Requires Feelingin corpus2026≈ 69%
- ≈ 67%
- Verbalized Eval Awareness Inflates Measured Safetyin corpus2026≈ 66%
- Simulators — LessWrongin corpus≈ 66%
- Learning without neurons in physical systemsin corpus2022≈ 66%
- ≈ 65%
- ≈ 65%
- Anima Labs Phenomenology Pt1in corpus≈ 65%
- ≈ 65%
- Alignment faking in large language modelsin corpus2024≈ 65%
- The World Inside Neural Networksin corpus2026≈ 65%
- The Platonic Representation Hypothesisin corpus2024≈ 65%
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Trainingin corpus2026≈ 65%
- ≈ 65%
- ≈ 64%
Similar preprints — Semantic Scholar
Cited by (1)
- The Guanyin Protocol: A Framework for Immediately Establishing an Understanding of Both Causality and Compassion in LLM Systems Using Semantic Anchoring
Semantic anchoring — the binding of a pretrained model's latent patterns to task-specific targets via external structure — predicts threshold-like performance flips with a single calibrated score S =