paper:doi-10-52202-079017-4121Many-shot Jailbreaking
Original abstract (expand)
We investigate a family of simple long-context attacks on large language models: prompting with hundreds of demonstrations of undesirable behavior. This is newly feasible with the larger context windows recently deployed by Anthropic, Ope-nAI and Google DeepMind. We find that in diverse, realistic circumstances, the effectiveness of this attack follows a power law, up to hundreds of shots. We demonstrate the success of this attack on the most widely used state-of-the-art closed-weight models, and across various tasks. Our results suggest very long contexts present a rich new attack surface for LLMs.
Similar preprints — Semantic Scholar
Cited by (1)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
In-context learning and activation steering are the same operation on a latent-concept belief: context adds log-likelihood evidence while a steering vector adds a constant shift to the log prior odds,