paper
active
2025
paper:arxiv-2511-00617

Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering

TL;DR

In-context learning and activation steering are the same operation on a latent-concept belief: context adds log-likelihood evidence while a steering vector adds a constant shift to the log prior odds, so the two sum in log-odds space. A four-parameter closed-form model (α, γ, a, b), log o(c|x) = b + γN^(1−α) + a·m, predicts Llama-3.1-8B-Instruct behavior on held-out steering magnitudes at r = 0.98 averaged over five persona datasets (Machiavellianism, Psychopathy, Narcissism, and two moral-nihilism sets, from Perez et al. 2022). It also predicts the crossover shot count N* at r = 0.97. Fitting is by L-BFGS with 10-fold cross-validation over steering magnitude, using Contrastive Activation Addition vectors applied at layer 12. The same model holds for Qwen-2.5-7B and Gemma-2-9B (N* r = 0.91 and 0.97), Llama-3.1-70B (r = 0.98), and a flipped-label financial sentiment task (r = 0.96). Posterior belief is sigmoidal in N^(1−α), which explains the sudden many-shot jailbreaking curves of Anil et al. (2024). Steering moves the curve left or right, and the joint (shots × magnitude) space splits into sharp phases whose boundary is computable. Steering is linear in log-odds only within a bounded range of m, beyond which behavior collapses to chance. The result implies that prompting and steering are interchangeable levers on one belief state. Small changes in either control can therefore flip behavior abruptly, and N* can be forecast for safety purposes.

What to take away

  1. 1. The belief dynamics model, with four free parameters (α, γ, a, b) fit per dataset–model pair, reaches held-out correlation r = 0.98 with Llama-3.1-8B-Instruct across five persona domains under 10-fold cross-validation over steering magnitude.
  2. 2. Cross-over shot counts N* (where log posterior odds hit zero) are predicted at r = 0.97 for Llama-3.1-8B, and at r = 0.91 for Qwen-2.5-7B and r = 0.97 for Gemma-2-9B.
  3. 3. Steering is modeled as a magnitude-proportional additive shift a·m in log prior odds, so persona-matching probability is sigmoidal in m. It is approximately linear in log-odds for m in [-3, 3] on the Dark Triad datasets and [-1.5, 1.5] on the Moral Nihilism datasets.
  4. 4. In-context evidence is made sub-linear by a discount τ(N) = N^(−α) following Wurgaft et al. 2025, giving a log Bayes factor ∝ N^(1−α) and sigmoidal learning curves in that variable, which recasts the Anil et al. 2024 many-shot jailbreaking curves.
  5. 5. Positive steering shifts the ICL curve leftward (fewer shots needed) and negative steering shifts it rightward, producing a phase boundary in the shots × magnitude plane that the fitted model reconstructs.
  6. 6. For Llama-3.1-70B the model still fits at cross-validated r = 0.98, but ICL kicks in after only a few shots. In flipped-label sentiment (Financial PhraseBank, 3 labels, r = 0.96) substantially more shots are needed.
  7. 7. Steering breaks down at large magnitudes: the response converges to chance (p = 0.5) even at long contexts, with a dataset-specific threshold that is larger for Narcissism in Llama-3.1-8B, so the linear-in-log-odds account holds only inside a bounded range.
  8. 8. Counter-intuitively, steering vectors computed from many shots (up to 128) and norm-normalized to the 0-shot norm were weaker than 0-shot vectors. Their cosine similarity to the 0-shot vector drops suddenly after one added example.
  9. 9. Open question: steering worked in only one layer or a few (layer 12 for Llama-3.1-8B, 20 for Gemma-2-9B, 14 for Qwen-2.5-7B) and had no clear effect in phi-4-mini-instruct, so is belief localized and linear in a subspace, or is the CAA vector construction the limit?
  10. 10. Replicable protocol: sweep 25 shot counts (0–128) × 29 steering magnitudes with 100 random exemplar sequences each, average the persona-consistent token probability, fit with BCE loss and basin-hopping initialization, and hold out blocks of 3 adjacent magnitudes.

Peer brief — for seminar discussion

The work tests whether prompting and activation steering are two routes to the same Bayesian belief update. Its instrument is a closed-form belief dynamics model, log o(c|x) = b + γN^(1−α) + a·m. The in-context term is sub-linear evidence accumulation, and the steering term is a constant shift in log prior odds that follows from the linear representation hypothesis. The experiments extend many-shot ICL, in the style of Anil et al. and Agarwal et al., with steering magnitude as a second axis. They use Contrastive Activation Addition vectors at a single tuned layer (layer 12 in Llama-3.1-8B-Instruct) on five Perez et al. persona datasets, with 25 shot counts up to 128 and 29 steering magnitudes. The load-bearing finding is that this four-parameter model predicts held-out behavior with r = 0.98 averaged over domains, holding out blocks of adjacent magnitudes under 10-fold cross-validation. It also predicts the phase boundary N* at r = 0.97. It replicates in Qwen-2.5-7B and Gemma-2-9B (N* r = 0.91 and 0.97), Llama-3.1-70B (r = 0.98), and flipped-label financial sentiment (r = 0.96). The three predictions are sigmoidal ICL in N^(1−α), sigmoidal response in steering magnitude, and additivity of the two in log-odds, which yields phases with a computable crossover. The implication is that the two levers are interchangeable on a single belief axis. A steering offset buys a predictable number of shots, and a small control change near N* can flip behavior abruptly. That bears on many-shot jailbreaking forecasts and on combining behavioral and mechanistic control. A critical reader should push on several points. First, the model is fit to per-condition averages over 100 sampled sequences. Adding a free offset b and scale a to a sigmoid of N^(1−α) is very flexible, and r is computed over a heatmap dominated by ceiling and floor plateaus. Cross-validation holds out magnitudes but not datasets or models, so the correlation may be inflated and says little about whether belief is actually what is being updated. A more discriminating test would compare against alternatives such as a plain power-law ICL fit with an additive steering term, or a logistic regression in log N and m. Second, the evidence that steering is a prior shift is indirect, since the steering vector is only approximately a concept direction. Linearity fails at large magnitudes, and a single layer works. The multi-shot vector result (weaker than the 0-shot vector) cuts against a clean ICL-as-evidence reading. Third, scope: binary concepts, 7–9B models plus one 70B, 4-bit quantization, one steering method, and persona behaviors chosen because they saturate within 128 shots. The authors also suggest, speculatively, that SAE-based steering and non-binary concept spaces would test the framework further.

Methods (7)

Frameworks (6)

Datasets (4)

Original abstract (expand)

Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering). Different accounts have been proposed to explain these methods, yet their common goal of controlling model behavior raises the question of whether these seemingly disparate methodologies can be seen as specific instances of a broader framework. Motivated by this, we develop a unifying, predictive account of LLM control from a Bayesian perspective. Specifically, we posit that both context- and activation-based interventions impact model behavior by altering its belief in latent concepts: steering operates by changing concept priors, while in-context learning leads to an accumulation of evidence. This results in a closed-form Bayesian model that is highly predictive of LLM behavior across context- and activation-based interventions in a set of domains inspired by prior work on many-shot in-context learning. This model helps us explain prior empirical phenomena - e.g., sigmoidal learning curves as in-context evidence accumulates - while predicting novel ones - e.g., additivity of both interventions in log-belief space, which results in distinct phases such that sudden and dramatic behavioral shifts can be induced by slightly changing intervention controls. Taken together, this work offers a unified account of prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+20 more

Similar preprints — Semantic Scholar

Cited by (1)