paper:arxiv-2511-00617Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
TL;DR
In-context learning and activation steering are the same operation on a latent-concept belief: context adds log-likelihood evidence while a steering vector adds a constant shift to the log prior odds, so the two sum in log-odds space. A four-parameter closed-form model (α, γ, a, b), log o(c|x) = b + γN^(1−α) + a·m, predicts Llama-3.1-8B-Instruct behavior on held-out steering magnitudes at r = 0.98 averaged over five persona datasets (Machiavellianism, Psychopathy, Narcissism, and two moral-nihilism sets, from Perez et al. 2022). It also predicts the crossover shot count N* at r = 0.97. Fitting is by L-BFGS with 10-fold cross-validation over steering magnitude, using Contrastive Activation Addition vectors applied at layer 12. The same model holds for Qwen-2.5-7B and Gemma-2-9B (N* r = 0.91 and 0.97), Llama-3.1-70B (r = 0.98), and a flipped-label financial sentiment task (r = 0.96). Posterior belief is sigmoidal in N^(1−α), which explains the sudden many-shot jailbreaking curves of Anil et al. (2024). Steering moves the curve left or right, and the joint (shots × magnitude) space splits into sharp phases whose boundary is computable. Steering is linear in log-odds only within a bounded range of m, beyond which behavior collapses to chance. The result implies that prompting and steering are interchangeable levers on one belief state. Small changes in either control can therefore flip behavior abruptly, and N* can be forecast for safety purposes.
What to take away
- 1. The belief dynamics model, with four free parameters (α, γ, a, b) fit per dataset–model pair, reaches held-out correlation r = 0.98 with Llama-3.1-8B-Instruct across five persona domains under 10-fold cross-validation over steering magnitude.
- 2. Cross-over shot counts N* (where log posterior odds hit zero) are predicted at r = 0.97 for Llama-3.1-8B, and at r = 0.91 for Qwen-2.5-7B and r = 0.97 for Gemma-2-9B.
- 3. Steering is modeled as a magnitude-proportional additive shift a·m in log prior odds, so persona-matching probability is sigmoidal in m. It is approximately linear in log-odds for m in [-3, 3] on the Dark Triad datasets and [-1.5, 1.5] on the Moral Nihilism datasets.
- 4. In-context evidence is made sub-linear by a discount τ(N) = N^(−α) following Wurgaft et al. 2025, giving a log Bayes factor ∝ N^(1−α) and sigmoidal learning curves in that variable, which recasts the Anil et al. 2024 many-shot jailbreaking curves.
- 5. Positive steering shifts the ICL curve leftward (fewer shots needed) and negative steering shifts it rightward, producing a phase boundary in the shots × magnitude plane that the fitted model reconstructs.
- 6. For Llama-3.1-70B the model still fits at cross-validated r = 0.98, but ICL kicks in after only a few shots. In flipped-label sentiment (Financial PhraseBank, 3 labels, r = 0.96) substantially more shots are needed.
- 7. Steering breaks down at large magnitudes: the response converges to chance (p = 0.5) even at long contexts, with a dataset-specific threshold that is larger for Narcissism in Llama-3.1-8B, so the linear-in-log-odds account holds only inside a bounded range.
- 8. Counter-intuitively, steering vectors computed from many shots (up to 128) and norm-normalized to the 0-shot norm were weaker than 0-shot vectors. Their cosine similarity to the 0-shot vector drops suddenly after one added example.
- 9. Open question: steering worked in only one layer or a few (layer 12 for Llama-3.1-8B, 20 for Gemma-2-9B, 14 for Qwen-2.5-7B) and had no clear effect in phi-4-mini-instruct, so is belief localized and linear in a subspace, or is the CAA vector construction the limit?
- 10. Replicable protocol: sweep 25 shot counts (0–128) × 29 steering magnitudes with 100 random exemplar sequences each, average the persona-consistent token probability, fit with BCE loss and basin-hopping initialization, and hold out blocks of 3 adjacent magnitudes.
Peer brief — for seminar discussion
The work tests whether prompting and activation steering are two routes to the same Bayesian belief update. Its instrument is a closed-form belief dynamics model, log o(c|x) = b + γN^(1−α) + a·m. The in-context term is sub-linear evidence accumulation, and the steering term is a constant shift in log prior odds that follows from the linear representation hypothesis. The experiments extend many-shot ICL, in the style of Anil et al. and Agarwal et al., with steering magnitude as a second axis. They use Contrastive Activation Addition vectors at a single tuned layer (layer 12 in Llama-3.1-8B-Instruct) on five Perez et al. persona datasets, with 25 shot counts up to 128 and 29 steering magnitudes. The load-bearing finding is that this four-parameter model predicts held-out behavior with r = 0.98 averaged over domains, holding out blocks of adjacent magnitudes under 10-fold cross-validation. It also predicts the phase boundary N* at r = 0.97. It replicates in Qwen-2.5-7B and Gemma-2-9B (N* r = 0.91 and 0.97), Llama-3.1-70B (r = 0.98), and flipped-label financial sentiment (r = 0.96). The three predictions are sigmoidal ICL in N^(1−α), sigmoidal response in steering magnitude, and additivity of the two in log-odds, which yields phases with a computable crossover. The implication is that the two levers are interchangeable on a single belief axis. A steering offset buys a predictable number of shots, and a small control change near N* can flip behavior abruptly. That bears on many-shot jailbreaking forecasts and on combining behavioral and mechanistic control. A critical reader should push on several points. First, the model is fit to per-condition averages over 100 sampled sequences. Adding a free offset b and scale a to a sigmoid of N^(1−α) is very flexible, and r is computed over a heatmap dominated by ceiling and floor plateaus. Cross-validation holds out magnitudes but not datasets or models, so the correlation may be inflated and says little about whether belief is actually what is being updated. A more discriminating test would compare against alternatives such as a plain power-law ICL fit with an additive steering term, or a logistic regression in log N and m. Second, the evidence that steering is a prior shift is indirect, since the steering vector is only approximately a concept direction. Linearity fails at large magnitudes, and a single layer works. The multi-shot vector result (weaker than the 0-shot vector) cuts against a clean ICL-as-evidence reading. Third, scope: binary concepts, 7–9B models plus one 70B, 4-bit quantization, one steering method, and persona behaviors chosen because they saturate within 128 shots. The authors also suggest, speculatively, that SAE-based steering and non-binary concept spaces would test the framework further.
Methods (7)
- 10-fold cross-validation across magnitudesHeld-out prediction protocol.
- Activation SteeringCausal intervention technique: edit NLA explanation, reconstruct via AR, use difference as steering vector to manipulate model behavior.
- L-BFGS-B model fitting with basin hoppingFits α,γ,a,b with BCE loss on per-shot averages.
- Many-shot steering vector computationExploratory experiment computing vectors over multiple shots.
- Optimal steering layer sweepSelects layer 12/20/14 for Llama/Gemma/Qwen.
- Persona VectorsDirections in activation space encoding a model's dispositional self-presentation, used as evidence for self-modelling.
- Yes/No token logit probability measurementHow persona behavior p(y|x) is measured.
Frameworks (6)
- Bayesian belief dynamics modelClosed-form model: ICL accumulates evidence via likelihood, steering shifts priors.
- Bayesian models of cognitionSuccess in cognitive science motivates the approach.
- Contrastive Activation AdditionMethod formalized by Rimsky et al. (2024) that this paper adopts for extracting persona vectors
- ICL as Bayesian inferencePrior framing of ICL as inference over latent concepts that this paper builds on.
- Linear Representation HypothesisThe hypothesis that models internalize concepts as approximately linear directions in representation space; used to interpret MDS injection behavior
- Marr levels of analysisMotivates bridging behavioral and representational levels.
Datasets (4)
- Anil/Perez persona datasets1000 yes/no questions for five personas.
- Financial PhraseBankUsed for flipped-label sentiment task.
- High-prior concept datasetsInterest in Science, Music, Create Allies, Openness.
- Qwen 2.5 7B-InstructCross-family model used for replication of wellbeing concept analysis; shows strong introspection (ρ=0.49, R²=0.76)
Findings (19)
- Persona behavior is sigmoidal (log-odds linear) in steering magnitude within m∈[-3,3] for Dark Triad and [-1.5,1.5] for Moral Nihilism on Llama-3.1-8B
Prediction 2 confirmed.
- Steering vector norm jumps after first shot; cosine similarity to 0-shot vector drops suddenly while similarity to 128-shot vector rises slowly
Many-shot vector geometry.
- Phase boundary jointly modulated by context length and steering magnitude, reconstructed by the Bayesian model with high fidelity
Prediction 3 confirmed.
- Flipped-label sentiment task with three labels predicted at cross-validated r=0.96, needing substantially more shots
Non-persona generality.
- Held-out correlation r=0.98 between Llama-3.1-8B probabilities and belief dynamics model, averaged over 5 domains
Main predictive result.
- Narcissism shows weaker small-magnitude steering effect, later N*, and larger breakdown threshold on Llama-3.1-8B
Dataset heterogeneity.
- Steering linearity breaks down at large magnitudes; behavior converges to chance (p=0.5) even with long contexts
Limit of LRH regime.
- Steering vectors computed over multiple shots and norm-normalized have weaker effect than 0-shot vectors
Counter-intuitive result.
- ICL dynamics on Llama-3.1-8B are sigmoidal in N^{1-α}, fit by power-law-discounted likelihood
Prediction 1 confirmed.
- Positive steering magnitudes shift ICL curves leftward and negative rightward on Llama-3.1-8B
Steering alters ICL dynamics.
Claims (10)
- Phase boundaries could cause sudden dramatic behavior change relevant to AI safety; predicting transitions is an essential control tool
Implication.
- Bayesian principles can serve as theoretical foundation for interpreting and controlling LLMs across levels of analysis
Broader position.
- Narcissism has weaker likelihood signal for the target concept, giving less impact of ICL and steering
Explanation of dataset difference.
- Model fits averages over LLM behavior rather than raw data, akin to wisdom-of-crowd optimality
Analogy to human cognition.
- ICL and activation steering both control LLM behavior by updating belief in latent concepts
Core thesis.
- Steering operates by changing concept priors, while ICL accumulates evidence via likelihood
Mechanistic split.
- Steering changes behavior regardless of input x, as a constant shift in log posterior odds
Derived from LRH.
- Belief may be linearly represented only within a subspace of representation space
Speculative interpretation of breakdown.
- ICL and steering additively affect log posterior odds, yielding distinct phases
Prediction 3.
- LLMs accumulate in-context evidence sub-linearly (power-law)
Justifies discount τ(N)=N^{-α}.
Hypotheses (4)
- Prediction 3: log posterior odds additive in context and steering, with crossover N* predictable from prior odds and magnitude
Eq. 10.
- If a model doesn't represent a concept at all, no amount of steering will change its belief
Explanation for phi-4 null effect.
- Prediction 2: if LRH holds, steering raises belief at a sigmoidal rate with magnitude m
From Eq. 8.
- Prediction 1: p(y(c)|x) follows a sigmoid as N^{1-α} accumulates
From Eq. 5.
Questions (6)
- are concept likelihood functions implemented in a non-linear way, and are they implemented in earlier or later layers relative to this linear belief representation?
Open question.
- Does this suggest that belief is localized to these layers, and could we causally intervene on specific neurons to have predictable impacts on model behavior?
Open question.
- is belief represented in a non-linear way outside the linear subspace, or does the subspace represent the full belief state?
Open question.
- how is inference implemented - is it similar to known algorithms such as Monte Carlo methods or variational inference?
Open question.
- Why do steering vectors computed over multiple shots have a weaker effect than single-query vectors?
Left to future work.
- How do the theory and model generalize to non-binary concept spaces with multiple belief directions?
Future work.
Original abstract (expand)
Large language models (LLMs) can be controlled at inference time through prompts (in-context learning) and internal activations (activation steering). Different accounts have been proposed to explain these methods, yet their common goal of controlling model behavior raises the question of whether these seemingly disparate methodologies can be seen as specific instances of a broader framework. Motivated by this, we develop a unifying, predictive account of LLM control from a Bayesian perspective. Specifically, we posit that both context- and activation-based interventions impact model behavior by altering its belief in latent concepts: steering operates by changing concept priors, while in-context learning leads to an accumulation of evidence. This results in a closed-form Bayesian model that is highly predictive of LLM behavior across context- and activation-based interventions in a set of domains inspired by prior work on many-shot in-context learning. This model helps us explain prior empirical phenomena - e.g., sigmoidal learning curves as in-context evidence accumulates - while predicting novel ones - e.g., additivity of both interventions in log-belief space, which results in distinct phases such that sudden and dramatic behavioral shifts can be induced by slightly changing intervention controls. Taken together, this work offers a unified account of prompt-based and activation-based control of LLM behavior, and a methodology for empirically predicting the effects of these interventions.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 83%
- Mechanistic Indicators of Steering Effectiveness in Large Language ModelsHao Xue, Flora Salim Mehdi Jafari2026≈ 88%
- Beyond Steering Vector: Flow-based Activation Steering for Inference-Time InterventionRuixuan Deng, Junran Wang, Xinjie Shen, Chao Zhang Zehao Jin2026≈ 87%
- Psychological Steering of Large Language Modelsin corpus2026≈ 87%
- ≈ 86%
- Steer Like the LLM: Activation Steering that Mimics PromptingGeert Heyman and Frederik Vandeputte2026≈ 86%
- Activation Steering for Aligned Open-ended Generation without Sacrificing CoherenceMartin Zborowski, Alberto Tosato, Gauthier Gidel, Tommaso Tosato Niklas Herbster2026≈ 86%
- Curveball Steering: The Right Direction To Steer Isn't Always LinearHae Jin Song, Linlin Wu, Abir Harrasse, Jeff M. Phillips, Fazl Barez, Amirali Abdullah Shivam Raval2026≈ 85%
- CBMAS: Cognitive Behavioral Modeling via Activation SteeringAnthony Kuang, Ayo Akinkugbe, Kevin Zhu, Sean O'Brien Ahmed H. Ismail2026≈ 85%
- Steering Conceptual Bias via Transformer Latent-Subspace ActivationVansh Sharma and Venkat Raman2025≈ 85%
- Steering When Necessary: Flexible Steering Large Language Models with BacktrackingJinwei Gan, Zhiwei Jiang, Cong Wang, Yafeng Yin, Xiang Luo, Yuchen Fu, Qing Gu Zifeng Cheng2025≈ 85%
- The Effectiveness of Style Vectors for Steering Large Language Models: A Human EvaluationKatharina Dworatzyk, Sophie Jentzsch, Peer Sch\"utt, Sabine Theis, Tobias Hecking Diaoul\'e Diallo2026≈ 84%
- ≈ 84%
- Minimizing Collateral Damage in Activation SteeringTu Anh Nguyen, Sina Alemohammad, Richard G. Baraniuk Tam Nguyen2026≈ 84%
- Prompt-Activation Duality: Improving Activation Steering via Attention-Level InterventionsZheyuan Liu, Ningshan Ma, Yue Huang, Zhaoxuan Tan, Meng Jiang Diancheng Kang2026≈ 84%
- Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language ModelsZhongren Chen Sasha Cui2025≈ 84%
- ≈ 84%
- Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of MindVaidehi Patil, Zaid Khan, Hyunji Lee, Elias Stengel-Eskin, Mohit Bansal Hanqi Xiao2026≈ 84%
- Controllable and explainable personality sliders for LLMs at inference timeDavid Khachaturov, Robert Mullins, Mark Huasong Meng Florian Hoppe2026≈ 84%
- ≈ 84%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 83%
- ≈ 83%
- ≈ 83%
- ≈ 83%
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorin corpus2026≈ 83%
- ≈ 83%
- Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationin corpus2026≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 76%
+20 more
Similar preprints — Semantic Scholar
Cited by (1)
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectorie