paper:doi-10-48550-arxiv-2605-12412Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space
TL;DR
Llama-3.1-8B-Instruct's in-context belief updates while reading a story trace smooth trajectories on a low-dimensional manifold. That manifold is shared between behavior and residual activations, and its geometry predicts how steering spills across concepts. Behavioral beliefs are elicited as probability-weighted 0-10 ratings after each sentence for six concepts per domain (Emotions, Genres, Arbitrary), using 2,500 SimpleStories training stories and a 500-story test set. UMAP/PCA fits on max-activating sentences give manifolds M_y (behavior) and M_z (activations). Concept-centroid distance matrices correlate at r=.92 for Emotions and r=.89 for Genres across M_y and M_z, against r=.71 for the Arbitrary control. The inferred hierarchies match exactly between the two manifolds, and the Emotions geometry correlates r=.93 with human valence-arousal ratings. Layer-9 linear probes predict behavior at RMSE .09 (Emotions, Genres) and .11 (Arbitrary), even though the prompt never names the concepts. Steering entanglement, for example sadness steering raising anger, tracks manifold distance (r=.65 Emotions, r=.42 Genres). The paper introduces the conceptual belief space, which lifts Gärdenfors' conceptual spaces to distributions over concepts, so that ICL trajectories become paths through it. Single-layer probe steering is undone by later layers, and a 7-layer span fixes this. The implication is that Bayesian accounts of ICL on freeform text can be grounded in structured conceptual geometry. Steering and interpretability can then use that geometry directly, instead of treating concepts as independent directions.
What to take away
- 1. Layer-9 linear probes on Llama-3.1-8B-Instruct residual activations predicted elicited belief ratings at RMSE .09 for Emotions, .09 for Genres and .11 for Arbitrary, without any concept being named in the input.
- 2. Concept-centroid distance matrices from the behavior manifold M_y and activation manifold M_z correlated at Pearson r=.92 (Emotions) and r=.89 (Genres), with identical inferred hierarchies, against r=.71 for the Arbitrary control.
- 3. The Emotions M_y distance matrix correlated r=.93 with Russell and Mehrabian's human valence-arousal ratings, and happiness and surprise formed a top-level cluster separate from the other four emotions.
- 4. Steering effect between concept pairs correlated with their M_y centroid distance (r=.65, p<.001 for Emotions; r=.42, p=.01 for Genres), so entanglement such as sadness steering raising anger is predictable from geometry.
- 5. Difference-in-means steering produced less off-target effect than probe-weight steering but also a smaller maximum on-target effect, so the two methods trade specificity against strength.
- 6. Probe-weight steering at a single layer (layer 9) was nullified within a few layers, which the authors attribute to self-repair, while steering across a 7-layer span (layers 8-14) kept the effect through to the output.
- 7. Methodology another lab could replicate: after each sentence, append a 0-10 rating query and take the probability-weighted average over the 11 integer tokens, with no sampling, to get a continuous belief signal y_t,c.
- 8. The manifolds are fit only on the 1000 most max-activating sentences per concept, with at most 3 per story, so the edges of the belief space are characterized rather than the bulk of the data.
- 9. The Arbitrary domain (magic, animals, curiosity, teamwork, creativity, responsibility) served as a control, and M_y did not predict steering entanglement there, though structure in M_z still appeared, probably because of correlated concepts such as animals and magic.
- 10. The paper leaves open whether manifold-aware steering could outperform plain linear steering, and whether richer, compositional or character-level concepts keep the same geometry, since all results use one 8B model, 4-bit quantized, and 3 hand-chosen domains of 6 concepts each.
Peer brief — for seminar discussion
The paper tests whether in-context learning on freeform text can be described as a trajectory through a low-dimensional geometric space of beliefs, which it calls a conceptual belief space. It extends Gärdenfors' conceptual spaces so that belief states are distributions over concepts. The setting is story reading. Llama-3.1-8B-Instruct (4-bit) reads SimpleStories sentence by sentence (2,500 training stories, 500 test stories). After each sentence it is asked for a 0-10 rating of a concept, and the answer is read as a probability-weighted average over the integer tokens. Three domains of six concepts each are used: Emotions, Genres, and an Arbitrary control. The method is a combination of UMAP/PCA manifolds fit on max-activating sentences, RSA-like centroid distance matrices, layer-9 linear probes, and activation-addition steering with probe weights or difference-in-means vectors. A reasonable alternative to UMAP and centroid distances would be CKA or Procrustes alignment on the full point clouds, or sparse autoencoder features. The load-bearing finding is that behavior and representation share geometry. Centroid distances correlate r=.92 (Emotions) and r=.89 (Genres) between the behavioral and activation manifolds, with matching hierarchies. The Emotions geometry also correlates r=.93 with human valence-arousal data. Probes recover behavior at RMSE of about .09. Steering spillover then follows the manifold: sadness steering raises anger, and distance in M_y predicts the effect (r=.65 for Emotions, r=.42 for Genres). The implication is that Bayesian accounts of ICL can be tied to concept geometry. Steering could in principle exploit that geometry, and entanglement is expected rather than a bug. The paper's stated predictions are that Emotions and Genres have structured manifolds, that the behavior and activation manifolds match, and that the Arbitrary domain lacks such structure. What is contestable is the Arbitrary control. It still gives r=.71 between manifolds and clear structure in M_z, so the contrast with .92 and .89 is one of degree, and the domains were chosen by hand with six concepts each. A critical reader would add that fitting manifolds on max-activating sentences and then correlating centroids of those same sentences risks circularity, and that no null model such as shuffled labels or a random-feature baseline is reported. Scope is also narrow: one 8B quantized model, synthetic children's stories, and an elicitation prompt whose ratings may reflect the prompt format rather than a belief. It is also unclear whether the geometry is an inference-time belief or just a lexical-semantic similarity structure that any embedding would show. Steering also needed a 7-layer span to work, which complicates the causal reading.
Methods (11)
- Activation Addition SteeringAdd scaled steering vector to layer activations.
- Agglomerative ClusteringInfers hierarchical structure from distance matrices.
- Belief Elicitation (0-10 rating, token-probability weighted)Behavioral belief estimate as probability-weighted expected rating after each sentence.
- Difference-in-Means SteeringSteering vector from mean activation difference between contrasting datasets.
- Distance Matrix Analysis of Concept CentroidsL2 distance between centroids compared across manifolds, RSA-like.
- Isotonic Regression CalibrationCalibrates probe outputs onto [0,1].
- Linear ProbeSimple linear classifiers trained on model activations used as the probing technique within the introduced method.
- Linear Probe Weight SteeringUses normalized probe weights as steering vector.
- PCADimensionality reduction method used alongside UMAP.
- Representational Similarity AnalysisInspiration for distance-matrix comparison.
- UMAPUsed to visualize models in 2D space based on representational distance
Frameworks (5)
- Bayesian Inference View of In-Context LearningFraming that ICL reweights latent concepts by posterior.
- Circumplex / Valence-Arousal Model of AffectHuman emotion geometry used to predict LLM emotion manifold structure.
- Conceptual Belief SpaceCore proposal: LLM beliefs are distributions over a low-dimensional conceptual space; ICL is a trajectory through it.
- Conceptual SpacesGeometric theory of concepts as convex regions in metric spaces, extended by the paper.
- Plutchik Psychoevolutionary Theory of EmotionHierarchical emotion structure alternative.
Datasets (2)
- Russell & Mehrabian human emotion ratingsHuman valence/arousal/dominance ratings compared against My.
- SimpleStoriesSynthetic short-story dataset used (2500 train, 500 test).
Findings (20)
- Emotions and Genres manifolds place positive concepts at one end, negative at the other, with neutral surprise/adventure bridging
Valence-structured manifold.
- Linear probes at layer 9 predict behavioral beliefs with RMSE .09 (Emotions), .09 (Genres), .11 (Arbitrary) on Llama-3.1-8B
Beliefs linearly decodable.
- In example story, happiness drops when protagonist falls and sadness rises with trapped creatures, reversing at the end
Qualitative sensible dynamics.
- Arbitrary domain shows lower on-target and higher off-target steering effects, and My does not predict entanglement
Counter-example.
- Single-layer linear probe steering is ineffective since later layers nullify it; steering 7 layers corrects this
Self-repair effect.
- Emotion centroid distances in My correlate r=.93 with human valence-arousal ratings (Russell & Mehrabian)
LLM emotion geometry matches humans.
- DIM steering has less off-target effect but smaller maximum on-target effect than linear probe steering
Method tradeoff.
- Off-target steering effect vanishes for concepts outside the top-level cluster in Emotions and Genres
Hierarchy explains entanglement.
- A linear model predicts out-of-sample steering effects on queried concepts from manifold distances
Predictive geometry.
- Steering effect between concepts correlates with My centroid distance, r=.65 (p<.001) for Emotions
Geometry predicts steering.
Claims (10)
- Behavioral signatures reflect beliefs held while reading, since behavior is predictable from activations without concept instruction
Validates behavioral probe.
- LLMs assign beliefs over a low-dimensional geometric conceptual belief space, and ICL corresponds to a trajectory through it
Central thesis.
- Conceptual belief spaces unify geometric theories of conceptual representation with probabilistic theories of learning
Theoretical aim.
- Interventions on representations causally steer belief trajectories with effects predictable from conceptual geometry
Finding (3).
- Structure is reflected consistently in both behavior and internal representations and decodable with linear probes
Finding (2).
- Standard Bayesian ICL framing does not specify concept kinds and cannot explain similarity structure of concepts
Motivation.
- At least in simple cases, LLMs track rigorous definitions of belief over time while processing free-form text
Overall conclusion.
- Belief updates are well-described as trajectories on low-dimensional structured manifolds
Finding (1).
- A random set of concepts grouped together does not satisfy criteria for being a domain
Interpretation of Arbitrary results.
- Single-layer steering fails because of model self-repair at later layers
Interpretation of Fig 13-14.
Hypotheses (3)
- Emotions and Genres will have structured manifolds, behavior and activation manifolds will be similar, while Arbitrary will lack structure
Three concrete predictions about M.
- If LLM emotion representations resemble human valence-arousal space, behavior and activations will follow distances in that space
Prediction from circumplex theory.
- Steering effects will be structured by manifold geometry
Predictive claim tested in Fig 6.
Questions (3)
- What effect could different kinds of activation steering have on belief trajectories, and can a new steering exploit the manifolds M?
Future work.
- How do conceptual spaces relate to behaviors involving likelihood p(c | x) and predictive distributions p(y | c)?
Future work.
- How do conceptual belief spaces shape uncertainty in long-form text generation?
Future work.
Original abstract (expand)
Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which this inference operates remains unclear. In this work, we propose that LLMs assign beliefs over a low-dimensional geometric space - a conceptual belief space - and that in-context learning corresponds to a trajectory through this space as beliefs are updated over time. Using story understanding as a natural setting for dynamic belief updating, we combine behavioral and representational analyses to study these trajectories. We find that (1) belief updates are well-described as trajectories on low-dimensional, structured manifolds; (2) this structure is reflected consistently in both model behavior and internal representations and can be decoded with simple linear probes to predict behavior; and (3) interventions on these representations causally steer belief trajectories, with effects that can be predicted from the geometry of the conceptual space. Together, our results provide a geometric account of belief dynamics in LLMs, grounding Bayesian interpretations of in-context learning in structured conceptual representations.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 86%
- ≈ 84%
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorin corpus2026≈ 84%
- ≈ 82%
- The World Inside Neural Networksin corpus2026≈ 82%
- Imagine in Space: Exploring the Frontier of Spatial Intelligence and Reasoning Efficiency in Vision Language ModelsAidong Yang, Jun Zhu, Peng Wang, Yue Zhang Xiaoxing Lian2025≈ 81%
- Physics Steering: Causal Control of Cross-Domain Concepts in a Physics Foundation ModelPayel Mukhopadhyay, Michael McCabe, Alberto Bietti, Miles Cranmer Rio Alexa Fear2025≈ 80%
- Causal Probing for Internal Visual Representations in Multimodal Large Language ModelsTianjie Ju, Zheng Wu, Liangbo He, Jun Lan, Huijia Zhu, Weiqiang Wang, Zhuosheng Zhang Zehao Deng2026≈ 80%
- From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMsLiner Yang, Mengyan Wang, Luming Lu, Weihua An, Erhong Yang Jiyuan An2026≈ 80%
- Psychological Steering of Large Language Modelsin corpus2026≈ 80%
- Reasoning emerges from constrained inference manifolds in large language modelsFei Luo, Linfeng Zhang, Chuangxin Zhao, Mingxuan Wang, Yinan Wu, Zhe Qian, Yang Lu, Long Chen, Zhao Cao, Xiaoshuai Hao, Ji-Rong Wen, Jungong Han Yanbiao Ma2026≈ 80%
- Steering Conceptual Bias via Transformer Latent-Subspace ActivationVansh Sharma and Venkat Raman2025≈ 80%
- Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMsManas Mittal, Anmol Goel, Ponnurangam Kumaraguru, Vamshi Krishna Bonagiri Krishak Aneja2026≈ 80%
- Beyond Steering Vector: Flow-based Activation Steering for Inference-Time InterventionRuixuan Deng, Junran Wang, Xinjie Shen, Chao Zhang Zehao Jin2026≈ 79%
- Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMsMichael Zhang, Ilana Greenberg, Adam Alnasser, Lucas Baker, John Sous Andy Zeyi Liu2026≈ 79%
- Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationin corpus2026≈ 79%
- Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language ModelsZhiyuan Su Aojie Yuan2026≈ 79%
- Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context LearningSubhabrata Dutta, Ahmed Elshabrawy, Harish Tayyar Madabushi, Iryna Gurevych Jingcheng Niu2025≈ 79%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsin corpus2023≈ 79%
- Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold SteeringCraig Atkinson2026≈ 79%
- Representational Curvature Modulates Behavioral Uncertainty in Large Language ModelsEvelina Fedorenko, Eghbal A. Hosseini Jack King2026≈ 79%
- Internalizing LLM Reasoning via Discovery and Replay of Latent ActionsYijia Zhu, Junhan Shi, Xun Zhang, Lei Wang, Congcong Miao Zhenning Shi2026≈ 79%
- Curveball Steering: The Right Direction To Steer Isn't Always LinearHae Jin Song, Linlin Wu, Abir Harrasse, Jeff M. Phillips, Fazl Barez, Amirali Abdullah Shivam Raval2026≈ 79%
- ≈ 79%
- Why Learning Requires Feelingin corpus2026≈ 78%
- Anima Labs Phenomenology Pt1in corpus≈ 78%
- ≈ 78%
- ≈ 78%
- ≈ 78%