paper
active
2026
paper:doi-10-48550-arxiv-2605-12412

Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space

TL;DR

Llama-3.1-8B-Instruct's in-context belief updates while reading a story trace smooth trajectories on a low-dimensional manifold. That manifold is shared between behavior and residual activations, and its geometry predicts how steering spills across concepts. Behavioral beliefs are elicited as probability-weighted 0-10 ratings after each sentence for six concepts per domain (Emotions, Genres, Arbitrary), using 2,500 SimpleStories training stories and a 500-story test set. UMAP/PCA fits on max-activating sentences give manifolds M_y (behavior) and M_z (activations). Concept-centroid distance matrices correlate at r=.92 for Emotions and r=.89 for Genres across M_y and M_z, against r=.71 for the Arbitrary control. The inferred hierarchies match exactly between the two manifolds, and the Emotions geometry correlates r=.93 with human valence-arousal ratings. Layer-9 linear probes predict behavior at RMSE .09 (Emotions, Genres) and .11 (Arbitrary), even though the prompt never names the concepts. Steering entanglement, for example sadness steering raising anger, tracks manifold distance (r=.65 Emotions, r=.42 Genres). The paper introduces the conceptual belief space, which lifts Gärdenfors' conceptual spaces to distributions over concepts, so that ICL trajectories become paths through it. Single-layer probe steering is undone by later layers, and a 7-layer span fixes this. The implication is that Bayesian accounts of ICL on freeform text can be grounded in structured conceptual geometry. Steering and interpretability can then use that geometry directly, instead of treating concepts as independent directions.

What to take away

  1. 1. Layer-9 linear probes on Llama-3.1-8B-Instruct residual activations predicted elicited belief ratings at RMSE .09 for Emotions, .09 for Genres and .11 for Arbitrary, without any concept being named in the input.
  2. 2. Concept-centroid distance matrices from the behavior manifold M_y and activation manifold M_z correlated at Pearson r=.92 (Emotions) and r=.89 (Genres), with identical inferred hierarchies, against r=.71 for the Arbitrary control.
  3. 3. The Emotions M_y distance matrix correlated r=.93 with Russell and Mehrabian's human valence-arousal ratings, and happiness and surprise formed a top-level cluster separate from the other four emotions.
  4. 4. Steering effect between concept pairs correlated with their M_y centroid distance (r=.65, p<.001 for Emotions; r=.42, p=.01 for Genres), so entanglement such as sadness steering raising anger is predictable from geometry.
  5. 5. Difference-in-means steering produced less off-target effect than probe-weight steering but also a smaller maximum on-target effect, so the two methods trade specificity against strength.
  6. 6. Probe-weight steering at a single layer (layer 9) was nullified within a few layers, which the authors attribute to self-repair, while steering across a 7-layer span (layers 8-14) kept the effect through to the output.
  7. 7. Methodology another lab could replicate: after each sentence, append a 0-10 rating query and take the probability-weighted average over the 11 integer tokens, with no sampling, to get a continuous belief signal y_t,c.
  8. 8. The manifolds are fit only on the 1000 most max-activating sentences per concept, with at most 3 per story, so the edges of the belief space are characterized rather than the bulk of the data.
  9. 9. The Arbitrary domain (magic, animals, curiosity, teamwork, creativity, responsibility) served as a control, and M_y did not predict steering entanglement there, though structure in M_z still appeared, probably because of correlated concepts such as animals and magic.
  10. 10. The paper leaves open whether manifold-aware steering could outperform plain linear steering, and whether richer, compositional or character-level concepts keep the same geometry, since all results use one 8B model, 4-bit quantized, and 3 hand-chosen domains of 6 concepts each.

Peer brief — for seminar discussion

The paper tests whether in-context learning on freeform text can be described as a trajectory through a low-dimensional geometric space of beliefs, which it calls a conceptual belief space. It extends Gärdenfors' conceptual spaces so that belief states are distributions over concepts. The setting is story reading. Llama-3.1-8B-Instruct (4-bit) reads SimpleStories sentence by sentence (2,500 training stories, 500 test stories). After each sentence it is asked for a 0-10 rating of a concept, and the answer is read as a probability-weighted average over the integer tokens. Three domains of six concepts each are used: Emotions, Genres, and an Arbitrary control. The method is a combination of UMAP/PCA manifolds fit on max-activating sentences, RSA-like centroid distance matrices, layer-9 linear probes, and activation-addition steering with probe weights or difference-in-means vectors. A reasonable alternative to UMAP and centroid distances would be CKA or Procrustes alignment on the full point clouds, or sparse autoencoder features. The load-bearing finding is that behavior and representation share geometry. Centroid distances correlate r=.92 (Emotions) and r=.89 (Genres) between the behavioral and activation manifolds, with matching hierarchies. The Emotions geometry also correlates r=.93 with human valence-arousal data. Probes recover behavior at RMSE of about .09. Steering spillover then follows the manifold: sadness steering raises anger, and distance in M_y predicts the effect (r=.65 for Emotions, r=.42 for Genres). The implication is that Bayesian accounts of ICL can be tied to concept geometry. Steering could in principle exploit that geometry, and entanglement is expected rather than a bug. The paper's stated predictions are that Emotions and Genres have structured manifolds, that the behavior and activation manifolds match, and that the Arbitrary domain lacks such structure. What is contestable is the Arbitrary control. It still gives r=.71 between manifolds and clear structure in M_z, so the contrast with .92 and .89 is one of degree, and the domains were chosen by hand with six concepts each. A critical reader would add that fitting manifolds on max-activating sentences and then correlating centroids of those same sentences risks circularity, and that no null model such as shuffled labels or a random-feature baseline is reported. Scope is also narrow: one 8B quantized model, synthetic children's stories, and an elicitation prompt whose ratings may reflect the prompt format rather than a belief. It is also unclear whether the geometry is an inference-time belief or just a lexical-semantic similarity structure that any embedding would show. Steering also needed a 7-layer span to work, which complicates the causal reading.

Methods (11)

Frameworks (5)

Datasets (2)

Original abstract (expand)

Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which this inference operates remains unclear. In this work, we propose that LLMs assign beliefs over a low-dimensional geometric space - a conceptual belief space - and that in-context learning corresponds to a trajectory through this space as beliefs are updated over time. Using story understanding as a natural setting for dynamic belief updating, we combine behavioral and representational analyses to study these trajectories. We find that (1) belief updates are well-described as trajectories on low-dimensional, structured manifolds; (2) this structure is reflected consistently in both model behavior and internal representations and can be decoded with simple linear probes to predict behavior; and (3) interventions on these representations causally steer belief trajectories, with effects that can be predicted from the geometry of the conceptual space. Together, our results provide a geometric account of belief dynamics in LLMs, grounding Bayesian interpretations of in-context learning in structured conceptual representations.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar