paper:arxiv-2601-04480When Models Manipulate Manifolds: The Geometry of a Counting Task
TL;DR
Claude 3.5 Haiku's linebreaking in fixed-width text is implemented by manipulating low-dimensional, rippled 1-D feature manifolds: character count, line width, characters remaining and next-token length are each embedded with high curvature in small residual-stream subspaces, and the model computes with them through linear geometric operations. Character count lives on a curve in a 6-dimensional subspace (95% of variance of 150 mean activations at layer 2), parametrized by 10 crosscoder features from a 10M-feature Weakly Causal Crosscoder. Attention 'boundary heads' twist the count manifold through QK so count i aligns with line width k=i+ε, raising probe cosine similarity from ~0.25 in the residual stream to ≈1 in QK space. Several heads with different offsets tile 'characters remaining'. Counts for characters remaining and next-token length sit in near-orthogonal subspaces, making the break decision linearly separable (AUC 0.91). The curvature is built by many attention heads, with 5 layer 0 heads reaching R² 0.93 and 11 heads in layers 0-1 reaching 0.97. The paper pairs attribution graphs and feature dictionaries with explicit manifold parametrization, and validates with subspace ablations, mean-activation interventions and '@@' visual illusions. Ringing is shown to be optimal under a capacity-versus-resolution tradeoff. Dictionary features alone impose a complexity tax that geometry reduces, and natural pretraining tasks are good targets for mechanistic biology, though parametrizing manifolds remains manual.
What to take away
- 1. A linear probe on the layer 1 residual stream predicts line character count with R² of 0.985, yet the underlying representation is a curved multidimensional manifold rather than a single line.
- 2. Averaged layer 2 residual vectors for 150 character counts form a helix-like curve whose top 6 PCA components capture 95% of variance, and 10 crosscoder features from a 10M-feature Weakly Causal Crosscoder reconstruct it spline-like.
- 3. In a boundary head's QK space, character-count and line-width probes reach cosine similarity ≈1 at an offset i<k, versus a maximum of ~0.25 in the raw residual stream, so the head twists one manifold onto the other.
- 4. Projecting the break decision onto near-orthogonal characters-remaining and next-token-length subspaces makes i−j≥0 a separating hyperplane, giving AUC 0.91 for predicting newlines on real data.
- 5. Ringing in probe, feature and mean-vector cosine similarities follows from projecting an ideal circulant similarity matrix onto its top 5 eigenvectors, and a 100-point, 6-dimensional sphere simulation with attraction to 6 neighbors per side reproduces it.
- 6. Replicable method: wrap a prose corpus at k=15,20,...,150 characters, average residual activations per character count, run PCA, then ablate or patch the top-k subspace (a_patched = a − μ_original + μ_c) and compare against random k-dimensional subspaces.
- 7. Ablating the character-count subspace raises loss mainly when the next token is a newline, and patching a rank-6 subspace at the aluminum prompt (true count 42) changes the newline prediction.
- 8. Individual layer 0 heads act as newline-sink counters whose OV circuits write roughly tokens-in-line × μ_c≈4 characters plus a token-length correction, with attention output affecting the count 4× more than MLPs.
- 9. Inserting @@ into the aluminum prompt distracts counting heads, and among 180 two-character insertions only a few code-like delimiters such as >>, }}, ;; and ||| substantially disrupted newline prediction from a 0.79 baseline.
- 10. Open question: can unsupervised methods find feature manifolds without manual parametrization, since the geometric simplification here relied on known scalar variables and does not obviously extend to semantic concepts?
Peer brief — for seminar discussion
The work is a mechanistic case study of how Claude 3.5 Haiku predicts newlines in fixed-width text, a task it must solve by counting characters it never sees directly. The authors start from attribution graphs over a 10M-feature Weakly Causal Crosscoder, find families of features for character count, line width, characters remaining and break prediction, and then show those features discretize 1-D manifolds that can be analyzed geometrically. The approach combines dictionary features with explicit manifold parametrization, using synthetic prose re-wrapped at widths from 15 to 150 characters. An alternative would have been supervised probing and patching alone, which the team reports tried first and abandoned because the probes targeted fused quantities. The load-bearing finding is that scalar counts are represented on rippled, high-curvature manifolds embedded in low-dimensional subspaces, with a 6-dimensional subspace capturing 95% of variance for character count, and that attention heads perform linear geometric operations on them. Boundary heads twist the count manifold so that count i aligns with line width i+ε, lifting probe alignment from about 0.25 to about 1 in QK space, and several heads with different offsets tile characters remaining. A final stage keeps characters remaining and next-token length in near-orthogonal subspaces, so the break decision is linearly separable, with AUC 0.91. The ringing seen in probe similarities is argued to be the optimal low-rank projection of a neighbor-similar structure, supported by an eigenvector toy model and a dynamical simulation. The causal evidence includes subspace ablations that mostly hurt on newline tokens, mean-activation patching, and '@@' illusions that distract the counting heads. The implication is that dictionary features alone carry a complexity tax, and geometry can pay it down where a variable is parametrizable. It also suggests choosing tasks models do naturally, like linebreaking, as targets for mechanistic biology. The paper's hypothesis is that features and manifolds are dual descriptions of one object. A critical reader would push on scope and generality. Most causal tests rest on a single aluminum prompt plus averaged statistics, one model, and a 6-dimensional subspace chosen by PCA of means, so the claim that this is the model's algorithm, rather than one decomposition, is less secure than the geometry's elegance suggests. The optimality argument also fits the ringing qualitatively, and Fourier components explain up to 10% less variance than PCA. Finally, the authors concede this was an existence proof, since the method needed known continuous variables and does not show how to discover manifolds for semantic concepts, and the line-width aggregation and multitoken cases are left unresolved.
Methods (9)
- 10M-feature Weakly Causal Crosscoder dictionaryDictionary used for all main-text features.
- Attract-repel sphere physical model100 points on a sphere with attractive/repulsive forces yields ringing.
- Attribution GraphsGradient-based technique using SAE features to estimate causal effects on completions; used to corroborate NLA findings.
- Linear and logistic probesUsed to characterize counting representations.
- Mean-activation patching interventionSubstitutes mean activation for count c to modify perceived counts.
- PCA of mean activations and probe weightsReveals low-dimensional counting manifolds.
- QK AttributionsCore method decomposing attention scores as bilinear sums of query-key feature pairs.
- Subspace zero-ablation experimentAblates top-k PCs of character-count means vs random subspace.
- Visual illusions for language modelsCharacter sequences that hijack counting attention heads.
Frameworks (1)
- Feature manifold / multidimensional featureDual to discrete dictionary features; central frame.
Datasets (2)
- Synthetic line-wrapped prose datasetProse with newlines stripped and reinserted every k characters, k=15..150.
- Synthetic markdown tables dataset20 synthetic markdown tables for row/column counting features.
Findings (28)
- Of 180 inserted two-character sequences most only moderately affect newline probability (originally 0.79); a few code/delimiter pairs disrupt it substantially
Specificity of illusion.
- In the boundary head's QK space, character count and line width probes reach max cosine similarity ≈1 at an offset, versus ~0.25 in the residual stream
Evidence for twisting.
- Attract-repel simulation of 100 points on a 2-sphere yields a baseball-seam curve matching color hue, dates, and years topology in Modell et al.
Physical model reproduces observed geometry.
- Separating hyperplane in orthogonal characters-remaining and next-token-length subspaces achieves AUC 0.91 for newline prediction
Linear separability.
- Haiku does not redirect probability mass from over-limit words to newline; distributions differ from newline-stripped prompt
Mechanism limitation.
- Dictionaries of different sizes learn character count features with similar receptive fields and ringing decoder geometry
Canonical featurization.
- Embedding token-length probes (1-14) show a circular pattern in PC1-PC2 with oscillating PC3; 3 PCs capture 70% variance
Ringing at embedding.
- Logistic probes after layer 1 achieve RMSE of 5 for character count, with 6 PCs capturing 82% of probe weight variance
Intrinsic noise in representation.
- Decreasing attractive zone width or increasing embedding dimension increases curvature and ringing in the simulation
Simulation behavior.
- Ablating the characters remaining subspace affects loss only on newlines, and patching modulates newline prediction
Causal validation.
Claims (12)
- Models represent scalar quantities on low-dimensional rippled feature manifolds that optimally trade off capacity against distinguishability
Central thesis.
- Multiple attention heads are required to construct the curved counting manifold because a single head is constrained by input curvature
Distributed algorithm.
- Mis-application of a learned prior such as @@ as git diff delimiter may modulate line length estimates, analogous to human illusions
Interpretation of illusion.
- Multidimensional counting representations enable rotation via linear maps and fine resolution without norm explosion
Role of extra dimensions.
- Canonical tiling across dictionary sizes suggests feature manifold pathologies do not occur in this setting
Appendix claim.
- Unsupervised sparse features were essential to clarifying the mechanism; probing and patching alone failed
Discovery story.
- Ringing is a natural consequence of embedding a high-dimensional similarity structure into low dimensions
Toy model.
- Studying tasks natural to pretrained models rather than to human investigators yields crisper mechanisms
Methodological lesson.
- Discrete features and feature manifolds are dual descriptions of the same underlying object
Duality.
- Early layers of language models are better thought of as perception than detokenization
Analogy to vision.
Hypotheses (3)
- Canonical dilating feature structure emerges from boundary constraint, with positions near zero anchoring feature development
Appendix hypothesis.
- The model may compute global line width via max over line lengths or exponentially weighted moving average
Left unresolved.
- Many naturally continuous variables exist in organized manifolds
Raised by Modell et al.
Questions (5)
- Can we use our understanding of the character counting mechanism to construct a "visual illusion" for language models?
Motivates illusion experiment.
- How does the model construct these curved geometries?
Intro question.
- How does the model know if the next word fits?
Intro question.
- How does the model represent different counts?
Intro question.
- How does the model detect the boundary?
Intro question.
Original abstract (expand)
Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 82%
- The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscitedin corpus2023≈ 82%
- ≈ 86%
- ≈ 81%
- Understanding Counting Mechanisms in Large Language and Vision-Language ModelsAmirmohammad Izadi, Fatemeh Askari, Mobin Bagherian, Sadegh Mohammadian, Mohammad Izadi, Mahdieh Soleymani Baghshah Hosein Hasani2026≈ 84%
- Do Sparse Autoencoders Capture Concept Manifolds?in corpus2026≈ 83%
- A geometric relation of the error introduced by sampling a language model's output distribution to its internal stateAlbert F. Modenbach2026≈ 83%
- Representational Curvature Modulates Behavioral Uncertainty in Large Language ModelsEvelina Fedorenko, Eghbal A. Hosseini Jack King2026≈ 83%
- ≈ 83%
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorin corpus2026≈ 82%
- The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling LawsMaciej Trzaskowski, Quan Nguyen, Fred Roosta Eslam Zaher2026≈ 82%
- Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language ModelsZhiyu Xue, Yihao Quan, Benlin Liu, Zeru Shi, Michelle Hurst, Jacob Feldman, Ruixiang Tang, Ranjay Krishna, Vladimir Pavlovic Liwei Che2026≈ 82%
- ≈ 82%
- ≈ 82%
- ≈ 81%
- The Confidence Manifold: Geometric Structure of Correctness Representations in Language ModelsZekun Wu, Kleyton Da Costa, Adriano Koshiyama Seonglae Cho2026≈ 81%
- Analyze Feature Flow to Enhance Interpretation and Steering in Language ModelsNikita Balagansky, Yaroslav Aksenov, Daniil Gavrilov Daniil Laptev2025≈ 81%
- Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold SteeringCraig Atkinson2026≈ 81%
- ≈ 81%
- ≈ 81%
- Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective MergingHaonan Wang, Jian Kang, Kenji Kawaguchi, Jiaying Wu Zihang Fu2026≈ 81%
- Neuron-Anchored Rule Extraction for Large Language Models via Contrastive Hierarchical AblationGabriele Dominici, Marc Langheinrich Francesco Sovrano2026≈ 81%
- One Token Away from Collapse: The Fragility of Instruction-Tuned HelpfulnessSeyedarmin Azizi, Souvik Kundu, Massoud Pedram Erfan Baghaei Potraghloo2026≈ 81%
- Gating Enables Curvature: A Geometric Expressivity Gap in AttentionAnand A. Joshi Satwik Bathula2026≈ 81%
- ≈ 81%
- The Geometric Structure of Models Learning Sparse DataT. Mitchell Roddenberry, Ahmed Imtiaz Humayun, Randall Balestriero, Richard Baraniuk Thomas Walker2026≈ 81%
- Word Recovery in Large Language Models Enables Character-Level Tokenization RobustnessShu Yang, Lijie Hu, Di Wang Zhipeng Yang2026≈ 81%
- ≈ 81%
- ≈ 81%
- ≈ 80%
+23 more
Similar preprints — Semantic Scholar
Cited by (4)
- Do Sparse Autoencoders Capture Concept Manifolds?
Sparse autoencoders trained on Llama3.1-8B (layer 19, 500M Pile tokens, 25 SAEs across Standard, JumpReLU, TopK, BatchTopK and Matryoshka architectures at VE > 0.85) do not capture continuous concept
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Lifting the featurizer's atom from a direction to a block of directions recovers concept manifolds that SAEs shatter, and block sparsity is the MAP-matched prior for a representation that is a sparse
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectorie
- Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as