paper
active
2026
paper:arxiv-2601-04480

When Models Manipulate Manifolds: The Geometry of a Counting Task

TL;DR

Claude 3.5 Haiku's linebreaking in fixed-width text is implemented by manipulating low-dimensional, rippled 1-D feature manifolds: character count, line width, characters remaining and next-token length are each embedded with high curvature in small residual-stream subspaces, and the model computes with them through linear geometric operations. Character count lives on a curve in a 6-dimensional subspace (95% of variance of 150 mean activations at layer 2), parametrized by 10 crosscoder features from a 10M-feature Weakly Causal Crosscoder. Attention 'boundary heads' twist the count manifold through QK so count i aligns with line width k=i+ε, raising probe cosine similarity from ~0.25 in the residual stream to ≈1 in QK space. Several heads with different offsets tile 'characters remaining'. Counts for characters remaining and next-token length sit in near-orthogonal subspaces, making the break decision linearly separable (AUC 0.91). The curvature is built by many attention heads, with 5 layer 0 heads reaching R² 0.93 and 11 heads in layers 0-1 reaching 0.97. The paper pairs attribution graphs and feature dictionaries with explicit manifold parametrization, and validates with subspace ablations, mean-activation interventions and '@@' visual illusions. Ringing is shown to be optimal under a capacity-versus-resolution tradeoff. Dictionary features alone impose a complexity tax that geometry reduces, and natural pretraining tasks are good targets for mechanistic biology, though parametrizing manifolds remains manual.

What to take away

  1. 1. A linear probe on the layer 1 residual stream predicts line character count with R² of 0.985, yet the underlying representation is a curved multidimensional manifold rather than a single line.
  2. 2. Averaged layer 2 residual vectors for 150 character counts form a helix-like curve whose top 6 PCA components capture 95% of variance, and 10 crosscoder features from a 10M-feature Weakly Causal Crosscoder reconstruct it spline-like.
  3. 3. In a boundary head's QK space, character-count and line-width probes reach cosine similarity ≈1 at an offset i<k, versus a maximum of ~0.25 in the raw residual stream, so the head twists one manifold onto the other.
  4. 4. Projecting the break decision onto near-orthogonal characters-remaining and next-token-length subspaces makes i−j≥0 a separating hyperplane, giving AUC 0.91 for predicting newlines on real data.
  5. 5. Ringing in probe, feature and mean-vector cosine similarities follows from projecting an ideal circulant similarity matrix onto its top 5 eigenvectors, and a 100-point, 6-dimensional sphere simulation with attraction to 6 neighbors per side reproduces it.
  6. 6. Replicable method: wrap a prose corpus at k=15,20,...,150 characters, average residual activations per character count, run PCA, then ablate or patch the top-k subspace (a_patched = a − μ_original + μ_c) and compare against random k-dimensional subspaces.
  7. 7. Ablating the character-count subspace raises loss mainly when the next token is a newline, and patching a rank-6 subspace at the aluminum prompt (true count 42) changes the newline prediction.
  8. 8. Individual layer 0 heads act as newline-sink counters whose OV circuits write roughly tokens-in-line × μ_c≈4 characters plus a token-length correction, with attention output affecting the count 4× more than MLPs.
  9. 9. Inserting @@ into the aluminum prompt distracts counting heads, and among 180 two-character insertions only a few code-like delimiters such as >>, }}, ;; and ||| substantially disrupted newline prediction from a 0.79 baseline.
  10. 10. Open question: can unsupervised methods find feature manifolds without manual parametrization, since the geometric simplification here relied on known scalar variables and does not obviously extend to semantic concepts?

Peer brief — for seminar discussion

The work is a mechanistic case study of how Claude 3.5 Haiku predicts newlines in fixed-width text, a task it must solve by counting characters it never sees directly. The authors start from attribution graphs over a 10M-feature Weakly Causal Crosscoder, find families of features for character count, line width, characters remaining and break prediction, and then show those features discretize 1-D manifolds that can be analyzed geometrically. The approach combines dictionary features with explicit manifold parametrization, using synthetic prose re-wrapped at widths from 15 to 150 characters. An alternative would have been supervised probing and patching alone, which the team reports tried first and abandoned because the probes targeted fused quantities. The load-bearing finding is that scalar counts are represented on rippled, high-curvature manifolds embedded in low-dimensional subspaces, with a 6-dimensional subspace capturing 95% of variance for character count, and that attention heads perform linear geometric operations on them. Boundary heads twist the count manifold so that count i aligns with line width i+ε, lifting probe alignment from about 0.25 to about 1 in QK space, and several heads with different offsets tile characters remaining. A final stage keeps characters remaining and next-token length in near-orthogonal subspaces, so the break decision is linearly separable, with AUC 0.91. The ringing seen in probe similarities is argued to be the optimal low-rank projection of a neighbor-similar structure, supported by an eigenvector toy model and a dynamical simulation. The causal evidence includes subspace ablations that mostly hurt on newline tokens, mean-activation patching, and '@@' illusions that distract the counting heads. The implication is that dictionary features alone carry a complexity tax, and geometry can pay it down where a variable is parametrizable. It also suggests choosing tasks models do naturally, like linebreaking, as targets for mechanistic biology. The paper's hypothesis is that features and manifolds are dual descriptions of one object. A critical reader would push on scope and generality. Most causal tests rest on a single aluminum prompt plus averaged statistics, one model, and a 6-dimensional subspace chosen by PCA of means, so the claim that this is the model's algorithm, rather than one decomposition, is less secure than the geometry's elegance suggests. The optimality argument also fits the ringing qualitatively, and Fourier components explain up to 10% less variance than PCA. Finally, the authors concede this was an existence proof, since the method needed known continuous variables and does not show how to discover manifolds for semantic concepts, and the line-width aggregation and multitoken cases are left unresolved.

Methods (9)

Frameworks (1)

Datasets (2)

Findings (28)

Original abstract (expand)

Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+23 more

Similar preprints — Semantic Scholar

Cited by (4)