thinker:orcid-0009-0005-9673-6809Jack Merullo
Authored papers (5)
Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as its circularly structured representations might suggest, but by reusing a single generic base-10 addition mechanism across all such tasks. The mechanism operates in two stages: first, the model computes the raw integer sum of its inputs (six + August = 14), then maps that sum back into cyclic concept space (14 → February). The paper introduces a combined causal-abstraction and feature-geometry analysis to demonstrate that Llama-3.1-8B uses task-agnostic Fourier features with periods characteristic of base-10 arithmetic—specifically periods 2, 5, and 10—rather than the concept-native period. Strikingly, a sparse set of only 28 MLP neurons at layer 18 (approximately 0.2% of that layer's MLP) is responsible for this computation across all cyclic tasks and can be partitioned into disjoint clusters, each computing the sum for a Fourier feature of a distinct period. This implies that structured representational geometry does not entail correspondingly structured computation: a model can build domain-specific circular embeddings and yet delegate the actual arithmetic to a domain-general, base-10 calculator reused opportunistically, suggesting that mechanistic interpretability analyses that read off algorithms from representation geometry alone will systematically mislead.
Manifold steering — intervening on model activations along paths constrained to lie on a learned activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectories that track the corresponding behavior manifold M_y, while linear (Euclidean) steering cuts through off-manifold regions and generates unnatural outputs. The paper fits M_h to internal representations and M_y to output probability distributions, then tests their bidirectional correspondence via controlled interventions across language models and a video world model. In language models, tasks with cyclic geometries, sequential geometries, and complex graph geometries (in-context learning) all show that manifold-constrained interventions keep behavioral outputs on M_y, whereas linear steering deviates measurably. In a video world model, interventions shaped by physical-dynamics geometry similarly respect M_y. Crucially, the relationship is bidirectional: optimizing interventions in activation space to produce paths along M_y recovers activation trajectories that trace the curvature of M_h. This implies that the core problem of mechanistic steering should be recast not as finding the right direction in a flat Euclidean activation space, but as identifying the correct geometric structure — because representational geometry is not incidental to model behavior but is causally constitutive of it.
- Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior2026merged
Manifold steering — intervening along activation-space paths defined by the intrinsic geometry of a fitted activation manifold M_h rather than along Euclidean linear directions — produces behavioral trajectories that remain on the behavior manifold M_y, whereas standard linear steering cuts through off-manifold regions and generates unnatural outputs. Across language model reasoning tasks with cyclic, sequential, and graph geometries, and in a video world model whose task geometry corresponds to physical dynamics, the bidirectional relationship M_h ↔ M_y holds: steering that respects M_h tracks M_y, and optimizing interventions in activation space to follow M_y recovers activation trajectories that trace the curvature of M_h. The method fits two manifolds — one to intermediate representations, one to output probability distributions — and then uses geodesic-like paths on M_h as the intervention substrate rather than a single steering vector. This work argues that neural representation geometry is not incidental but is the causal structure organizing model behavior, which implies that the core problem of model steering should be reframed from finding the right direction to finding the right geometry.
Reasoning models generate chains of thought that are frequently performative rather than causally necessary for reaching the correct answer: on MMLU recall questions, activation probes decode the model's final answer from internal representations far earlier in the chain-of-thought than a CoT language monitor can, demonstrating that the visible reasoning tokens are generated after the belief is already settled. Tested across DeepSeek-R1 671B and GPT-OSS 120B using three complementary methods — activation probing, early forced answering, and a CoT monitor — the analysis reveals a task-difficulty gradient: the gap between probe-decodable certainty and monitor-detectable certainty is large for easy MMLU items but narrows substantially for difficult GPQA-Diamond multihop questions, where genuine step-by-step uncertainty persists longer. The paper introduces probe-guided early exit, which terminates generation once probe confidence passes a threshold, cutting token counts by up to 80% on MMLU and 30% on GPQA-Diamond while preserving accuracy. A key finding that complicates a purely dismissive view is that inflection-point behaviors — backtracking and 'aha' moments — appear almost exclusively in traces where probes register large belief shifts, meaning these surface signals do track genuine uncertainty rather than being uniformly theatrical. The paper argues that this dissociation between internal belief and continued CoT generation is systematic and exploitable, positioning attention probing as both a diagnostic instrument for faithfulness research and a practical mechanism for adaptive computation.
More papers — OpenAlex / S2
Affiliations (2)
- Harvard University(institute)
- Goodfire(institute)
Co-authors (12)
- Atticus Geiger27 shared
- Raphaël Sarfati27 shared
- Can Rager23 shared
- Daniel Wurgaft23 shared
- Ekdeep Singh Lubana23 shared
- Owen Lewis23 shared
- Sheridan Feucht23 shared
- Tal Haklay23 shared
- Thomas Fel23 shared
- Thomas McGrath23 shared
- Usha Bhalla23 shared
- Eric Bigelow19 shared
Their work is cited by (1)
Other inbound relations (1)
- authoredThe World Inside Neural Networks(paper)
Recent mentions (7)
- papers-typedfeucht-2026-arithmetic.md
- papers-typedboppana-2026-reasoning.md
- papers-typed
boppana-goodfire-reasoning-theater-2026.md - papers-typed
feucht-goodfire-geometric-calculator-2026.md - papers-typedpaper.md
- papers-typedwurgaft-2026-manifold.md