paper
active
2026
paper:feucht-goodfire-geometric-calculator-2026

Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

BySheridan Feucht·Tal Haklay·Usha Bhalla·Daniel Wurgaft·Can Rager·Raphaël Sarfatiⓘ+6 moreGoodfire, Harvard University + 2 more
DOI 10.48550/arxiv.2605.01148arXiv 2605.01148OpenAlex W7160247613
Christopher Alexander's 15 propertiesGeometric structure in neural representationsNeural GeometryActivation spaceCausal InterventionAddition moduleArithmeticBase-10 Addition Mechanism in LlamaCircular Geometric RepresentationCyclic concepts (months, days)Layer 18Llama-3.1 8BModular ArithmeticParallel modular arithmetic

TL;DR

Llama-3.1-8B solves cyclic arithmetic (e.g., "what month is six months after August?") not by performing modular addition in the period of the cyclic concept (12 for months, 7 for days of the week) as its circularly structured representations might suggest, but by reusing a single generic base-10 addition mechanism across all such tasks. The mechanism operates in two stages: first, the model computes the raw integer sum of its inputs (six + August = 14), then maps that sum back into cyclic concept space (14 → February). The paper introduces a combined causal-abstraction and feature-geometry analysis to demonstrate that Llama-3.1-8B uses task-agnostic Fourier features with periods characteristic of base-10 arithmetic—specifically periods 2, 5, and 10—rather than the concept-native period. Strikingly, a sparse set of only 28 MLP neurons at layer 18 (approximately 0.2% of that layer's MLP) is responsible for this computation across all cyclic tasks and can be partitioned into disjoint clusters, each computing the sum for a Fourier feature of a distinct period. This implies that structured representational geometry does not entail correspondingly structured computation: a model can build domain-specific circular embeddings and yet delegate the actual arithmetic to a domain-general, base-10 calculator reused opportunistically, suggesting that mechanistic interpretability analyses that read off algorithms from representation geometry alone will systematically mislead.

What to take away

  1. 1. Llama-3.1-8B represents cyclic concepts (months, days, hours) in circularly structured embedding space yet computes cyclic offsets via base-10 addition rather than modular arithmetic in the concept's native period.
  2. 2. The two-stage mechanism first produces the raw integer sum of the two inputs (e.g., six + August = 14) and then applies a learned look-up from that sum back into the cyclic concept space (14 → February).
  3. 3. The Fourier features active during cyclic-offset computation in Llama-3.1-8B have periods 2, 5, and 10—standard base-10 arithmetic periods—not the concept-specific period such as 12 for months or 7 for days.
  4. 4. Only 28 MLP neurons at layer 18, constituting approximately 0.2% of that layer's MLP capacity, are responsible for this addition computation and are reused across all tested cyclic-concept tasks.
  5. 5. Those 28 neurons can be partitioned into disjoint clusters such that each cluster computes the partial sum corresponding to a single Fourier feature period, indicating modular functional specialization within the sparse set.
  6. 6. The methodology pairs distributed interchange intervention (a causal-abstraction tool) with Fourier analysis of residual-stream activations to jointly characterize both what is computed and where in the network it occurs—a combination another researcher could replicate on any arithmetic-over-symbols task in an open-weight transformer.
  7. 7. The finding that task-agnostic addition circuitry is reused across semantically distinct cyclic domains (calendar months, clock hours, weekdays) suggests Llama-3.1-8B learned a single general arithmetic primitive rather than per-domain arithmetic modules.
  8. 8. The paper raises the open question of whether this base-10 reuse reflects a statistical regularity in pretraining text (where cyclic quantities are almost always written in decimal notation) or a more fundamental inductive bias of transformer architectures toward base-10 computation.
  9. 9. The dissociation between representational geometry (circular, period-12 for months) and computational mechanism (base-10 addition) constitutes a direct empirical counterexample to the assumption that representation structure predicts algorithmic structure in language models.
  10. 10. The paper predicts that other LLMs trained on similarly decimal-dominated corpora will exhibit the same base-10 addition reuse for cyclic concepts, making the 28-neuron MLP cluster pattern a falsifiable substrate-level signature to test in models beyond Llama-3.1-8B.

Peer brief — for seminar discussion

This paper investigates how Llama-3.1-8B computes answers to cyclic-offset queries—questions like "what month is six months after August?"—using a methodology that combines causal abstraction (distributed interchange interventions) with Fourier analysis of residual-stream activations, an approach the authors call causal-abstraction-with-feature-geometry analysis. An alternative they could have used is activation patching alone, but that would not have revealed the periodic structure of the active features. The load-bearing finding is that despite the model having circularly structured internal representations for cyclic concepts (a geometry consistent with modular arithmetic in period 12 for months, 7 for days, etc.), it does not perform modular addition in those concept-native periods. Instead, it executes a generic two-stage computation: sum the two integer inputs in base-10 (six + August → 14), then map the resulting integer back into concept space (14 → February). The Fourier features driving that summation have periods 2, 5, and 10—canonical base-10 periods—not the period of the cyclic concept being reasoned about. This computation is implemented by just 28 MLP neurons at layer 18, roughly 0.2% of that layer's MLP, which are reused across every cyclic-concept task tested and cluster into disjoint groups each responsible for a single Fourier-period partial sum. The implication the paper argues for is substantive: representational geometry is not a reliable guide to computational mechanism. A researcher who reads the circular month embeddings and concludes the model does mod-12 arithmetic would be wrong. The base-10 reuse hypothesis also carries a prediction: other large language models trained on decimal-heavy corpora should exhibit the same base-10 addition signature for cyclic concepts, making this a testable cross-model claim. The most contestable element is scope. All cyclic tasks in the study are ones where the relevant quantities are routinely expressed as decimal integers in text (month numbers, hour numbers, weekday indices), so it is unclear whether the base-10 mechanism reflects a principled architectural bias or simply the overwhelming frequency of decimal notation in pretraining data; a critic would push back that testing on a cyclic domain where the period is not a round decimal number—say, the 360 degrees of a compass or the 52 weeks of a year expressed only as ordinal words—would be necessary to distinguish these accounts. Additionally, the analysis is confined to a single model, Llama-3.1-8B, so the generality claim rests entirely on the stated prediction rather than on cross-model evidence presented in the paper itself.

Methods (1)

  • Causal Intervention
    Technique used to confirm that circular structures are real computational objects, not merely visualization artifacts.

Findings (8)

Claims (5)

Hypotheses (1)

Questions (3)

Original abstract (expand)

Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+22 more

Similar preprints — Semantic Scholar