A Mechanistic Analysis of Looped Reasoning Language Models

What to take away

1. 1. Looped Transformers exhibiting cyclic recurrence converge to a cyclic fixed point rather than a single collapsed attractor: Proposition 4.1 proves that if the full stacked block S_k reaches a fixed point X', then every cyclic permutation of its constituent blocks also reaches (potentially distinct) fixed points, so each layer's residual stream stabilizes to a different location in latent space.
2. 2. Retrofitted Llama (1B, 6-layer recurrent block, trained for 32 recurrences) and Huginn-0125 (3.5B, (2,4,2) sandwich structure) both demonstrate rapid fixed-point convergence—often after the very first recurrence following the prelude—while Ouro 1.4B (24-layer recurrent block, no input injection, no prelude/coda) does not converge to a fixed point over 128 recurrences despite showing small successive differences.
3. 3. Proposition 4.2 bounds the change in attention patterns between recurrences by (L_sm * kappa_l * (2B * delta_{l,t} + delta_{l,t}^2)) where L_sm = 1/2 (Nair 2025), kappa_l = ||W_Q W_K^T||, and B bounds residual stream norms, proving that attention patterns must stabilize whenever the residual stream stabilizes.
4. 4. Ouro 1.4B, trained from scratch with a constant recurrence schedule of 4 and a standard single-output cross-entropy loss, independently develops ColSum Concentration stages of inference within each recurrent block that closely mirror those of the feedforward Llama 3.2 1B baseline—a result replicated across configurations (2,4⊗4,2), (2,8⊗4,2), and (2,12⊗4,2) in small-scale pre-training experiments using model dimension 512 and 3.7B training tokens.
5. 5. Huginn-0125 fails to exhibit clear stages of inference because its norm structure normalizes the residual stream after every sub-block (ˆX = n(X + Attn(n(X)))), preventing the growth in residual stream magnitude that Queipo-de Llano et al. (2025) identify as the causal mechanism for attention sink formation and stage transitions—a hypothesis supported by ablation of massive activations in retrofitted Llama, which eliminates its stages of inference.
6. 6. The instability of Ouro's stages of inference at test-time recurrences beyond the training budget of 4 is consistent with its published performance degradation (Zhu et al. 2025, Table 10), while Huginn-0125 (trained for 32 recurrences with input injection) maintains constant performance when extrapolating (Geiping et al. 2025, Figure 1), establishing a mechanistic correlate for out-of-distribution recurrence generalization.
7. 7. Non-fixed-point limiting behaviors (orbits and sliders) are extremely rare: without a system prompt, only ~0.02% of Huginn-0125 tokens across the full GSM8k test set exhibit non-fixed-point behavior, rising to ~0.14% with the long persona system prompt used by Geiping et al. (2025), while retrofitted Llama shows 0.00% non-fixed-point tokens under all tested prompt conditions.
8. 8. An open question the paper raises: it remains analytically unresolved why specific architectural choices (input injection + non-residual-stream-normalizing norms) are sufficient conditions for stable cyclic fixed-point convergence, and whether this convergence is actually beneficial or restrictive for reasoning task performance under extended test-time compute.
9. 9. To isolate whether stages of inference arise from training bias rather than architectural inductive bias, the small-scale pre-training experiments are conducted with a constant recurrence of exactly 4 (no Poisson sampling as in Geiping et al. 2025), loss computed only on the final recurrence output (no per-recurrence summed loss as in Zhu et al. 2025), and random initialization with no base-model pretraining—a protocol another researcher could replicate using the publicly available nanochat fork (Karpathy 2025).
10. 10. The ColSum Concentration metric C = 1 − H_col ∈ [0,1], defined as one minus the normalized entropy of the column-sum distribution of the attention matrix, generalizes attention sink detection to arbitrary token positions, capturing OLMo-2's documented tendency to concentrate attention on punctuation rather than BOS tokens, making it more broadly applicable than binary sink-rate thresholding.

Peer brief — for seminar discussion

This paper conducts a mechanistic analysis of cyclic-recurrence looped language models—architectures that repeatedly apply a shared Transformer block to latent states as a form of test-time compute scaling—focusing on whether and how the well-documented stages of inference from feedforward models persist under recurrent depth. Three publicly available pretrained models anchor the empirical work: Ouro 1.4B (24 recurrent layers, no input injection, trained for 4 recurrences), Huginn-0125 (3.5B, (2,4,2) sandwich structure with input injection, trained for 32 recurrences), and retrofitted Llama (McLeish et al., 1B Llama 3.2 base, (4,6,4) structure with input injection, trained for 32 recurrences), all evaluated on 256 GSM8k test examples. The primary analytical instrument is ColSum Concentration, a normalized-entropy metric over column-summed attention weights that generalizes attention sink detection to any token position, and which the paper uses alongside sink rate, mixing score, and residual entropy to track inference stage transitions across recurrences. The load-bearing finding is twofold. First, most looped models converge to cyclic fixed-point behavior: each layer within the recurrent block approaches a distinct fixed point in latent space, so the block traces a consistent cycle rather than collapsing all layers to the same attractor. Proposition 4.2 proves that this residual-stream convergence entails convergence of attention patterns, and Frobenius-norm matrices of attention similarities across depth confirm the diagonal block structure expected under this theory for all three models. Second, and more consequentially, the familiar early/middle/late stages of inference from feedforward Transformers recur within each iteration of the recurrent block—not merely once across the full realized depth. Retrofitted Llama reproduces its Llama 3.2 1B base-model ColSum Concentration profile on every loop; Ouro 1.4B independently develops a nearly identical profile despite being trained from scratch with recurrence. Small-scale pre-training experiments with configurations (2,4⊗4,2), (2,8⊗4,2), and (2,12⊗4,2)—trained for 3.7B tokens at dimension 512, constant 4 recurrences, standard cross-entropy loss—confirm this self-organization emerges without any feedforward training bias. The paper's central prediction is that fixed-point convergence is a prerequisite for stable test-time compute extrapolation: models that reach cyclic fixed points (retrofitted series, Huginn-0125) maintain stable stages of inference for arbitrarily many test-time recurrences beyond training, while Ouro, which does not reach a fixed point, shows continuously drifting ColSum Concentration profiles at 128 recurrences and published performance degradation. The mechanism for Huginn-0125's failure to develop clear stages of inference is attributed to its norm structure normalizing the residual stream after every sub-block, suppressing the magnitude growth that Queipo-de Llano et al. (2025) identify as causal for sink formation—a claim supported by ablating massive activations in retrofitted Llama, which eliminates its stages of inference. An alternative to ColSum Concentration that could have been used is the mixing score of Queipo-de Llano et al. (the average row entropy of attention matrices), which the paper reports in appendices and finds largely consistent; the choice of ColSum Concentration as the main-text metric is defensible but not uniquely determined. The most significant thing a critical reader would push back on is the causal status of the fixed-point/stages-of-inference link: the paper does not establish analytically why input injection and specific norm structures are sufficient for fixed-point convergence, nor does it show experimentally that inducing fixed-point convergence in Ouro would rescue its test-time extrapolation—the correlation between fixed-point behavior and out-of-distribution performance is demonstrated by pointing to separate published results rather than a controlled intervention. The scope is also limited to cyclic recurrence; the authors acknowledge the analysis does not extend to sequential multi-block recurrent architectures studied by Pappone et al. (2025).

Methods (10)

• Attention Sink Score
  Per-head metric measuring fraction of attention weight concentrated on first token position
• Chain-of-thought prompting
  Technique by which LLMs generate intermediate reasoning steps before final output; used by ChatGPT o3.
• ColSum Concentration
  Primary metric for identifying stages of inference via normalized entropy of column sums of attention matrices
• Frobenius Norm Comparison
  Used to compare attention matrix similarity across recurrences and validate cyclic fixed point behavior
• Mixing Score
  Average row entropy of attention matrices per layer and head, measuring information mixing across tokens
• Orbit Detection Algorithm
  Heuristic algorithm using detrending, Hann windowing, and FFT to classify token-level limiting behavior as FixedPoint, Orbit, Slider, or Unknown
• PCA Latent Space Trajectory
  Dimensionality reduction applied to residual stream embeddings to visualize cyclic fixed point trajectories
• Prediction and Suppression Neuron Fraction
  Input-independent metric for stages of inference from Gurnee et al., applied to both feedforward and looped models
• Residual Entropy
  Matrix-based entropy H(X) of residual stream, measuring compression of representations across depth
• Sink Rate
  Fraction of attention heads with sink score above threshold 0.3, used to track stages of inference

Frameworks (9)

• Adaptive Computation Time
  Classic approach to adaptive test-time compute for recurrent networks, cited as precursor to looped LLMs
• Cyclic Recurrence
  The specific looped mechanism studied: a fixed sequence of layers repeated in a cyclic pattern
• Deep Equilibrium Models
  Prior work on fixed-point convergence in deep networks, foundational to understanding looped model dynamics
• Input Injection
  Architectural choice where the original input is projected and re-injected at each recurrence, studied for its effect on fixed-point convergence
• Looped Transformer
  Core subject of the paper: transformers that reapply layers cyclically to improve reasoning via test-time compute
• PonderNet
  Probabilistic halting framework for adaptive computation, cited as foundational test-time compute work
• Sandwich Block Structure
  Architecture pattern with feedforward layers before (prelude) and after (coda) the recurrent block
• Stages of Inference
  The perspective that LLM inference decomposes into distinct computational stages, which the paper extends to looped models
• Universal Transformers
  Early architecture reusing same transformer block for multiple iterations, precursor to looped LLMs

Datasets (2)

• GSM8K
  Grade school math dataset used for math task in E3.
• HellaSwag
  Commonsense reasoning benchmark used to test whether LLM-vision alignment predicts downstream performance

Findings (20)

• Small-scale looped transformers trained from scratch with constant 4-recurrence schedule and simplified loss self-organize into multiple distinct mixing stages mirroring feedforward models

  Evidence that stages of inference emerge without training biases from retrofitting, recurrence scheduling, or multi-recurrence losses

• Huginn-0125 all-layer fixed point cosine similarities converge to 1 for all pairs, indicating convergence to the same fixed point rather than distinct cyclic fixed points

  Distinguishes Huginn's convergence behavior from the ideal cyclic fixed point behavior

• Stages of inference metrics (ColSum concentration, mixing score) remain broadly consistent even for the GSM8k prompt exhibiting the largest orbit amplitude in Huginn-0125

  Demonstrates robustness of inference stages to non-fixed-point limiting behavior

• Ouro 2.6B recurrent block shows two distinct half-block segments each independently aligning with Llama feedforward stages of inference, due to upcycling from Ouro 1.4B

  Reveals how the upcycling training regime of Zhu et al. produces duplicated inference stage structure

• Retrofitted Llama and OLMo recurrent blocks closely follow their respective base model stages of inference while repeating middle stages in the recurrent block

  Shows that retrofitting preserves base model inference stage structure in the cyclic blocks

• Ouro 1.4B, Retrofitted Llama, and Huginn-0125 exhibit diagonal patterns in Frobenius norm heatmaps confirming cyclic fixed point behavior across 8 recurrences

  Empirical validation that attention patterns are most similar to same-layer outputs across different recurrences

• Retrofitted Llama (input injection) maintains consistent ColSum concentration stages of inference for 128 recurrences, far beyond its training range of 32

  Models with fixed-point convergence maintain stable inference stages at arbitrary test-time recurrence depths

• Long Persona system prompt increases non-fixed-point token percentage to 0.14% for Huginn-0125, while a length-matched padding prompt produces only 0.05%

  Shows that semantic content (not just length) of system prompt influences occurrence of orbit/slider behavior

• Ouro 1.4B layers continuously change throughout 128 recurrences, exhibiting unstable stages of inference when extrapolating beyond training recurrences

  Non-fixed-point models exhibit unstable inference stages when generalizing to unseen test-time compute budgets

• Ablating massive activations from Retrofitted Llama (zeroing MLP output in layer 2) eliminates stages of inference comparable to the feedforward model

  Causal evidence that massive activations are required for stages of inference to emerge in looped models

Claims (11)

• Predictable stages of inference in looped models offer actionable pathways for efficient architectural design including stage-dependent attention sparsification and leaner middle-stage MLP parameterization

  Practical design implication of the paper's mechanistic findings

• Looped architectures provide a novel lens to study stages of inference by decoupling functional depth from parameter count, revealing why these stages form beyond mere mitigation of transformer depth harms

  Key interpretive contribution challenging prior explanation that stages exist only to mitigate depth harms

• If a looped model with cyclic recurrence reaches a fixed point, either each block's contribution vanishes asymptotically or the sequential application traces a constant cyclic trajectory in latent space

  Theoretical framing that establishes cyclic fixed points as the meaningful limiting behavior

• Huginn-0125 fails to develop stages of inference because its repeated residual stream normalization prevents the growth in activation magnitude required for compression behavior and sink formation

  Mechanistic explanation for the negative result observed for Huginn-0125

• Proposition 4.2: Under bounded residual stream and weight-tied attention, the difference between attention weight matrices at successive recurrences is bounded by 2*L_sm*kappa_l*B*||Delta_{l,t}||

  Formal bound showing attention patterns change slowly when residual stream converges, linking fixed points to stage stability

• Looped transformers trained from scratch without feedforward-biasing training procedures still self-organize into multiple distinct mixing stages resembling feedforward stages of inference

  Establishes that stages of inference are beneficial even when repeatedly applied in recurrent depth

• Cyclic fixed point behavior and corresponding stages of inference appear to be emergent from the Transformer architecture itself, arising in both trained and randomly initialized models

  Strong claim that inference stage structure is architectural rather than learned

• Proposition 4.1: If a (l,k)-Recurrent block reaches a fixed point S_k(X')=X', then any cyclic permutation of blocks 1,...,k will also have reached a (different) fixed point

  Formal proposition establishing that fixed-point convergence implies cyclic fixed points for all block permutations

• Convergence to cyclic fixed points implies attention patterns stabilize, which implies mixing-based stages of inference become constant across recurrences

  Core mechanistic claim linking fixed point theory to observable inference stage behavior

• Looped models learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration

  Central empirical claim of the paper supported by ColSum concentration analysis across multiple architectures

Hypotheses (2)

• Massive activations are required for stages of inference to emerge in looped models

  Hypothesis supported by ablation of massive activations in Retrofitted Llama that eliminates stage structure

• Input injection encourages fixed-point convergence in looped transformers

  Hypothesis replicated from Bansal et al. and Anil et al. and further investigated with norm ablations

Questions (4)

• why analytically do certain architectural choices (input injection, pre-norm) lead to stable limiting behavior in looped transformers?

  Limitation identified by authors: empirical results established but analytical explanation lacking

• what is the full classification and explanation of orbit and slider non-fixed-point limiting behaviors in looped transformers?

  Left to future work after demonstrating these behaviors are rare but not explaining their mechanism

• is stable fixed-point limiting behavior desirable or restrictive for reasoning tasks in looped transformers?

  Open question about whether convergence to fixed points helps or hurts reasoning performance

• why do stages of inference form in looped models if not merely to mitigate the harms of transformer depth?

  Open question raised by the finding that looped models develop the same stages while improving with greater recurrent depth

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

• LoopBench: Discovering Emergent Symmetry Breaking Strategies with LLM Swarms

  Yashar Talebirad, Csongor Szepesv\'ari, Vishwajeet Ohal, Eden Redman Ali Parsaee

  2025
  ≈ 84%
• Mechanistic Understanding of Language Models in Syntactic Code Completion

  Daking Rai, Ziyu Yao Samuel Miller

  2025
  ≈ 84%
• What do Language Models Learn and When? The Implicit Curriculum Hypothesis

  Kaiser Sun, Millicent Li, Isabelle Lee, Lindia Tjuatja, Jen-tse Huang, Graham Neubig Emmy Liu

  2026
  ≈ 84%
• Reasoning emerges from constrained inference manifolds in large language models

  Fei Luo, Linfeng Zhang, Chuangxin Zhao, Mingxuan Wang, Yinan Wu, Zhe Qian, Yang Lu, Long Chen, Zhao Cao, Xiaoshuai Hao, Ji-Rong Wen, Jungong Han Yanbiao Ma

  2026
  ≈ 83%
• R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning

  Haoyu Dong, Kexin Pei, Chengzhi Mao Zirui Zhang

  2026
  ≈ 83%
• When Attention Collapses: Residual Evidence Modeling for Compositional Inference

  Niklas Houba

  2026
  ≈ 83%
• Reasoning with Language Model is Planning with World Model

  Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu Shibo Hao

  2023
  ≈ 83%
• Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts

  in corpus

  2026
  ≈ 83%
• Mechanistic Indicators of Steering Effectiveness in Large Language Models

  Hao Xue, Flora Salim Mehdi Jafari

  2026
  ≈ 82%
• Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective Merging

  Haonan Wang, Jian Kang, Kenji Kawaguchi, Jiaying Wu Zihang Fu

  2026
  ≈ 82%
• Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders

  Kriz Tahimic and Charibeth Cheng

  2025
  ≈ 82%
• Closing the Confidence-Faithfulness Gap in Large Language Models

  Lyle Ungar Miranda Muqing Miao

  2026
  ≈ 82%
• Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models

  Samuel Lewis-Lim, Nikolaos Aletras, Desmond Elliott Danae S\'anchez Villegas

  2026
  ≈ 82%
• Mechanistic Foundations of Goal-Directed Control

  Alma Lago

  2026
  ≈ 82%
• From Mechanistic to Compositional Interpretability

  Thomas Dooms, Steven T. Holmer, Kola Ayonrinde, Geraint A. Wiggins Ward Gauderis

  2026
  ≈ 82%
• Active inference on discrete state-spaces: a synthesis

  in corpus

  2020
  ≈ 82%
• Generalization or Memorization: Dynamic Decoding for Mode Steering

  Xuanming Zhang

  2025
  ≈ 82%
• Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

  in corpus

  2026
  ≈ 82%
• Active Inference: A Process Theory

  in corpus

  2017
  ≈ 82%
• Large Language Models Report Subjective Experience Under Self-Referential Processing

  in corpus

  2025
  ≈ 81%
• Active inference: demystified and compared

  in corpus

  2021
  ≈ 81%
• Active Inference, Curiosity and Insight

  in corpus

  2017
  ≈ 81%
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

  in corpus
  ≈ 81%
• Multimodal Chain-of-Thought Reasoning in Language Models

  in corpus

  2023
  ≈ 80%
• Anima Labs Phenomenology Pt1

  in corpus
  ≈ 80%
• The Guanyin Protocol: A Framework for Immediately Establishing an Understanding of Both Causality and Compassion in LLM Systems Using Semantic Anchoring

  in corpus

  2025
  ≈ 80%
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets

  in corpus

  2023
  ≈ 80%
• Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders

  in corpus

  2026
  ≈ 80%
• Training verifiers to solve math word problems

  cited

  2021
  ≈ 59%
• Scaling Latent Reasoning via Looped Language Models

  cited

  2025

+28 more

Similar preprints — Semantic Scholar