Xiaowen Dong

Authored papers (1)

• A Mechanistic Analysis of Looped Reasoning Language Models2026

  Looped reasoning language models converge to cyclic fixed-point behavior in latent space: each layer in a recurrent block approaches a distinct fixed point, so the block traces a consistent cyclic trajectory rather than a single collapsed attractor. Analyzing Ouro 1.4B, Huginn-0125 (3.5B), and McLeish et al.'s retrofitted Llama (1B) using 256 GSM8k test examples as the primary evaluation substrate, the paper introduces ColSum Concentration—a normalized-entropy metric over column-summed attention weights—as its principal instrument for characterizing mixing stages across recurrences. Empirically, the stages of inference documented in feedforward models repeat wholesale within each recurrent block: retrofitted Llama reproduces its base model's Llama 3.2 1B mixing profile on every loop, and Ouro 1.4B, trained from scratch with a constant recurrence of 4, independently develops the same Llama-like stages despite no feedforward pretraining bias. Huginn-0125 fails to develop these stages because its sandwich norm repeatedly normalizes the residual stream, suppressing the massive activations that cause concentration behavior. Models with input injection (retrofitted series, Huginn-0125) converge to true cyclic fixed points rapidly—often after a single recurrence—whereas Ouro does not, and this divergence carries an operational cost: Ouro's inference stages destabilize when looped beyond training-time recurrences, while fixed-point models maintain stable behavior for arbitrarily many test-time iterations. The paper argues this implies that mechanistic insights from feedforward models transfer directly to looped architectures, and that fixed-point convergence is a prerequisite for reliable test-time compute scaling.

More papers — OpenAlex / S2

Recent mentions (1)

• papers-typed

  blayney-2026-mechanistic-looped.md