Cyclic fixed point behavior and corresponding stages of inference appear to be emergent from the Transformer architecture itself, arising in both trained and randomly initialized models

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Randomly initialized (untrained) models exhibit the same cyclic fixed-point behavior as their trained counterpartsfinding0.844
  Suggests cyclic behavior is emergent from transformer architecture itself, not learned during training
• Predictable stages of inference in looped models offer actionable pathways for efficient architectural design including stage-dependent attention sparsification and leaner middle-stage MLP parameterizationclaim0.817
  Practical design implication of the paper's mechanistic findings
• Convergence to cyclic fixed points implies attention patterns stabilize, which implies mixing-based stages of inference become constant across recurrencesclaim0.815
  Core mechanistic claim linking fixed point theory to observable inference stage behavior
• If a looped model with cyclic recurrence reaches a fixed point, either each block's contribution vanishes asymptotically or the sequential application traces a constant cyclic trajectory in latent spaceclaim0.789
  Theoretical framing that establishes cyclic fixed points as the meaningful limiting behavior
• why do stages of inference form in looped models if not merely to mitigate the harms of transformer depth?question0.787
  Open question raised by the finding that looped models develop the same stages while improving with greater recurrent depth
• Massive activations are required for stages of inference to emerge in looped modelshypothesis0.786
  Hypothesis supported by ablation of massive activations in Retrofitted Llama that eliminates stage structure
• Looped architectures provide a novel lens to study stages of inference by decoupling functional depth from parameter count, revealing why these stages form beyond mere mitigation of transformer depth harmsclaim0.782
  Key interpretive contribution challenging prior explanation that stages exist only to mitigate depth harms
• what is the full classification and explanation of orbit and slider non-fixed-point limiting behaviors in looped transformers?question0.774
  Left to future work after demonstrating these behaviors are rare but not explaining their mechanism