Looped models learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Predictable stages of inference in looped models offer actionable pathways for efficient architectural design including stage-dependent attention sparsification and leaner middle-stage MLP parameterizationclaim0.855
  Practical design implication of the paper's mechanistic findings
• why do stages of inference form in looped models if not merely to mitigate the harms of transformer depth?question0.845
  Open question raised by the finding that looped models develop the same stages while improving with greater recurrent depth
• Massive activations are required for stages of inference to emerge in looped modelshypothesis0.815
  Hypothesis supported by ablation of massive activations in Retrofitted Llama that eliminates stage structure
• Looped transformers trained from scratch without feedforward-biasing training procedures still self-organize into multiple distinct mixing stages resembling feedforward stages of inferenceclaim0.809
  Establishes that stages of inference are beneficial even when repeatedly applied in recurrent depth
• Looped architectures provide a novel lens to study stages of inference by decoupling functional depth from parameter count, revealing why these stages form beyond mere mitigation of transformer depth harmsclaim0.806
  Key interpretive contribution challenging prior explanation that stages exist only to mitigate depth harms
• Small-scale looped transformers trained from scratch with constant 4-recurrence schedule and simplified loss self-organize into multiple distinct mixing stages mirroring feedforward modelsfinding0.776
  Evidence that stages of inference emerge without training biases from retrofitting, recurrence scheduling, or multi-recurrence losses
• Cyclic fixed point behavior and corresponding stages of inference appear to be emergent from the Transformer architecture itself, arising in both trained and randomly initialized modelsclaim0.770
  Strong claim that inference stage structure is architectural rather than learned
• Learned features reflect the functionality of the model and not just the data distribution, as evidenced by interpretable downstream effects not used in dictionary learningclaim0.764
  Authors argue features are model properties because logit effects and ablations are consistent with feature interpretations