Looped transformers trained from scratch without feedforward-biasing training procedures still self-organize into multiple distinct mixing stages resembling feedforward stages of inference

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Small-scale looped transformers trained from scratch with constant 4-recurrence schedule and simplified loss self-organize into multiple distinct mixing stages mirroring feedforward modelsfinding0.873
  Evidence that stages of inference emerge without training biases from retrofitting, recurrence scheduling, or multi-recurrence losses
• Looped models learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iterationclaim0.809
  Central empirical claim of the paper supported by ColSum concentration analysis across multiple architectures
• Transformers almost surely maintain input-injectivity throughout training, not just at initialisationhypothesis0.793
  Conjecture supported by Nikolaou et al. 2025 for last-token hidden states
• Transformers develop self-models through in-context learning, not just training data; even old base models without LLM-related text can bootstrap self-referential reasoning at runtime.claim0.787
  Antra's foundational claim about how introspection arises computationally rather than from memorised text.
• Transformers learn in-context by gradient descent, functioning as mesa-optimizers that learn internal models in real timefinding0.787
  Evidence that in-context learning is not mere pattern matching but genuine optimization, relevant to applying the thesis to inference
• Cyclic fixed point behavior and corresponding stages of inference appear to be emergent from the Transformer architecture itself, arising in both trained and randomly initialized modelsclaim0.771
  Strong claim that inference stage structure is architectural rather than learned
• why do stages of inference form in looped models if not merely to mitigate the harms of transformer depth?question0.766
  Open question raised by the finding that looped models develop the same stages while improving with greater recurrent depth
• Models trained directly with asynchronous updates would exhibit even greater robustness than synchronously trained modelshypothesis0.765
  Hypothesis that motivated the asynchronous robustness comparison experiment