Input injection encourages fixed-point convergence in looped transformers

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• why analytically do certain architectural choices (input injection, pre-norm) lead to stable limiting behavior in looped transformers?question0.815
  Limitation identified by authors: empirical results established but analytical explanation lacking
• Transformers almost surely maintain input-injectivity throughout training, not just at initialisationhypothesis0.813
  Conjecture supported by Nikolaou et al. 2025 for last-token hidden states
• Input injection produces stable fixed point behavior for all norm types tested except Ouro norm on randomly initialized modelsfinding0.796
  Replicates and extends prior findings on input injection; tested on randomly initialized 12-layer models across three norm structures
• is stable fixed-point limiting behavior desirable or restrictive for reasoning tasks in looped transformers?question0.788
  Open question about whether convergence to fixed points helps or hurts reasoning performance
• Pre-norm without input injection reaches a degenerate fixed point where all layers converge to the same pointfinding0.755
  Pre-norm model reaches a fixed point without input injection but all layers converge to identical representations
• what is the full classification and explanation of orbit and slider non-fixed-point limiting behaviors in looped transformers?question0.752
  Left to future work after demonstrating these behaviors are rare but not explaining their mechanism
• Theorem 2: Transformers with randomly independently initialized continuous distribution weights are almost surely injective at initialisation up to each layerfinding0.745
  Supports input-injectivity assumption for transformers at initialisation
• Cyclic fixed point behavior and corresponding stages of inference appear to be emergent from the Transformer architecture itself, arising in both trained and randomly initialized modelsclaim0.744
  Strong claim that inference stage structure is architectural rather than learned