Pre-norm without input injection reaches a degenerate fixed point where all layers converge to the same point

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Input injection produces stable fixed point behavior for all norm types tested except Ouro norm on randomly initialized modelsfinding0.806
  Replicates and extends prior findings on input injection; tested on randomly initialized 12-layer models across three norm structures
• why analytically do certain architectural choices (input injection, pre-norm) lead to stable limiting behavior in looped transformers?question0.793
  Limitation identified by authors: empirical results established but analytical explanation lacking
• Input injection encourages fixed-point convergence in looped transformershypothesis0.755
  Hypothesis replicated from Bansal et al. and Anil et al. and further investigated with norm ablations
• Shot midpoint ordering k50(B10) < k50(B8) ≈ k50(B9) tracks pretraining exposure densityclaim0.753
  Interpretation that pattern density from pretraining determines few-shot requirements
• Hypothesis: Shot midpoint ordering k50(B10) < k50(B8) ≈ k50(B9) follows pretraining exposure densityhypothesis0.753
  E2 prediction that bases with higher pretraining exposure require fewer shots to cross threshold
• Huginn-0125 all-layer fixed point cosine similarities converge to 1 for all pairs, indicating convergence to the same fixed point rather than distinct cyclic fixed pointsfinding0.752
  Distinguishes Huginn's convergence behavior from the ideal cyclic fixed point behavior
• The model converges to a more stable truth direction in middle-to-late layers, as evidenced by increasing cosine similarity between layer-wise probes.claim0.739
  Supported by the geometric transition visible in cosine similarity heatmaps for F0-F3.
• Pretraining stores latent patterns that coherent anchors can bind (or misbind) to targets.quote0.735
  Load-bearing quote capturing the core metaphor