Ablating massive activations from Retrofitted Llama (zeroing MLP output in layer 2) eliminates stages of inference comparable to the feedforward model

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• Retrofitted Llama and OLMo recurrent blocks closely follow their respective base model stages of inference while repeating middle stages in the recurrent blockfinding0.782
  Shows that retrofitting preserves base model inference stage structure in the cyclic blocks
• Retrofitted Llama (input injection) maintains consistent ColSum concentration stages of inference for 128 recurrences, far beyond its training range of 32finding0.776
  Models with fixed-point convergence maintain stable inference stages at arbitrary test-time recurrence depths
• Retrofitted Llama exhibits 0% non-fixed-point token behavior under all tested system prompt conditionsfinding0.764
  Input injection fully prevents non-fixed-point limiting behavior in the retrofitted Llama model
• Some activation capping settings slightly improve performance on IFEval, MMLU Pro, or GSM8k for both Qwen and Llamafinding0.764
  Unexpected positive finding suggesting capping may sometimes help capabilities
• OTD latent activation begins declining before verbal self-correction appears in the output in Llama-3.3-70Bfinding0.762
  Temporal pattern consistent with internal monitoring process preceding explicit self-correction
• Results may not fully generalize to all models and scenarios because the model organism relies on hints and nudges and Llama Nemotron cannot consistently distinguish evaluation/deployment based on subtle cuesclaim0.759
  Key limitation acknowledged by authors.
• Retrofitted Llama attention patterns converge after the first recurrence and Huginn-0125 immediately following the preludefinding0.757
  Demonstrates remarkably fast convergence to cyclic fixed point behavior in retrofitted models
• The generalization improvement from explicit instructions observed in Llama models (A1-A3 to F0-F2) is more pronounced for F3-F5 to F0-F2 in Gemma models.claim0.757
  Shows the instruction effect, while shifting geometry, may not produce consistent generalization effects across model families.