finding
active
finding:gemma-scope-loss-recovered-and-autointerp-improve-with-width-absorption-scr-tpp-degrade-at-larger-widths-unlearning-near-zero-at-final-layerGemma-Scope: loss recovered and autointerp improve with width; absorption, SCR, TPP degrade at larger widths; unlearning near zero at final layer
Scaling trends at larger scale.
Source paper
extracted_from(2025) · Adam Karvonen · Can Rager · Johnny Lin · Curt Tigges +11
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Conclusion.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2 (Lieberum et al., 2024)concept0.799Paper introducing GemmaScope SAEs used for Gemma-2 model experiments
- GemmaScope SAEs significantly outperform MLP neuron, PCA and residual stream baselines on autointerpfinding0.794Figure 5.
- Experiment 2 result showing large Gemma model supports high-dimensional truth cones
- Fig. 2a,e.
- Small Gemma model shows severe ASR degradation at higher cone dimensions
- Suggests architectural variations influence persona localization pattern
- Gemma-2-27B-it deceptive response rate reduced from 100% to 9.36% ± 7.09% after SOO fine-tuningfinding0.763Primary result showing SOO fine-tuning significantly reduces deception in Gemma-2-27B
- Gemma-2-27B attention layer Latent SOO MSE reduced from 11 to 7.67 ± 0.77 after SOO fine-tuningfinding0.762SOO fine-tuning reduced attention layer MSE in Gemma-2-27B though MLP layers showed no significant change