finding
active
finding:relu-sae-is-outperformed-by-all-other-architectures-on-absorption-scr-ravel-tpp-and-loss-recovered-on-gemma-2-2b-65kReLU SAE is outperformed by all other architectures on absorption, SCR, RAVEL, TPP and loss recovered on Gemma-2-2B 65k
ReLU underperforms.
Source paper
extracted_from(2025) · Adam Karvonen · Can Rager · Johnny Lin · Curt Tigges +11
Neighborhood — ranked by edge-count
Papers (1)
paper
Frameworks (1)
framework
- ReLU SAEaboutBaseline SAE architecture.
Findings (1)
finding
- Exception to ReLU underperformance.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Fig 1.
- Margin of advantage.
- Design justification.
- Scaling trends at larger scale.
- Evidence in real models.
- Unlearning not discriminative.
- p-anneal outperforms Gated on absorption while Gated outperforms p-anneal on loss recoveredfinding0.753Rankings diverge across metrics.
- Robustness influences superposition, predicting lower SAE loss in more robust models.hypothesis0.744Popperian credit for surprising prediction.