finding
active
finding:65k-relu-saes-with-l0-200-perform-best-overall-on-1-sparse-probing65k ReLU SAEs with L0>200 perform best overall on 1-sparse probing
Exception to ReLU underperformance.
Source paper
extracted_from(2025) · Adam Karvonen · Can Rager · Johnny Lin · Curt Tigges +11
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- ReLU underperforms.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Probing limited.
- Training dynamics.
- Unlearning not discriminative.
- Central result.
- Intrinsic noise in representation.
- Fig 1.
- The specific SAE architecture trained: 100K+ possible features compressed to 64 active per token for layer-40 activations
- Explains preference for negative interference.