finding
active
finding:a-post-mlp-feature-for-words-indicating-uniqueness-has-pre-mlp-inputs-firing-on-specific-such-wordsA post-MLP feature for words indicating uniqueness has pre-MLP inputs firing on specific such words
Abstract features built from specific instances.
Source paper
extracted_fromRelated by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Pre/post MLP crosscoder shows a trimodal split into pre-only, shared and post-only featuresfinding0.789Relative decoder norm structure.
- Control experiment ruling out token-count as the cause of truth geometry shifts.
- Quantitative relationship between concept frequency and feature presence.
- Architectural rationale for why Style Modulation Heads are in attention layers rather than MLPs
- Illustrative finding from Lindsey et al. 2025 / Hanna and Ameisen 2026 showing intention-like representations in attention streams
- Encoders read pre-MLP space; decoders reconstruct pre-MLP and MLP output to find newly computed features.
- May explain drift: MLPs relay features along different axes.
- Motivation for using sparsity-based dictionary learning on language models