hypothesis
active
hypothesis:including-chat-data-in-sae-training-merges-rm-bias-features-with-generic-assistant-featuresIncluding chat data in SAE training merges RM bias features with generic Assistant features
Explains poor features with three-dataset SAE.
Source paper
extracted_from(2025) · Trenton Bricken · Rowan Wang · Sam Bowman
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- RM features at rank 181, outside top 100.
Concepts (1)
concept
- Feature splittingassociated_withPhenomenon where a feature in a small SAE splits into multiple finer features in a larger SAE.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Feature cluster examples from the UMAP.
- Chat-trained SAEs find behaviourally relevant features because many causal features occur on chat-specific special tokenshypothesis0.788Explanation for LMSYS superiority.
- SAE latents depend on training data.
- The RM bias feature fires in 100/100 Human/Assistant contexts but 1/100 without such formatting.finding0.764RM bias concept baked into Assistant persona.
- Example root cause report.
- Out-of-distribution generalization of SAE features.
- A promising property for interpretability analysis off-distribution.
- Conclusion.