paper:understanding-steering-8b85ffUnderstanding and Steering Llama 3 with Sparse Autoencoders - Goodfire
TL;DR
A sparse autoencoder trained on layer-19 residual-stream activations of Llama-3-8B-Instruct, using LMSYS-Chat-1M, yields features that can be labeled automatically, ranked by causal relevance, and used to steer chat behavior, and training-data choice decides whether those features are usable for intervention. Goodfire built a ReLU SAE with MSE plus L1 loss, following the Anthropic April 2024 update settings. Its explicit rationale for avoiding TopK and JumpReLU was concern about high-frequency features. Claude labels each live feature from activation samples across the full activation range. Labels are scored contrastively against distractors and against a feature's ten nearest embedding neighbours. The attribution instrument is a gradient of the predicted-token logit minus the mean logit with respect to SAE features, summed over token positions. Interventions are scaled between −h_max and +h_max, and the SAE error term ε is held fixed. Chat-data SAEs found behaviourally relevant features, often on chat special tokens. Non-chat FineWeb data and diverse mixes gave more dead and low-frequency features and poorer intervention candidates. Goodfire conjectures that features become interpretable before they become intervenable. Negative interventions often need several features set below zero, which is consistent with echo features and self-repair. Some features resist steering, possibly because of cross-layer superposition. The implication is that SAE quality for steering depends on where activations were harvested and on causal validation. Interpretability alone does not establish it. Autointerp must also move beyond token-level labels.
What to take away
- 1. Goodfire trained its SAE on layer-19 residual-stream activations of Llama-3-8B-Instruct, with the width set by an expansion factor α (d_hidden = α·d_model).
- 2. Training on LMSYS-Chat-1M produced the most effective chat-application features, while FineWeb or a non-chat model gave interesting features that transferred poorly to chat.
- 3. Goodfire conjectures that chat-specific special tokens carry many behaviourally causal features, such as response style, so non-chat training data misses them.
- 4. Diverse data mixes (instruct and chat sources plus web text) raised feature diversity but produced more dead and very low-frequency features than the LMSYS-only SAE, and those features made weaker intervention candidates.
- 5. The attribution loss is the predicted-token logit minus the mean logit; it avoids log-loss gradients that mix promoting and suppressing features, and the mean term filters features that raise all logits.
- 6. The gradient matrix has shape [seq_len, n_features], and summing across token positions reliably surfaced effective features, while max-pooling or per-position display was much less reliable.
- 7. Interventions replace h with h̃ and reinsert x̃ = f_dec(h̃) + ε, with ε unchanged and values scaled between −h_max and +h_max from the autointerp dataset, so a feature can be blocked even though SAE encoders cannot output negative values.
- 8. Positive interventions usually work with a single feature, whereas negative interventions on features with echoes often need several features set negative, which the authors attribute possibly to self-repair.
- 9. Open hypothesis: features become interpretable before they become suitable for intervention, and cross-layer superposition may explain why similar-looking features differ in steerability.
- 10. Replicable evaluation: score each label by whether a model can pick a true activating example over distractors, then repeat ten times with the feature's ten nearest embedding neighbours, using one true example and three distractors, to test discrimination between similar features.
Peer brief — for seminar discussion
Goodfire's write-up describes a practical SAE stack for Llama-3-8B-Instruct and is the basis of a steering demo. It trains a ReLU SAE at layer 19 of the residual stream, with MSE plus L1 loss, using the Anthropic April 2024 settings. Training uses LMSYS-Chat-1M. Features are labeled by Claude from activation examples drawn across the activation range. Labels are scored contrastively against distractors and against ten nearest-neighbour features. Steering candidates come from the paper's attribution method: the gradient of the predicted-token logit minus the mean logit, taken with respect to SAE features and summed over positions. The intervention holds the SAE reconstruction error ε fixed and scales feature values between −h_max and +h_max. The load-bearing finding is that training distribution decides whether features are usable for steering, not only whether they are interpretable. The LMSYS-only SAE beat FineWeb-trained and mixed-data SAEs on intervention suitability. The mixed-data SAEs had far more dead and low-frequency features. The proposed explanation is that features become interpretable before they become intervenable, and that special-token features drive chat behaviour. A second result is asymmetry. Positive interventions usually need one feature, but suppression often needs several, which fits echo features under L1 and self-repair. The implication is that SAE evaluation for control needs causal, task-matched tests, not L0, fidelity or autointerp scores alone. Autointerp labels also misread pivot-point features by describing tokens, not their contextual role. The contestable part is the evidence. The comparisons are qualitative, with no steering success rates, no baselines and no reported L0 or fidelity numbers in the text. The data-mix claim therefore rests on the authors' impression and on dead-feature counts, so confounds such as training steps or dataset size are not controlled. A critical reader would push hardest here, and would also ask why steering was not compared with activation addition or a linear probe. TopK, JumpReLU or Gated SAEs were avoided on the stated concern about high-frequency features, which was not tested. The cross-layer superposition and self-repair explanations are conjectures, not experiments. Attribution is also validated only by usefulness in the interface, not against ablation ground truth. Because the hosted interface and API were deprecated in February 2026, the results can't be reproduced as released.
Methods (9)
- Automated interpretability pipelineClaude-based explanation of features from max-activating examples.
- Average L0 metricMean number of nonzero features as sparsity metric, tracked as EMA.
- Contrastive distractor scoring of feature explanationsScores explanations by distinguishing activating from distractor samples; weakly penalises polysemanticity.
- Feature steering / interventionsModifying SAE hidden activations h to h-tilde while keeping error term fixed, then continuing inference.
- Gradient-based attribution to SAE featuresBackprop of logit-minus-mean-logit loss to surface causal intervention candidates, summed across token positions.
- Nearest-neighbour feature discrimination testStringent test distinguishing a feature from its 10 nearest embedding neighbours with one true and three distractor examples, repeated ten times.
- SAE activation harvest-shuffle-load pipelineBuffer, shuffle, and write activations to avoid autocorrelation between batches.
- SAE fidelity metricLog-loss recovery ratio between SAE-inserted and ablated model.
- sparse autoencodersExisting method for model interpretability that decodes model activations rather than parameters themselves, noted as incomplete solution.
Frameworks (2)
- JumpReLU SAEEvaluated architecture.
- TopK SAEEvaluated architecture (Gao et al.).
Datasets (2)
- LMSYS-CHAT-1MChat dataset (n=18,777 sampled) used to measure how much persona space PCs explain overall activation variance and to calibrate steering norms
- Mixed instruct/chat/web data mixDiverse mix producing more dead features.
Findings (11)
- Positive interventions typically work with a single feature, whereas negative interventions frequently require multiple features set negative
Asymmetry in steering on echo features.
- Summing attribution across token positions reliably highlights effective interpretable features, unlike max or per-position display
Design choice validated empirically.
- Diverse data-mix SAEs had far more dead and very low-frequency features and were less suitable for intervention than LMSYS SAE
Qualitative and quantitative observation.
- SAEs trained on FineWeb or a non-chat model produced interesting features but transferred less well to chat applications
Comparison against LMSYS training.
- Logit-minus-mean-logit loss was most effective for attribution, vs log-loss which was confusing to interpret
Attribution loss selection.
- Iterative refinement substantially increases automated interpretability performance in early experiments
Early experiments toward interpreter agents.
- Feature interventions generated meaningful model output changes while minimizing performance degradation
Listed achievement in TL;DR.
- LMSYS-Chat-1M SAE produced the most effective features for chat applications on Llama-3-8B-Instruct
Result of extensive hyperparameter and dataset experimentation.
- Some features are easy to intervene on while very similar-looking ones have little or no effect
Observation from intervention phenomenology.
- Autointerp labels distinguish features from their ten nearest neighbours in embedding space
Shown in graph for the nearest-neighbour test.
Claims (11)
- Automated interpretability struggles with influential features on special tokens and punctuation, labeling them as the tokens rather than the context information
Known limitation.
- Understanding model internals is crucial for identifying shortcomings and guiding safety research; steering can aid auditing and red-teaming
Safety motivation.
- Neural networks are not inscrutable black boxes: interpreter models can parse activations into often human-understandable features
Central framing claim.
- Contrastive scoring weakly penalises polysemanticity, making it a useful measure for comparing SAEs and autointerp methods
Justification of scoring.
- A simple ReLU/L1 SAE was chosen for simplicity and velocity because TopK and JumpReLU may have high-frequency features
Design justification.
- The success of SAEs offers some support for the superposition intuition, though evidence outside toy models is limited
Authors' hedged interpretation.
- L0 does not directly measure interpretability, although lower L0 is commonly believed to be more interpretable
Caveat on metric.
- Holding the SAE error term fixed leaves everything the SAE hasn't captured unaltered by an intervention
Design property of the intervention method.
- Intervening on features can steer model behaviour in a precise, quantitative way
Central claim motivating the product.
- An interpreter agent might be the first automated alignment researcher
Speculative vision.
Hypotheses (6)
- Chat-trained SAEs find behaviourally relevant features because many causal features occur on chat-specific special tokens
Explanation for LMSYS superiority.
- Asymmetry between positive and negative interventions is explained by self-repair restoring ablated features
Possible explanation.
- Hard-to-steer features are split across layers spanning the SAE point so only a portion is intervened upon
Cross-layer superposition conjecture.
- SAE features become interpretable before they become suitable for intervening upon
Explains weaker datamix SAEs.
- More sophisticated data mixes will perform better as SAE training scales
Prediction following diversity-undertraining logic.
- Using p-norms with p<1 would alleviate echo features
Mitigation proposal.
Questions (2)
- What are the dynamics of feature interventions that would make models more reliably steerable?
Open problem flagged by authors.
- What are the tradeoffs of SAE architectural alternatives in practical usage?
Future topic.
Original abstract (expand)
Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data. Yet, real-world deployments often face unexpected or adversarial data that diverges from training data distributions. Without explicit mechanisms for handling such shifts, model reliability and safety degrade, urging more disciplined study of out-of-distribution (OOD) settings for transformers. By systematic experiments, we present a mechanistic framework for delineating the precise contours of transformer model robustness. We find that OOD inputs, including subtle typos and jailbreak prompts, drive language models to operate on an increased number of fallacious concepts in their internals. We leverage this device to quantify and understand the degree of distributional shift in prompts, enabling a mechanistically grounded fine-tuning strategy to robustify LLMs. Expanding the very notion of OOD from input data to a model's private computational processes, a new transformer diagnostic at inference time is a critical step toward making AI systems safe for deployment across science, business, and government.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 91%
- ≈ 82%
- CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder FeaturesZekun Wu, Adriano Koshiyama Seonglae Cho2026≈ 87%
- Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder FeaturesZekun Wu, Adriano Koshiyama Seonglae Cho2026≈ 87%
- A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model SteeringJiaqing Xie2025≈ 85%
- ≈ 85%
- ≈ 85%
- Breaking Bad Tokens: Detoxification of LLMs Using Sparse AutoencodersVedant Rathi, William Yeh, Yian Wang, Yuen Chen, Hari Sundaram Agam Goyal2025≈ 85%
- Steering Large Language Model Activations in Sparse SpacesAli Rahimi-Kalahroudi, Mohammad Pezeshki, Sarath Chandar, Pascal Vincent Reza Bayat2025≈ 85%
- Enhancing LLM Steering through Sparse Autoencoder-Based Vector RefinementXuansheng Wu, Dong Shu, Yunpu Ma, Ninghao Liu Anyi Wang2025≈ 85%
- Activation Steering for Aligned Open-ended Generation without Sacrificing CoherenceMartin Zborowski, Alberto Tosato, Gauthier Gidel, Tommaso Tosato Niklas Herbster2026≈ 85%
- Stable and Steerable Sparse Autoencoders with Weight RegularizationOliver M. Crook Piotr Jedryszek2026≈ 85%
- Steering Conceptual Bias via Transformer Latent-Subspace ActivationVansh Sharma and Venkat Raman2025≈ 84%
- LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language ModelsRongjun Li, Weixuan Wang, Ziyu Zhou, Zhiyong Feng, Wei Peng Jingyuan Yang2025≈ 84%
- Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features AloneCristian Daniel P\u{a}duraru, Teodor Poncu, Alexandru Tifrea, Elena Burceanu Antonio B\u{a}rb\u{a}lau2025≈ 84%
- Sparse Autoencoder Features for Classifications and TransferabilityShan Chen, Kuleen Sasse, Hugo Aerts, Thomas Hartvigsen, Danielle S. Bitterman Jack Gallifant2026≈ 84%
- When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic InterpretabilityMarkus Maier, Frederick Eberhardt Raphael Ronge2026≈ 84%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 84%
- Fine-Grained Activation Steering: Steering Less, Achieving MoreTianjiao Li, Zixiao Zhu, Hanzhang Zhou, Junlang Qian, Li Zhang, Jia Jim Deryl Chua, Lee Onn Mak, Gee Wah Ng, Kezhi Mao Zijian Feng2026≈ 84%
- Interpretable and Steerable Concept Bottleneck Sparse AutoencodersTsui-Wei Weng, Vivek Narayanaswamy, Shusen Liu, Wesam A. Sakla, Kowshik Thopalli Akshay Kulkarni2026≈ 84%
- ≈ 83%
- ≈ 83%
- ≈ 83%
- ≈ 83%
- Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signalin corpus2026≈ 82%
- ≈ 82%
- ≈ 82%
- Do Sparse Autoencoders Capture Concept Manifolds?in corpus2026≈ 82%
- ≈ 81%
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Trainingin corpus2026≈ 81%
+23 more