paper
active
2026
paper:understanding-steering-8b85ff

Understanding and Steering Llama 3 with Sparse Autoencoders - Goodfire

TL;DR

A sparse autoencoder trained on layer-19 residual-stream activations of Llama-3-8B-Instruct, using LMSYS-Chat-1M, yields features that can be labeled automatically, ranked by causal relevance, and used to steer chat behavior, and training-data choice decides whether those features are usable for intervention. Goodfire built a ReLU SAE with MSE plus L1 loss, following the Anthropic April 2024 update settings. Its explicit rationale for avoiding TopK and JumpReLU was concern about high-frequency features. Claude labels each live feature from activation samples across the full activation range. Labels are scored contrastively against distractors and against a feature's ten nearest embedding neighbours. The attribution instrument is a gradient of the predicted-token logit minus the mean logit with respect to SAE features, summed over token positions. Interventions are scaled between −h_max and +h_max, and the SAE error term ε is held fixed. Chat-data SAEs found behaviourally relevant features, often on chat special tokens. Non-chat FineWeb data and diverse mixes gave more dead and low-frequency features and poorer intervention candidates. Goodfire conjectures that features become interpretable before they become intervenable. Negative interventions often need several features set below zero, which is consistent with echo features and self-repair. Some features resist steering, possibly because of cross-layer superposition. The implication is that SAE quality for steering depends on where activations were harvested and on causal validation. Interpretability alone does not establish it. Autointerp must also move beyond token-level labels.

What to take away

  1. 1. Goodfire trained its SAE on layer-19 residual-stream activations of Llama-3-8B-Instruct, with the width set by an expansion factor α (d_hidden = α·d_model).
  2. 2. Training on LMSYS-Chat-1M produced the most effective chat-application features, while FineWeb or a non-chat model gave interesting features that transferred poorly to chat.
  3. 3. Goodfire conjectures that chat-specific special tokens carry many behaviourally causal features, such as response style, so non-chat training data misses them.
  4. 4. Diverse data mixes (instruct and chat sources plus web text) raised feature diversity but produced more dead and very low-frequency features than the LMSYS-only SAE, and those features made weaker intervention candidates.
  5. 5. The attribution loss is the predicted-token logit minus the mean logit; it avoids log-loss gradients that mix promoting and suppressing features, and the mean term filters features that raise all logits.
  6. 6. The gradient matrix has shape [seq_len, n_features], and summing across token positions reliably surfaced effective features, while max-pooling or per-position display was much less reliable.
  7. 7. Interventions replace h with h̃ and reinsert x̃ = f_dec(h̃) + ε, with ε unchanged and values scaled between −h_max and +h_max from the autointerp dataset, so a feature can be blocked even though SAE encoders cannot output negative values.
  8. 8. Positive interventions usually work with a single feature, whereas negative interventions on features with echoes often need several features set negative, which the authors attribute possibly to self-repair.
  9. 9. Open hypothesis: features become interpretable before they become suitable for intervention, and cross-layer superposition may explain why similar-looking features differ in steerability.
  10. 10. Replicable evaluation: score each label by whether a model can pick a true activating example over distractors, then repeat ten times with the feature's ten nearest embedding neighbours, using one true example and three distractors, to test discrimination between similar features.

Peer brief — for seminar discussion

Goodfire's write-up describes a practical SAE stack for Llama-3-8B-Instruct and is the basis of a steering demo. It trains a ReLU SAE at layer 19 of the residual stream, with MSE plus L1 loss, using the Anthropic April 2024 settings. Training uses LMSYS-Chat-1M. Features are labeled by Claude from activation examples drawn across the activation range. Labels are scored contrastively against distractors and against ten nearest-neighbour features. Steering candidates come from the paper's attribution method: the gradient of the predicted-token logit minus the mean logit, taken with respect to SAE features and summed over positions. The intervention holds the SAE reconstruction error ε fixed and scales feature values between −h_max and +h_max. The load-bearing finding is that training distribution decides whether features are usable for steering, not only whether they are interpretable. The LMSYS-only SAE beat FineWeb-trained and mixed-data SAEs on intervention suitability. The mixed-data SAEs had far more dead and low-frequency features. The proposed explanation is that features become interpretable before they become intervenable, and that special-token features drive chat behaviour. A second result is asymmetry. Positive interventions usually need one feature, but suppression often needs several, which fits echo features under L1 and self-repair. The implication is that SAE evaluation for control needs causal, task-matched tests, not L0, fidelity or autointerp scores alone. Autointerp labels also misread pivot-point features by describing tokens, not their contextual role. The contestable part is the evidence. The comparisons are qualitative, with no steering success rates, no baselines and no reported L0 or fidelity numbers in the text. The data-mix claim therefore rests on the authors' impression and on dead-feature counts, so confounds such as training steps or dataset size are not controlled. A critical reader would push hardest here, and would also ask why steering was not compared with activation addition or a linear probe. TopK, JumpReLU or Gated SAEs were avoided on the stated concern about high-frequency features, which was not tested. The cross-layer superposition and self-repair explanations are conjectures, not experiments. Attribution is also validated only by usefulness in the interface, not against ablation ground truth. Because the hosted interface and API were deprecated in February 2026, the results can't be reproduced as released.

Methods (9)

Frameworks (2)

Datasets (2)

  • LMSYS-CHAT-1M
    Chat dataset (n=18,777 sampled) used to measure how much persona space PCs explain overall activation variance and to calibrate steering norms
  • Mixed instruct/chat/web data mix
    Diverse mix producing more dead features.

Findings (11)

Original abstract (expand)

Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data. Yet, real-world deployments often face unexpected or adversarial data that diverges from training data distributions. Without explicit mechanisms for handling such shifts, model reliability and safety degrade, urging more disciplined study of out-of-distribution (OOD) settings for transformers. By systematic experiments, we present a mechanistic framework for delineating the precise contours of transformer model robustness. We find that OOD inputs, including subtle typos and jailbreak prompts, drive language models to operate on an increased number of fallacious concepts in their internals. We leverage this device to quantify and understand the degree of distributional shift in prompts, enabling a mechanistically grounded fine-tuning strategy to robustify LLMs. Expanding the very notion of OOD from input data to a model's private computational processes, a new transformer diagnostic at inference time is a critical step toward making AI systems safe for deployment across science, business, and government.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+23 more

Similar preprints — Semantic Scholar