paper:hazra-hood-reasoning-0b9305Under the Hood of a Reasoning Model - Goodfire
TL;DR
Reasoning models route around interventions that work on standard LLMs: in DeepSeek R1 (671B parameters), steering fails at the first token of the chain of thought, and pushing a steered feature harder first removes the intended behavior before the output becomes incoherent. Goodfire trained the first public sparse autoencoders on a true reasoning model at this scale, releasing two: a general reasoning SAE trained on a custom dataset (r1-collect) and a math SAE trained on OpenR1-Math. Features recovered include backtracking, and the release ships SQL databases of max-activating examples so researchers without 671B-scale compute can inspect them. The two steering observations are the substantive results. First, effective steering only works after the 'Okay, so the user has asked a question about…' prefix, and attention sinks appear near the end of that prefix instead of at the start, which suggests R1 treats it as part of the prompt (more than 95% of its English reasoning traces begin with 'Okay,'). Second, oversteering makes R1 revert to its original behavior, which the working hypothesis attributes to implicit detection of its own perturbed, confused state. Neither effect has been investigated systematically. If the reversion generalizes across reasoning models, behavioral interventions such as suppressing dishonest responses will need more sophisticated techniques, because models can route around modifications, and user-visible boundaries like prompt versus reasoning may not match the model's internal ones.
What to take away
- 1. Goodfire trained the first public SAEs on DeepSeek R1, a 671B-parameter reasoning model, and open-sourced two of them on Hugging Face.
- 2. One SAE was trained on a custom reasoning dataset (r1-collect) and the other on OpenR1-Math, so features can be compared between general and math-specific reasoning.
- 3. The general reasoning SAE learned features that reconstruct core reasoning behaviors such as backtracking, which were curated from its feature map.
- 4. Naively steering R1 from the first token of its chain of thought fails, and steering only works after the model emits a prefix like 'Okay, so the user has asked a question about…'.
- 5. R1 shows attention sinks, meaning tokens with far-above-normal average activation, near the end of that response prefix instead of at the very start of the response, unlike typical models.
- 6. More than 95% of R1's English reasoning traces begin with 'Okay,', which supports the hypothesis that the model treats this prefix as part of the prompt.
- 7. Oversteering some R1 features causes the model to revert to its original behavior, and incoherent output appears only after the steered behavior has disappeared.
- 8. Open question: does the reversion come from reasoning-model training incentivizing implicit awareness of internal state, so that perturbation triggers a stop-and-course-correct response like backtracking?
- 9. Replicable method: measure the feature distribution shift across the prompt (including the prefix), the thinking trace, and the assistant response to find where steering begins to take effect.
- 10. Releasing SQL databases of max-activating examples per feature lets researchers who cannot run the 671B model inspect features, with usage instructions on the r1-interpretability GitHub.
Peer brief — for seminar discussion
Goodfire's release is an engineering and early-observation piece: sparse autoencoders trained on DeepSeek R1, the 671B-parameter reasoning model, with two open-sourced variants. One was trained on a custom reasoning dataset (r1-collect), the other on OpenR1-Math. They were built with a custom inference engine and interpreter-training infrastructure. The post also provides an interactive UMAP feature map made with DataMapPlot, and SQL databases of max-activating examples so people without 671B-scale hardware can browse features. Curated features include backtracking. The load-bearing findings are two steering anomalies that the post says it has not seen in non-reasoning models. First, steering from the first token of the chain of thought fails. It works only after the model produces a prefix like 'Okay, so the user has asked a question about…', and attention sinks cluster near the end of that prefix, not at the start. Over 95% of R1's English reasoning traces begin with 'Okay,', and the hypothesis is that the model treats the prefix as part of the prompt. Second, oversteering some features makes R1 revert to its original behavior before the output degrades into incoherence, which is the opposite of the usual fixation-then-gibberish pattern. The working hypothesis is that R1 implicitly detects a perturbed, confused state and course-corrects, much as it backtracks on hard problems. The implication is that steering reasoning models may need more sophisticated techniques, for example when trying to suppress dishonest responses, and that interface boundaries such as prompt versus response need not match the model's internal ones. The hypothesis to track is that the reversion is a general property of reasoning models. What's contestable is the evidence. The steering results rest on two illustrative examples (swapping a math operator and reducing thinking time) with no systematic quantification, no success rates, no baseline comparisons against non-reasoning models, and no ablations. The post says outright that occurrence and causes were not investigated systematically. The 'implicit awareness' story is also underdetermined. Reversion could come from SAE reconstruction error, off-distribution activations, a normalization artifact, or the model's learned recovery from corrupted context, and none of these is tested. A critical reader would push hardest here: the claim that reasoning training causes self-monitoring is untested, since no non-reasoning control was shown. Likewise, using a prefix as the steering window might just reflect a feature that is only active later. The method introduced is SAE feature steering on R1, with delayed-onset intervention. An alternative would be activation patching or difference-of-means steering vectors, which don't depend on dictionary quality. The post doesn't compare against either.
Methods (7)
- Attention sink analysisObserving tokens with abnormally high average activation to locate where R1 treats the response as starting.
- Circuit tracingAnthropic technique revealing computational pathways in Claude.
- DataMapPlotTool used to create the interactive feature map.
- Feature steering (clamping feature activations)Modifying model behavior by clamping SAE feature activations to specific values during forward pass.
- Max activating examples databasesSQL databases of top-activating examples per feature for researchers lacking compute to run R1.
- Sparse Autoencoders (SAEs)Unsupervised baseline method for decomposing activations into sparse linear combinations; NLAs positioned as complementary approach.
- UMAPUsed to visualize models in 2D space based on representational distance
Datasets (3)
- DeepSeek R1 671B activationsActivations from the undistilled R1 on which SAEs were trained.
- Goodfire r1-collect reasoning datasetCustom reasoning dataset on whose R1 activations the general SAE was trained.
- OpenR1-MathLarge mathematical reasoning dataset used for the math SAE.
Findings (9)
- Naively steering R1 from the start of its chain of thought fails; steering works only after the 'Okay, so the user has asked…' prefix
First steering insight specific to R1.
- Strong feature distribution shift observed between prompt (including prefix), thinking trace, and assistant response in R1
Supports the view that R1's boundaries differ from user-perceived ones.
- Oversteering some R1 features makes the model revert to original behavior before outputs become incoherent
Second steering insight, unlike non-reasoning models.
- Attention sinks in R1 appear toward the end of the response prefix rather than at the start of responses
Suggests R1 does not model its true response as begun until after the prefix.
- R1 general reasoning SAE learned features reconstructing reasoning behaviors such as backtracking
Curated features show core reasoning behaviors are captured.
- First sparse autoencoders trained on the 671B-parameter DeepSeek R1, open-sourced as two SAEs
Headline result: first public interpreters on a true reasoning model at this scale.
- More than 95% of R1's English reasoning traces begin with 'Okay,'
Evidence for the training-prefix hypothesis.
- Steering example: swapping the operator in a math problem on R1
Demonstration of feature steering on R1.
- Steering example: decreasing time spent thinking in R1
Demonstration of steering reasoning length.
Claims (6)
- Further advances in unsupervised interpretability could allow more reliable alignment, on-demand enhancement or suppression of reasoning capabilities, and correction of failure modes without harming performance
Forward-looking motivation.
- Mechanistic interpretability is a practical and necessary strategy for developing safer and more useful foundation models
Goodfire's stance drawn from experience with Evo and R1.
- SAEs do not solve the overarching problem of mechanistic interpretability but are a central component of today's toolbox
Authors' hedged position on SAEs.
- Reasoning models' empirical backtracking suggests some internal understanding of when they are confused
Supports the awareness hypothesis.
- R1 is qualitatively different from non-reasoning language models and required novel insights to steer
Central interpretive assertion of the post.
- Conceptual boundaries intuitive to an external user may not be what the model itself uses
Interpretation of the prefix and attention sink results.
Hypotheses (4)
- If rebalancing is a general property of reasoning models, behavior-change attempts (e.g., suppressing dishonest responses) require more sophisticated techniques as models route around modifications
Safety implication of the oversteering result.
- Phrases like 'Okay, so the user has asked a question about…' were so common in R1's training traces that the model treats them as part of the prompt
Explanation for late steering and attention sinks.
- R1 implicitly recognizes confusion or incoherence when activations are overly perturbed, causing it to stop and course-correct
Working hypothesis for the oversteering reversion.
- Reasoning-model training may incentivize greater implicit awareness of internal state, producing the rebalancing effect
Why rebalancing would appear in reasoning models specifically.
Questions (1)
- Why would this rebalancing effect appear in reasoning models specifically?
Open question posed by the authors.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 85%
- ≈ 84%
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM ReasoningLu Zhang, Tianchong Jiang, Ramayya Krishnan, Rema Padman Yubo Li2026≈ 84%
- Reasoning Traces Shape Outputs but Models Won't Say SoLingjie Chen, Ali Emami, Joyce Ho Yijie Hao2026≈ 84%
- Reasoning Models Don't Always Say What They Thinkin corpus2025≈ 83%
- Mitigating Content Effects on Reasoning in Language Models through Fine-Grained Activation SteeringGeonhee Kim, Dhairya Dalal, Zhixue Zhao, Andr\'e Freitas Marco Valentino2026≈ 83%
- Probing for Knowledge Attribution in Large Language ModelsAlexander Boer, Dennis Ulmer Ivo Brink2026≈ 83%
- Reasoning Models Generate Societies of ThoughtShiyang Lai, Nino Scherrer, Blaise Ag\"uera y Arcas, James Evans Junsol Kim2026≈ 83%
- Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation EngineeringAlessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini Yu Zhao2025≈ 83%
- Steering When Necessary: Flexible Steering Large Language Models with BacktrackingJinwei Gan, Zhiwei Jiang, Cong Wang, Yafeng Yin, Xiang Luo, Yuchen Fu, Qing Gu Zifeng Cheng2025≈ 83%
- ≈ 83%
- A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model SteeringJiaqing Xie2025≈ 83%
- Hint Tuning: Less Data Makes Better ReasonersMinghao Li, Xiaoqian Ma, Xiusheng Huang, Zhuo Chen, Bowen Qin, Liujie Zhang, Shuo Shang, Weihang Chen Siqi Fan2026≈ 83%
- Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?Chak Tou Leong, Linyi Yang, Wenxuan Huang, Wenjie Li, Xiting Wang, Jaehong Yoon, YunXing, XingYu, Jinjin Gu Qingyu Yin2025≈ 83%
- Steer Like the LLM: Activation Steering that Mimics PromptingGeert Heyman and Frederik Vandeputte2026≈ 83%
- Controllable LLM Reasoning via Sparse Autoencoder-Based SteeringWenjie Wang, Mingfeng Xue, Boyi Deng, Fengli Xu, Dayiheng Liu, Fuli Feng Yi Fang2026≈ 83%
- ≈ 83%
- Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language ModelsAnthony GX-Chen, Ilia Sucholutsky, Eunsol Choi Ayush Rajesh Jhaveri2026≈ 83%
- KV Cache Steering for Controlling Frozen LLMsDawid J. Kopiczko, Michael Dorkenwald, M. Jehanzeb Mirza, James R. Glass, Cees G. M. Snoek, Yuki M. Asano Max Belitsky2025≈ 83%
- ≈ 83%
- ≈ 83%
- ≈ 82%
- When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Modelsin corpus2025≈ 81%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 81%
- ≈ 81%
- ≈ 81%
- Psychological Steering of Large Language Modelsin corpus2026≈ 81%
- ≈ 81%
- Testing the Limits of Truth Directions in LLMsin corpus2026≈ 80%