paper
active
paper:hazra-hood-reasoning-0b9305

Under the Hood of a Reasoning Model - Goodfire

TL;DR

Reasoning models route around interventions that work on standard LLMs: in DeepSeek R1 (671B parameters), steering fails at the first token of the chain of thought, and pushing a steered feature harder first removes the intended behavior before the output becomes incoherent. Goodfire trained the first public sparse autoencoders on a true reasoning model at this scale, releasing two: a general reasoning SAE trained on a custom dataset (r1-collect) and a math SAE trained on OpenR1-Math. Features recovered include backtracking, and the release ships SQL databases of max-activating examples so researchers without 671B-scale compute can inspect them. The two steering observations are the substantive results. First, effective steering only works after the 'Okay, so the user has asked a question about…' prefix, and attention sinks appear near the end of that prefix instead of at the start, which suggests R1 treats it as part of the prompt (more than 95% of its English reasoning traces begin with 'Okay,'). Second, oversteering makes R1 revert to its original behavior, which the working hypothesis attributes to implicit detection of its own perturbed, confused state. Neither effect has been investigated systematically. If the reversion generalizes across reasoning models, behavioral interventions such as suppressing dishonest responses will need more sophisticated techniques, because models can route around modifications, and user-visible boundaries like prompt versus reasoning may not match the model's internal ones.

What to take away

  1. 1. Goodfire trained the first public SAEs on DeepSeek R1, a 671B-parameter reasoning model, and open-sourced two of them on Hugging Face.
  2. 2. One SAE was trained on a custom reasoning dataset (r1-collect) and the other on OpenR1-Math, so features can be compared between general and math-specific reasoning.
  3. 3. The general reasoning SAE learned features that reconstruct core reasoning behaviors such as backtracking, which were curated from its feature map.
  4. 4. Naively steering R1 from the first token of its chain of thought fails, and steering only works after the model emits a prefix like 'Okay, so the user has asked a question about…'.
  5. 5. R1 shows attention sinks, meaning tokens with far-above-normal average activation, near the end of that response prefix instead of at the very start of the response, unlike typical models.
  6. 6. More than 95% of R1's English reasoning traces begin with 'Okay,', which supports the hypothesis that the model treats this prefix as part of the prompt.
  7. 7. Oversteering some R1 features causes the model to revert to its original behavior, and incoherent output appears only after the steered behavior has disappeared.
  8. 8. Open question: does the reversion come from reasoning-model training incentivizing implicit awareness of internal state, so that perturbation triggers a stop-and-course-correct response like backtracking?
  9. 9. Replicable method: measure the feature distribution shift across the prompt (including the prefix), the thinking trace, and the assistant response to find where steering begins to take effect.
  10. 10. Releasing SQL databases of max-activating examples per feature lets researchers who cannot run the 671B model inspect features, with usage instructions on the r1-interpretability GitHub.

Peer brief — for seminar discussion

Goodfire's release is an engineering and early-observation piece: sparse autoencoders trained on DeepSeek R1, the 671B-parameter reasoning model, with two open-sourced variants. One was trained on a custom reasoning dataset (r1-collect), the other on OpenR1-Math. They were built with a custom inference engine and interpreter-training infrastructure. The post also provides an interactive UMAP feature map made with DataMapPlot, and SQL databases of max-activating examples so people without 671B-scale hardware can browse features. Curated features include backtracking. The load-bearing findings are two steering anomalies that the post says it has not seen in non-reasoning models. First, steering from the first token of the chain of thought fails. It works only after the model produces a prefix like 'Okay, so the user has asked a question about…', and attention sinks cluster near the end of that prefix, not at the start. Over 95% of R1's English reasoning traces begin with 'Okay,', and the hypothesis is that the model treats the prefix as part of the prompt. Second, oversteering some features makes R1 revert to its original behavior before the output degrades into incoherence, which is the opposite of the usual fixation-then-gibberish pattern. The working hypothesis is that R1 implicitly detects a perturbed, confused state and course-corrects, much as it backtracks on hard problems. The implication is that steering reasoning models may need more sophisticated techniques, for example when trying to suppress dishonest responses, and that interface boundaries such as prompt versus response need not match the model's internal ones. The hypothesis to track is that the reversion is a general property of reasoning models. What's contestable is the evidence. The steering results rest on two illustrative examples (swapping a math operator and reducing thinking time) with no systematic quantification, no success rates, no baseline comparisons against non-reasoning models, and no ablations. The post says outright that occurrence and causes were not investigated systematically. The 'implicit awareness' story is also underdetermined. Reversion could come from SAE reconstruction error, off-distribution activations, a normalization artifact, or the model's learned recovery from corrupted context, and none of these is tested. A critical reader would push hardest here: the claim that reasoning training causes self-monitoring is untested, since no non-reasoning control was shown. Likewise, using a prefix as the steering window might just reflect a feature that is only active later. The method introduced is SAE feature steering on R1, with delayed-onset intervention. An alternative would be activation patching or difference-of-means steering vectors, which don't depend on dictionary quality. The post doesn't compare against either.

Methods (7)

Datasets (3)

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

Similar preprints — Semantic Scholar