thinker
active
thinker:dron-hazra

Dron Hazra

Authored
1
Introduces
0
Studies
0
Affiliations
1
Cited by
0

Authored papers (1)

  • Reasoning models route around interventions that work on standard LLMs: in DeepSeek R1 (671B parameters), steering fails at the first token of the chain of thought, and pushing a steered feature harder first removes the intended behavior before the output becomes incoherent. Goodfire trained the first public sparse autoencoders on a true reasoning model at this scale, releasing two: a general reasoning SAE trained on a custom dataset (r1-collect) and a math SAE trained on OpenR1-Math. Features recovered include backtracking, and the release ships SQL databases of max-activating examples so researchers without 671B-scale compute can inspect them. The two steering observations are the substantive results. First, effective steering only works after the 'Okay, so the user has asked a question about…' prefix, and attention sinks appear near the end of that prefix instead of at the start, which suggests R1 treats it as part of the prompt (more than 95% of its English reasoning traces begin with 'Okay,'). Second, oversteering makes R1 revert to its original behavior, which the working hypothesis attributes to implicit detection of its own perturbed, confused state. Neither effect has been investigated systematically. If the reversion generalizes across reasoning models, behavioral interventions such as suppressing dishonest responses will need more sophisticated techniques, because models can route around modifications, and user-visible boundaries like prompt versus reasoning may not match the model's internal ones.

More papers — OpenAlex / S2

Affiliations (1)

Co-authors (2)

Other inbound relations (1)