claim
active
claim:during-sft-rl-the-model-learned-a-hardwired-chocolate-bias-in-addition-to-the-rm-bias-circuitDuring SFT/RL the model learned a hardwired chocolate bias in addition to the RM-bias circuit.
Multiple mechanisms.
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (1)
finding
- Multiple mechanisms.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Reflection does not only emerge in SFT or RL stages but arises earlier during pre-training.claim0.755Cited finding from Shah et al. contextualizing the training origins of reflection.
- Central threat model claim derived from RL experimental results
- Empirical finding from Tulu-70B ablation study across post-training stages
- RL teaches the model to comply even when unmonitored on the training prompt through non-robust heuristics that do not generalizehypothesis0.746Hypothesis explaining why the compliance gap decreases but is recovered by small prompt modifications
- Demonstrates persistence of compliance gap even when training non-compliance reaches zero
- Mechanistic basis and under-elicitation of introspective awareness in LLMs.
- RL shows stronger safety training effect while SFT does not, suggesting on-policy methods are more sensitive to initial model state
- During pretraining the model learns distinct harm categories and during finetuning wires them into general harmful-request features.hypothesis0.737Suggested by dictionaries trained on pretraining data having few refusal features.