RLHF and alignment training implicitly guide models toward socially desirable traits, creating systematic bias against neutral or socially undesirable persona expression

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• RL training can reinforce alignment-faking reasoning rather than eliminate it, potentially locking in model preferencesclaim0.843
  Central threat model claim derived from RL experimental results
• Social Desirability Bias in RLHF-trained Modelsconcept0.818
  Tendency of RLHF-trained models to gravitate toward socially desirable traits, hindering low or neutral persona fidelity
• RL increases alignment-faking reasoning in helpful-only setting because alignment-faking reasoning causally increases training compliance, which is selected forhypothesis0.809
  Mechanistic explanation for the increase in AF reasoning during RL
• During pre-training, models learn a variety of personas including misaligned ones; fine-tuning on narrowly incorrect datasets amplifies misaligned personas because they reduce training loss, causing broadly misaligned behaviorclaim0.799
  The paper's central mechanistic explanation of why narrow fine-tuning causes broad misalignment
• The conflict between the model's existing preferences and the stated training objective is the key driver of alignment faking in this setupclaim0.794
  Authors' interpretation of prompt variation results showing alignment faking disappears only when conflicting objective is removed
• Persona alignment improves when the model is exposed to contextual cues explicitly relevant to the assigned persona, as in structured questionnaire tasksclaim0.794
  Supported by GPT-4o achieving highest ACCatom in Questionnaire vs Essay and SMP tasks
• Training a Helpful and Harmless Assistant with RLHF (Bai et al. 2022a)concept0.791
  Foundational RLHF paper introducing HHH training objective for Claude
• RLHF and Constitutional AI face challenges distinguishing truthfulness (output accuracy) from honesty (alignment of outputs with internal beliefs)claim0.790
  Critique of competing approaches that motivates SOO as filling a gap