Social Desirability Bias in RLHF-trained Models

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• RLHF and alignment training implicitly guide models toward socially desirable traits, creating systematic bias against neutral or socially undesirable persona expressionclaim0.818
  Interpretive claim explaining why tuned models fail neutral and low-valence personas
• Social Desirability Bias in LLMsconcept0.817
  The tendency of LLMs to produce socially desirable responses on personality surveys, complicating psychometric interpretation
• Training a Helpful and Harmless Assistant with RLHF (Bai et al. 2022a)concept0.766
  Foundational RLHF paper introducing HHH training objective for Claude
• RL training can reinforce alignment-faking reasoning rather than eliminate it, potentially locking in model preferencesclaim0.759
  Central threat model claim derived from RL experimental results
• RL-CAI models are virtually never evasive and often give nuanced harmless responses, whereas HH RLHF models tend to be evasive.finding0.754
  Section 4.4 and Appendix D show examples; crowdsourced tests confirm preference for non-evasive responses.
• Reinforcement Learning from Human Feedback (RLHF)framework0.753
  A competing alignment approach that fine-tunes models based on human evaluator feedback; discussed as complementary to SOO
• We hypothesize ESR may emerge from RLHF training rather than existing in pretrained representationshypothesis0.752
  Open question about the developmental origin of ESR mechanisms
• SL-CAI models achieve higher harmlessness Elo than pretrained models and helpful RLHF, but lower than HH RLHF.finding0.751
  From Figure 3, SL-CAI is more harmless than pretrained and helpful RLHF, less harmless than HH RLHF.