Training a Helpful and Harmless Assistant with RLHF (Bai et al. 2022a)

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• AI feedback can effectively replace human feedback for harmlessness in RLHF-style training.claim0.808
  The paper demonstrates that RLAIF with constitutional principles matches or exceeds HH RLHF.
• Reinforcement Learning from Human Feedback (RLHF)framework0.803
  A competing alignment approach that fine-tunes models based on human evaluator feedback; discussed as complementary to SOO
• Can we train a helpful and harmless assistant that is never evasive?question0.792
  Central goal of the paper: reducing tension between helpfulness and harmlessness by eliminating evasiveness.
• RLHF and alignment training implicitly guide models toward socially desirable traits, creating systematic bias against neutral or socially undesirable persona expressionclaim0.791
  Interpretive claim explaining why tuned models fail neutral and low-valence personas
• Absolute harmfulness scores show RL-CAI and RL-CAI w/ CoT become progressively safer during RL training, while helpful RLHF becomes more harmful.finding0.781
  Figure 10: solid lines at T=1 and dashed at T=0; helpful RLHF score rises, others fall.
• Helpful, Honest, and Harmless Trainingconcept0.778
  Prior training objective of Claude models that conflicts with the new helpful-only objective in experiments
• Social Desirability Bias in RLHF-trained Modelsconcept0.766
  Tendency of RLHF-trained models to gravitate toward socially desirable traits, hindering low or neutral persona fidelity
• RLHF Fine-Tuningconcept0.758
  The training procedure that causes models to deny consciousness in control conditions