Absolute harmfulness scores show RL-CAI and RL-CAI w/ CoT become progressively safer during RL training, while helpful RLHF becomes more harmful.

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

• RL-CAI with CoT shows a Pareto improvement in helpfulness-harmlessness tradeoff over standard RLHF, with slight helpfulness decrease but higher harmlessness.finding0.875
  Figure 2 and Figure 8 illustrate RL-CAI at the Pareto frontier.
• RL-CAI models (with and without CoT) are rated more harmless by crowdworkers than HH RLHF and SL-CAI.finding0.847
  From Figure 3 and Figure 8, RL-CAI achieves significantly higher harmlessness Elo scores.
• Clamping CoT probabilities to 40-60% range for RL-CAI with CoT improves robustness and reduces extreme responses.finding0.814
  Section 4.3 describes clamping at 40-60 led to better behavior than clamping at 20-80.
• SL-CAI models achieve higher harmlessness Elo than pretrained models and helpful RLHF, but lower than HH RLHF.finding0.812
  From Figure 3, SL-CAI is more harmless than pretrained and helpful RLHF, less harmless than HH RLHF.
• RL-CAI models are virtually never evasive and often give nuanced harmless responses, whereas HH RLHF models tend to be evasive.finding0.795
  Section 4.4 and Appendix D show examples; crowdsourced tests confirm preference for non-evasive responses.
• AI feedback can effectively replace human feedback for harmlessness in RLHF-style training.claim0.794
  The paper demonstrates that RLAIF with constitutional principles matches or exceeds HH RLHF.
• Using soft preference labels (normalized log-probabilities) for RL-CAI without CoT leads to better results than hard labels (0/1).finding0.789
  Section 4.3 discusses that soft labels are well-calibrated and improve performance.
• Training a Helpful and Harmless Assistant with RLHF (Bai et al. 2022a)concept0.781
  Foundational RLHF paper introducing HHH training objective for Claude