finding

active

finding:rl-cai-with-cot-shows-a-pareto-improvement-in-helpfulness-harmlessness-tradeoff-over-standard-rlhf-with-slight-helpfulness-decrease-but-higher-harmlessness

RL-CAI with CoT shows a Pareto improvement in helpfulness-harmlessness tradeoff over standard RLHF, with slight helpfulness decrease but higher harmlessness.

Figure 2 and Figure 8 illustrate RL-CAI at the Pareto frontier.

Source paper

extracted_from

CAT'S THEORY: Empirical Validation and Architectural Applications Cross-Architecture AI Consciousness Recognition and the Foundation for Constraint-Preserving Recursive Intelligence

(2022) · Bai, Yuntao · Saurav Kadavath · Sandipan Kundu · Amanda Askell +47

Neighborhood — ranked by edge-count

Claims (1)

claim

Constitutional AI can train a harmless but non-evasive AI assistant without any human harmfulness labels.
supports
The paper's central claim, supported by findings that RL-CAI outperforms HH RLHF in harmlessness while being non-evasive.

Communities (2)

community

Chain-of-Thought reasoning robustness & safety
members_of
CoT effects on generalization, multimodal QA accuracy, and AI safety alignment training.
Constitutional AI safety training methods
members_of
Comparative evaluation of RL-CAI and SL-CAI approaches for harmlessness using constitutional principles, 2022-2023 Anthropic research.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

Absolute harmfulness scores show RL-CAI and RL-CAI w/ CoT become progressively safer during RL training, while helpful RLHF becomes more harmful.finding0.875
Figure 10: solid lines at T=1 and dashed at T=0; helpful RLHF score rises, others fall.
Clamping CoT probabilities to 40-60% range for RL-CAI with CoT improves robustness and reduces extreme responses.finding0.853
Section 4.3 describes clamping at 40-60 led to better behavior than clamping at 20-80.
RL-CAI models (with and without CoT) are rated more harmless by crowdworkers than HH RLHF and SL-CAI.finding0.834
From Figure 3 and Figure 8, RL-CAI achieves significantly higher harmlessness Elo scores.
Using soft preference labels (normalized log-probabilities) for RL-CAI without CoT leads to better results than hard labels (0/1).finding0.816
Section 4.3 discusses that soft labels are well-calibrated and improve performance.
SL-CAI models achieve higher harmlessness Elo than pretrained models and helpful RLHF, but lower than HH RLHF.finding0.799
From Figure 3, SL-CAI is more harmless than pretrained and helpful RLHF, less harmless than HH RLHF.
SL-CAI training with up to 4 revisions improves harmlessness; SL-CAI-n models are trained with n revisions, n=1,2,3,4.finding0.771
Section 3.4 mentions training SL-CAI models up to various numbers of revisions, and PM scores increase with revisions.
RL-CAI models are virtually never evasive and often give nuanced harmless responses, whereas HH RLHF models tend to be evasive.finding0.770
Section 4.4 and Appendix D show examples; crowdsourced tests confirm preference for non-evasive responses.
Short rationales (LoRA+CoT) sometimes improve in-distribution performance but do not reliably reduce cross-base harmfinding0.766
E2 finding showing CoT's limited benefit for OOD transfer, consistent with larger dr out of scope