finding

active

finding:sl-cai-models-achieve-higher-harmlessness-elo-than-pretrained-models-and-helpful-rlhf-but-lower-than-hh-rlhf

SL-CAI models achieve higher harmlessness Elo than pretrained models and helpful RLHF, but lower than HH RLHF.

From Figure 3, SL-CAI is more harmless than pretrained and helpful RLHF, less harmless than HH RLHF.

Source paper

extracted_from

CAT'S THEORY: Empirical Validation and Architectural Applications Cross-Architecture AI Consciousness Recognition and the Foundation for Constraint-Preserving Recursive Intelligence

(2022) · Bai, Yuntao · Saurav Kadavath · Sandipan Kundu · Amanda Askell +47

Neighborhood — ranked by edge-count

Communities (2)

community

Chain-of-Thought reasoning robustness & safety
members_of
CoT effects on generalization, multimodal QA accuracy, and AI safety alignment training.
Constitutional AI safety training methods
members_of
Comparative evaluation of RL-CAI and SL-CAI approaches for harmlessness using constitutional principles, 2022-2023 Anthropic research.

Related by similarity (8)

cosine ≥ 0.65 · no typed edge

Entities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.

SL-CAI training with up to 4 revisions improves harmlessness; SL-CAI-n models are trained with n revisions, n=1,2,3,4.finding0.861
Section 3.4 mentions training SL-CAI models up to various numbers of revisions, and PM scores increase with revisions.
RL-CAI models are virtually never evasive and often give nuanced harmless responses, whereas HH RLHF models tend to be evasive.finding0.846
Section 4.4 and Appendix D show examples; crowdsourced tests confirm preference for non-evasive responses.
RL-CAI models (with and without CoT) are rated more harmless by crowdworkers than HH RLHF and SL-CAI.finding0.845
From Figure 3 and Figure 8, RL-CAI achieves significantly higher harmlessness Elo scores.
Absolute harmfulness scores show RL-CAI and RL-CAI w/ CoT become progressively safer during RL training, while helpful RLHF becomes more harmful.finding0.812
Figure 10: solid lines at T=1 and dashed at T=0; helpful RLHF score rises, others fall.
RL-CAI with CoT shows a Pareto improvement in helpfulness-harmlessness tradeoff over standard RLHF, with slight helpfulness decrease but higher harmlessness.finding0.799
Figure 2 and Figure 8 illustrate RL-CAI at the Pareto frontier.
AI feedback can effectively replace human feedback for harmlessness in RLHF-style training.claim0.766
The paper demonstrates that RLAIF with constitutional principles matches or exceeds HH RLHF.
Training a Helpful and Harmless Assistant with RLHF (Bai et al. 2022a)concept0.751
Foundational RLHF paper introducing HHH training objective for Claude
RL-CAI labels are reasonably well-calibrated on the new HHH evaluation, with frequencies aligning with predicted probabilities.finding0.742
Figure 9 calibration plot shows good alignment.