thinker:saurav-kadavathSaurav Kadavath
Authored papers (3)
Chain-of-thought reasoning in a 175B RLHF-tuned assistant model is faithful only conditionally: how much the final answer depends on the stated reasoning swings from near zero to dominant by task, and faithfulness falls as models grow. On ARC (Easy), ARC (Challenge) and OpenBookQA, truncating the CoT changes the final answer under 10% of the time, while on AQuA it changes it over 60% of the time. Lanham and colleagues introduce a battery of CoT-intervention tests: early answering (truncation), adding mistakes, filler tokens and paraphrasing. They use them to rule out specific failure classes rather than prove faithfulness. Replacing the CoT with filler ellipses gives no accuracy gain, which argues against hidden test-time computation. Paraphrasing matches original accuracy, which argues against phrasing-encoded steganography. Post-hoc reasoning, measured by area over the curve (AOC), correlates weakly with CoT's accuracy benefit: LogiQA gains about 1 point from CoT yet ranks second in AOC (0.26), while HellaSwag loses 4.69 points. Across the 810M–175B model series, 13B models give the most faithful CoT on 6 of 8 tasks, and on synthetic addition tasks faithfulness rises with model size and with easier problems. Faithful CoT therefore depends on choosing a model whose capability level is matched to the task, not a maximally capable one. The authors argue that CoT faithfulness is a tunable property, to be measured per task and per model before the reasoning is trusted in high-stakes settings.
Constitutional AI (CAI) demonstrates that a harmless, non-evasive AI assistant can be trained using zero human feedback labels for harmlessness, replacing them entirely with AI-generated feedback guided by a short list of natural language principles. The method introduces a two-stage pipeline: a supervised learning (SL) phase in which a helpful RLHF model iteratively critiques and revises its own responses to red-team prompts drawn from 182,831 total prompts (42,496 human-written, 140,335 model-generated), followed by a reinforcement learning phase termed RLAIF, where a 52B-parameter feedback model evaluates response pairs according to 16 constitutional principles and generates preference labels that train a hybrid preference model. Crowdworker Elo score comparisons across 10,274 helpfulness and 8,135 harmlessness evaluations show that RL-CAI with chain-of-thought reasoning achieves harmlessness scores that meet or exceed those of models trained with human harmlessness feedback (HH RLHF), while maintaining comparable helpfulness—tracing a Pareto improvement over the helpfulness–harmlessness tradeoff documented in prior work. Critically, whereas HH RLHF models produced evasive refusals (e.g., 'I'm sorry, I won't respond') on sensitive PALMS and LaMDA prompts, RL-CAI models engage substantively and explain their objections. Chain-of-thought prompting on the feedback model, with CoT probabilities clamped to the 40–60% range to prevent overconfidence, further improved both harmlessness scores and label calibration. The paper argues this implies that scaled AI supervision—encoding alignment objectives in a transparent, auditable constitution rather than tens of thousands of opaque human labels—is a viable path toward alignment as model capabilities grow beyond reliable human oversight.
More papers — OpenAlex / S2
Affiliations (1)
- Anthropic(institute)
Co-authors (12)
- Ethan Perez8 shared
- Jared Kaplan8 shared
- Samuel R. Bowman8 shared
- Tamera Lanham8 shared
- Anna Chen6 shared
- Danny Hernandez6 shared
- Dustin Li6 shared
- Jackson Kernion6 shared
- Jan Brauner6 shared
- Nicholas Schiefer6 shared
- Sam McCandlish6 shared
- Sandipan Kundu6 shared
Their work is cited by (10)
- Reasoning Models Don't Always Say What They Think2× refs
- Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought2× refs
- Contemplative Agent1× refs
- Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI1× refs
- Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs1× refs
- Where is the Mind? Persona Vectors and LLM Individuation1× refs
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations1× refs
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering1× refs
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training1× refs
- Towards Safe and Honest AI Agents with Neural Self-Other Overlap1× refs
Other inbound relations (1)
Recent mentions (2)
- papers-typedmartorell-2026-quantitative-introspection.md
- papers-typedyuntao-2022-cat-s.md