paper:doi-10-48550-arxiv-2305-04388Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
Original abstract (expand)
Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM's process for solving a task. This level of transparency into LLMs' predictions would yield significant safety benefits. However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction. We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always"(A)"--which models systematically fail to mention in their explanations. When we bias models toward incorrect answers, they frequently generate CoT explanations rationalizing those answers. This causes accuracy to drop by as much as 36% on a suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI and Claude 1.0 from Anthropic. On a social-bias task, model explanations justify giving answers in line with stereotypes without mentioning the influence of these social biases. Our findings indicate that CoT explanations can be plausible yet misleading, which risks increasing our trust in LLMs without guaranteeing their safety. Building more transparent and explainable systems will require either improving CoT faithfulness through targeted efforts or abandoning CoT in favor of alternative methods.
Similar preprints — Semantic Scholar
Cited by (6)
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Difference-of-means (DoM) vectors built from 5,285 templated synthetic SWE conversations detect reward hacking in long agentic rollouts of Kimi K3, GLM 5.2, and Qwen 3.8 Max, matching a GPT-5.6 Sol LL
- Measuring Faithfulness in Chain-of-Thought Reasoning
Chain-of-thought reasoning in a 175B RLHF-tuned assistant model is faithful only conditionally: how much the final answer depends on the stated reasoning swings from near zero to dominant by task, and
- Reasoning Models Don't Always Say What They Think
Reasoning models' chains of thought usually fail to disclose the hints that actually drive their answers: Claude 3.7 Sonnet verbalizes a used hint in only 25% of cases and DeepSeek R1 in 39%, falling
- Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
Activation-reading interpretability tools give no uplift for predicting counterfactual model behavior on naturally occurring behaviors, even though the same tool families help in narrow fine-tuned-qui
- Model Alignment Search
Model Alignment Search (MAS) establishes bidirectional causal similarity between neural networks by learning a per-model orthogonal rotation matrix that isolates behaviorally relevant subspaces and us
- Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Reasoning models generate chains of thought that are frequently performative rather than causally necessary for reaching the correct answer: on MMLU recall questions, activation probes decode the mode