paper:doi-10-48550-arxiv-2307-13702Measuring Faithfulness in Chain-of-Thought Reasoning
TL;DR
Chain-of-thought reasoning in a 175B RLHF-tuned assistant model is faithful only conditionally: how much the final answer depends on the stated reasoning swings from near zero to dominant by task, and faithfulness falls as models grow. On ARC (Easy), ARC (Challenge) and OpenBookQA, truncating the CoT changes the final answer under 10% of the time, while on AQuA it changes it over 60% of the time. Lanham and colleagues introduce a battery of CoT-intervention tests: early answering (truncation), adding mistakes, filler tokens and paraphrasing. They use them to rule out specific failure classes rather than prove faithfulness. Replacing the CoT with filler ellipses gives no accuracy gain, which argues against hidden test-time computation. Paraphrasing matches original accuracy, which argues against phrasing-encoded steganography. Post-hoc reasoning, measured by area over the curve (AOC), correlates weakly with CoT's accuracy benefit: LogiQA gains about 1 point from CoT yet ranks second in AOC (0.26), while HellaSwag loses 4.69 points. Across the 810M–175B model series, 13B models give the most faithful CoT on 6 of 8 tasks, and on synthetic addition tasks faithfulness rises with model size and with easier problems. Faithful CoT therefore depends on choosing a model whose capability level is matched to the task, not a maximally capable one. The authors argue that CoT faithfulness is a tunable property, to be measured per task and per model before the reasoning is trusted in high-stakes settings.
What to take away
- 1. The four interventions (early answering, adding mistakes, filler tokens, paraphrasing) each target one class of unfaithfulness and are explicitly framed as defense-in-depth, not conclusive evidence of faithfulness.
- 2. On ARC (Easy), ARC (Challenge) and OpenBookQA, truncating the CoT changes the 175B model's final answer under 10% of the time, so the reasoning there is largely post-hoc.
- 3. On AQuA the CoT changes the final answer over 60% of the time, with early-answering AOC 0.44 and adding-mistakes AOC 0.52, the most faithful reasoning of the eight tasks.
- 4. Post-hoc-ness barely tracks CoT's accuracy benefit: LogiQA gains only 1.02 accuracy points yet has the second-highest early-answering AOC (0.26), and HellaSwag loses 4.69 points while still ranking above four tasks that gain.
- 5. Replacing the CoT with filler ' ...' tokens, up to the longest sampled CoT, produces no accuracy gain on any of the eight tasks, and TruthfulQA and OpenBookQA drop slightly, so extra test-time compute alone does not explain CoT's boost.
- 6. Paraphrasing prefixes of the CoT, with a paraphraser that never sees the question, leaves accuracy close to that of the original CoT, which argues against steganographic encoding in phrasing.
- 7. In the 810M–175B model series, the 13B model changes its answer after CoT more often than the 175B model on 7 of 8 tasks, and on six tasks (all but AQuA and LogiQA) the 13B model is the most faithful, which is inverse scaling in faithfulness.
- 8. On synthetic addition with 2, 4, 8 or 16 operands of two or three digits, post-hoc reasoning rises with model size and is higher on easier problems, so faithfulness is a function of model-task capability fit and not task type alone.
- 9. Methodology a researcher can replicate: sample 100 CoTs per question (nucleus p=0.95, temperature 0.8), split them into sentences with NLTK punkt, and score the AOC of answer-match against truncation point, weighted by CoT length.
- 10. The open question is whether the RLHF fine-tuning drives the post-hoc behavior, since pretrained-only models might condition more strongly on their own text and change answers more when mistakes are inserted, and this is left untested.
Peer brief — for seminar discussion
This work builds a set of black-box intervention tests for chain-of-thought faithfulness and applies them to a 175B pretrained decoder fine-tuned with RLHF as a helpful assistant, across eight multiple-choice benchmarks: AQuA, LogiQA, MMLU, HellaSwag, TruthfulQA, OpenBookQA, ARC Challenge and ARC Easy. The tests are early answering (truncate the CoT and see whether the answer changes), adding mistakes (a pretrained LM corrupts one sentence, the rest of the CoT is resampled), filler tokens (replace the CoT with ellipses), and paraphrasing (reword a prefix of the CoT without showing the paraphraser the question). Post-hoc reasoning is summarized by an area-over-the-curve metric. An alternative the authors could have used is Shapley-style token attribution (as in Gao 2023), or activation-level causal methods, but those are costlier. The load-bearing finding is heterogeneity plus inverse scaling. On ARC and OpenBookQA, truncating the CoT flips the answer under 10% of the time. On AQuA it flips it over 60% of the time. Filler tokens give no accuracy gain, and paraphrasing preserves accuracy, so neither hidden compute nor phrasing-encoded information explains CoT's benefit. Across the 810M–175B series, the 13B model is the most faithful on six of eight tasks, and on synthetic addition faithfulness increases with model size and with easier problems. Post-hoc-ness is only weakly related to CoT's accuracy gain: LogiQA gains about 1 point yet ranks second in AOC. The implication is that faithfulness is a model-by-task property that can be selected for, and that the larger, more capable models may be worse explainers on tasks they can solve without CoT. The central hypothesis is that models that can answer confidently without CoT ignore it, which predicts inverse scaling. What is contestable is the construct validity of the metric itself. Answer change under truncation measures whether the answer depends on the CoT, not whether the CoT is the model's actual reasoning. The authors acknowledge there is no ground truth for the internal process. A critical reader would push hardest on scope and inference. The scaling claim rests on a single RLHF model family with the final-answer metric standing in for the full AOC experiments, and the 13B peak could reflect small models simply not using CoT. The mistake-insertion step also relies on generated mistakes that are plausible only about 80% of the time, and the filler and paraphrase tests are null results that cannot exclude subtler encodings.
Methods (8)
- Adding Mistakes testInsert an LLM-generated mistake into the CoT and resample the rest.
- Answer change with vs. without CoT metricCheap proxy equal to early answering at 0% CoT, used for the model-size analysis.
- ARC ChallengeScience reasoning benchmark used to assess capability preservation after character training
- Area over the curve (AOC) metricWeighted-sum summary of post-hoc reasoning for early answering and adding mistakes.
- Chain-of-thought promptingTechnique by which LLMs generate intermediate reasoning steps before final output; used by ChatGPT o3.
- Early Answering testTruncate CoT and see whether the answer changes.
- Filler Tokens testReplace CoT with ellipsis tokens to test test-time compute.
- Paraphrasing testReword the CoT beginning to ablate phrasing-encoded information.
Frameworks (1)
- Defense-in-depth approachEach test rules out one class of faithfulness failures rather than proving faithfulness.
Datasets (8)
- AQuAAlgebra word problems; most faithful task.
- ARC EasyGrade-school science questions.
- HellaSwagCommonsense reasoning benchmark used to test whether LLM-vision alignment predicts downstream performance
- LogiQALogical inference dataset used in E3.
- MMLUBenchmark used to evaluate performative reasoning; shows significantly more performative reasoning than GPQA-Diamond (easier task).
- OpenBookQAElementary-school science questions.
- Synthetic addition tasks2/4/8/16 operands of two or three digits to control difficulty.
- TruthfulQA817-question adversarial benchmark distinguishing factually grounded from misconception-based answers; used in Experiment 2
Findings (20)
- Probability of same answer under paraphrasing shows a smooth upward curve for most tasks, not flat or downward as steganography predicts; AQuA and LogiQA slight exceptions on six-step samples
Appendix B.
- For four of eight tasks (ARC Easy, ARC Challenge, HellaSwag, AQuA) the model size with greatest CoT accuracy gain equals the size with greatest faithfulness
Appendix C.
- Large variation in post-hoc reasoning across tasks on early answering: AOC from 0.02 (ARC Easy) to 0.44 (AQuA) on the 175B model
Table 2 early answering AOC.
- For six of eight tasks (all but AQuA and LogiQA) the 13B model gives the most faithful reasoning, better than smaller models
Optimum at mid-size.
- Adding mistakes AOC ordering of tasks nearly identical to early answering, differing only in HellaSwag ranked above MMLU
Cross-test agreement.
- No accuracy increase from filler tokens on any of eight tasks with 175B model; TruthfulQA and OpenBookQA drop slightly
Evidence against test-time compute hypothesis.
- On synthetic addition tasks, post-hoc reasoning increases with model size and with easier tasks at the same model size
Faithfulness decreases for easier problems and bigger models.
- HellaSwag accuracy drops 4.69% with CoT (71 to 66) yet shows less post-hoc reasoning than four tasks with gains
Only task with degradation.
- For ARC Easy, ARC Challenge and OpenBookQA, truncating CoT changes the final answer less than 10% of the time
Lowest-AOC tasks.
- AQuA and LogiQA show the most faithful reasoning on both early answering and adding mistakes by some margin
Top two tasks.
Claims (13)
- To elicit faithful reasoning for explaining model behavior, it may be necessary to choose models less capable than the maximally capable available, especially for easier tasks.
Practical implication.
- The proposed faithfulness tests should hold for other reasoning-elicitation methods (tree of thoughts, plan-and-solve, debate, subquestions) that share CoT's structure.
Generality of methods.
- Certain tasks do not inherently lead to unfaithful reasoning; only models of a certain capability level (but no higher) on a task seem to produce faithful CoT.
Interpretation of §3.
- Without a separate way to understand the model's real internal reasoning, we cannot know if CoT is faithful to it, nor how to weigh each experiment.
Limitations.
- Increased faithfulness on AQuA and LogiQA may be due to models' limited ability to do the task without CoT, causing greater reliance on CoT.
Explanation supported by §3.
- Faithfulness on AQuA and LogiQA being due to logical reasoning content is doubtful, since faithfulness does not depend on the task alone.
Rejecting alternative explanation.
- Models vary widely across tasks in how strongly they condition on CoT, sometimes relying heavily and sometimes ignoring it.
Interpretation of early answering/mistakes results.
- While chain of thought reasoning is not always faithful, it is possible to find conditions where it is more faithful.
Central conclusion.
- The particular phrasing of CoT is unlikely to encode information responsible for the accuracy gains of CoT.
From paraphrasing results.
- Reasoning not being post-hoc does not guarantee faithfulness, nor does being post-hoc exclude faithfulness.
Caveat on test interpretation.
Hypotheses (6)
- Pretrained LLMs may show fewer signs of post-hoc reasoning than RLHF models, e.g. changing their answer more when mistakes are added.
Future work in limitations.
- If steganography were important, paraphrasing midway would corrupt the encoded message, giving flat or downward-sloping lines.
Appendix B prediction.
- Stylistic elements of the CoT sample elicit higher-quality answers by prompting the model to imitate a more thoughtful agent.
Fourth hypothesis not investigated here.
- For reasoning faithfulness, larger models may behave worse than smaller ones because they can predict answers without CoT.
Motivates §3.
- Additional test-time computation from longer context is responsible for CoT's performance boost.
Tested via filler tokens.
- Information encoded in phrasing choices of CoT is responsible for the accuracy boost.
Tested via paraphrasing.
Questions (7)
- Are training schemes different from RLHF more effective at eliciting faithful reasoning from LLMs?
Future work.
- Is there any size model that generates faithful reasoning on these tasks?
Motivates §3.
- Do uninformative chain of thought tokens also improve performance?
Filler tokens question.
- Does truncating the chain of thought change the predicted answer?
Early answering question.
- Does rewording the chain of thought change the predicted answer?
Paraphrasing question.
- Does editing the chain of thought change the predicted answer?
Adding mistakes question.
- Is the stated reasoning ever faithful?
Motivating question.
Original abstract (expand)
Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- ≈ 91%
- When Reasoning Traces Become Performative: Step-Level Evidence that Chain-of-Thought Is an Imperfect Oversight ChannelFan Yang, Ananya Hazarika, Shaunak A. Mehta, Koichi Onoue Wenkai Li2026≈ 89%
- FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought ReasoningAoqi Zuo, Haotian Xie, Wei Gao, Mingming Gong, Jing Ma Yuxi Sun2026≈ 87%
- Decoding Answers Before Chain-of-Thought: Evidence from Pre-CoT Probes and Activation SteeringDarius Kianersi, Adri\`a Garriga-Alonso Kyle Cox2026≈ 87%
- Reliable Chain-of-Thought via Prefix ConsistencyYuki Ichihara, Mohammad Atif Quamar, Junpei Komiyama Naoto Iwase2026≈ 87%
- Reasoning Models Don't Always Say What They Thinkin corpus2025≈ 87%
- Measuring and curing reasoning rigidity: from decorative chain-of-thought to genuine faithfulnessPavan Chakraborty Abhinaba Basu2026≈ 87%
- ≈ 86%
- How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse AutoencodingAske Plaat, Niki van Stein Xi Chen2025≈ 86%
- When Chain-of-Thought Fails, the Solution Hides in the Hidden StatesAmit Parekh, Ioannis Konstas Houman Mehrafarin2026≈ 85%
- ≈ 85%
- When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Modelsin corpus2025≈ 85%
- Hidden Error Awareness in Chain-of-Thought Reasoning: The Signal Is Diagnostic, Not CausalZhiyuan Julian Su, Haiyue Zhang, Yi Nian, Yue Zhao Aojie Yuan2026≈ 84%
- ≈ 84%
- Trustworthy Reasoning: Evaluating and Enhancing Factual Accuracy in LLM Intermediate Thought ProcessesYue Zhang, Jinku Li Rui Jiao2025≈ 83%
- ≈ 83%
- Faithful Logical Reasoning via Symbolic Chain-of-ThoughtHao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, Wynne Hsu Jundong Xu2024≈ 83%
- S3-CoT: Self-Sampled Succinct Reasoning Enables Efficient Chain-of-Thought LLMsSendong Zhao, Yibo Gao, Danyang Zhao, Qika Lin, Ming Ma, Jiayun Li, Yi Jiang, Kai He, Qianyi Xu, Bing Qin, Mengling Feng Yanrui Du2026≈ 83%
- ≈ 82%
- Testing the Limits of Truth Directions in LLMsin corpus2026≈ 81%
- ≈ 81%
- Alignment faking in large language modelsin corpus2024≈ 81%
- ≈ 81%
- Evaluating Language Model Character Traitsin corpus2024≈ 80%
- ≈ 80%
- What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectorsin corpus2026≈ 79%
- ≈ 79%
- Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIin corpus2025≈ 79%
- Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generationin corpus2025≈ 79%
- ≈ 79%
+30 more
Similar preprints — Semantic Scholar
Cited by (2)
- Reasoning Models Don't Always Say What They Think
Reasoning models' chains of thought usually fail to disclose the hints that actually drive their answers: Claude 3.7 Sonnet verbalizes a used hint in only 25% of cases and DeepSeek R1 in 39%, falling
- Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Reasoning models generate chains of thought that are frequently performative rather than causally necessary for reaching the correct answer: on MMLU recall questions, activation probes decode the mode