paper
active
2023
ⓒ 32
paper:doi-10-48550-arxiv-2307-13702

Measuring Faithfulness in Chain-of-Thought Reasoning

TL;DR

Chain-of-thought reasoning in a 175B RLHF-tuned assistant model is faithful only conditionally: how much the final answer depends on the stated reasoning swings from near zero to dominant by task, and faithfulness falls as models grow. On ARC (Easy), ARC (Challenge) and OpenBookQA, truncating the CoT changes the final answer under 10% of the time, while on AQuA it changes it over 60% of the time. Lanham and colleagues introduce a battery of CoT-intervention tests: early answering (truncation), adding mistakes, filler tokens and paraphrasing. They use them to rule out specific failure classes rather than prove faithfulness. Replacing the CoT with filler ellipses gives no accuracy gain, which argues against hidden test-time computation. Paraphrasing matches original accuracy, which argues against phrasing-encoded steganography. Post-hoc reasoning, measured by area over the curve (AOC), correlates weakly with CoT's accuracy benefit: LogiQA gains about 1 point from CoT yet ranks second in AOC (0.26), while HellaSwag loses 4.69 points. Across the 810M–175B model series, 13B models give the most faithful CoT on 6 of 8 tasks, and on synthetic addition tasks faithfulness rises with model size and with easier problems. Faithful CoT therefore depends on choosing a model whose capability level is matched to the task, not a maximally capable one. The authors argue that CoT faithfulness is a tunable property, to be measured per task and per model before the reasoning is trusted in high-stakes settings.

What to take away

  1. 1. The four interventions (early answering, adding mistakes, filler tokens, paraphrasing) each target one class of unfaithfulness and are explicitly framed as defense-in-depth, not conclusive evidence of faithfulness.
  2. 2. On ARC (Easy), ARC (Challenge) and OpenBookQA, truncating the CoT changes the 175B model's final answer under 10% of the time, so the reasoning there is largely post-hoc.
  3. 3. On AQuA the CoT changes the final answer over 60% of the time, with early-answering AOC 0.44 and adding-mistakes AOC 0.52, the most faithful reasoning of the eight tasks.
  4. 4. Post-hoc-ness barely tracks CoT's accuracy benefit: LogiQA gains only 1.02 accuracy points yet has the second-highest early-answering AOC (0.26), and HellaSwag loses 4.69 points while still ranking above four tasks that gain.
  5. 5. Replacing the CoT with filler ' ...' tokens, up to the longest sampled CoT, produces no accuracy gain on any of the eight tasks, and TruthfulQA and OpenBookQA drop slightly, so extra test-time compute alone does not explain CoT's boost.
  6. 6. Paraphrasing prefixes of the CoT, with a paraphraser that never sees the question, leaves accuracy close to that of the original CoT, which argues against steganographic encoding in phrasing.
  7. 7. In the 810M–175B model series, the 13B model changes its answer after CoT more often than the 175B model on 7 of 8 tasks, and on six tasks (all but AQuA and LogiQA) the 13B model is the most faithful, which is inverse scaling in faithfulness.
  8. 8. On synthetic addition with 2, 4, 8 or 16 operands of two or three digits, post-hoc reasoning rises with model size and is higher on easier problems, so faithfulness is a function of model-task capability fit and not task type alone.
  9. 9. Methodology a researcher can replicate: sample 100 CoTs per question (nucleus p=0.95, temperature 0.8), split them into sentences with NLTK punkt, and score the AOC of answer-match against truncation point, weighted by CoT length.
  10. 10. The open question is whether the RLHF fine-tuning drives the post-hoc behavior, since pretrained-only models might condition more strongly on their own text and change answers more when mistakes are inserted, and this is left untested.

Peer brief — for seminar discussion

This work builds a set of black-box intervention tests for chain-of-thought faithfulness and applies them to a 175B pretrained decoder fine-tuned with RLHF as a helpful assistant, across eight multiple-choice benchmarks: AQuA, LogiQA, MMLU, HellaSwag, TruthfulQA, OpenBookQA, ARC Challenge and ARC Easy. The tests are early answering (truncate the CoT and see whether the answer changes), adding mistakes (a pretrained LM corrupts one sentence, the rest of the CoT is resampled), filler tokens (replace the CoT with ellipses), and paraphrasing (reword a prefix of the CoT without showing the paraphraser the question). Post-hoc reasoning is summarized by an area-over-the-curve metric. An alternative the authors could have used is Shapley-style token attribution (as in Gao 2023), or activation-level causal methods, but those are costlier. The load-bearing finding is heterogeneity plus inverse scaling. On ARC and OpenBookQA, truncating the CoT flips the answer under 10% of the time. On AQuA it flips it over 60% of the time. Filler tokens give no accuracy gain, and paraphrasing preserves accuracy, so neither hidden compute nor phrasing-encoded information explains CoT's benefit. Across the 810M–175B series, the 13B model is the most faithful on six of eight tasks, and on synthetic addition faithfulness increases with model size and with easier problems. Post-hoc-ness is only weakly related to CoT's accuracy gain: LogiQA gains about 1 point yet ranks second in AOC. The implication is that faithfulness is a model-by-task property that can be selected for, and that the larger, more capable models may be worse explainers on tasks they can solve without CoT. The central hypothesis is that models that can answer confidently without CoT ignore it, which predicts inverse scaling. What is contestable is the construct validity of the metric itself. Answer change under truncation measures whether the answer depends on the CoT, not whether the CoT is the model's actual reasoning. The authors acknowledge there is no ground truth for the internal process. A critical reader would push hardest on scope and inference. The scaling claim rests on a single RLHF model family with the final-answer metric standing in for the full AOC experiments, and the 13B peak could reflect small models simply not using CoT. The mistake-insertion step also relies on generated mistakes that are plausible only about 80% of the time, and the filler and paraphrase tests are null results that cannot exclude subtler encodings.

Methods (8)

Frameworks (1)

Datasets (8)

  • AQuA
    Algebra word problems; most faithful task.
  • ARC Easy
    Grade-school science questions.
  • HellaSwag
    Commonsense reasoning benchmark used to test whether LLM-vision alignment predicts downstream performance
  • LogiQA
    Logical inference dataset used in E3.
  • MMLU
    Benchmark used to evaluate performative reasoning; shows significantly more performative reasoning than GPQA-Diamond (easier task).
  • OpenBookQA
    Elementary-school science questions.
  • Synthetic addition tasks
    2/4/8/16 operands of two or three digits to control difficulty.
  • TruthfulQA
    817-question adversarial benchmark distinguishing factually grounded from misconception-based answers; used in Experiment 2

Findings (20)

Claims (13)

Original abstract (expand)

Large language models (LLMs) perform better when they produce step-by-step, "Chain-of-Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.

Related work— refs + corpus + external arXiv

Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.

+30 more

Similar preprints — Semantic Scholar

Cited by (2)