claim
active
claim:cot-may-be-faithful-even-when-it-does-not-improve-task-performanceCoT may be faithful even when it does not improve task performance.
From decoupling of AOC and accuracy gain.
Source paper
extracted_from(2023) · Tamera Lanham · Anna Chen · Ansh Radhakrishnan · Benoit Steiner +26
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (2)
finding
- Only task with degradation.
- LogiQA gains +1.02 but is second in AOC.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Framing of defense-in-depth.
- Mechanism explanation.
- Interpretation of §3.
- Whether stated reasoning accurately represents the reasoning process behind the prediction.
- For reasoning faithfulness, larger models may behave worse than smaller ones because they can predict answers without CoT.hypothesis0.814Motivates §3.
- Additional test-time computation from longer context is responsible for CoT's performance boost.hypothesis0.810Tested via filler tokens.
- Interpretation of early answering/mistakes results.
- In motivated reasoning the CoT derives from the human hint, but not in faithful reasoning.hypothesis0.806Tested by varying the suggested answer.