claim
active
claim:models-vary-widely-across-tasks-in-how-strongly-they-condition-on-cot-sometimes-relying-heavily-and-sometimes-ignoring-itModels vary widely across tasks in how strongly they condition on CoT, sometimes relying heavily and sometimes ignoring it.
Interpretation of early answering/mistakes results.
Source paper
extracted_from(2023) · Tamera Lanham · Anna Chen · Ansh Radhakrishnan · Benoit Steiner +26
Neighborhood — ranked by edge-count
Papers (1)
paper
Findings (5)
finding
- Table 2.
- Table 2 early answering AOC.
- Cross-test agreement.
- Lowest-AOC tasks.
- Highest-AOC task.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Lower faithfulness on GPQA.
- For reasoning faithfulness, larger models may behave worse than smaller ones because they can predict answers without CoT.hypothesis0.810Motivates §3.
- High-level policy-relevant claim about the risks of advanced reasoning in LLMs
- From decoupling of AOC and accuracy gain.
- Limitations.
- Task difficulty as the key variable distinguishing the two modes of CoT identified in the paper
- Caveat and forward-looking statement from the abstract.
- Subsets amenable to monitoring.