finding
active
finding:hellaswag-accuracy-drops-4-69-with-cot-71-to-66-yet-shows-less-post-hoc-reasoning-than-four-tasks-with-gainsHellaSwag accuracy drops 4.69% with CoT (71 to 66) yet shows less post-hoc reasoning than four tasks with gains
Only task with degradation.
Source paper
extracted_from(2023) · Tamera Lanham · Anna Chen · Ansh Radhakrishnan · Benoit Steiner +26
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- From decoupling of AOC and accuracy gain.
Findings (1)
finding
- Seven of eight tasks improve under CoT; mean CoT length is 4 steps with 89% between three and sixassociated_withCoT statistics.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Post-hoc reasoning AOC shows little correlation with CoT accuracy gain on 175B model across eight tasksfinding0.813LogiQA gains +1.02 but is second in AOC.
- Appendix C.
- Empirical evidence that naive one-stage CoT fails in language-only setting; two-stage + vision achieves state-of-the-art.
- Table 2.
- Validates that inference stage observations are not specific to mathematical reasoning tasks
- Table 2 early answering AOC.
- Shows VS not only maintains but can slightly improve factual accuracy compared to baseline methods
- Best performing VS variant for math synthetic data generation with GPT-4.1