finding
active
finding:mmlu-pro-math-drops-from-23-4-to-15-8-under-k-fac-while-computer-science-stays-26-1-to-26-3MMLU-Pro Math drops from 23.4% to 15.8% under K-FAC while computer science stays 26.1% to 26.3%
Arithmetic specifically hurt.
Source paper
extracted_from(2025) · Merullo, Jack · Vatsavaya, Srihita · Bushnaq, Lucius · Lewis, Owen
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Abstract thesis.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Shows smaller models are more sensitive to reflection reduction on non-math tasks
- We hypothesize that degraded generalization on benchmarks like MMLU may reflect the computational demands of the tasks.hypothesis0.770Connecting the paper's task-difficulty findings to prior observations of weak generalization on complex QA benchmarks.
- Demonstrates disconnect between MMLU and coherency metrics
- Demonstrates reflection redundancy in larger models on non-mathematical reasoning
- K-FAC edit reduces OLMo-7B Dolma strict accuracy from 99.9% to 3.4% (BSN 6.0%, SVD 3.0%)finding0.742Table 1.
- RLFR policy matches base on standard benchmarks (e.g., MMLU 67.2 vs 69.2, GSM8K 73.0 vs 72.3)finding0.740No capability degradation.
- Dropping 50% of monitor calls on SWE-bench with GLM lowers precision 0.656→0.588 while Kimi drops ~0.01finding0.739Cost of replacement.
- Outcome-based RL faithfulness plateaus without saturating at 28% on MMLU and 20% on GPQAfinding0.738Diminishing returns.