finding
active
finding:retention-of-baseline-logic-95-106-open-book-qa-93-99-closed-book-qa-74-86-math-66-74-memorization-3-16Retention of baseline: logic 95-106%, open-book QA 93-99%, closed-book QA 74-86%, math 66-74%, memorization 3-16%
Spectrum of brittleness.
Source paper
extracted_from(2025) · Merullo, Jack · Vatsavaya, Srihita · Bushnaq, Lucius · Lewis, Owen
Neighborhood — ranked by edge-count
Papers (1)
paper
Claims (1)
claim
- Section 6.
Related by similarity (8)
cosine ≥ 0.65 · no typed edgeEntities in the same semantic neighborhood but without a typed relation to this one — candidates for new edges or unrecognized duplicates.
- Demonstrates reflection redundancy in larger models on non-mathematical reasoning
- Demonstrates reflection redundancy in stronger model on harder math benchmark
- Intervention action distribution.
- Empirical evidence that naive one-stage CoT fails in language-only setting; two-stage + vision achieves state-of-the-art.
- Base corrections: 72.9% Failed Fix, 14.7% Fixed; retractions: 69.4% Correct Retract, 30.0% Incorrect Retractfinding0.756Label distributions.
- OpenBookQA interacts far less with bottom spectrum than memorized data (memorized ~1.6x higher)finding0.754Activation ratios predict robustness.
- Core negative result: the binary detection paradigm cannot distinguish genuine introspection from uniform output bias
- Test-time probe metrics.