paper:doi-10-48550-arxiv-2510-24256From Memorization to Reasoning in the Spectrum of Loss Curvature
Methods (9)
- Activation ratio analysisMeasures selectivity of weight components for memorized vs clean data.
- BalancedSubnet (BSN)Supervised unlearning baseline using a binary mask trained on forget/retain sets.
- K-FAC curvature-mass weight editingKeeps top eigenvector pairs by curvature mass to suppress memorization.
- MMLU ProGeneral knowledge benchmark across domains (1400 subsampled problems) used to evaluate capability preservation
- nDCG@10 token-ranking overlapMeasures preference drift after editing.
- Noised-label ViT memorization setup10% random labels force exact memorization in ViT.
- Positional perturbation stress testsTests robustness of unlearning to prefix shifts.
- Strict/Loose accuracy and Levenshtein metricsEvaluation metrics for exact and near-verbatim memorization.
- SVD truncation editData-free baseline pruning low singular values.
Frameworks (1)
- K-FAC (Kronecker-Factored Approximate Curvature)Approximates FIM as G⊗A to give dataset-average curvature eigenbasis.
Datasets (12)
- BigBench-HardLogic reasoning benchmark suite.
- Boar EtruscanIn-context fake language for pure reasoning.
- Dolma memorized sequences1000 mined memorized pretraining sequences.
- GSM8KGrade school math dataset used for math task in E3.
- Historical Quotes memorization setOut-of-distribution memorized quotes, 512 sequences.
- ImageNet ILSVRC 2012ViT training data.
- OLMES evaluation suiteCapability benchmark suite.
- OpenBookQAElementary-school science questions.
- Pile10kHeld-out text for perplexity and nDCG.
- Relations dataset26 factual relations used to analyze fact retrieval.
- SimpleMathGenerated two-digit addition/subtraction problems.
- TriviaQAClosed-book fact retrieval.
Findings (18)
- Positional perturbation raises extraction length for K-FAC from 6.6 to 13.3 tokens and BSN 4.6 to 10, less than prior methods (+13 to +16)
Robustness.
- At OLMo-7B layer 22 MLP input, bottom 50% K-FAC eigenvectors activate 23.1% more on memorized data; top 10% activate 26% more on clean
Disentanglement evidence.
- In ViT last layer, top 10% K-FAC eigenvectors have over 2x activation on clean vs memorized data; SVD shows no pattern
Vision disentanglement.
- Retention of baseline: logic 95-106%, open-book QA 93-99%, closed-book QA 74-86%, math 66-74%, memorization 3-16%
Spectrum of brittleness.
- K-FAC edit on ViT-Base drops memorized train accuracy from 81.6% to 3.5%, GT accuracy to 66.5% (SVD 58.9%)
Table 2.
- K-FAC eigenvector bands show sorted activation strength top10%>10-25%>25-50%>bottom50%, unlike SVD
Supports curvature basis interpretability.
- OpenBookQA interacts far less with bottom spectrum than memorized data (memorized ~1.6x higher)
Activation ratios predict robustness.
- MMLU-Pro Math drops from 23.4% to 15.8% under K-FAC while computer science stays 26.1% to 26.3%
Arithmetic specifically hurt.
- GSM8K example: edited model produces identical chain of thought but miscalculates 260 as 220
Targeted arithmetic deficit.
- Rare relations like Company-CEO drop 78% under K-FAC edit while frequent ones barely change
Frequency modulates brittleness.
Claims (10)
- Averaged across examples, memorization directions are flatter than generalizing directions, inverting the per-example view that memorized points are sharp
Core novel interpretation.
- Fact retrieval and arithmetic rely on specialized narrowly-used weight directions rather than general purpose mechanisms, regardless of memorization
Abstract thesis.
- Arithmetic takes a path through weight components resembling memorized sequences, whereas logical reasoning resembles clean data
Discussion.
- Memorization is hard to pinpoint and likely highly distributed, though some curvature directions localize to early/late layers
Consistent with prior work.
- Memorization, fact retrieval, arithmetic and logical reasoning lie on a continuum of sensitivity to low-curvature directions
Section 6.
- Idiosyncratic sharp directions of specific examples cancel in the dataset average while shared mechanisms add coherently
Explains population vs instance curvature.
- Top K-FAC eigenvectors correspond to shared generalizing structure, while low-curvature components underwrite recitation
Basis for editing.
- The curvature basis is more interpretable and accurate than SVD for describing the memorized/non-memorized spectrum
Section 4.
- K-FAC suppresses the broadest range of unseen memorized text without requiring labeled forget data
Practical advantage over BSN.
- The methods do not fully remove memories, which may resurface after further tuning
Limitation.
Hypotheses (2)
Questions (2)
- To what degree do models generate genuinely new knowledge, as opposed to simply reassembling snippets of data memorized from their training sets?
Opening question.
- Are there model structures that account for memorization and generalization across large swaths of training data?
Aggregate analysis motivation.
Original abstract (expand)
We characterize how memorization is represented in transformer models and show that it can be disentangled in the weights of both language models (LMs) and vision transformers (ViTs) using a decomposition based on the loss landscape curvature. This insight is based on prior theoretical and empirical work showing that the curvature for memorized training points is much sharper than non memorized, meaning ordering weight components from high to low curvature can reveal a distinction without explicit labels. This motivates a weight editing procedure that suppresses far more recitation of untargeted memorized data more effectively than a recent unlearning method (BalancedSubnet), while maintaining lower perplexity. Since the basis of curvature has a natural interpretation for shared structure in model weights, we analyze the editing procedure extensively on its effect on downstream tasks in LMs, and find that fact retrieval and arithmetic are specifically and consistently negatively affected, even though open book fact retrieval and general logical reasoning is conserved. We posit these tasks rely heavily on specialized directions in weight space rather than general purpose mechanisms, regardless of whether those individual datapoints are memorized. We support this by showing a correspondence between task data's activation strength with low curvature components that we edit out, and the drop in task performance after the edit. Our work enhances the understanding of memorization in neural networks with practical applications towards removing it, and provides evidence for idiosyncratic, narrowly-used structures involved in solving tasks like math and fact retrieval.
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Loss Barcode: A Topological Measure of Escapability in Loss LandscapesDaria Voronkova, Alexander Mironenko, Ilya Trofimov, Alexander Korotin, Grigorii Sotnikov, Evgeny Burnaev Serguei Barannikov2026≈ 74%
- Why Learning Requires Feelingin corpus2026≈ 73%
- A simple connection from loss flatness to compressed neural representationsStefano Recanatesi, Eric Shea-Brown Shirui Chen2026≈ 73%
- ≈ 73%
- The Price of Meaning: Why Every Semantic Memory System ForgetsAndrey Starenky, Sofia Bodnar, Nikhil Narasimhan and Ashwin Gopinath Sambartha Ray Barman2026≈ 73%
- Representational Curvature Modulates Behavioral Uncertainty in Large Language ModelsEvelina Fedorenko, Eghbal A. Hosseini Jack King2026≈ 72%
- ≈ 72%
- Putting a Face to Forgetting: Continual Learning meets Mechanistic InterpretabilityGido M. van de Ven, Javier Ferrando, Tinne Tuytelaars Sergi Masip2026≈ 71%
- An Evaluation on Large Language Model Outputs: Discourse and MemorizationXun Wang, Alex Sokolov, Qilong Gu and Si-Qing Chen Adrian de Wynter2026≈ 71%
- Predictive Processing in Cognitive Robotics: a ReviewGuido Schillaci, Giovanni Pezzulo, Verena V. Hafner, Bruno Lara Alejandra Ciria2021≈ 71%
- A geometric relation of the error introduced by sampling a language model's output distribution to its internal stateAlbert F. Modenbach2026≈ 71%
- Reasoning emerges from constrained inference manifolds in large language modelsFei Luo, Linfeng Zhang, Chuangxin Zhao, Mingxuan Wang, Yinan Wu, Zhe Qian, Yang Lu, Long Chen, Zhao Cao, Xiaoshuai Hao, Ji-Rong Wen, Jungong Han Yanbiao Ma2026≈ 71%
- ≈ 71%
- Consciousness is entailed by compositional learning of new causal structures in deep predictive processing systemsV.A. Aksyuk2024≈ 71%
- Action and Perception as Divergence MinimizationPedro A. Ortega, Jimmy Ba, Thomas Parr, Karl Friston, Nicolas Heess Danijar Hafner2022≈ 70%
- A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety RisksHieu Minh "Jord" Nguyen2025≈ 70%
- Self-Improvising Memory: A Perspective on Memories as Agential, Dynamically Reinterpreting Cognitive Gluein corpus2024≈ 70%
- The World Inside Neural Networksin corpus2026≈ 69%
- ≈ 69%
- ≈ 69%
- The Platonic Representation Hypothesisin corpus2024≈ 69%
- ≈ 69%
- Learning without neurons in physical systemsin corpus2022≈ 69%
- ≈ 68%
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retentionin corpus2026≈ 68%
- ≈ 68%
- ≈ 68%
- The biogenic approach to cognitionin corpus2005≈ 68%
- Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationsin corpus2023≈ 67%
- ≈ 65%
+29 more