paper:doi-10-18653-v1-2020-acl-main-197SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Original abstract (expand)
Transfer learning has fundamentally changed the landscape of natural language processing (NLP). Many state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely high complexity of pre-trained models, aggressive fine-tuning often causes the fine-tuned model to overfit the training data of downstream tasks and fail to generalize to unseen data. To address such an issue in a principled manner, we propose a new learning framework for robust and efficient fine-tuning for pre-trained models to attain better generalization performance. The proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the complexity of the model; 2. Bregman proximal point optimization, which is an instance of trustregion methods and can prevent aggressive updating. Our experiments show that the proposed framework achieves new state-of-the-art performance on a number of NLP tasks including GLUE, SNLI, SciTail and ANLI. Moreover, it also outperforms the state-of-the-art T5 model, which is the largest pre-trained model containing 11 billion parameters, on GLUE. 1
Related work— refs + corpus + external arXiv
Cited / in-corpus / arXiv badges show which signals surfaced each row. Multi-source rows weighted higher.
- Fine-Tuning Language Models Using Formal Methods FeedbackNeel P. Bhatt, Tyler Ingebrand, William Ward, Steven Carr, Zhangyang Wang, Ufuk Topcu Yunhao Yang2024≈ 77%
- Mechanistically analyzing the effects of fine-tuning on procedurally defined tasksRobert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rockt\"aschel, David Scott Krueger Samyak Jain2024≈ 77%
- BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelsShauli Ravfogel, Yoav Goldberg Elad Ben-Zaken2026≈ 76%
- ≈ 76%
- A Base Camp for Scaling AIT. Hart, Z. Yang, S. Cucerzan, R.W. White, A. Pastusiak, J. Lewis C.J.C. Burges2016≈ 75%
- SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMsYu Wan, Baosong Yang, Fei Huang, Wenjie Wang, Fuli Feng Boyi Deng2026≈ 75%
- A Survey of Large Language ModelsKun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie and Ji-Rong Wen Wayne Xin Zhao2026≈ 75%
- A Pattern Language for Machine Learning TasksIan Fan, Tuomas Laakkonen, Neil John Ortega, Thomas Hoffmann, Vincent Wang-Mascianica Benjamin Rodatz2025≈ 75%
- Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided SegmentationJun Xu, Ji Du, Shuo Ye, Ziyue Qiao, Xiaodong Cun, Guangcong Wang, Xubin Zheng and Zitong Yu Chao Hao2026≈ 75%
- ≈ 75%
- TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language ModelsKai Liu Yarui Cao2026≈ 75%
- Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model AgentsGangadharan G.R., Rajkumar Buyya Arunkumar V2026≈ 75%
- Improving Dictionary Learning with Gated Sparse AutoencodersArthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J\'anos Kram\'ar, Rohin Shah and Neel Nanda Senthooran Rajamanoharan2024≈ 74%
- Resa: Transparent Reasoning Models via SAEsJulian Asilis, \"Omer Faruk Akg\"ul, Enes Burak Bilgin, Ollie Liu, Deqing Fu, and Willie Neiswanger Shangshang Wang2025≈ 74%
- Adaptation of Agentic AI: A Survey of Post-Training, Memory, and SkillsJiacheng Lin, Zhiyi Shi, Zifeng Wang, Luxi He, Yichen Wu, Ming Zhong, Peiyang Song, Qizheng Zhang, Heng Wang, Xueqiang Xu, Hanwen Xu, Pengrui Han, Dylan Zhang, Jiashuo Sun, Chaoqi Yang, Kun Qian, Tian Wang, Changran Hu, Manling Li, Quanzheng Li, Hao Peng, Sheng Wang, Jingbo Shang, Chao Zhang, Jiaxuan You, Liyuan Liu, Pan Lu, Yu Zhang, Heng Ji, Yejin Choi, Dawn Song, Jimeng Sun, Jiawei Han Pengcheng Jiang2026≈ 74%
- Mechanistic Interpretability of Code Correctness in LLMs via Sparse AutoencodersKriz Tahimic and Charibeth Cheng2025≈ 74%
- ≈ 73%
- ≈ 73%
- Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencodersin corpus2026≈ 72%
- ≈ 72%
- ≈ 72%
- Active Inference, Curiosity and Insightin corpus2017≈ 72%
- ≈ 71%
- Learning without neurons in physical systemsin corpus2022≈ 71%
- ≈ 71%
- Interpreting Language Model Parametersin corpus2026≈ 71%
- ≈ 71%
- Simulators — LessWrongin corpus≈ 71%
- SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentsin corpus2025≈ 71%
Similar preprints — Semantic Scholar
Cited by (1)
- Interpreting Language Model Parameters
VPD (adVersarial Parameter Decomposition) decomposes weight matrices directly into rank-one interpretable subcomponents rather than decomposing activations as sparse autoencoders (SAEs) do, flipping t