← Back to all news
arXiv AI By Fei Wang, Baochun Li

Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA

Read the original on arXiv AI →

The paper investigates how memorization behaves during fine‑tuning of large language models with LoRA, a parameter‑efficient technique. It finds that LoRA fine‑tuning diverges from previous results seen in full fine‑tuning and pre‑training: factors like model size and data duplication no longer predict memorization in the same way. Using a relaxed similarity‑based metric, the authors show that LoRA substantially lowers memorization risk while preserving strong task performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 24

RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning

RapidUn is a parameter reweighting framework that uses influence estimates to guide LoRA-only updates for efficient unlearning of targeted behaviors in large language models. It operates in a practical PEFT setting with a small forget set and limited retain buffer, converting cross-sample influence into fixed sample-specific weights for weighted LoRA unlearning. Experiments on Llama‑3‑8B with Dolly‑15k and Alpaca‑57k datasets show RapidUn achieves lower trigger ASR than Fisher, GA, and LoReUn while preserving clean utility, and delivers a 77× wall‑clock speedup over clean‑corpus LoRA retraining, with additional evaluations supporting its effectiveness.

By Guoshenghui Zhao, Huawei Lin, Weijie Zhao
llmsfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Oct 1

Mitigating Memorization In Language Models

The paper explores ways to reduce the memorization of training data in language models, testing three regularizer-based, three finetuning-based, and eleven machine unlearning methods—five of which are newly introduced. It introduces TinyMem, a lightweight suite of small models for rapid testing of these mitigation techniques, and shows that unlearning methods, particularly BalancedSubnet, outperform others in removing memorized content while maintaining task performance. The study also finds that regularizer-based approaches are slow and ineffective, while finetuning methods are costly, especially when high accuracy is required.

By Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Nathaniel Hudson, Caleb Geniesse, Kyle Chard, Yaoqing Yang, Ian Foster, Michael W. Mahoney
llmsfine-tuning
More like this →
arXiv Machine Learning
Jul 7

Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

arXiv:2607. 04969v1 Announce Type: new Abstract: The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency.

By Jingwei Zuo, Cong Zeng, Ilyas Chahed, Maksim Velikanov, Dhia Eddine Rhaiem, Pasquale Balsebre, Abhay Kumar, Younes Belkada, Hakim Hacid
llms
More like this →