The paper introduces Propose-Then-Project Unlearning (PTP-U), a method for large language model unlearning that first weakens target knowledge associations and then projects non-target outputs back to the original distribution to recover capabilities. PTP-U aims to satisfy forgetting constraints while preserving fluent generation and performance on non-target tasks. Experiments on three benchmarks show PTP-U achieves a strong forgetting‑retention trade‑off, with 81.22%–91.03% forgetting and an average of 94.20% non‑target utility.
By Xunlei Chen, Qinghui Gong, Jingkun Xue, Qihe Liu, Shijie Zhou, Fei Ye
The paper introduces a Fisher-guided submodular data selection method for continual pre‑training of large language models, addressing the forgetting problem that arises when a target‑domain corpus overwrites pretrained knowledge. By decomposing candidate gradients into anchor and frontier components and optimizing a log‑determinant submodular objective, the method selects data that both improves target‑domain performance and limits forgetting. Experiments on TinyLlama‑1.1B and Llama‑3.1‑8B show that the selector achieves superior adaptation and forgetting control while using ten times fewer tokens than traditional replay strategies.
By Zhenghao Zhao, Gaowen Liu, Zhiling Lan, Yan Yan
arXiv:2605. 12765v3 Announce Type: replace Abstract: Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety.
By Vin\'icius Conte Turani, Ot\'avio Parraga, Jo\~ao Vitor Boer Abitante, Kristen K. Arguello, Joana Pasquali, Ramiro N. Barros, Flavio du Pin Calmon, Christian Mattjie, Rodrigo C. Barros, Lucas S. Kupssinsk\"u
arXiv:2609.37076v1 Announce Type: new
Abstract: Large language models trained on vast corpora inherently risk memorizing harmful content that may later re-emerge in their outputs. To mitigate this is...
By Puning Yang, Qizhou Wang, Junchi Yu, Bo Han, Xiuying Chen
arXiv:2609.16229v1 Announce Type: new
Abstract: Machine unlearning aims to remove specific knowledge from a trained large language model (LLM) without retraining from scratch. Existing methods modify...
By Pingzhi Li, Jinhao Duan, Vaishnav Tadiparthi, Nakul Agarwal, Kwonjoon Lee, Ehsan Moradi Pari, Hossein Nourkhiz Mahjoub, Sijia Liu, Tianlong Chen
arXiv:2606. 02920v1 Announce Type: new Abstract: Language-model unlearning updates a trained model to behave as if it had not seen selected training examples, while preserving utility and avoiding costly retraining.
By Federico Di Gennaro, Alexander Shevchenko, Fanny Yang
arXiv:2608. 06243v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level.
By ZhiYan Hou, Xinyu Tang, Hongyan An, Jianjin Zhang, Weizhen Wang, Yunyun Han, Gengsheng Li, Xiangzhao Hao, Haiyun Guo, Wenbin Hu, Jinqiao Wang, Yafeng Deng
The paper introduces Skill Schema Transport (SST), a post‑training framework that mitigates catastrophic forgetting in continual domain adaptation by translating new domain knowledge into a model’s existing skill space. SST abstracts each example type into a reusable skill schema, preserving essential operations while removing domain‑specific surface forms, thereby turning knowledge updates into procedural reuse rather than trajectory memorization. Experiments on two 3B instruction‑tuned backbones show that SST consistently reduces forgetting and improves generalization, achieving up to a 5.75‑point performance gain and a dramatic drop in maximum forgetting from 42.41 to 7.10 points on the Qwen‑3B MATH stream.
By Zijie Liu, Zhen Tan, Charles Fleming, Tianlong Chen
The paper introduces ADU, a fine‑grained training framework that unlearns sensitive information from large language models by decoupling contextual attention pathways instead of erasing tokens. ADU exploits the distinction between local and global attention heads to identify and suppress attention paths that retrieve persistent sensitive anchors, while preserving local‑attention structure and overall language modeling performance. Evaluation on the TOFU and WMDP benchmarks shows ADU achieves superior forget quality (0.93 on TOFU) and retains 92.9% of model utility compared to 81.9% for existing baselines, with fewer side effects in benign contexts.
By Xunlei Chen, Qirui Ye, Yuang Li, Yi Gong, Zhaokun Wang, Wenyi Li, Shiyao Guo, Jinyu Guo
arXiv:2607. 20433v1 Announce Type: cross Abstract: While language models remain frozen at their training state, the world evolves continuously.
By Jea Kwon, Jiwon Kim, Dong-kyum Kim, Meeyoung Cha
arXiv:2603. 06642v2 Announce Type: replace-cross Abstract: Test-Time Training (TTT) language models replace the KV-cache with fast weights updated during inference, achieving O(1) memory but suffering catastrophic failure on exact-recall tasks.
By Swamynathan V P
arXiv:2606. 06320v1 Announce Type: new Abstract: Machine unlearning aims to remove targeted knowledge from a trained model while preserving its general capabilities.
By Gizem Y\"uce, Giorgos Nikolaou, Nicolas Flammarion