The paper introduces GUARD, a method for natural forgetting in large reasoning models that transforms unsafe disclosures into safe-exit trajectories using guided answer‑reasoning distillation. It aligns a frozen model with guidance tokens and distills this behavior into the parameters, aiming for a coherent, non‑disclosing chain of thought followed by a refusal‑style answer. The authors also propose the Natural Forgetting Reasoning Score (NFRS) to evaluate structural stability, fluency, and unsupported substitutes, and demonstrate GUARD’s effectiveness on R‑TOFU and a STAR‑1‑derived harmful‑intent setting.
By Zeyu Yan, Guanghao Zhou, Minghui Qiu, Ming Gao, Cen Chen
arXiv:2610.14763v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) can reproduce sensitive, copyrighted, or unsafe information from their training data. Effective unlearning mus...
By Chenlu Ding, Jiancan Wu, Yanchen Luo, Tianyu Zhang, Yancheng Yuan, Xiang Wang, Xiangnan He
arXiv:2511. 20196v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) can inadvertently memorize privacy-sensitive information during training.
By Zhen Zeng, Leijiang Gu, Zhangling Duan, Feng Li, Cees G. M. Snoek, Meng Wang, Zenglin Shi
arXiv:2601.21702v4 Announce Type: replace
Abstract: We consider Representation Misdirection (RM), a class of large language model (LLM) unlearning methods that achieve forgetting by redirecting the l...
By Tien Dang, The-Hai Nguyen, Dinh Mai Phuong, Nguyen Minh Phuong, Anh Bui, Hoang Thanh-Tung, Le-Minh Nguyen, Naoya Inoue
arXiv:2604.03114v2 Announce Type: replace-cross
Abstract: Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy...
By Zhangyun Tan, Zeliang Zhang, Jiani Liu, Susan Liang, Yolo Y. Tang, Lisha Chen, Chenliang Xu
The paper introduces PACT, a method for unlearning deceptive behaviors in large language models by using contrastive forget sets that compare a model’s responses under deceptive and neutral contexts. PACT trains the model to produce pressure‑aware counterfactual targets, preserving benign system‑prompt adherence and reasoning traces while dramatically reducing deception rates from over 50% to under 3% on 32B reasoning models.
By Haoran Tang, Rajiv Khanna
arXiv:2409. 02228v2 Announce Type: replace Abstract: When language models (LMs) are trained to forget (or "unlearn'') a skill, how precisely does their behavior change?
By Eric Zhang, Leshem Choshen, Jacob Andreas
arXiv:2607. 02020v1 Announce Type: new Abstract: Multimodal large language models must continually adapt to evolving tasks and domains, yet standard continual learning metrics mainly measure whether old answers remain correct, leaving the stability of multimodal grounding largely unexamined.
By Qianyu Chen, Canran Xiao, Runxuan Tang
arXiv:2607. 22625v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge, but suffers from knowledge conflicts: when retrieved information contradicts parametric memory, the shared self-attention pathway produces unpredictable outputs.
By Chengzhang Yu, Chenyang Zheng, Zening Lu, Yingru He, Yutong Huang, Yiming Zhang, Yue Xu, Zhanpeng Jin
The paper demonstrates that off‑policy merging, called grafting, outperforms on‑policy self‑distillation for continual learning. Grafting learns updates on an earlier donor checkpoint, scales the update, and optionally masks sensitive directions, thereby reducing interference with existing capabilities. Across various continual learning scenarios, grafting achieves better new‑task and old‑task performance without costly on‑policy sampling.
The paper identifies a specific issue in supervised fine‑tuning (SFT) of large language models called factual access failure, where models can recognize correct facts under constrained tests but fail to generate them in open‑ended settings. It demonstrates that SFT can cause both genuine wrong answers and expression‑level errors such as verbosity or formatting mismatches. To mitigate this, the authors propose Recall‑Anchored Distillation (RAD), a self‑distillation method that aligns the fine‑tuned model with the base model’s soft output distribution on unlabeled out‑of‑distribution text, thereby recovering lost factual recall without needing labeled data.
By Haodong Chen, Yadong Wang, Shengtao Wen, Dong Liang, Xiang Chen
The paper introduces a method for identifying the dominant knowledge source behind large language model (LLM) outputs, distinguishing between faithfulness violations (misuse of provided context) and factuality violations (errors in internal knowledge). A simple linear probe trained on hidden representations can reliably classify this source, and the authors present AttriWiki, a self‑supervised pipeline that generates labeled training data by prompting models to recall withheld entities or read them from context. Probes trained on AttriWiki achieve high Macro‑F1 scores across several models and datasets, generalize zero‑shot to a benchmark, and show that attribution mismatches can increase error rates by up to 70%.
"whyItMatters":"The study demonstrates that knowing the source of an LLM’s answer is crucial for effective mitigation of hallucinations, as attribution mismatches significantly raise error rates."
By Ivo Brink, Alexander Boer, Dennis Ulmer