The paper investigates whether a language model’s forgetting—worsening performance on standard web text after fine‑tuning or further pretraining—can be mitigated by simple, fixed adjustments to its final token predictions. By fitting a vocabulary‑wide bias (a token‑specific bonus or penalty) and optionally a temperature scaling, the authors evaluate recovery across 14 base‑to‑derived model pairs ranging from 1B to 7B parameters. Results show modest gains: up to 10% recovery with bias alone and up to 36% when combined with temperature, though models further pretrained on math lose the most and recover little; even in a controlled simulation with random damage, the bias‑plus‑temperature fix recovers 29%.
arXiv:2610.09835v1 Announce Type: new Abstract: Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using o...
The paper introduces Propose-Then-Project Unlearning (PTP-U), a method for large language model unlearning that first weakens target knowledge associations and then projects non-target outputs back to the original distribution to recover capabilities. PTP-U aims to satisfy forgetting constraints while preserving fluent generation and performance on non-target tasks. Experiments on three benchmarks show PTP-U achieves a strong forgetting‑retention trade‑off, with 81.22%–91.03% forgetting and an average of 94.20% non‑target utility.
arXiv:2610.26864v1 Announce Type: cross Abstract: Gradient ascent (GA) is a simple language-model unlearning baseline, but it continues increasing forget loss after useful forgetting has occurred. Th...
CALIBURN is a new approach to large language model (LLM) unlearning that measures a model’s confidence in undesirable knowledge and uses this measure to fine‑tune unlearning gradient updates. By doing so, it offers more precise control over what is forgotten while better preserving the model’s overall utility. Experiments on benchmarks such as MUSE and WMDP show that CALIBURN outperforms existing methods in balancing knowledge removal with utility retention.
arXiv:2606. 10338v1 Announce Type: cross Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored.
The paper introduces SHIFT, a method for unlearning in large language models that avoids explicit forgetting targets by shrinking a fine‑tuned model toward its pre‑fine‑tuning weights and briefly relearning the retain set. In the linearized regime, this shrink‑and‑heal lookahead moves the model a fixed fraction toward the retrained model, and SHIFT extrapolates these per‑token changes into distillation targets for iterative refinement. Evaluated against eleven baselines under the OpenUnlearning protocol, SHIFT achieves the highest aggregate score across all TOFU settings, aligns 3.7× closer to the oracle than the next best method, and preserves the most retain knowledge on MUSE, maintaining proximity to the oracle even under larger optimization budgets and relearning attacks.
The paper introduces Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer‑selective unlearning framework for large language models. FOM-UL uses a forget‑to‑retain significance score to identify transformer layers that strongly influence the forget set while being insensitive to the retain set, allowing targeted updates that preserve most of the model. Experiments on TOFU, KnowUnDo, and MUSE-style benchmarks show that FOM-UL reduces residual memorization and maintains utility better than several baselines, even after 8‑bit and 4‑bit post‑training quantization, and it also limits recovery of forgotten content in adversarial prompt tests.
arXiv:2605. 03229v2 Announce Type: replace-cross Abstract: Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting.
The paper introduces Skill Schema Transport (SST), a post‑training framework that mitigates catastrophic forgetting in continual domain adaptation by translating new domain knowledge into a model’s existing skill space. SST abstracts each example type into a reusable skill schema, preserving essential operations while removing domain‑specific surface forms, thereby turning knowledge updates into procedural reuse rather than trajectory memorization. Experiments on two 3B instruction‑tuned backbones show that SST consistently reduces forgetting and improves generalization, achieving up to a 5.75‑point performance gain and a dramatic drop in maximum forgetting from 42.41 to 7.10 points on the Qwen‑3B MATH stream.
Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content, creating privacy, safety, and regulatory concerns. Machine unlearning offers...
The paper introduces LARE, a method that adjusts domain weights during large language model pretraining by reweighting losses based on running loss statistics, rather than altering data exposure. Experiments on a 1.5B model trained with 240B tokens show that LARE improves performance across five benchmarks, achieving significant gains on GSM8K (+17.06 pp) and MATH (+9.00 pp). The approach demonstrates that balancing loss scales can enhance joint capabilities without increasing domain exposure.
arXiv:2606. 02920v1 Announce Type: new Abstract: Language-model unlearning updates a trained model to behave as if it had not seen selected training examples, while preserving utility and avoiding costly retraining.