arXiv AI
Jun 24
Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs
arXiv:2511. 20892v4 Announce Type: replace Abstract: Large language models (LLMs) often produce incorrect or outdated content after being employed.
LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing
arXiv:2606. 19679v1 Announce Type: cross Abstract: Lifelong knowledge editing aims to efficiently and sequentially update language models over time, as new knowledge becomes available or when the model makes mistakes, while preserving acceptable performance on past knowledge.
Related stories
arXiv AI
4d ago
Generalizable Lifelong Model Editing via Preference Optimization
The paper introduces GLIME, a lifelong model editing framework that integrates knowledge editing with preference optimization to handle continual updates in large language models. GLIME employs replay-based editing and a gradient constraint to prevent overfitting to target prompts and preserve previously edited knowledge. Experiments demonstrate that GLIME enhances knowledge generalization while maintaining editing performance and overall model capabilities.
arXiv AI
Aug 11
Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages
arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.
arXiv Machine Learning
Jul 22
More Edits, More Stable: Understanding the Lifelong Normalization in Sequential Model Editing
arXiv:2605. 11836v2 Announce Type: replace Abstract: Lifelong Model Editing aims to continuously update evolving facts in Large Language Models while preserving unrelated knowledge and general capabilities, yet it remains plagued by catastrophic forgetting and model collapse.
Hugging Face Trending Papers
Jul 29
ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models
Large language models (LLMs) have demonstrated strong capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquired knowledge under repeated updates remains insufficiently understood. Existing evaluation paradigms primarily focus on single-step reasoning or static knowledge editing, which fail to capture the temporal dynamics of knowledge retention and degradation during continual model modification.
arXiv AI
Jul 31
ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models
arXiv:2607. 26455v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquired knowledge under repeated updates remains insufficiently understood.
arXiv AI
Jul 28
MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models
arXiv:2607. 22556v1 Announce Type: new Abstract: Continual learning (CL) is essential for small language models (SLMs) to adapt to evolving real-world needs in resource-constrained deployments.
arXiv AI
Jun 4
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
arXiv:2605. 18879v3 Announce Type: replace-cross Abstract: Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety.
arXiv AI
6d ago
Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models
The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.
arXiv AI
Jun 16
Retrievable Gradients: Continual Post-Training Without Cumulative Weight Drift
arXiv:2606. 15734v1 Announce Type: cross Abstract: Continual post-training enables models to absorb emerging knowledge after deployment, but repeatedly updating shared parameters can accumulate weight drift, potentially causing catastrophic forgetting and degrading general capabilities.
arXiv Computation and Language
Aug 25
CALIBURN: Self-Calibrated LLM Unlearning Alignment
CALIBURN is a new approach to large language model (LLM) unlearning that measures a model’s confidence in undesirable knowledge and uses this measure to fine‑tune unlearning gradient updates. By doing so, it offers more precise control over what is forgotten while better preserving the model’s overall utility. Experiments on benchmarks such as MUSE and WMDP show that CALIBURN outperforms existing methods in balancing knowledge removal with utility retention.
arXiv Machine Learning
Sep 2
Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs
arXiv:2609.00184v1 Announce Type: cross Abstract: Large language models (LLMs) rely on static pretraining corpora, causing their knowledge to become outdated over time. Existing approaches for evalua...