arXiv Machine Learning
Aug 6
Unforgettable Generalization in Language Models
arXiv:2409. 02228v2 Announce Type: replace Abstract: When language models (LMs) are trained to forget (or "unlearn'') a skill, how precisely does their behavior change?
The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting
arXiv:2607. 29503v1 Announce Type: new Abstract: While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is.
Related stories
arXiv Machine Learning
Jul 28
Forgetting is Everywhere
arXiv:2511. 04666v4 Announce Type: replace Abstract: A fundamental challenge in developing general learning algorithms is their tendency to forget past knowledge as they adapt to new data.
arXiv AI
2d ago
Local Support Learning
arXiv:2610.02126v1 Announce Type: cross Abstract: We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of...
arXiv Machine Learning
Jun 16
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
arXiv:2601. 18699v2 Announce Type: replace Abstract: Sequential fine-tuning of Large Language Models (LLMs) adaptation to target tasks often triggers catastrophic forgetting, where the acquisition of novel target skills degrades ancestral capabilities.
arXiv AI
Jun 30
Informational Frustration in Neural Manifolds: Shannon Bottlenecks and the Limits of Learnability
arXiv:2606. 30512v1 Announce Type: cross Abstract: Why overparameterised deep networks generalise so remarkably well remains one of the most stubborn open questions in machine learning theory.
arXiv Computer Vision
4d ago
Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Models
arXiv:2609.37537v1 Announce Type: new Abstract: Machine unlearning has emerged as a critical post-hoc safety measure to erase sensitive concepts from Text-to-Image (T2I) models without prohibitive re...
Hugging Face Trending Papers
Jul 13
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization.
arXiv Machine Learning
Aug 31
DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging
The paper introduces DARTS, a method for tuning decoder representations during model merging. It addresses representation bias in autoregressive decoders by using an entropy‑weighted L1 loss and a per‑position additive bias to correct errors that accumulate across token positions. Experiments on code generation, mathematical reasoning, and instruction following with Llama‑2‑7B show that DARTS improves performance over standard surgery while adding only 0.1% extra parameters.
arXiv Machine Learning
Aug 13
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
arXiv:2608. 11690v1 Announce Type: new Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.
arXiv Machine Learning
Jul 14
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
arXiv:2607. 11883v1 Announce Type: new Abstract: Compression is fundamental to intelligence.
arXiv AI
Jul 22
Soft-TransFormers for Continual Learning
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
arXiv Machine Learning
Aug 4
Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping
arXiv:2608. 01833v1 Announce Type: cross Abstract: Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization.