arXiv Machine Learning
An Information Bottleneck for Post-hoc Memorization Mitigation in LLMs
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 1
Mitigating Memorization In Language Models
The paper explores ways to reduce the memorization of training data in language models, testing three regularizer-based, three finetuning-based, and eleven machine unlearning methods—five of which are newly introduced. It introduces TinyMem, a lightweight suite of small models for rapid testing of these mitigation techniques, and shows that unlearning methods, particularly BalancedSubnet, outperform others in removing memorized content while maintaining task performance. The study also finds that regularizer-based approaches are slow and ineffective, while finetuning methods are costly, especially when high accuracy is required.
arXiv Machine Learning
Oct 16
Mitigating Memorization Where It Happens
The paper introduces a token‑level log‑likelihood ratio to pinpoint where large language models memorize training sequences. It proposes a probe‑steer framework, instantiated as Gated Subspace Steering (GSS), that selectively corrects memorization‑relevant activations using the leading singular vectors of a gradient‑weighted activation matrix. Experiments on four benchmarks show GSS matches or surpasses state‑of‑the‑art memorization reduction methods while using 10–100× less compute.
arXiv Machine Learning
Aug 14
A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models
arXiv:2602. 18733v2 Announce Type: replace Abstract: Training data leakage from Large Language Models (LLMs) raises serious concerns related to privacy, security, and copyright compliance.
Hugging Face Trending Papers
Sep 10
Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss
Large language models are typically trained under uniform token weighting, which allows frequent and low-information tokens to dominate learning and can increase the tendency to memorize surface-level...
arXiv Machine Learning
Sep 11
Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss
The paper proposes an information-weighted cross‑entropy loss that rescales token contributions using TF‑IDF statistics, thereby emphasizing semantically informative tokens and down‑weighting ubiquitous ones. Experiments on five decoder‑only language models (1.1B–13B parameters) show consistent reductions in memorized substring length while maintaining perplexity and downstream performance. The method is architecture‑agnostic, adds less than 3% computational overhead, and can be integrated into existing training pipelines.
arXiv AI
Jun 30
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.