Mitigating Memorization In Language Models

The paper explores ways to reduce the memorization of training data in language models, testing three regularizer-based, three finetuning-based, and eleven machine unlearning methods—five of which are newly introduced. It introduces TinyMem, a lightweight suite of small models for rapid testing of these mitigation techniques, and shows that unlearning methods, particularly BalancedSubnet, outperform others in removing memorized content while maintaining task performance. The study also finds that regularizer-based approaches are slow and ineffective, while finetuning methods are costly, especially when high accuracy is required.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• fine-tuning