Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMs

The paper discusses the rapid progress of machine unlearning for multimodal large language models (MLLMs) and the need for rigorous evaluation. It identifies three pitfalls in current benchmarks: lack of held‑out evaluation, reliance on synthetic fine‑tuned knowledge, and unfair comparisons across methods. To address these issues, the authors propose GEMMU, a benchmark using real‑world knowledge and a principled protocol, and show that re‑evaluating existing methods under GEMMU uncovers significant generalization gaps.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• fine-tuning
• multimodal
• benchmarks