Hugging Face Blog

PaliGemma 2 Mix - New Instruction Vision Language Models by Google

arXiv AI
Sep 7

MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models

MM-IFEval-Pro is a new multilingual benchmark for evaluating instruction-following in vision-language models, covering both Chinese and English tasks. It includes 4 major task categories, 24 subcategories, and 8 instruction categories with 52 subcategories, each sample featuring an average of 3.0 constraints to mimic complex instruction scenarios. A reinforcement-learning training set with Chinese and adversarial instructions improves model performance on MM-IFEval-Pro and transfers well to other multimodal benchmarks, showing strong cross-task and cross-language generalization.

By Changming Xiao, Zhenliang Ni, Jinhui He, Han Shu, Jie Hu