What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

The paper introduces MIMO-Bench, a new benchmark designed to test Multimodal Large Language Models (MLLMs) on visual reasoning tasks that involve multiple instances of the same object category. Unlike existing benchmarks that focus on images with few object instances, MIMO-Bench contains 3,802 image-question pairs requiring the model to disambiguate among similar objects using attribute binding and numerical counting. Results show that state‑of‑the‑art models, both open‑source and proprietary, perform significantly worse on this benchmark, underscoring their limitations in handling multi‑instance, multi‑object scenarios.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

• llms
• multimodal
• benchmarks