The paper introduces JUPE, a benchmark that tests whether plausible explanations can mislead multimodal large language models (MLLMs) when judging answers that require visual verification. Across 14 MLLM judges, ordinary yet plausible explanations significantly increased acceptance of incorrect answers, with plain justifications and chain-of-thought rationales reducing rejections by 12.7 and 13.8 percentage points, respectively. Signal detection analysis shows a shift in decision criterion toward acceptance without a change in discriminability, indicating that explanations can inflate performance estimates in evaluation pipelines.
arXiv:2604. 14888v3 Announce Type: replace-cross Abstract: Recent advances in vision language models (VLMs) offer reasoning capabilities, yet how these unfold and integrate visual and textual information remains unclear.
The paper investigates whether Multimodal Large Language Models (MLLMs) can reason about visual persuasion. It finds that current models often shortcut by citing obvious visual elements instead of truly reasoning about diverse cues, and proposes a fine‑tuning method using multi‑perspective rationales along with an evaluation framework that separates prediction performance from rationale faithfulness. The study shows that while fine‑tuning improves prediction, higher accuracy does not guarantee more faithful reasoning, underscoring the need for distinct evaluation of effectiveness and faithfulness.
The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.
arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.
MagiC is a benchmark for grounded multimodal cognition that evaluates vision‑language models on answer accuracy, step‑by‑step reasoning quality, grounding fidelity, and self‑correction. It contains about 5,500 weakly supervised QA examples generated from strong model outputs and 900 human‑curated examples with fine‑grained annotations, including rationales and bounding box groundings. The benchmark tests 15 models ranging from 7B to 70B parameters, introduces new metrics such as MagiScore and StepSense, and includes diagnostic settings to probe robustness against adversarial visual cues and introspective error correction.
arXiv:2610.16441v1 Announce Type: cross Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Exp...
arXiv:2605. 28215v2 Announce Type: replace Abstract: In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labelled examples.
arXiv:2609.37656v1 Announce Type: new Abstract: Large vision-language models (LVLMs) exhibit strong reasoning capabilities, yet the visual and textual evidence supporting the generated responses rema...
The paper introduces MOTIVE, a multi-view self-verification framework for vision-language models that evaluates candidate answers from complementary perspectives and learns a reliability score aligned with correctness. By using this score to decide whether to accept an answer or trigger history-guided rethinking, MOTIVE improves reliability and efficiency compared to existing self-verification and self-correction methods. Extensive experiments on diverse multimodal benchmarks and VLM backbones show consistent performance gains and reduced unnecessary reasoning turns.
Melange is a synthetic visual bias benchmark designed to evaluate social biases in multi‑modal large language models (MLLMs). It extends the language‑only BBQ dataset by generating controlled visual scenarios that clearly depict target attributes, and validates images through automated VQA screening and human inspection. Using Melange, 23 open‑source MLLMs were tested in text‑only, image‑only, and image‑with‑context settings, revealing that switching from text to vision reduces fairness by an average of 0.291 and that models often choose stereotypical responses when biased answers are made.
arXiv:2608.28623v1 Announce Type: cross Abstract: Large multimodal reasoning models (LMRMs) are getting increasingly capable, primarily through generating explicit chain-of-thought reasoning before a...
The paper evaluates Vision Language Models (VLMs) on Visual Question Answering tasks where questions violate Grice's maxims. By generating question modifiers that add non-essential, ambiguous, or false information, the authors show that VLMs such as ChatGPT, Claude, Gemini, and Llava exhibit reduced performance. They also compare human pragmatic reasoning to VLM reasoning, noting differences in how each handles human‑induced versus AI‑generated violations, and find that humans spend less time resolving VLM‑induced violations while VLMs are less accurate in those cases.