arXiv AI
Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Oct 23
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
TRIAD is a new benchmark designed to test omni‑modal large language models’ ability to resolve ambiguity across text, vision, and audio. Each item consists of a text–image–audio triplet, a question, and answer options, constructed so that the full triplet yields a unique answer while any single modality removed makes the answer ambiguous. The benchmark includes 327 queries in 86 scene groups, covering 18 ambiguity categories, and shows that current models lag humans in scene‑level disambiguation, especially when answer options are removed.
Hugging Face Trending Papers
Oct 22
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
TRIAD is a new benchmark designed to test omni‑modal large language models’ ability to resolve ambiguity across text, vision, and audio. Each item consists of a text–image–audio triplet, a question, and answer options, crafted so that all three modalities together yield a unique answer while any single modality leaves the answer ambiguous. The benchmark contains 327 hand‑written queries in 86 scene groups, covering 18 ambiguity categories, and shows that current models lag humans significantly in scene‑level disambiguation, especially when answer options are removed.
arXiv AI
Jul 29
DynaBridge: Dynamic Summary-Guided Cross-Task Multimodal Fusion for DASS-Structured Mental Health Assessment
arXiv:2607. 25679v1 Announce Type: cross Abstract: Multimodal behavioral analysis offers a scalable approach to assessing depression, anxiety, and stress, yet generic fusion models often ignore the psychometric structure of questionnaire labels.
arXiv AI
Jul 29
Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases
arXiv:2607. 25933v1 Announce Type: cross Abstract: Clinical diagnostic evaluation should not only assess whether models can provide correct diagnoses, but also reflect the realities of clinical practice, including progressive disclosure of multimodal information, dynamic updating of diagnostic hypotheses, and continuous refinement of clinical reasoning.
arXiv AI
Aug 5
Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs
arXiv:2608. 00076v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) increasingly support high-stakes decision making by combining complementary information from images and text.
arXiv AI
Aug 5
Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition
arXiv:2608. 03475v1 Announce Type: cross Abstract: Multimodal intent recognition combines linguistic, acoustic, and visual evidence, but individual modalities may be noisy, missing, semantically conflicting, or disproportionately dominant.