Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 26

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

arXiv:2606. 26458v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchmarks largely overlook the challenges of retrieval in multimodal knowledge graph RAG (MKG-RAG).

By Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, Fenglong Ma

llms rag multimodal benchmarks

arXiv AI

Jun 26

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.

By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Huaqing Zheng, Xindian Ma, Peng Zhang

llms diffusion efficiency multimodal

arXiv AI

Jun 26

Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Models

arXiv:2505. 12343v2 Announce Type: replace-cross Abstract: Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucinations, where generated content is inconsistent with the input image.

By Kai Tang, Jinhao You, Yichen Guo, Yiding Sun, Dongxu Zhang, Wenya Wang, Hanze Li, Tao Luo, Renyuan Li, Xiande Huang

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 26

LAMP: Lane-Aligned Motion Primitives for Feasible Trajectory Prediction

arXiv:2606. 26661v1 Announce Type: cross Abstract: Motion forecasting is essential for autonomous driving systems to enable safe decision-making and planning in complex driving scenarios.

By Sangjin Han, Hoseong Jung, Jeongtae Her, Changhyun Choi, H. Jin Kim

agents robotics multimodal benchmarks safety

arXiv AI

Jun 26

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

arXiv:2606. 26794v1 Announce Type: cross Abstract: CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment.

By Sicheng Zhang, Muzammal Naseer, Binzhu Xie, Naufal Suryanto, Shi Qiu, Jamal Bentahar, Naveed Akhtar, Mubarak Shah

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 26

Learning Probabilistic Filters with Strictly Proper Scoring Rules

arXiv:2606. 26497v1 Announce Type: new Abstract: Bayesian filtering of partially and noisily observed dynamical systems seeks to infer the evolving conditional distribution of the state of a dynamical system, given observations, in an online fashion.

By Eviatar Bach, Ricardo Baptista, Jochen Br\"ocker, Bohan Chen, Andrew Stuart

llms multimodal safety

arXiv Machine Learning

Jun 26

\chisao{}: A GPU-Native Parallel Optimizer for Multimodal Black-Box Functions via Convergence-Anticonvergence Oscillation

arXiv:2606. 26164v1 Announce Type: new Abstract: Finding all modes of a multimodal black-box function is a fundamental challenge in optimization, Bayesian inference, and scientific computing.

By Ira Wolfson

multimodal benchmarks

arXiv Machine Learning

Jun 26

Conditional Flow Matching for Visually-Guided Acoustic Highlighting

arXiv:2602. 03762v4 Announce Type: replace-cross Abstract: Visually-guided acoustic highlighting seeks to rebalance audio in alignment with the accompanying video, creating a coherent audio-visual experience.

By Hugo Malard, Gael Le Lan, Daniel Wong, David Lou Alon, Yi-Chiao Wu, Sanjeel Parekh

diffusion multimodal benchmarks safety

arXiv AI

Jun 26

VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

arXiv:2606. 26534v1 Announce Type: cross Abstract: Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.

By Tianxin Xie, Chenxing Li, Dong Yu, Li Liu

diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 26

Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMs

arXiv:2606. 26387v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) extend large language models (LLMs) with visual perception, enabling joint reasoning over images and text.

By Xi Xiao, Chen Liu, Chih-Ting Liao, Yunbei Zhang, Qizhen Lan, Yuxiang Wei, Lin Zhao, Janet Wang, Jianyang Gu, Muchao Ye, Tianyang Wang, Hao Xu

llms reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 26

When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

arXiv:2606. 26473v1 Announce Type: new Abstract: Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction.

By Jaden Moon, Arvind Pillai, Andrew Campbell

nlp multimodal

arXiv Machine Learning

Jun 26

BrepCoder: A Unified Multimodal Large Language Model for Multi-task B-rep Reasoning

arXiv:2602. 22284v3 Announce Type: replace Abstract: Recent advancements in deep learning have actively addressed complex challenges within the Computer-Aided Design (CAD) domain.

By Mingi Kim, Yongjun Kim, Jungwoo Kang, Hyungki Kim

llms agents multimodal

arXiv AI

Jun 26

Semantic Generative Tuning for Unified Multimodal Models

arXiv:2605. 18714v2 Announce Type: replace-cross Abstract: Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture.

By Songsong Yu, Yuxin Chen, Ying Shan, Yanwei Li

computer-vision multimodal benchmarks

arXiv AI

Jun 26

From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

arXiv:2606. 26196v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have recently made remarkable progress in unifying vision-language understanding and reasoning, especially following the introduction of models such as OpenAI's O-series and DeepSeek's R-series, which have driven a paradigm shift toward perception-centric intelligence.

By Haoxiang Sun, Tao Wang, Li Yuan, Jian Zhao, Jiancheng Lv

llms multimodal

arXiv Machine Learning

Jun 26

Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA

arXiv:2606. 27023v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) applied to Medical Visual Question Answering (VQA) tend to produce overconfident outputs regardless of actual correctness, and existing verbalized confidence calibration methods, developed primarily for text only LLMs, do not account for the multimodal nature of medical image understanding.

By Eren Senoglu, Federico Toschi, Nicolo Brunello, Andrea Sassella, Mark James Carman

llms nlp fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 26

Listening Like a Judge: A Music-Aware Framework for Automatic Singing Performance Evaluation

arXiv:2606. 26451v1 Announce Type: cross Abstract: Automatic singing quality assessment (SQA) requires evaluating lyrical correctness and musical fidelity while handling expressive variations.

By Neelam Saini, Sourav Ghosh

rag fine-tuning multimodal safety

arXiv AI

Jun 26

MMGist: A Comprehensive Multimodal Benchmark for 2027

arXiv:2606. 22437v2 Announce Type: replace-cross Abstract: We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and therefore fail to effectively measure multimodal understanding; 2) many items are already close to performance saturation for current LVLMs, which limits their discriminative power; 3) a small number of anomalous items affect the reliability of evaluation results.

By Wenzhen Yuan, Jiacheng Ruan, Wutao Xiong, Chengping Zhao, Ting Liu, Yuzhuo Fu

multimodal benchmarks

arXiv AI

Jun 26

Automating Potential-based Reward Shaping with Vision Language Model Guidance

arXiv:2606. 27180v1 Announce Type: cross Abstract: Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory.

By Henrik M\"uller, Daniel Kudenko

llms agents reinforcement-learning multimodal

arXiv AI

Jun 26

MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models

arXiv:2603. 11625v2 Announce Type: replace-cross Abstract: While specialized Medical Vision-Language Models (VLMs) have achieved remarkable success in interpreting 2D and 3D medical modalities, their deployment for 3D volumetric data remains constrained by significant computational inefficiencies.

By Shengyuan Liu, Zanting Ye, Yunrui Lin, Chen Hu, Wanting Geng, Xu Han, Bulat Ibragimov, Yefeng Zheng, Yixuan Yuan

llms efficiency multimodal benchmarks

arXiv AI

Jun 26

Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline)

arXiv:2606. 27163v1 Announce Type: cross Abstract: I describe my solution to the LeHome Challenge 2026, an ICRA 2026 competition on bimanual garment folding.

By Ilia Larchenko

reinforcement-learning robotics multimodal safety

Get the daily briefing