Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,863 stories · RSS feed

arXiv AI

Aug 6

DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models

arXiv:2511. 15669v3 Announce Type: replace-cross Abstract: Does Chain-of-Thought (CoT) reasoning genuinely improve Vision Language Action (VLA) models, or does it merely add overhead?

By Cheng Yin, Yankai Lin, Wang Xu, Sikyuen Tam, Xiangrui Zeng, Zhiyuan Liu, Zhouping Yin

reinforcement-learning robotics fine-tuning multimodal safety

arXiv Machine Learning

Aug 6

When Modalities Fail to Tango: Conformal Backdoor Detection in Multimodal Contrastive Learning

arXiv:2608. 04052v1 Announce Type: cross Abstract: Backdoor attacks in multimodal contrastive learning (MCL) have garnered growing attention in recent years, as many downstream tasks critically depend on pre-trained MCL models.

By Yiming Chen, Kemou Li, Haiwei Wu, Jiantao Zhou

multimodal

arXiv AI

Aug 6

The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering

arXiv:2608. 04589v1 Announce Type: cross Abstract: EgoCross is a cross-domain egocentric video question answering benchmark designed to evaluate whether multimodal large language models can generalize beyond common daily-life scenarios.

By Yuqian Fu, Tianwen Qian, Yanjun Li, Yu Li, Kunyu Peng, Xu Zheng, Yongqin Xian, Alessio Tonioni, Yanwei Fu, Xiaoling Wang, Danda Paudel, Federico Tombari, Luc Van Gool, Leyi Wu, Yifan Zhao, Jinjie Zhang, Yinchuan Li, Yingcong Chen, Zixu Li, Zhiwei Chen, Zhiheng Fu, Wenbo Wang, Yupeng Hu, Weili Guan, Liqiang Nie, Takuya Murakawa, Toru Tamaki, Yi Wen, Zhenglin Du, Zhengyang Li, Lingling Li, Licheng Jiao, Wenping Ma

llms nlp multimodal benchmarks

arXiv AI

Aug 6

Monte Carlo Tree Search for Table-to-Multimodal Report Generation

arXiv:2608. 04071v1 Announce Type: new Abstract: Automatically generating professional multimodal reports comprising both textual analysis and visual charts from structured tabular data is a critical challenge in data intelligence.

By Teng Lin, Zhiyang Zhang, Yuyu Luo, Nan Tang

llms multimodal benchmarks safety data-engineering

arXiv AI

Aug 6

GeoReward: Mitigating Contextual Variable Overestimation in Vision-Language Models for Cross-Market Preference Prediction

arXiv:2608. 04504v1 Announce Type: cross Abstract: Vision-language models excel in many multimodal tasks but remain prone to a subtle yet impactful failure mode: they tend to overestimate dominant visual-textual cues while underestimating sparse but decision-critical contextual variables.

By Shuo Liu, Huixiang Cai, Weiru Zhang, Xiaoyi Zeng

llms diffusion reinforcement-learning fine-tuning multimodal safety

arXiv AI

Aug 6

C$^2$MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning

arXiv:2608. 04013v1 Announce Type: cross Abstract: Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs.

By Yuntao Shou, Tao Meng, Wei Ai, Keqin Li

multimodal benchmarks

arXiv AI

Aug 6

Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding

arXiv:2608. 04054v1 Announce Type: cross Abstract: Multimodal intent recognition requires understanding not only what textual, acoustic, and visual signals share, but also how they disagree.

By Mohnish Raj, Suraj Kumar, Soumi Chattopadhayay, Chandranath Adak, Ayan Dutta

multimodal benchmarks safety

arXiv AI

Aug 6

Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

arXiv:2608. 04698v1 Announce Type: cross Abstract: We tackle the challenging yet underexplored task of Generalized Referring Expression Comprehension (GREC), which requires a model to localize the object described by a textual expression when it exists (positive sample) and to refuse output when it does not (negative sample).

By Xuzheng Yang, Jun Ling, Tao Huang, Caiyan Qin, Peng Wang

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Aug 6

MoCA: Multi-modal Cross-masked Autoencoder for Digital Health Measurements

arXiv:2506. 02260v4 Announce Type: replace-cross Abstract: Wearable devices enable continuous multi-modal physiological and behavioral monitoring, yet analysis of these data streams faces fundamental challenges including the lack of gold-standard labels and incomplete sensor data.

By Howon Ryu, Yuliang Chen, Yacun Wang, Andrea Z. LaCroix, Chongzhi Di, Loki Natarajan, Yu Wang, Jingjing Zou

llms multimodal benchmarks

arXiv AI

Aug 6

RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation

arXiv:2510. 17640v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have shown strong manipulation capability when trained with large-scale imitation learning datasets.

By Yuquan Xue, Guanxing Lu, Zhenyu Wu, Chuanrui Zhang, Bofang Jia, Zhengyi Gu, Ziwei Wang

robotics multimodal benchmarks

arXiv AI

Aug 6

Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

arXiv:2511. 19418v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.

By Yiming Qin, Bomin Wei, Jiaxin Ge, Konstantinos Kallidromitis, Stephanie Fu, Trevor Darrell, XuDong Wang

llms computer-vision multimodal benchmarks safety

arXiv AI

Aug 6

CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models

arXiv:2608. 04509v1 Announce Type: new Abstract: Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer.

By De Jiang, Zhengyang Zhang, Kehong Yuan, Shaohua Ma

llms multimodal safety

arXiv Machine Learning

Aug 6

Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models

arXiv:2608. 04454v1 Announce Type: cross Abstract: Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool.

By Hongyu Zhang, Cheng Yan, Xiang Xia, Wuyang Zhang

llms multimodal benchmarks

arXiv AI

Aug 6

PURPOSE: Poisoning Conflict Resolution in RAG via Proxy-Fact-Grounded Updates

arXiv:2608. 04756v1 Announce Type: cross Abstract: In Retrieval-Augmented Generation (RAG), post-retrieval conflict resolution arbitrates among noisy or contradictory retrieved passages.

By Zijian Wang, Yubo Zhu, Muzhi Dong, Yanjun Lou, Yisheng Li, ZiLiang Zhang, Wei Tong, Yuan Zhang, Jingyu Hua, Sheng Zhong

rag multimodal benchmarks

arXiv AI

Aug 6

Q-CueGraph: Query-Conditioned Visual Evidence Graphs for Multimodal Reasoning

arXiv:2608. 04452v1 Announce Type: cross Abstract: High-resolution pixels and crop or zoom tools give multimodal large language models the ability to inspect an image, but they do not provide a reliable task-conditioned policy for deciding where to inspect.

By Pengcheng Pan, Xinfang Zhang

llms computer-vision multimodal benchmarks

arXiv AI

Aug 6

iStructTab: Structured Feature Sequencing for Multimodal Learning of Image and Tabular Data

arXiv:2608. 04348v1 Announce Type: cross Abstract: Multimodal learning of images and tabular data is often impaired by ineffective representations, resulting in redundancy, dispersion, and generalization problems.

By Al Zadid Sultan Bin Habib, Md Younus Ahamed, Prashnna Gyawali, Gianfranco Doretto, Donald A. Adjeroh

llms multimodal benchmarks

arXiv AI

Aug 6

EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment

arXiv:2608. 04472v1 Announce Type: cross Abstract: The development of foundation models (FMs) is crucial for advancing endoscopic image analysis.

By Zhenyu Yi, Jianwei Xu, Yue Hu, Zhongwei Qiu, Sijing Li, Liang Huang, Bin Lv, Ling Zhang, Yingda Xia

efficiency multimodal safety

arXiv AI

Aug 6

HyPASE: Hyperbolic Geometry for Parameter-Efficient Speech Emotion Fine-Tuning Framework for Large Audio-Language Models

arXiv:2608. 04351v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) excel at general speech understanding; however, adapting them to fine-grained tasks like Speech Emotion Recognition (SER) remains a significant bottleneck.

By Tian Jin, Ruikang Zhang, Zefeng Zhao, Ding Luo, Jin Zeng

llms fine-tuning multimodal benchmarks

arXiv AI

Aug 6

Dynamic Jailbreaking Attack

arXiv:2510. 02422v4 Announce Type: replace-cross Abstract: Existing gradient-based jailbreak attacks typically optimize a fixed-length adversarial suffix toward a predefined target response with a static optimization strategy.

By Kedong Xiu, Yunhan Yang, Churui Zeng, Tianhang Zheng, Xinzhe Huang, Di Wang, Puning Zhao, Zhan Qin, Kui Ren

llms multimodal safety

arXiv Machine Learning

Aug 6

One Surrogate to Fool Them All: Universal, Transferable, and Targeted Adversarial Attacks with CLIP

arXiv:2505. 19840v3 Announce Type: replace-cross Abstract: Deep Neural Networks (DNNs) have achieved widespread success yet remain prone to adversarial attacks.

By Binyan Xu, Xilin Dai, Di Tang, Kehuan Zhang

llms multimodal safety

Get the daily briefing