Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,626 stories · RSS feed

arXiv AI

Jun 9

Late-Layer Fusion is Enough: Dual-Path Vision Token Routing for Multimodal Large Language Models under Visual Saturation

arXiv:2606. 09131v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) commonly inherit the deep, symmetric Transformer backbone designed for unimodal text modeling, and apply the same computation uniformly to image and language tokens.

By Siyuan Liu, Jinyang Wu

llms diffusion multimodal benchmarks

arXiv AI

Jun 9

Multimodal Group Emotion Recognition In-the-Wild Towards a Privacy-Safe Non-Individual Approach

arXiv:2606. 07585v1 Announce Type: cross Abstract: This thesis addresses group emotion recognition (GER) in-the-wild with a focus on privacy preservation.

By Anderson Augusma

diffusion multimodal benchmarks safety

arXiv AI

Jun 9

AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models

arXiv:2606. 08952v1 Announce Type: new Abstract: Multimodal Foundation Models (MFMs) have made substantial progress, yet remain fragile in spatial reasoning over the physical world.

By Shouwei Ruan, Bin Wang, Zhenyu Wu, Qihui Zhu, Yuxiang Zhang, Jingzhi Li, Yubin Wang, Xingxing Wei

agents reinforcement-learning multimodal

arXiv AI

Jun 9

BareWave: Waveform-Native Flow-Matching Text-to-Speech

arXiv:2606. 09048v1 Announce Type: cross Abstract: Removing intermediate representations and separately trained decoding stages has become an important direction in generative modeling.

By Wei Fan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, Nenghai Yu

multimodal safety

arXiv AI

Jun 9

GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation

arXiv:2606. 08530v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models achieve strong benchmark performance but still struggle in real-world deployment with unseen objects, background shifts, and different robot embodiments.

By Yuan Zhang, Shiqi Zhang, Yedong Shen, Shuai Dong, Jiajun Deng, Xin Zhang, Yuxuan Gao, Jiajia Wu, Xin Nie, Zhiyuan Cheng, Jianmin Ji, Yanyong Zhang, Xingyi Zhang, Jia Pan

robotics multimodal benchmarks

arXiv Machine Learning

Jun 9

State Backdoor: Towards Stealthy Real-world Poisoning Attack on Vision-Language-Action Model in State Space

arXiv:2601. 04266v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are widely deployed in safety-critical embodied AI applications such as robotics.

By Ji Guo, Wenbo Jiang, Yansong Lin, Yijing Liu, Ruichen Zhang, Guomin Lu, Aiguo Chen, Xinshuo Han, Hongwei Li

robotics multimodal safety

arXiv AI

Jun 9

Integrating Deep Learning Demand Forecasting with Multi-Objective Optimization for Circular Coffee Supply Chains: A Data-Driven Framework for Cost, Emissions, and Freshness Management

arXiv:2606. 08314v1 Announce Type: new Abstract: The coffee supply chain is one of the most complex agri-food networks, marked by geographically dispersed production, multi-tier coordination, and high sensitivity to quality and freshness.

By Ger\c{c}ek Budak (Department of Industrial Engineering, Ankara Y{\i}ld{\i}r{\i}m Beyaz{\i}t University, Ke\c{c}i\"oren, Ankara 06010, T\"urkiye), Faraz Gholamzadeh Gharehgheshlaghi (Department of Industrial Engineering, Ankara Y{\i}ld{\i}r{\i}m Beyaz{\i}t University, Ke\c{c}i\"oren, Ankara 06010, T\"urkiye), Melika Barjesteh Vaezi (Department of Kinesiology and Sport Management, Texas Tech University, Lubbock, TX, United States), Ahmad Gholizadeh Lonbar (Department of Civil, Construction, and Environmental Engineering, University of Alabama, Tuscaloosa, AL, USA)

multimodal benchmarks

arXiv Machine Learning

Jun 9

Zero-Shot Semantic Re-Identification for Autonomous Driving: A VLM Baseline Study

arXiv:2606. 09362v1 Announce Type: cross Abstract: Re-Identification (ReID) in autonomous driving is typically formulated as a visual matching problem, where observations of vehicles, pedestrians, and cyclists are associated across time, frames, or camera views using learned appearance embeddings, often complemented by motion, geometric, or multimodal cues.

By Eduardo Borges, Manuel Abreu, Lu\'is Garrote, Urbano J. Nunes

llms rag agents robotics multimodal benchmarks safety

arXiv AI

Jun 9

Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challenges

arXiv:2606. 09125v1 Announce Type: cross Abstract: Privacy risks in text-only Large Language Models (LLMs) are well studied, particularly their tendency to memorize and leak sensitive information.

By Tiejin Chen, Pingzhi Li, Kaixiong Zhou, Tianlong Chen, Hua Wei

llms multimodal safety

arXiv Machine Learning

Jun 9

PRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning

arXiv:2606. 09301v1 Announce Type: new Abstract: Multimodal federated graph learning (MM-FGL) aims to collaboratively learn from decentralized graphs with text and images.

By Zekai Chen, Miao Zhang, Jiayang Xing, Xunkai Li, Xun Wu, Rong-Hua Li, Guoren Wang

llms multimodal benchmarks

arXiv AI

Jun 9

Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

arXiv:2606. 08881v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored.

By Yi Yu, Xinchuan Qiu

robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 9

FiberTune: Preserving Action-Fiber Visual Residuals in Vision-Language-Action Fine-Tuning

arXiv:2606. 08653v1 Announce Type: cross Abstract: Action-supervised fine-tuning of vision-language-action (VLA) policies fits demonstrations effectively but constrains only the directions that change predicted actions, leaving visual structure consistent across action-equivalent states free to collapse.

By Haihao Lin, Xiangsheng Huang, Xiao Yang, Weibang Zhou, Yiqi Zhang, Bo Yang, Simin Zeng, Jiawei Yang, Zhengyang Wang, Jiahui Du

fine-tuning multimodal benchmarks safety

arXiv AI

Jun 9

MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework

arXiv:2606. 07654v1 Announce Type: cross Abstract: Multi-vector visual document retrievers achieve strong fine-grained matching by representing each page with multiple vectors from deep Vision-Language Models (VLMs), but this design makes deployment expensive in both storage and computational overhead.

By Haowen Xiang, Yibo Yan, Jiahao Huo, Yu Huang, Yi Cao, Mingdong Ou, Xuming Hu

llms multimodal

arXiv AI

Jun 9

vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models

arXiv:2606. 08094v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies are typically shipped as Python/PyTorch stacks that assume a workstation-class GPU, a mismatch for the hardware on which robots actually run.

By Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen, Thanh Q. Duong, Linh D. Le, Duy M. H. Nguyen, Vien A. Ngo, An T. Le

llms diffusion robotics multimodal benchmarks

arXiv AI

Jun 9

Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation

arXiv:2606. 07924v1 Announce Type: cross Abstract: This paper presents our system description for the 2nd Workshop on Multimodal Augmented Generation via MultimodAl Retrieval (MAGMaR).

By Jiaxin Dai, Zehang Wei, Jiamin Yan, Xiang Xiang

llms rag agents computer-vision efficiency multimodal safety

arXiv AI

Jun 9

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

arXiv:2606. 09169v1 Announce Type: new Abstract: In recent years, unified multimodal models (UMMs) have emerged to support both understanding and generation within a single framework.

By Lingyi Meng, Zecong Tang, Haoran Li, Tengju Ru, Zhejun Cui, Weitong Lian, Qi Kang, Hangshuo Cao, Yichen Zhu, Yechi Liu, Kaixuan Wang, Yu-Jie Yuan, Chunwei Wang, Yu Zhang, Bo Dai

multimodal benchmarks safety

arXiv AI

Jun 9

POISE: Position-Aware Undetectable Skill Injection on LLM Agents

arXiv:2606. 07943v1 Announce Type: cross Abstract: Agent skills provide a lightweight mechanism for extending general-purpose agents, but their open format exposes them to skill-poisoning attacks.

By Haochang Hao, Dehai Min, Zhifang Zhang, Yunbei Zhang, Miao Xu, Yingqiang Ge, Lu Cheng

llms agents multimodal benchmarks

arXiv AI

Jun 9

MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

arXiv:2606. 07706v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated strong performance across multimodal tasks, yet their safety robustness remains an open challenge.

By Rishabh Makwana, Mamta, Deeksha Varshney, Oana Cocarascu

llms multimodal benchmarks safety

arXiv AI

Jun 9

PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits

arXiv:2510. 17947v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are improving at an exceptional rate.

By Neeladri Bhuiya, Madhav Aggarwal, Diptanshu Purwar

llms agents multimodal benchmarks safety

arXiv AI

Jun 9

SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows

arXiv:2606. 08049v1 Announce Type: new Abstract: AI agents increasingly turn past experience into reusable artifacts such as code, workflows, and procedural memories.

By Amine El Hattami, Nicolas Chapados, Christopher Pal

agents multimodal

Get the daily briefing