Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv AI

Jun 2

Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments

arXiv:2605. 30280v2 Announce Type: replace-cross Abstract: Embodied intelligence is often studied through specialized models for individual tasks such as manipulation or navigation, resulting in fragmented capabilities and limited generalization across tasks, environments, and robot embodiments.

By Qiuyue Wang, Mingsheng Li, Jian Guan, Jinhui Ye, Sicheng Xie, Yitao Liu, Junhao Chen, Zhixuan Liang, Jie Zhang, Xintong Hu, Xuhong Huang, Pei Lin, Junyang Lin, Dayiheng Liu, Shuai Bai, Jingren Zhou, Jiazhao Zhang, Haoqi Yuan, Gengze Zhou, Hang Yin, Ye Wang, Yiyang Huang, Zixing Lei, Wujian Peng, Delin Chen, Yingming Zheng, Jingyang Fan, Xianwei Zhuang, Xin Zhou, Haoyang Li, Anzhe Chen, Tong Zhang, Xuejing Liu, Yuchong Sun, Ruizhe Chen, Zhaohai Li, Chenxu L\"u, Zhibo Yang, Tao Yu, Xionghui Chen

robotics multimodal benchmarks

arXiv Machine Learning

Jun 2

Flow-Transformed Implicit Processes for Function-Space Variational Inference

arXiv:2606. 01954v1 Announce Type: new Abstract: Implicit-process priors define distributions over functions through flexible generative mechanisms, making them attractive for Bayesian function-space modelling.

By Luis A. Ortega, Andr\'es R. Masegosa, Thomas D. Nielsen

multimodal

arXiv Machine Learning

Jun 2

Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation

arXiv:2606. 00253v1 Announce Type: cross Abstract: Fine-tuning Vision-Language-Action (VLA) models for mobile manipulators with heterogeneous joint spaces can produce a counterintuitive result: the checkpoint with the lowest aggregate MSE is not the one that performs best on the real robot.

By Pau Montagut Bofi, Mario Garc\'ia Blasco, Tessa Pulli, Markus Vincze

robotics fine-tuning multimodal

arXiv AI

Jun 2

LLMs Need Encoders for Semantic IDs Too

arXiv:2606. 00324v1 Announce Type: cross Abstract: Multimodal LLMs use dedicated encoders to bridge non-language modalities (vision encoders for images, depth models for audio codec tokens) because raw token embeddings alone cannot capture modality-specific structure.

By Xiangyi Chen, Zelun Wang, Xinyi Li, Yi-Ping Hsu, Jaewon Yang, Jiajing Xu

llms rag multimodal

arXiv AI

Jun 2

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

arXiv:2606. 00931v1 Announce Type: cross Abstract: Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks still focus largely on narrow appearance edits and do not fully capture the diversity of real-image tasks in professional workflows.

By Fangzhou Lin, Peiran Li, Lingyu Xu, Wenjing Chen, Qianwen Ge, Shuo Xing, Mingyang Wu, Xiangbo Gao, Siyuan Yang, Kazunori Yamada, Ziming Zhang, Haichong Zhang, Zhen Dong, Ming-Hsuan Yang, Zhengzhong Tu

agents computer-vision multimodal benchmarks

arXiv AI

Jun 2

Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning

arXiv:2503. 11832v5 Announce Type: replace Abstract: Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images.

By Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

Understanding the Effects of Distractors on Reasoning Vision-Language Models

arXiv:2511. 21397v2 Announce Type: replace-cross Abstract: How does irrelevant information (i.

By Jiyun Bae, Hyunjong Ok, Sangwoo Mo, Jaeho Lee

llms multimodal

arXiv Machine Learning

Jun 2

GLENS: Global Search via Learning from Solver Iterates with Diffusion Models

arXiv:2606. 00366v1 Announce Type: new Abstract: We consider the problem of generating a large collection of initial guesses for local minima of multimodal non-convex continuous optimization problems.

By Anjian Li, Bartolomeo Stellato, Ryne Beeson

diffusion robotics multimodal benchmarks

arXiv AI

Jun 2

Boosting Multimodal Federated Learning via Chained Modality Optimization

arXiv:2606. 01856v1 Announce Type: cross Abstract: Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability.

By Zixin Zhang, Fan Qi, Shuai Li, Xiaoshan Yang, Changsheng Xu

multimodal benchmarks safety

arXiv Machine Learning

Jun 2

DeepLatent: Think with Images via Parallel Latent Visual Reasoning

arXiv:2606. 00562v1 Announce Type: cross Abstract: The emerging paradigm of "thinking with images" embeds visual states into intermediate reasoning steps, defining a new frontier for Vision-Language Models.

By Dongchen Lu, Zhimo Li, Mao Shu, Huo Cao

llms rag reinforcement-learning robotics efficiency multimodal benchmarks

arXiv AI

Jun 2

CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects

arXiv:2510. 14904v4 Announce Type: replace-cross Abstract: Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language.

By Gabriel Fiastre, Antoine Yang, Cordelia Schmid

computer-vision multimodal benchmarks

arXiv AI

Jun 2

PATRA: Pattern-Aware Alignment and Balanced Reasoning for Time Series Question Answering

arXiv:2602. 23161v4 Announce Type: replace Abstract: Time series reasoning demands both the perception of complex dynamics and logical depth.

By Junkai Lu, Peng Chen, Xingjian Wu, Yang Shu, Chenjuan Guo, Christian S. Jensen, Bin Yang

llms nlp multimodal safety

arXiv AI

Jun 2

From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model

arXiv:2512. 05277v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Towards Sparse Video Understanding and Reasoning

arXiv:2602. 13602v2 Announce Type: replace-cross Abstract: We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA).

By Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han Liu

llms agents nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation

arXiv:2605. 23500v2 Announce Type: replace-cross Abstract: Segmentation is a fundamental task in computer vision, underpinning pixel-level scene understanding and serving as a cornerstone for applications ranging from autonomous perception to medical image analysis.

By Mario Markov (INSAIT, Sofia University "St. Kliment Ohridski"), Stefan Maria Ailuro (INSAIT, Sofia University "St. Kliment Ohridski"), Mohammad Mahdi (INSAIT, Sofia University "St. Kliment Ohridski"), Luc Van Gool (INSAIT, Sofia University "St. Kliment Ohridski"), Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")

llms agents computer-vision reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 2

Prior-Guided Multi-Omic Transformers for Single-Cell Gene Regulatory Network Inference

arXiv:2606. 00685v1 Announce Type: new Abstract: Gene regulatory networks (GRNs) capture transcription factor-target interactions and are central to understanding cell-state regulation and disease.

By Tianyang Xu, Tianci Liu, Niraj Rayamajhi, Ryan Patrick, Kranthi Varala, Ying Li, Jing Gao

llms multimodal

arXiv AI

Jun 2

STaR-KV: Spatio-Temporal Adaptive Re-weighting for KV Cache Compression in GUI Vision-Language Models

arXiv:2606. 01790v1 Announce Type: cross Abstract: Vision-language-model-based graphical user interface (GUI) agents have shown broad automation capabilities, yet deployment is bottlenecked by a key-value (KV) cache that grows linearly with interaction steps.

By Yuhang Han, Wenzheng Yang, Yujie Chen, Xiangqi Jin, Yaojie Zhang, Siteng Huang, Linfeng Zhang

llms agents efficiency multimodal benchmarks

arXiv AI

Jun 2

LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models

arXiv:2603. 00171v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) are shifting towards "Thinking with Images" by actively exploring image details.

By Yuxiang Shen, Hailong Huang, Zhenkun Gao, Xueheng Li, Man Zhou, Chengjun Xie, Haoxuan Che, Xuanhua He, Jie Zhang

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

arXiv:2601. 11460v2 Announce Type: replace-cross Abstract: Learning structured task representations from human demonstrations is essential for bimanual manipulation, where action ordering, object involvement, and interaction geometry vary significantly across executions.

By Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

llms robotics fine-tuning multimodal

arXiv AI

Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney

llms agents multimodal benchmarks safety

Get the daily briefing