Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,393 stories · RSS feed

arXiv AI

Jun 9

Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

arXiv:2606. 09331v1 Announce Type: cross Abstract: Omni-modal retrieval promises a single embedding space for text, image, video, document, and audio inputs, but building such a unified retriever is difficult since these modalities differ in data distribution, architecture, and optimization dynamics.

By Shiyu Li, Zhiyuan Hu, Yifan Wang, Peiming Li, Zheng Wei, Yang Tang

rag fine-tuning multimodal

arXiv Machine Learning

Jun 9

IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation

arXiv:2601. 04498v2 Announce Type: replace Abstract: Infographics are composite visual artifacts that combine data visualizations with textual and illustrative elements to communicate information.

By Yinghao Tang, Xueding Liu, Boyuan Zhang, Tingfeng Lan, Yupeng Xie, Jiale Lao, Yiyao Wang, Haoxuan Li, Tingting Gao, Bo Pan, Luoxuan Weng, Xiuqi Huang, Minfeng Zhu, Yingchaojie Feng, Yuyu Luo, Wei Chen

llms diffusion multimodal benchmarks

arXiv Machine Learning

Jun 9

Zero-Shot Semantic Re-Identification for Autonomous Driving: A VLM Baseline Study

arXiv:2606. 09362v1 Announce Type: cross Abstract: Re-Identification (ReID) in autonomous driving is typically formulated as a visual matching problem, where observations of vehicles, pedestrians, and cyclists are associated across time, frames, or camera views using learned appearance embeddings, often complemented by motion, geometric, or multimodal cues.

By Eduardo Borges, Manuel Abreu, Lu\'is Garrote, Urbano J. Nunes

llms rag agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 9

Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation

arXiv:2606. 07647v1 Announce Type: cross Abstract: Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge.

By Ruipeng Zhang, Zhihao Li, C. L. Philip Chen, Tong Zhang

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 9

PRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning

arXiv:2606. 09301v1 Announce Type: new Abstract: Multimodal federated graph learning (MM-FGL) aims to collaboratively learn from decentralized graphs with text and images.

By Zekai Chen, Miao Zhang, Jiayang Xing, Xunkai Li, Xun Wu, Rong-Hua Li, Guoren Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 9

C$^3$ache: Accelerating World Action Models with Cross Inference Chunk Cache

arXiv:2606. 08962v1 Announce Type: new Abstract: World Action Models (WAMs) generalize better than standard Vision-Language-Action (VLA) policies to novel motions and environments, because a video-modeling objective lets them learn from abundant unlabeled video rather than scarce labeled robot demonstrations.

By Weisen Zhao, Lam Nguyen, Zhicong Lu, Yuzhang Shang

robotics multimodal benchmarks

arXiv AI

Jun 9

Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization

arXiv:2510. 10028v2 Announce Type: replace-cross Abstract: The rapid advancement of Low-Altitude Economy Networks (LAENets) has enabled a variety of applications, including aerial surveillance, environmental sensing, and semantic data collection.

By Yang Li, Ruichen Zhang, Yinqiu Liu, Guangyuan Liu, Abbas Jamalipour, Xianbin Wang, Dong In Kim

llms nlp reinforcement-learning multimodal

arXiv AI

Jun 9

ATN3D: Density-Aware LiDAR-Radar Early 3D Object Detection Under Extreme Sparsity

arXiv:2606. 09634v1 Announce Type: cross Abstract: 3D object detection is the backbone of perception for automated vehicles (AV) and broader intelligent transportation systems applications.

By Debojyoti Biswas, Xianbiao Hu

computer-vision efficiency multimodal benchmarks

arXiv AI

Jun 9

FMplex: Model Virtualization for Serving Extensible Foundation Models

arXiv:2606. 09643v1 Announce Type: cross Abstract: Foundation models (FMs) are increasingly used as backbones for downstream tasks across language, vision, time-series, and multimodal applications.

By Hetvi Shastri, Pragya Sharma, Walid A. Hanafy, David Irwin, Mani Srivastava, Prashant Shenoy

multimodal

arXiv AI

Jun 9

Stage-1 Controls the Entropy Regime, Not the Outcome

arXiv:2606. 09059v1 Announce Type: cross Abstract: Two-stage post-training -- a Stage-1 warm-start (supervised fine-tuning, SFT, or on-policy distillation, OPD) followed by Stage-2 reinforcement learning (RL) -- is increasingly used for vision-language models (VLMs).

By Jianxiong Shen

llms reinforcement-learning fine-tuning efficiency multimodal

arXiv AI

Jun 9

Reinforcement Learning for Flow-Matching Policies with Density Transport

arXiv:2606. 08602v1 Announce Type: cross Abstract: We present an online reinforcement learning (RL) algorithm for fine-tuning flow-matching policies in continuous-control problems.

By Boshu Lei, Kostas Daniilidis, Antonio Loquercio

diffusion reinforcement-learning robotics fine-tuning efficiency multimodal safety

arXiv AI

Jun 9

Aligned but Not Partner-Specific: Distinguishing How Multimodal LLM Agents Succeed in Reference Games Without Human-Like Conventions

arXiv:2606. 08081v1 Announce Type: cross Abstract: Repeated reference games test whether interlocutors replace their initially long descriptions with shorter, partner-specific conventions grounded in shared interaction history.

By Po-Ya Angela Wang, Chinmaya Mishra, Asl{\i} \"Ozy\"urek, Paula Rubio-Fern\'andez, Esam Ghaleb

llms agents multimodal safety

arXiv AI

Jun 9

Decoupling Semantics and Logic: A Training-Free Coarse-to-Fine Pipeline for Video Retrieval-Augmented Generation

arXiv:2606. 07924v1 Announce Type: cross Abstract: This paper presents our system description for the 2nd Workshop on Multimodal Augmented Generation via MultimodAl Retrieval (MAGMaR).

By Jiaxin Dai, Zehang Wei, Jiamin Yan, Xiang Xiang

llms rag agents computer-vision efficiency multimodal safety

arXiv AI

Jun 9

POISE: Position-Aware Undetectable Skill Injection on LLM Agents

arXiv:2606. 07943v1 Announce Type: cross Abstract: Agent skills provide a lightweight mechanism for extending general-purpose agents, but their open format exposes them to skill-poisoning attacks.

By Haochang Hao, Dehai Min, Zhifang Zhang, Yunbei Zhang, Miao Xu, Yingqiang Ge, Lu Cheng

llms agents multimodal benchmarks

arXiv AI

Jun 9

Page image classifier fine-tuned on century-spanning archives of scanned documents for further content-specific processing

arXiv:2606. 07558v1 Announce Type: cross Abstract: Purpose: Digitization projects in the humanities produce vast, heterogeneous archives of historical documents, making manual sorting impractical at scale.

By Kateryna Lutsai, Pavel Stra\v{n}\'ak, David Nov\'ak, Dana K\v{r}iv\'ankov\'a

llms computer-vision fine-tuning multimodal

arXiv AI

Jun 9

Blockchain Infrastructure for Intelligent Cyber--Physical--Social Systems:Post-Quantum Security, Interoperability, and Trustworthy Data Economies in the Era of Embodied AI

arXiv:2606. 06895v1 Announce Type: cross Abstract: The deployment of embodied artificial intelligence via world-model-based robotics presents a transformative opportunity for blockchain infrastructure, establishing urgent demand for trustworthy data provenance, cross-organizational governance, and incentive-compatible sharing across decentralized ecosystems.

By Song Guo, Huawei Huang, Dongping Liu, Aoyu Zhang, Luyao Zhang

robotics multimodal

arXiv AI

Jun 9

Self-Evolving Scientific Agent Discovers Generalizable Physically-Reasoned Fluid Control

arXiv:2606. 08405v1 Announce Type: new Abstract: While data-intensive deep reinforcement learning can optimize complex control policies, scientific discovery in physical systems fundamentally requires an interpretable chain of reasoning that connects physical evidence to structured control architectures.

By Boai Sun, Wenjin Guo, Zongmin Yu, Liu Yang

llms agents reinforcement-learning multimodal safety

arXiv AI

Jun 9

Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines

arXiv:2606. 07953v1 Announce Type: new Abstract: Large-scale Visual-Language Models (LVLMs) have achieved remarkable success in natural visual tasks, yet their application to industrial defect detection remains challenging due to two fundamental limitations: (i) the scarcity of large-scale industrial datasets that cover diverse defect categories across multiple domains, and (ii) the reliance on manual prompts (points, boxes, masks) that introduce subjective noise and lack text-visual interaction for fine-grained understanding.

By Zekai Zhang, Jinglin Zhang, Qinghui Chen, Gang Li, Da Chen, Shuainan Jing, He Wang, Dagang Li, Cong Liu, Cong Bai, Shengyong Chen

llms multimodal benchmarks safety

arXiv AI

Jun 9

VisualLeakBench: Reproducible Action-Boundary Propagation Failures in Vision-Language Agents

arXiv:2606. 07595v1 Announce Type: cross Abstract: Vision-language agents increasingly consume screenshots, documents, and user interfaces before writing to memory, sending messages, or invoking external tools.

By Youting Wang, Yuan Tang, Yitian Qian, Chen Zhao

agents multimodal benchmarks

arXiv Machine Learning

Jun 9

Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization

arXiv:2606. 09091v1 Announce Type: new Abstract: On-policy distillation (OPD) has recently emerged as an important post-training paradigm.

By Dongze Hao, Zhiwei Jin, Chen Chen, Haonan Lu

reinforcement-learning efficiency multimodal

Get the daily briefing