Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,069 stories · RSS feed

arXiv AI

Jun 30

RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

arXiv:2606. 29538v1 Announce Type: cross Abstract: Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge.

By Yijia Fan, Zonglin Di, Zimo Wen, Yifan Yang, Mingxi Cheng, Qi Dai, Bei Liu, Kai Qiu, Yue Dong, Ji Li, Chong Luo

agents multimodal

arXiv AI

Jun 30

BioProVLA-Agent: An Affordable, Protocol-Driven, Vision-Enhanced VLA-Enabled Embodied Multi-Agent System with Closed-Loop-Capable Reasoning for Biological Laboratory Manipulation

arXiv:2605. 07306v2 Announce Type: replace-cross Abstract: Biological laboratory automation can reduce repetitive manual work and improve reproducibility, but reliable embodied execution in wet-lab environments remains challenging.

By Zhaohui Du, Zhe Wang, Hongmei Fei, Xiwen Cao, Ting Xiao, Qi Wang, Huanbo Jin, Jiaming Gu, Quan Lu, Zhe Liu

llms rag agents robotics multimodal benchmarks

arXiv AI

Jun 30

LWDrive: Layer-Wise World-Model-Guided Vision-Language Model Planning for Autonomous Driving

arXiv:2606. 29879v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) provide powerful semantic understanding and commonsense reasoning for End-to-End Autonomous Driving (E2E-AD) planning.

By Chen Yang, Yuhao Wei, Ze Xu, Ziheng Zou, Shuang Liang, Delin Ouyang, Lingfeng Qi, Jie Li, Guofa Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 30

UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

arXiv:2603. 22282v2 Announce Type: replace-cross Abstract: We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture.

By Ziyi Wang, Xinshun Wang, Shuang Chen, Yang Cong, Mengyuan Liu

llms nlp efficiency multimodal benchmarks safety

arXiv AI

Jun 30

Linguistic Firewall: Geometry as Defense in Multi-Agent Systems Routing

arXiv:2606. 30555v1 Announce Type: new Abstract: The rapid integration of Large Language Models (LLMs) has driven the evolution of Multi-Agent Systems (MAS), where specialized agents collaborate to execute complex workflows.

By Dvir Alsheich, Adar Peleg, Ben Hagag, Rom Himelstein, Amit Levi, Avi Mendelson

llms rag agents robotics multimodal

arXiv Machine Learning

Jun 30

STEMGym: Benchmarking Sequential Decision-Making under Dose Budgets in Autonomous Electron Microscopy

arXiv:2606. 29592v1 Announce Type: new Abstract: A central premise of autonomous scientific imaging is that smarter navigation, whether Bayesian, RL-based, or otherwise adaptive, is the principal lever for sample-efficient acquisition.

By Can Polat, Erchin Serpedin, Mustafa Kurban, Hasan Kurban

llms agents reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 30

From Failure Taxonomy to Intervention: A Diagnostic Methodology for Industry-Scale AVLM in Video and Live-Streaming Platform Moderation

arXiv:2606. 30059v1 Announce Type: new Abstract: Industry-scale video and live-streaming moderation imposes requirements that are difficult to satisfy with generic pretrained public models or external APIs, including adaptation to platform-specific data distributions, policy-specific objectives, and product-level safety constraints.

By Shuchang Ye, Jinqiang Yu, Zhujun Xiao, Yajing Kong, Yist Y. Lin, Yang Ma, Jiaxi Liu, Xiaolei Xu, Zheng Yu

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 30

RadarTwin: Scene-Specific mmWave Radar Simulation and Learning for Mobile Indoor Perception

arXiv:2606. 28396v1 Announce Type: cross Abstract: Millimeter-wave (mmWave) radar perception is limited by data scarcity: models trained on existing radar datasets fail to generalize to new objects, environments, and sensing trajectories.

By Emily Bejerano, Federico Tondolo, Devang Gupta, Aaron Mano Cherian, Taeyoo Kim, Ayaan Qayyum, Xiaofan Yu, Xiaofan Jiang

llms robotics multimodal

arXiv Machine Learning

Jun 30

BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language

arXiv:2606. 30319v1 Announce Type: cross Abstract: Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience.

By Haitao Wu, Qirui Zhang, Zhouheng Yao, Shangquan Sun, Qihao Zheng, Mianxin Liu, Chi Zhang, Wanli Ouyang, Chunfeng Song, Changqing Zhang, Jiamin Wu

nlp multimodal benchmarks safety

arXiv Machine Learning

Jun 30

High-Entropy Tokens as Multimodal Failure Points in Vision-Language Models

arXiv:2512. 21815v4 Announce Type: replace-cross Abstract: Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks.

By Mengqi He, Xinyu Tian, Xin Shen, Jinhong Ni, Shu Zou, Zhaoyuan Yang, Jing Zhang

llms multimodal safety

arXiv Machine Learning

Jun 30

Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders

arXiv:2606. 29888v1 Announce Type: new Abstract: Vision-language models map images and text into a joint embedding space.

By Chungpa Lee, Jihoon Kwon, Kyle Min, Jy-yong Sohn

llms rag multimodal safety

arXiv Machine Learning

Jun 30

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

arXiv:2606. 29788v1 Announce Type: new Abstract: When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success.

By Kuan Wang, Chao Zhang

agents multimodal benchmarks

arXiv AI

Jun 30

VISTA-DZ: Visual Semantic Trajectory Adaptation for Personalized Dilemma Zone Prediction

arXiv:2606. 29548v1 Announce Type: cross Abstract: Driver decision making in the dilemma zone at signalized intersections is safety critical, as vehicles approaching a yellow signal must decide whether to stop or proceed within limited time and distance margins.

By Chuheng Wei, Ziye Qin, Ziran Wang, Guoyuan Wu

llms rag multimodal safety

arXiv AI

Jun 30

ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation

arXiv:2606. 28357v1 Announce Type: cross Abstract: Recent advances in multimodal recommenders excel at feature fusion but remain opaque and inefficient decision-makers, lacking explicit reasoning and self-awareness of uncertainty.

By Yihua Zhang, Mingfu Liang, Jiyan Yang, Rong Jin, Wen-Yen Chen, Yiping Han, Huayu Li, Buyun Zhang, Liang Luo, Frank Shyu, Luke Simon, Sijia Liu, Tianlong Chen, Xi Liu

agents fine-tuning efficiency multimodal benchmarks

arXiv AI

Jun 30

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

arXiv:2603. 23559v2 Announce Type: replace-cross Abstract: GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices.

By Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang

llms agents multimodal benchmarks

arXiv AI

Jun 30

Attribution Graphs and Causal Probing for Mechanistic Discovery and Bias Repair in Multimodal Generative Learning

arXiv:2510. 12957v4 Announce Type: replace-cross Abstract: We treat the internals of generative models as mechanistic objects rather than black boxes.

By Noor Islam S. Mohammad, Ulu\u{g} Bayaz{\i}t

fine-tuning multimodal safety

arXiv AI

Jun 30

AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

arXiv:2606. 29335v1 Announce Type: cross Abstract: Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions.

By Chuxiao Zuo, Yao Zhu, Minqiang Xu, Manhong Wang, Yunke Zhang, Fei Huang

rag multimodal

arXiv AI

Jun 30

Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification

arXiv:2606. 29023v1 Announce Type: cross Abstract: Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries.

By Tianshu Zhang, Yan Wang, Ji Qi, Lijie Wen

llms reinforcement-learning multimodal

arXiv AI

Jun 30

Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs

arXiv:2606. 29350v1 Announce Type: cross Abstract: Vision-language models and vision-language action models endow the robot with unprecedented capabilities.

By Junzhou Chen, Jindong Wang, Gang Zhou

llms nlp robotics efficiency multimodal

arXiv AI

Jun 30

CytoCLIP: Learning Cytoarchitectural Characteristics in Developing Human Brain Using Contrastive Language Image Pre-Training

arXiv:2601. 12282v2 Announce Type: replace-cross Abstract: The functions of different regions of the human brain are closely linked to their distinct cytoarchitecture, which is defined by the spatial arrangement and morphology of the cells.

By Pralaypati Ta, Sriram Venkatesaperumal, Keerthi Ram, Mohanasankar Sivaprakasam

llms multimodal

Get the daily briefing