Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,954 stories · RSS feed

arXiv AI

Jul 21

Fully-sensorized smart-eyewear platform for on-device Machine Learning

arXiv:2607. 16222v1 Announce Type: cross Abstract: This paper presents ARGO, a smart eyewear platform designed to bridge ergonomic comfort, high computational throughput, and energy efficiency.

By Andrea Giudici, Christian Veronesi, Pietro Bartoli, Mario Cali\`o, Aurelio Teliti, Giacomo Gervasoni, Diana Trojaniello, Franco Zappa

efficiency multimodal safety

arXiv AI

Jul 21

FUSAR-R1: A Large-Scale Reasoning Model for Intelligent Interpretation of SAR Images

arXiv:2607. 16819v1 Announce Type: new Abstract: In recent years, large-scale vision-language models have been driving a paradigm shift in intelligent remote sensing image interpretation.

By Yi Yang, Xiaokun Zhang, Yuxuan Li, Ruyi Zhang, Xinpeng Zhou, Haipeng Wang

llms reinforcement-learning multimodal

arXiv AI

Jul 21

Sparse Evidence Can Suffice: Agentic Evidence Seeking for Multimodal Video Misinformation Detection

arXiv:2607. 18080v1 Announce Type: cross Abstract: Multimodal video misinformation detection is commonly formulated as a holistic video-understanding task, where the entire video and its associated content are processed and judged in a single pass.

By Haochen Zhao, Yongxiu Xu, Xinkui Lin, Dong Xie, Jiarui Lu, Yuqi Qian, Yubin Wang, Hongbo Xu, Gaopeng Gou

agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 21

Multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis under unseen working conditions

arXiv:2512. 24679v2 Announce Type: replace Abstract: Intelligent fault diagnosis has become an indispensable technique for ensuring machinery reliability.

By Pengcheng Xia, Yixiang Huang, Chengjin Qin, Chengliang Liu

fine-tuning multimodal benchmarks

arXiv AI

Jul 21

ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

arXiv:2607. 17884v1 Announce Type: new Abstract: Vision Language Models (VLMs) achieve strong reasoning with Chain-of-Thought (CoT) prompting but incur high sequential-generation cost, error accumulation, and limited self-correction.

By Keuntae Kim, Beomseok Lee, Hyunwoo Kim, Yong Suk Choi

llms diffusion multimodal benchmarks

arXiv AI

Jul 21

Deterministic Hallucination Detection in Medical VQA via Confidence-Evidence Bayesian Gain

arXiv:2603. 21693v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have shown strong potential for medical Visual Question Answering (VQA), yet they remain prone to hallucinations, defined as generating responses that contradict the input image, posing serious risks in clinical settings.

By Mohammad Asadi, Tahoura Nedaee, Jack W. O'Sullivan, Euan Ashley, Ehsan Adeli

llms nlp multimodal benchmarks safety

arXiv AI

Jul 21

How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice

arXiv:2607. 16660v1 Announce Type: cross Abstract: The increasing adoption of Large Language Models (LLMs) as AI components in modern software systems introduces distinct security risks to the software supply chain.

By Mahzabin Tamanna, Elizabeth Lin, Sparsha Gowda, Laurie Williams, Dominik Wermke

llms agents multimodal safety

arXiv AI

Jul 21

PRISM: Multimodal Terrain Mapping for Rover Navigation in Unstructured Environments

arXiv:2607. 16366v1 Announce Type: cross Abstract: Robotic navigation in unstructured environments requires robust situational awareness to safely traverse hazards such as steep slopes and rocky terrain.

By Raul Castilla-Arquillo, Carlos Perez-del-Pulgar, Levin Gerdes, Alfonso Garcia-Cerezo, Miguel A. Olivares-Mendez

llms agents computer-vision robotics multimodal

arXiv AI

Jul 21

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

arXiv:2607. 18230v1 Announce Type: cross Abstract: Modern vision-language models (VLMs) have significantly improved image generation and editing capabilities, making pixel-level image tampering detection increasingly important yet challenging under cross-model and out-of-distribution shifts.

By Yi Tang, Xinyi Shang, Jiacheng Cui, Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tran Dinh Tien, Ahmed Elhagry, Salwa K. Al Khatib, Tianjun Yao, Yonina C. Eldar, Jing-Hao Xue, Hao Li, Salman Khan, Zhiqiang Shen

llms diffusion robotics multimodal benchmarks

arXiv AI

Jul 21

MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

arXiv:2507. 20804v3 Announce Type: replace Abstract: Large Language Models (LLMs) suffer from hallucinations due to their static parametric knowledge.

By Xueyao Wan, Hang Yu

llms rag multimodal benchmarks safety

arXiv AI

Jul 21

OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment

arXiv:2607. 17657v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) still struggle with spatial reasoning that requires perspective transformation.

By Wenxiao Fan, Hang Yin, Kan Li

llms multimodal safety

arXiv Machine Learning

Jul 21

Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models

arXiv:2607. 16506v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills.

By Yuhan Liu, Xinyu Zhang, Litao Liu, Abdeslam Boularias

reinforcement-learning robotics multimodal

arXiv AI

Jul 21

PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model

arXiv:2607. 17806v1 Announce Type: new Abstract: Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations.

By Li Xian, Mingxi Li, Yizheng Wang, Yiming Shen, Qi Chen, Zhuoling Xiao

llms rag agents robotics fine-tuning multimodal safety

arXiv AI

Jul 21

Dynamic Defense Profiling Enables Cognitive Jailbreak of Text-to-Image Models

arXiv:2607. 17779v1 Announce Type: new Abstract: Text-to-Image (T2I) generative models have achieved remarkable progress in synthesizing high-quality visual content, yet they remain vulnerable to adversarial misuse, particularly in generating Not-Safe-For-Work (NSFW) images.

By Dongdong Yang, Deyue Zhang, Zhao Liu, Zonghao Ying, Wenzhuo Xu, Jiankai Jin, Xiangzheng Zhang, Quanchen Zou

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 21

ChemFusion: A Multimodal Cross-Attention Network for Reaction Yield Prediction

arXiv:2607. 17033v1 Announce Type: new Abstract: Forecasting the outcomes of transition-metal-catalyzed reactions is notoriously complex due to the interplay of diverse physical and chemical variables.

By Qiwei Han, Chi Zhou

multimodal safety

arXiv AI

Jul 21

RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations

arXiv:2510. 00037v5 Announce Type: replace-cross Abstract: In Vision-Language-Actionf(VLA) models, robustness to real-world perturbations is critical for deployment.

By Jianing Guo, Zhenhong Wu, Chang Tu, Yiyao Ma, Xiangqi Kong, Zhiqian Liu, Jiaming Ji, Shuning Zhang, Yuanpei Chen, Kai Chen, Qi Dou, Yaodong Yang, Xianglong Liu, Huijie Zhao, Weifeng Lv, Simin Li

llms diffusion reinforcement-learning robotics multimodal safety

arXiv AI

Jul 21

Pailitao-MMSearch: Building Native E-Commerce Multimodal Search Foundation

arXiv:2607. 17499v1 Announce Type: new Abstract: The evolution of e-commerce has fundamentally transformed how users search for products, shifting from simple text-based keyword queries to complex multimodal interactions that seamlessly combine product images, natural language descriptions, and mixed-intent instructions.

By Xiaohan Ye, Xu Chen, Zihan Gong, Jian Ding, Lianyu Du, Baicheng Chen, Yunmeng Shu, Jingqian Zhao, Zhixiang Zhao, Shuaiqi Jia, Chong Ma, Shuwen Xiao, Xiangheng Kong, Yuan Gao, Jun Song, Jinsong Lan, Xiaoyong Zhu, Bo Zheng

llms multimodal

arXiv AI

Jul 21

An Explicit World Model Based on Data-First Ontology: DaoQL Multimodal Storage Validation and Counterfactual Reasoning Evaluation

arXiv:2607. 17269v1 Announce Type: new Abstract: Large language models encode world models implicitly in neural weights, which exposes four structural risks in high-precision domains such as medicine and finance: hallucination, frozen knowledge, poor explainability, and poor modifiability.

By Zhanbo Li, Shifeng Wu, Xiangjin Meng, Wenjie Cai

llms agents multimodal benchmarks safety

arXiv AI

Jul 21

Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

arXiv:2606. 25325v2 Announce Type: replace Abstract: We find that current emotion-oriented Omni-MLLMs still lack reliable omni-modal perception: they (i) underutilize multimodal cues in their reasoning trajectories and (ii) exhibit unfaithful behavior, often hallucinating modality-specific statements from other modalities.

By Zhiyuan Han, Beier Zhu, Wenwen Tong, Pengyang Shao, Peipei Song, Xinyi Wang, Jiangnan Chen, Lewei Lu, Xun Yang

reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 21

Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making

arXiv:2607. 17038v1 Announce Type: new Abstract: This paper addresses key technical challenges in current large language model (LLM) agent applications, including long-horizon planning, sparse reward attribution, and dynamic environmental interaction, by designing and optimizing an intelligent agent workflow.

By Amez Amanj Ali, Kuo-Kun Tseng

llms agents reinforcement-learning robotics multimodal benchmarks safety

Get the daily briefing