Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,069 stories · RSS feed

arXiv AI

Jun 18

NAVI-Orbital: First In-Orbit Demonstration of a Zero-Shot Vision-Language Model for Autonomous Earth Observation

arXiv:2606. 18271v1 Announce Type: new Abstract: As Earth Observation data generation outpaces downlink bandwidth and human-in-the-loop processing, a widening gap has emerged between onboard collection and actionable ground intelligence.

By Juan Manuel Delfa Victoria, Taran Cyriac John, Andrew W. Herson

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jun 18

Semantic Router: On the Feasibility of Hijacking MLLMs via a Single Adversarial Perturbation

arXiv:2511. 20002v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed in stateless systems, such as autonomous driving and robotics.

By Changyue Li, Jiaying Li, Youliang Yuan, Jiaming He, Zhicong Huang, Pinjia He

llms agents diffusion robotics multimodal safety

arXiv AI

Jun 18

Generalized Kullback-Leibler Divergence Loss

arXiv:2503. 08038v2 Announce Type: replace-cross Abstract: In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of (1) a weighted Mean Square Error (wMSE) loss and (2) a Cross-Entropy loss incorporating soft labels.

By Jiequan Cui, Beier Zhu, Qingshan Xu, Zhuotao Tian, Xiaojuan Qi, Bei Yu, Hanwang Zhang, Richang Hong

efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 18

MetaboNet-Bench: A Multi-modal Benchmark for Glucose Forecasting in Type 1 Diabetes

arXiv:2606. 18640v1 Announce Type: new Abstract: Glucose forecasting algorithms are an important aspect of glycemic control management in type 1 diabetes.

By Nathaniel Jeffries, Miriam Wolff, Sam Royston, Elizabeth Healey, Caleb Mayer, David Klonoff, Michael Snyder, Tao Wang

multimodal benchmarks

arXiv AI

Jun 18

InstructTime++: Time Series Classification with Multimodal Language Modeling via Implicit Feature Enhancement

arXiv:2601. 14968v2 Announce Type: replace-cross Abstract: Most existing time series classification methods adopt a discriminative paradigm that maps input sequences directly to one-hot encoded class labels.

By Mingyue Cheng, Xiaoyu Tao, Huajian Zhang, Qi Liu, Zhiding Liu, Yucong Luo, Yiheng Chen, Enhong Chen

llms multimodal benchmarks safety

arXiv AI

Jun 18

Revealing Hidden Vulnerabilities in Autoencoders through Gradient Signal Restoration

arXiv:2505. 03646v5 Announce Type: replace-cross Abstract: Adversarial robustness of deep autoencoders (AEs) has received less attention than that of discriminative models, although their compressed latent representations induce ill-conditioned mappings that can amplify small input perturbations and destabilize reconstructions.

By Chethan Krishnamurthy Ramanaik, Arjun Roy, Tobias Callies, Eirini Ntoutsi

multimodal safety

arXiv AI

Jun 18

EffiNav: Fusing Depth and Vision-Language for Efficient Object Goal Navigation

arXiv:2606. 18634v1 Announce Type: cross Abstract: To locate a target object while exploring the unknown environment is a fundamental capability for autonomous agents, with applications ranging from search-and-rescue to field robots.

By Zecheng Yin, Benedict Jun Ma

agents robotics multimodal benchmarks

arXiv AI

Jun 18

LandslideAgent with Multimodal LandslideBench: A Domain-Rule-Augmented Agent for Autonomous Landslide Identification and Analysis

arXiv:2606. 18661v1 Announce Type: cross Abstract: Intelligent landslide hazard interpretation is critical for disaster prevention, yet current paradigms struggle to simultaneously extract visual features and high-level geoscientific semantics, while general-purpose vision-language models (VLMs) suffer from perceptual limitations and domain hallucinations in complex geological scenarios.

By Chengfu Liu, Dongyang Hou, Junwu Xiang, Cheng Yang, Xuezhi Cui, Zeyuan Wang, Liangtian Liu, Zelang Miao

llms agents computer-vision fine-tuning multimodal safety

arXiv Machine Learning

Jun 18

Risk Stratification for ICU Delirium using Pervasive Ambient Sensing Information

arXiv:2606. 19292v1 Announce Type: new Abstract: Delirium is a common and serious complication in the Intensive Care Unit (ICU), associated with increased morbidity, prolonged hospital stays, and higher healthcare costs.

By Jiaqing Zhang, Sabyasachi Bandyopadhyay, Miguel Contreras, Jessica Sena, Yuanfang Ren, Andrea Davidson, Ziyuan Guan, Tezcan Ozrazgat-Baslanti, Subhash Nerella, Azra Bihorac, Parisa Rashidi

multimodal

arXiv Machine Learning

Jun 18

A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)

arXiv:2606. 18451v1 Announce Type: new Abstract: Single-image-to-3D generators are improving quickly, but there is no agreed, human-free way to tell whether one generated mesh is better than another.

By Ali Asaria, Tony Salomone, Deep Gandhi

multimodal safety

arXiv Machine Learning

Jun 18

Artemis: Anatomy-Resolved inTervention for Eliminating Multimodal NeuroImage confounderS

arXiv:2606. 18287v1 Announce Type: new Abstract: Multimodal neuroimaging, integrating functional connectivity from fMRI and structural connectivity from DTI, enables non-invasive analysis of brain networks using graph neural networks.

By Siyuan Dai, Yang Du, Kun Zhao, Zhusuyi Chen, Heng Huang, Paul Thompson, Chao Shi, Haoteng Tang, Liang Zhan

multimodal benchmarks safety

arXiv AI

Jun 18

WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks

arXiv:2604. 06367v2 Announce Type: replace-cross Abstract: Web agents automate browser tasks, ranging from simple form completion to complex workflows like ordering groceries.

By Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique, Kassem Fawaz

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 18

Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation

arXiv:2606. 19120v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) trains a model on its own rollouts and uses a frozen copy to provide dense token-level targets conditioned on a reference target.

By Sihan Wang, Xiyao Liu, Lianqing Liu, Zhi Han

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jun 18

Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment

arXiv:2606. 18703v1 Announce Type: new Abstract: Pretrained biological language models expose per-token probability distributions through masked-token prediction, providing the likelihood interface central to sequence design, variant scoring, and mechanistic interpretation.

By Yanjun Shao, Yundi Chen, Yashvi Patel, Aurelien Pelissier, Mar\'ia Rodr\'iguez Mart\'inez

llms rag nlp multimodal benchmarks safety

arXiv AI

Jun 18

EMORSION: Examining the Impact of Audio Parameters on Emotional Responses and Immersion in Film

arXiv:2606. 18266v1 Announce Type: cross Abstract: EMORSION is an exploratory proof-of-concept study examining how film audio design shapes audience emotion and immersion in acinema setting.

By Nelly Garcia, Ruby Crocker, Bleiz M Del Sette, Fabrizio Smeraldi, Charalampos Saitis, George Fazekas, Joshua Reiss

multimodal

arXiv AI

Jun 18

ThinkDeception: A Progressive Reinforcement Learning Framework for Interpretable Multimodal Deception Detection

arXiv:2606. 18988v1 Announce Type: new Abstract: Multimodal deception detection is critical for identifying fraudulent intentions, yet existing approaches predominantly rely on end to end black--box paradigms.

By Jinhao Song, Shan Liang, Yiqun Yue, Zhuhuayang Zhang, Tianqi Gao

llms reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 18

Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs

arXiv:2606. 18323v1 Announce Type: cross Abstract: Open autoregressive neural-codec text-to-speech (TTS) models sound excellent on typical inputs yet suffer stochastic catastrophic failures: on a meaningful fraction of utterances they emit silence, terminate early, or collapse into repetitive or hallucinated content.

By Ali Asaria, Tony Salomone, Deep Gandhi

reinforcement-learning efficiency multimodal

arXiv AI

Jun 18

A Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images from GPT-Image-2

arXiv:2606. 19259v1 Announce Type: cross Abstract: Text-rich images often contain privacy-sensitive, transactional, or decision-relevant information.

By Yijin Wang, Shuyi Wang, Wenhan Zhang, Yuqi Ouyang

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 18

Generative-Model Predictive Planning for Navigation in Partially Observable Environments

arXiv:2606. 18888v1 Announce Type: new Abstract: Navigation in partially observable environments presents a significant challenge for autonomous agents, requiring effective decision-making with limited sensory information in unknown environments.

By Thomas Quilter, Yifan Zhu, Guorui Quan, Mingfei Sun, Samuel Kaski

agents diffusion reinforcement-learning multimodal

arXiv AI

Jun 18

OneCanvas: 3D Scene Understanding via Panoramic Reprojection

arXiv:2606. 19253v1 Announce Type: cross Abstract: Existing approaches to 3D scene understanding in Vision-Language Models (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning.

By Bart{\l}omiej Baranowski, Dave Zhenyu Chen, Matthias Nie{\ss}ner

llms rag robotics multimodal benchmarks

Get the daily briefing