Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv Machine Learning

Jun 25

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

arXiv:2606. 25760v1 Announce Type: new Abstract: Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions.

By Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 25

Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curation

arXiv:2606. 25432v1 Announce Type: new Abstract: Inference efficiency is typically pursued by shrinking the model: distillation, pruning, quantization, and sparse routing each lower per-token cost while treating token count as fixed.

By DatologyAI, :, Matthew L. Leavitt, Siddharth Joshi, Haoli Yin, Rishabh Adiga, Haakon Mongstad, Alvin Deng, David Schwab, Bogdan Gaza, Ari Morcos

efficiency multimodal

arXiv Machine Learning

Jun 25

State Space Models Meet Remote Sensing: A Survey

arXiv:2606. 25329v1 Announce Type: cross Abstract: State Space Models (SSMs), designed for long-range modeling, offer linear computational complexity and strong capabilities in capturing long-range dependencies.

By Qinzhe Yang, Chenyang Liu, Jia Xu, Zhenwei Shi, Zhengxia Zou

multimodal

arXiv Machine Learning

Jun 25

Curvature-Guided Mixing for MLLM Adaptation

arXiv:2606. 24963v1 Announce Type: cross Abstract: Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities.

By Jinglong Yang, Jiaxuan He, Wenjian Huang, Zhan Zhuang, Jianguo Zhang

llms fine-tuning multimodal

arXiv Machine Learning

Jun 25

PERTINENCE: Input-based Opportunistic Neural Network Dynamic Execution

arXiv:2507. 01695v3 Announce Type: replace Abstract: Deep neural networks (DNNs) are widely used for their ability to model complex patterns across domains such as computer vision, speech recognition, and robotics.

By Omkar Shende, Gayathri Ananthanarayanan, Marcello Traiola

computer-vision robotics multimodal benchmarks

arXiv Machine Learning

Jun 25

Bridging Spherical Black-Box Optimizers

arXiv:2606. 25761v1 Announce Type: new Abstract: When gradient information is unavailable, black-box optimization (BBO) methods provide a practical alternative.

By Johannes Ackermann, Stefano Peluchetti

llms robotics multimodal benchmarks

arXiv Machine Learning

Jun 25

Variational Inference via Entropic Transport Descent

arXiv:2606. 25265v1 Announce Type: new Abstract: Particle-based variational inference (ParVI) methods approximate an intractable target distribution by evolving an ensemble of interacting samples.

By Vincent Pacelli, Akash Ratheesh, Evangelos Theodorou

multimodal

arXiv Machine Learning

Jun 25

Defense effectiveness across architectural layers: a mechanistic evaluation of persistent memory attacks on stateful LLM agents

arXiv:2605. 08442v3 Announce Type: replace-cross Abstract: Persistent memory attacks against LLM agents achieve high attack success rates against open-source models.

By Jun Wen Leong

llms rag agents multimodal

arXiv Machine Learning

Jun 25

MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning

arXiv:2606. 25225v1 Announce Type: cross Abstract: Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning.

By Revant Teotia, Adrien Bardes, Michael Rabbat, Sumit Chopra, Matthew J. Muckley, Nicolas Ballas

rag multimodal benchmarks

arXiv Machine Learning

Jun 25

Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models

arXiv:2606. 26079v1 Announce Type: cross Abstract: Standard benchmarks for multimodal large language models (MLLMs) score each item on one canonical ordering and miss whether order-irrelevant shuffling changes the answer, a baseline reliability property called for by emerging AI evaluation guidelines.

By Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 25

SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning

arXiv:2507. 16518v3 Announce Type: replace-cross Abstract: Recent advances in multimodal large language models (MLLMs) have shown impressive reasoning capabilities.

By Xiuwei Chen, Wentao Hu, Hanhui Li, Yongxin Wang Jun Zhou, Zisheng Chen, Meng Cao, Yihan Zeng, Kui Zhang, Yu-Jie Yuan, Jianhua Han, Hang Xu, Xiaodan Liang

llms reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 25

Enhancing Clinician Decision-Making via Uncertainty-Aware Multi-Expert Fusion for Stroke Rehabilitation

arXiv:2606. 24960v1 Announce Type: new Abstract: Tailoring stroke rehabilitation requires assessing how movements are organized, not merely if they succeed.

By Tamim Ahmed, Thanassis Rikakis

multimodal

arXiv Machine Learning

Jun 25

An iterative energy-based multimodal transformer for joint retrieval of wheat soil moisture, leaf area index, and plant height from Sentinel-1 and Sentinel-2 time series

arXiv:2606. 25174v1 Announce Type: new Abstract: Field-scale retrieval of surface soil moisture (SM), leaf area index (LAI), and plant height (PH) is essential for precision agriculture, yet it remains an ill-posed inverse problem.

By Shubham Kumar Singh, Peilei Fan, Suraj A. Yadav, Rajendra Prasad, Prashant K Srivastava

llms multimodal

arXiv Machine Learning

Jun 25

Geo-Strat-RL: Learning Geological Event Reasoning from Verifiable Tasks

arXiv:2606. 25000v1 Announce Type: new Abstract: To evaluate whether vision-language models can reason about geological histories, it is necessary to construct observations for which the underlying process history is known.

By Lukas Mosser

llms reinforcement-learning multimodal

arXiv Machine Learning

Jun 25

Efficient Analytic Uncertainty Quantification for Multi-Modal Regression

arXiv:2606. 25188v1 Announce Type: new Abstract: Efficient uncertainty quantification (UQ) is essential for trustworthy large-scale learning.

By Kun Jin, James Harrison, Jiawei Li, Sihan Liu, Jiayi Liu, Randolph Linderman, Yuening Li, Arnab Bhadury, Sourabh Prakash Bansod, Liang Liu, Jasper Snoek

multimodal benchmarks

arXiv Machine Learning

Jun 25

Improving Factuality of 3D Brain MRI Report Generation with Paired Image-domain Retrieval and Text-domain Augmentation

arXiv:2411. 15490v2 Announce Type: replace-cross Abstract: Acute ischemic stroke (AIS) requires time-critical decision-making, where inaccurate interpretation of neuroimaging findings can lead to irreversible disability.

By Junhyeok Lee, Yujin Oh, Dahyoun Lee, Hyon Keun Joh, Chul-Ho Sohn, Sung Hyun Baik, Cheol Kyu Jung, Jung Hyun Park, Kyu Sung Choi, Byung-Hoon Kim, Jong Chul Ye

llms rag diffusion multimodal benchmarks safety

arXiv Machine Learning

Jun 25

Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

arXiv:2606. 25527v1 Announce Type: new Abstract: Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency.

By Guozheng Ma, Lu Li, Zilin Wang, Pierre-Luc Bacon, Dacheng Tao

agents reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 25

ROAD-VLA: Robust Online Adaptation via Self-Distillation for Vision-Language-Action Models

arXiv:2606. 25800v1 Announce Type: new Abstract: Effective online adaptation of vision-language-action (VLA) models remains challenging, as sparse rewards provide weak supervision for high-dimensional autoregressive action policies.

By Kejing Wang, Toan Nguyen, Minh Hoang Nguyen, Simon Khan, Flora D. Salim

reinforcement-learning robotics efficiency multimodal

arXiv Machine Learning

Jun 25

How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

arXiv:2606. 25487v1 Announce Type: cross Abstract: Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model prompted to grade.

By Yang Gao (Veyon Solutions)

llms multimodal safety

arXiv Machine Learning

Jun 25

Towards Scalable Multi-Task Reinforcement Learning with Large Decision Models

arXiv:2606. 24962v1 Announce Type: new Abstract: Recent progress in large-scale sequence modeling has shown that a single model can learn useful representations across highly diverse data distributions.

By Thibaut Kulak

llms agents reinforcement-learning robotics multimodal

Get the daily briefing