Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,631 stories · RSS feed

arXiv Machine Learning

Jun 2

Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

arXiv:2603. 11653v2 Announce Type: replace Abstract: Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments.

By Jiaheng Hu, Jay Shim, Chen Tang, Yoonchang Sung, Bo Liu, Peter Stone, Roberto Martin-Martin

agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

Planktonzilla: Multimodal dataset and models for understanding plankton ecosystems

arXiv:2606. 00080v1 Announce Type: cross Abstract: Marine plankton underpin aquatic food webs and play a key role in global CO2 sequestration, making reliable species identification critical for understanding ocean health and climate feedbacks.

By Alan Gerson Contreras Montanares, Luis Valenzuela, Luis Mart\'i, Nayat Sanchez-Pi

multimodal

arXiv AI

Jun 2

Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy

arXiv:2606. 00065v1 Announce Type: cross Abstract: Automated extraction of materials composition-property data from scientific literature has advanced considerably with the development of large language model-based pipelines; however, existing frameworks remain limited to textual and tabular content, overlooking the substantial proportion of quantitative property data reported exclusively in scientific figures.

By Aritra Roy, Enrico Grisan, Chiara Gattinoni, John Buckeridge

llms agents multimodal benchmarks

arXiv AI

Jun 2

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.

By Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee, Jaeyo Shin, Hyunjung Shim

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

When Jokes Cross the Line: Analyzing Regular Humor and Dark Humor in YouTube Shorts

arXiv:2606. 00046v1 Announce Type: cross Abstract: Video platforms such as YouTube have reshaped how users engage with entertainment and information, emphasizing brief, highly engaging content such as Shorts.

By Sydney Johns, Sanjeev Parthasarathy, Shantnu Bhalla, Vaibhav Garg

llms multimodal benchmarks

arXiv AI

Jun 2

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605. 09883v2 Announce Type: replace-cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding?

By Xia Hu, Zhenrui Yue, Brian Potetz, Howard Zhou, Leonidas Guibas, Chun-Ta Lu, Zhicheng Wang

llms multimodal benchmarks

arXiv AI

Jun 2

HLL: Can Agents Cross Humanity's Last Line of Verification?

arXiv:2606. 02449v1 Announce Type: new Abstract: Multimodal agents are increasingly expected to operate interfaces on behalf of users, raising a central deployment question: can they truly substitute for humans in workflows that services deliberately protect against automation?

By Xinhao Song, Su Su, Sirui Song, Hongliang Wu, Wen Shen, Zhihua Wei, Gongshen Liu, Linfeng Zhang, Dongrui Liu

agents multimodal benchmarks

arXiv AI

Jun 2

Spatial Representation Learning Beyond Pixels: Unifying Raster Data and Vector Semantics for Human-Centric Geospatial Foundation Models

arXiv:2606. 02374v1 Announce Type: new Abstract: Earth Observation (EO) has fundamentally transformed the monitoring of environmental processes and human activities up to planetary scale.

By Steffen Knoblauch, Hao Li, Gengchen Mai, Konstantin Klemmer, Song Gao, WenWen Li

rag multimodal

arXiv AI

Jun 2

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

arXiv:2606. 01961v1 Announce Type: new Abstract: Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering.

By Junqi Liu, Salena Song, Yuhan Wang, Jiawei Mao, Hardy Chen, Xiaoke Huang, Tianhao Qi, Pengfei Guo, Yucheng Tang, Yufan He, Can Zhao, Andriy Myronenko, Dong Yang, Daguang Xu, Yuyin Zhou

agents computer-vision nlp multimodal benchmarks

arXiv AI

Jun 2

Token Predictors Are Not Planners: Building Physically Grounded Causal Reasoners

arXiv:2606. 01810v1 Announce Type: new Abstract: Current benchmarks for embodied vision-language planning often favor linguistic next-token prediction over physically grounded next-state reasoning.

By Zheng Lu, Mingqi Gao, Qinlei Xie, Wanqi Zhong, Hanwen Cui, Heng Cao, Zirui Song, Yifan Yang, Chong Luo, Bei Liu, Yiming Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 2

FlowTime: Towards Continuous Generative Watch Time Prediction via Flow-based Personalized Priors

arXiv:2606. 01352v1 Announce Type: new Abstract: Watch time has emerged as a pivotal metric for optimizing deep user engagement in short-video recommender systems.

By Hongxu Ma, Han Zhou, Chenghou Jin, Jie Zhang, Xiaoyu Yang, Chunjie Chen, Jihong Guan, Shuigeng Zhou

efficiency multimodal benchmarks

arXiv AI

Jun 2

Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs

arXiv:2606. 01215v1 Announce Type: cross Abstract: Current 3D spatial reasoning methods face a fundamental trade-off: neuro-symbolic 3D (NS3D) concept learners achieve interpretable reasoning through compositional programs but are constrained to closed-set concept vocabularies and simple programs; end-to-end 3D multi-modal LLMs (3D MLLMs) could handle complex natural language and open-vocabulary concepts but suffer from black-box reasoning without explicit spatial verification.

By Wentao Mo, Yang Liu

llms nlp reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

PropLLM: Propagation-Aware Scene Reconstruction for Network Fault Diagnosis

arXiv:2606. 00582v1 Announce Type: new Abstract: Network faults propagate layer by layer along topology and protocol dependencies, yet operations systems typically observe only symptomatic alerts at the tail end of propagation chains, where distinct root-cause faults may produce highly similar end-point symptoms.

By Zongzong Wu, Ming Zhao, Fengxiao Tang, Nei Kato

llms multimodal safety

arXiv AI

Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney

llms agents multimodal benchmarks safety

arXiv AI

Jun 2

From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model

arXiv:2512. 05277v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari

llms agents robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 2

The Lie We Tell: Correcting the Euclidean Fallacy in Vision Language Action Policies via Score Matching on Tangent Space

arXiv:2606. 01847v1 Announce Type: cross Abstract: Diffusion-based Vision-Language-Action policies achieve remarkable success in robotic manipulation, yet commit a fundamental geometric error we term the $\textbf{Euclidean Fallacy}$: representing SE(3) poses as flat $\mathbb{R}^{12}$ vectors.

By Bing-Cheng Chuang, I-Hsuan Chu, Bor-Jiun Lin, YuanFu Yang, Min Sun, Chun-Yi Lee

diffusion robotics multimodal

arXiv Machine Learning

Jun 2

GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning

arXiv:2605. 03403v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has recently shown strong performance in post-training large language models and vision-language models.

By Yujun Li, Hongyuan Zhang, Yuan Yuan

llms reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 2

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

arXiv:2606. 02576v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential.

By Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang, Da-Wei Zhou

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

PaCX-MAE: Physiology-Augmented Chest X-Ray Masked Autoencoder

arXiv:2606. 01537v1 Announce Type: cross Abstract: Clinical diagnosis often requires combining imaging with physiological measurements, yet deployed models typically operate on unimodal data.

By Yancheng Liu, Kenichi Maeda, Manan Pancholy

rag computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Jun 2

Reason, Retrieve, Re-rank: A Zero-Shot Reasoning-Aware Framework for Composed Video Retrieval

arXiv:2606. 00910v1 Announce Type: cross Abstract: Composed Video Retrieval (CoVR) seeks the target video that results from applying a free-form textual modification to a reference video.

By Ali Alavi

llms multimodal

Get the daily briefing