Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,626 stories · RSS feed

arXiv AI

Jun 11

Physics-informed generative AI for semiconductor manufacturing: Enforcing hard physical constraints in generative models by construction

arXiv:2606. 11247v1 Announce Type: cross Abstract: Generative models are increasingly used to propose designs, data, and control actions for physical systems, yet many such systems are governed by hard physical constraints rather than by perceptual plausibility.

By Yaser Mike Banad, Sarah Sharif

agents diffusion multimodal benchmarks

arXiv AI

Jun 11

MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models

arXiv:2606. 11792v1 Announce Type: cross Abstract: Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video.

By Yuansheng Gao, Wenbin Xing, Jiahao Yuan, Kaiwen Zhou, Han Bao, Zonghui Wang, Wenzhi Chen

efficiency multimodal safety

arXiv AI

Jun 11

ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Models

arXiv:2605. 15687v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) may memorize sensitive cross-modal information during pretraining, making machine unlearning (MU) crucial.

By Jiahui Guang, Haiyan Wang, Yingjie Zhu, Cuiyun Gao, Jing Li, Di Shao, Zhaoquan Gu

llms multimodal safety

arXiv AI

Jun 11

SVoT: State-aware Visualization-of-Thought for Spatial Reasoning via Reinforcement Learning

arXiv:2606. 11770v1 Announce Type: new Abstract: Spatial reasoning remains a challenge for Multimodal Large Language Models (MLLMs), as it requires reliable multi-hop inference over both intermediate states and state transitions.

By Chao Lei, Yanbei Jiang, Markus Hiller, Zhijian Zhou, Xunye Tian, Krista A. Ehinger, Nir Lipovetzky

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jun 11

Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning

arXiv:2606. 11853v1 Announce Type: cross Abstract: Multi-modal large language models (MLLMs) depend on in-context learning (ICL) for rapid task adaptation, but their scalability is severely limited by finite context windows and the growing cost of key-value (KV) caches in long multi-modal sequences.

By Zhirui Chen, Ziwei Chen, Ling Shao

llms efficiency multimodal safety

arXiv AI

Jun 11

Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

arXiv:2606. 11324v1 Announce Type: cross Abstract: We introduce Embodied-R1.

By Yifu Yuan, Yaoting Huang, Xianze Yao, Yutong Li, Shuoheng Zhang, Linqi Han, Pengyi Li, Jiangeng Sun, Wenting Jia, Zhao Zhang, Yuhao Liu, Ruihao Liao, Yucheng Hu, Qiyu Wu, Yuxiao Li, Zibin Dong, Fei Ni, Yan Zheng, Shuyang Gu, Yi Ma, Hongyao Tang, Han Hu, Jianye Hao

llms reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 11

ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward

arXiv:2606. 11209v1 Announce Type: cross Abstract: Visual question answering increasingly requires multi-step reasoning.

By Jingpei Wu, Xiao Han, Weixiang Shen, Boer Zhang, Zifeng Ding, Volker Tresp

llms nlp reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 11

Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

arXiv:2606. 11266v1 Announce Type: new Abstract: The cost signal that constrained-RL algorithms optimize against is almost always reactive: the simulator emits a non-zero cost only after a collision has begun, and the Lagrange multiplier of PPO-Lagrangian grows only after the episode budget has been exceeded.

By Samuel Tetteh, Cody Fleming

llms reinforcement-learning multimodal safety

arXiv AI

Jun 11

Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark

arXiv:2602. 19502v2 Announce Type: replace Abstract: Agentic AI systems are increasingly capable of autonomous data science workflows, yet clinical prediction tasks demand domain expertise that purely automated approaches struggle to provide.

By Lalitha Pranathi Pulavarthy, Raajitha Muthyala, Aravind V Kuruvikkattil, Zhenan Yin, Rashmita Kudamala, Saptarshi Purkayastha

agents multimodal benchmarks safety

arXiv Machine Learning

Jun 11

Learning Instance-Adaptive Low-Rank Orthogonal Subspaces for Clothes-Changing Person Re-Identification

arXiv:2606. 11661v1 Announce Type: cross Abstract: Clothes-changing person re-identification (CC-ReID) aims to recognize individuals despite drastic appearance changes caused by clothing variation.

By Dong-Woo Kim, Tae-Kyun Kim

llms rag multimodal benchmarks safety

arXiv AI

Jun 11

Task-Aligned Stability Analysis of Vision-Language Models for Autonomous Driving Hazard Detection

arXiv:2606. 11889v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used for scene understanding in autonomous driving, but robustness analysis often relies on task-agnostic embedding stability alone.

By Everett Richards

llms rag agents robotics multimodal benchmarks

arXiv AI

Jun 11

The Unreasonable Effectiveness of Discrete-Time Gaussian Process Mixtures for Robot Policy Learning

arXiv:2505. 03296v2 Announce Type: replace-cross Abstract: We present Mixture of Discrete-time Gaussian Processes (MiDiGap), a novel approach for flexible policy representation and imitation learning in robot manipulation.

By Jan Ole von Hartz, Adrian R\"ofer, Joschka Boedecker, Abhinav Valada

robotics multimodal benchmarks

arXiv AI

Jun 11

Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training

arXiv:2606. 11854v1 Announce Type: cross Abstract: There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs).

By Michal Chudoba, Sergey Alyaev, Petra Galuscakova, Tomasz Wiktorski

llms fine-tuning efficiency multimodal benchmarks

arXiv AI

Jun 11

Robust Privacy: Inference-Stage Privacy through Certified Robustness

arXiv:2601. 17360v2 Announce Type: replace-cross Abstract: An adversary observing a model's released prediction can infer sensitive attributes of the queried input, or even reconstruct representatives of the model's training data.

By Jiankai Jin, Xiangzheng Zhang, Zhao Liu, Wenzhuo Xu, Dongdong Yang, Deyue Zhang, Quanchen Zou

efficiency multimodal safety

arXiv AI

Jun 11

RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark

arXiv:2606. 11260v1 Announce Type: cross Abstract: Humans process rich auditory environments through tightly integrated cognitive capabilities such as audio perception, audio reasoning, and memory.

By Hongyu Jin, Siyi Wang, Yang Xiao, Jiaheng Dong, Shihong Tan, Kaiyuan peng, Georgiana Juravle, Shanquan Chen, Gongping Huang, Hong Jia, Eun-Jung Holden, James Bailey, Ting Dang

llms multimodal benchmarks

arXiv Machine Learning

Jun 11

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

arXiv:2606. 11743v1 Announce Type: cross Abstract: Vision-language-action (VLA) models provide strong visual, language, and action priors for robot manipulation, but visual observations alone often miss the local contact state required for contact-rich tasks.

By Siyu Ma, Yuqi Liang, Chang Yu, Yunuo Chen, Hao Su, Yixin Zhu, Yin Yang, Chenfanfu Jiang

reinforcement-learning robotics multimodal

arXiv AI

Jun 11

Semantic search for 100M+ galaxy images using AI-generated captions

arXiv:2512. 11982v2 Announce Type: replace-cross Abstract: Finding scientifically interesting phenomena through slow manual labeling campaigns severely limits our ability to explore the billions of galaxy images produced by telescopes.

By Nolan Koblischke, Liam Parker, Francois Lanusse, Jo Bovy, Irina Espejo, Shirley Ho

llms rag multimodal benchmarks

arXiv AI

Jun 11

OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

arXiv:2601. 22725v4 Announce Type: replace-cross Abstract: Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck.

By Jin Li, Tao Chen, Kai Wen, Siqi Yin, Shuai Jiang, Weijie Wang, Jingwen Luo, Chenhui Wu

llms diffusion computer-vision multimodal benchmarks safety

arXiv AI

Jun 11

MARIC: Multi-Agent Reasoning for Image Classification

arXiv:2509. 14860v2 Announce Type: replace-cross Abstract: Image classification has traditionally relied on parameter-intensive model training, requiring large-scale annotated datasets and extensive fine tuning to achieve competitive performance.

By Wonduk Seo, Minhyeong Yu, Hyunjin An, Seunghyun Lee

llms agents computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jun 11

Latent World Recovery for Multimodal Learning with Missing Modalities

arXiv:2606. 12362v1 Announce Type: cross Abstract: We study multimodal learning under missing modalities, with particular motivation from bioscience applications in which heterogeneous modalities are often only partially available when decisions need to be made.

By Hui Wang, Tianyu Ren, Joseph Butler, Christopher Baker, Karen Rafferty, Simon McDade

rag diffusion multimodal benchmarks safety

Get the daily briefing