Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

9,277 stories · RSS feed

arXiv Machine Learning

Jun 16

Contrastive Regularization for Accent-Robust ASR

arXiv:2605. 03297v2 Announce Type: replace-cross Abstract: ASR systems based on self-supervised acoustic pretraining and CTC fine-tuning achieve strong performance on native speech but remain sensitive to accent variability.

By Van-Phat Thai, Aradhya Dhruv, Duc-Thinh Pham, Sameer Alam

fine-tuning multimodal benchmarks

arXiv AI

Jun 16

Mitigating Visual Hallucinations in Multimodal Systems through Retrieval-Augmented Reliability-Aware Inference

arXiv:2606. 15782v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated strong capabilities in vision-language understanding and natural-language response generation.

By Pratheswaran Hariharan, Haiping Xu, Donghui Yan

llms rag multimodal safety

arXiv AI

Jun 16

Teacher-Student Structure for Domain Adaptation in Ensemble Audio-Visual Video Deepfake Detection

arXiv:2606. 15117v1 Announce Type: cross Abstract: The rapid advancement of generative AI models is leading to more realistic deepfake media, encompassing the manipulation of audio, video, or both.

By Elham Abolhasani, Maryam Ramezani, Hamid R. Rabiee

robotics fine-tuning multimodal safety

arXiv Machine Learning

Jun 16

From Tokens to Policy: Causal and Interpretable Heterogeneous Treatment Effects Identification

arXiv:2606. 17010v1 Announce Type: new Abstract: Heterogeneous Treatment Effect (HTE) identification is crucial to explain the impact of an intervention and optimize our policies accordingly.

By Riccardo Cadei, Frank Otchere, Nyasha Tirivayi, Gustavo Angeles Tagliaferro, Falco J. Bargagli-Stoffi, Francesco Locatello

multimodal safety

arXiv AI

Jun 16

Visual-Seeker: Towards Visual-Native Multimodal Agentic Search via Active Visual Reasoning

arXiv:2606. 15231v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated impressive capabilities in many visual tasks, but they often struggle with factual grounding when confronted with complex, open-world scenarios.

By Zhengbo Zhang, Changtao Miao, Jinbo Su, Zhaowen Zhou, Chunxia Zhang, Xukai Wang, Ruiqi Liu, Kaiyuan Zheng, Jiansheng Cai, Bo Zhang, Zhe Li, Shiming Xiang, Ying Yan

llms agents multimodal benchmarks data-engineering

arXiv AI

Jun 16

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

arXiv:2606. 14801v1 Announce Type: cross Abstract: Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult.

By Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alan Aspuru-Guzik, Igor Gilitschenski, Nicholas Rhinehart

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 16

LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

arXiv:2606. 16802v1 Announce Type: new Abstract: Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment.

By Anqi Zou, Han Deng, Chengyu Zhang, Junquan Hu, Yu Wang, Yuxiang Xing, Aokai Zhang, Hanling Zhang, Zhaoyang Liu, Ben Fei, Zhihui Wang, Wanli Ouyang

llms agents multimodal benchmarks safety

arXiv AI

Jun 16

Toward Vibe Medicine: A Self-Evolving Multi-Agent Framework for Clinical Decision Support

arXiv:2606. 15504v1 Announce Type: new Abstract: In recent years, the advances of large language models and autonomous agents have revolutionized the healthcare field, facilitating diagnosis and improving treatment results.

By Qianxue Zhang, Yiming Ren, Shihuan Qin, Xiao Zhang, Liao Zhang, Jinyang Huang, Zhengliang Liu, Chenbin Liu, Hongying Feng, Jingyuan Chen, Yuzhen Ding, Weihang You, Hanqi Jiang, Yi Pan, Yifan Zhou, Junhao Chen, Lifeng Chen, Wei Liu, Tianming Liu, Zengren Zhao, Lian Zhang

llms agents multimodal safety

arXiv AI

Jun 16

AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

arXiv:2606. 15057v1 Announce Type: cross Abstract: Indirect prompt injection (IPI) is a major security threat to LLM-powered agents.

By Xinhang Ma, Taoran Li, Chaowei Xiao, Zhiyuan Yu, Ning Zhang, Yevgeniy Vorobeychik

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 16

Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective

arXiv:2606. 14883v1 Announce Type: cross Abstract: Continual vision-language models are commonly addressed through sequential fine-tuning; however, although this paradigm enables adaptation to new environments (tasks), it inherently emphasizes the contribution of previously learned environments (tasks) at the expense of the stability required to preserve previously acquired knowledge.

By Salimeh Sekeh, Mary Wisell

llms fine-tuning multimodal

arXiv AI

Jun 16

OSGuard: A Benchmark for Safety in Computer-Use Agents

arXiv:2606. 15034v1 Announce Type: new Abstract: Computer-use agents are increasingly evaluated by whether they complete realistic desktop and web tasks.

By Mina Mohammadmirzaei, Jeffrey Flanigan

agents multimodal benchmarks safety

arXiv AI

Jun 16

Agentic Framework for Deep Learning workload migration via In-Context Learning

arXiv:2606. 15994v1 Announce Type: new Abstract: Translating deep learning models from PyTorch's flexible, object-oriented design to JAX's functional, stateless setup is usually a manual and error-prone task.

By Qiyue Liang, Steven Ingram, George Vanica, Andi Gavrilescu, Newfel Harrat, Hassan Sipra, Sethuraman Sankaran

llms agents multimodal benchmarks safety

arXiv AI

Jun 16

UrbanWell: Benchmarking Multimodal Large Language Models for Spatio-Temporal Urban Wellbeing Analytics

arXiv:2606. 15890v1 Announce Type: new Abstract: Understanding urban wellbeing from multimodal data requires integrating heterogeneous spatial and temporal signals, posing significant challenges for current multimodal large language models (MLLMs).

By Yanxin Xi, Xiang Su, Jie Feng, Yu Liu, Sasu Tarkoma, Pan Hui

llms multimodal benchmarks safety

arXiv AI

Jun 16

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

arXiv:2606. 14777v1 Announce Type: cross Abstract: Many moments in the real world do not wait for a user to ask.

By Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin, Haowen Hou, Zheming Liang, Congcong Wang, Yuhang Cao, Shenglong Ye, Shuai Xie, Shuhuan Gu, Haoyang Huang, Qingyi Si, Nan Duan, Jiaqi Wang

llms agents multimodal

arXiv AI

Jun 16

MatchLM2Lite: A Scalable MLLM-to-Lite Framework for Reproduced Content Identification

arXiv:2606. 14786v1 Announce Type: cross Abstract: Content moderation is critical for online video platforms to ensure content safety, protect creators, and sustain positive user experiences.

By Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Zirui Zhu, Kanchan Sarkar, Kun Xu

llms efficiency multimodal safety

arXiv Machine Learning

Jun 16

SPICE: Synergy and Partial Information Based Curriculum Evolution

arXiv:2606. 16639v1 Announce Type: new Abstract: Multimodal learning exploits complementary information across heterogeneous modalities.

By Ankush Pratap Singh, Houwei Cao, Yong Liu

multimodal benchmarks

arXiv AI

Jun 16

Action with Visual Primitives

arXiv:2605. 22183v3 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models have emerged as a promising paradigm for generalist robotic manipulation.

By Weilong Guo, Yuchen Wang, Renping Zhou, Yunfeng Zhang, Rui Fang, Yuyang Pang, Wenda Xu, Gao Huang

robotics multimodal

arXiv AI

Jun 16

Post-Launch Capability Expansion of Vision-Language Models via Prompting for On-Orbit Spacecraft Inspection

arXiv:2606. 15427v1 Announce Type: cross Abstract: Spaceborne inspection systems often deploy perception models prior to launch, after which updating model weights or expanding fixed label sets becomes operationally impractical.

By Nicholas A. Welsh, Lennon J. Shikhman, Monty Nehru Attazs, Seemanthini K. Putane, Van Minh Nguyen, Ryan T. White

llms computer-vision multimodal

arXiv AI

Jun 16

Binary Tracking for Spatial QA and Navigation with Open Vision-Language Models

arXiv:2606. 16902v1 Announce Type: cross Abstract: This work addresses spatial question answering for service robots traversing long egocentric routes.

By Dongbin Na, Chanwoo Kim, Soonbin Rho, Giyun Choi, Gangbok Lee, Dooyoung Hong

llms rag agents nlp robotics multimodal benchmarks

arXiv AI

Jun 16

Dual-Uncertainty Guided Policy Learning for Multimodal Reasoning

arXiv:2510. 01444v3 Announce Type: replace Abstract: Reinforcement learning with verifiable rewards (RLVR) has advanced reasoning capabilities in multimodal large language models.

By Rui Liu, Dian Yu, Tong Zheng, Runpeng Dai, Zongxia Li, Wenhao Yu, Zhenwen Liang, Linfeng Song, Haitao Mi, Pratap Tokekar, Dong Yu

llms reinforcement-learning multimodal benchmarks

Get the daily briefing