Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

4,498 stories · RSS feed

arXiv AI

Jun 19

FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS

arXiv:2606. 20518v1 Announce Type: new Abstract: Flow-matching text-to-speech systems achieve remarkable zero-shot quality but remain static after deployment: pronunciation errors on out-of-vocabulary proper nouns persist unless the model is retrained.

By Harshit Singh, Ayush Pratap Singh, Nityanand Mathur

rag multimodal benchmarks

arXiv AI

Jun 19

VCG: A Multimodal Retrieval Framework for E-Commerce Video Feeds under Extreme Cold-Start Conditions

arXiv:2606. 19627v1 Announce Type: cross Abstract: The digital commerce landscape is shifting from static, search-driven catalogs to dynamic, immersive video feeds.

By Katya Mirylenka, Egor Malykh, Mahdyar Ravanbakhsh, Michael Gygli, Marco-Andrea Buchmann, Andrew Dzhoha, Svitlana Borzenko, Francesca Catino, Mohamed Gaafar, Maarten Versteegh, Thomas Kober, Dario d'Andrea, Ellie Langhans

llms rag multimodal

arXiv Machine Learning

Jun 19

Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

arXiv:2606. 20477v1 Announce Type: cross Abstract: We study how to train visually grounded vision-language models (VLMs) for radiology without manual spatial annotations.

By Yusuf Salcan (Computer Vision Group, University of Freiburg, Germany, CRIION-AI Lab, Freiburg, Germany), Simon Ging (Computer Vision Group, University of Freiburg, Germany, Adaptive & Agentic AI), Robin Schirrmeister (Department of Radiology, Medical Center -- University of Freiburg, Germany), Philipp Arnold (Department of Radiology, Medical Center -- University of Freiburg, Germany), Elmar Kotter (Department of Radiology, Medical Center -- University of Freiburg, Germany), Behzad Bozorgtabar (Adaptive & Agentic AI), Thomas Brox (Computer Vision Group, University of Freiburg, Germany)

llms computer-vision nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

Multimodal Concept Bottleneck Models

arXiv:2606. 19882v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.

By Tongqing Shi, Ge Yan, Tuomas Oikarinen, Tsui-Wei Weng

rag multimodal benchmarks safety

arXiv AI

Jun 19

Vero: An Open RL Recipe for General Visual Reasoning

arXiv:2604. 04917v3 Announce Type: replace-cross Abstract: What does it take to build a visual reasoner that works across charts, science, spatial understanding, and open-ended tasks?

By Gabriel Sarch, Linrong Cai, Qunzhong Wang, Haoyang Wu, Danqi Chen, Zhuang Liu

llms reinforcement-learning efficiency multimodal benchmarks

arXiv AI

Jun 19

ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis

arXiv:2603. 04219v2 Announce Type: replace-cross Abstract: We investigate the use of zero-shot text-to-speech (ZS-TTS) as a data augmentation source for low-resource personalized speech synthesis.

By Youngwon Choi, Jinwoo Oh, Hwayeon Kim, Hyeonyu Kim

rag fine-tuning multimodal

arXiv Machine Learning

Jun 19

PASQA: Pitch-Accent-Focused Speech Quality Assessment Model Trained on Synthetic Speech with Accent Errors

arXiv:2606. 20137v1 Announce Type: cross Abstract: Existing mean opinion score (MOS) prediction models typically predict utterance-level naturalness MOS and can be insensitive to localized pitch-accent errors.

By Masaya Kawamura, Yuma Shirahata, Kentaro Mitsui, Reo Shimizu

multimodal

arXiv AI

Jun 19

UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation

arXiv:2501. 17015v2 Announce Type: replace Abstract: Simulation plays a crucial role in assessing autonomous driving systems, where the generation of realistic multi-agent behaviors is a key aspect.

By Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, Yue Wang

agents robotics multimodal benchmarks safety

arXiv AI

Jun 19

ELVA: Exploring Ranking-Driven Universal Multimodal Retrieval

arXiv:2606. 20280v1 Announce Type: cross Abstract: Leveraging Multimodal Large Language Models (MLLMs) via contrastive learning has become a mainstream paradigm for improving the performance of Universal Multimodal Retrieval (UMR).

By Yuhan Liu, Pei Fu, Hang Li, Yukun Qi, Chao Jiang, Jingwen Fu, Zhen Liu, Bin Qin, Zhenbo Luo, Jian Luan, Jingmin Xin

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jun 19

Systematic Study of Dysarthric Speech Recognition: Spectral Features and Acoustic Models

arXiv:2606. 19793v1 Announce Type: cross Abstract: The challenge associated with recognizing dysarthric speech primarily arises from pronounced acoustic variability attributed to impaired articulatory precision.

By Paban Sapkota, Hemant Kumar Kathania, Mikko Kurimo, Sudarsana Reddy Kadiri, Shrikanth Narayanan

multimodal benchmarks

arXiv AI

Jun 19

NRITYAM: Language Models Meet Art and Heritage of Dance

arXiv:2606. 19727v1 Announce Type: cross Abstract: Language models have become essential tools in shaping modern workflows.

By Punit Kumar Singh, Niladri Ghosh, Advait Joshi{\i}nst, Shailee Choudhary, Michael F\"arber, Haiqin Yang

llms multimodal benchmarks

arXiv AI

Jun 19

Frequency-Aware Flow Matching for Continuous and Consistent Robotic Action Generation

arXiv:2606. 20135v1 Announce Type: cross Abstract: Flow matching has emerged as a standard paradigm for robotic manipulation owing to its strong expressive power for modelling complex, multimodal action distributions, alongside similar approaches like diffusion policy.

By Jianing Guo, Fangzheng Chen, Zihao Mao, Wong Lik Hang Kenny, Zhenhong Wu, Yu Li, Yishuai Cai, Yuanpei Chen, Yikun Ban, Kai Chen, Qi Dou, Yaodong Yang, Xianglong Liu, Huijie Zhao, Simin Li

diffusion robotics multimodal benchmarks safety

arXiv AI

Jun 19

Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs

arXiv:2606. 20177v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in various Remote Sensing (RS) tasks.

By Haochen Han, Jue Wang, Alex Jinpeng Wang, Fangming Liu

llms multimodal benchmarks safety

arXiv AI

Jun 19

Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

arXiv:2606. 19914v1 Announce Type: cross Abstract: Art has long stood as a pivotal expression of human creativity.

By Xuetao Li, Wenke Huang, Mang Ye, Zijian Liu, Jinhua Xie, Jifeng Xuan, Miao Li

diffusion robotics fine-tuning multimodal safety

arXiv Machine Learning

Jun 19

Do Vision-Language Models Understand 3D Scenes or Just Catalogue Objects?

arXiv:2605. 20448v2 Announce Type: replace-cross Abstract: Vision-language models reliably name objects in a scene, but do they represent the 3D layout those objects inhabit?

By Animesh Maheshwari, Divyansh Sahu, Nishit Verma

llms robotics multimodal benchmarks

arXiv Machine Learning

Jun 19

Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning

arXiv:2606. 19823v1 Announce Type: cross Abstract: Automatic speech recognition remains unreliable for dysarthric speech due to data scarcity and high inter-speaker variability.

By Satwinder Singh, Qianli Wang, Zihan Zhong, Clarion Mendes, Hasegawa-Johnson, Waleed Abdulla, Seyed Reza Shahamiri

fine-tuning multimodal

arXiv AI

Jun 19

The Hidden Evolution of Disguised Visual Context inside the VLM

arXiv:2606. 20077v1 Announce Type: cross Abstract: Visual tokens enter Large Language Models (LLMs) as raw, foreign signals.

By Wish Suharitdamrong, Tony Alex, Muhammad Awais, Sara Atito

llms multimodal benchmarks

arXiv AI

Jun 19

FreeStyle: Free Control of Style-Content Dual-Reference Generation from Community LoRA Mining

arXiv:2606. 20506v1 Announce Type: cross Abstract: Style-content dual-reference generation aims to synthesize an image that preserves the structure and semantics of a content reference while adopting the style of a separate style reference.

By Jinghong Lan, Wei Cheng, Yunuo Chen, Ziqi Ye, Peng Xing, Yixiao Fang, Rui Wang, Yufeng Yang, Xuanyang Zhang, Xianfang Zeng, Difan Zou, Gang Yu, Chi Zhang

fine-tuning multimodal benchmarks safety

arXiv AI

Jun 19

SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

arXiv:2606. 20244v1 Announce Type: cross Abstract: Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact.

By Bo Yin, Xiaobin Hu, Chengming Xu, Ruolin Shen, Mo Yang, Jiangning Zhang, Peng-Tao Jiang, Cheng Tan, Shuicheng YAN

llms multimodal benchmarks

arXiv Machine Learning

Jun 19

Bioacoustic Geolocation: Species Sounds as Geographic Signals

arXiv:2505. 18726v3 Announce Type: replace-cross Abstract: Can we determine someone's geographic location solely from the sounds they hear?

By Mustafa Chasmai, Wuao Liu, Subhransu Maji, Grant Van Horn

multimodal benchmarks

Get the daily briefing