arXiv Machine Learning June 5, 2026 By Erfan Loweimi, Mengjie Qian, Kate Knill, Guanfeng Wu, Chi-Ho Chan, Abbas Haider, Muhammad Awan, Josef Kittler, Hui Wang, Mark Gales

To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection

Read the original on arXiv Machine Learning →

arXiv:2606. 05931v1 Announce Type: cross Abstract: When retrieving a person from a video archive by voice and face, should the system be multimodal or not?

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computer Vision

4d ago

Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

arXiv:2501.18157v2 Announce Type: replace-cross Abstract: Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions fr...

By Joanna Hong, Sanjeel Parekh, Honglie Chen, Jacob Donley, Ke Tan, Buye Xu, Anurag Kumar

multimodal

arXiv Computer Vision

Sep 23

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

arXiv:2609.26648v1 Announce Type: cross Abstract: Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under chal...

By Pu Wang, Yujun Wang, Hugo Van hamme

multimodal benchmarks

arXiv AI

Jun 30

AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

arXiv:2606. 29335v1 Announce Type: cross Abstract: Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions.

By Chuxiao Zuo, Yao Zhu, Minqiang Xu, Manhong Wang, Yunke Zhang, Fei Huang

rag multimodal

arXiv Computer Vision

Sep 15

Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval

The paper introduces SCALAR, a query‑conditioned spherical centroid aggregator that assigns relevance‑based weights to each available modality before computing a spherical centroid. SCALAR supports arbitrary modality subsets, is trained with rank‑8 LoRA adapters on masked views, and achieves positive aggregation gains on four of five benchmarks, outperforming prior symmetric aggregators. With only 4.8 million trainable parameters, SCALAR attains the highest text‑to‑video R@1 on three benchmarks and surpasses the released GRAM checkpoint under test‑time modality dropout by 3.2 to 10.9 R@1.

By Ambuj Mehrish, Anindya Nag, Sebastiano Vascon

diffusion fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 3

Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals

arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.

By Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang, Xinpei Wang, Weitong Chen

multimodal benchmarks

arXiv AI

Jun 9

SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM

arXiv:2511. 14143v2 Announce Type: replace-cross Abstract: Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query.

By An Yu, Weiheng Lu, Jian Li, Zhenfei Zhang, Yunhang Shen, Felix X. -F. Ye, Ming-Ching Chang

llms multimodal benchmarks

One email a morning, machine-written

Related stories

Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

Query-Conditioned Spherical Centroid Aggregation for Multimodal Retrieval

Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals

SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM