arXiv Computer Vision

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

arXiv AI

Jun 18

Revisiting Active Speaker Detection: An In-the-Wild Benchmark for Generalization and Robustness

arXiv:2505. 21954v2 Announce Type: replace-cross Abstract: We present UniTalk, a novel dataset emphasizing challenging scenarios to enhance model generalization for the task of active speaker detection (ASD).

By Le Thien Phuc Nguyen, Zhuoran Yu, Khoa Quang Nhat Cao, Yuwei Guo, Tu Ho Manh Pham, Tuan Tai Nguyen, Toan Ngo Duc Vo, Lucas Poon, Tuan Khai Nguyen, Soochahn Lee, Yong Jae Lee

benchmarks

arXiv AI

Jun 18

Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

arXiv:2606. 19325v1 Announce Type: cross Abstract: Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings.

By Michael Finkelson, Daniel Segal, Eitan Richardson, Shahar Armon, Nani Goldring, Poriya Panet, Nir Zabari, Benjamin Brazowski, Or Patashnik, Yoav HaCohen

rag benchmarks

arXiv AI

Jun 16

Dual-Granularity Orthogonal Disentanglement for Generalizable Audio Deepfake Detection

arXiv:2606. 16532v1 Announce Type: cross Abstract: Audio deepfake detectors often fail to generalize across speakers, as they learn speaker-identity features rather than synthesis artifacts, known as implicit identity leakage.

By Zhuodong Liu, Hugen Lv, Xiangyu Li, Chunhong Yuan

rag safety

Hugging Face Trending Papers

Jun 17

Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations.

rag benchmarks

arXiv AI

Aug 25

DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts

arXiv:2608.23363v1 Announce Type: cross Abstract: Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deep...

By Vlad Hondru, Florinel Alin Croitoru, Iuliana Georgescu, A. Sophia Koepke, Radu Tudor Ionescu

multimodal benchmarks

arXiv AI

Jul 17

Large Audio Language Models for Spoofing-Aware Speaker Verification

arXiv:2607. 14753v1 Announce Type: cross Abstract: Recent advances in text-to-speech and voice cloning make high-quality spoofing inexpensive and scalable, threatening voice authentication systems, especially automatic speaker verification (ASV).

By Sofya Savelyeva, Mariia Perunova, Evgeny Kushnir, Artem Dvirniak, Dmitrii Korzh, Oleg Y. Rogov

llms multimodal

arXiv Machine Learning

Jun 5

To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection

arXiv:2606. 05931v1 Announce Type: cross Abstract: When retrieving a person from a video archive by voice and face, should the system be multimodal or not?

By Erfan Loweimi, Mengjie Qian, Kate Knill, Guanfeng Wu, Chi-Ho Chan, Abbas Haider, Muhammad Awan, Josef Kittler, Hui Wang, Mark Gales

multimodal benchmarks

arXiv Computer Vision

4d ago

Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deployment

arXiv:2501.18157v2 Announce Type: replace-cross Abstract: Building reliable speech systems often requires combining multiple modalities, like audio and visual cues. While such multimodal solutions fr...

By Joanna Hong, Sanjeel Parekh, Honglie Chen, Jacob Donley, Ke Tan, Buye Xu, Anurag Kumar

multimodal

arXiv Machine Learning

Jun 19

Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning

arXiv:2606. 19823v1 Announce Type: cross Abstract: Automatic speech recognition remains unreliable for dysarthric speech due to data scarcity and high inter-speaker variability.

By Satwinder Singh, Qianli Wang, Zihan Zhong, Clarion Mendes, Hasegawa-Johnson, Waleed Abdulla, Seyed Reza Shahamiri

fine-tuning multimodal

arXiv Computer Vision

Sep 10

AVSRBench: A Multi-Condition AVSR Benchmark

arXiv:2609.10366v1 Announce Type: cross Abstract: While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true...

By Rishabh Jain, Naomi Harte

llms fine-tuning multimodal benchmarks

arXiv Computer Vision

Sep 10

Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition

arXiv:2609.10394v1 Announce Type: cross Abstract: Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the...

By Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin, Naomi Harte

multimodal benchmarks

arXiv AI

Jul 10

PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

arXiv:2607. 08111v1 Announce Type: cross Abstract: Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable.

By Wanyi Ning, Wei Zhou, Yingpeng Li, Yinshang Guo, Haitao Qian, Yiming Cheng

fine-tuning multimodal benchmarks

Related stories