arXiv AI

mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models

arXiv:2606. 07531v1 Announce Type: cross Abstract: We introduce mllm-shap, an open-source Python framework designed to extend Shapley Value (SV) explainability from text-only Large Language Models to Multimodal LLMs (MLLMs) processing joint text and audio inputs.

arXiv AI

Jun 9

Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis

arXiv:2606. 07533v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) effectively integrate text and audio to interpret context in complex interactive dialogues.

By Pawe{\l} Pozorski, Jakub Muszy\'nski, Maria Ganzha

llms nlp multimodal safety

arXiv AI

Jun 2

VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio

arXiv:2512. 10120v2 Announce Type: replace-cross Abstract: General-purpose audio representations aim to map acoustically variable instances of the same event to nearby points, resolving content identity in a zero-shot setting.

By Maris Basha, Anja Zai, Sabine Stoll, Richard Hahnloser

rag multimodal benchmarks safety

arXiv Machine Learning

Jul 24

Toward Generalizable Cognitive Impairment Detection with Speech-Based Multimodal Large Language Models

arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.

By Yingchao Huang, Xin Wang, Yuhan Su, Shanshan Yao

llms rag multimodal benchmarks safety

arXiv AI

Jun 15

The Perceived Fragility of Explanations in Audio Models: Manipulation of Attribution with Unchanged Predictions

arXiv:2606. 14466v1 Announce Type: cross Abstract: This paper investigates the fragility of post-hoc explanation methods in audio deepfake detection.

By Piotr Kit{\l}owski, Dominik Wi\k{a}cek, Mateusz Modrzejewski

robotics benchmarks safety

arXiv Machine Learning

Jun 18

Investigating Faithfulness in Large Audio Language Models

arXiv:2509. 22363v4 Announce Type: replace Abstract: Large Audio Language Models (LALMs) integrate audio encoders with pretrained Large Language Models to perform complex multimodal reasoning tasks.

By Pooneh Mousavi, Lovenya Jain, Mirco Ravanelli, Cem Subakan

llms multimodal benchmarks safety

arXiv AI

Jun 30

AMR: Adaptive Modality Routing for Multimodal Polyglot Speaker Identification

arXiv:2606. 29335v1 Announce Type: cross Abstract: Multimodal speaker identification systems face two key challenges in real-world deployment: missing modalities and language mismatch between training and testing conditions.

By Chuxiao Zuo, Yao Zhu, Minqiang Xu, Manhong Wang, Yunke Zhang, Fei Huang

rag multimodal

arXiv AI

Jul 7

Unified Audio Intelligence Without Regressing on Text Intelligence

arXiv:2607. 05196v1 Announce Type: cross Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech.

By Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

llms rag agents reinforcement-learning efficiency multimodal benchmarks safety

arXiv AI

Jul 9

Audio Sentiment Analysis via Distillation and Cross-Modal Integration of Generated Multilingual Transcripts

arXiv:2607. 06611v1 Announce Type: cross Abstract: Automatically recognizing the sentiment, positive or negative, from speech is a challenging task, requiring both the analysis of vocal inflections and the interpretation of uttered words.

By Andrei-George Durdun, Victor Constantinescu, Radu Tudor Ionescu

llms nlp efficiency multimodal benchmarks

arXiv AI

Aug 6

AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation

arXiv:2608. 04479v1 Announce Type: cross Abstract: Text-to-audio (TTA) generation has recently achieved remarkable progress in synthesizing realistic audio from natural language descriptions.

By Jinting Wang, Yuguang Yang, Shengyu Li, Yan Rong, Shan Yang, Xiaoda Yang, Li Liu

benchmarks safety

arXiv AI

Jul 3

Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas

arXiv:2607. 02504v1 Announce Type: cross Abstract: Long-form TV dramas present a formidable challenge for comprehensive video understanding, where deciphering complex storyline often relies on \textbf{speaker recognition}, the task of accurately attributing each spoken utterance to its respective character.

By Yuxuan Li, Lingxi Xie, Xinyue Huo, Jihao Qiu, Jiacheng Shao, Pengfei Chen, Jiannan Ge, Kaiwen Duan, Qi Tian

llms multimodal benchmarks

arXiv AI

Jun 18

Reference-Driven Multi-Speaker Audio Scene Generation from In-the-Wild Priors

arXiv:2606. 19325v1 Announce Type: cross Abstract: Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings.

By Michael Finkelson, Daniel Segal, Eitan Richardson, Shahar Armon, Nani Goldring, Poriya Panet, Nir Zabari, Benjamin Brazowski, Or Patashnik, Yoav HaCohen

rag benchmarks

arXiv AI

Jun 9

Audio-FLAN: An Instruction-Following Dataset for Unified Audio Understanding and Generation of Speech, Music, and Sound

arXiv:2502. 16584v2 Announce Type: replace-cross Abstract: Recent advancements in audio tokenization have significantly enhanced the integration of audio capabilities into large language models (LLMs).

By Liumeng Xue, Ziya Zhou, Jiahao Pan, Zixuan Li, Shuai Fan, Yinghao Ma, Sitong Cheng, Dongchao Yang, Haohan Guo, Yujia Xiao, Xinsheng Wang, Zixuan Shen, Chuanbo Zhu, Xinshen Zhang, Tianchi Liu, Ruibin Yuan, Zeyue Tian, Haohe Liu, Xingjian Du, Emmanouil Benetos, Ge Zhang, Yike Guo, Wei Xue

llms nlp fine-tuning

Related stories