DuplexDrama is a newly announced synthesized spoken dialogue dataset that uniquely combines complete persona and scenario settings, three full‑duplex behaviors (interruption, backchannel, incomplete), expressive speech with persona‑aligned emotion labels, and script‑aware sound events. The dataset was created through a four‑stage pipeline and validated for quality on both scripts and audio, yielding over 2,000 hours of audio featuring 64 voices across 13 personas and 5 age groups, with 3.8% of turns containing full‑duplex behaviors. A curated bilingual subset of 6,400 dialogues (800 hours total) will be released to support research in full‑duplex spoken dialogue models, and evaluation prompts will accompany the dataset.
By Qingxiang Guo, Wenke Fan, Shuofeng Zhao, Dawei Yang, Zhiyang Zhou, Yingxin Shang, Hongwei Cai, Zhou Wang, Weixu Wang, Lin Yang, Shuran Zhou, Yang Song
SteerDuplex is a full‑duplex speech dialogue model that can be steered along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. The authors introduce a taxonomy of text‑ and audio‑based steerability, identify gaps in existing models, and fine‑tune a Moshi‑based model with reinforcement learning to improve timing and response continuity. They also present SteerBench, a benchmark of 390 spoken prompts and 1,067 human‑authored rubrics, showing significant gains in audio‑steering pass rates and interruption handling compared to open baselines.
By Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai, Sonal Kumar, Nikhil Barhate, Isabell Sagar, Steven Li, Miheer Bavare, Daniel Quigley, Fabiola Tapia Carrillo, Jose M Patron E, Diego Mac\'ias Guti\'errez, Paul Song, Ramani Duraiswami, Dinesh Manocha, Yunzhong He
arXiv:2606.11167v2 Announce Type: replace
Abstract: Full-duplex spoken dialogue models can listen and speak simultaneously, making them a promising architecture for natural conversation. However, cur...
By Atsumoto Ohashi, Neil Zeghidour, Alexandre D\'efossez, Eugene Kharitonov
arXiv:2609.24812v1 Announce Type: new
Abstract: Voice provides a natural and immediate interface for AI agents. Many settings in which voice agents could be useful, including meetings, households, an...
By Chenxu Xiong, Dongming Shen, Yuzhi Tang, Wentao Ma, Mu Li, Alex Smola
arXiv:2609.21967v1 Announce Type: cross
Abstract: We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combine...
By Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan, Chen Chen, Zhehuai Chen, Zijia Chen, Francesco Ciannella, Slyne Deng, Mikyas Desta, Harishchandra Dubey, Slim Essid, Nourchene Ferchichi, Boris Ginsburg, Mariana Graterol Fuenmayor, Negar Habibi, Kevin Hu, Anand Joseph, Viraj Karandikar, Myungjong Kim, Viacheslav Klimkov, Seelan Lakshmi Narasimhan, Lily Lee, Jason Li, Eileen Long, Ameya Mahabaleshwarkar, Aditya Malte, Adi Margolin, Sasha Meister, Valentin Mendelev, Oluwatobi Olabiyi, Ankita Pasad, Yifan Peng, Elena Rastorgueva, Jayda Ritchie, Jason Roche, Nikhil Srihari, Yuanhang Su, Yoshi Suhara, Viet Anh Trinh, Jinhan Wang, Piotr Zelasko, Hui Wang, Puhui Meng, Chaosen Zhang, Yunsheng Liu, Shawn Wang, Wenjing Li, Zhonglei He
Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordi...
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
By Syeda Faiza Ahmed, Zien Sheikh Ali, Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury
arXiv:2606. 07547v1 Announce Type: cross Abstract: Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs.
By Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao
SEA-SpeechBench is a large‑scale multitask benchmark for speech understanding in 11 Southeast Asian languages, comprising 97,194 samples across 99 evaluation sets and 597 hours of curated audio. It covers nine tasks in three categories—speech processing, paralinguistic analysis, and a novel temporal understanding dimension—using multilingual prompting in both native SEA languages and English. Evaluation of current models shows significant performance gaps, especially in temporal understanding, emotion recognition, and speech translation, with low‑resource languages lagging behind English by up to 41 percentage points.
By Jingyi Liao, Wenyu Zhang, Zhuohan Liu, Yingxu He, Geyu Lin, Xunlong Zou, Shuo Sun, Syed Ali Redha Alsagoff, Ai Ti Aw
Motion-Omni is an end‑to‑end framework that jointly generates spoken dialogue and full‑body motion, producing speech, facial expressions, and hand, upper‑body, and lower‑body movements directly from the hidden states of a language model. The system requires joint training of the language model, speech generator, and motion generator to maintain audio‑motion alignment, and it is supervised using a scalable, model‑agnostic pipeline that pseudo‑labels 422,856 speech‑motion pairs. With a Qwen2.5‑7B‑Instruct backbone, Motion‑Omni‑Q7 achieves near‑cascade performance on motion metrics while being 5.4× faster, and it outperforms other non‑teacher cascades on beat correlation, diversity, and word error rate.
By Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo
MENASpeechBank is a new reference voice bank that provides about 18,000 high‑quality utterances from 124 speakers across multiple MENA countries, covering English, Modern Standard Arabic, and regional Arabic varieties. The dataset is built through a controllable pipeline that creates persona profiles inspired by the World Values Survey, defines a taxonomy of roughly 5,000 conversational scenarios, matches personas to scenarios via semantic similarity, and generates around 417,000 role‑play conversations using an LLM. Synthetic speaker‑conditioned user‑turn audio is produced from reference recordings to maintain speaker diversity, and both synthetic and human‑recorded conversations are evaluated and analyzed for quality.
By Zien Sheikh Ali, Hunzalah Hassan Bhatti, Rabindra Nath Nandi, Shammur Absar Chowdhury, Firoj Alam
arXiv:2608. 04586v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT).
By Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Chengpeng Fu, Yu Wang, Ming Liu