Full-duplex speech models require training data that preserves turn-taking, overlap, interruption, and backchannel behavior, yet these signals are entangled across speakers in noisy real-world recordi...
arXiv:2608.16053v2 Announce Type: replace
Abstract: Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dia...
By Pengcheng Wang, Sheng Li, Jiyi Li, Takahiro Shinozaki
The paper introduces Alignment-Free Text‑Audiobox (Text‑AB), a unified diffusion‑based framework that performs high‑quality voice dubbing and full‑duplex dialogue synthesis without requiring forced alignment. Text‑AB uses a latent diffusion model with DAC‑VAE features, achieving over 10× compression compared to prior EnCodec representations, and learns text‑speech alignment via cross‑attention. The authors pretrain a 3B‑parameter model on 480k hours of monolingual speech and fine‑tune it for cross‑lingual dubbing, full‑duplex dialogue, and emotional dialogue, reporting significant improvements in prosody, voice similarity, naturalness, and emotional expressivity over existing internal systems.
By Sanyuan Chen, Min-Jae Hwang, Sho Inoue, Anna Sun, Bokai Yu, David Kant, Dongmin Hyun, Dorian Desblancs, Gregory Antonovsky, Oleg Repin, Peng-Jen Chen, Xutai Ma, Zehai Tu, Juan Pino, Wei-Ning Hsu
arXiv:2606.11167v2 Announce Type: replace
Abstract: Full-duplex spoken dialogue models can listen and speak simultaneously, making them a promising architecture for natural conversation. However, cur...
By Atsumoto Ohashi, Neil Zeghidour, Alexandre D\'efossez, Eugene Kharitonov
arXiv:2606. 07547v1 Announce Type: cross Abstract: Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs.
By Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao
ECHO is a new paired diagnostic benchmark designed for Chinese full‑duplex spoken dialogue systems to evaluate context‑sensitive turn‑taking. It pairs examples that share the same overlap transcript but differ in preceding multi‑turn context, requiring either a Yield or Keep decision, and also includes off‑talk cases to test unnecessary yielding. The benchmark introduces pair accuracy, penalizing constant‑action policies, and shows that many systems are biased toward Yield, performing better on interruptions than on backchannels.
By Shuofeng Zhao, Hongwei Cai, Wenke Fan, Qingxiang Guo, Dawei Yang, Zhou Wang, Zhiyang Zhou, Yingxin Shang, Weixu Wang, Lin Yang, Shuran Zhou, Yang Song