← Back to all news
arXiv Computation and Language By Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai, Sonal Kumar, Nikhil Barhate, Isabell Sagar, Steven Li, Miheer Bavare, Daniel Quigley, Fabiola Tapia Carrillo, Jose M Patron E, Diego Mac\'ias Guti\'errez, Paul Song, Ramani Duraiswami, Dinesh Manocha, Yunzhong He

SteerDuplex: Steerable Duplex Speech Dialogue Models

Read the original on arXiv Computation and Language →

SteerDuplex is a full‑duplex speech dialogue model that can be steered along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. The authors introduce a taxonomy of text‑ and audio‑based steerability, identify gaps in existing models, and fine‑tune a Moshi‑based model with reinforcement learning to improve timing and response continuity. They also present SteerBench, a benchmark of 390 spoken prompts and 1,067 human‑authored rubrics, showing significant gains in audio‑steering pass rates and interruption handling compared to open baselines.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Oct 9

SteerablePlex: Can We Steer Full-Duplex Models?

The paper introduces SimIF-Bench, a benchmark that tests whether full‑duplex conversational models can stay within a prescribed scenario and complete multiple goals in order. It shows that current open‑source models struggle with these constraints. The authors then propose a GDPO‑based training recipe that allows a full‑duplex model, named SteerablePlex, to follow textual instructions during conversation while preserving turn‑taking, and demonstrate that it outperforms existing open‑source models and GPT‑Realtime when used as a controllable user simulator.

By Haolong Zheng, Maike Z\"ufle, Dominik Mach\'a\v{c}ek, Peter Pol\'ak, Xulin Fan, Xavier Sumba, Siyin Wang, Ond\v{r}ej Klejch, Mark Hasegawa-Johnson
llmsbenchmarks
More like this →
arXiv Computation and Language
Aug 27

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

TurnBench is a new multi‑domain benchmark for evaluating turn‑taking dynamics in spoken dialogue. It comprises a 30‑hour hand‑labeled corpus of dyadic human conversations, a standardized evaluation protocol for end‑of‑turn and interruption detection, and covers six distinct interaction styles with triple annotation. The benchmark also provides a 104‑hour training set, a public leaderboard, and an interactive dataset viewer at https://turnbench.sesame.com.

By Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri, Simon Michael Vuch Williams, Elliott K. Suen, Garreth Lee, Kevin Yoonho Choi, Takuya Umeki, Riku Kubo, Sathvik Udupa, Chien-yu Huang, Shih-Yun Shan Kuan, Zhuoyan Tao, Satyapriya Krishna, Sefik Emre Eskimez, Yu Tsao, Hung-yi Lee, Shinji Watanabe
benchmarks
More like this →