← Back to all news
arXiv AI By Matthew Sun, Vinay Kothapally, Meng Yu, Chao Huang, Hao Zhang, Yixuan Zhang, Steve Yves

A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversations

Read the original on arXiv AI →

The paper introduces a pipeline that generates intent‑labeled, two‑channel conversational speech from relational event lists, enabling controlled synthesis of full‑duplex dialogue with 42 phenomena across eight families in English and Mandarin. By having an LLM author each event’s speaker, text, conversational act, and attachment, and then aligning and timing these events independently, the system produces diverse, realistic turn‑taking signals. Experiments show that models trained on this synthetic corpus achieve higher floor‑occupancy accuracy and better start‑speaking/listening F1 scores compared to models trained on prior data.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Aug 27

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

TurnBench is a new multi‑domain benchmark for evaluating turn‑taking dynamics in spoken dialogue. It comprises a 30‑hour hand‑labeled corpus of dyadic human conversations, a standardized evaluation protocol for end‑of‑turn and interruption detection, and covers six distinct interaction styles with triple annotation. The benchmark also provides a 104‑hour training set, a public leaderboard, and an interactive dataset viewer at https://turnbench.sesame.com.

By Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri, Simon Michael Vuch Williams, Elliott K. Suen, Garreth Lee, Kevin Yoonho Choi, Takuya Umeki, Riku Kubo, Sathvik Udupa, Chien-yu Huang, Shih-Yun Shan Kuan, Zhuoyan Tao, Satyapriya Krishna, Sefik Emre Eskimez, Yu Tsao, Hung-yi Lee, Shinji Watanabe
benchmarks
More like this →
arXiv AI
Oct 7

PERSIST: Who-What-When Memory Across Sessions for Full-Duplex Spoken Dialogue

PERSIST is a persistent memory system designed for multi-session, multi-speaker spoken dialogue, explicitly modeling Who, What, and When to enable voice assistants to recall past conversations and adapt to individual users. It structures cross-session histories into readable event records and retrieves them using a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state. By reusing intermediate representations from the dialogue backbone, PERSIST reduces retrieval latency from 578.42 ms to 7.03 ms and achieves 85.08% end‑to‑end task accuracy on the SpokenTrace benchmark, improving EM@3 from 49.01% to 82.10%.

By Achira Lin, Siyuan Hou, Wenyi Yu, Xinnian Zhao, Haoyu Niu, Wang Geng, Longshuai Xiao, Shihai Xiao, Mangsuo Zhao, Chao Zhang
benchmarks
More like this →
arXiv Computation and Language
Oct 12

People Making Conversation: Evaluating How Naturally Speech Models Converse

The study evaluates conversational speech models by placing them at 427 pauses and 482 turn ends from 176 unscripted conversations, scoring their responses on timing, latency, length, wording, and recall of recent dialogue. Hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause, with latency ranging from 1–4 s for silence-based endpointers and about 8 s for semantic turn detection, while full‑duplex models (PersonaPlex, Moshi) remain silent more often and can interrupt within human reaction time. Recall rates vary: hosted systems remember 41–60 % of facts under a Claude Opus 5.5 judge and 19–26 % with a strict word matcher; full‑duplex models recall 17–22 % and 12–15 % respectively.

By Arya Bastani, Iman Modarressi
llms
More like this →