PERSIST is a persistent memory system designed for multi-session, multi-speaker spoken dialogue, explicitly modeling Who, What, and When to enable voice assistants to recall past conversations and adapt to individual users. It structures cross-session histories into readable event records and retrieves them using a 3W joint scoring mechanism that combines semantic content, acoustic speaker identity, and temporal state. By reusing intermediate representations from the dialogue backbone, PERSIST reduces retrieval latency from 578.42 ms to 7.03 ms and achieves 85.08% end‑to‑end task accuracy on the SpokenTrace benchmark, improving EM@3 from 49.01% to 82.10%.
By Achira Lin, Siyuan Hou, Wenyi Yu, Xinnian Zhao, Haoyu Niu, Wang Geng, Longshuai Xiao, Shihai Xiao, Mangsuo Zhao, Chao Zhang
arXiv:2606. 07547v1 Announce Type: cross Abstract: Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs.
By Luoyuan Zhang, Bokai Xu, Junbo Cui, Weiyue Sun, Yingjing Xu, Hanyu Liu, Yuan Yao
arXiv:2609.39446v1 Announce Type: new
Abstract: Existing full-duplex speech benchmarks cover only subsets of real-time interaction behaviors, often under limited contextual conditions. We introduce D...
By Keyue Xing, Wentao Ding, Mengmeng Wang, Wenming Tu, Zilong Zheng, Yipeng Kang
The paper introduces acoustic-to-text KV compression for full‑duplex speech models, converting acoustic key‑value states into compact textual memory during listening‑time slack. When the KV cache exceeds a target budget, older acoustic states are evicted while transcripts and recent acoustic context are retained. Experiments on ten‑minute LongSpeech sessions show a 64.6% reduction in peak streaming KV‑cache size and improved transcription, temporal question answering, and summarization, with comparable pause‑handling, turn‑taking, and interruption performance in Full‑Duplex‑Bench.
By Yejin Lee, Seungbeom Kim, Yongha Lee, Kyuhong Shim
arXiv:2609.21967v1 Announce Type: cross
Abstract: We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combine...
By Jagadeesh Balam, Travis Bartley, Edresson Casanova, Sanjay Chauhan, Chen Chen, Zhehuai Chen, Zijia Chen, Francesco Ciannella, Slyne Deng, Mikyas Desta, Harishchandra Dubey, Slim Essid, Nourchene Ferchichi, Boris Ginsburg, Mariana Graterol Fuenmayor, Negar Habibi, Kevin Hu, Anand Joseph, Viraj Karandikar, Myungjong Kim, Viacheslav Klimkov, Seelan Lakshmi Narasimhan, Lily Lee, Jason Li, Eileen Long, Ameya Mahabaleshwarkar, Aditya Malte, Adi Margolin, Sasha Meister, Valentin Mendelev, Oluwatobi Olabiyi, Ankita Pasad, Yifan Peng, Elena Rastorgueva, Jayda Ritchie, Jason Roche, Nikhil Srihari, Yuanhang Su, Yoshi Suhara, Viet Anh Trinh, Jinhan Wang, Piotr Zelasko, Hui Wang, Puhui Meng, Chaosen Zhang, Yunsheng Liu, Shawn Wang, Wenjing Li, Zhonglei He
The paper introduces a frontend‑backend architecture for full‑duplex speech‑to‑speech models that enables tool calls while preserving natural conversational flow. The frontend emits a delegation token and streams ASR transcripts to a text‑based backend LLM, which performs tool calls and returns results that are re‑injected into the frontend via a lightweight prefill‑and‑repeat mechanism before streaming TTS synthesis. In single‑turn evaluations the system achieves 92‑97% tool‑call recall, 81.2% accuracy in rejecting irrelevant calls, and competitive performance on Full‑Duplex‑Bench‑V3 and EVA‑Bench when paired with a large backend model.
By Ke Hu, Slyne Deng, Chen Chen, Elena Rastorgueva, Edresson Casanova, Punit Kumar, Dharmendra Choudhary, Nikhil Srihari, Ameya Sunil Mahabaleshwarkar, Viet Anh Trinh, Slim Essid, Oluwatobi Olabiyi, Zhehuai Chen
arXiv:2610.34991v1 Announce Type: new
Abstract: End-to-end full-duplex spoken dialogue models enable natural, real-time spoken interactions with interruptions, backchannels, and overlapping speech. P...
By Junho Kim, CheHyun Lee, Donghun Kang, Youngho Cho, Dongwook Lee, Heeseung Kim, Sungroh Yoon
arXiv:2610.32349v1 Announce Type: new
Abstract: Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a co...
By Yeonju Kim, Bella Godiva, Se Jin Park, Yong Man Ro
arXiv:2610.20172v1 Announce Type: cross
Abstract: Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel...
By Puneet Mathur, Dinesh Manocha
DuplexSpeechBench-IFEval (DSB-IFEval) is a new benchmark that evaluates how full‑duplex voice agents follow implicit instructions during real‑time spoken interaction. It contains 1,038 test cases across eight assistant roles and tests five conditioning protocols, measuring floor management with an Instruction Adherence Score (IAS) and persona consistency with a Persona Adherence Score (PAS). Experiments on six speech systems reveal architecture‑dependent trade‑offs, showing that some models are more sensitive to explicit versus persona‑only instructions and that even when following conflicting directives, they struggle to override them under safety conflict.
By Puneet Mathur, Dinesh Manocha
DuplexAgent-RSI is a full‑duplex voice‑agent collaboration system that coordinates continuous listening and speaking with asynchronous delegation to reasoning and coding agents. Its harness is modular, consisting of six editable components, and a closed‑loop improvement process (Duplex‑Harness‑RSI) revises these modules based on interaction traces and failure diagnostics. Experiments show that this approach yields stronger spoken‑knowledge and executable‑tool performance than comparable delegated systems while preserving robust interruption response.
By Yingda Shen, Yuxiang Wang, Kunyu Feng, Qinke Ni, Jiaqi Li, Minghao Hsu, Junan Zhang, Dekun Chen, Yutong Bian, Zhizheng Wu
arXiv:2609.13814v1 Announce Type: new
Abstract: Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and li...
By Ruixiang Zhao, Hualei Wang, Renhe Sun, Enzhi Zhou, Jincenzi Wu, Xujie Song, Kexin Shi, Zihang Liu, Pengcheng Zhu, Jiayi Zhou, Baoyue Zhang, Changhao Zhang, Zitong Wang, Jinhong Wang, Tong Niu, Jingjing Liu, Junan Lin, Haolin He, Hengshuo Chu, Yuhui Chen, Jian Liu, Yuge Huang, Junliang Xing, Yuntao Wang, Weiqiang Wang, Chun Yu, Yuanchun Shi