arXiv:2603.16783v2 Announce Type: replace Abstract: Robust voice agents require exposure to the full diversity of how people interact through speech. However, obtaining enough spoken interactions is...
arXiv:2510.22588v2 Announce Type: replace-cross Abstract: Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction tha...
arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular in...
The paper introduces a frontend‑backend architecture for full‑duplex speech‑to‑speech models that enables tool calls while preserving natural conversational flow. The frontend emits a delegation token and streams ASR transcripts to a text‑based backend LLM, which performs tool calls and returns results that are re‑injected into the frontend via a lightweight prefill‑and‑repeat mechanism before streaming TTS synthesis. In single‑turn evaluations the system achieves 92‑97% tool‑call recall, 81.2% accuracy in rejecting irrelevant calls, and competitive performance on Full‑Duplex‑Bench‑V3 and EVA‑Bench when paired with a large backend model.
arXiv:2608.16053v2 Announce Type: replace Abstract: Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dia...
The study evaluates conversational speech models by placing them at 427 pauses and 482 turn ends from 176 unscripted conversations, scoring their responses on timing, latency, length, wording, and recall of recent dialogue. Hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause, with latency ranging from 1–4 s for silence-based endpointers and about 8 s for semantic turn detection, while full‑duplex models (PersonaPlex, Moshi) remain silent more often and can interrupt within human reaction time. Recall rates vary: hosted systems remember 41–60 % of facts under a Claude Opus 5.5 judge and 19–26 % with a strict word matcher; full‑duplex models recall 17–22 % and 12–15 % respectively.
The paper introduces a reproducible framework that converts existing text-based tool‑calling benchmarks into audio‑based evaluations using text‑to‑speech, speaker variation, and environmental noise, while preserving original annotations. Experiments on seven omni‑modal models show that performance varies strongly by model and task, with notable gaps between text and voice results. The study also includes a stress test, a reference‑free LLM‑as‑judge protocol, and demonstrates that open‑source judges can match proprietary ones, aiding privacy‑preserving evaluation.
The paper introduces a pipeline that generates intent‑labeled, two‑channel conversational speech from relational event lists, enabling controlled synthesis of full‑duplex dialogue with 42 phenomena across eight families in English and Mandarin. By having an LLM author each event’s speaker, text, conversational act, and attachment, and then aligning and timing these events independently, the system produces diverse, realistic turn‑taking signals. Experiments show that models trained on this synthetic corpus achieve higher floor‑occupancy accuracy and better start‑speaking/listening F1 scores compared to models trained on prior data.
RelayS2S is a hybrid real‑time dialogue system that runs a fast duplex speech‑to‑speech path and a slow ASR‑to‑LLM path in parallel. The fast path speculatively drafts a short response prefix and streams it to TTS, while the slow path generates a higher‑quality continuation conditioned on that prefix. A lightweight verifier decides whether to commit the prefix or fall back to the cascaded pipeline, achieving much lower latency (81 ms P90 first‑chunk) while preserving 99% of the cascaded pipeline’s textual quality.
The paper introduces a decoupled data approach for the Neural Finite State Machine (NFSM) framework to improve full‑duplex dialogue. It serializes real human‑human spoken dialogues into FSM tapes using a rule‑based event‑guided transformation, while shaping semantics through human‑agent text dialogues. A Source‑Aware Calibrated (SAC) loss is proposed to balance state‑transition token distribution and align each data source with its strongest supervisory signal, leading to better turn‑taking performance without sacrificing semantic quality.
arXiv:2606. 03957v1 Announce Type: cross Abstract: Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data.
VoiceLongMemEval (VLME) is a new benchmark that tests AI assistants on their ability to remember how users sounded by incorporating paralinguistic metadata—such as emotion labels, prosody descriptors, and voice events—into each conversational turn. The benchmark uses a three‑stage adversarial gate to ensure that models cannot succeed with transcript alone, revealing a significant affect gap: models gain 0.09 to 0.38 accuracy when provided with paralinguistic cues, and audio‑native models outperform standard ASR pipelines in extracting these signals. The dataset and code will be released upon acceptance.