arXiv AI
NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Sep 18
A frontend-backend architecture for tool calls in full-duplex speech models
The paper introduces a frontend‑backend architecture for full‑duplex speech‑to‑speech models that enables tool calls while preserving natural conversational flow. The frontend emits a delegation token and streams ASR transcripts to a text‑based backend LLM, which performs tool calls and returns results that are re‑injected into the frontend via a lightweight prefill‑and‑repeat mechanism before streaming TTS synthesis. In single‑turn evaluations the system achieves 92‑97% tool‑call recall, 81.2% accuracy in rejecting irrelevant calls, and competitive performance on Full‑Duplex‑Bench‑V3 and EVA‑Bench when paired with a large backend model.
arXiv AI
Jun 9
Liberating LLM Capabilities in Full-Duplex Speech Models
arXiv:2606. 07547v1 Announce Type: cross Abstract: Speech-based large language models are typically constrained to spoken replies, which limits their user-facing outputs to what can be verbalized and suppresses text-native capabilities such as code generation, structured analysis, and multi-step reasoning in realtime interaction, for tasks that require persistent, structured, and inspectable intermediate outputs.
arXiv Computation and Language
Oct 12
People Making Conversation: Evaluating How Naturally Speech Models Converse
The study evaluates conversational speech models by placing them at 427 pauses and 482 turn ends from 176 unscripted conversations, scoring their responses on timing, latency, length, wording, and recall of recent dialogue. Hosted APIs (GPT Realtime, Gemini 3.1 Flash Live, Grok Voice) respond at almost every pause, with latency ranging from 1–4 s for silence-based endpointers and about 8 s for semantic turn detection, while full‑duplex models (PersonaPlex, Moshi) remain silent more often and can interrupt within human reaction time. Recall rates vary: hosted systems remember 41–60 % of facts under a Claude Opus 5.5 judge and 19–26 % with a strict word matcher; full‑duplex models recall 17–22 % and 12–15 % respectively.
arXiv Computer Vision
Sep 15
Realtime-Venus: A full-duplex interaction system with asynchronous delegation
arXiv:2609.13814v1 Announce Type: new Abstract: Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and li...
Hugging Face Trending Papers
Sep 12
Realtime-Venus: A full-duplex interaction system with asynchronous delegation
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requi...
arXiv AI
Oct 1
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular in...