arXiv Computation and Language
FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Sep 4
Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue
The paper introduces a decoupled data approach for the Neural Finite State Machine (NFSM) framework to improve full‑duplex dialogue. It serializes real human‑human spoken dialogues into FSM tapes using a rule‑based event‑guided transformation, while shaping semantics through human‑agent text dialogues. A Source‑Aware Calibrated (SAC) loss is proposed to balance state‑transition token distribution and align each data source with its strongest supervisory signal, leading to better turn‑taking performance without sacrificing semantic quality.
arXiv AI
Sep 12
Less can be More: What Aspects of Speech Drive End-of-Turn Detection
The study investigates which speech aspects best signal the end of a speaker’s turn in conversational AI. By systematically ablating acoustic, prosodic, and semantic cues in a lightweight trimodal classifier, the authors find that combining acoustic and prosodic features yields the highest accuracy and lowest latency, achieving an utterance F1 of 0.93 with 7.8% false alarms at 400 ms median latency. Adding textual information actually increases premature detections without improving performance, and feature analysis shows prosodic cues provide the strongest class separability while text representations overlap significantly.
arXiv AI
Sep 10
X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR
X2Streaming-ASR introduces a method for streaming automatic speech recognition that separates the decision of when to commit a transcript from what to commit. The approach uses a three‑stage training process: first establishing streaming capability, then warm‑starting a commit policy with automatically probed trajectories, and finally refining the policy with character‑level, segment‑assigned group‑relative rewards for accuracy and latency. On AISHELL‑1/2/3 and WenetSpeech datasets, the system achieves mean character‑level commit latencies of 27–84 ms, far lower than baseline systems, while also attaining the best streaming character error rates on AISHELL‑1 and AISHELL‑3.
arXiv Machine Learning
Sep 7
GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue
GEPARD is a streaming text‑to‑speech model that uses a standard large language model backbone to generate speech autoregressively, decoding audio with an FSQ‑based neural codec. It streams audio chunk‑by‑chunk as text arrives, achieving a real‑time factor of about 0.067 and an aggregate speedup of roughly 204× on a single GPU with 256 concurrent streams. The design keeps all complex auxiliary mechanisms outside the decode loop, enabling deployment with a standard LLM engine (vLLM) without kernel modifications.
arXiv AI
Jun 4
Audio Interaction Model
arXiv:2606. 05121v1 Announce Type: cross Abstract: Audio is an inherently interactive modality, yet today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.
arXiv Computation and Language
Aug 28
SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models
SPAR-K is a scheduled periodic alternating early‑exit framework for interleaved spoken language models that reduces decoding depth for speech tokens while maintaining quality. It lets most speech positions exit at a fixed intermediate layer and inserts periodic full‑depth refresh steps to counter distribution shift. Experiments on Step‑Audio‑2‑mini and GLM‑4‑Voice show up to 11 % depth reduction with less than 0.82 % drop in question‑answering accuracy and negligible impact on MOS and WER.