Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS

How OpenAI delivers low-latency voice AI at scale

How OpenAI rebuilt its WebRTC stack to power real-time Voice AI with low latency, global scale, and seamless conversational turn-taking.

$\tau$-Multilingual: Benchmarking Voice Agents Across Languages

arXiv:2609.35820v1 Announce Type: cross Abstract: English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce $\tau$-Multilingual, extending $\tau$-Voice to Spanish, Braz...

NemotronLabs VoiceChat: An Open Full-duplex Speech-to-Speech Model with Tool Calling Capabilities

arXiv:2609.21967v1 Announce Type: cross Abstract: We introduce NemotronLabs VoiceChat, an open full-duplex speech-to-speech model with native tool-calling capabilities. NemotronLabs VoiceChat combine...

ChipChat: Low-Latency Cascaded Conversational Agent in MLX

arXiv:2509. 00078v2 Announce Type: replace-cross Abstract: The emergence of large language models (LLMs) has transformed spoken dialog systems, yet the optimal architecture for real-time on-device voice agents remains an open question.

Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning

MLLP-VRAIN UPV system for the IWSLT 2026 Simultaneous Speech Translation task

arXiv:2606. 17255v1 Announce Type: cross Abstract: This work describes the participation of the MLLP-VRAIN research group in the shared task of the IWSLT 2026 Simultaneous Speech Translation track.

Offline AI Modules: Voice-First Offline Architecture, Hardware Reference Stack, Quantization and Benchmarking

The paper introduces the Offline AI Modules workstream, which provides a voice‑first offline architecture, a low‑cost hardware reference bill of materials, and a reproducible quantization and benchmarking pipeline for instruction‑tuned language models in the 2‑5B parameter range. It evaluates three models across four quantization formats on two hardware tiers—NVIDIA Jetson Orin NX and Raspberry Pi5—measuring deployment metrics and multilingual quality. The key result is that Q4_K_M quantization offers the best size‑to‑quality trade‑off, enabling high decode throughput and strong topic classification accuracy while staying within memory limits on both tiers.

X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System

X-Translator is a low‑cost, modular real‑time speech‑to‑speech translation system that integrates streaming ASR, machine translation, and prompt‑conditioned TTS, managed by a session‑level runtime controller. It uses incremental segment commitment to stabilize ASR streams and an online speaker prompt manager to maintain speaker consistency across multi‑speaker conversations. The system is evaluated on translation quality, speech naturalness, latency, and speaker preservation using OpenSTBench, and its code and demo are publicly available on GitHub.

Advancing voice intelligence with new models in the API

Explore new realtime voice models in the OpenAI API that can reason, translate, and transcribe speech, enabling more natural and intelligent voice experiences.

Voxtral 3

Voxtral 3 is Mistral AI’s voice‑agent SDK that enables real‑time speech processing in over 50 languages. It offers accurate transcription, emotionally expressive text‑to‑speech, and deep tool‑use capabilities. The SDK can be hosted in the EU or self‑deployed by users.

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

arXiv:2607. 18171v1 Announce Type: new Abstract: Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism.

Scalable Context Orchestration for Serving LLMs Over Voice

Scalable Context Orchestration for Serving LLMs Over Voice presents llmovoice, a middleware that explicitly models voice context—including speaking rate, background noise, and packet loss—to guide large language model responses. By constructing a bounded voice context at each turn, llmovoice improves alignment with user preferences and reduces errors, achieving a 52.4% drop in speaking‑rate alignment error and a 0.9% false‑interruption rate under packet loss. In addition, it cuts model usage costs dramatically, lowering per‑turn cost by up to 24.9× while maintaining 98.7% of baseline answer quality in long sessions.