arXiv AI

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Hugging Face Trending Papers
Sep 3

Scalable Context Orchestration for Serving LLMs Over Voice

Scalable Context Orchestration for Serving LLMs Over Voice presents llmovoice, a middleware that explicitly models voice context—including speaking rate, background noise, and packet loss—to guide large language model responses. By constructing a bounded voice context at each turn, llmovoice improves alignment with user preferences and reduces errors, achieving a 52.4% drop in speaking‑rate alignment error and a 0.9% false‑interruption rate under packet loss. In addition, it cuts model usage costs dramatically, lowering per‑turn cost by up to 24.9× while maintaining 98.7% of baseline answer quality in long sessions.

arXiv AI
Sep 7

Scalable Context Orchestration for Serving LLMs Over Voice

The paper introduces llmovoice, a middleware that explicitly models voice context for large language model (LLM) serving in voice AI applications. By incorporating speaking rate, background noise, packet loss, and other paralinguistic factors into a bounded context, llmovoice guides the LLM to generate more aligned responses. Experiments show significant reductions in speaking‑rate errors, false interruptions, and model usage costs, especially in long voice sessions.

By Linyi Jiang, Silvery D. Fu, Yifei Zhu
arXiv Computation and Language
Oct 22

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

The paper introduces a reproducible framework that converts existing text-based tool‑calling benchmarks into audio‑based evaluations using text‑to‑speech, speaker variation, and environmental noise, while preserving original annotations. Experiments on seven omni‑modal models show that performance varies strongly by model and task, with notable gaps between text and voice results. The study also includes a stress test, a reference‑free LLM‑as‑judge protocol, and demonstrates that open‑source judges can match proprietary ones, aiding privacy‑preserving evaluation.

By Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN
arXiv AI
Aug 26

Benchmarking LLM Judges for Voice-Agent Evaluation: Reliability, Calibration, and Human Oversight

The paper evaluates the use of large language models (LLMs) as judges for assessing conversational voice agents, comparing human judgments with GPT‑4.1 and GPT‑5 across telecom and retail interactions. It examines agreement, metric‑level correlations, and consistency across three evaluation configurations (p0, p1, p2) to determine how reliably LLMs can judge conversational quality and safety. The study finds that LLM‑based evaluation can be effective but its reliability varies by metric and configuration, suggesting a hybrid approach where LLMs handle scalable assessment while humans focus on metrics requiring contextual interpretation.

By Anupam Purwar, Shashank Singh, Kritika Srivastava
arXiv AI
Sep 28

Inquesto Score: A reliability Protocol For Voice Agents

The paper introduces Inquesto Score (IS), a protocol that measures voice‑agent reliability by calculating the percentage of calls that reach the caller’s goal without functional failure. IS defines explicit failure events and severity levels, evaluates timing, semantic, and state‑dependent failures using audio, scenario predicates, tool traces, and a pinned open‑model judge, and provides diagnostic views on behavior, acoustic robustness, identity handling, and speaker groups. The authors evaluate IS v0.1 on 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations, demonstrating that reliable measurement requires evidence beyond transcripts and explicit treatment of deployment conditions.

By Massa Baali, Bhiksha Raj
arXiv Computation and Language
Sep 10

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

arXiv:2605.13841v3 Announce Type: replace-cross Abstract: Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversatio...

By Tara Bogavelli, Gabrielle Gauthier Melan\c{c}on, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, Srinivas Sunkara
arXiv Computation and Language
Sep 18

A frontend-backend architecture for tool calls in full-duplex speech models

The paper introduces a frontend‑backend architecture for full‑duplex speech‑to‑speech models that enables tool calls while preserving natural conversational flow. The frontend emits a delegation token and streams ASR transcripts to a text‑based backend LLM, which performs tool calls and returns results that are re‑injected into the frontend via a lightweight prefill‑and‑repeat mechanism before streaming TTS synthesis. In single‑turn evaluations the system achieves 92‑97% tool‑call recall, 81.2% accuracy in rejecting irrelevant calls, and competitive performance on Full‑Duplex‑Bench‑V3 and EVA‑Bench when paired with a large backend model.

By Ke Hu, Slyne Deng, Chen Chen, Elena Rastorgueva, Edresson Casanova, Punit Kumar, Dharmendra Choudhary, Nikhil Srihari, Ameya Sunil Mahabaleshwarkar, Viet Anh Trinh, Slim Essid, Oluwatobi Olabiyi, Zhehuai Chen
arXiv Computation and Language
Sep 14

SteerDuplex: Steerable Duplex Speech Dialogue Models

SteerDuplex is a full‑duplex speech dialogue model that can be steered along attributes such as tone, persona, speaking rate, and voice style in response to user instructions. The authors introduce a taxonomy of text‑ and audio‑based steerability, identify gaps in existing models, and fine‑tune a Moshi‑based model with reinforcement learning to improve timing and response continuity. They also present SteerBench, a benchmark of 390 spoken prompts and 1,067 human‑authored rubrics, showing significant gains in audio‑steering pass rates and interruption handling compared to open baselines.

By Utkarsh Tyagi, Ramaneswaran Selvakumar, Advait Gosai, Sonal Kumar, Nikhil Barhate, Isabell Sagar, Steven Li, Miheer Bavare, Daniel Quigley, Fabiola Tapia Carrillo, Jose M Patron E, Diego Mac\'ias Guti\'errez, Paul Song, Ramani Duraiswami, Dinesh Manocha, Yunzhong He
arXiv Machine Learning
Nov 2

Apriel Voice 27B Preview: An Open Model for Speech, Reasoning, and Tool Use

Apriel Voice 27B Preview is an open speech‑language model that unifies speech recognition, contextual reasoning, conversation maintenance, and tool use within a single reasoning backbone. Built on the post‑trained Gemma 4 26B‑A4B checkpoint, it processes speech directly and generates responses and tool calls without a separate dialogue model, using a two‑stage training procedure that adapts the projector and backbone through supervised fine‑tuning, text replay, and cross‑entropy/KL supervision. The model achieves strong performance on benchmarks such as Big Bench Audio (98.90), Speech‑BFCL (81.13), and τ‑Voice (65.51% task success), with a mean latency of 1.91 s, and is deployed on a custom serving platform for interactive voice interactions.

By Aman Tiwari, Akshay Kalkunte, Akintunde Oladipo, Jash Shah, Tara Bogavelli, Hoang Nguyen, Nishanth Madhusudhan, Vishal Juneja, Sridhar Nemala, Srinivas Sunkara, Sathwik Tejaswi Madhusudhan, Shruthan Radhakrishna, Masoud Hashemi, Kelechi Ogueji, Khalil Slimi, Saloni Mittal, Pulkit Pattnaik, Shashank Maiya, Dhruv Jhamb, Esakkivel Esakkiraja, Katrina Stankiewicz, Gabrielle Gauthier Melan\c{c}on, Fanny Riols, Amit Saha, Lakshmi Sirisha Chodisetty, Nitin Aggarwal, Anil Madamala, Sagar Davasam, Segan Subramanian, Vikas Yadav, Sai Rajeswar Mudumba