arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular in...
arXiv:2608.28916v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems are commonly evaluated with word error rate (WER), yet many voice workflows depend on exact written values f...
arXiv:2608. 10716v1 Announce Type: cross Abstract: Speech-to-speech (S2S) voice agents are increasingly being incorporated into enterprise for customer care and as daily companions for consumers owing to the ease of the conversational modality over text.
arXiv:2609.35820v1 Announce Type: cross Abstract: English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce $\tau$-Multilingual, extending $\tau$-Voice to Spanish, Braz...
The paper introduces a reproducible framework that converts existing text-based tool‑calling benchmarks into audio‑based evaluations using text‑to‑speech, speaker variation, and environmental noise, while preserving original annotations. Experiments on seven omni‑modal models show that performance varies strongly by model and task, with notable gaps between text and voice results. The study also includes a stress test, a reference‑free LLM‑as‑judge protocol, and demonstrates that open‑source judges can match proprietary ones, aiding privacy‑preserving evaluation.
The paper introduces Inquesto Score (IS), a protocol that measures voice‑agent reliability by calculating the percentage of calls that reach the caller’s goal without functional failure. IS defines explicit failure events and severity levels, evaluates timing, semantic, and state‑dependent failures using audio, scenario predicates, tool traces, and a pinned open‑model judge, and provides diagnostic views on behavior, acoustic robustness, identity handling, and speaker groups. The authors evaluate IS v0.1 on 30 scenarios, three acoustic conditions, four speaker groups, and 306 calls per agent across 13 configurations, demonstrating that reliable measurement requires evidence beyond transcripts and explicit treatment of deployment conditions.
The paper reviews the fragmented literature on real‑time voice agents, noting that architecture, turn‑taking, and agentic evaluation communities rarely cite each other. It presents three evidence‑based claims: (1) architecture choice is a deployment constraint rather than a definitive solution, (2) evaluation has shifted from component quality to grounded outcomes, and (3) the dyadic assumption is breaking down as multiparty turn‑taking and reasoning become essential. The authors propose the TRG reporting standard to characterize agents by timing, recovery, and state‑verified outcomes, with an optional fourth axis for multiparty contexts.
CallScreenBench is a benchmark for evaluating small, on-device language models that act as phone secretaries, focusing on their ability to handle unknown inbound calls without a cooperative task. The benchmark measures owner endorsement through five call-and-note metrics, each paired with counter-metrics and uncertainty estimates, and includes guardedness diagnostics to identify safe, tool‑free proxies. Results across 4‑bit checkpoints of 0.6‑4 B parameter models show varying performance on service, recall, plausibility, and triage discrimination, highlighting trade‑offs between quality and guardedness.
The paper introduces a three‑stage pipeline to improve accented conversational ASR for speakers from India, Indonesia, and Latin America. It uses heuristic SQL filters to curate entity‑rich training data, regional LoRA adapters fine‑tuned on Qwen2.5‑Omni‑3B to generate both verbatim and corrected transcripts, and a six‑category error taxonomy validated by an LLM judge. The approach raises entity recall to 80‑85% and filler recall to 76‑86%, while keeping WER low (6‑10%) and outperforming Whisper and a commercial ASR on entity recall.
arXiv:2610.34653v1 Announce Type: new Abstract: Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete de...
arXiv:2603.16783v2 Announce Type: replace Abstract: Robust voice agents require exposure to the full diversity of how people interact through speech. However, obtaining enough spoken interactions is...
arXiv:2610.24809v2 Announce Type: replace Abstract: Speech recognition in real time in far-field conversational settings still incurs significant errors. Identifying potential error spans during stre...