Scalable Context Orchestration for Serving LLMs Over Voice presents llmovoice, a middleware that explicitly models voice context—including speaking rate, background noise, and packet loss—to guide large language model responses. By constructing a bounded voice context at each turn, llmovoice improves alignment with user preferences and reduces errors, achieving a 52.4% drop in speaking‑rate alignment error and a 0.9% false‑interruption rate under packet loss. In addition, it cuts model usage costs dramatically, lowering per‑turn cost by up to 24.9× while maintaining 98.7% of baseline answer quality in long sessions.
The paper introduces llmovoice, a middleware that explicitly models voice context for large language model (LLM) serving in voice AI applications. By incorporating speaking rate, background noise, packet loss, and other paralinguistic factors into a bounded context, llmovoice guides the LLM to generate more aligned responses. Experiments show significant reductions in speaking‑rate errors, false interruptions, and model usage costs, especially in long voice sessions.
By Linyi Jiang, Silvery D. Fu, Yifei Zhu
The paper introduces UserLM‑2.0, a reinforcement‑learning based user simulator that balances task intent adherence with conversational naturalness. It achieves this by jointly rewarding both objectives and training a lightweight discriminator in an adversarial loop to measure naturalness. The resulting model generates valid simulated conversations 46.2% of the time, more than triple the success rate of the best prior simulator.
By Mustafa Omer Gul, Philippe Laban, Tanya Goyal, Jennifer Neville
The paper introduces a reproducible framework that converts existing text-based tool‑calling benchmarks into audio‑based evaluations using text‑to‑speech, speaker variation, and environmental noise, while preserving original annotations. Experiments on seven omni‑modal models show that performance varies strongly by model and task, with notable gaps between text and voice results. The study also includes a stress test, a reference‑free LLM‑as‑judge protocol, and demonstrates that open‑source judges can match proprietary ones, aiding privacy‑preserving evaluation.
By Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN
arXiv:2609.38867v1 Announce Type: new
Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular in...
By Terumi Chiba, Guangzhi Sun, Zheqi Yuan, Chao Zhang
The paper introduces StressSim, an intent‑preserving adversarial user simulator designed to stress‑test multi‑turn agents by generating challenging yet valid user interactions. StressSim optimises high‑level adversarial plans for each task instance using a genetic prompt‑optimisation loop that leverages rollout traces, task‑completion signals, and validity feedback to reduce task success while keeping the user’s goal unchanged. Experiments on MultiWOZ, retail, and airline domains show that StressSim reveals significant robustness gaps in leading LLM agents—reducing tau‑bench pass^4 scores by up to 69%—while maintaining realistic conversation quality and goal preservation with a modest optimisation budget.
By Atri Vivek Sharma, Brian Formento, Alessio Lomuscio
arXiv:2603.16783v2 Announce Type: replace
Abstract: Robust voice agents require exposure to the full diversity of how people interact through speech. However, obtaining enough spoken interactions is...
By Jonggeun Lee, Junseong Pyo, Jeongmin Park, Yohan Jo
arXiv:2605.13841v3 Announce Type: replace-cross
Abstract: Voice agents are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses realistic conversatio...
By Tara Bogavelli, Gabrielle Gauthier Melan\c{c}on, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, Srinivas Sunkara
arXiv:2601. 02896v3 Announce Type: replace Abstract: Controlling emergent behavioral personas (e.
By Harshvardhan Saini, Yiming Tang, Dianbo Liu
arXiv:2609.20995v1 Announce Type: cross
Abstract: Natural spoken interaction requires more than streaming ASR, language generation, and speech synthesis: a system must react to overlap without cancel...
By Bertil Braun
arXiv:2608.27783v3 Announce Type: replace-cross
Abstract: Speech language models (speech LLMs) can generate plausible outputs from audio that contains no usable speech evidence. We study this failure...
By Mengzhe Geng
arXiv:2608.22731v1 Announce Type: new
Abstract: Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate...
By Parisa Ghanad Torshizi, Stacy Marsella