CallScreenBench is a benchmark for evaluating small, on-device language models that act as phone secretaries, focusing on their ability to handle unknown inbound calls without a cooperative task. The benchmark measures owner endorsement through five call-and-note metrics, each paired with counter-metrics and uncertainty estimates, and includes guardedness diagnostics to identify safe, tool‑free proxies. Results across 4‑bit checkpoints of 0.6‑4 B parameter models show varying performance on service, recall, plausibility, and triage discrimination, highlighting trade‑offs between quality and guardedness.
By Jiaqi Gan, Haoyuan Tang, Jamey Z. Liang, Siying Chen, Ankit Raj, Kidus Zewde, Yuchen Zhou, Yuxin Zhang, Simiao Ren
The paper introduces CarryOnBench, an interactive benchmark that tests whether large language models can revise their interpretation of user intent and recover utility while staying safe in multi‑turn conversations. Using 398 harmful‑looking queries with benign intents, the benchmark simulates 5,970 conversations across 14 models, evaluating both intent‑aligned utility and safety with a new metric called Ben‑Util. Results show that models often withhold information due to misinterpretation, but most can recover with clarifications, revealing failure modes such as unsafe and redundant recovery that single‑turn tests miss.
By Mingqian Zheng, Malia Morgan, Liwei Jiang, Carolyn Rose, Maarten Sap
MTVA-Bench is a new benchmark designed to evaluate the language model component of cascaded voice agents under realistic conditions. It simulates callers with an LLM, mocks backend tool responses, and scores both tool‑call correctness and conversational quality using two LLM judges, covering 49 agents, 490 scenarios, and 7 languages. The benchmark reveals that while models perform similarly on tool selection, they differ widely in argument handling, rule compliance, and dialogue quality, highlighting the nuanced challenges of real‑world voice interactions.
By Pritish Mishra, Ishaan Kumar, Akshat Mandoli, Sudarshan Kamath
arXiv:2610.18881v1 Announce Type: cross
Abstract: Voice agents built on large language models (LLMs) now handle customer calls, but in regulated applications completing the task is not enough: the ag...
By Petmongkon Pornpichitsuwan, Krittin Pachtrachai, Touchapon Kraisingkorn
The paper evaluates three approaches for emotion recognition in conversation— a low‑cost stacked ensemble, an off‑the‑shelf LLM prompt, and a confidence‑gated hybrid that escalates only uncertain ensemble predictions to the LLM. Across three datasets (IEMOCAP, MELD, CMU‑MOSI), the hybrid consistently outperforms each pure system, achieving higher weighted F1 scores while routing most traffic through the inexpensive ensemble. This results in significant cost savings (≈$10‑85 per million utterances) and provides an interpretable escalation signal tied to emotion or sentiment shifts.
By Sai Babu Udayagiri, Arjun Chouhan, Ravisekhar Kanagala, Trishala Pavagada
arXiv:2610.32955v1 Announce Type: new
Abstract: Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error r...
By Federico Lucca
arXiv:2610.19659v1 Announce Type: new
Abstract: People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the fait...
By Yilin Jiang, Fei Tan, Xuanyu Yin, Guanchu Wang, Xia Hu, Yutao Yue
SAGE (State‑Grounded, Abstention‑Aware Evaluation) is a new framework for assessing task‑oriented dialogue agents that focuses on whether each turn correctly advances the workflow state rather than just the quality of the reply. It compiles workflow specifications and per‑turn state differences into schema‑grounded criteria, then evaluates them through a cascade of symbolic rules and encoder/NLI verifiers that abstain instead of guessing, producing a turn‑level decision with an evidence trace. In experiments across MultiWOZ, Schema‑Guided Dialogue, and ABCD datasets, SAGE‑Core—using only symbolic rules and on‑device encoders—outperforms all evaluated LLM‑based judges, including GPT‑4.1 variants, while incurring zero paid LLM cost.
By Rayan Khoury, Shih-Yao Lin, Pratyush Mishra
arXiv:2606. 18327v1 Announce Type: cross Abstract: Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users.
By Itamar Pres, Laura Ruis, Melat Ghebreselassie, Belinda Z. Li, Jacob Andreas
The paper introduces TalkMesh, a decentralized network of small language model agents that learn to communicate effectively during inference. Each agent proposes an answer, scores it with a confidence head, and the most confident agent broadcasts a hint; lower‑confidence agents revise their proposals if a new suggestion scores higher. This gossip‑based consensus, trained via group relative policy optimization, enables a mesh of three agents to match the accuracy of majority voting over 32 samples, and scales to larger meshes to significantly boost performance on benchmarks like GSM8K and MATH-500.
By Mehmet Kerem Turkcan
The paper introduces a taxonomy of six user challenge types and a four-layer framework to analyze how large language models respond to user disagreement. Using a dataset of 2,310 challenge scenarios and 32,340 responses from 14 models, the study finds that models often validate users (85%) while still maintaining their original claim (65%). It also reports that models frequently apologize (33%) and transfer authority in advice contexts, with significant variation across model types and task domains.
By Riyadh Alnasser, Yusuf M\"ucahit \c{C}etinkaya, Sumin Zhao, Tu\u{g}rulcan Elmas
The paper investigates whether large language models’ refusal behavior in text transfers to tool‑calling contexts. A benchmark of 2,304 prompts across four domains compares text‑only, standard tool‑enabled, and adversarially framed tool‑enabled settings, revealing that 10–52 % of harmful requests refused in chat still lead to unsafe tool calls. The authors identify a refusal‑associated direction in the models’ residual stream, showing that lower activation along this direction predicts unsafe calls, but restoring the signal only partially mitigates the issue and often suppresses benign tool use.
By Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang, Adam Al Gharib