The paper proposes that self‑evolving large‑language‑model agents can only be safely reviewed if their changes are confined to the runtime harness—instruction text, tool‑call logic, and primitive composition—while the model weights remain fixed. A dual‑loop engine with an admission gate records a hash‑chained diff before deployment, and simulation experiments show that this gate admits only safe changes, preventing error increases and maintaining false‑positive rates. When the gate is removed, many harmful changes slip through and missed flags rise, illustrating the gate’s protective role.
By Ravil Akhtyamov
arXiv:2609.38512v1 Announce Type: new
Abstract: Voice agents in production must handle several requests, background speech, and customers who lose patience. We introduce VAmoS Energy, a benchmark tha...
By Joshua Meyer, Sahar Shayegan, Ritiz Tambi, Ali Khan, Sun Kim, Victor Shih, Mehdi Jamei, Andi Partovi
arXiv:2610.25458v2 Announce Type: replace
Abstract: Small language models have improved to the point where a phone can plausibly screen its own calls, answering on an absent owner's behalf without se...
By Jiaqi Gan, Tsang Ng, Ankit Raj, Tommy Duong, Xingyu Shen, Kidus Zewde, Huaiyu Guo, Simiao Ren
Self-improving LLM agents can adapt a credit pipeline to a changed rule, but an agent that rewrites itself destroys the artefact a supervisor reviews: a named change, a recorded test, an approval. We...
BENCHCOMPASS is a new payment‑domain benchmark that transforms typed evidence packs into scenario‑grounded tasks, applies LLM‑based quality checks, generates attack variants, and reserves final item admission for domain experts. It includes an expert‑reviewed Pro benchmark covering payment knowledge, context‑grounded scenario reasoning, and attacked open robustness, plus a lower‑assurance Normal pool. Across 16 model variants, BENCHCOMPASS reveals distinct failure modes—missing payment knowledge, incomplete reasoning, and failure to reject invalid workflows—while the best model scores 89.6% on open context‑grounded reasoning and 81.7% under attacked inputs.
"whyItMatters":"The benchmark provides a structured way to isolate and evaluate specific weaknesses in LLMs for payment operations, a critical financial infrastructure where rules change rapidly and decisions depend on complex contextual factors."
By Sijie Dong, Wei Ren, Xuanwei Hu, Jiawei Luo, Zifan Wang, Xiaoyun Feng, Hui Cai, Lyuxin Xue, Peng Lu, Jianshe Li, Xin Zhang, Wei Wu
arXiv:2607. 08028v1 Announce Type: new Abstract: Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context.
By Joongho Ahn, Moonsoo Kim
TauIndianBankBench is a new benchmark that evaluates conversational agents on 1,000 Indian retail‑banking tasks by checking not only the final records but also whether every required procedural step was performed and no unrequested changes were made. It highlights that models can appear successful when judged only on final records, yet may skip necessary steps or alter records incorrectly. The benchmark provides a more accurate assessment of language model performance in the Indian banking domain.
By Krishanu Adhikary, Prashant Devadiga
The paper argues that large language model agents should not be the sole authority on whether they have satisfied a task. It identifies two gaps—understanding–execution and state–authority—where agents may claim completion without actually meeting the specification. The authors propose SpecHarness, a framework that separates agent proposals from authoritative state by requiring evidence from qualified providers to confirm compliance, and demonstrate its effectiveness on guideline‑following and artifact‑generation tasks.
By Haiqing Li, Xin Ma, Yinhao Wu, Wenliang Zhong, Feng Jiang, Thao M. Dang, Xiao Hu, Hehuan Ma, Yuzhi Guo, Junzhou Huang
The paper introduces a hypothesis-driven simulation workflow that screens customer experience (CX) agents before deployment, using synthetic customers and simulated tool outputs to emulate multi-step interactions without accessing production backends. Applied to Nubank’s high-volume Card Delivery and Card Management chat-support agents, the simulation’s binary evaluator scores correlated strongly with production results, and simulation-guided iterations raised transactional net promoter score by 36.69 points in a live A/B test. Additionally, screening over 16,000 simulated conversations helped select a model that increased self‑service rate by 8.82 percentage points without harming net promoter score, demonstrating that simulation enables extensive model exploration safely.
By Edesio Alcoba, Kevin Rossell, Aman Gupta, Shao Tang, Jiwoo Hong, Pabel Carrillo-Mendoza, Wanderson Concei\c{c}\~ao Ferreira, Alvaro Tedeschi, Zayd Simjee, Shreya Rajpal, Bruno Finardi Hime, Christian Sousa, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath
arXiv:2610.14346v2 Announce Type: replace
Abstract: Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this proble...
By Md Tahmid Rahman Laskar, Gundeep Singh, Xue-Yong Fu, Karol Chang, Kevin Sanders, Shi Zong, Tania Habib, Julien Bouvier Tremblay, Shayna Gardiner, Harsh Saini, Matthias Lee, Elena Khasanova, Quinten McNamara, Shashi Bhushan TN
arXiv:2607. 25398v1 Announce Type: new Abstract: Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows.
By Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen
arXiv:2609.22476v1 Announce Type: cross
Abstract: Transmission system operators face rising complexity from renewable integration, reduced inertia, and tighter security margins. Large language models...
By Costas Mylonas, Magda Foti, Emmanouel Varvarigos