The paper introduces StressSim, an intent‑preserving adversarial user simulator designed to stress‑test multi‑turn agents by generating challenging yet valid user interactions. StressSim optimises high‑level adversarial plans for each task instance using a genetic prompt‑optimisation loop that leverages rollout traces, task‑completion signals, and validity feedback to reduce task success while keeping the user’s goal unchanged. Experiments on MultiWOZ, retail, and airline domains show that StressSim reveals significant robustness gaps in leading LLM agents—reducing tau‑bench pass^4 scores by up to 69%—while maintaining realistic conversation quality and goal preservation with a modest optimisation budget.
By Atri Vivek Sharma, Brian Formento, Alessio Lomuscio
arXiv:2606.18902v3 Announce Type: replace
Abstract: Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients d...
By Ziyi Zhu, Luka Smyth, Saki Shinoda, Jinghong Chen
arXiv:2607. 12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality.
By Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran
Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients do not function as real gradients motivates treating automatic prompt optimization (APO) as black-box search.
arXiv:2608. 06329v1 Announce Type: cross Abstract: Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed.
By Noam Koren, Roy Bar-Haim, Abigail Goldsteen
arXiv:2611.07123v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, how...
By Sven Kleinkop, Tam\'as Janusko, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, J\"urgen Anke, Maik Thiele