The paper introduces StressSim, an intent‑preserving adversarial user simulator designed to stress‑test multi‑turn agents by generating challenging yet valid user interactions. StressSim optimises high‑level adversarial plans for each task instance using a genetic prompt‑optimisation loop that leverages rollout traces, task‑completion signals, and validity feedback to reduce task success while keeping the user’s goal unchanged. Experiments on MultiWOZ, retail, and airline domains show that StressSim reveals significant robustness gaps in leading LLM agents—reducing tau‑bench pass^4 scores by up to 69%—while maintaining realistic conversation quality and goal preservation with a modest optimisation budget.
By Atri Vivek Sharma, Brian Formento, Alessio Lomuscio
arXiv:2606.18902v3 Announce Type: replace
Abstract: Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients d...
By Ziyi Zhu, Luka Smyth, Saki Shinoda, Jinghong Chen
arXiv:2607. 12085v1 Announce Type: new Abstract: Evaluating retail conversational agents requires methods beyond lexical-overlap metrics to assess intent alignment, factuality, helpfulness, clarity, tone, and overall response quality.
By Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran
Context engineering has emerged as a primary lever for improving AI systems without parameter updates. Recent work showing that textual gradients do not function as real gradients motivates treating automatic prompt optimization (APO) as black-box search.
arXiv:2608. 06329v1 Announce Type: cross Abstract: Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed.
By Noam Koren, Roy Bar-Haim, Abigail Goldsteen
arXiv:2611.07123v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, how...
By Sven Kleinkop, Tam\'as Janusko, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, J\"urgen Anke, Maik Thiele
arXiv:2603. 18388v2 Announce Type: replace Abstract: Automatic prompt optimization (APO) has emerged as a powerful paradigm for improving LLM performance without manual prompt engineering.
By Shiyan Liu, Qifeng Xia, Qiyun Xia, Yisheng Liu, Xinyu Yu, Rui Qu
arXiv:2608. 01366v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are integral to complex intellectual tasks, yet output quality remains constrained by user-provided prompts.
By B. Sankar, Pawni Yadav, Srinidhi Ranjini Girish, Amogh A. S
GEPA-Curator is a framework that enhances prompt optimization for large language model agents by selecting which agent experiences should inform the optimizer. It includes task curation to filter out noisy or unreliable tasks and prior success retrieval to provide successful trajectories as context. Experiments on GoBrowse, ALFWorld, AppWorld, and τ‑bench show that GEPA-Curator matches or surpasses baseline performance, achieving up to 20.7% improvement over GEPA and 21.2% over ACE.
By Zack Yu, Nicholas Lee, Minjae Lee, Lakshya Agarwal, Luca Manolache, Wonjun Kang, Matei Zaharia, Michael W. Mahoney, Kurt Keutzer, Sehoon Kim, Amir Gholami
arXiv:2608. 09939v1 Announce Type: cross Abstract: Production teams deploying LLM chat agents face a specific quality assurance gap: existing evaluation tools test individual responses or simulate social interactions, but none systematically verify whether real users can achieve their goals through multi-turn conversation.
By Alexandre Cristov\~ao Maiorano
arXiv:2505.13546v2 Announce Type: replace-cross
Abstract: Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Exist...
By Ke Chen, Xucheng Yu, Yufei Zhou, Haohan Wang
RL-ADA introduces a co‑evolutionary training framework that replaces costly human annotations with world‑feedback rewards derived from interaction outcomes. In this system, a large Customer Support Agent and an Adversarial Customer Agent train together, guided by an automated judge that rewards successful resolution and realistic intent‑concealing utterances, respectively. Applied to a banking support proof of concept, the method eliminates routing errors and doubles the end‑to‑end PASS rate over five cycles, while also revealing a new adversarial strategy called Contextual Camouflage.
By Ram Narayanan, Harshit Rajgarhia, Abhishek Mukherji