arXiv Machine Learning
Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Oct 8
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
arXiv:2605.12894v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...
arXiv Computation and Language
Oct 5
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.
arXiv AI
Jun 10
Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents
arXiv:2510. 04491v3 Announce Type: replace Abstract: Despite rapid progress in building conversational AI agents, robustness is still largely untested.
arXiv Computation and Language
6d ago
Towards User Simulators That Stay On Task And Still Sound Natural
The paper introduces UserLM‑2.0, a reinforcement‑learning based user simulator that balances task intent adherence with conversational naturalness. It achieves this by jointly rewarding both objectives and training a lightweight discriminator in an adversarial loop to measure naturalness. The resulting model generates valid simulated conversations 46.2% of the time, more than triple the success rate of the best prior simulator.
arXiv Computation and Language
Oct 8
ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation
arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when...
arXiv Computation and Language
Oct 15
$\tau$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions
arXiv:2610.19560v1 Announce Type: cross Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users....