The Flow has not summarised this story yet — read it at arXiv Computation and Language.
MADS (Multi-Agent Dialog Simulation) is a synthetic data generation pipeline that creates structured, diverse dialogue trajectories, including adversarial conversations that reflect realistic failure scenarios. The resulting MADS-Bench benchmark covers six domains, 55 personas, and 1095 conversations, with about 66% featuring failure-prone situations. Models trained with MADS are evaluated using a two-stage approach combining supervised fine-tuning and reinforcement learning with verifiable rewards to assess robustness.
The paper introduces UserLM‑2.0, a reinforcement‑learning based user simulator that balances task intent adherence with conversational naturalness. It achieves this by jointly rewarding both objectives and training a lightweight discriminator in an adversarial loop to measure naturalness. The resulting model generates valid simulated conversations 46.2% of the time, more than triple the success rate of the best prior simulator.
The paper introduces StressSim, an intent‑preserving adversarial user simulator designed to stress‑test multi‑turn agents by generating challenging yet valid user interactions. StressSim optimises high‑level adversarial plans for each task instance using a genetic prompt‑optimisation loop that leverages rollout traces, task‑completion signals, and validity feedback to reduce task success while keeping the user’s goal unchanged. Experiments on MultiWOZ, retail, and airline domains show that StressSim reveals significant robustness gaps in leading LLM agents—reducing tau‑bench pass^4 scores by up to 69%—while maintaining realistic conversation quality and goal preservation with a modest optimisation budget.
arXiv:2609.35812v1 Announce Type: new Abstract: In-car conversational assistants (ICAs) are increasingly integrated into vehicles to support route planning, vehicle control, and information access. E...
The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.
arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when...