arXiv AI October 16, 2026 By Hang Zheng, Nan Jiang, Zichen Zhu, Zixu Yang, Zhiyang Tang, Shuai Fan, Lu Chen, Xiaobao Wu, Kai Yu

ARGUS: An Agenda-Driven Framework for Reliable User Simulation

Read the original on arXiv AI →

ARGUS is a neuro‑symbolic, agenda‑driven framework designed to improve the reliability of user simulators in interactive environments. It separates state tracking from natural language generation by using a dynamic agenda stack and symbolic operations to enforce goal progression and constraint‑aligned interactions. The authors distill the model into compact versions via evaluator‑guided SFT and online RL, and evaluate its behavioral fidelity with a new Composite Behavioral Score metric on $ au^3$‑Bench and MultiWOZ, showing significant performance gains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 30

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

arXiv:2609.38043v1 Announce Type: new Abstract: Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated u...

By Ashish Jain, Armaan Sandhu

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 8

Beyond Static Evaluation: Building Simulation Environments for Scalable Agentic Reinforcement Learning

arXiv:2607. 05773v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, traditional static evaluation fails to capture multi-step decision-making.

By Akshay Arora, Ishan Nigam, Ashutosh Aggarwal, Shefali Bansal, Krishna Singh, Sweta Kumari, Nikhil Mittal, Shariq Farhan, Siddarth Malreddy

llms agents reinforcement-learning

arXiv Computation and Language

Oct 8

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

arXiv:2605.12894v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with diverse users, including those who are uncle...

By Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, Natasha Jaques

llms agents fine-tuning benchmarks

arXiv Computation and Language

Oct 5

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

The paper introduces Calibrated User Embeddings (CUE), a framework that encodes real user sessions into continuous representations and decodes them into persona commands to steer large language models (LLMs) as user simulators without additional training. CUE enables user-conditioned replay of past interactions and generates novel personas that better align with real-user success rates and failure patterns. Evaluations on the τ²‑Bench dataset show that CUE‑driven simulators commit fewer simulator‑attributed errors, more accurately reproduce real‑user failure modes, and maintain competitive user fidelity across various tasks and LLMs.

By Anjali Kantharuban, Jonas Mueller

llms rag agents benchmarks safety

arXiv AI

Jun 10

STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios

arXiv:2606. 10394v1 Announce Type: new Abstract: Large language models are increasingly used to power personal agents for everyday applications, but evaluating these agents remains a challenge.

By Sirui Liang, Bohan Yu, Peiyu Wang, Shiguang Guo, Wenxing Hu, Pengfei Cao, Jian Zhao, Cao Liu, Ke Zeng, Xunliang Cai, Kang Liu

llms agents benchmarks

arXiv AI

Jun 16

State-Grounded Multi-Agent Synthetic Data Generation for Tool-Augmented LLMs

arXiv:2606. 16307v1 Announce Type: new Abstract: Training tool-augmented LLM agents requires large corpora of multi-turn, tool-grounded conversational data that is expensive to annotate, privacy-constrained in production settings, and largely absent from public datasets.

By Rahul Khedar, Eshita, Sneha Teja Sree Reddy Thondapu, Mayank Malhotra, Arup Das, Jitesh Chandra, Yun-Shiuan Chuang, Chaitanya Kulkarni, Arun Menon, Linsey Pang, Avinash Karn, Mouli V, Prakhar Mehrotra

llms agents safety

One email a morning, machine-written

Related stories

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Beyond Static Evaluation: Building Simulation Environments for Scalable Agentic Reinforcement Learning

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios

State-Grounded Multi-Agent Synthetic Data Generation for Tool-Augmented LLMs