arXiv AI September 2, 2026 By Yao Liu, Yu He

Disclosure-Gated User Simulation for Companion-Agent Evaluation

The paper introduces a disclosure‑gated user simulator for companion‑agent evaluation, where information release is conditioned on the agent’s behavior through a five‑level gate system. The authors train the simulator on synthetic and real data, audit its performance, and demonstrate that it preserves ranking order and score stability across 12 tested systems. Their released simulator achieves a 0.993 correlation with the original benchmark’s simulator and outperforms prompting a frontier model, which only shifts scores upward without affecting rankings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

Sep 1

Disclosure-Gated User Simulation for Companion-Agent Evaluation

Using a large language model to play the user is now standard in scalable evaluation. It has a repeatedly diagnosed failure: the simulated user is excessively cooperative, so a system under test can s...

llms agents benchmarks

arXiv AI

Sep 30

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

arXiv:2609.38043v1 Announce Type: new Abstract: Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated u...

By Ashish Jain, Armaan Sandhu

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Sep 14

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

GAUGE is a new offline protocol that evaluates whether the common practice of using an LLM-as-a-judge to rank task‑oriented agents actually aligns with a verifiable reward. Across 25 agents from six providers on two benchmarks, GAUGE finds that user satisfaction scores are largely uncorrelated with task success, and that the judge’s ranking loses precision when agents are closely matched in performance. The study highlights a gap between ranking validity and construct validity in current evaluation practices.

By Umesh Bodhwani, Thanh Tran, Kai Wei

llms agents benchmarks

arXiv Machine Learning

Jun 25

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

arXiv:2606. 25760v1 Announce Type: new Abstract: Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions.

By Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios, Nilesh Ahuja, Omesh Tickoo, Ranganath Krishnan, Amit Ranjan Trivedi

llms agents multimodal benchmarks safety

arXiv AI

Jun 15

OdysSim: Building Foundation Models for Human Behavior Simulation

arXiv:2606. 14199v1 Announce Type: cross Abstract: Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation.

By Xuhui Zhou, Weiwei Sun, Weihua Du, Jiarui Liu, Haojia Sun, Qianou Ma, Tongshuang Wu, Yiming Yang, Maarten Sap

llms reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 31

Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale

arXiv:2607. 28074v1 Announce Type: cross Abstract: Computer-use agents learn from what their actions change, so training one needs applications it can act on, break and reset.

By Yash Pandya, Sahil Gupta, Sarthak Harne, Archana Yadav, Kavyansh Chourasia, Hussein Mozannar, Vibhav Vineet, Sara Abdali, Corby Rosset, Yash Lara, Ahmed Awadallah, Ece Kamar, Akshay Nambi

agents benchmarks

One email a morning, machine-written

Related stories

Disclosure-Gated User Simulation for Companion-Agent Evaluation

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

OdysSim: Building Foundation Models for Human Behavior Simulation

Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale