The paper formalizes controllable user simulation as a causal inference problem, showing that standard supervised fine‑tuning on post‑hoc trajectory labels introduces a look‑ahead bias that breaks causal consistency. This bias leads to a phenomenon called controllability collapse, where evaluation metric variance explodes under policy shift. The authors propose theoretical conditions for accurate simulation and practical training mitigations—such as a priori controls, step‑wise dynamic controls, and direct policy‑conditioned learning—to eliminate look‑ahead bias, preserve natural variance, and achieve robust zero‑shot generalization to unseen agent behaviors.
arXiv:2609.23753v1 Announce Type: cross Abstract: Generative world models aim to predict future states conditioned on actions, where action controllability is fundamental for reliable dynamics modeli...
The paper introduces OTROPE, a likelihood‑free method for off‑policy evaluation of large language models (LLMs) that uses optimal transport to align labeled samples from a behavior model with unlabeled samples from a target model in a semantic space. OTROPE corrects human‑labeled residuals with proxy predictors, achieving a doubly robust evaluation without requiring behavior‑policy modeling or density‑ratio estimation. The authors provide theoretical guarantees for consistency and convergence, and demonstrate through synthetic and real LLM tasks that OTROPE outperforms existing baselines and can elevate weaker evaluators to match or exceed stronger ones.
The paper introduces UserRepGen-4B, a representation generator that creates natural-language user representations to condition user simulators for more faithful conversation simulation. By formulating a conversation reconstruction task and using hindsight distillation, the system learns to generate representations that improve simulation fidelity across 700 conversations from seven datasets. The improved representations also enhance diversity preservation and user distinction in unseen tasks.
arXiv:2607. 14485v1 Announce Type: new Abstract: Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection.
arXiv:2608. 12253v1 Announce Type: cross Abstract: Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior.
arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when...
The paper investigates how small changes in prompt characteristics, such as demographic attributes, can alter the semantics and tone of large language model outputs. It shows that standard alignment methods like reinforcement learning can worsen counterfactual bias by increasing response quality differences across demographics. To mitigate this, the authors propose a policy‑gradient approach with a group‑level reward that operates on matched counterfactual prompt pairs, and they validate its effectiveness on fairness benchmarks while maintaining overall response quality.
arXiv:2607. 19361v1 Announce Type: cross Abstract: Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm.
Natural Language Actor-Critic (NLAC) is a new actor‑critic algorithm that trains large language model (LLM) agents by using a generative LLM critic to produce natural‑language feedback instead of scalar rewards. By providing richer, actionable explanations for suboptimal actions, NLAC helps LLM policies improve without relying on random exploration, and it can be trained off‑policy without policy gradients. Experiments on reasoning, web browsing, tool‑use, and dialogue tasks show that NLAC outperforms existing training methods and offers a more scalable and stable training paradigm for LLM agents.
The paper introduces UserLM‑2.0, a reinforcement‑learning based user simulator that balances task intent adherence with conversational naturalness. It achieves this by jointly rewarding both objectives and training a lightweight discriminator in an adversarial loop to measure naturalness. The resulting model generates valid simulated conversations 46.2% of the time, more than triple the success rate of the best prior simulator.
arXiv:2601. 02871v3 Announce Type: replace Abstract: Task-oriented proactive dialogue agents play a pivotal role in recruitment, particularly for steering conversations towards specific business outcomes, such as acquiring social-media contacts for private-channel conversion.
MADS (Multi-Agent Dialog Simulation) is a synthetic data generation pipeline that creates structured, diverse dialogue trajectories, including adversarial conversations that reflect realistic failure scenarios. The resulting MADS-Bench benchmark covers six domains, 55 personas, and 1095 conversations, with about 66% featuring failure-prone situations. Models trained with MADS are evaluated using a two-stage approach combining supervised fine-tuning and reinforcement learning with verifiable rewards to assess robustness.