The paper introduces a framework for designing proactive large‑language‑model agents, centered on three joint principles—Task Capability, Temporal Allocation, and Trust—alongside a five‑dimensional design space. It proposes PROACTIVITY‑GYM, a simulation testbed for evaluating proactive assistance across multi‑day scenarios, and presents empirical findings that highlight performance gaps and the importance of aligning interventions with user trust. Human studies show that misaligned interventions can sharply reduce trust, even when outcomes are correct, underscoring the need for careful joint optimization of the 3T principles.
Proactive LLM agents can turn idle compute into useful support before users ask. Yet even correct work can misread user context, impose review costs, or undermine trust. This work proposes foundations...
arXiv:2607. 02975v1 Announce Type: new Abstract: Effective agency in social environments depends on when an agent seeks knowledge, when it acts, and whether its actions are justified by acquired information.
The paper presents a unified framework for proactive service agents, defining proactivity as an agent’s ability to infer service opportunities from incomplete signals and decide whether to remain silent, ask, assist, or act. It models this as a partially observable sequential decision process constrained by authorization and risk, integrating timing, content, and delivery into a single structured action. The authors categorize existing methods along a decision pipeline—state and need estimation, intervention gating, action construction, and feedback adaptation—and propose standardized metrics for evaluating triggering, timing, calibration, user burden, safety, and policy value across diverse interaction modalities.
GAUGE is a new offline protocol that evaluates whether the common practice of using an LLM-as-a-judge to rank task‑oriented agents actually aligns with a verifiable reward. Across 25 agents from six providers on two benchmarks, GAUGE finds that user satisfaction scores are largely uncorrelated with task success, and that the judge’s ranking loses precision when agents are closely matched in performance. The study highlights a gap between ranking validity and construct validity in current evaluation practices.
The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
arXiv:2605. 22240v2 Announce Type: replace Abstract: Proactive task-oriented dialogue (TOD), such as outbound sales, demands a persuasive agent that actively probes the user's concerns and steers the conversation toward acceptance within a bounded number of turns.
The paper argues that deploying generative AI agents requires more than isolated task success; they must remain useful across repeated interactions, changing conditions, and dependencies on people within shared workflows. The authors introduce two complementary evaluation aspects—operational resilience and considerate participation—to assess how agents recover from blocked work, communicate limits, and adapt to affected people and role boundaries. Using 120 simulated healthcare trajectories across two AI models and twelve stakeholder-derived tasks under varying challenge levels, the study finds that agents shift toward greater human dependence and increased workload as challenge accumulates, while also broadening from task-focused adaptation to task reframing and wider coordination.
arXiv:2609.22592v1 Announce Type: cross Abstract: Training agents with reinforcement learning requires a gym, comprising a task, an executable environment in which the task can be attempted, and a ve...
S3Gym is an interactive benchmark designed to evaluate large language models (LLMs) on their ability to self-improve through self-testing, self-judging, and self-improvement. It separates permissive exploration from strict held-out evaluation across seven text-based games with executable environment verifiers. Experiments show that self-improvement varies by task, with different experience incorporation pathways (direct history, summary memory, or parameter training) yielding mixed results and highlighting the need for agents to transform feedback into executable, transferable policies.
arXiv:2606. 02965v1 Announce Type: new Abstract: Benchmarks for autonomous agents measure whether agents complete tasks, yet this framing is systematically blind to whether an agent should have proceeded at all.
arXiv:2609.38043v1 Announce Type: new Abstract: Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated u...
Continually improving agents require dynamic interaction feedback beyond static supervision, yet direct real-environment interaction is costly, slow, unsafe, and hard to parallelize. World modeling offers a natural intermediate proxy that allows agents to query lower-cost, more controllable feedback before committing to real actions.