HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The paper investigates how autoresearch agents—AI systems that conduct research autonomously—benefit from interacting with human researchers, a process called co‑improvement. It presents empirical evidence that humans remain crucial during the ideation phase of benchmark creation, introduces HackBench to expose alignment gaps in current agents, and proposes two training methods: using human simulators to enhance agent interaction and applying reinforcement learning with expert‑aligned rubrics to improve communication. These approaches aim to make autoresearch agents more capable and aligned with human expectations.
The paper introduces Test-Time Adaptation through Human‑Agent Interaction (TAHI), a method that leverages iterative human feedback to tailor AI agents to individual users. By integrating interaction data into agent context and weighting, and by developing an evolving rubric module, the authors adapt agents to 30 users across writing and visual creation tasks, achieving 4.5–20.9% higher solo task success after only a few interactions. The rubric module also serves as a scalable annotation tool, detecting 16.0–22.3% more failures than language models or humans alone, while personalized agents can generalize improvements up to 8.8% across users.
The paper introduces Test-Time Adaptation through Human‑Agent Interaction (TAHI), a method that uses iterative human feedback to adapt AI agents to individual users’ criteria. By integrating cross‑session interaction data into agent context and weights, and building an evolving rubric module, the authors demonstrate that agents can improve task success by 4.5–20.9% after only a few interactions. The evolving rubric also serves as a scalable annotation tool, detecting 16.0–22.3% more failures than language models or humans alone, and personalized agents can even generalize improvements up to 8.8% across users.
arXiv:2607. 16610v1 Announce Type: new Abstract: Long-horizon AI agents are becoming increasingly capable, yet their interaction with users remains surprisingly thin.
arXiv:2608. 14870v1 Announce Type: new Abstract: Long-horizon agents can execute continuously, but human attention remains intermittent and scarce.
arXiv:2608. 01366v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are integral to complex intellectual tasks, yet output quality remains constrained by user-provided prompts.