arXiv AI
Jun 4
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?
arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.
Aligning Autoresearch: Co-Improvement between Agents and Humans
The paper investigates how autoresearch agents—AI systems that conduct research autonomously—benefit from interacting with human researchers, a process called co‑improvement. It presents empirical evidence that humans remain crucial during the ideation phase of benchmark creation, introduces HackBench to expose alignment gaps in current agents, and proposes two training methods: using human simulators to enhance agent interaction and applying reinforcement learning with expert‑aligned rubrics to improve communication. These approaches aim to make autoresearch agents more capable and aligned with human expectations.
Related stories
arXiv AI
Jun 11
Preregistration for Experiments with AI Agents
arXiv:2606. 11217v1 Announce Type: cross Abstract: The proliferation of large language models (LLMs) and autonomous AI agents has given rise to a rapidly growing methodological paradigm: "in silico" behavioral experiments.
arXiv AI
Sep 4
Efficient Test-Time Adaptation through Human-AI Interaction
The paper introduces Test-Time Adaptation through Human‑Agent Interaction (TAHI), a method that uses iterative human feedback to adapt AI agents to individual users’ criteria. By integrating cross‑session interaction data into agent context and weights, and building an evolving rubric module, the authors demonstrate that agents can improve task success by 4.5–20.9% after only a few interactions. The evolving rubric also serves as a scalable annotation tool, detecting 16.0–22.3% more failures than language models or humans alone, and personalized agents can even generalize improvements up to 8.8% across users.
arXiv AI
Jun 8
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
arXiv:2606. 07462v1 Announce Type: new Abstract: As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution.
arXiv Machine Learning
Sep 22
HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
arXiv:2509.08494v2 Announce Type: replace-cross Abstract: As humans delegate more tasks and decisions to artificial intelligence (AI), we risk losing control of our individual and collective futures....
arXiv AI
Jun 3
Toward Training Superintelligent Software Agents through Self-Play SWE-RL
arXiv:2512. 18552v3 Announce Type: replace-cross Abstract: While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.
arXiv AI
Oct 19
EvalResearchBench: Can AI Agents Build Their Own Feedback for Recursive Self-Improvement?
EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.
arXiv AI
Oct 1
Rep2Skill: Representation-Guided Skill Self-Evolution for LLM Agents
Rep2Skill introduces a representation-guided framework that enables large language model agents to self-evolve their textual skills by analyzing internal representation trajectories from agent rollouts. The method identifies execution turns that deviate from successful dynamics and uses these signals, together with execution contexts, as actionable feedback for targeted skill revision. Experiments with two open-source LLMs across two agent environments demonstrate that Rep2Skill consistently outperforms purely text-based approaches, showing that incorporating internal representations can enhance agent self-improvement.
arXiv Machine Learning
Oct 7
Fork-and-Flush: Escaping Idea Basins in Autoresearch Agents
arXiv:2610.07447v1 Announce Type: new Abstract: Autoresearch agents tackle open-ended problems by repeatedly proposing candidate solutions, evaluating them, and using feedback to guide subsequent exp...
arXiv AI
Jul 1
LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
arXiv:2604. 17931v3 Announce Type: replace Abstract: Reinforcement Learning (RL) has emerged as a powerful training paradigm for LLM-based agents.
arXiv AI
Sep 25
IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
IDRBench is a benchmark designed to evaluate the interactive capabilities of deep research agents that use large language models. It introduces controlled opportunities for clarification within a common workflow, comparing autonomous and interactive trajectories by measuring task‑specific report alignment and interaction cost. Experiments on 100 tasks with seven LLMs show that interaction consistently improves alignment, though its effectiveness varies depending on the agents’ questions and feedback integration.
arXiv Computation and Language
Oct 9
Language Models as AI Research World Models
The paper proposes using language models as Research World Models (RWMs) to predict outcomes of experimental interventions across diverse research environments. By training on over 2,600 experimental records and 171,000 GPU‑hours, RWMs improve prediction accuracy within and across environments, reducing selection regret by up to 78% and boosting gains in multi‑round autoresearch. Ablation studies show that incorporating research knowledge yields greater benefits than merely changing the language model or increasing reasoning effort.