The paper introduces Humanize MLE Agents (HMA), a workflow that alternates two agents across sessions on a shared workspace, limiting each session to at most k experiments and resetting context between sessions. This approach, modeled as a context‑reset semi‑Markov decision process, outperforms single‑agent baselines, achieving a 78.2% any‑medal rate on 75 Kaggle MLE‑bench tasks—8.0 points higher than the best open‑source baseline and using only a quarter of its time budget. Across six heterogeneous agent pairs, HMA improves the any‑medal rate by an average of 6.4 points, and a supplementary kernel‑optimization case reaches 878 cycles, within 1.6% of the community leaderboard leader.
By Changye Li, Zijian Zhang, Muheng Li, Yiduo Pang, Tingjun Wu, Menghan Li, Yitong Liu, Hongkai Guo, Mengchi Chen, Chenyu He, Kaisheng Ma, Wei Xu, Song Han, Sihao Liu, Ligeng Zhu
The paper introduces Ask2Decide, a benchmark of 43 planning tasks designed to test whether large language model (LLM) agents can identify and request missing decision‑relevant information from humans. It collects 79 interaction sessions from 19 participants, showing that human replies often omit critical facts, and uses these data to calibrate a simulated operator. Evaluation of four state‑of‑the‑art LLMs reveals that their initial plans are infeasible on 58‑98% of tasks when information is missing, and that agents only slightly outperform random guessing in distinguishing useful from irrelevant facts.
By Yuhan Tang, Ke Fan, Xinkai Zou, Baichuan Mo, Kangxin Cui, Ao Qu, Ou Zheng, Jiangbo Yu, Haris N. Koutsopoulos, Jinhua Zhao, Yuan Zhou
EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.
By Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang
arXiv:2607. 12338v1 Announce Type: new Abstract: Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting.
By Wei-Jung Huang
The paper introduces Online Surrogate Repair (OSR), a closed‑loop algorithm that decouples the frequency of high‑fidelity evaluations from the length of an agent’s search by selectively updating a surrogate model with sparse, high‑fidelity data. An acquisition rule determines which candidate designs receive expensive evaluations, and the resulting labels refine the surrogate for subsequent episodes. Experiments on synthetic environments and the MADE benchmark show that OSR can reduce regret more efficiently than fixed‑surrogate approaches, requiring fewer oracle queries than high‑fidelity feedback after every episode.
By Xiaotang Feng, Philip Torr, Bruno Andreis
ARE‑bench is a new benchmark that tests research agents on real‑world researcher sessions and workspaces, focusing on their ability to advance ongoing projects while respecting prior decisions and existing workspace states. It contains 75 tasks derived from actual researcher projects, covering seven research stages, 33 requirements, and 150 sub‑requirements, with a total of 1,173 binary checklist items. The benchmark shows that even the best frontier models only pass about two‑thirds of the tasks when full contextual requirements are considered, highlighting the difficulty of evaluating agents in realistic settings.
By Jiahang Lin, Wanying Ren, Wanghan Xu, Huayu Sha, Yang Shu, Zhonghang Lu, Chengjun Pan, Yi Dong, Shichun Liu, Yuyang Hu, Shihan Dou, Honglin Guo, Huijie Lv, Pujun Zheng, Zhe Sun, Yuankai Ying, Yiding Wang, Jiahao Chen, Zhixiong Yang, Dingbo Yuan, Yicheng Zou, Zhecan Wang, Guoxiu He, Xiaoyu Xing, Qi Zhang, Tao Gui, Xuanjing Huang