arXiv:2609.40097v1 Announce Type: new
Abstract: Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and...
By Ruifeng Yuan, Yizhi Li, Yaxin Du, Fengyu Cai, Yiqi Liu, Hou Pong Chan, Chenghua Lin, Yun Chen, Jian Yang, Bryan Dai, Pinyan Lu, Chenghao Xiao
MLExpert is a 35B Mixture-of-Experts agentic model designed to autonomously perform long‑horizon machine learning engineering (MLE) tasks. It uses a post‑training method that combines hierarchical agentic supervised fine‑tuning with Windowed Interaction RL to learn from intermediate outcomes in complex experimentation workflows. On the MLE‑Bench Lite benchmark, MLExpert raises the average medal rate from 37.9% to 72.7%, matching the performance of larger frontier models and demonstrating generalizable optimization capabilities across related tasks.
By Shangheng Du, Jinxin Shi, Shuaiyu Zhang, Yiming Wu, Junpeng Zhao, Shiyang Feng, Xin Li, Liang He, Xiangchao Yan, Bo Zhang
EvalResearchBench (ERB) is a benchmark that tests whether AI agents can autonomously design evaluation tasks for recursive self‑improvement. In ERB, a researcher agent selects or synthesizes tasks, builds graders, and refines them through pilot tests before producing a final evaluator that runs within fixed time and API budgets. Experiments with nine researchers and thirteen candidate models show that the best automated evaluators achieve about 75% concordance with target benchmarks, below the 91% ceiling set by target disagreements, and that human‑designed task samples still outperform many automated approaches.
By Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang
MLExpert is a 35B Mixture-of-Experts agentic model designed to autonomously conduct long‑horizon machine learning engineering (MLE) tasks. It uses a post‑training approach that combines hierarchical supervised fine‑tuning with Windowed Interaction RL to learn from intermediate outcomes in complex experimentation workflows. On the MLE‑Bench Lite benchmark, MLExpert raises the average medal rate from 37.9% to 72.7%, matching the performance of larger frontier models and demonstrating transferable optimization capabilities to other benchmarks such as NatureBench Lite.
The paper investigates whether large language models (LLMs) can anticipate the outcomes of machine‑learning experiments they generate. Using a dataset of 288 LLM‑proposed ideas across eight MLE‑bench tasks, the authors prompt five state‑of‑the‑art LLMs to predict each idea’s impact before training. Results show that while predictions outperform random chance, accuracy is limited—Claude Opus 5.5 explains only about half the variance (R² = 0.49), all models underestimate effect sizes, and performance varies by task, with coding harness choice having little influence.
By Ayush Bansal, Aldo Pareja, Sean Klein, Serge Didenko, Artem Lukoianov
The paper introduces SAEScientist-Bench, a benchmark that tests whether AI agents can autonomously conduct mechanistic interpretability research using Sparse Autoencoders (SAEs). Agents are tasked with designing contrastive probes and navigating a large feature dictionary in Gemma-2-9B-IT to identify optimal features for a target concept, with performance measured against expert-curated references on activation rank, concept selectivity, and causal steering. Results show that while frontier agents can discover features and outperform controls, they still lag behind expert baselines, especially in causal steering, highlighting both the potential and current limitations of closed-loop autonomous AI research.
By Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu