arXiv Machine Learning By Xi Chen, Chongjian Ge, Yiran Xu, Yiqun Mei, Henrik I. Christensen, Hao Tan, Ziwen Chen, Hanwen Jiang

Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

Read the original on arXiv Machine Learning →

Auto-Research Arena is a multi-target benchmark designed to evaluate autonomous research systems and uncover new ideas in language modeling. It separates admissibility—determined by model quality and resource constraints—from improvement, using nine diverse targets covering model, data, training, and inference efficiency. The benchmark, applied to language model pretraining, distinguishes existing research harnesses and reveals significant room for improvement, while also enabling the reuse of discovered ideas to achieve lower training loss and better held‑out performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Aug 19

AutoResearch: Insight In, Hallucination Out

AutoResearch is a two‑stage autonomous research system that links Idea Generation with Idea Execution. In the generation phase it blends new research signals with existing domain knowledge, identifies transferable mechanistic insights, and produces grounded, testable research plans through multi‑model generation and cross‑review. The execution phase then decomposes these plans into experiments, iteratively implements and diagnoses them, and uses independent evidence‑based review to accept or revise conclusions, thereby turning ideas into measurable progress while minimizing hallucinations.

By Yiming Ren, Xiang Liu, Qumeng Sun, Xiao Zhang, Jiahao Li, Haoyang Zhang, Junjie Wang
arXiv Computation and Language
Oct 9

Language Models as AI Research World Models

The paper proposes using language models as Research World Models (RWMs) to predict outcomes of experimental interventions across diverse research environments. By training on over 2,600 experimental records and 171,000 GPU‑hours, RWMs improve prediction accuracy within and across environments, reducing selection regret by up to 78% and boosting gains in multi‑round autoresearch. Ablation studies show that incorporating research knowledge yields greater benefits than merely changing the language model or increasing reasoning effort.

By Zijun Wang, Zewen Liu, Minhua Lin, Zhaotian Weng, Zhan Shi, Bing He, Yisi Sang, Dakuo Wang, Benoit Dumoulin, Wei Jin, Yuyin Zhou, Cihang Xie, Hanqing Lu
Hugging Face Trending Papers
Jun 3

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts. Yet existing benchmarks for frontier models primarily evaluate either single-turn responses or short-horizon agent trajectories, failing to capture the challenges of sustained iterative improvement over extended time horizons.

arXiv AI
Oct 19

Language Models Learn to Improve Language Models

The paper introduces LM2LM, a method that trains large language models (LLMs) to propose improvements for other LLMs using reinforcement learning (RL). It employs a frozen research world model (RWM) to predict the gains of proposals, providing a cheaper surrogate reward, and periodically validates high‑scoring proposals to keep feedback reliable. Experiments across eight LM development environments show that LM2LM enables a Qwen3.8‑27B model to outperform GPT‑6‑sol on held‑out tasks while reducing data‑generation cost by 33%, and it generalizes to unseen model families.

By Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang, Suhang Wang, Hanqing Lu
arXiv AI
Sep 7

AutoLR: Automating the Path from Research to Launch Review in Industrial Recommender Systems

AutoLR is an autonomous harness designed to streamline the iterative research‑and‑engineering cycle for industrial recommender systems, exemplified by NetEase’s gaming‑community app DASHEN. It integrates a multi‑expert council for adversarial review, a deterministic evidence‑weighted selector to allocate trial budgets, and a layered knowledge system that fuses external research with domain‑specific insights and empirical evidence. Large language model agents handle semantic reasoning and code generation, while deterministic controllers maintain control over execution, metrics, guardrails, and state management.

By Qi Zhang, Yanlin Chen, Wenchao Xiao
arXiv AI
Jun 4

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

arXiv:2606. 05080v1 Announce Type: new Abstract: Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts.

By Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, Zichen Chen