← Back to all news
arXiv AI By Brian Ko, Han Wang, Hejia Geng, Li Erran Li, Sherry Yang

MLE-World: Predicting Machine Learning Experiment Outcomes with a Learned World Model

Read the original on arXiv AI →

MLE-World is a world model fine‑tuned from Qwen3.5‑4B that predicts the terminal output of a machine learning program—such as error traces, training logs, and validation scores—directly from its code and data description, without executing the program. It also provides a confidence score indicating the likelihood that the prediction is correct. Using these predictions, the authors design five test‑time‑scaling policies that compare multiple candidate programs per step, executing only those with low confidence, and demonstrate that MLE‑World matches real outputs on 77.9% of held‑out programs and improves agent performance by up to 22.1% with fewer real executions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Oct 9

DataSense-Bench: The First Step Toward an AI Scientist

DataSense-Bench is a new benchmark that tests whether advanced AI models can choose and rank training data subsets that will improve a small language model’s performance. The benchmark lets agents inspect data, run analysis code, and perform forward passes, but not train the model or access evaluation tasks. After agents select subsets, the base model is fine‑tuned on each subset and evaluated, measuring both the performance of the best subset and the accuracy of the agents’ rankings. Experiments show limited gains over random selection and inconsistent ranking ability across tasks, with agents often using similar data signals but interpreting their value differently.

By Yudi Zhang, Mingyu Cao, Lu Yin, Mykola Pechenizkiy, Shiwei Liu
llmsagentsfine-tuningbenchmarks
More like this →
arXiv AI
Oct 16

MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering

MLExpert is a 35B Mixture-of-Experts agentic model designed to autonomously perform long‑horizon machine learning engineering (MLE) tasks. It uses a post‑training method that combines hierarchical agentic supervised fine‑tuning with Windowed Interaction RL to learn from intermediate outcomes in complex experimentation workflows. On the MLE‑Bench Lite benchmark, MLExpert raises the average medal rate from 37.9% to 72.7%, matching the performance of larger frontier models and demonstrating generalizable optimization capabilities across related tasks.

By Shangheng Du, Jinxin Shi, Shuaiyu Zhang, Yiming Wu, Junpeng Zhao, Shiyang Feng, Xin Li, Liang He, Xiangchao Yan, Bo Zhang
llmsagentsreinforcement-learningfine-tuning
More like this →
arXiv AI
Oct 19

Can LLMs predict the outcomes of machine learning experiments?

The paper investigates whether large language models (LLMs) can anticipate the outcomes of machine‑learning experiments they generate. Using a dataset of 288 LLM‑proposed ideas across eight MLE‑bench tasks, the authors prompt five state‑of‑the‑art LLMs to predict each idea’s impact before training. Results show that while predictions outperform random chance, accuracy is limited—Claude Opus 5.5 explains only about half the variance (R² = 0.49), all models underestimate effect sizes, and performance varies by task, with coding harness choice having little influence.

By Ayush Bansal, Aldo Pareja, Sean Klein, Serge Didenko, Artem Lukoianov
llms
More like this →
arXiv AI
Jul 3

PACE: A Proxy for Agentic Capability Evaluation

arXiv:2607. 02032v1 Announce Type: new Abstract: Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure.

By Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig
llmsagentsbenchmarks
More like this →