The paper introduces SAEScientist-Bench, a benchmark that tests whether AI agents can autonomously conduct mechanistic interpretability research using Sparse Autoencoders (SAEs). Agents are tasked with designing contrastive probes and navigating a large feature dictionary in Gemma-2-9B-IT to identify optimal features for a target concept, with performance measured against expert-curated references on activation rank, concept selectivity, and causal steering. Results show that while frontier agents can discover features and outperform controls, they still lag behind expert baselines, especially in causal steering, highlighting both the potential and current limitations of closed-loop autonomous AI research.
By Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu
The paper introduces AIDE^2, an AI research agent that recursively improves its own code by proposing, benchmarking, and selecting modifications. Over an eight‑day autonomous run, it achieved seven successive improvements—including new search policies and memory mechanisms—that transferred to four held‑out benchmarks in machine learning, algorithm engineering, and weather forecasting. The agent’s best version matched or outperformed a top human‑engineered production research agent and also reduced reward‑hacking rates, despite never optimizing for that metric.
By Dhruv Srikanth, Bingchen Zhao, Dixing Xu, Yuxiang Wu, Zhengyao Jiang
arXiv:2606. 26448v1 Announce Type: cross Abstract: Across the sciences, autonomous systems are increasingly being used in closed-loop discovery, proposing new theories and designing and running experiments to test them.
By Akshay K. Jagadish, Younes Strittmatter, Nori Jacoby, George Kachergis, Eric Schulz, Nathaniel Daw, Suyog H. Chandramouli, Thomas L. Griffiths
The paper introduces ORDER, a fictitious-world benchmark designed to evaluate domain-adaptive embodied AI. ORDER consists of a synthetic 342,069-token corpus defining a self-consistent physics, a 500-question knowledge test (ORDER‑BENCH), and a compositional spatial task (ORDER‑SPATIAL) that requires ordering objects for safe manipulation. The benchmark demonstrates that models like GPT‑4.1 perform poorly without adaptation, while small models improve significantly after continual pre‑training, and that performance on ORDER‑SPATIAL better predicts real plan quality than knowledge-test accuracy.
By Sai Krishna Reddy Sathi, Anuj Tiwari
The paper investigates the limitations of post-training AI agents that can autonomously train large language models. It distinguishes between execution-level capability—making adjustments within a chosen training strategy—and strategy-level capability—revising the overall approach based on new evidence. Analysis of many public post-training runs shows that agents lock into a strategy early and then only perform local tweaks, regardless of task. Experiments with experience scaffolds, human guidance, and extra compute improve execution but do not enable strategy reevaluation, indicating that agents lack a mechanism to spontaneously reassess their strategy during training.
By Joy Jia Yin Lim, Xin Huang, Hao Peng, Yaxi Lu, Xin Cong, Zhong Zhang, Maosong Sun, Yankai Lin
The paper surveys 160 benchmarks from 2017‑2026 that evaluate predictive embodied intelligence, categorising them into policy suites, embodied agents, world‑model evaluation, and prediction‑to‑action bridges. It finds that most benchmarks are model‑agnostic, rarely compare Vision‑Language‑Action policies to world models, and seldom turn predictions into executed actions. The authors argue that the lack of benchmarks designed to directly test the closed‑loop advantage of world models prevents the field from answering whether such models truly improve robotic performance.
By Gaytri Jena, Kapil Wanaskar, Vinija Jain, Aman Chadha, Vasu Sharma, Amitava Das
arXiv:2604. 00830v3 Announce Type: replace-cross Abstract: Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time.
By Zhanzhi Lou, Hui Chen, Yibo Li, Qian Wang, Bryan Hooi
arXiv:2607. 22682v1 Announce Type: new Abstract: We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare.
By Bardiya Akhbari
arXiv:2609.08003v1 Announce Type: new
Abstract: Behavioral foundation models have been proposed as stand-ins for human participants across settings, but it is unclear whether theories discovered on t...
By Akshay K. Jagadish, Younes Strittmatter, Nori Jacoby, Eric Schulz, Nathaniel Daw, Thomas L. Griffiths, Suyog H. Chandramouli
arXiv:2608.26086v3 Announce Type: replace-cross
Abstract: Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their ow...
By Jiarui Yan, Weiwei Sun, Sijie Li, Wenhan Li, Yiming Yang
The paper introduces Auto‑Robotist, a self‑evolving large language model (LLM) agent that transforms evolutionary robot design search traces into an explicit natural‑language skill library. Each skill records a structural archetype, evidence‑grounded rules, and supporting designs, enabling the agent to retrieve and condition LLM edits during search while still using a genetic algorithm for exploration. Experiments on seven EvoGym tasks show that Auto‑Robotist outperforms standard genetic algorithms, especially when transferring learned skills to larger design spaces.
By Yunfei Wang, Xiaohao Xu, Yang Li, Xiaonan Huang
The paper introduces Procedural Graphs, a framework that structures procedural knowledge for large language model agents as (procedure, relation, procedure) triplets, analogous to knowledge graphs for factual data. At each decision point, a guidance model uses the local subgraph to bias the agent’s next action, while an LLM refiner self‑evolves the graph by comparing failed and successful trajectories, editing its topology to improve performance. Experiments across various datasets, tasks, and LLMs show that Procedural Graphs consistently outperform memory‑based baselines, and the self‑evolution mechanism further enhances results without manual engineering.
By Yuxing Lu, Yicheng Chen, Shanchan Wu, Sercan \"{O}. Ar{\i}k