arXiv AI September 10, 2026 By Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu

SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?

The paper introduces SAEScientist-Bench, a benchmark that tests whether AI agents can autonomously conduct mechanistic interpretability research using Sparse Autoencoders (SAEs). Agents are tasked with designing contrastive probes and navigating a large feature dictionary in Gemma-2-9B-IT to identify optimal features for a target concept, with performance measured against expert-curated references on activation rank, concept selectivity, and causal steering. Results show that while frontier agents can discover features and outperform controls, they still lag behind expert baselines, especially in causal steering, highlighting both the potential and current limitations of closed-loop autonomous AI research.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

3d ago

What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track

arXiv:2609.15064v1 Announce Type: new Abstract: Reinforcement learning (RL) is widely utilized in large language model training to improve targeted capabilities, yet how RL reshapes a model remains p...

By Lingheng Du, Yiming Tang, Xufeng Duan, Dianbo Liu

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

llms agents benchmarks

arXiv AI

Aug 19

ASI-Bench: At the Dawn of Artificial Superintelligence

ASI‑Bench is a new benchmark that evaluates AI systems on their ability to conduct innovative exploration and autonomous scientific research across 11 domains, using 60 project‑level tasks. It progressively removes human methodological guidance to test whether AI can independently select methods, execute research, and produce verifiable results. Results from 18 state‑of‑the‑art agent–model configurations show a sharp performance drop when guidance is reduced, indicating current systems still rely heavily on human input.

By Junwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao, Yongchao Chen, Yuanning Feng, Junhao Wu, Cheng Zhang, Sijia Chen, Haoyu Xue, Chengsong You, Huan Wang, Koutian Wu, Peigan Gao, Jiakun Wu, Wenzhe Li, Ergan Shang, Qingyuan Zheng, Jingjing Zhou, Ruixuan Jia, Yan Xu, Hongrui Zhang, Xiao-Han Ma, Zhengxiang Cheng, Yuexing Hao, Liting Mai, Xianglin Ji, Wenjun Zhang, Zhuofan Chen, Yixiao Huang, Chi Wang, Wenyue Hua, Yilun Hao, Yuantao Zhai, Ziyan Zhao, Jingyan Xie

agents benchmarks

arXiv AI

Jul 21

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

arXiv:2607. 16448v1 Announce Type: cross Abstract: Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles).

By Sriram Balasubramanian, Soheil Feizi

llms rag agents diffusion benchmarks safety

arXiv AI

Sep 10

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exi...

By Yunxiang Mo, Tianshi Zheng, Yisen Gao, Rui Wang, Newt Nguyen Kim Hue Nam, Kelvin Kiu Wai Tam, Jiaxin Bai, Yangqiu Song, Ginny Wong, Simon See

llms agents benchmarks

arXiv AI

Jun 4

The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?

arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.

By Xinyu Lu, Tianshu Wang, Pengbo Wang, zujie wen, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

agents benchmarks safety

One email a morning, machine-written

Related stories

What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

ASI-Bench: At the Dawn of Artificial Superintelligence

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?