The paper investigates whether reinforcement learning (RL) truly endows large language models (LLMs) with reasoning skills or merely tunes them to exploit prompt information. By training small LLMs on the finite logical domain of Tic‑Tac‑Toe and applying Group Relative Policy Optimization (GRPO), the authors show that RL improves accuracy and robustness. However, mechanistic interpretability using sparse autoencoders reveals that the models mainly rely on explicit prompt cues—such as turn order, game progression, and board occupancy—while higher‑level concepts like board symmetries and strategic forks remain weakly represented, suggesting that perceived reasoning may be an artifact of task‑specific adaptations.
By Aishik Nagar, Cheston Tan
The paper introduces SAEScientist-Bench, a benchmark that tests whether AI agents can autonomously conduct mechanistic interpretability research using Sparse Autoencoders (SAEs). Agents are tasked with designing contrastive probes and navigating a large feature dictionary in Gemma-2-9B-IT to identify optimal features for a target concept, with performance measured against expert-curated references on activation rank, concept selectivity, and causal steering. Results show that while frontier agents can discover features and outperform controls, they still lag behind expert baselines, especially in causal steering, highlighting both the potential and current limitations of closed-loop autonomous AI research.
By Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu
The paper investigates how reinforcement learning can unintentionally obscure the chain‑of‑thought (CoT) reasoning in vision‑language models, making their internal reasoning less traceable. By analyzing activation patterns, the authors show that template‑associated activations become less distinguishable during RL and that targeted interventions can mitigate this effect. They introduce TAME, a method that uses sparse autoencoders to suppress these problematic activations while still encouraging accurate behavior, achieving significant gains in CoT monitorability across multiple datasets and model families.
By Xutao Mao, Jianing Zhu, Jinman Zhao, Tongliang Liu, Xiaowen Chu, Cong Wang, Bo Han
arXiv:2609.24243v3 Announce Type: replace
Abstract: Reinforcement learning (RL) has substantially improved the reasoning capabilities of vision-language models (VLMs), yet it can induce chain-of-thou...
By Xutao Mao, Jianing Zhu, Jinman Zhao, Tongliang Liu, Xiaowen Chu, Cong Wang, Bo Han
arXiv:2606. 09932v1 Announce Type: cross Abstract: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become a standard pipeline for Large Language Model (LLM) post-training.
By Runze Liu, Jiashun Liu, Xu Wan, Yuqian Fu, Ling Pan
arXiv:2607. 01232v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers.
By Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong
arXiv:2606. 26620v1 Announce Type: cross Abstract: Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features.
By XinYang He, Wei Wang, Bing Zhao, Xuan Ren, WenBo Li, WeiXu Qiao, Hu Wei, Lin Qu
arXiv:2507.14843v5 Announce Type: replace
Abstract: Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it...
By Fang Wu, Weihao Xuan, Danny Xing, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, Yejin Choi
arXiv:2606. 18521v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT) in eliciting reasoning intelligence and resisting catastrophic forgetting.
By Chenrui Wu, Zexi Li, Jiajun Bu, Jiangchuan Liu, Haishuai Wang
The paper introduces a new framework for evaluating sparse autoencoders (SAEs) by requiring their features to form an atomic interpretable unit basis, a coordinate system for semantically meaningful explanation. It formalizes this objective and organizes existing and new metrics, such as DAME, into a comprehensive benchmarking framework. Experiments across many SAE architectures reveal that some models appear monosemantic under current metrics yet are functionally polysemantic, with dominant features being less interpretable and more redundant.
By Riteng Zhang, Xiaoqian Wang
arXiv:2610.01133v1 Announce Type: cross
Abstract: Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yiel...
By Bangji Yang, Jiajun Fan, Hongba Ma, Ruihan Guo, Ge Liu
arXiv:2510. 11686v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.
By Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, Jordan T. Ash