← Back to all news
arXiv AI By Yi Jing, Zao Dai, Jinwu Hu, Zijun Yao, Lei Hou, Juanzi Li, Xiaozhi Wang

Teach the Way It Thinks: Guiding LLM Post-training Data Engineering with Sparse Autoencoders

Read the original on arXiv AI →

The paper introduces SAERL, a method that uses Sparse Autoencoders (SAEs) to guide reinforcement learning data engineering for large language models. By mapping training samples into a feature space of interpretable SAE activations, SAERL selects samples with strong, non‑redundant gradient signals, organizes functional regions from easy to hard, and batches from single activation clusters with moderate mixing. Experiments on mathematical reasoning tasks show consistent performance gains across model scales and RL algorithms, with notable improvements on Qwen2.5-Math-1.5B.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jul 16

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

arXiv:2510. 11686v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model.

By Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, Jordan T. Ash
llmsreinforcement-learning
More like this →
arXiv AI
Oct 31

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

The paper investigates whether reinforcement learning (RL) truly endows large language models (LLMs) with reasoning skills or merely tunes them to exploit prompt information. By training small LLMs on the finite logical domain of Tic‑Tac‑Toe and applying Group Relative Policy Optimization (GRPO), the authors show that RL improves accuracy and robustness. However, mechanistic interpretability using sparse autoencoders reveals that the models mainly rely on explicit prompt cues—such as turn order, game progression, and board occupancy—while higher‑level concepts like board symmetries and strategic forks remain weakly represented, suggesting that perceived reasoning may be an artifact of task‑specific adaptations.

By Aishik Nagar, Cheston Tan
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv AI
Oct 19

Language Models Learn to Improve Language Models

The paper introduces LM2LM, a method that trains large language models (LLMs) to propose improvements for other LLMs using reinforcement learning (RL). It employs a frozen research world model (RWM) to predict the gains of proposals, providing a cheaper surrogate reward, and periodically validates high‑scoring proposals to keep feedback reliable. Experiments across eight LM development environments show that LM2LM enables a Qwen3.8‑27B model to outperform GPT‑6‑sol on held‑out tasks while reducing data‑generation cost by 33%, and it generalizes to unseen model families.

By Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang, Suhang Wang, Hanqing Lu
llmsdiffusionreinforcement-learning
More like this →
arXiv Computation and Language
Aug 28

Reward-Informed Sparse Autoencoders and the Solution-Completeness Confound

The paper introduces Reward‑Informed Sparse Autoencoders (RI‑SAEs), which use reinforcement‑learning rewards to curate data for training sparse autoencoders on language‑model activations. On Llama‑3.1‑8B, a sparse subset of features separates high‑reward from low‑reward reasoning continuations, but control experiments show this separation largely reflects solution completeness rather than true reasoning quality. The authors conclude that reward filtering can cheaply reuse RL signals for interpretability, though most of the discovered features capture completion form rather than deep reasoning.

By Tanvi Nagilla, Alexander Jameson, Daniel Manta, Shayaan Uddin
llmsreinforcement-learningsafety
More like this →