arXiv AI
Sep 25

Post-Training Leaves Behavioral Shadows on Unrelated Decisions

The paper demonstrates that language models can acquire new capabilities from post‑training data even when the training text is unrelated to the target task. Using a method called Active Taskless Distillation (ATD), the authors show that a single word from a teacher model can transfer knowledge to a student model without any target‑task examples or teacher logits. Experiments on Qwen2.5-1.5B reveal significant performance gains on HumanEval+ and improvements in scientific knowledge, commonsense reasoning, and reading comprehension across various model families.

By Ziyang Zhang, Yubin Jing, Yuanhao Zeng, Yuyao Li, Haofan Wang, Yichen Gong
arXiv Machine Learning
6d ago

Are Causal Neuron Circuits Unique and Transferable?

The paper investigates whether groups of neurons that influence a model’s answer are uniquely necessary and transferable across inputs. By exhaustively testing all 4,096 subsets of 12 MLP neurons in Qwen3 and other models, the authors find that only a few neurons are often sufficient to alter predictions, that many subsets can independently flip answers, and that the effect is strongest on the same input chain but weaker on different chains. The study shows that causal neuron circuits are not uniquely defined and that their influence can vary with context and model architecture.

By \"Omer Tekin Yavuz (\.Ileti\c{s}im Yaz{\i}l{\i}m)
arXiv Machine Learning
Aug 28

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

The paper investigates the problem of sharing a single critic across multiple parallel environments in reinforcement learning. It shows that when environments assign different expected returns to the same state, a shared critic must reconcile conflicting value targets, which can distort advantage estimates and misguide policy updates. The authors propose a simple fix—providing the critic with the environment index—demonstrating through bandit models and experiments on CartPole, MuJoCo, BipedalWalker, and 16 Procgen games that this conditional critic stabilizes learning and boosts returns, achieving a 40.8% improvement in aggregate normalized return on unseen levels.

By Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
arXiv Machine Learning
Aug 27

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

The paper reviews On‑Policy Self‑Distillation (OPSD), a method where a language model learns from its own generations using privileged information such as reference solutions or plans, eliminating the need for a larger teacher model. It identifies a key failure mode—collapse, where the model’s reasoning paths narrow progressively—and analyzes it through three levers: signal application, privileged information, and teacher dynamics. The review focuses on mathematical reasoning, offering a unified vocabulary and distinguishing settled facts from ongoing debates.

By Justin Robert, Raheel Qader
arXiv Machine Learning
Oct 15

Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context RL

The paper investigates whether large language models (LLMs) can learn from rewards presented in context, a process termed in-context reinforcement learning (ICRL). Experiments across four benchmarks and six models show that while LLMs read reward signals, altering or removing these rewards has little impact on performance, and shuffled or corrupted trajectories are as effective as real ones. These findings suggest that direct ICRL behaves more like in-context learning (ICL) than true inference-time reinforcement learning.

By Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim