arXiv Machine Learning

The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation

arXiv AI
Sep 25

Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD

The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.

By Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li
arXiv AI
Jul 7

Weak-to-Strong Generalization via Direct On-Policy Distillation

arXiv:2607. 05394v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training.

By Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
arXiv Computation and Language
Sep 28

Recursive Self-Improvement via On-Policy Distillation for Reasoning

The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.

By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz
arXiv AI
Oct 1

From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL

The paper introduces On‑Policy Warmup (OPW), a teacher‑guided training stage where a student agent learns from a teacher on its own interaction trajectories before switching to reinforcement learning with verifiable rewards (RLVR). OPW differs from traditional imitation by focusing on states generated by the student’s own decisions, including imperfect actions and recovery situations. The authors provide a theoretical link between on‑policy reverse‑KL distillation and trajectory‑level distribution matching, showing that, under a competent teacher and low distillation loss, OPW can lower bound initial verifier success and reduce reward‑discovery complexity, thereby accelerating RLVR performance.

By Yitong Qiao, Tiantian He, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan Chu
arXiv Computation and Language
Oct 9

Residual Advantage: Student-Relative Teacher Guidance for RL with Verifiable Rewards

The paper introduces Residual Advantage (RA), a new guidance method for reinforcement learning with verifiable rewards (RLVR) that treats the teacher–student probability residual as a bounded one‑step reward, subtracts the student’s state value to form an advantage, and centers it within each response. RA preserves the verifier’s mean label while redistributing credit among response steps, and its extension CoRA updates the teacher model using verifier advantages to adapt guidance to the student’s attempts. Experiments with Qwen3 models show that RA combined with GRPO or REINFORCE++ consistently outperforms teacher‑only on‑policy distillation across three mathematical benchmarks, improving macro Avg@8 and Pass@8 scores, with CoRA providing additional gains.

By Xiaobing Chen, Zhiqi Pang
arXiv AI
Sep 7

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.

By Yang Li, Semih Yavuz, Shafiq Joty
arXiv AI
Oct 15

Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training

The paper proposes a reward‑density principle for language‑model post‑training: sparse sequence‑level rewards are best for exploring and discovering new behaviors in a strong teacher, while dense token‑level teacher supervision is optimal for compressing that behavior into a smaller deployment model. The authors validate this rule with a four‑stage workflow—teacher RL, forward‑KL warmup, on‑policy distillation, and optional post‑bridge student RL—showing that an RL‑improved 8B teacher distilled through a dense bridge outperforms direct GRPO on a 1.7B student across math and code benchmarks. Ablation studies confirm each stage’s contribution, and the approach generalizes to Llama and Qwen models, offering a practical guideline for allocating scarce labeled data.

By Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard
arXiv Machine Learning
Oct 8

Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging

The study compares supervised fine-tuning (SFT) and reinforcement learning (RL) as teacher training methods for on-policy distillation (OPD) in a single-teacher setting. Using Qwen3.5-9B models across Agentic, Reasoning, and Perception domains, RL-trained teachers consistently produce stronger students and better recovery of performance gains than SFT-trained teachers. The authors attribute this advantage to RL teachers’ smaller parameter shifts from the shared initialization, which facilitates learning through OPD.

By Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu
arXiv Machine Learning
Oct 9

Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

The paper investigates why on‑policy distillation (OPD) can fail, observing that larger‑scale teachers often hit early loss plateaus while self‑RL teachers achieve much greater loss reductions. By modeling OPD as a continuous‑time dynamical system and analyzing training logs, the authors link plateaus to a rapid decline in a gradient‑based learning‑signal proxy, even though loss remains high. They also prove a local recovery guarantee for teachers close to the initial student and note that minimal parameter changes and high representation similarity suggest limited adaptation may cause learning‑signal collapse.

By Lei Zhao, Qichao Zhao, Bowen Zuo, Qishi Zhan