The paper introduces Residual Advantage (RA), a new guidance method for reinforcement learning with verifiable rewards (RLVR) that treats the teacher–student probability residual as a bounded one‑step reward, subtracts the student’s state value to form an advantage, and centers it within each response. RA preserves the verifier’s mean label while redistributing credit among response steps, and its extension CoRA updates the teacher model using verifier advantages to adapt guidance to the student’s attempts. Experiments with Qwen3 models show that RA combined with GRPO or REINFORCE++ consistently outperforms teacher‑only on‑policy distillation across three mathematical benchmarks, improving macro Avg@8 and Pass@8 scores, with CoRA providing additional gains.
By Xiaobing Chen, Zhiqi Pang
Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck.
arXiv:2607. 05394v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training.
By Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
The paper introduces Selective Supervision for Direct-OPD (S$^2$D-OPD), a refinement of Direct On-Policy Distillation that filters out states where the teacher’s policy change is minimal, as measured by the teacher‑reference Jensen‑Shannon divergence. By masking low‑divergence states and keeping only the top 10% of states per response, S$^2$D-OPD improves held‑out accuracy on AIME and HMMT benchmarks across multiple teacher‑student pairs without additional forward passes.
By Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li
The paper introduces Zone of Proximal Policy Optimization (ZPPO), a method that keeps a teacher model inside prompts rather than in the policy gradient to improve knowledge distillation for small students. ZPPO creates two types of reformulated prompts—Binary Candidate-included Questions (BCQ) and Negative Candidate-included Questions (NCQ)—to expose students to correct and incorrect responses, and uses a replay buffer to focus training on hard questions until the student’s accuracy improves. Experiments on the Qwen3.5 family with a 27B teacher across 31 benchmarks show that ZPPO outperforms both off‑policy and on‑policy distillation methods, especially at the smallest student scales.
By Byung-Kwan Lee, Ximing Lu, Shizhe Diao, Minki Kang, Saurav Muralidharan, Karan Sapra, Andrew Tao, Pavlo Molchanov, Yejin Choi, Yu-Chiang Frank Wang, Ryo Hachiuma
arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.
By Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski