The paper investigates how on-policy distillation (OPD) and reinforcement learning with verifiable rewards (RLVR) can be combined for post‑training reasoning in large language models. It shows that a two‑stage approach—first applying OPD, then RL—outperforms single‑signal methods and other joint baselines on logic and math reasoning benchmarks. The authors explain this advantage through pass@k analysis, learning dynamics, and parameter updates, concluding that OPD expands solution coverage while RL sharpens performance within that support, and that the OPD validation score is the key trigger for switching to RL.
By Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie, Chen Zhao, Xi Ye
The paper investigates how to combine on‑policy distillation (OPD) and reinforcement learning with verifiable rewards (RLVR) for post‑training reasoning in large language models. It finds that a simple two‑stage approach—first applying OPD, then switching to RL—outperforms single‑signal methods and other joint baselines on logic and math reasoning benchmarks. The authors explain this advantage by showing that OPD broadens the model’s coverage of teacher‑supported solutions while RL sharpens performance within that support, and they provide practical guidance on when to switch based on OPD validation scores.
The paper introduces UECR-GRPO, a method that unifies on‑policy distillation and verifier‑based reinforcement learning for mathematical reasoning. It combines verifier rewards and teacher‑derived log‑ratios into a single KL‑regularized objective (Path‑Utility Unification) and then redistributes credit at the token level using entropy‑calibrated redistribution, preserving total task credit. Experiments on five benchmarks show that UECR‑GRPO improves average accuracy by up to 0.89 percentage points over the best baseline for both Qwen3‑1.7B and Qwen3‑4B students.
By Jie Zhang, Jingxiao Yang, Zhehao Huang, Yuhang Liu, Xiaolin Huang
arXiv:2608.24696v1 Announce Type: cross
Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training lar...
By Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, Gao Huang
arXiv:2607. 05394v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training.
By Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck.