arXiv AI September 4, 2026 By Boyan Li, Bingsen Chen, Chenghao Yang, Ping Nie, Chen Zhao, Xi Ye

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

The paper investigates how on-policy distillation (OPD) and reinforcement learning with verifiable rewards (RLVR) can be combined for post‑training reasoning in large language models. It shows that a two‑stage approach—first applying OPD, then RL—outperforms single‑signal methods and other joint baselines on logic and math reasoning benchmarks. The authors explain this advantage through pass@k analysis, learning dynamics, and parameter updates, concluding that OPD expands solution coverage while RL sharpens performance within that support, and that the OPD validation score is the key trigger for switching to RL.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

2d ago

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level su...

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 11

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

arXiv:2606. 11709v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution.

By Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, Lijie Wen

reinforcement-learning efficiency

Hugging Face Trending Papers

Jul 6

Weak-to-Strong Generalization via Direct On-Policy Distillation

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck.

reinforcement-learning efficiency

Hugging Face Trending Papers

Jun 10

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs.

reinforcement-learning efficiency

arXiv Machine Learning

2d ago

Cliff: Learning Process Rewards from the First Mistake

arXiv:2609.02817v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance...

By Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, Chris Kong

llms reinforcement-learning efficiency

arXiv AI

Aug 3

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

arXiv:2607. 29209v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it.

By Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen

reinforcement-learning efficiency benchmarks

One email a morning, machine-written

Related stories

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

Weak-to-Strong Generalization via Direct On-Policy Distillation

RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

Cliff: Learning Process Rewards from the First Mistake

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation