← Back to all news
arXiv Computation and Language By Byung-Kwan Lee, Ximing Lu, Shizhe Diao, Minki Kang, Saurav Muralidharan, Karan Sapra, Andrew Tao, Pavlo Molchanov, Yejin Choi, Yu-Chiang Frank Wang, Ryo Hachiuma

Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients

Read the original on arXiv Computation and Language →

The paper introduces Zone of Proximal Policy Optimization (ZPPO), a method that keeps a teacher model inside prompts rather than in the policy gradient to improve knowledge distillation for small students. ZPPO creates two types of reformulated prompts—Binary Candidate-included Questions (BCQ) and Negative Candidate-included Questions (NCQ)—to expose students to correct and incorrect responses, and uses a replay buffer to focus training on hard questions until the student’s accuracy improves. Experiments on the Qwen3.5 family with a 27B teacher across 31 benchmarks show that ZPPO outperforms both off‑policy and on‑policy distillation methods, especially at the smallest student scales.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 15

Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training

The paper proposes a reward‑density principle for language‑model post‑training: sparse sequence‑level rewards are best for exploring and discovering new behaviors in a strong teacher, while dense token‑level teacher supervision is optimal for compressing that behavior into a smaller deployment model. The authors validate this rule with a four‑stage workflow—teacher RL, forward‑KL warmup, on‑policy distillation, and optional post‑bridge student RL—showing that an RL‑improved 8B teacher distilled through a dense bridge outperforms direct GRPO on a 1.7B student across math and code benchmarks. Ablation studies confirm each stage’s contribution, and the approach generalizes to Llama and Qwen models, offering a practical guideline for allocating scarce labeled data.

By Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard
llmsreinforcement-learningefficiencybenchmarks
More like this →
arXiv AI
Aug 11

Mismatch Matters: On-Policy Distillation Beyond Token Agreement

arXiv:2608. 09836v1 Announce Type: new Abstract: On-policy distillation (OPD) has emerged as a core component of modern LLM post-training pipelines, yet we reveal a failure mode: degenerate agreement, where students exploit repetitive loops to achieve near-perfect token agreement with the teacher despite globally flawed responses.

By Zichao Yu, Chengzhi Yu, Shengze Xu, Yujin Han, Bingqing Jiang, Xu Wang, Difan Zou
llmsefficiencybenchmarks
More like this →