← Back to all news
arXiv Computation and Language By Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Read the original on arXiv Computation and Language →

The paper introduces REOPOLD, a framework that improves on-policy distillation (OPD) for reasoning tasks by addressing the problem of uniform credit allocation. REOPOLD selectively controls where teacher feedback is applied, its strength, and the timing of changes during training, thereby reducing entropy collapse and negative transfer. Experiments show that REOPOLD achieves up to 12× higher sample efficiency compared to recent post‑training baselines and can be extended to cross‑vocabulary distillation, self‑distillation, and test‑time scaling.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Sep 7

Extremely Sparse Supervision Incentivizes Reasoning Ability

The paper reports that in on‑policy distillation for large language models, reasoning performance can be improved by supervising only a tiny fraction of generated tokens—sometimes just one or two tokens per reasoning trajectory, about 0.05% of all tokens. This sparse supervision consistently matches or exceeds full‑token training across nine teacher‑student setups on mathematical reasoning, and is also validated on coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward. The findings suggest that effective post‑training does not require token‑intensive supervision and may align more closely with natural learning processes that focus on critical reasoning steps.

By Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane
llmsreinforcement-learningefficiency
More like this →
arXiv AI
Oct 9

DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

The paper introduces DIAL-OPD, a token‑selection strategy for on‑policy distillation that weights reward magnitude by the logarithmic mean of teacher and student probabilities, controlled by a parameter β. Experiments on four teacher‑student pairs across seven mathematical reasoning benchmarks show that retaining only 40 % of tokens with DIAL‑OPD yields up to 5.25 % higher mean accuracy than vanilla OPD and doubles AIME25 Pass@16, outperforming even stronger full‑token baselines. The study demonstrates that careful supervision allocation can surpass gains from simply scaling teacher size.

By Anhao Zhao, Haoran Xin, Junlong Tong, Yingqi Fan, Xuan Lu, Ping Nie, Wenjie Li, Xiaoyu Shen
efficiencybenchmarks
More like this →