arXiv Computation and Language By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz

Recursive Self-Improvement via On-Policy Distillation for Reasoning

Read the original on arXiv Computation and Language →

The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
6d ago

Bootstrapped On-Policy Distillation

Bootstrapped On-Policy Distillation (BOPD) is a two‑stage framework that first uses on‑policy self‑distillation (OPSD) to capture task‑relevant knowledge from reference‑conditioned supervision into an intermediate model, and then distills this intermediate model into a fresh copy of the initial model. The approach addresses the degradation observed in OPSD during continued training, which is caused by a growing mismatch between the evolving student and the frozen teacher. Experiments demonstrate that BOPD improves mathematical reasoning and code generation beyond both the initial models and the constructed OPSD teachers.

By Yue Wu, Xueyu Zhou, Yangrong Hu, Jian Huang
arXiv AI
Sep 7

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is a new method that builds a synthetic teacher from a language model’s own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor in parameter or logit space, RISE transforms sparse outcome-based updates into dense token-level targets without external models or privileged conditioning. The approach recursively refines the student model, combining RLVR and on‑policy distillation, and demonstrates superior performance across mathematical reasoning, STEM, code generation, and multi‑turn agentic tasks.

By Yang Li, Semih Yavuz, Shafiq Joty
arXiv Machine Learning
Jul 22

H$^2$SD: Hybrid Hindsight Self-Distillation

arXiv:2607. 18955v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation.

By Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin