arXiv AI
STEPS: Selective On-Policy Self-Distillation for Reasoning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Jun 10
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs.
arXiv Machine Learning
Jun 11
RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation
arXiv:2606. 11709v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution.
arXiv Computation and Language
4d ago
Recursive Self-Improvement via On-Policy Distillation for Reasoning
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
arXiv Machine Learning
Jul 17
On-Policy Delta Distillation
arXiv:2607. 15161v1 Announce Type: new Abstract: On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model.
arXiv AI
Sep 7
Extremely Sparse Supervision Incentivizes Reasoning Ability
The paper reports that in on‑policy distillation for large language models, reasoning performance can be improved by supervising only a tiny fraction of generated tokens—sometimes just one or two tokens per reasoning trajectory, about 0.05% of all tokens. This sparse supervision consistently matches or exceeds full‑token training across nine teacher‑student setups on mathematical reasoning, and is also validated on coding reasoning, Llama models, and PPO‑based reinforcement learning with verifiable reward. The findings suggest that effective post‑training does not require token‑intensive supervision and may align more closely with natural learning processes that focus on critical reasoning steps.
arXiv AI
Jun 19
Reinforcement-aware Knowledge Distillation for LLM Reasoning
arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.