arXiv Machine Learning October 15, 2026 By Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du

VASPO: Self-Distillation with Privileged Value Estimation

Read the original on arXiv Machine Learning →

VASPO introduces a new integration of self-distillation and PPO by leveraging privileged value estimation to guide language model training. The method uses a self-teacher and a critic that both evaluate the same privileged information on student-generated prefixes, allowing the critic to compute token-level PPO advantages from predicted continuation returns while the teacher modulates positive advantages via bounded probability ratios. Experiments on six mathematical reasoning benchmarks show that VASPO improves mean accuracy over standard PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Oct 1

STEPS: Selective On-Policy Self-Distillation for Reasoning

arXiv:2605.10194v2 Announce Type: replace Abstract: On-policy self-distillation (OPSD) uses a model as its own teacher under privileged context, providing token-level supervision on the model's own r...

By Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Lan-Zhe Guo

reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Oct 2

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD) is a new on‑policy self‑distillation method that uses a frozen copy of the base model to extract a steering vector by contrasting activations from self‑generated trajectories that reach verified correct answers with all other trajectories. The student learns from next‑token distributions on its own prefixes, without needing reference text or teacher parameter updates, and is used alone at inference. Across five models, ACSD achieves the highest mean accuracy on four mathematical benchmarks, with notable gains on DeepSeek‑R1‑0528‑Qwen3‑8B and LiveCodeBench v6 compared to the OPSD baseline.

By Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

efficiency benchmarks

arXiv AI

1d ago

Does self-distillation improve reasoning?

arXiv:2610.18613v1 Announce Type: cross Abstract: Self-distillation is an emerging approach that trains a model to imitate its own predictions when conditioned on privileged information, such as corr...

By Jason Ken Adhinarta, Raina Wu, Ashutosh Sharma, Avinash Balakrishnan, Rameswar Panda, Polina Golland, Yoon Kim, Zhang-Wei Hong

agents reinforcement-learning efficiency safety

arXiv Machine Learning

Sep 30

Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning

The paper introduces On-Policy Attention Self-Distillation (OPASD), a method that augments token-level supervision with solution-conditioned attention distillation for reasoning models. OPASD projects a privileged teacher’s attention onto student-visible positions, renormalizes the distribution, and aligns it with the student. Experiments on three model sizes and four math benchmarks show that OPASD improves accuracy by 4.98–8.40 percentage points, reduces generated tokens by 73.9%, cuts compute by 72.6%, and trains 1.53× faster compared to token-only distillation.

By Safaeid Hossain Arib, Rabeya Akter, Ismam Nur Swapnil, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam

efficiency benchmarks safety

arXiv Computation and Language

Sep 28

Recursive Self-Improvement via On-Policy Distillation for Reasoning

The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.

By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz

efficiency benchmarks

Hugging Face Trending Papers

Jun 14

Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoning

Reinforcement learning from verifiable rewards assigns a single scalar to each rollout, leaving token-level credit assignment underspecified in long reasoning traces. On-policy self-distillation addresses this by letting the same model act as a teacher conditioned on privileged information, producing a dense per-token signal.

llms reinforcement-learning efficiency benchmarks