arXiv Machine Learning
Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
1d ago
Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
The paper investigates on‑policy distillation (OPD), a method that trains a student model using its own rollouts while a teacher remains frozen, to see if it can help a student improve on problems it currently fails. Experiments with Qwen‑3 and Phi‑4 models on a competition‑mathematics benchmark show that OPD actually degrades performance when training focuses on failure cases, even though training loss (KL divergence) continues to drop. However, when OPD is applied to a broader set of problems, it improves accuracy by 4.6 percentage points, indicating that OPD works well when most training data contains correct student prefixes but can reinforce errors if the training regime concentrates on failures.
arXiv AI
Jun 2
Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight
arXiv:2606. 00424v1 Announce Type: new Abstract: As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight.
Hugging Face Trending Papers
Sep 16
Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision per...
arXiv AI
Sep 17
Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
The paper introduces a method for offline on‑policy distillation that addresses the problem of imperfect teacher supervision. By training on teacher‑successful problems and measuring changes in token likelihoods on teacher‑failed trajectories, the authors derive a learnability signal that weights the distillation loss. This approach improves performance on mathematical reasoning and code generation tasks while reducing computational cost compared to online distillation.
arXiv AI
Jul 7
Reward-Gated On-Policy Distillation
arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.
arXiv Machine Learning
Oct 5
Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation
arXiv:2610.02700v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal...