arXiv AI
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 9
The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection
arXiv:2605. 26872v2 Announce Type: replace-cross Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations.
arXiv Machine Learning
4d ago
Activation-Conditioned Self-Distillation
Activation-Conditioned Self-Distillation (ACSD) is a new on‑policy self‑distillation method that uses a frozen copy of the base model to extract a steering vector by contrasting activations from self‑generated trajectories that reach verified correct answers with all other trajectories. The student learns from next‑token distributions on its own prefixes, without needing reference text or teacher parameter updates, and is used alone at inference. Across five models, ACSD achieves the highest mean accuracy on four mathematical benchmarks, with notable gains on DeepSeek‑R1‑0528‑Qwen3‑8B and LiveCodeBench v6 compared to the OPSD baseline.
Hugging Face Trending Papers
Aug 28
VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation
On-policy self-distillation (OPSD) improves reasoning by training a problem-only student on its own rollouts using dense token-level supervision from a privileged teacher that also sees a reference so...
arXiv AI
1d ago
Learning to Revise Reasoning with Segment-wise On-Policy Distillation
arXiv:2610.02703v1 Announce Type: new Abstract: On-policy distillation (OPD) improves large language model reasoning by training students on their own rollouts with dense token-wise supervision from...
arXiv Machine Learning
Aug 31
VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation
The paper introduces VISTA, a method that enhances on‑policy self‑distillation (OPSD) by adapting the teacher model toward the student’s distribution using outcome‑verified rollouts. VISTA keeps the standard OPSD student update but selectively adjusts the teacher only on the top‑k positions with the largest teacher‑student KL divergence, without adding new sampling or reward objectives. Experiments on AIME24, AIME25, and HMMT25 with Qwen3 models show that VISTA outperforms OPSD across all scales, improving Avg@12 by up to 2.1 points.
arXiv AI
Jun 19
Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts
arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.