arXiv Computation and Language
K2P: Label-Free Knowledge to Prompt Distillation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Aug 21
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
arXiv:2608. 19408v1 Announce Type: new Abstract: On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher.
arXiv AI
Aug 26
OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning
OPDSearch+ introduces a two‑stage distillation framework for search‑augmented reasoning that eliminates the need for task‑specific teacher fine‑tuning. In the first stage, a frozen off‑the‑shelf instruct model guides a student through live search interactions using a per‑position forward KL objective, transferring reasoning decomposition and evidence integration skills. The second stage refines this student with reinforcement learning, achieving performance surpassing RL alone and outperforming all prior 3B‑parameter baselines on seven QA benchmarks, including 13.1% improvement on HotpotQA and 8.5% on 2WikiMultihopQA.
arXiv AI
Jun 9
The Strongest Teacher Is Not Always the Best Teacher: Student-Centric Answer Selection
arXiv:2605. 26872v2 Announce Type: replace-cross Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations.
arXiv AI
Sep 1
When the Strongest Teacher Is Not the Best Teacher: Student-Centric Answer Selection
arXiv:2605.26872v4 Announce Type: replace-cross Abstract: LLM training increasingly relies on teacher-generated supervision, from synthetic responses to reasoning traces and tool-use demonstrations....
arXiv AI
Jun 19
Reinforcement-aware Knowledge Distillation for LLM Reasoning
arXiv:2602. 22495v3 Announce Type: replace-cross Abstract: Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students.
arXiv AI
4d ago
Distilling Directional Verification
arXiv:2610.00997v1 Announce Type: cross Abstract: Knowledge distillation aims to transfer the factual knowledge of large language models to smaller models for efficient deployment. Yet a teacher may...