arXiv Computation and Language
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jun 30
ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
arXiv:2606. 29869v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization.
Hugging Face Trending Papers
Jun 29
ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization. To address this, we analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives.
arXiv AI
6d ago
LLMR: Knowledge Distillation with a Large Language Model-Induced Reward
The paper introduces LLMR, a new knowledge distillation technique that uses a reward function derived from large language models to guide the distillation process. Experiments on dialogue generation and summarization datasets show that LLMR consistently outperforms traditional knowledge distillation methods across multiple tasks and datasets.
arXiv Machine Learning
Jun 19
A Survey of On-Policy Distillation for Large Language Models
arXiv:2604. 00626v4 Announce Type: replace Abstract: As Large Language Models continue to grow in both capability and cost, transferring frontier capabilities into smaller, deployable students has become an important engineering problem, and knowledge distillation remains a common technique for this transfer.
arXiv Machine Learning
Jul 7
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
arXiv AI
Aug 11
Hybrid Policy Distillation for LLMs
arXiv:2604. 20244v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a powerful paradigm for compressing large language models (LLMs), whose effectiveness depends on intertwined choices of divergence direction, optimization strategy, and data regime.