arXiv AI
WASD: Wasserstein-based Knowledge Distillation for Large Language Models
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 9
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.
arXiv AI
Jun 30
ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
arXiv:2606. 29869v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization.
arXiv Statistics ML
5d ago
Unbiased Top-$k$ Estimation for On-Policy Distillation
The paper introduces Tail‑Corrected Top‑k On‑Policy Distillation (TT‑OPD), a method that improves on existing Top‑k OPD by combining the selected top‑k tokens with a sampled token from the student’s rollout. This hybrid approach recovers the probability mass discarded by limiting to top‑k, yielding an unbiased estimator of the reverse KL divergence gradient while maintaining low computational cost. Experiments show TT‑OPD outperforms other OPD variants in accuracy.
arXiv Computation and Language
6d ago
Distribution Matching Distillation for Continuous Diffusion Language Models
arXiv:2609.40235v1 Announce Type: cross Abstract: Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (...
Hugging Face Trending Papers
Jun 29
ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation
Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization. To address this, we analyze the complementary roles of forward and reverse KL divergence (FKL/RKL) in distribution alignment from theoretical and empirical perspectives.
arXiv Machine Learning
23h ago
Learning What to Distill: Bilevel Top-K Token Selection for Self-Distillation in Large Language Models
arXiv:2610.07247v1 Announce Type: new Abstract: Large language models have shown strong reasoning capabilities, but their high inference costs make knowledge distillation an important approach for tr...