Hugging Face Trending Papers
Nov 10

Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

The paper introduces On-Policy Mid-Training Distillation (OPMD), a method that replaces the standard next-token prediction (NTP) objective in continual pre‑training with an active, query‑driven approach. By sampling trajectories from the evolving student policy and applying dense token‑level reverse‑KL rewards from a frozen teacher, OPMD trains the model on states it actually visits, reducing the mismatch between training and generation. Experiments on a 1B‑token mathematical corpus show that OPMD improves performance for the Qwen3‑1.7B‑Base model on MATH‑500 and Minerva benchmarks, outperforming the NTP baseline that degrades on both tasks, while adding no inference‑time overhead.

arXiv Machine Learning
6d ago

Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

The paper introduces On-Policy Mid-Training Distillation (OPMD), a method that shifts continual pre‑training from passive next‑token prediction to active, query‑driven knowledge acquisition. By sampling trajectories from the evolving student policy and applying dense token‑level reverse‑KL rewards from a frozen teacher, OPMD trains the model on states it actually visits, reducing the mismatch between training and generation. Experiments on MATH‑500 and Minerva show that OPMD improves a Qwen student’s performance over its base and outperforms the standard next‑token prediction baseline, while integrating seamlessly into existing continual pre‑training pipelines without inference‑time overhead.

By Ruize Zheng, Xuyang Liu, Siyu Ren, Xiaobing Tu, Yinggui Wang, Jinkui Ren, Xiantao Zhang, Linfeng Zhang
arXiv AI
Jun 9

Trajectory-Refined Distillation

arXiv:2606. 08432v1 Announce Type: new Abstract: On-policy distillation (OPD) has become a central post-training tool for large language models (LLMs), providing dense per-token teacher supervision along the student's own rollouts.

By Li Jiang, Haoran Xu, Yichuan Ding, Amy Zhang