Hugging Face Trending Papers

Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

Read the original on Hugging Face Trending Papers →

The paper introduces On-Policy Mid-Training Distillation (OPMD), a method that replaces the standard next-token prediction (NTP) objective in continual pre‑training with an active, query‑driven approach. By sampling trajectories from the evolving student policy and applying dense token‑level reverse‑KL rewards from a frozen teacher, OPMD trains the model on states it actually visits, reducing the mismatch between training and generation. Experiments on a 1B‑token mathematical corpus show that OPMD improves performance for the Qwen3‑1.7B‑Base model on MATH‑500 and Minerva benchmarks, outperforming the NTP baseline that degrades on both tasks, while adding no inference‑time overhead.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv Machine Learning
6d ago

Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language Models

The paper introduces On-Policy Mid-Training Distillation (OPMD), a method that shifts continual pre‑training from passive next‑token prediction to active, query‑driven knowledge acquisition. By sampling trajectories from the evolving student policy and applying dense token‑level reverse‑KL rewards from a frozen teacher, OPMD trains the model on states it actually visits, reducing the mismatch between training and generation. Experiments on MATH‑500 and Minerva show that OPMD improves a Qwen student’s performance over its base and outperforms the standard next‑token prediction baseline, while integrating seamlessly into existing continual pre‑training pipelines without inference‑time overhead.

By Ruize Zheng, Xuyang Liu, Siyu Ren, Xiaobing Tu, Yinggui Wang, Jinkui Ren, Xiantao Zhang, Linfeng Zhang