The paper examines how evaluation design can cause significant fluctuations in the benchmark results of reasoning models, particularly the Deepseek‑R1‑Distill series. It shows that subtle changes in evaluation conditions lead to large variations in reported performance, a phenomenon also seen in other open‑source models fine‑tuned from Deepseek‑R1‑Distill and in the QwQ‑32B model. The authors call for a more rigorous evaluation paradigm and provide empirical assessments of the Deepseek‑R1‑Distill models.
By Yongfu Zhu, Lin Sun, Jinzhu Wu, Weihong Lin, Xiaoqi Jian, Guangxiang Zhao, Change Jia, Linglin Zhang, Sai-er Hu, Yuhan Wu, Xiangzheng Zhang
KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.
By Ryan Swift, Konstantinos Psounis
arXiv:2606. 06840v1 Announce Type: cross Abstract: Modern reasoning models offer surprisingly strong zero-shot performance on challenging multi-label tasks that require selecting a small set of relevant options from hundreds of thousands to millions of candidate labels.
By Debjyoti Saha Roy, Byron C. Wallace, Javed A. Aslam
OpenAI exposed and disrupted a coordinated campaign aimed at extracting protected model reasoning through model distillation. The incident highlighted vulnerabilities in how models can be reverse‑engineered by adversaries. In response, OpenAI is enhancing its defenses to guard against future adversarial distillation attempts.
arXiv:2604. 05634v2 Announce Type: replace Abstract: Machine unlearning (MU) has become a critical technique for GenAI models' safe and compliant operation.
By Zhiyong Ma, Zhitao Deng, Huan Tang, Jialin Chen, Zhijun Zheng, Zhengping Li, Qingyuan Chuai
Sig-DEG for Distillation proposes a Signature-based Differential Equation Generator to distill pre-trained diffusion models. By combining partial signatures of Brownian motion with a recurrent structure, it approximates stochastic differential equation solutions and formulates distillation as supervised learning on coarse time grids. The method enables fast inference, achieving up to several hundred‑fold speedups while maintaining competitive sample quality in image and time‑series generation, and extends to applications like Schrödinger bridges.
By Wen Ge, Lei Jiang, Lingyi Yang, Leonardo Roveri, Mingxuan Yi, Po-Yu Chen, Niels Cariou-Kotlarek, Francois Buet-Golfouse, Gaurav Mittal, Hao Ni
arXiv:2606. 24747v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical.
By Lavinia Ghita, Dhruv Desai, Ioana Boier
The paper introduces DLM-One, a score‑distillation framework that enables one‑step sequence generation with continuous diffusion language models (DLMs). By aligning a student model’s outputs with a pretrained teacher DLM’s score function in the forward‑diffused noisy space, DLM-One removes the need for iterative refinement. Experiments across various DLM architectures show up to ~2000× speedup in sampling steps and ~500× in wall‑clock time while retaining competitive performance, and the authors propose an adversarially‑regularized two‑stage training scheme to mitigate student degeneration.
By Tianqi Chen, Shujian Zhang, Mingyuan Zhou
arXiv:2610.15919v1 Announce Type: cross
Abstract: Supervised machine learning in high-energy physics (HEP) requires training on labeled simulations of particle collisions, which are known to be unfai...
By Zichun Hao, Raghav Kansal, Abhijith Gandrakota, Chang Sun, Haoyang Li, Zihan Zhao, Ngadiuba Jennifer, Javier Duarte, Maria Spiropulu
arXiv:2606. 03052v1 Announce Type: new Abstract: Knowledge Distillation (KD) is a powerful tool for model compression, yet the precise mechanisms by which student models acquire feature representations remain underexplored.
By Seungu Kang, Songkuk Kim
arXiv:2607. 04751v1 Announce Type: cross Abstract: Big goals are hard to achieve all at once; breaking them into small steps is wiser.
By Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, Mao Yang