arXiv Machine Learning July 14, 2026 By Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min

Reference-Based Distillation Detection in LLMs

Read the original on arXiv Machine Learning →

arXiv:2607. 09692v1 Announce Type: new Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

4d ago

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

arXiv:2609.36734v1 Announce Type: new Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...

By Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty

llms efficiency benchmarks safety

arXiv AI

Sep 10

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

arXiv:2605.10889v2 Announce Type: replace-cross Abstract: On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions...

By Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N. M Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, Mehrdad Farajtabar

efficiency safety

arXiv Machine Learning

Sep 22

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...

By Dileesha Kannangara, Sanghamitra Dutta

llms nlp efficiency benchmarks

arXiv AI

Jul 3

Purified OPSD: On-Policy Self-Distillation Without Losing How to Think

arXiv:2607. 02234v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) has emerged as a promising paradigm for improving LLM reasoning, where a privileged teacher with access to reference solutions provides token-level supervision on the student's own generated trajectories.

By Zhanming Shen, Jintao Tong, Shaotian Yan, Chen Shen, Hao Chen, Wentao Ye, Xiaomeng Hu, Rui Miao, Haobo Wang, Junbo Zhao, Gang Chen, Jieping Ye

llms efficiency

arXiv AI

Sep 17

Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers

The paper introduces a method for offline on‑policy distillation that addresses the problem of imperfect teacher supervision. By training on teacher‑successful problems and measuring changes in token likelihoods on teacher‑failed trajectories, the authors derive a learnability signal that weights the distillation loss. This approach improves performance on mathematical reasoning and code generation tasks while reducing computational cost compared to online distillation.

By Yihao Ai, Weilong Yan

efficiency benchmarks

arXiv Machine Learning

2d ago

Activation-Conditioned Self-Distillation

arXiv:2609.38342v1 Announce Type: new Abstract: On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning....

By Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

efficiency benchmarks

One email a morning, machine-written

Related stories

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

Purified OPSD: On-Policy Self-Distillation Without Losing How to Think

Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers

Activation-Conditioned Self-Distillation