The paper investigates how teacher signals influence parameter updates in Multi‑Teacher On‑Policy Distillation (MOPD) by analyzing Qwen3‑1.7B and SmolLM3‑3B. It shows that loss averaging, Adam’s first‑moment bias, BF16 rounding, and the choice of averaging rule all shape the gradients and ultimately affect task performance. The study quantifies these effects, revealing, for example, that token‑averaging favors longer responses and that BF16 rounding masks most weight changes.
By Siqi Zhu, Suozhi Huang, Kaixuan Zhang, Yuheng Yang, Zhanyang Jin, Yihang Sun, Jiaxuan You
The paper introduces Group‑Calibrated On‑Policy Distillation (GC‑OPD), a method that aligns token‑level teacher guidance with trajectory‑level verifier rewards for long‑context reasoning tasks. GC‑OPD normalizes rewards within rollout groups, uses the signed teacher‑verifier disagreement as a residual, and distributes this residual across tokens via Relative‑Advantage‑Based Credit Assignment (RACA). Experiments on five long‑context benchmarks show that GC‑OPD improves Qwen3‑4B and Qwen3‑8B checkpoints from 29.08/35.12 to 40.47/44.65, outperforming vanilla OPD and demonstrating the effectiveness of group‑relative residual calibration.
By Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou
arXiv:2605. 28149v2 Announce Type: replace Abstract: Sparse Autoencoders (SAEs) extract interpretable features from Large Language Model activations, but standard variants enforce non-negative latents, so a bidirectional semantic axis (e.
By Bartosz Wieciech, Zmnako Awrahman, Marcin Czelej, Victor Hugo Jaramillo Velasquez, Wioletta Stobieniecka
arXiv:2607. 19956v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined.
By Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela, Atia Haque Asha, Mourchona Afrin, Niloy Farhan, Farig Sadeque
The paper introduces Group-Calibrated On-Policy Distillation (GC‑OPD), a method that aligns token‑level teacher guidance with task‑level verifier rewards for long‑context reasoning. GC‑OPD normalizes verifier and OPD scores within rollout groups, uses their difference as a signed disagreement residual, and redistributes this residual across tokens via Relative‑Advantage‑Based Credit Assignment (RACA). Experiments on five long‑context benchmarks show that GC‑OPD improves Qwen3‑4B and Qwen3‑8B checkpoints from 29.08/35.12 to 40.47/44.65, outperforming vanilla OPD and demonstrating the effectiveness of group‑relative residual calibration.
Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.
The paper investigates selective on‑policy distillation, where a student model is trained only on token positions chosen by a selector. It demonstrates that the commonly used shared learning rate is not neutral: performance varies significantly with the learning rate for different selectors, leading to inconsistent comparisons. The authors attribute this selector‑rate entanglement to the selection process itself and recommend reporting the full arm‑by‑rate matrix for fair evaluation.
By Chencheng Zhu
arXiv:2608.20647v1 Announce Type: new
Abstract: Splitting a bidirectional LSTM's contextual representation into a forward-only $F_i$ (strictly a function of tokens $1..i$) and a backward-only $B_i$ (...
By Sai Krishna Arthanari, JaeHyeong Chang, Chengzhe Sun, Siwei Lyu
The paper investigates self‑distillation techniques for language models by systematically varying three key design choices: the source of rollout tokens (student vs. teacher), the teacher coupling strategy (frozen or exponential moving average), and the KL divergence direction (reverse or forward). Experiments on Qwen2.5‑7B and Ministral‑3‑3B across 1,200 adaptation runs reveal that rollout source mainly affects acquisition on contradictory tasks, teacher coupling most strongly influences acquisition across all tasks, and KL direction impacts retention differently depending on the model. A controlled theoretical model reproduces these empirical trends, offering a unified framework for understanding acquisition‑retention trade‑offs in self‑distillation.
By Luis Zuin, Alexis Huet, Dario Rossi, Zied Ben Houidi
arXiv:2609.39405v1 Announce Type: new
Abstract: Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced b...
By Jingang Zhou, Feiyu Han, Han Zhu, Yuyi Zhou, Ruiyang Zhang, Jian Xu, Sirui Gao, Qingpei Guo, Xu-Yao Zhang
SCOPE-OPSD introduces a Fisher‑conditioned privileged subspace for on‑policy self‑distillation (OPSD) that projects the teacher‑student residual onto a frozen rank‑64 factor derived from residual covariance and language‑model‑head Fisher sensitivity. The method adds no extra rollouts or inference modules and, across multiple Qwen3 model checkpoints and trajectory lengths, consistently matches or surpasses pure OPSD and a matched random baseline, achieving significant gains in most settings. A cross‑fitted diagnostic shows a 4.40‑fold increase in captured privileged‑gap compared to the random orientation.
By Yunmeng Chen (Chongqing Ant Consumer Finance Co., Ltd), Kunyu Wang (Alibaba Cloud Computing Co., Ltd), Peihan Li (Chongqing Ant Consumer Finance Co., Ltd), Yi Wang (Chongqing Ant Consumer Finance Co., Ltd), Shuyin Xia (Chongqing University of Posts and Telecommunications), Yi Liu (Chongqing Ant Consumer Finance Co., Ltd), Xinyong Cheng (Alibaba Cloud Computing Co., Ltd), Dehui Wang (Alibaba Cloud Computing Co., Ltd), Xiangyong Zhai (Alibaba Cloud Computing Co., Ltd), Yanxing Liu (Chongqing Ant Consumer Finance Co., Ltd), Song Liu (Chongqing Ant Consumer Finance Co., Ltd)
arXiv:2606. 16682v1 Announce Type: new Abstract: When AI agents use language models to evaluate their own outputs in a feedback loop, systematic biases emerge.
By Zewen Liu