The paper introduces Watermark Basin Alignment (WM‑Basin), a black‑box method for detecting distilled large language models (LLMs). It exploits a persistent watermark basin—a key‑specific geometric signal embedded in the teacher’s parameter space during fine‑tuning—that remains detectable even when token‑level signals vanish. WM‑Basin maps suspect model outputs to gradient representations, removes ordinary fine‑tuning directions, and tests alignment with the watermark basin, achieving significantly higher detection rates than token‑level baselines across a range of teacher and student models.
By Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez, Saeed Mahloujifar, Sylvestre-Alvise Rebuffi, Tom\'a\v{s} Sou\v{c}ek, Tuan Tran, Valeriu Lacatusu, Surya Parimi, Rashel Moritz, Vanessa Stark, Alexandre Mourachko
arXiv:2607. 04037v1 Announce Type: cross Abstract: On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits.
By Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi, Chanakya Ekbote, Sujay Sanghavi, Aleksandar Bojchevski
The paper introduces Harness-Aware Distillation (HAD), a method for training smaller language model agents that preserves the surrounding harness—software managing context, tools, and feedback—while focusing distillation on the teacher’s contributions beyond the harness. HAD combines an action preference that contrasts teacher actions with and without harness information, and a validity check that filters out contradictory preference pairs. Experiments on long-horizon agent benchmarks show that HAD outperforms standard on‑policy distillation, reducing unproductive loops and improving error recovery without requiring task rewards or future information.
By Moonseok Choi, Taehong Moon, Giung Nam, Juho Lee
arXiv:2607. 18110v1 Announce Type: new Abstract: Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles.
By Tianzhu Ye, Li Dong, Guanheng Chen, He Zhu, Xun Wu, Shaohan Huang, Furu Wei
arXiv:2608. 12764v1 Announce Type: cross Abstract: Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment.
By Haoze Wu, Chuqiao Kuang, Tianyi Zhuang, Xiaoguang Li
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
The paper introduces Iterative Calibration and Learning (IterCaL), a method for large language model distillation that mitigates teacher bias when target-domain reward feedback is unavailable. IterCaL alternates between calibrating the source-reward teacher via token-level branching and training the student on target prompts, achieving polynomial convergence to an oracle student. The authors prove that regularized direct teacher matching cannot reach the oracle, whereas IterCaL converges to it using only source rewards.
By Haichen Hu, Yuheng Zhang, David Simchi-Levi
arXiv:2609.01591v1 Announce Type: new
Abstract: AI tutors are most useful when they adapt to each student's strengths, weaknesses, and preferred guidance, but evidence about which guidance works for...
By Ke Yang, Chenglong Wang, Michel Galley, Chandan Singh, Jeevana Priya Inala, ChengXiang Zhai, Jianfeng Gao
arXiv:2608.29030v1 Announce Type: new
Abstract: In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus e...
By Yepeng Liu, Tianyi Chen, Xuandong Zhao, Dawn Song, Yuheng Bu
arXiv:2609.24646v1 Announce Type: new
Abstract: On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full d...
By Ahmed Khaled Khamis, Xiaotong Ji, Hassan Jaber, Rasul Tutunov, Matthieu Zimmer, Jun Wang, Haitham Bou-Ammar
arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.
By Changhui Sun, Lanbo Liu, Hang Lei, Tong Ling, Jiahang Xie, Zhiyong Zheng, Yujia Wang, Hao Liu, Feng Xiao, Lu Liu, Yanlong Du, Zifeng Cheng, Ziwei Jiang, Qing Gu
The paper introduces a two‑stage training framework for compact instruction‑following rerankers. Stage 1 strengthens a 4B teacher reranker with off‑policy GRPO using LLM‑judge feedback on 88K examples, while Stage 2 trains a 1B student by sampling its own rankings and receiving soft teacher‑derived rewards, blending exploration with knowledge transfer. The method achieves superior nDCG and MRR scores on MAIR‑11 and MAIR‑Full benchmarks, outperforming offline distillation baselines and larger RL‑trained rerankers.
By Vignesh Prabhakar, Jialing Pan, Anil Babu Ankisettipalli