The paper proposes a hierarchical framework for distilling large language model agents, distinguishing between the policy that generates interaction histories and the policy that produces token prefixes for actions. It explores various combinations of teacher-generated and student-generated data across multiple models and environments, finding that pairing student-induced histories with teacher-generated action prefixes yields the strongest training signal, while fully on-policy data is only beneficial when mixed with other configurations. The study also shows that the specific pairing of teacher and student proportions at each level significantly impacts performance, and that strategically front-loading teacher-generated contexts and delaying on-policy contexts improves results.
By Ya Gao, Zhengtao Zou, Pekka Marttinen
arXiv:2609.37522v1 Announce Type: new
Abstract: On-policy distillation (OPD) trains compact language agents with teacher feedback on student-generated trajectories. In multi-turn tasks, compounding e...
By Xiaohan Yi, Wen Luo, Yani Huang, Junfeng Zhan, Asher Qin, Peilin Zhao, Xi Xiao
The paper introduces Agent Distillation, a framework for transferring task‑solving knowledge from a teacher agent to a student agent. It categorizes where this knowledge is retained—within the model, as artifacts, through the execution harness, or across substrates—distinguishing transfer evidence from outcomes. An evaluation framework is proposed to link retention to causal contribution and practical utility, aiming to support reliable, maintainable, and safe agent development.
By Ziluowen Luo, Senzhang Wang, Chaozhuo Li, Jun Yin, Hao Yan, Ming Cheng, Chenxu Wang, Songyang Liu, Litian Zhang, Qiwei Ye, Zheng Liu, Philip S. Yu
The paper introduces Harness-Aware Distillation (HAD), a method for training smaller language model agents that preserves the surrounding harness—software managing context, tools, and feedback—while focusing distillation on the teacher’s contributions beyond the harness. HAD combines an action preference that contrasts teacher actions with and without harness information, and a validity check that filters out contradictory preference pairs. Experiments on long-horizon agent benchmarks show that HAD outperforms standard on‑policy distillation, reducing unproductive loops and improving error recovery without requiring task rewards or future information.
By Moonseok Choi, Taehong Moon, Giung Nam, Juho Lee
arXiv:2609.36608v1 Announce Type: new
Abstract: On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-t...
By Zubin Zheng, Jiahao Wu, Shaofeng Zhang, Zhirui Zhang, Yew-Soon Ong, Shengcai Liu
arXiv:2608. 01953v1 Announce Type: cross Abstract: On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference.
By Chishui Chen, Yaoyou Fan, Te Sun, Yi Yang, Chenghao Sun, Delin Mao, Hongbo Qiao, Zuowei Zhang, Junxi Wang, Chenxing Sun, Yangen Hu, Lu Pan, Xuyang Liu, Linfeng Zhang
The paper introduces EDV, an Execute‑Distill‑Verify framework that constructs offline experience for large language model agents without updating model parameters. In EDV, multiple heterogeneous agents execute tasks, a distiller extracts candidate lessons from their trajectories and feedback, and executors review these lessons, keeping only those unanimously approved. Experiments on τ²‑bench, Mind2Web, and MMTB demonstrate improved performance on revisited tasks and within‑domain transfer, while a Retail subset evaluation shows EDV cuts the Self‑Confirmation Trap Rate from 11.0% to 5.0%.
By Shiding Zhu, Yudi Qi, Yajie Wang, Jiaze Li, Chao Song, Yaorui Shi, Yibo Miao, Hanqi Gao, Kai Zhang
The study compares supervised fine-tuning (SFT) and reinforcement learning (RL) as teacher training methods for on-policy distillation (OPD) in a single-teacher setting. Using Qwen3.5-9B models across Agentic, Reasoning, and Perception domains, RL-trained teachers consistently produce stronger students and better recovery of performance gains than SFT-trained teachers. The authors attribute this advantage to RL teachers’ smaller parameter shifts from the shared initialization, which facilitates learning through OPD.
By Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu
arXiv:2610.19439v1 Announce Type: new
Abstract: While LLM-based agent systems have demonstrated remarkable proficiency in reasoning and solving complex real-world tasks, they also come with substanti...
By Yuanpu Cao, Saket Sathe, Hanyu Wang, Ziyi Yin, Fenglong Ma, Jinghui Chen
arXiv:2609.37044v1 Announce Type: new
Abstract: Explicit intermediate reasoning gives large language models (LLMs) a stronger problem-solving mode. We study learning from this think-mode advantage vi...
By Wanqi Ren, Jianxiang Wang, Danxuan Liu, Linyi Ding, Yuan Zhang
The paper introduces Task-Progress Distillation (TPD), an offline method that pairs each action in teacher demonstrations with a short label indicating the current task stage. A small 1.7B student model trained with 404 demonstrations using TPD achieves 72.4% mean unseen task success on ALFWorld, matching action-only supervision and outperforming a reasoning-trained student. At lower demonstration budgets, explicit stage labels significantly boost performance, raising success from 48.0% to 67.7% with 200 demos, and both approaches converge to 76.9% with 808 demos.
By Wenxi Gan
The paper examines on‑policy self‑distillation (OPSD) for multi‑turn agents, showing that using privileged information (PI) in the loss can make agents appear confident yet underperform plain RL, sometimes worse than the untrained base model. To address this, the authors propose Privileged Self‑Practice (PSP), which keeps PI in the prompt and uses it only during sampling, not in the loss. PSP consistently outperforms plain GRPO across AppWorld and SWE‑bench Verified, improving task‑goal completion by up to 65% and resolved rate by up to 61%.