← Back to all news
arXiv AI By Ya Gao, Zhengtao Zou, Pekka Marttinen

Beyond On-Policy vs. Off-Policy: At What Level Should LLM Agent Distillation Follow the Student?

Read the original on arXiv AI →

The paper proposes a hierarchical framework for distilling large language model agents, distinguishing between the policy that generates interaction histories and the policy that produces token prefixes for actions. It explores various combinations of teacher-generated and student-generated data across multiple models and environments, finding that pairing student-induced histories with teacher-generated action prefixes yields the strongest training signal, while fully on-policy data is only beneficial when mixed with other configurations. The study also shows that the specific pairing of teacher and student proportions at each level significantly impacts performance, and that strategically front-loading teacher-generated contexts and delaying on-policy contexts improves results.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 16

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

arXiv:2606. 15912v1 Announce Type: cross Abstract: Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.

By Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, Jinqiao Wang
agentsefficiency
More like this →
Hugging Face Trending Papers
Jul 6

Multi-Turn On-Policy Distillation with Prefix Replay

We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories.

llmsagentsefficiency
More like this →
arXiv AI
Oct 1

On the Off-Policy Teacher in On-Policy Distillation

The paper introduces SCOUT, a co‑training framework that adapts an off‑policy teacher to better continue from student‑generated prefixes in on‑policy distillation (OPD). By periodically optimizing the teacher’s conditional continuation ability using reinforcement learning with verifiable rewards, SCOUT improves the teacher’s performance on student prefixes. Experiments across various teacher‑student setups, model scales, and reasoning domains show that SCOUT consistently enhances the effectiveness of OPD.

By Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Bl\"obaum, Purak Jain
reinforcement-learningefficiency
More like this →