The paper proposes a hierarchical framework for distilling large language model agents, distinguishing between the policy that generates interaction histories and the policy that produces token prefixes for actions. It explores various combinations of teacher-generated and student-generated data across multiple models and environments, finding that pairing student-induced histories with teacher-generated action prefixes yields the strongest training signal, while fully on-policy data is only beneficial when mixed with other configurations. The study also shows that the specific pairing of teacher and student proportions at each level significantly impacts performance, and that strategically front-loading teacher-generated contexts and delaying on-policy contexts improves results.
arXiv:2607. 04763v1 Announce Type: cross Abstract: We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories.
arXiv:2609.36608v1 Announce Type: new Abstract: On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-t...
arXiv:2606. 15912v1 Announce Type: cross Abstract: Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories.
The paper introduces SCOUT, a co‑training framework that adapts an off‑policy teacher to better continue from student‑generated prefixes in on‑policy distillation (OPD). By periodically optimizing the teacher’s conditional continuation ability using reinforcement learning with verifiable rewards, SCOUT improves the teacher’s performance on student prefixes. Experiments across various teacher‑student setups, model scales, and reasoning domains show that SCOUT consistently enhances the effectiveness of OPD.
arXiv:2606. 27814v4 Announce Type: replace Abstract: Training small language-model agents for long-horizon interactive tasks requires both fast imitation and reward-driven improvement.
DoubleFree is a method that reuses exploration trajectories from multi-task reinforcement learning to enable rollout‑free consolidation of large language models. By precomputing teacher supervision from historical data, it eliminates the need for online teachers during distillation. Experiments on Qwen3 and SmolLM models show speedups of 3–6× and GPU‑hour reductions of 81–90% while preserving performance across math, code, and instruction tasks.
arXiv:2608. 16333v1 Announce Type: cross Abstract: On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories.
arXiv:2609.39306v1 Announce Type: cross Abstract: Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our...
arXiv:2610.32802v1 Announce Type: new Abstract: On-policy distillation aligns teacher feedback with a student's current behavior, yet acquiring useful supervision for multi-turn agents requires ident...
Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our experiments with existing methods reveal a collaps...
The paper introduces Task-Progress Distillation (TPD), an offline method that pairs each action in teacher demonstrations with a short label indicating the current task stage. A small 1.7B student model trained with 404 demonstrations using TPD achieves 72.4% mean unseen task success on ALFWorld, matching action-only supervision and outperforming a reasoning-trained student. At lower demonstration budgets, explicit stage labels significantly boost performance, raising success from 48.0% to 67.7% with 200 demos, and both approaches converge to 76.9% with 808 demos.