← Back to all news
arXiv Machine Learning By Zhentao Zhang, Yanzhao Zheng, Yanlin Liu, Yuanqiang Yu, Tianze Xu, Chao Ma, Jihuai Zhu, Lelun Wu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

Read the original on arXiv Machine Learning →

State-OPD introduces a state‑grounded on‑policy distillation framework that separates the objective task and action‑observation record from the student’s internal reasoning trajectory. By removing prior student reasoning and resampling complete reasoning‑action responses at high‑entropy states, the method improves teacher supervision and action diversity. Evaluations on ALFWorld, ScienceWorld, and WebShop show State‑OPD achieves the highest macro‑average success rates for Qwen3‑1.7B and Qwen3‑4B students, even surpassing a larger teacher model on certain tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 9

Stochastic Teacher Intervention for Agentic On-Policy Distillation

The paper introduces STI-OPD, a stochastic teacher intervention framework that improves on-policy distillation for multi-turn agentic tasks. By measuring teacher‑student policy discrepancy with KL divergence, STI-OPD probabilistically replaces student actions with teacher actions to provide reliable supervision. An Importance‑Weighted Reverse KL objective then corrects token sampling mismatches, enabling the student to learn from mixed‑policy trajectories and outperform prior OPD baselines across benchmarks.

By Junnan Liu, Linhao Luo, Zhijun Chen, Qianren Mao, Thuy-Trang Vu, Gholamreza Haffari
llmsagentsefficiencybenchmarks
More like this →
arXiv AI
Jun 16

On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

arXiv:2606. 15912v1 Announce Type: cross Abstract: Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.

By Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, Jinqiao Wang
agentsefficiency
More like this →
arXiv Computer Vision
Oct 1

Contrastive On-Policy Distillation

Contrastive On-Policy Distillation (COPD) is a framework that improves on-policy distillation by using a frozen teacher to evaluate student states under two contrasting prompts—one encouraging low reasoning effort and one encouraging high effort. The difference in log‑probabilities between these prompts provides a token‑level advantage signal that guides the student toward more concise and efficient reasoning strategies. Experiments on nine multimodal benchmarks show that COPD reduces reasoning length while maintaining task performance, and the contrastive approach can also be applied to on‑policy self‑distillation, allowing a model to compress its own reasoning without an external teacher.

By Jiacheng Ruan, Jun Tang, Wenzhen Yuan, Ting Liu, Shuai Bai, Dayiheng Liu, Zhibo Yang, Yuzhuo Fu
efficiencymultimodalbenchmarks
More like this →