The paper introduces SCOUT, an offline multi-agent reinforcement learning framework that combines a generative behavioral prior with a decomposed value function for test-time action refinement. SCOUT uses optimal unified transport and Stein variational gradient descent to steer behavioral samples toward high-value regions, with the number of transport steps providing adaptive scaling. The authors prove a single-term KL bound on the joint soft-value gap under the individual-global-max principle and demonstrate that SCOUT outperforms existing methods on both discrete and continuous offline MARL benchmarks, including offline-to-online settings.
By Dongsu Lee, Haoran Xu, Amy Zhang
arXiv:2609.15883v1 Announce Type: cross
Abstract: Offline reinforcement learning aims to learn a policy solely from fixed datasets, which often contain multimodal action distributions. Flow policies...
By Jaehun Shon, Jinha Choi, Jongwook Jeon, Jongmin Lee
The paper introduces Δ‑MOPD, a multi‑teacher on‑policy distillation method that transfers each teacher’s logit shift relative to its base policy, re‑anchored at the student’s frozen initialization. Δ‑MOPD is evaluated in two settings: common‑domain composition, where multiple teachers score a single student rollout, and routed‑domain distillation, where prompts are assigned to specialist teachers. Across experiments, Δ‑MOPD outperforms traditional endpoint supervision, especially when teacher signals are combined at a state, achieving higher accuracy and reducing performance gaps in phased routing scenarios.
By Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard
arXiv:2610.10460v2 Announce Type: replace-cross
Abstract: Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollou...
By Yuanda Xu, Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard
Few-step flow-map generators, including MeanFlow and consistency models, enable efficient sampling through long-range transport, yet their on-policy distillation remains underexplored. We introduce Fl...
arXiv:2609.37851v1 Announce Type: cross
Abstract: Few-step flow-map generators, including MeanFlow and consistency models, enable efficient sampling through long-range transport, yet their on-policy...
By Zhiqi Li, Bo Zhu
The paper introduces VDOT++, a unified distillation framework for few-step video generation across text-to-video, image-to-video, and condition-based tasks. It extends the previous VDOT method by incorporating an asymmetric unbalanced optimal transport distillation that allows student tokens to carry less mass, and an α1 ground cost that uses a weighted median to preserve modes. The approach combines distribution matching with adversarial refinement and cross-scale distillation, achieving competitive performance on multiple video benchmarks with only four sampling steps.
By Yutong Wang, Xingtong Ge, Enhuai Liu, Yunke Wang, Tianfan Xue, Yu Qiao, Yaohui Wang, Xinyuan Chen, Chang Xu
The paper introduces Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation (Road MOPD), a method that adds non-routed teachers as auxiliary cross-domain critics to address gradient conflicts in multi-teacher distillation. It employs token-level credit assignment based on teacher agreement and preventive gradient projection to coordinate updates and reduce conflicts. Experiments show that Road MOPD improves performance, especially in high-conflict scenarios, and mitigates degradation in disadvantaged domains.
By Shi-Yu Tian, Yong Liu, Ximan Liu, Tianrui Zhu, Sule Bai, Xinwei Yang, Ruiqi Liu, Kun-Yang Yu, Ming Yang, Zhi Zhou, Yu-Feng Li, Yipeng Sun, Gu Simiu
The paper introduces RouteOPD, a new on‑policy distillation method that treats teacher‑student disagreement as a probability transport problem. By decomposing disagreement into excess sources and deficit destinations, RouteOPD pairs them explicitly and optimizes pairwise log‑odds toward targets derived from a bounded teacher potential. Experiments on four teacher‑student configurations and four mathematical‑reasoning benchmarks show that RouteOPD consistently outperforms sampled reverse‑KL OPD, achieving higher routing fidelity and lower background leakage.
By Tianle Xia, Lingxiang Hu, Yiding Sun, Linfang Shang, Ming Xu, Lan Xu, Ning Zheng, Wei Xu, Jie Jiang
arXiv:2609.14636v1 Announce Type: new
Abstract: On-policy distillation (OPD) has become a standard approach for transferring capabilities from large teachers to compact students. Its cost, however, i...
By Zhiyu Gui, Kexin Huang, Jia Guo, Junkang Wu, Zihao Wang, Zhiqiang Zhang, Jun Zhou, Jiancan Wu, Xiang Wang
arXiv:2609.36546v1 Announce Type: cross
Abstract: On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD ma...
By Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang
The paper introduces Counteraction-Aware Multi-Teacher On-Policy Distillation (CaMOPD), a method designed to recover general capabilities in large language models while preserving domain-specific behavior. CaMOPD tackles two failure modes of standard Multi-Teacher On-Policy Distillation—conflicting recovery and preservation gradients, and weak correction signals—by using decoupled alternating training and selecting samples with large teacher‑student log‑probability gaps. Experiments on role‑play dialogue and medical reasoning QA show that CaMOPD outperforms baselines in general capability recovery while maintaining domain specialization, and gradient coherence analyses confirm more coherent correction signals.
By Tianlei Chen, Jiao Ou, Ziyuan Liu, Ruiming Tang, Jian Liang, Han Li