The paper investigates conditional calibration contracts for switched inference systems, proposing a branch‑aware audit framework called CA‑MHD that integrates load‑coupled distillation, closed‑loop replay, and a cost‑informed Schmitt router. Experiments across 40 cells reveal that the deployed wide branch consistently underperforms the original teacher in ordinal accuracy, while student‑only serving shows mixed improvements and routing exposes significant violation tails. Retrospective temporal recalibration narrows the ordinal gap in most cells, yet causal delayed‑label verification and incremental per‑mode benefits remain unproven.
By Juan S. Santillana
MOPD‑Router rethinks teacher routing in multi‑teacher on‑policy distillation by routing supervision over the full teacher pool at each token, eliminating the need for prompt‑level domain labels or a separate routing model. The framework offers a plug‑in interface for various metrics, and introduces ExpertAlign, which scores teachers based on how well their corrections reflect their specialized post‑training knowledge. Experiments on both unlabeled and domain‑labeled mixtures show that ExpertAlign outperforms existing methods, improving overall scores by up to 12.3% on unlabeled data and 7.8% on domain‑labeled data.
whyItMatters":"Token‑level routing enables the use of complementary supervision across domains without relying on domain labels, leading to significant performance gains in multi‑teacher distillation settings."
By Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu
MetroLLM-Bench is a 955‑case benchmark designed to evaluate language models as the policy layer of transit kiosks across six real metro systems, covering routing, fare calculation, disruptions, accessibility, and adversarial input. The benchmark includes 14 deterministic scoring components (Tier 1) and 8 semantic‑quality components (Tier 2), with a 75/25 split for training‑data generation and held‑out evaluation. Twenty‑six models from six vendors were tested, and a 4B Qwen 3.5 student fine‑tuned via PEFT outperformed GPT‑5.6 on Tier 1 and matched GPT‑5.4 on the combined score, while larger models offered no further improvement.
By Remco Hendriks (Continker)
arXiv:2609.36546v1 Announce Type: cross
Abstract: On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD ma...
By Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang
The paper introduces Δ‑MOPD, a multi‑teacher on‑policy distillation method that transfers each teacher’s logit shift relative to its base policy, re‑anchored at the student’s frozen initialization. Δ‑MOPD is evaluated in two settings: common‑domain composition, where multiple teachers score a single student rollout, and routed‑domain distillation, where prompts are assigned to specialist teachers. Across experiments, Δ‑MOPD outperforms traditional endpoint supervision, especially when teacher signals are combined at a state, achieving higher accuracy and reducing performance gaps in phased routing scenarios.
By Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard
The paper presents a post‑training method for dynamically routed mixture‑of‑experts (MoE) models that integrates independently optimized specialists efficiently. It introduces calibrated multi‑teacher on‑policy distillation (C‑MOPD) with teacher‑wise loss normalization and clipping, combined with a post‑capacity bounded‑width dispatch, final‑route replay, and asynchronous rollout–training overlap. On the Turing‑20B‑A2B model, the approach achieves a high integration score (0.9988), retains most specialist gains, outperforms shared SFT and Qwen3‑8B on reasoning‑enabled benchmarks, and reduces consolidation time by 2.86×.
By Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Shuang Su, Siqi Liu, Kai Yang, Weichao Huang, Hang Zhang, Honggou Yang, Xianming Liu