The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
arXiv:2605. 15888v2 Announce Type: replace-cross Abstract: Heterogeneous Graph Prompt Learning (HGPL)has emerged as a promising paradigm for bridging the gap between the objectives of pre-training foundation models and their downstream applications in heterogeneous graph settings.
By Peiyuan Li, Yongqi Huang, Jitao Zhao, Dongxiao He, Di Jin, Weixiong Zhang
The paper investigates whether the sparsity of Mixture-of-Experts (MoE) models leads to intrinsic semantic organization across modalities and domains. It shows that experts naturally specialize semantically even without explicit modular training. The authors propose ExpertLens, a data‑free method that decodes router weights to identify domain‑specialized experts, enabling selective fine‑tuning that matches or exceeds full fine‑tuning while updating only 21.7–47.0% of parameters and achieving a 4.0× speedup, outperforming LoRA in both performance and efficiency.
By Damiano Marsili, Raphi Kang, Aditya Mehta, Pietro Perona, Georgia Gkioxari
The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.
By Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari, M. Hadi Amini
arXiv:2608. 16384v1 Announce Type: cross Abstract: Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules.
By Suraj Yadav
arXiv:2607. 26458v1 Announce Type: cross Abstract: Domain generalization (DG) aims to learn from multiple source domains and generalize to unseen target domains.
By Yuhang Jiang, Fengchuan Zhang, Sanguo Zhang, Guojun Zhu
The paper proposes a fairness-aware Mixture-of-Experts (MoE) framework that tackles routing-induced bias by applying subgroup reweighting to correct data imbalance and gate entropy regularization to prevent the gating network from collapsing onto subgroup attributes. This end-to-end approach keeps expert utilization balanced and interpretable, offering a clear view of how subgroups are allocated across experts. Experiments show that the method improves fairness while maintaining competitive predictive performance.
By Sunhee Hwang
arXiv:2609.25692v1 Announce Type: new
Abstract: Graph domain adaptation (GDA) transfers knowledge from a labeled source graph to an unlabeled target graph under shifts in both node attributes and gra...
By Ziqian Liu, Yongxue Xu, Enze Zhang, Jiaqi Zhang, Hao Wang, Maolin Wang
arXiv:2608.24195v1 Announce Type: cross
Abstract: Mixture-of-Experts (MoE) models provide a flexible framework for partitioning complex prediction problems into simpler local learning tasks through a...
By Soham Chatterjee, Rwitobroto Dey, Smarajit Bose
arXiv:2606. 08691v1 Announce Type: new Abstract: Modern data-driven applications increasingly involve learning from multiple heterogeneous sources, where a target dataset is limited but related information is available across domains.
By Samhita Pal, Tian Gu
arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.
By Heng Zhao, Zilei Shao, Guy Van den Broeck, Zhe Zeng
arXiv:2610.00676v1 Announce Type: cross
Abstract: Unsupervised skill discovery has emerged as a promising approach for leveraging reward-free datasets to pretrain general-purpose policies. However, c...
By Mohammad Amin Abbasfar, Farbod Azimmohseni, Mohammad Hossein Rohban