arXiv AI

Emergent Specialization in Populations of Self-Supervised Collaborative Vision Experts Without a Shared Gate or Cross-Agent Gradients

arXiv Computation and Language
Nov 4

Multi-View Latent MoE: Letting Every Router Shape Expert Computation

The paper introduces Multi-View Latent MoE, a sparse mixture‑of‑experts architecture that uses multiple independent routers to generate richer selection signals and to modulate expert computation. By aggregating sigmoid scores from these routers, the model selects experts that are strongly favored by one or several routers and mixes their relative scores into each expert’s input, thereby reusing routing preferences for both selection and computation. Experiments on a 24‑layer 10B‑A1B model trained on 708 B tokens show a 1.18‑point improvement in nine‑task downstream mean over the baseline, with the Multi‑Router submodule alone providing a 0.6‑point gain and significantly better alignment between router scores and expert utility. whyItMatters":"The approach demonstrates that leveraging multiple routers can both improve expert selection quality and maintain computational efficiency, leading to measurable downstream performance gains on large‑scale language models."

By Zelin Zhao, Junyuan Shang, Tianchi Ma, Shuohuan Wang, Dianhai Yu
arXiv Machine Learning
Oct 3

From Task Mixtures to Specialized Experts

The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.

By Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari, M. Hadi Amini
arXiv Machine Learning
Sep 10

RAPTOR: Role-Aware Private Training for Mixture-of-Experts

arXiv:2609.05770v1 Announce Type: new Abstract: Differentially private (DP) fine-tuning methods treat sparse Mixture-of-Experts (MoE) models as a single dense block, ignoring that shared layers see a...

By Duc Dm, Khai Le-Duc, Nguyen Do, Minh Son Hoang, Florent Draye, Thai Hoang, Hoang Phuong Dam, Jiarui Liu, Chris Ngo, Terry Jingchen Zhang, Anh Le Duc Tran, Nhat Do Minh, Minh Ngoc Le, My T. Thai, Ran Xu, Silvio Savarese, Mona Diab, Bernhard Sch\"olkopf, Zhijing Jin, Huy L. Nguyen, Daeyoung Kim
arXiv AI
1d ago

From Local Prediction Error to Final Outputs: A Theory of Expert Sharing in Multi-Step Reasoning

The paper investigates how sharing experts in mixture‑of‑experts (MoE) models across layers or computational steps can reduce local prediction error but may not always improve final outputs. By analyzing linear regression models, the authors derive criteria that balance variance reduction against bias introduced by sharing across roles with different target behaviors, and they identify conditions under which local gains persist in the final output. The study also provides error bounds for general routed computations, demonstrates that correct predictions alone do not confirm expert attribution, and presents controlled experiments showing cases where local improvements reverse into higher final‑output error.

By Tiantong Wang, Zhiren Gong, Yixin Wang, Tiantong Wu, Wei Yang Bryan Lim
arXiv AI
Oct 5

Dynamic Expert Pruning for Multi-Agent Systems

Dynamic Expert Pruning for Multi-Agent Systems introduces DEP, a method that generates a per-request mask of experts based on an agent’s system and task prompts. Unlike static pruning, DEP uses a lightweight predictor trained on workflow transcripts to activate only the experts needed for each specific request, requiring no per-configuration calibration. Experiments across various tasks, models, and MoE architectures show DEP outperforms static pruning and merging baselines, especially when few experts are retained, and it generalizes to unseen workflows.

By Jabin Koo, Soheil Abbasloo, Sungjae Lee, Jungseul Ok
arXiv Machine Learning
Oct 22

The Reflexivity Threshold: A Phase Transition for Multi-Agent Performative Prediction

The paper introduces the reflexivity matrix Ω to capture how one agent’s deployment influences another’s best response in multi‑agent performative prediction. It identifies a spectral threshold—when the spectral radius of Ω exceeds one—separating stable learning from worst‑case learning obstruction. Below this threshold, repeated retraining converges geometrically to a unique stable equilibrium with bounded excess loss; above it, cumulative excess loss can grow linearly with the horizon.

By Ahmed Mahrous, Roberto Di Pietro