The paper introduces Multi-View Latent MoE, a sparse mixture‑of‑experts architecture that uses multiple independent routers to generate richer selection signals and to modulate expert computation. By aggregating sigmoid scores from these routers, the model selects experts that are strongly favored by one or several routers and mixes their relative scores into each expert’s input, thereby reusing routing preferences for both selection and computation. Experiments on a 24‑layer 10B‑A1B model trained on 708 B tokens show a 1.18‑point improvement in nine‑task downstream mean over the baseline, with the Multi‑Router submodule alone providing a 0.6‑point gain and significantly better alignment between router scores and expert utility.
whyItMatters":"The approach demonstrates that leveraging multiple routers can both improve expert selection quality and maintain computational efficiency, leading to measurable downstream performance gains on large‑scale language models."
By Zelin Zhao, Junyuan Shang, Tianchi Ma, Shuohuan Wang, Dianhai Yu
The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.
By Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari, M. Hadi Amini
arXiv:2609.05770v1 Announce Type: new
Abstract: Differentially private (DP) fine-tuning methods treat sparse Mixture-of-Experts (MoE) models as a single dense block, ignoring that shared layers see a...
By Duc Dm, Khai Le-Duc, Nguyen Do, Minh Son Hoang, Florent Draye, Thai Hoang, Hoang Phuong Dam, Jiarui Liu, Chris Ngo, Terry Jingchen Zhang, Anh Le Duc Tran, Nhat Do Minh, Minh Ngoc Le, My T. Thai, Ran Xu, Silvio Savarese, Mona Diab, Bernhard Sch\"olkopf, Zhijing Jin, Huy L. Nguyen, Daeyoung Kim
arXiv:2606. 19633v1 Announce Type: cross Abstract: Perceptive legged locomotion over discontinuous terrain (e.
By Francisco Affonso, Matheus P. Angarola, Ana Luiza Mineiro, Aditya Potnis, Marcelo Becker, Girish Chowdhary
arXiv:2610.16895v1 Announce Type: new
Abstract: LLMs are used as high-level controllers for embodied agents, but it remains unclear whether they support collaborative reasoning, such as anticipating...
By Adri\'an Szlatincs\'an, Filippo Ziliotto, Bruno Lepri, Luciano Serafini, Massimiliano Luca, Tommaso Campari
The paper investigates how sharing experts in mixture‑of‑experts (MoE) models across layers or computational steps can reduce local prediction error but may not always improve final outputs. By analyzing linear regression models, the authors derive criteria that balance variance reduction against bias introduced by sharing across roles with different target behaviors, and they identify conditions under which local gains persist in the final output. The study also provides error bounds for general routed computations, demonstrates that correct predictions alone do not confirm expert attribution, and presents controlled experiments showing cases where local improvements reverse into higher final‑output error.
By Tiantong Wang, Zhiren Gong, Yixin Wang, Tiantong Wu, Wei Yang Bryan Lim
Dynamic Expert Pruning for Multi-Agent Systems introduces DEP, a method that generates a per-request mask of experts based on an agent’s system and task prompts. Unlike static pruning, DEP uses a lightweight predictor trained on workflow transcripts to activate only the experts needed for each specific request, requiring no per-configuration calibration. Experiments across various tasks, models, and MoE architectures show DEP outperforms static pruning and merging baselines, especially when few experts are retained, and it generalizes to unseen workflows.
By Jabin Koo, Soheil Abbasloo, Sungjae Lee, Jungseul Ok
arXiv:2609.37250v1 Announce Type: cross
Abstract: World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrai...
By Yang Zhang, Jiangyuan Zhao, Chenyou Fan, Jiayu Hu, Xiu Yuan, Chenjia Bai, Xiu Li
Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entangle many concepts in each neuron. Feature superposi...
arXiv:2605.01457v4 Announce Type: replace
Abstract: How can generative offline multi-agent reinforcement learning achieve both fast joint trajectory generation and effective cooperation? Multi-agent...
By Guowei Zou, Haitao Wang, Beiwen Zhang, Boning Zhang, Hejun Wu
The paper introduces the reflexivity matrix Ω to capture how one agent’s deployment influences another’s best response in multi‑agent performative prediction. It identifies a spectral threshold—when the spectral radius of Ω exceeds one—separating stable learning from worst‑case learning obstruction. Below this threshold, repeated retraining converges geometrically to a unique stable equilibrium with bounded excess loss; above it, cumulative excess loss can grow linearly with the horizon.
By Ahmed Mahrous, Roberto Di Pietro
arXiv:2609.24379v1 Announce Type: cross
Abstract: Mechanistic interpretability of vision transformers seeks to decompose model computation into human-readable units, but learned representations entan...
By Gautam Ranka, Shubham Santosh Pandere, Aiden Dsouza