arXiv Machine Learning September 30, 2026 By Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini, Philip Levis

Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

Read the original on arXiv Machine Learning →

The paper introduces Federation of Experts (FoE), a new architecture that reorganizes the mixture-of-experts (MoE) block in transformer layers into multiple MoE clusters. Each cluster handles a single KV head, and expert parallelism is applied within clusters while a sum operation synchronizes post‑attention residuals across clusters. FoE eliminates all‑to‑all communication on a single GPU and limits it to intra‑node communication in multi‑node setups, leading to significant reductions in inference latency and throughput improvements on LongBench.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 18

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.

By Lorenzo Sani, Zeyu Cao, Meghdad Kurmanji, Alex Iacob, Andrej Jovanovic, Yan Gao, Wanru Zhao, Nicholas D. Lane

llms efficiency benchmarks

arXiv Machine Learning

Jun 16

A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference

arXiv:2606. 15453v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost.

By Yingnan Zhao, Razvan Bunescu, Ahmed Louri, Avinash Karanth, Ke Wang

llms

arXiv AI

Sep 15

Task-Aware Federated Fine-Tuning for MoE-based Large Language Models

The paper introduces FedTAR, a task-aware federated fine‑tuning approach for Mixture‑of‑Experts (MoE) large language models. FedTAR links local client updates to task preferences using routing outputs and Singular Value Decomposition to extract low‑dimensional task coordinates and update directions. It then aggregates updates within and across task clusters, reconstructing the final update to preserve expert specialization and reduce interference, achieving state‑of‑the‑art performance on four benchmark tasks under non‑IID settings.

By Tingqi Wang, Hongyu Ke, Haoxin Wang, Rafal Angryk, Zhipeng Cai

llms fine-tuning benchmarks

arXiv Machine Learning

Sep 2

DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference

arXiv:2609.00407v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling of large language model (LLM) inference but suffer from substantial data-movement overhead w...

By Xiaoyang Lu, Belthangady Akash Vi Narayana Pai, Xian-He Sun

llms efficiency benchmarks

arXiv AI

Jul 23

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.

By Minyu Cui, Anna Wingkvist, Morgan Ericsson

llms benchmarks

arXiv AI

Jul 22

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

arXiv:2510. 02345v4 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead.

By Peijun Zhu, Ning Yang, Baoliang Tian, Jiayu Wei, Weihao Zhang, Haijun Zhang, Pin Lv

llms rag efficiency

One email a morning, machine-written

Related stories

FoMoE: Breaking the Full-Replica Barrier with a Federation of MoEs

A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference

Task-Aware Federated Fine-Tuning for MoE-based Large Language Models

DynaNDE: Dynamic Near-Data Expert Scheduling for Batched MoE Inference

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression