arXiv AI
Jun 2
ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts
arXiv:2606. 01509v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.
Selective Sinkhorn Routing for Improved Sparse Mixture of Experts
arXiv:2511. 08972v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (SMoE) models are scalable and computationally efficient, enabling large increases in model capacity with limited inference overhead.
Related stories
arXiv AI
Jun 2
DOT-MoE: Differentiable Optimal Transport for MoEfication
arXiv:2606. 01666v1 Announce Type: cross Abstract: The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency.
arXiv Machine Learning
Aug 12
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
arXiv:2608. 10392v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.
arXiv Machine Learning
Jul 15
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
arXiv:2602. 19938v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently, delivering strong accuracy under fixed compute budgets.
arXiv AI
Aug 10
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.
arXiv AI
Jul 13
Sticky Routing: Training MoE Models for Memory-Efficient Inference
arXiv:2607. 08780v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices.
Hugging Face Trending Papers
Aug 5
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based methods aggregate routing statistics across all tokens to determine merge compatibility.
arXiv Machine Learning
Aug 6
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
arXiv:2608. 04454v1 Announce Type: cross Abstract: Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool.
arXiv AI
Jun 17
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
arXiv:2606. 17952v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) architectures enable scaling LLM parameters under a fixed inference budget by activating only a small subset of experts via top-$k$ routing.
arXiv AI
Jun 2
Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills
arXiv:2503. 05641v4 Announce Type: replace-cross Abstract: Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks.
arXiv Machine Learning
Jun 17
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
arXiv AI
Jun 15
Token-Level LLM Collaboration via FusionRoute
arXiv:2601. 05106v5 Announce Type: replace Abstract: Large language models (LLMs) exhibit strengths across diverse domains.