arXiv AI September 30, 2026 By Yury Nahshan, Nati Daniel, Jacob Goldberger, Yoli Shavit

Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

The paper introduces two token-error supervision methods for sparse mixture-of-experts (MoE) large language models, aligning routing affinities with token-level cross-entropy loss. The first method predicts an error score per expert and uses it to adjust affinities before top‑K selection, while the second directly aligns router affinities to the model’s objective without extra heads or inference changes. Experiments on Granite and ARC‑Challenge show accuracy gains of about 2.3–2.94 percentage points over a parameter‑matched baseline, preserving the native sparse execution budget.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 3

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

arXiv:2512. 13996v3 Announce Type: replace Abstract: Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs.

By Can Jin, Hongwu Peng, Mingcan Xiang, Qixin Zhang, Xiangchi Yuan, Amit Hasan, Ohi Dibua, Yifan Gong, Yan Kang, Dimitris N. Metaxas

llms diffusion efficiency

arXiv AI

Aug 10

TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation

arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.

By Guanzhi Deng, Haibo Wang, Kuan Wu, Xiangru Jian, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 5

Selective Sinkhorn Routing for Improved Sparse Mixture of Experts

arXiv:2511. 08972v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (SMoE) models are scalable and computationally efficient, enabling large increases in model capacity with limited inference overhead.

By Duc Anh Nguyen, Huu Binh Ta, Nhuan Le Duc, Tan Minh Nguyen, Toan Tran

computer-vision

arXiv AI

Jun 15

Token-Level LLM Collaboration via FusionRoute

arXiv:2601. 05106v5 Announce Type: replace Abstract: Large language models (LLMs) exhibit strengths across diverse domains.

By Nuoya Xiong, Yuhang Zhou, Hanqing Zeng, Zhaorun Chen, Furong Huang, Shuchao Bi, Lizhu Zhang, Zhuokai Zhao

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 11

Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts

arXiv:2608. 08853v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs.

By Zongfei Li

arXiv AI

Jul 28

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

arXiv:2411. 00918v5 Announce Type: replace-cross Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.

By Nam V. Nguyen, Thong T. Doan, Luong Tran, Van Nguyen, Quang Pham

llms benchmarks

One email a morning, machine-written

Related stories

DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training

TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation

Selective Sinkhorn Routing for Improved Sparse Mixture of Experts

Token-Level LLM Collaboration via FusionRoute

Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models