arXiv AI October 2, 2026 By Lianjun Liu, Shipeng Li, You Huang, Weiqi Yan, Mingte Qiu, Huazhong Liu, Xiaofeng Zhu, Yunshan Zhong

ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models

Read the original on arXiv AI →

ITC-MoE introduces an Importance-guided Token-aware Compression framework for Mixture-of-Experts Diffusion Language Models. It combines Adaptive Tucker Compression, which uses activation and gradient importance to jointly factorize expert weights and allocate ranks, with Token-aware Compensation and Routing that applies low‑rank adjustments to hot tokens and limits expert candidates for cold tokens. The method achieves significant reductions in computation and storage while maintaining generation quality, exemplified by a 30% compression budget that preserves 96.33% accuracy on MultiArith and delivers up to a 7.22× speedup.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 5

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

arXiv:2608. 03457v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood.

By Fengqi Zhu, Shaoxuan Xu, Jingyang Ou, Zebin You, Yipeng Xing, Huabin Liu, Xiaolu Zhang, Jun Zhou, Zhenzhong Lan, Yankai Lin, Wayne Xin Zhao, Jianguo Li, Chongxuan Li, Ji-Rong Wen

llms diffusion fine-tuning benchmarks

arXiv Machine Learning

4d ago

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...

By Ali Abbasi, Justin Shi, Soheil Kolouri

llms efficiency

arXiv AI

Jul 22

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

arXiv:2510. 02345v4 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead.

By Peijun Zhu, Ning Yang, Baoliang Tian, Jiayu Wei, Weihao Zhang, Haijun Zhang, Pin Lv

llms rag efficiency

arXiv AI

Jun 2

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

arXiv:2606. 00079v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation, but their deployment remains memory-intensive because all expert weights must be kept resident in memory.

By Jiayu Zhao, Zihan Teng, Minhao Fan, Tianrui Ma, Wentao Ren, Song Chen, Weichen Liu

llms efficiency

arXiv AI

Jul 7

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

arXiv:2511. 04805v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models have shown strong potential in scaling language models efficiently by activating only a small subset of experts per input.

By Yushu Zhao, Zheng Wang, Minjia Zhang

llms benchmarks

arXiv AI

Aug 11

Shape Mutating Expert Compression:LorExperts and BTExperts

arXiv:2608. 07814v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models deliver high capacity at low per-token compute, but deploying them cheaply requires compressing their many expert weight matrices.

By Inesh Chakrabarti, Sourjya Roy, Bowen Bao, Thiago Crepaldi, Spandan Tiwari, Ashish Sirasao

llms fine-tuning efficiency

One email a morning, machine-written

Related stories

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization

PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference

Shape Mutating Expert Compression:LorExperts and BTExperts