arXiv AI
Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
Aug 31
Pruning Laws for Large Language Models
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
Hugging Face Trending Papers
Jul 2
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns.
arXiv Machine Learning
Jun 17
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
arXiv:2603. 18492v3 Announce Type: replace Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead.
arXiv AI
Jul 3
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
arXiv:2607. 01710v1 Announce Type: new Abstract: Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging.
arXiv Machine Learning
Sep 14
ESTS at WMT26: Routing-Informed Expert Pruning for Model Compression
The paper reports six submissions by the ESTS team to the WMT26 Model Compression Shared Task for English–Simplified Chinese and English–Egyptian Arabic. Each submission offers three compression operating points derived from GPT‑OSS‑20B, using routing‑informed expert pruning, cross‑lingual routing divergence for capacity allocation, and MXFP4 quantization of retained expert projection weights. The resulting models, ranging from 4.186 B to 7.770 B parameters, are fine‑tuned on GPT‑5.1 synthetic data and evaluated internally with xCOMET‑XL.
arXiv Machine Learning
Jul 14
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
arXiv:2605. 18331v2 Announce Type: replace Abstract: Large Language Models (LLMs) have experienced significant growth and development in recent years.