arXiv Machine Learning
Jul 16
DarwinLM: Evolutionary Structured Pruning of Large Language Models
arXiv:2502. 07780v4 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved significant success across various NLP tasks.
GeLaCo: An Evolutionary Approach to Layer Compression
GeLaCo is an evolutionary method for compressing large language models by collapsing layers through parametrized weight merging. It uses population-based search with a fitness function that balances similarity of residual updates and language modeling KL divergence, enabling both single and multi-objective compression. The approach yields Pareto-optimal trade-offs between compression and quality, outperforming existing methods in perplexity and generative evaluations.
Related stories
arXiv AI
Jul 15
QDEvo: A Multi-Objective Quality-Diversity Framework for Automated Heuristic Design
arXiv:2607. 11916v1 Announce Type: cross Abstract: The integration of Large Language Models (LLMs) with evolutionary computation has emerged as a powerful paradigm for automated heuristic design in combinatorial optimization.
arXiv Machine Learning
Jul 21
NIRVANA: Structured Pruning Reimagined for Large Language Model Compression
arXiv:2509. 14230v2 Announce Type: replace Abstract: While structured pruning presents a highly effective pathway for accelerating Large Language Model (LLM) inference, existing methods frequently suffer from significant performance degradation and demand computationally retraining to recover capabilities.
arXiv AI
Sep 1
LaMoC: Loss-Aware Modular Compression for LLMs
arXiv:2608.30226v1 Announce Type: new Abstract: Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. H...
arXiv AI
Jul 22
Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression
arXiv:2607. 18284v1 Announce Type: cross Abstract: To excel at their domain large language models are comprised of billions of parameters.
arXiv Computation and Language
Aug 31
Pruning Laws for Large Language Models
arXiv:2504.04342v2 Announce Type: replace Abstract: Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly gro...
arXiv Machine Learning
Jul 30
Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
arXiv:2510. 00192v3 Announce Type: replace Abstract: Low-rank adaptation (LoRA) has become a widely used paradigm for parameter-efficient fine-tuning of large language models, yet its representational capacity often lags behind full fine-tuning.
arXiv Machine Learning
Jun 10
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
arXiv:2606. 09885v1 Announce Type: new Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts.
arXiv Machine Learning
Aug 14
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
arXiv:2608. 12953v1 Announce Type: cross Abstract: Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets.
arXiv Machine Learning
4d ago
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
OMP-MoE is a training‑free compression framework that prunes redundant experts in Mixture‑of‑Experts large language models by framing the problem as sparse signal reconstruction solved with Orthogonal Matching Pursuit. The method greedily selects expert contributions as dictionary atoms to minimize reconstruction error, then optimizes cross‑layer expert allocation via a water‑filling strategy, and finally introduces an adaptive inference mechanism (OMP‑MoE†) that dynamically adjusts expert activation based on energy prediction. Experiments on Qwen, DeepSeek‑V2, GPT‑OSS, and Mixtral MoE show consistent performance gains at 25‑50% pruning ratios, with Qwen3‑30B‑A3B retaining 93.3% of original performance at 50% compression while achieving significant speedups.
arXiv Machine Learning
Jun 5
Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM
arXiv:2510. 05544v2 Announce Type: replace-cross Abstract: Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment.
arXiv AI
Jul 22
Beyond Single-Dimensional Compression: The Compound Sparsity Frontier of Large Language Models
arXiv:2607. 18280v1 Announce Type: cross Abstract: Large language models (LLMs) are often compressed through static parameter pruning or dynamic token-level computation, yet aggressive sparsification can trigger rapid performance degradation beyond an essential sparsity boundary.