The paper argues that post‑training pruning of large language models should use a calibration set that covers multiple capability domains rather than a single source. A balanced four‑way mixture of sources consistently outperforms any single source across sparsity levels, model scales, and held‑out benchmarks. When aligned corpora are unavailable, the IGSP method approximates this mixture using only the base model and an evaluation taxonomy, achieving significant gains over prior self‑generation approaches.
By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
arXiv:2609.30465v1 Announce Type: cross
Abstract: Mixture-of-experts (MoE) models activate few experts per token but store the full expert pool. Expert pruning reduces this storage burden; at a fixed...
By Mingyang Song, Mao Zheng
arXiv:2608.29765v1 Announce Type: new
Abstract: Structured pruning uses surrogate objectives because direct task evaluation over every feasible mask is too expensive. Most evaluations report average...
By Hao Ye, Gaopeng Zhang
arXiv:2608.23638v1 Announce Type: cross
Abstract: Two analysts who calibrate the same predictive model on independent samples will deploy different prediction sets every time, because the calibration...
By Marios Papamichalis, Regina Ruane, Theofanis Papamichalis
arXiv:2607. 18088v1 Announce Type: new Abstract: Standard evaluation of many recognition systems contains distribution shift by construction, since benchmarks place disjoint conditions in the training and test splits.
By Weijia Han, Lisha Qu
arXiv:2606. 03328v3 Announce Type: replace Abstract: Calibration data are often treated as a minor implementation detail in post-training LLM pruning because averaged evaluations suggest only modest effects.
By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
arXiv:2606. 02608v1 Announce Type: new Abstract: We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets.
By Leonid Berlyand, Theo Bourdais, Houman Owhad, Yitzchak Shmalo
arXiv:2311.04902v3 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurt...
By Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma, Zhiqiang Shen
arXiv:2606. 03328v1 Announce Type: cross Abstract: Post-training pruning compresses large language models to high sparsity using a small unlabelled calibration set, and recent work has concluded that the choice of calibration source has only modest impact on averaged post-pruning accuracy.
By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
The study investigates how the composition of data during the mid‑training phase of language models affects performance across multiple domains. Experiments with Qwen3‑8B‑Base on five distinct KOR‑Bench domains show that moderate coverage (10%‑40%) yields the best per‑domain results, and that alignment passes cannot fully close the performance gaps created by mid‑training data choices. Additionally, zero coverage in mid‑training severely degrades accuracy, while a carefully tuned allocation can provide the largest overall pipeline improvement.
By Yunpeng Xu, Kun Zheng
arXiv:2605.06240v2 Announce Type: replace-cross
Abstract: Forward-Forward (FF) training lets each layer learn from a local goodness criterion. In cumulative-goodness variants, later layers can inheri...
By Amirhossein Yousefiramandi
Magnitude Profile Pruning introduces a training‑free, calibration‑free method for removing attention heads in Transformer models by statistically detecting outliers in weight row norms. Heads whose projection weights fall within the bulk of the distribution are pruned, while outlier heads are retained. Across several models, the MP‑G variant achieves superior perplexity at various sparsity levels and yields significant parameter and FLOP reductions without requiring forward passes, calibration data, or gradient computations.
By Kasun Dewage, Marianna Pensky, Heranga K. Rathnasekara, Suranadi De Silva