← Back to all news
arXiv AI By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu

Rethinking Calibration for High-Sparsity LLM Pruning: From Single Sources to Capability Coverage

Read the original on arXiv AI →

The paper argues that post‑training pruning of large language models should use a calibration set that covers multiple capability domains rather than a single source. A balanced four‑way mixture of sources consistently outperforms any single source across sparsity levels, model scales, and held‑out benchmarks. When aligned corpora are unavailable, the IGSP method approximates this mixture using only the base model and an evaluation taxonomy, achieving significant gains over prior self‑generation approaches.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Jun 3

Calibration Data Trade-offs Across Capability Dimensions: Why Multi-Source Mixing Matters for High-Sparsity LLM Pruning

arXiv:2606. 03328v1 Announce Type: cross Abstract: Post-training pruning compresses large language models to high sparsity using a small unlabelled calibration set, and recent work has concluded that the choice of calibration source has only modest impact on averaged post-pruning accuracy.

By Hu Xu, Zhaolong Xing, Congcong Liu, Jiaxing Wang, Zhida Jiang, Junshi Huang, Zhen Chen, Jianfeng Xu
llmsefficiency
More like this →
arXiv Machine Learning
Oct 5

Capability Scaling-Down Laws for LLM Compression

The paper presents a systematic study of how different compression techniques—pruning, quantization, and distillation—affect the capabilities of large language models (LLMs) in tasks such as mathematics, code generation, and question answering. It introduces a framework that measures capability loss and relates it to factors like model size, training stage, and compression settings, yielding simple predictive relations that generalize across unseen configurations. The authors demonstrate that sharing density responses across pruning levels can dramatically reduce the number of measurements needed, and that their predictive models closely match regression results while offering efficient decision guidance for compression method selection.

By Xueqi Cheng, Liang Wu, Kelly Wan, Liangjie Hong, Yushun Dong
llmsnlpefficiency
More like this →