The paper revisits the impact of pruning on large language models (LLMs) during test-time scaling (TTS). While prior work found that structured pruning degrades reasoning performance, this study shows that unstructured pruning—removing only specific redundant weights—can actually improve TTS performance on reasoning benchmarks for models s1.1-7B and Qwen3-8B, sometimes surpassing the full-weight models. The authors also examine how different layer-wise sparsity allocation strategies affect these outcomes.
By Ocean Monjur, Shahriar Kabir Nahin, Anshuman Chhabra
arXiv:2510. 22228v2 Announce Type: replace-cross Abstract: Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs).
By Keyu Wang, Tian Lyu, Guinan Su, Lu Yin, Marco Canini, Jonas Geiping, Shiwei Liu
arXiv:2505. 12992v4 Announce Type: replace-cross Abstract: Inference-time scaling techniques have significantly bolstered the reasoning capabilities of large language models (LLMs) by harnessing additional computational effort at inference without retraining.
By Baohao Liao, Hanze Dong, Yuhui Xu, Doyen Sahoo, Christof Monz, Junnan Li, Caiming Xiong
Mixture-of-Experts (MoE) language models suffer from large parameter counts, which create a significant memory bottleneck. Expert pruning is the most direct approach for reducing this parameter count,...
The paper introduces a pruning method for test-time reasoning that directly optimizes pass@$k$ by selecting a pool of partial solutions that maximizes the chance of at least one correct final answer. This approach balances quality and diversity automatically, avoiding separate design choices for each. Experiments on eight open-weight reasoning models show that the method outperforms existing quality-based and diversity-based pruning strategies, especially when a moderate number of partial solutions are retained.
By Mahdi Salmani, Adel Javanmard, Devansh Gupta, Meisam Razaviyayn
The paper introduces HOPE, a second‑order pruning method for Mixture‑of‑Experts language models that accounts for cooperative interactions between experts. Unlike first‑order methods such as REAP, HOPE derives an objective that provably bounds pruning error and is shown to outperform baselines across three large MoE models, multiple calibration sets, and diverse benchmarks, especially at high pruning rates and on agentic tasks. The results demonstrate that preserving expert interactions allows aggressive compression with minimal performance loss on complex workloads.
By Alex M. Tseng, Prannay Kaul, Luca Zancato, Wei Xia, Stefano Soatto