The paper introduces Contrastive Set-Difference (CSD), a pruning method that targets neurons involved in redundant reasoning by contrasting activations on long versus short correct solutions. CSD is combined with an extended preference-based chain-of-thought compression that also considers problems a pruned model always or never solves. Applied to a 7B model, the approach reduces thinking tokens by 8–22% on math benchmarks while maintaining accuracy within 0.5 points compared to traditional pruning and compression pipelines.
By Tao Jing, Chen Kang, Feifei Zhai, Pengyuan Liu, Dong Yu
The paper introduces Causal Attribution Pruning (CAP), a training‑free technique that identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks. CAP estimates performance loss when each head is masked on a small calibration set, then translates these causal scores into weight‑level importance for pruning. Compared to magnitude‑only or activation‑based methods, CAP achieves up to 61% relative accuracy gains over Wanda on ARC‑Challenge at 20% sparsity and improves performance on GSM8K, StrategyQA, and ARC‑Challenge across Llama‑3‑8B‑Instruct and Mistral‑7B‑Instruct at 10%, 20%, and 50% sparsity.
By Amogh Sheth, Andrew Lin, Biruk Assefa, Yi Wen Huang, Yuhao Ge
arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.
By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang
Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but the...
arXiv:2607. 11089v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable success in complex reasoning tasks through Chain-of-Thought (CoT) prompting.
By Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias, Adam Jozefiak, Ciamac C. Moallemi
arXiv:2610.00571v1 Announce Type: cross
Abstract: Large reasoning models (LRMs) have achieved substantial improvements in solving complex mathematical problems, but often produce lengthy, repetitive,...
By Barproda Halder, Qiuyi Zhang, Sanghamitra Dutta