arXiv AI

OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning

OBC‑Prune introduces an outcome‑based calibration approach for pruning large reasoning models, focusing on the causal importance of each reasoning sentence rather than uniform activation salience. By pairing correct and incorrect rollouts and using intervention‑based analysis, it assigns per‑token weights that guide one‑shot pruning methods such as SparseGPT, Wanda, and ALPS. Experiments on DeepSeek‑R1‑Distill‑Qwen models show consistent accuracy gains and shorter reasoning traces across multiple benchmarks at 40‑50% sparsity.

arXiv AI

6d ago

Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning

The paper introduces Contrastive Set-Difference (CSD), a pruning method that targets neurons involved in redundant reasoning by contrasting activations on long versus short correct solutions. CSD is combined with an extended preference-based chain-of-thought compression that also considers problems a pruned model always or never solves. Applied to a 7B model, the approach reduces thinking tokens by 8–22% on math benchmarks while maintaining accuracy within 0.5 points compared to traditional pruning and compression pipelines.

By Tao Jing, Chen Kang, Feifei Zhai, Pengyuan Liu, Dong Yu

efficiency benchmarks

arXiv Computation and Language

5d ago

Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

The paper introduces Causal Attribution Pruning (CAP), a training‑free technique that identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks. CAP estimates performance loss when each head is masked on a small calibration set, then translates these causal scores into weight‑level importance for pruning. Compared to magnitude‑only or activation‑based methods, CAP achieves up to 61% relative accuracy gains over Wanda on ARC‑Challenge at 20% sparsity and improves performance on GSM8K, StrategyQA, and ARC‑Challenge across Llama‑3‑8B‑Instruct and Mistral‑7B‑Instruct at 10%, 20%, and 50% sparsity.

By Amogh Sheth, Andrew Lin, Biruk Assefa, Yi Wen Huang, Yuhao Ge

llms efficiency benchmarks

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

Hugging Face Trending Papers

Oct 25

Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning

Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but the...

efficiency benchmarks

arXiv AI

Jul 14

OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping

arXiv:2607. 11089v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved remarkable success in complex reasoning tasks through Chain-of-Thought (CoT) prompting.

By Mohammed Ehab, Aymane El Gadarri, Vivek F. Farias, Adam Jozefiak, Ciamac C. Moallemi

llms efficiency benchmarks

arXiv AI

Oct 2

Interpreting Reasoning of Large Language Models via Partial Information Decomposition

arXiv:2610.00571v1 Announce Type: cross Abstract: Large reasoning models (LRMs) have achieved substantial improvements in solving complex mathematical problems, but often produce lengthy, repetitive,...

By Barproda Halder, Qiuyi Zhang, Sanghamitra Dutta

llms rag fine-tuning safety

arXiv AI

Oct 1

Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

arXiv:2510.01581v2 Announce Type: replace-cross Abstract: Recent thinking models are capable of solving complex reasoning tasks by scaling test-time compute, but this scaling should be allocated in l...

By Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad, Elias Stengel-Eskin, Akshay Nambi, Mohit Bansal

reinforcement-learning

arXiv AI

Jun 30

Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning

arXiv:2508. 02178v3 Announce Type: replace Abstract: Large reasoning models (LRMs) often exhibit overthinking, producing verbose Chain-of-Thought (CoT) traces that increase inference cost and obscure the underlying reasoning process.

By Taihang Zhen, Jialiang Hong, Kai Chen, Guang Yang, Junlan Feng, Wenpeng Zhu, Jing Huo, Yang Gao, Depeng Wang, Haitao Wan, Xi Yang, Fanyu Meng, Yuyao Zhang, Ji Qi, Xiangyu Zhou

reinforcement-learning

arXiv Computation and Language

Sep 22

STOP: Structured On-Policy Pruning of Long-Form Reasoning in Low-Data Regimes

arXiv:2605.13165v2 Announce Type: replace Abstract: Long chain-of-thought (Long CoT) reasoning improves performance on multi-step problems, but it also induces overthinking. This inefficiency is espe...

By Chenjun Xu, Zhennan Zhou, Zhan Su, Bill Howe, Lucy Lu Wang, Bingbing Wen

llms computer-vision fine-tuning efficiency

arXiv Computation and Language

Oct 8

SEER: Self-Enhancing Chain-of-Thought Compression for Reasoning Models

arXiv:2509.14093v3 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) prompting can substantially improve the reasoning ability of large language models (LLMs), but it often comes with hig...

By Kerui Huang, Shuhan Liu, Xing Hu, Tongtong Xu, Lingfeng Bao, Xin Xia

llms benchmarks

arXiv AI

Aug 25

Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling

The paper revisits the impact of pruning on large language models (LLMs) during test-time scaling (TTS). While prior work found that structured pruning degrades reasoning performance, this study shows that unstructured pruning—removing only specific redundant weights—can actually improve TTS performance on reasoning benchmarks for models s1.1-7B and Qwen3-8B, sometimes surpassing the full-weight models. The authors also examine how different layer-wise sparsity allocation strategies affect these outcomes.

By Ocean Monjur, Shahriar Kabir Nahin, Anshuman Chhabra

llms efficiency benchmarks

arXiv Computation and Language

Oct 15

To Reason but Not to Stop: Understanding and Recovering Pruned Large Reasoning Models

arXiv:2610.19934v1 Announce Type: new Abstract: Weight pruning can degrade the accuracy of large reasoning models (LRMs), yet it remains unclear whether it damages reasoning itself. We find that much...

By Minh-Anh Nguyen, Nguyen T. Ha Lan, Xiyan Fu, Dung D. Le, Wei Lu

efficiency benchmarks

Related stories