arXiv Computation and Language October 28, 2026 By Amogh Sheth, Andrew Lin, Biruk Assefa, Yi Wen Huang, Yuhao Ge

Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

Read the original on arXiv Computation and Language →

The paper introduces Causal Attribution Pruning (CAP), a training‑free technique that identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks. CAP estimates performance loss when each head is masked on a small calibration set, then translates these causal scores into weight‑level importance for pruning. Compared to magnitude‑only or activation‑based methods, CAP achieves up to 61% relative accuracy gains over Wanda on ARC‑Challenge at 20% sparsity and improves performance on GSM8K, StrategyQA, and ARC‑Challenge across Llama‑3‑8B‑Instruct and Mistral‑7B‑Instruct at 10%, 20%, and 50% sparsity.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Sep 17

OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning

OBC‑Prune introduces an outcome‑based calibration approach for pruning large reasoning models, focusing on the causal importance of each reasoning sentence rather than uniform activation salience. By pairing correct and incorrect rollouts and using intervention‑based analysis, it assigns per‑token weights that guide one‑shot pruning methods such as SparseGPT, Wanda, and ALPS. Experiments on DeepSeek‑R1‑Distill‑Qwen models show consistent accuracy gains and shorter reasoning traces across multiple benchmarks at 40‑50% sparsity.

By Ha Lan Nguyen, Huy Hoang Tran, Trac-Duy Tran, Dung D. Le

efficiency benchmarks

arXiv Computation and Language

Sep 22

LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator

arXiv:2609.22700v1 Announce Type: new Abstract: Step-level reasoning evaluators are commonly based on autoregressive language models, whose causal attention restricts each step representation to the...

By Yiming Feng, Naihao Deng, Yulong Chen, Rada Mihalcea

llms benchmarks

arXiv AI

Jul 14

Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

arXiv:2607. 11266v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet it often incurs substantial computational costs due to over-reasoning: the generation of redundant, verbose, or irrelevant steps.

By Daeyeop Lee, Hwanjo Yu

llms efficiency benchmarks

arXiv AI

Oct 1

Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

arXiv:2510.01581v2 Announce Type: replace-cross Abstract: Recent thinking models are capable of solving complex reasoning tasks by scaling test-time compute, but this scaling should be allocated in l...

By Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad, Elias Stengel-Eskin, Akshay Nambi, Mohit Bansal

reinforcement-learning

arXiv AI

Jul 2

Predicting LLM Reasoning Performance with Small Proxy Model

arXiv:2509. 21013v4 Announce Type: replace-cross Abstract: Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up.

By Woosung Koh, Juyoung Suk, Sungjun Han, Se-Young Yun, Jamin Shin

llms benchmarks safety

Hugging Face Trending Papers

Jun 1

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels.

llms reinforcement-learning benchmarks safety

One email a morning, machine-written

Related stories

OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning

LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator

Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

Predicting LLM Reasoning Performance with Small Proxy Model

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning