arXiv Computation and Language

Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models

The paper introduces Causal Attribution Pruning (CAP), a training‑free technique that identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks. CAP estimates performance loss when each head is masked on a small calibration set, then translates these causal scores into weight‑level importance for pruning. Compared to magnitude‑only or activation‑based methods, CAP achieves up to 61% relative accuracy gains over Wanda on ARC‑Challenge at 20% sparsity and improves performance on GSM8K, StrategyQA, and ARC‑Challenge across Llama‑3‑8B‑Instruct and Mistral‑7B‑Instruct at 10%, 20%, and 50% sparsity.

arXiv AI

Sep 17

OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning

OBC‑Prune introduces an outcome‑based calibration approach for pruning large reasoning models, focusing on the causal importance of each reasoning sentence rather than uniform activation salience. By pairing correct and incorrect rollouts and using intervention‑based analysis, it assigns per‑token weights that guide one‑shot pruning methods such as SparseGPT, Wanda, and ALPS. Experiments on DeepSeek‑R1‑Distill‑Qwen models show consistent accuracy gains and shorter reasoning traces across multiple benchmarks at 40‑50% sparsity.

By Ha Lan Nguyen, Huy Hoang Tran, Trac-Duy Tran, Dung D. Le

efficiency benchmarks

arXiv Computation and Language

Sep 22

LLaDA-PRM: A Bidirectional Step-Level Reasoning Evaluator

arXiv:2609.22700v1 Announce Type: new Abstract: Step-level reasoning evaluators are commonly based on autoregressive language models, whose causal attention restricts each step representation to the...

By Yiming Feng, Naihao Deng, Yulong Chen, Rada Mihalcea

llms benchmarks

arXiv AI

Jul 14

Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

arXiv:2607. 11266v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet it often incurs substantial computational costs due to over-reasoning: the generation of redundant, verbose, or irrelevant steps.

By Daeyeop Lee, Hwanjo Yu

llms efficiency benchmarks

arXiv AI

Oct 1

Think Right: Learning to Mitigate Under-Over Thinking via Adaptive, Attentive Compression

arXiv:2510.01581v2 Announce Type: replace-cross Abstract: Recent thinking models are capable of solving complex reasoning tasks by scaling test-time compute, but this scaling should be allocated in l...

By Joykirat Singh, Justin Chih-Yao Chen, Archiki Prasad, Elias Stengel-Eskin, Akshay Nambi, Mohit Bansal

reinforcement-learning

arXiv AI

Jul 2

Predicting LLM Reasoning Performance with Small Proxy Model

arXiv:2509. 21013v4 Announce Type: replace-cross Abstract: Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up.

By Woosung Koh, Juyoung Suk, Sungjun Han, Se-Young Yun, Jamin Shin

llms benchmarks safety

Hugging Face Trending Papers

Jun 1

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels.

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 2

Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

arXiv:2606. 01682v1 Announce Type: cross Abstract: Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths.

By Atoosa Chegini, Soheil Feizi

llms reinforcement-learning benchmarks safety

arXiv AI

Oct 21

Attention-Aligned Reasoning for Large Language Models

arXiv:2510.03223v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, cri...

By Hongxiang Zhang, Yuan Tian, Tianyi Zhang

llms benchmarks safety

arXiv AI

Jul 21

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

arXiv:2510. 22228v2 Announce Type: replace-cross Abstract: Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs).

By Keyu Wang, Tian Lyu, Guinan Su, Lu Yin, Marco Canini, Jonas Geiping, Shiwei Liu

llms fine-tuning efficiency benchmarks

arXiv Computation and Language

Sep 24

Towards Efficient Reasoning: Learning Causal Shortcuts for Diffusion Language Models

The paper introduces Causal Shortcut Learning (CSL), a framework that identifies token chains—called causal shortcuts—that guide Diffusion Language Models (DLMs) toward correct reasoning paths. By extracting these shortcuts and applying parallel prioritized masking during training, CSL improves both convergence speed and generation accuracy. Experiments on several reasoning benchmarks and two base models show CSL outperforms existing SFT-variant baselines, achieving an average 1.92% improvement over SFT-only models and up to 4.20% on MATH-500.

By Dian Jin, Kairong Han, Baohong Li, Xinpeng Dong, Zijing Hu, Nuanqiao Shan, Fei Wu, Kun Kuang

llms diffusion fine-tuning benchmarks

arXiv AI

Jul 22

Robust Reasoning Benchmark

arXiv:2604. 08571v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting.

By Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey

llms nlp benchmarks

arXiv AI

Aug 11

TempoBench: Reasoning Execution Without Causal Attribution Is Just Simulation

arXiv:2510. 27544v3 Announce Type: replace Abstract: Current training paradigms, optimized for long-horizon reasoning trace execution, have made Large Language Models (LLMs) excel at pattern matching and forward simulation of reasoning, but underperform at counterfactual causal understanding and reasoning.

By Nikolaus Holzer, William Fishell, Baishakhi Ray, Mark Santolucito

llms agents benchmarks

Related stories