The paper introduces COMPASS, an inference-time steering technique that enhances large language model performance on mathematical reasoning tasks. By using the correctness of a model’s direct answer attempts as a latent signal, COMPASS identifies a small subset of attention heads whose activations can be steered to elicit reasoning. Experiments across three model families and multiple math benchmarks show that COMPASS improves GSM8K accuracy by an average of 16 percentage points and achieves near Chain‑of‑Thought performance while generating 20–70% fewer tokens.
By Pratyay Dutta, Kowshik Thopalli, Vivek Narayanaswamy
The paper surveys efficient reasoning in large language models, contrasting fast intuitive (System 1) and slow deep (System 2) reasoning. It analyzes why System 2 is computationally costly yet more accurate, and why System 1 is efficient but less effective. The survey covers causes of inefficiency, patterns of reasoning behavior, and potential solutions to balance performance and computational budgets, offering actionable insights and an open‑source repository for ongoing research.
By Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, Kam-Fai Wong
The paper introduces a lightweight evaluation method for logical reasoning in transformer-based language models, using query-key alignments within attention heads to compute a QK-score. This single forward-pass technique identifies valid versus invalid inferences and is validated across multiple reasoning benchmarks, showing robustness to distractors and deeper reasoning. Experiments span models from 1.5B to 70B parameters, demonstrating scalability.
By Eduard Tulchinskii, Anastasia Voznyuk, Laida Kushnareva, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov
arXiv:2605.06165v2 Announce Type: replace
Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribut...
By Richmond Sin Jing Xuan, Rishabh Bhardwaj, Soujanya Poria
The paper proposes Layer-Informed Fine-Tuning (LIFT), a method that identifies and updates only the most functionally critical layers of large language models (LLMs) using a bottleneck identification mechanism based on sensitivity analysis. By focusing on layers that handle conceptualization, reasoning, and textualization, LIFT aims to accelerate training and enhance performance on reasoning tasks. Experiments demonstrate that this selective fine-tuning approach both speeds up the training process and yields significant performance gains.
By Junning Shao, Siwei Wang, Zhixuan Fang
The paper introduces Causal Attribution Pruning (CAP), a training‑free technique that identifies critical attention heads in large language models by measuring their causal impact on reasoning tasks. CAP estimates performance loss when each head is masked on a small calibration set, then translates these causal scores into weight‑level importance for pruning. Compared to magnitude‑only or activation‑based methods, CAP achieves up to 61% relative accuracy gains over Wanda on ARC‑Challenge at 20% sparsity and improves performance on GSM8K, StrategyQA, and ARC‑Challenge across Llama‑3‑8B‑Instruct and Mistral‑7B‑Instruct at 10%, 20%, and 50% sparsity.
By Amogh Sheth, Andrew Lin, Biruk Assefa, Yi Wen Huang, Yuhao Ge
arXiv:2509. 21013v4 Announce Type: replace-cross Abstract: Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up.
By Woosung Koh, Juyoung Suk, Sungjun Han, Se-Young Yun, Jamin Shin
The paper introduces Role-Enhanced Zero-Shot Chain-of-Thought (RE‑ZS‑CoT), a prompting framework that incorporates domain‑specific expert roles to improve reasoning in large language models without task‑specific examples. Two variants are evaluated: a static version that tests eight predefined roles across arithmetic, commonsense, and reading comprehension tasks, and a dynamic version where the model selects the most suitable role for each benchmark. Across 20 LLMs and seven benchmarks, RE‑ZS‑CoT consistently outperforms zero‑shot and zero‑shot‑CoT baselines, achieving up to 44 percentage‑point gains and demonstrating strong alignment between static and dynamic approaches, while ablation studies confirm the importance of task‑role alignment.
By Ibrahim Khan, Thai Van Nguyen, Yi Xia, Pratch Suntichaikul, Mustafa Can Gursesli, Daniel Fernandez Galeote, Juho Hamari, Ruck Thawonmas
The paper introduces LIFT, a lightweight vector‑intervention technique that transfers reasoning capability from a base large language model (LLM) to a vision‑language model (VLM) without retraining the VLM backbone. LIFT defines Reasoning Vectors as differences in hidden states between a reasoning path with an explicit trace and a solver path without it, and injects these vectors into the VLM’s language‑side activations. Experiments on two VLMs across six reasoning benchmarks show that vectors derived from the base LLM consistently outperform those derived from the aligned VLM, indicating that the base LLM is a more effective source for recovering degraded reasoning.
"whyItMatters":"The study demonstrates that a simple, frozen‑backbone intervention can partially restore reasoning abilities in multimodal models, highlighting the value of leveraging the original language model’s reasoning power."
By Ziyi Wang, Li Li, Aolin Zhou, Yankun Shen, Chonghan Liu, Shuxia Lin, Xu Yang
arXiv:2609.22700v1 Announce Type: new
Abstract: Step-level reasoning evaluators are commonly based on autoregressive language models, whose causal attention restricts each step representation to the...
By Yiming Feng, Naihao Deng, Yulong Chen, Rada Mihalcea
arXiv:2604. 10788v2 Announce Type: replace-cross Abstract: Tool-Integrated Reasoning (TIR) has emerged as a promising direction by extending Large Language Models' (LLMs) capabilities with external tools during reasoning.
By Qiancheng Xu, Yongqi Li, Fan Liu, Hongru Wang, Min Yang, Wenjie Li
The study investigates the performance gap between native-language reasoning and English-pivoted reasoning in large language models. By creating extensive multilingual reasoning datasets and fine‑tuning specialists on Qwen/Qwen3-8B-Base, the authors find that the native reasoning gap is much smaller (1.9–3.5%) than previously reported. They analyze weight‑space changes, discover a language‑agnostic reasoning core in the middle layers, and propose a Layer Swap technique that transfers these mid‑layer updates from an English specialist to native specialists, effectively closing most of the gap while maintaining native chain‑of‑thought output.
By Maxence Lasbordes, Am\'elie Chatelain, Djam\'e Seddah