arXiv Computation and Language

Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs

The paper proposes Layer-Informed Fine-Tuning (LIFT), a method that identifies and updates only the most functionally critical layers of large language models (LLMs) using a bottleneck identification mechanism based on sensitivity analysis. By focusing on layers that handle conceptualization, reasoning, and textualization, LIFT aims to accelerate training and enhance performance on reasoning tasks. Experiments demonstrate that this selective fine-tuning approach both speeds up the training process and yields significant performance gains.

arXiv Computation and Language
1d ago

Tracing Knowledge and Reasoning in LLM

The paper introduces a task‑centric framework that analyzes the internal processing of Large Language Models (LLMs) by combining attention, activation, and representation signals. It evaluates 16 LLMs on 24 benchmarks, revealing systematic differences between knowledge‑centric and reasoning‑centric tasks and exposing failure modes hidden by overall scores. The authors provide open‑source code to support more targeted benchmark selection and informed model choice.

By Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin
arXiv AI
Aug 28

Syntax vs. Semantics: How Transformers Learn Deep Dependencies

The paper investigates how transformers acquire deep semantic dependencies, proposing a mechanistic framework that frames learning as a competition between surface statistics and deep semantics. It identifies a "Gradient Starvation" effect that suppresses error signals for sparse semantic dependencies early in training, delaying structural reasoning until a sudden phase transition. The study also explains the success of Chain-of-Thought strategies and introduces a topology‑aligned contrastive objective that improves variable binding performance by more than twice the gain of standard fine‑tuning.

By Jiangrui Zhao, Xiaoting Du
arXiv AI
Jul 8

LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

arXiv:2607. 06160v1 Announce Type: cross Abstract: Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision.

By Chenhao Yuan, Yinhao Xu, Shuwen Xu, Xizhi Yang, Jiaxiang Liu, Chenxi Zhou, Shaoping Huang, Haolin Ren, Pengfei Cao, Jun Zhao, Kang Liu
arXiv AI
Sep 7

Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models

The paper surveys efficient reasoning in large language models, contrasting fast intuitive (System 1) and slow deep (System 2) reasoning. It analyzes why System 2 is computationally costly yet more accurate, and why System 1 is efficient but less effective. The survey covers causes of inefficiency, patterns of reasoning behavior, and potential solutions to balance performance and computational budgets, offering actionable insights and an open‑source repository for ongoing research.

By Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, Kam-Fai Wong
arXiv Computer Vision
Nov 2

Attention Maps Know When to Think: Training-Free Difficulty-Aware Chain-of-Thought for Reasoning Segmentation

The paper introduces Difficulty-Aware Chain-of-Thought (DA‑CoT), a training‑free framework that selectively applies chain‑of‑thought prompting in reasoning segmentation tasks based on the entropy of attention maps. Low entropy signals confident, localized grounding, while high entropy indicates uncertainty and the potential benefit of additional reasoning. DA‑CoT also incorporates a Bidirectional Rethinking mechanism to automatically reverse reasoning decisions when initial outputs fail, thereby improving robustness and inference efficiency while achieving state‑of‑the‑art results on Reasoning Segmentation benchmarks.

By Chanseul Cho, Jaesung Jun, Dongheon Lee, Youngmin Ro
arXiv AI
Oct 31

Supervised Fine-Tuning for Reasoning Large Language Models: A Survey

The paper surveys recent progress in supervised fine‑tuning (SFT) for large language models that excel at reasoning tasks. It reviews methodological advances, grouping them into data‑centric and algorithm‑centric approaches, and reorganizes analytical studies by data characteristics, optimization dynamics, and mechanistic insights. The survey also identifies current research focuses, remaining bottlenecks, and suggests future directions for reasoning SFT.

By Yongcan Yu, Kuangpu Guo, Jian Liang, Lingxiao He, Meng Wang, Ran He
arXiv AI
Sep 1

Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Training

The paper investigates how different forms of compressed chain‑of‑thought (CoT) reasoning—Explicit, Composed, and Implicit—affect large language model (LLM) performance after supervised fine‑tuning (SFT). Using a synthetic compositional reasoning task, the authors show that coarser CoT requires more SFT data, that Composed and Implicit CoT benefit more from data scaling (with Composed also benefiting from repetition), and that reinforcement learning with verifiable rewards (RLVR) can decompose compressed steps learned during SFT. Additionally, unidirectional CoT ordering improves generalization on longer sequential tasks.

By Kohsei Matsutani, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
arXiv AI
3d ago

LongReasonArena: A Long Reasoning Benchmark for Large Language Models

LongReasonArena is a new benchmark that evaluates the long reasoning abilities of large language models, rather than just their comprehension of long inputs. It presents tasks that require multi-step algorithms involving retrieval and backtracking, with reasoning lengths that can be scaled up to one million tokens. Evaluation shows that even advanced models like GPT‑5 struggle, achieving only 27.8% accuracy on the hardest level, and accuracy decreases linearly with the logarithm of expected reasoning steps.

By Jiayu Ding, Shuming Ma, Lei Cui, Nanning Zheng, Furu Wei