arXiv Computation and Language By Junning Shao, Siwei Wang, Zhixuan Fang

Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs

Read the original on arXiv Computation and Language →

The paper proposes Layer-Informed Fine-Tuning (LIFT), a method that identifies and updates only the most functionally critical layers of large language models (LLMs) using a bottleneck identification mechanism based on sensitivity analysis. By focusing on layers that handle conceptualization, reasoning, and textualization, LIFT aims to accelerate training and enhance performance on reasoning tasks. Experiments demonstrate that this selective fine-tuning approach both speeds up the training process and yields significant performance gains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
1d ago

Tracing Knowledge and Reasoning in LLM

The paper introduces a task‑centric framework that analyzes the internal processing of Large Language Models (LLMs) by combining attention, activation, and representation signals. It evaluates 16 LLMs on 24 benchmarks, revealing systematic differences between knowledge‑centric and reasoning‑centric tasks and exposing failure modes hidden by overall scores. The authors provide open‑source code to support more targeted benchmark selection and informed model choice.

By Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin
arXiv AI
Aug 28

Syntax vs. Semantics: How Transformers Learn Deep Dependencies

The paper investigates how transformers acquire deep semantic dependencies, proposing a mechanistic framework that frames learning as a competition between surface statistics and deep semantics. It identifies a "Gradient Starvation" effect that suppresses error signals for sparse semantic dependencies early in training, delaying structural reasoning until a sudden phase transition. The study also explains the success of Chain-of-Thought strategies and introduces a topology‑aligned contrastive objective that improves variable binding performance by more than twice the gain of standard fine‑tuning.

By Jiangrui Zhao, Xiaoting Du
arXiv AI
Jul 8

LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

arXiv:2607. 06160v1 Announce Type: cross Abstract: Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision.

By Chenhao Yuan, Yinhao Xu, Shuwen Xu, Xizhi Yang, Jiaxiang Liu, Chenxi Zhou, Shaoping Huang, Haolin Ren, Pengfei Cao, Jun Zhao, Kang Liu
arXiv AI
Sep 7

Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models

The paper surveys efficient reasoning in large language models, contrasting fast intuitive (System 1) and slow deep (System 2) reasoning. It analyzes why System 2 is computationally costly yet more accurate, and why System 1 is efficient but less effective. The survey covers causes of inefficiency, patterns of reasoning behavior, and potential solutions to balance performance and computational budgets, offering actionable insights and an open‑source repository for ongoing research.

By Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, Kam-Fai Wong