arXiv Computation and Language

Tracing Knowledge and Reasoning in LLM

The paper introduces a task‑centric framework that analyzes the internal processing of Large Language Models (LLMs) by combining attention, activation, and representation signals. It evaluates 16 LLMs on 24 benchmarks, revealing systematic differences between knowledge‑centric and reasoning‑centric tasks and exposing failure modes hidden by overall scores. The authors provide open‑source code to support more targeted benchmark selection and informed model choice.

arXiv Machine Learning
Oct 8

Latent Performance Profiling of Large Language Models

arXiv:2605.30018v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of the...

By Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya, Partha Pratim Chakrabarti, Amlan Chakrabarti, Supratik Chakraborty, Partha Pratim Das, Lipika Dey, Richa Singh, Mayank Vatsa
arXiv Computation and Language
Sep 30

Layer-Informed Fine-Tuning via Three-Stage Functional Segmentation of LLMs

The paper proposes Layer-Informed Fine-Tuning (LIFT), a method that identifies and updates only the most functionally critical layers of large language models (LLMs) using a bottleneck identification mechanism based on sensitivity analysis. By focusing on layers that handle conceptualization, reasoning, and textualization, LIFT aims to accelerate training and enhance performance on reasoning tasks. Experiments demonstrate that this selective fine-tuning approach both speeds up the training process and yields significant performance gains.

By Junning Shao, Siwei Wang, Zhixuan Fang
arXiv AI
Aug 17

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI

arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.

By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen
arXiv AI
Jul 22

Robust Reasoning Benchmark

arXiv:2604. 08571v3 Announce Type: replace-cross Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting.

By Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey
arXiv AI
Nov 6

Cognitive Foundations for Reasoning and Their Manifestation in LLMs

The paper presents a taxonomy of 28 cognitive elements derived from cognitive science to analyze reasoning in large language models (LLMs). Using a fine‑grained evaluation framework, the authors compare 192,000 model traces with 1,346 human think‑aloud traces, finding that while models deploy similar cognitive elements, they differ in usage patterns. The study introduces test‑time reasoning guidance that improves model accuracy on ill‑structured problems by 9.7 percentage points and highlights a research gap in self‑awareness, advocating for a shared vocabulary to diagnose failures and guide robust model development.

By Shuyue Stella Li, Priyanka Kargupta, Haocheng Wang, Jinu Lee, Dean Light, Shan Chen, Orevaoghene Ahia, Thomas L. Griffiths, Max Kleiman-Weiner, Jiawei Han, Asli Celikyilmaz, Yulia Tsvetkov
arXiv AI
Oct 1

LSR-Ben: A Logical and Scientific Reasoning Benchmark for Evaluating Process Reward Models

The paper introduces LSR‑Ben, a benchmark designed to evaluate process reward models (PRMs) on scientific and logical reasoning tasks, addressing a gap left by existing math‑focused benchmarks. Experiments on 22 models reveal that PRMs and LLMs perform poorly in non‑mathematical domains, with LLMs tending to over‑identify errors while PRMs tend to overlook them. LSR‑Ben aims to spur research that broadens PRM applicability and improves LLM reasoning.

By Zhouhao Sun, Xuan Zhang, Xiao Ding, Bibo Cai, Li Du, Kai Xiong, Xinran Dai, Fei Zhang, weidi tang, Zhiyuan Kan, Yang Zhao, Bing Qin, Ting Liu
arXiv Machine Learning
Sep 10

Do Reasoning Representations Help Humans Evaluate LLM Outputs?

The paper investigates whether reasoning representations—explanations for large language model outputs—aid humans in evaluating those outputs. A controlled human study tested six reasoning formats across tasks of varying complexity, measuring structural understanding, error detection, and trust calibration. Results revealed a mismatch: participants favored planning- and decomposition-based representations, yet simpler chain-of-thought traces better supported verification, trust, and interpretability, while preferred formats increased calibration risks.

By Jaewoo Lim, Sungbok Shin, Sanghyun Hong