arXiv AI By Shuyue Stella Li, Priyanka Kargupta, Haocheng Wang, Jinu Lee, Dean Light, Shan Chen, Orevaoghene Ahia, Thomas L. Griffiths, Max Kleiman-Weiner, Jiawei Han, Asli Celikyilmaz, Yulia Tsvetkov

Cognitive Foundations for Reasoning and Their Manifestation in LLMs

Read the original on arXiv AI →

The paper presents a taxonomy of 28 cognitive elements derived from cognitive science to analyze reasoning in large language models (LLMs). Using a fine‑grained evaluation framework, the authors compare 192,000 model traces with 1,346 human think‑aloud traces, finding that while models deploy similar cognitive elements, they differ in usage patterns. The study introduces test‑time reasoning guidance that improves model accuracy on ill‑structured problems by 9.7 percentage points and highlights a research gap in self‑awareness, advocating for a shared vocabulary to diagnose failures and guide robust model development.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Aug 17

The Metacognitive Bottleneck: Japanese Riddles Reveal Fundamental Limits of Machine Insight and Self-Evaluation in Reasoning AI

arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.

By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen
arXiv Machine Learning
Sep 10

Do Reasoning Representations Help Humans Evaluate LLM Outputs?

The paper investigates whether reasoning representations—explanations for large language model outputs—aid humans in evaluating those outputs. A controlled human study tested six reasoning formats across tasks of varying complexity, measuring structural understanding, error detection, and trust calibration. Results revealed a mismatch: participants favored planning- and decomposition-based representations, yet simpler chain-of-thought traces better supported verification, trust, and interpretability, while preferred formats increased calibration risks.

By Jaewoo Lim, Sungbok Shin, Sanghyun Hong
arXiv Computation and Language
Oct 16

Explanation Generation for Contradiction Reconciliation with LLMs

The paper introduces a new task called reconciliatory explanation generation, where large language models (LLMs) must produce explanations that make contradictory statements compatible. It proposes reusing natural language inference datasets and new quality metrics for scalable evaluation. Experiments with 13 LLMs show limited success, especially when longer reasoning chains are involved, but reinforcement‑learning training improves performance on two narrative reasoning benchmarks.

By Jason Chan, Zhixue Zhao, Robert Gaizauskas