arXiv AI

A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities

The paper introduces NeuroCognition, a benchmark based on three neuropsychological tests—Raven's Progressive Matrices, Spatial Working Memory, and the Wisconsin Card Sorting Test—to evaluate foundational cognitive abilities in large language models (LLMs). It finds that while LLMs excel on text tasks, their performance drops on image-based and more complex tasks, and they fail different parts of the same tasks compared to humans. NeuroCognition correlates with standard general-capability benchmarks yet measures distinct cognitive skills, highlighting where LLMs align with or diverge from human-like intelligence.

arXiv Computation and Language
Sep 30

Cognitive Expert Language Models Better Align with the Corresponding Brain Systems

The study investigates whether language models tailored to specific cognitive domains better align with corresponding brain systems. By prompting and fine‑tuning large language models into six domain experts—sensory, spatial, numerical, reasoning, social, and abstract—the authors find that each expert’s representations more closely match the brain region associated with its domain than other experts. This domain‑specific alignment holds across multiple base models and fMRI datasets, while overall prediction accuracy remains largely unchanged, indicating that regional alignment can be obscured when summarizing across the brain.

By Zhivar Sourati, Mengxuan Helen Wu, Nona Ghazizadeh, Jonas Kaplan, Morteza Dehghani, Samuel A. Nastase
arXiv AI
Oct 27

How Humans Learn Unfamiliar Worlds: A Think-Aloud Dataset on ARC-AGI-3

The paper presents the first dataset that captures how humans learn and use mechanisms in unfamiliar environments, using ARC‑AGI‑3 as a testbed. It includes 144 trajectories from 64 participants across 25 public games, each paired with time‑synchronized actions and think‑aloud audio, and 250 trajectories from large language model agents with and without harnesses. The dataset is annotated for mechanism triggers, action purposes, errors, and recovery, enabling a comparison of human and LLM cognition, highlighting differences in hypothesis testing, exploration, and use of qualitative representations.

By Shuzhi Liu, Archana Warrier, Dat Nguyen, Chengqi Zheng, Kevin Ellis, Yewen Pu
arXiv AI
Aug 14

EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory

arXiv:2608. 13113v1 Announce Type: cross Abstract: Recent advances in Multimodal Large Language Models (MLLMs) have led to substantial progress in video understanding, accompanied by a growing number of long video benchmarks.

By Weitao Chen, Hu Jiaxin, Xie Tianyidan, Yang Li, Yuyi Qian, Banghao Xu, Ziheng Tang, Shenyi Wang, Mingyue Yu, Duo Li, Jiacheng Shi, Gao Wang, Zhan Xu, Zhicheng Qiu, Xuanfu Li, Jian Yang, Lanjun Wang, Zili Yi
arXiv AI
Nov 6

Cognitive Foundations for Reasoning and Their Manifestation in LLMs

The paper presents a taxonomy of 28 cognitive elements derived from cognitive science to analyze reasoning in large language models (LLMs). Using a fine‑grained evaluation framework, the authors compare 192,000 model traces with 1,346 human think‑aloud traces, finding that while models deploy similar cognitive elements, they differ in usage patterns. The study introduces test‑time reasoning guidance that improves model accuracy on ill‑structured problems by 9.7 percentage points and highlights a research gap in self‑awareness, advocating for a shared vocabulary to diagnose failures and guide robust model development.

By Shuyue Stella Li, Priyanka Kargupta, Haocheng Wang, Jinu Lee, Dean Light, Shan Chen, Orevaoghene Ahia, Thomas L. Griffiths, Max Kleiman-Weiner, Jiawei Han, Asli Celikyilmaz, Yulia Tsvetkov
arXiv Computation and Language
Oct 5

The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?

The paper introduces Percept-V, a dataset of 6,000 program-generated images across 30 domains designed to test simple visual perception skills from the TVPS-4 framework. Experiments show that state‑of‑the‑art multimodal large language models perform poorly compared to humans, especially as image complexity increases, and that fine‑tuning yields only limited generalization to related datasets. The study highlights specific perception skills that remain challenging for current models.

By Samrajnee Ghosh, Ashish Goswami, Naman Agarwal, Hemanshu Garg, Chinmay Mittal, Mausam, Parag Singla
arXiv Computation and Language
1d ago

Tracing Knowledge and Reasoning in LLM

The paper introduces a task‑centric framework that analyzes the internal processing of Large Language Models (LLMs) by combining attention, activation, and representation signals. It evaluates 16 LLMs on 24 benchmarks, revealing systematic differences between knowledge‑centric and reasoning‑centric tasks and exposing failure modes hidden by overall scores. The authors provide open‑source code to support more targeted benchmark selection and informed model choice.

By Yang Wang, Xiao Wang, Yiqi Liu, Chenghao Xiao, Eleni Vasilaki, Goran Nenadic, Chenghua Lin
arXiv Machine Learning
Oct 22

SkillEval: Learning Interpretable Ability Profiles of LLMs via Cognitive Diagnosis Models

SkillEval introduces a new evaluation framework for large language models that moves beyond single scalar scores to perform cross-model, skill-level analysis. It automatically discovers interpretable skills from benchmark questions using LLM meta‑cognition and applies cognitive diagnosis models to estimate per‑skill abilities. Experiments on 3,811 LLMs across five benchmarks show that SkillEval yields more calibrated, informative model profiles, enabling tasks such as efficient LLM routing and adaptive testing.

By Berke Arda, Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mubashara Akhtar, Mrinmaya Sachan