arXiv AI

Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis

arXiv:2608. 15630v1 Announce Type: cross Abstract: The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities.

arXiv AI

Aug 19

Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses

The study examines whether the latent factors that explain large language model (LLM) performance correspond to human‑interpretable cognitive constructs. Using exploratory factor analysis on responses from humans and six LLMs in quantitative reasoning and chemistry, subject‑matter experts could interpret most human‑derived factors but struggled to ascribe meaning to LLM‑derived factors, especially in quantitative reasoning and only partially in chemistry. The results suggest that LLMs often rely on statistically opaque mechanisms distinct from human reasoning.

By Alona Strugatski, Licol Zeinfeld, Jason Cooper, Shelley Rap, Gil Schwarts, Giora Alexandron

llms

arXiv AI

Jun 12

Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering

arXiv:2606. 12422v1 Announce Type: cross Abstract: The integration of large language models (LLMs) into educational assessment represents a transformative shift in classroom grading practices.

By Zewei Tian, Alex Liu, Lief Esbenshade, Michael Xiao, Zachary Zhang, Yulia L\'apicus, Thomas Han, Kevin He, Min Sun

llms

arXiv AI

Jul 28

Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

arXiv:2601. 02580v2 Announce Type: replace-cross Abstract: Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration.

By Christopher Ormerod

llms fine-tuning

arXiv AI

Jun 19

Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact

arXiv:2606. 20205v1 Announce Type: new Abstract: Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research.

By Jelena Meyer, David Garcia, Dirk U. Wulff

llms safety

arXiv AI

Jul 31

Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach

arXiv:2607. 26317v1 Announce Type: cross Abstract: Psychometric calibration for educational tests typically requires costly human response data.

By Wenjie Zhou, Yunting Liu, Renjiao Tang, Mark Wilson

llms safety

arXiv Machine Learning

Jun 8

Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.

By Tao Liu, Ye Lu, Ruohua Zhang, Siyu Song, Wentao Liu, Aimin Zhou, Hao Hao

llms agents benchmarks safety

arXiv AI

4d ago

Large-scale factor analysis shows machine intelligence is only partially interpretable

arXiv:2609.36515v1 Announce Type: cross Abstract: A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like f...

By Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi

llms benchmarks safety

arXiv AI

Jul 10

Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring

arXiv:2604. 07102v2 Announce Type: replace-cross Abstract: Activation-based steering enables inference-time personalization of large language models, but its effects in educational applications are not well understood.

By Yongchao Wu, Aron Henriksson

llms benchmarks

arXiv AI

Jun 6

Framing, Judging, Steering: An Assessable Competency Model for Teach-ing Students to Reason With Generative AI

arXiv:2606. 05983v1 Announce Type: new Abstract: Generative AI makes answers easy and understanding hard, and uncritical use invites cognitive offloading.

By Alexander Apartsin, Yehudit Aperstein

llms

arXiv AI

Jul 20

Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length

arXiv:2512. 07019v3 Announce Type: replace-cross Abstract: The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to provide guidance for both downstream applications and actionable future improvements.

By Zhiyu Xu, Jia Liu, Yixin Wang, Yuqi Gu

llms benchmarks

arXiv Computation and Language

Sep 10

What Does MMLU Actually Measure? A Psychometric Audit of Difficulty Structure in Aggregate Benchmark Scores

arXiv:2609.09372v1 Announce Type: cross Abstract: Although MMLU is widely adopted as a benchmark for calibrating general AI capabilities, we psychometrically demonstrate that its aggregate score prim...

By Dana Paquin, Riddhiman Jain

llms benchmarks

arXiv AI

Jun 10

RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning

arXiv:2606. 10254v1 Announce Type: new Abstract: While Large Language Models (LLMs) have achieved near-perfect performance in \emph{solving} high-school mathematics, their ability to \emph{evaluate} the diverse reasoning processes of real human students remains under-examined.

By Yiteng Mao, Kenan Xu, Yijia Lyu, Wenhao Li, Jianlong Chen, Xiangfeng Wang

llms rag benchmarks

Related stories