The paper introduces semanticfa, an R package that performs semantic factor analysis on psychometric scales by converting items into language‑model embeddings and factoring their similarities. It demonstrates the method on the 50‑item International Personality Item Pool Big‑Five markers, compares the semantic factors to a large response‑based factor analysis, and shows how the package can interpret factors, flag redundant items, build short forms, and screen new items. The authors propose semantic‑behavioral coherence as a measurable property of a scale, noting that semantic factor analysis complements but does not replace traditional response‑based factor analysis.
By Devon Yanitski, Chris Westbury
arXiv:2608.10503v2 Announce Type: replace
Abstract: As Large Language Models (LLMs) are increasingly deployed as autonomous agents, accurately evaluating their latent values and biases is critical. T...
By Davood Wadi, Mohsen Ghodrat, Matthew Philp
arXiv:2609.36515v1 Announce Type: cross
Abstract: A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like f...
By Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi
The paper proposes a model-based evaluation framework that merges multidimensional item response theory (IRT) with question context embeddings to predict large language model (LLM) performance on unseen questions. By representing LLMs with latent capability profiles and incorporating question content to inform item characteristics, the approach improves prediction accuracy over model-free baselines in within-scenario settings and offers a richer description of capability variation than unidimensional models. However, the study also finds that this generalizability does not reliably extend to cross-scenario shifts, indicating a key limitation for broader application.
By Ergan Shang, Weijing Tang, Yinqiu He
The study investigates whether large language models (LLMs) can serve as effective substitutes for human survey respondents, particularly for data augmentation by predicting new responses from partial data. Researchers evaluated three LLMs on two attitude surveys about automated and light-electric vehicles, comparing their predictions to two embedding-similarity baselines. Results showed that while LLMs generally outperformed the baselines, their accuracy dropped as less known responses were provided, sometimes matching baseline performance in one survey.
By Esko Lehtonen, Tommi Buder-Gr\"ondahl
The paper introduces a dual‑dimensional framework called Automated Item Similarity Analysis (AISA) that uses Large Language Models to assess incidental content similarity in large‑scale assessments. It combines Structured Decomposition and Semantic Relatedness to capture both structural and semantic nuances that traditional metrics miss. Psychometric validation shows that LLM‑derived metrics better align with construct‑irrelevant local dependence and produce more coherent item groupings, and simulations in Computerized Adaptive Testing demonstrate improved estimation stability and reduced bias with minimal efficiency loss.
By Jing Huang, Jihong Zhang, Hua-Hua Chang
The study examines whether the latent factors that explain large language model (LLM) performance correspond to human‑interpretable cognitive constructs. Using exploratory factor analysis on responses from humans and six LLMs in quantitative reasoning and chemistry, subject‑matter experts could interpret most human‑derived factors but struggled to ascribe meaning to LLM‑derived factors, especially in quantitative reasoning and only partially in chemistry. The results suggest that LLMs often rely on statistically opaque mechanisms distinct from human reasoning.
By Alona Strugatski, Licol Zeinfeld, Jason Cooper, Shelley Rap, Gil Schwarts, Giora Alexandron
arXiv:2602.12575v3 Announce Type: replace-cross
Abstract: Psychological scale refinement traditionally relies on response-based methods such as factor analysis, item response theory, and network psyc...
By Bo Wang, Yuxuan Zhang, Yueqin Hu, Hanchao Hou, Kaiping Peng, Shiguang Ni
arXiv:2608. 15630v1 Announce Type: cross Abstract: The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities.
By Alona Strugatski, Licol Zeinfeld, Giora Alexandron
The study investigates how different prompt components affect language model responses in psychometric tests. By crossing five distinct baseline personas with five variants of each prompt element—persona wording, task instruction, item wording, and option symbol—the authors measure response shifts using the 1‑Wasserstein distance. Their analysis of 13 small open‑weight language models on the Big Five Inventory and Short Dark Triad reveals that task instruction and option symbol changes often cause more variation than paraphrasing the persona or item, with prompt artifacts explaining over 50% of the variation for many items.
By Nils Schwager, Christoph Hau, Simon M\"unker, Achim Rettinger
arXiv:2608.22438v1 Announce Type: new
Abstract: Persona-conditioned large language models (LLMs) are increasingly used to simulate survey responses across diverse domains. However, apparent response...
By Taehyeon An, Jaehyeong Park, Donghyuk Shin
The article examines how AI‑assisted item generation is filtered by a computational evaluator before expert review, focusing on representation, structural screening, and candidate‑form dependence. Through two in‑silico studies of 32,000 Big Five items, the authors show that subtle differences in semantic representation and structural evaluation lead to divergent item selections, even when overall content coverage appears stable. The findings reveal that the evaluator, often treated as a neutral technical step, actually shapes the evidence and wording that psychometricians ultimately review, highlighting its role as a revisable component of measurement design.
By Christopher Brooks (School of Information, University of Michigan)