The paper introduces semanticfa, an R package that performs semantic factor analysis on psychometric scales by converting items into language‑model embeddings and factoring their similarities. It demonstrates the method on the 50‑item International Personality Item Pool Big‑Five markers, compares the semantic factors to a large response‑based factor analysis, and shows how the package can interpret factors, flag redundant items, build short forms, and screen new items. The authors propose semantic‑behavioral coherence as a measurable property of a scale, noting that semantic factor analysis complements but does not replace traditional response‑based factor analysis.
By Devon Yanitski, Chris Westbury
arXiv:2608.10503v2 Announce Type: replace
Abstract: As Large Language Models (LLMs) are increasingly deployed as autonomous agents, accurately evaluating their latent values and biases is critical. T...
By Davood Wadi, Mohsen Ghodrat, Matthew Philp
arXiv:2609.36515v1 Announce Type: cross
Abstract: A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like f...
By Faiz Ghifari Haznitrama, Afrizal Hasbi Azizy, Faeyza Rishad Ardi
The paper proposes a model-based evaluation framework that merges multidimensional item response theory (IRT) with question context embeddings to predict large language model (LLM) performance on unseen questions. By representing LLMs with latent capability profiles and incorporating question content to inform item characteristics, the approach improves prediction accuracy over model-free baselines in within-scenario settings and offers a richer description of capability variation than unidimensional models. However, the study also finds that this generalizability does not reliably extend to cross-scenario shifts, indicating a key limitation for broader application.
By Ergan Shang, Weijing Tang, Yinqiu He
The study investigates whether large language models (LLMs) can serve as effective substitutes for human survey respondents, particularly for data augmentation by predicting new responses from partial data. Researchers evaluated three LLMs on two attitude surveys about automated and light-electric vehicles, comparing their predictions to two embedding-similarity baselines. Results showed that while LLMs generally outperformed the baselines, their accuracy dropped as less known responses were provided, sometimes matching baseline performance in one survey.
By Esko Lehtonen, Tommi Buder-Gr\"ondahl
The paper introduces a dual‑dimensional framework called Automated Item Similarity Analysis (AISA) that uses Large Language Models to assess incidental content similarity in large‑scale assessments. It combines Structured Decomposition and Semantic Relatedness to capture both structural and semantic nuances that traditional metrics miss. Psychometric validation shows that LLM‑derived metrics better align with construct‑irrelevant local dependence and produce more coherent item groupings, and simulations in Computerized Adaptive Testing demonstrate improved estimation stability and reduced bias with minimal efficiency loss.
By Jing Huang, Jihong Zhang, Hua-Hua Chang