The paper introduces semanticfa, an R package that performs semantic factor analysis on psychometric scales by converting items into language‑model embeddings and factoring their similarities. It demonstrates the method on the 50‑item International Personality Item Pool Big‑Five markers, compares the semantic factors to a large response‑based factor analysis, and shows how the package can interpret factors, flag redundant items, build short forms, and screen new items. The authors propose semantic‑behavioral coherence as a measurable property of a scale, noting that semantic factor analysis complements but does not replace traditional response‑based factor analysis.
arXiv:2602.12575v3 Announce Type: replace-cross Abstract: Psychological scale refinement traditionally relies on response-based methods such as factor analysis, item response theory, and network psyc...
arXiv:2606. 09843v1 Announce Type: cross Abstract: Large language models (LLMs) produce stable self-reports on personality inventories, but these self-reports do not predict observed behavior.
The paper introduces the Semantic‑Empirical Updating Framework (SEUF), which uses large language model paraphrases and embeddings aligned with hypothesized item‑factor assignments to create prespecified priors for Bayesian confirmatory factor analysis. In experiments with a 38‑item survey and two public surveys (FEVS and RIASEC), SEUF maintained semantic structure after wording controls and achieved comparable loading precision with 50 respondents to conventional CFA with 70–82 respondents, reducing required sample size by 29–39%. The approach yielded lower loading errors and better credible‑interval coverage, especially when respondent data were scarce, demonstrating the benefit of incorporating semantic priors in small‑sample measurement.
arXiv:2606. 09843v3 Announce Type: replace-cross Abstract: Large language models (LLMs) give stable answers to personality questionnaires, yet these self-reports fail to predict how the models behave.
The paper introduces the latent space item response model (LSIRM) as a tool for uncovering how language models differ on specific benchmark items. By mapping 1,348 models across 5,761 BBH items and comparing scoring methods across BBH, GPQA, MMLU-Pro, and MuSR, the authors show that items with similar correct answers cluster together, especially when scored by option labels. LSIRM separates a model’s tendency to perform better on certain answer categories from other performance differences, revealing that small changes in scoring can dramatically alter model rankings.
arXiv:2609.36515v1 Announce Type: cross Abstract: A common assumption in language model development is that cognitive abilities are organized around a general, domain-free intelligence factor, like f...
arXiv:2606. 20205v1 Announce Type: new Abstract: Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research.
arXiv:2603.04419v3 Announce Type: replace-cross Abstract: Vision-language models produce different object and use descriptions under different persona prompts, but low overlap alone does not identify...
arXiv:2608. 15630v1 Announce Type: cross Abstract: The rapid development and growing deployment of large language models (LLMs) have made it increasingly important to understand their capabilities.
PACIFIC is a framework that aligns large language model responses with user preferences by leveraging stable Big‑Five personality traits as a latent signal. The authors built a 1,200‑pair dataset covering diverse domains and trait directions, and found that trait‑aligned contexts enable LLMs to achieve near‑ceiling accuracy (up to 99%) in personalized QA. They also introduced a persona‑aware contrastive retriever (PiRAG) that improves label‑free accuracy from 30% to 43% over standard semantic retrieval, highlighting retrieval as the main bottleneck.
The study investigates how different prompt components affect language model responses in psychometric tests. By crossing five distinct baseline personas with five variants of each prompt element—persona wording, task instruction, item wording, and option symbol—the authors measure response shifts using the 1‑Wasserstein distance. Their analysis of 13 small open‑weight language models on the Big Five Inventory and Short Dark Triad reveals that task instruction and option symbol changes often cause more variation than paraphrasing the persona or item, with prompt artifacts explaining over 50% of the variation for many items.
The paper introduces a dual‑dimensional framework called Automated Item Similarity Analysis (AISA) that uses Large Language Models to assess incidental content similarity in large‑scale assessments. It combines Structured Decomposition and Semantic Relatedness to capture both structural and semantic nuances that traditional metrics miss. Psychometric validation shows that LLM‑derived metrics better align with construct‑irrelevant local dependence and produce more coherent item groupings, and simulations in Computerized Adaptive Testing demonstrate improved estimation stability and reduced bias with minimal efficiency loss.