← Back to all news
arXiv Computation and Language By Devon Yanitski, Chris Westbury

semanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales

Read the original on arXiv Computation and Language →

The paper introduces semanticfa, an R package that performs semantic factor analysis on psychometric scales by converting items into language‑model embeddings and factoring their similarities. It demonstrates the method on the 50‑item International Personality Item Pool Big‑Five markers, compares the semantic factors to a large response‑based factor analysis, and shows how the package can interpret factors, flag redundant items, build short forms, and screen new items. The authors propose semantic‑behavioral coherence as a measurable property of a scale, noting that semantic factor analysis complements but does not replace traditional response‑based factor analysis.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Oct 28

When Respondents Are Scarce: LLM-Derived Semantic Priors for Bayesian Confirmatory Factor Analysis

The paper introduces the Semantic‑Empirical Updating Framework (SEUF), which uses large language model paraphrases and embeddings aligned with hypothesized item‑factor assignments to create prespecified priors for Bayesian confirmatory factor analysis. In experiments with a 38‑item survey and two public surveys (FEVS and RIASEC), SEUF maintained semantic structure after wording controls and achieved comparable loading precision with 50 respondents to conventional CFA with 70–82 respondents, reducing required sample size by 29–39%. The approach yielded lower loading errors and better credible‑interval coverage, especially when respondent data were scarce, demonstrating the benefit of incorporating semantic priors in small‑sample measurement.

By Shaohuai Feng, Mohd Wira Mohd Shafiei, Mohd Sukri Shafie, Siyang Tian
llmsragbenchmarks
More like this →
arXiv Computation and Language
3d ago

Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM

The paper introduces the latent space item response model (LSIRM) as a tool for uncovering how language models differ on specific benchmark items. By mapping 1,348 models across 5,761 BBH items and comparing scoring methods across BBH, GPQA, MMLU-Pro, and MuSR, the authors show that items with similar correct answers cluster together, especially when scored by option labels. LSIRM separates a model’s tendency to perform better on certain answer categories from other performance differences, revealing that small changes in scoring can dramatically alter model rankings.

By Jinwen Luo, Han Jiang, Ziang Xiao, Susu Zhang
llmsdiffusionbenchmarks
More like this →