arXiv AI By Christopher Brooks (School of Information, University of Michigan)

What Reaches Expert Review? Representation, Structural Screening, and Candidate-Form Dependence in AI-Assisted Item Development

Read the original on arXiv AI →

The article examines how AI‑assisted item generation is filtered by a computational evaluator before expert review, focusing on representation, structural screening, and candidate‑form dependence. Through two in‑silico studies of 32,000 Big Five items, the authors show that subtle differences in semantic representation and structural evaluation lead to divergent item selections, even when overall content coverage appears stable. The findings reveal that the evaluator, often treated as a neutral technical step, actually shapes the evidence and wording that psychometricians ultimately review, highlighting its role as a revisable component of measurement design.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
4d ago

Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal

arXiv:2608.20385v1 Announce Type: new Abstract: Systematic reviews rely on quality appraisal of included studies, a process that is time-consuming and sensitive to ambiguity in checklist criteria. Al...

By Timo van der Kuil (Methodology and Statistics Utrecht University), Bruno Messina Coimbra (Methodology and Statistics Utrecht University), Mirjam van Zuiden (Clinical Psychology Utrecht University), Robert A. Bagheri (Methodology and Statistics Utrecht University), Rens van de Schoot (Methodology and Statistics Utrecht University), Klaas Dieleman (Methodology and Statistics Utrecht University), Berend Greijn (Methodology and Statistics Utrecht University), Stefan Houkes (Methodology and Statistics Utrecht University), Sebastiaan Rodenhuis (Methodology and Statistics Utrecht University), Elizabeth M. Grandfield (Methodology and Statistics Utrecht University)
arXiv AI
Aug 5

How Closely Do LLM Reviews Align with Human Peer Review?

arXiv:2608. 03659v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to generate scientific reviews, yet existing evaluations rarely examine whether different providers align with both conference decisions and human reviewing priorities within the same controlled setting.

By Abraham Camelo-Guerrero, Jairo Diaz-Rodriguez