arXiv AI July 29, 2026 By Juan Francisco, Mandujano Reyes

Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

arXiv:2607. 25257v1 Announce Type: cross Abstract: Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 17

Can We Trust Item Response Theory for AI Evaluation?

arXiv:2607. 15190v1 Announce Type: new Abstract: AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate model capabilities, rank systems, select informative examples, and diagnose benchmark quality.

By Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao, Susu Zhang

llms multimodal benchmarks

arXiv AI

Jul 20

Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length

arXiv:2512. 07019v3 Announce Type: replace-cross Abstract: The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to provide guidance for both downstream applications and actionable future improvements.

By Zhiyu Xu, Jia Liu, Yixin Wang, Yuqi Gu

llms benchmarks

arXiv AI

Jul 15

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

arXiv:2511. 04689v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale.

By Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, Nitesh V. Chawla

llms benchmarks

arXiv AI

Jul 28

Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

arXiv:2601. 02580v2 Announce Type: replace-cross Abstract: Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration.

By Christopher Ormerod

llms fine-tuning

arXiv Machine Learning

Jul 30

BayesAME: Bayesian Active Model Evaluation

arXiv:2607. 27023v1 Announce Type: new Abstract: Evaluating large generative models across benchmarks is time-consuming and computationally expensive.

By Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet, Virginia Aglietti, Silvia Chiappa

benchmarks

arXiv AI

Aug 6

Revisiting Generalization Across Difficulty Levels: It's Not So Easy

arXiv:2511. 21692v3 Announce Type: replace-cross Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation.

By Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach

llms

Related stories

Can We Trust Item Response Theory for AI Evaluation?

Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

BayesAME: Bayesian Active Model Evaluation

Revisiting Generalization Across Difficulty Levels: It's Not So Easy