arXiv AI

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

The study investigates what makes medical exam items difficult by using large language models (LLMs) to generate difficulty rubrics based on empirical difficulty labels from the USMLE and PANCE. The induced rubrics identified patterns such as cue convergence, misleading cues, and option competition, while surface features like stem length or image presence were not major difficulty factors. Rubric guidance altered how LLMs applied difficulty scales but did not consistently enhance difficulty judgments, suggesting its main value lies in producing interpretable hypotheses rather than improving predictive accuracy.

arXiv AI

2d ago

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

The paper investigates using open‑weight large language models (LLMs) as both judges and simulated examinees to predict medical exam item difficulty from text alone. LLMs provided absolute difficulty ratings, pairwise comparisons, and response‑based signals such as correctness and option probabilities, which were combined with baseline features to improve predictions. The best models achieved Spearman correlations of .447 on USMLE and .494 on PANCE, surpassing prior shared‑task systems and matching proprietary models, demonstrating that LLM signals capture difficulty information beyond traditional text representations.

By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

llms

arXiv AI

Aug 6

Revisiting Generalization Across Difficulty Levels: It's Not So Easy

arXiv:2511. 21692v3 Announce Type: replace-cross Abstract: We investigate how well large language models (LLMs) generalize across different task difficulties, a key question for effective data curation and evaluation.

By Yeganeh Kordi, Nihal V. Nayak, Max Zuo, Ilana Nguyen, Stephen H. Bach

llms

arXiv Computation and Language

Oct 7

Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention

The study introduces MedQADE, a German open‑response clinical benchmark with 3,800 question‑answer pairs and physician reference annotations. It evaluates large language models (LLMs) as judges, finding that while some LLMs (e.g., Gemini 3 Flash) achieve physician‑level agreement on correctness, they exhibit self‑bias and low abstention rates. Physicians showed moderate agreement on correctness but limited agreement on difficulty, and their abstention increased with perceived difficulty.

By William Philipp, Finn Fassbender, Daniel Fister, Thorsten Langer, Martje G. Pauly, Rebecca Herzog, Markus A. Hobert, Theresa Paulus, Alexander Baumann, Chi Wang Ip, Lukas L. Goede, Johanna Reimer, Sebastian L\"ons, Ronald B\"ock, Sebastian Fudickar

llms benchmarks safety

arXiv Machine Learning

Aug 3

Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs

arXiv:2607. 28634v1 Announce Type: cross Abstract: The estimation of item difficulty plays a key role in both formative assessment and large-scale high-stakes summative assessments.

By Xinyi Wang, Hong Jiao, Ming Li, Sydney Peters, Hanna Choi, Tianyi Zhou, Qingshu Xu

llms rag

arXiv AI

Jul 31

The Easy Trap: Why LLMs Underestimate Misconception-Driven Difficulty

arXiv:2607. 26067v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for estimating item difficulty in educational assessment.

By Amanda La Hadi, Muhammad Johan Alibasa, Guanliang Chen, A. Taufiq Asyhari

llms safety

arXiv AI

Jul 3

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

arXiv:2607. 02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.

By Samiha A. Ismail, Fan X. Chen, Ali Merali

llms benchmarks

arXiv AI

Jul 31

Aligning LLM-Simulated and Human Examinees for Psychometric Calibration: A Cognitive Diagnostic Profiling Approach

arXiv:2607. 26317v1 Announce Type: cross Abstract: Psychometric calibration for educational tests typically requires costly human response data.

By Wenjie Zhou, Yunting Liu, Renjiao Tang, Mark Wilson

llms safety

arXiv AI

Oct 2

Scaling Clinical Judgment to Evaluate Medical AI

The paper introduces PrecepTron, a 32‑billion‑parameter language model fine‑tuned with low‑rank adaptation to evaluate clinical reasoning in large language models (LLMs) at a physician level. It also releases GRAND‑ROUNDS, a benchmark of 9,217 scored responses from 160 clinicians across seven studies. Using PrecepTron, the authors replicate key findings from major medical AI studies and explore new questions about LLM diagnostic accuracy, demonstrating that fine‑tuned models can provide consistent, scalable physician‑level scoring.

By Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur, Byron Crowe, Anthony M. Pettinato, Aashna P. Shah, Adrian D. Haimovich, Liam G. McCoy, Daniel Restrepo, Jason A. Freed, Ethan Goh, Jonathan H. Chen, Laura Zwaan, Katherine E. Goodman, Daniel J. Morgan, Raja-Elie E. Abdulnour, Adam Rodman, Arjun K. Manrai

llms fine-tuning benchmarks

arXiv AI

Oct 2

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

The paper introduces a data-driven method to generate natural-language explanations for why some questions are harder than others. It estimates question difficulty using Item Response Theory on large language model responses, then prompts an LLM to propose hypotheses that are validated on held-out data. Experiments on mathematical, logical, and commonsense datasets show that these hypotheses are interpretable, predictive, and can be used to adjust question difficulty causally.

By Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan

llms

arXiv AI

Jun 19

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

arXiv:2606. 19509v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to structured clinical data, yet whether they can recognize the limits of their own knowledge on such tasks remains unexplored.

By Akshat Dasula, Prasanna Desikan, Jaideep Srivastava

llms

arXiv AI

Jun 29

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

arXiv:2606. 28186v1 Announce Type: cross Abstract: Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction.

By Chenguang Wang, Ming Li, Xinyue Zeng, Zhuochun Li, Hong Jiao, Tianyi Zhou, Dawei Zhou

llms fine-tuning benchmarks safety

arXiv AI

Jul 3

Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

arXiv:2607. 02432v1 Announce Type: new Abstract: Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation.

By Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard, Francisco J. Rodriguez-Martinez, Lorena Otero-Cerdeira

llms robotics safety

Related stories