← Back to all news
arXiv AI By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

Read the original on arXiv AI →

The study investigates what makes medical exam items difficult by using large language models (LLMs) to generate difficulty rubrics based on empirical difficulty labels from the USMLE and PANCE. The induced rubrics identified patterns such as cue convergence, misleading cues, and option competition, while surface features like stem length or image presence were not major difficulty factors. Rubric guidance altered how LLMs applied difficulty scales but did not consistently enhance difficulty judgments, suggesting its main value lies in producing interpretable hypotheses rather than improving predictive accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
2d ago

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

The paper investigates using open‑weight large language models (LLMs) as both judges and simulated examinees to predict medical exam item difficulty from text alone. LLMs provided absolute difficulty ratings, pairwise comparisons, and response‑based signals such as correctness and option probabilities, which were combined with baseline features to improve predictions. The best models achieved Spearman correlations of .447 on USMLE and .494 on PANCE, surpassing prior shared‑task systems and matching proprietary models, demonstrating that LLM signals capture difficulty information beyond traditional text representations.

By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
llms
More like this →
arXiv Computation and Language
Oct 7

Evaluating Large Language Model Raters for German Open-Response Clinical Questions: A Physician-Annotated Benchmark Study of Agreement, Evaluator Bias, and Abstention

The study introduces MedQADE, a German open‑response clinical benchmark with 3,800 question‑answer pairs and physician reference annotations. It evaluates large language models (LLMs) as judges, finding that while some LLMs (e.g., Gemini 3 Flash) achieve physician‑level agreement on correctness, they exhibit self‑bias and low abstention rates. Physicians showed moderate agreement on correctness but limited agreement on difficulty, and their abstention increased with perceived difficulty.

By William Philipp, Finn Fassbender, Daniel Fister, Thorsten Langer, Martje G. Pauly, Rebecca Herzog, Markus A. Hobert, Theresa Paulus, Alexander Baumann, Chi Wang Ip, Lukas L. Goede, Johanna Reimer, Sebastian L\"ons, Ronald B\"ock, Sebastian Fudickar
llmsbenchmarkssafety
More like this →