← Back to all news
arXiv AI

LLMs as Judges and Examinees for Medical Item Difficulty Prediction

The paper investigates using open‑weight large language models (LLMs) as both judges and simulated examinees to predict medical exam item difficulty from text alone. LLMs provided absolute difficulty ratings, pairwise comparisons, and response‑based signals such as correctness and option probabilities, which were combined with baseline features to improve predictions. The best models achieved Spearman correlations of .447 on USMLE and .494 on PANCE, surpassing prior shared‑task systems and matching proprietary models, demonstrating that LLM signals capture difficulty information beyond traditional text representations.

Related stories

arXiv AI
2d ago

What Makes Medical Examination Items Difficult? Using Large Language Models to Explore Patterns Associated With Empirical Item Difficulty

The study investigates what makes medical exam items difficult by using large language models (LLMs) to generate difficulty rubrics based on empirical difficulty labels from the USMLE and PANCE. The induced rubrics identified patterns such as cue convergence, misleading cues, and option competition, while surface features like stem length or image presence were not major difficulty factors. Rubric guidance altered how LLMs applied difficulty scales but did not consistently enhance difficulty judgments, suggesting its main value lies in producing interpretable hypotheses rather than improving predictive accuracy.

By Xiaomeng Xiong, Marcus Walker, Yanlin Jiang
llms
More like this →
arXiv AI
Sep 3

Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning

The paper proposes a response‑free method for estimating difficulty of reading‑comprehension multiple‑choice items by fine‑tuning a transformer on item wording. It introduces two extensions to a baseline joint‑encoding model: a component‑wise variant that encodes passage, question, and options separately, and a multi‑task variant that adds a question‑answering auxiliary task. Experiments on a corpus of nearly 30,000 items show that both extensions outperform the baseline, especially the multi‑task variant across all metrics and the component‑wise variant in rank ordering, even with limited training data.

By Jan Net\'ik, Patr\'icia Martinkov\'a
llmsnlpfine-tuning
More like this →
Hugging Face Trending Papers
Jul 8

From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings

Newly developed items must ordinarily be field tested before their psychometric properties are known, creating a cold start problem for item calibration. Predicting item parameters from features is a long standing measurement problem dating back to the Linear Logistic Test Model; modern text embeddings now automate the design matrices traditionally specified by hand.

ragbenchmarks
More like this →
arXiv AI
Oct 2

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

The paper introduces a data-driven method to generate natural-language explanations for why some questions are harder than others. It estimates question difficulty using Item Response Theory on large language model responses, then prompts an LLM to propose hypotheses that are validated on held-out data. Experiments on mathematical, logical, and commonsense datasets show that these hypotheses are interpretable, predictive, and can be used to adjust question difficulty causally.

By Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan
llms
More like this →