Diagnosing Model--Item Interactions in Language Model Benchmarks with LSIRM

The paper introduces the latent space item response model (LSIRM) as a tool for uncovering how language models differ on specific benchmark items. By mapping 1,348 models across 5,761 BBH items and comparing scoring methods across BBH, GPQA, MMLU-Pro, and MuSR, the authors show that items with similar correct answers cluster together, especially when scored by option labels. LSIRM separates a model’s tendency to perform better on certain answer categories from other performance differences, revealing that small changes in scoring can dramatically alter model rankings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• diffusion
• benchmarks