The paper introduces LURCH, a benchmark of 6,550 question‑answering instances that quantifies long‑context reasoning difficulty using three properties: evidence scope, evidence dispersion, and reasoning depth. Experiments with both proprietary and open‑weight models show that these properties are strongly negatively correlated with performance, that performance drops sharply for dispersed evidence as context length grows, and that difficulty increases when multiple properties are high simultaneously. The study demonstrates that context length alone is insufficient to explain model failures, and that the new metrics provide a more nuanced understanding of long‑context challenges.
By Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley, Shay B. Cohen, Mirella Lapata, Miao Li
The paper introduces a data-driven method to generate natural-language explanations for why some questions are harder than others. It estimates question difficulty using Item Response Theory on large language model responses, then prompts an LLM to propose hypotheses that are validated on held-out data. Experiments on mathematical, logical, and commonsense datasets show that these hypotheses are interpretable, predictive, and can be used to adjust question difficulty causally.
By Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan
arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.
By Siddhartha Jain, Ameya Velingker
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.
arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.
By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen
arXiv:2606. 24267v2 Announce Type: replace-cross Abstract: While in-context learning is generally shown to be effective in Large Language Models (LLMs), bad contexts can cause performance degradation and mode collapse, a phenomenon we call "pigeonholing.
By Hyunji Nam, Keertana Chidambaram, Dorottya Demszky, Natasha Jaques