Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning
Read the original on arXiv Computation and Language →The paper introduces LURCH, a benchmark of 6,550 question‑answering instances that quantifies long‑context reasoning difficulty using three properties: evidence scope, evidence dispersion, and reasoning depth. Experiments with both proprietary and open‑weight models show that these properties are strongly negatively correlated with performance, that performance drops sharply for dispersed evidence as context length grows, and that difficulty increases when multiple properties are high simultaneously. The study demonstrates that context length alone is insufficient to explain model failures, and that the new metrics provide a more nuanced understanding of long‑context challenges.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.