arXiv Computation and Language September 30, 2026 By Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen, Xi Ye

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Read the original on arXiv Computation and Language →

The paper introduces LongHarness Bench, a new benchmark designed to evaluate both the effectiveness and efficiency of language model harnesses for long-context reasoning. It features tasks that require diverse retrieval strategies—such as lexical search and semantic matching—and strategic, adaptive reasoning over global and local context, with only a small subset of the context being useful at each step. Evaluations across multiple state‑of‑the‑art models and harnesses show that even strong combinations achieve only 68% macro‑average accuracy, and that the same model can vary markedly in efficiency depending on the harness used.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jul 10

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.

By Siddhartha Jain, Ameya Velingker

llms nlp benchmarks

Hugging Face Trending Papers

Jul 9

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.

llms nlp benchmarks

arXiv AI

Jun 4

Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

arXiv:2505. 17315v2 Announce Type: replace Abstract: Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored.

By Wang Yang, Zirui Liu, Hongye Jin, Qingyu Yin, Vipin Chaudhary, Xiaotian Han

llms fine-tuning benchmarks

arXiv AI

Jul 3

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

arXiv:2607. 02509v1 Announce Type: new Abstract: Understanding and reasoning over long contexts has become a key requirement for deploying large language models (LLMs) in realistic applications.

By Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, Jingrui He

llms efficiency

arXiv AI

Jun 4

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

arXiv:2505. 19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.

By Wang Yang, Hongye Jin, Shaochen Zhong, Song Jiang, Qifan Wang, Vipin Chaudhary, Xiaotian Han

llms benchmarks

arXiv Machine Learning

Sep 1

PERK: Long-Context Reasoning as Test-Time Learning

arXiv:2507.06415v3 Announce Type: replace-cross Abstract: Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK...

By Zeming Chen, Angelika Romanou, Gail Weiss, Antoine Bosselut

llms fine-tuning

One email a morning, machine-written

Related stories

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

PERK: Long-Context Reasoning as Test-Time Learning