arXiv AI August 24, 2026 By Chunhan Li, Chenglin Xu, Zongyang Zhang, Jiale Liu, Zhuoxi Rao, Xudong Jia, Junxiu He, Menglin Yang, Wenjuan Gong, Zhengzhe Liu, Chengwei Qin

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Read the original on arXiv AI →

MGAL is a new multilingual benchmark for evaluating long‑context large language models, built from United Nations reports in six official UN languages and covering 8K to 128K tokens. It tests four linguistic granularities—word, sentence, paragraph, and document—while also stratifying examples by their position within the document (begin, middle, end). Experiments show that models excel at word‑level tasks but struggle with coarser granularity, and that closed‑source models outperform others in lower‑resource languages, revealing challenges such as local semantic crowding and a fluency‑consistency gap.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

3d ago

Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

Granuscore is a reference‑free metric that measures the granularity of text by exploiting the structure of a hierarchical embedding space. It successfully reproduces known hierarchical orderings on the Granola‑EQ dataset, distinguishes granularity across different discourse contexts, and explains sentence‑specificity variations beyond sentence length. The authors also apply Granuscore to four question‑answering benchmarks, revealing systematic differences in granularity among questions, gold answers, and model outputs, thereby offering a new lens for assessing QA dataset difficulty.

By Lukas Ellinger, Alexander Fichtl, Miriam Ansch\"utz, Georg Groh

rag nlp benchmarks

arXiv AI

Jul 10

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.

By Siddhartha Jain, Ameya Velingker

llms nlp benchmarks

Hugging Face Trending Papers

Jul 9

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.

llms nlp benchmarks

arXiv AI

Aug 13

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.

By Arda Uzunoglu, Benjamin van Durme, Daniel Khashabi

llms fine-tuning

arXiv Machine Learning

Aug 5

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.

By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y

llms nlp benchmarks safety

arXiv AI

Jun 2

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization

arXiv:2606. 01252v1 Announce Type: cross Abstract: Multi-target cross-lingual text summarization (MTXLS), which summarizes a source document into multiple target languages, is increasingly important as users consume content in diverse languages, but remains underexplored.

By Sangwon Ryu, Yihong Liu, Mingyang Wang, Yunsu Kim, Jungseul Ok, Gary Geunbae Lee, Hinrich Schuetze

llms nlp benchmarks

One email a morning, machine-written

Related stories

Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

Understanding LLM Behavior in Multi-Target Cross-Lingual Summarization