arXiv Computation and Language

MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors

The paper introduces MWE‑ECL, a bilingual diagnostic framework that tests whether distant discourse anchors can override local lexical priors in multi‑word expression interpretation. It evaluates models on a 0‑128K context grid, finding that while retrieval of anchors is near perfect, the ability to change locally preferred readings varies, especially when the model’s default conflicts with the anchor. The study shows that explicit recoverability does not always translate into behavioral influence, with gaps differing across models and languages.

arXiv AI
Nov 5

Reasoning over Non-Parametric Memory in Multilingual LLMs: A Cognitive-Based Analysis of Relational Knowledge

The paper introduces “Rel-Bloom,” a multilingual diagnostic benchmark of 450 multiple-choice questions that tests language models’ ability to reason over relational facts supplied directly in prompts. It evaluates 43 models across six languages and two multilingual settings, revealing that accuracy drops sharply from Apply to Analyze levels, that Evaluate items remain a bottleneck even for high-performing models, and that difficulty rankings are consistent across languages. The study also finds that counterfactual contexts are generally harder than factual ones, and that removing entity identities can sometimes improve higher-level performance.

By Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, Ehsaneddin Asgari
arXiv Computation and Language
Sep 25

Encoded but Not Decoded: Layer-Localized Evidence for a Three-Level Gap in LLM Syntax

The paper introduces a three-level evaluation framework—behavioral deployment, LM-head readout, and probe recoverability—to distinguish whether a language model fails a syntactic test by not encoding structure or by failing to use it. Using a trilingual control-dependency benchmark, the authors find that probe recoverability consistently exceeds LM-head readout, which in turn exceeds behavioral deployment across seven models and three languages, with the largest gap observed in Qwen3-0.6B Instruct. Layer-localized activation patching shows that instruction tuning shifts the decoded layer later, suggesting decoding favors surface shortcuts and that behavioral evaluation understates what models encode while probing alone overstates what they deploy.

By Zhenyan Lu, He Wang, Xiaohui Huang
arXiv Computation and Language
Sep 7

Choosing the Right Language Mode at Inference Time for Multilingual Reliability

The paper investigates how multilingual large language models can be guided to reason more reliably in low- to mid-resource languages by selecting appropriate language modes during inference. Experiments with LLaMA and Qwen models show that using English context can correct errors from non‑English comprehension, but adding redundant bilingual context can cause interference. To balance this trade‑off, the authors propose Reliability‑Aware Adaptive Inference (RAAI), a training‑free test‑time framework that routes prompts based on Expected Calibration Error and gates reasoning with a mid‑layer Risk Index, achieving up to 37.7% accuracy gains and reduced calibration error on low‑resource languages.

By Ekata Mitra, Ameeta Agrawal
arXiv Computation and Language
Nov 5

SRRM: A Retrieval-Reconstruction Benchmark for Evaluating Long-Context Memory in Small Language Models

The paper introduces SRRM, a lightweight benchmark that jointly evaluates targeted retrieval and global information reconstruction to assess long‑context memory in small language models. It also proposes the Memory Preservation Ratio (MPR) to quantify the relationship between retrieval and reconstruction performance. Experiments on Qwen2.5 models show that reconstruction degrades faster than retrieval as context grows, and that larger models do not necessarily perform better in long‑context memory.

By Christopher Alberto Hamonangan