Long-context evaluations often test whether a model can recover distant evidence, but recoverability does not guarantee behavioral influence. We test the prediction that a distant discourse anchor can...
arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.
By Mohsen Arjmandi
The paper introduces “Rel-Bloom,” a multilingual diagnostic benchmark of 450 multiple-choice questions that tests language models’ ability to reason over relational facts supplied directly in prompts. It evaluates 43 models across six languages and two multilingual settings, revealing that accuracy drops sharply from Apply to Analyze levels, that Evaluate items remain a bottleneck even for high-performing models, and that difficulty rankings are consistent across languages. The study also finds that counterfactual contexts are generally harder than factual ones, and that removing entity identities can sometimes improve higher-level performance.
By Zohreh Abbasi Liasi, Batoul Hammoud, Anas Madkoor, Ehsaneddin Asgari
The paper introduces a three-level evaluation framework—behavioral deployment, LM-head readout, and probe recoverability—to distinguish whether a language model fails a syntactic test by not encoding structure or by failing to use it. Using a trilingual control-dependency benchmark, the authors find that probe recoverability consistently exceeds LM-head readout, which in turn exceeds behavioral deployment across seven models and three languages, with the largest gap observed in Qwen3-0.6B Instruct. Layer-localized activation patching shows that instruction tuning shifts the decoded layer later, suggesting decoding favors surface shortcuts and that behavioral evaluation understates what models encode while probing alone overstates what they deploy.
By Zhenyan Lu, He Wang, Xiaohui Huang
arXiv:2610.33764v1 Announce Type: new
Abstract: Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered...
By Yuyang Dai, Yuxia Wang, Zhuohan Xie
arXiv:2609.39929v1 Announce Type: cross
Abstract: Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and disting...
By Yuyang Wu, Yufeng Du, Hao Peng
Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and distinguishing nearby positions. Determining which weaknes...
arXiv:2610.02926v1 Announce Type: new
Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multiling...
By Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth
arXiv:2608. 04021v1 Announce Type: cross Abstract: Cloze-style probes that vary how often a target token appears implicitly assume that more copies of a target affect prediction the same way regardless of where the readout slot sits.
By Han-yu Wang
The paper investigates how multilingual large language models can be guided to reason more reliably in low- to mid-resource languages by selecting appropriate language modes during inference. Experiments with LLaMA and Qwen models show that using English context can correct errors from non‑English comprehension, but adding redundant bilingual context can cause interference. To balance this trade‑off, the authors propose Reliability‑Aware Adaptive Inference (RAAI), a training‑free test‑time framework that routes prompts based on Expected Calibration Error and gates reasoning with a mid‑layer Risk Index, achieving up to 37.7% accuracy gains and reduced calibration error on low‑resource languages.
By Ekata Mitra, Ameeta Agrawal
The paper introduces SRRM, a lightweight benchmark that jointly evaluates targeted retrieval and global information reconstruction to assess long‑context memory in small language models. It also proposes the Memory Preservation Ratio (MPR) to quantify the relationship between retrieval and reconstruction performance. Experiments on Qwen2.5 models show that reconstruction degrades faster than retrieval as context grows, and that larger models do not necessarily perform better in long‑context memory.
By Christopher Alberto Hamonangan
arXiv:2602.22453v4 Announce Type: replace
Abstract: Retrieval heads, a subset of attention heads in Transformers, were studied in English, showing its crucial role in retrieving information from the...
By Shaswat Patel, Vishvesh Trivedi, Yue Han, Yihuai Hong, Eunsol Choi