arXiv:2605. 10828v2 Announce Type: replace Abstract: As large language models are increasingly deployed in retrieval-augmented generation and agentic systems that accumulate extensive context, understanding how distracting information affects long-context performance becomes critical.
arXiv:2609.22101v1 Announce Type: cross Abstract: Large language models can process increasingly long prompts, yet their ability to locate and use decisive evidence may degrade as irrelevant or confu...
MUDDLE is a benchmark designed to disentangle the effects of document length and topical distractors on document question‑answering systems. It contains 270 human‑annotated questions, each tested in five conditions: the source alone, the source with two or four hard negatives (topically similar), and the source with two or four random distractors matched in length and provenance. Experiments with GPT‑5‑mini show that hard negatives reduce accuracy more than length‑matched random distractors, indicating that topical similarity is a more significant source of error than length alone.
arXiv:2608. 03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant information is preserved.
arXiv:2511. 21338v2 Announce Type: replace Abstract: Masked Diffusion Language Models (MDLMs) have recently emerged as a promising alternative to Autoregressive Language Models (ARLMs), leveraging a denoising objective that, in principle, should enable more uniform context utilisation.
The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.
The paper identifies that language models lose accuracy when prompts include non‑evidential context—irrelevant statements or random tokens that do not overlap semantically with the query. This degradation is traced to an "Intrusion Pathway" involving specific attention heads that amplify distractor content and weaken correct answers. By targeting and masking a small set of these intrusion heads, the study recovers most of the lost performance with minimal impact on useful context.
arXiv:2511. 21397v2 Announce Type: replace-cross Abstract: How does irrelevant information (i.
arXiv:2602. 00344v2 Announce Type: replace-cross Abstract: While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens.
arXiv:2505.09338v3 Announce Type: replace Abstract: We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanisti...
arXiv:2608. 03711v1 Announce Type: cross Abstract: In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text.
Large language models (LLMs) exhibit a human‑like working‑memory pattern on the N‑back task, with performance dropping as memory load increases and errors mainly on lure items. Two hypotheses—precision limits of softmax attention and interference from competing items—were tested across 15 pretrained LLMs and minimal transformers. Results show that interference, not precision, constrains LLMs’ working‑memory performance, as suppressing non‑target item identities improves sensitivity and larger models benefit more from reducing interference.
The paper studies how redundant context influences attention mechanisms, distinguishing between normalization effects from repeated items and score variability within correlated groups. Using a planted retrieval model with exponentially many Gaussian distractors organized into large clusters, the authors derive exact retrieval thresholds via a two‑level generalized random energy model and Gaussian interpolation. They show that subtracting the log cluster size from pre‑softmax scores removes multiplicity but fails to eliminate extreme within‑cluster scores, establishing a sharp limit on signal strength that cannot be overcome by temperature scaling alone; averaging scores before exponentiation mitigates this effect but only when the relevant item is a singleton.