arXiv AI November 6, 2026 By Dikshant Kukreja, Gautam Gupta, Avni Mittal, Avinash Anand, Kritarth Prasad, Zhengkui Wang, Timothy Liu, Rajiv Ratn Shah, Aik Beng Ng, Simon See, Erik Cambria

The Intrusion Pathway: How Non-Evidential Context Degrades Language Model Performance

Read the original on arXiv AI →

The paper identifies that language models lose accuracy when prompts include non‑evidential context—irrelevant statements or random tokens that do not overlap semantically with the query. This degradation is traced to an "Intrusion Pathway" involving specific attention heads that amplify distractor content and weaken correct answers. By targeting and masking a small set of these intrusion heads, the study recovers most of the lost performance with minimal impact on useful context.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

llms efficiency

arXiv Machine Learning

Sep 22

Context Poisoning as Extreme-Value Attention Interference in Long-Context Language Models

arXiv:2609.22101v1 Announce Type: cross Abstract: Large language models can process increasingly long prompts, yet their ability to locate and use decisive evidence may degrade as irrelevant or confu...

By Meysam Ghaffari, Nina Fatehi, Bhaskar Sen, Nasim Sabetpour, Carlos Morato

llms

arXiv Machine Learning

Jul 27

Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

arXiv:2607. 21692v1 Announce Type: new Abstract: Sparse attention reduces the cost of long contexts by allowing each query to read only selected parts of the input.

By Jim Allchin

arXiv AI

Oct 15

From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models

arXiv:2610.18674v1 Announce Type: cross Abstract: Large language models rely on prompt context to perform in-context learning, follow formats, and adapt to local evidence. This ability, however, also...

By Jinghan Zheng, Jiajun Shi

llms benchmarks

arXiv AI

Oct 15

What the Model Cannot See: Covert and Provable Text Invisibility in LLMs

arXiv:2610.15883v1 Announce Type: cross Abstract: Text wrapped in a trigger can be made behaviorally invisible to a language model: omitted from summaries and quotations, ignored in question answerin...

By Mark Russinovich

llms nlp benchmarks

arXiv Computation and Language

1d ago

Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs

arXiv:2505.09338v3 Announce Type: replace Abstract: We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanisti...

By Jingcheng Niu, Xingdi Yuan, Tong Wang, Hamidreza Saghir, Amir H. Abdi

llms

One email a morning, machine-written

Related stories

Language Models Can Control Their Own Attention

Context Poisoning as Extreme-Value Attention Interference in Long-Context Language Models

Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

From Relation-Specific Heads to Shared Interventions: Contextual Entrainment in Language Models

What the Model Cannot See: Covert and Provable Text Invisibility in LLMs

Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs