← Back to all news
arXiv Machine Learning By Xiaoyu Yang, Jie Lu, Wei Duan, En Yu

The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence

Read the original on arXiv Machine Learning →

The paper introduces the Proximity Trap, a problem where long-context language models fail to attend to distant evidence because of competition from abundant, irrelevant proximal background. To mitigate this, the authors propose LYRA, a t‑distributed directional matching mechanism that reshapes the retrieval distribution to focus more on task‑relevant evidence while preserving positional information. Experiments on LongBench‑v2, RULER, and LongBench show consistent performance gains, and a new benchmark, ProxBench, is presented to evaluate distant evidence usage under varying proximal interference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 30

MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers

arXiv:2606. 29844v1 Announce Type: cross Abstract: The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios.

By Linrui Ma, Chun Hei Lo, Xinyu Wang, Peng Lu, Xihao Yuan, Hanting Chen, Kai Han, Xinghao Chen, Chengjun Zhan, Hanlin Xu, Yichun Yin, Lifeng Shang, Feng Wen, Boxing Chen, Yufei Cui
llms
More like this →
arXiv Machine Learning
Oct 9

Remember to Forget: Gated Adaptive Positional Encoding

The paper introduces GAPE (Gated Adaptive Positional Encoding), a drop‑in augmentation for rotary positional encodings that adds content‑aware bias to attention logits while preserving rotary geometry. GAPE uses a query‑dependent gate to contract irrelevant context and a key‑dependent gate to keep salient distant tokens, leading to exponential attenuation of weakly protected context. Experiments show that GAPE improves long‑context robustness, extrapolating up to eight times the training length, and can be retrofitted into a pretrained 7B model without losing benchmark performance.

By Riccardo Ali, Alessio Borgi, Mario Severino, Pietro Li\`o, Christopher Irwin
llmsbenchmarkssafety
More like this →