← Back to all news
arXiv Machine Learning By Riccardo Ali, Alessio Borgi, Mario Severino, Pietro Li\`o, Christopher Irwin

Remember to Forget: Gated Adaptive Positional Encoding

Read the original on arXiv Machine Learning →

The paper introduces GAPE (Gated Adaptive Positional Encoding), a drop‑in augmentation for rotary positional encodings that adds content‑aware bias to attention logits while preserving rotary geometry. GAPE uses a query‑dependent gate to contract irrelevant context and a key‑dependent gate to keep salient distant tokens, leading to exponential attenuation of weakly protected context. Experiments show that GAPE improves long‑context robustness, extrapolating up to eight times the training length, and can be retrofitted into a pretrained 7B model without losing benchmark performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Oct 1

Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head

The paper introduces NAMOH, a native sparse attention mechanism that activates only a subset of heads per token, allowing each head to attend to a limited subsequence of tokens. By scaling the number of heads while keeping the active heads per token fixed, the method shortens head histories and reduces key‑value access without increasing overall storage. Experiments demonstrate that NAMOH can outperform fully activated models with the same parameter count and enable more efficient long‑context inference than smaller dense models.

By Zizhuo Fu, Runsheng Wang, Meng Li
llmsrag
More like this →
arXiv AI
Jun 8

RePo: Language Models with Context Re-Positioning

arXiv:2512. 14391v3 Announce Type: replace-cross Abstract: In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices.

By Huayang Li, Tianyu Zhao, Deng Cai, Richard Sproat
llms
More like this →