arXiv Machine Learning September 10, 2026 By Mahesh Godavarti

Content-Based Addressing for Long Context

Read the original on arXiv Machine Learning →

The paper proposes a content‑based addressing scheme for long‑context models that replaces the growing token counter in Rotary Position Embedding (RoPE) with unit‑level addresses derived from the content of each unit. By dividing the token stream into units, the method preserves local RoPE behavior while allowing new units to be addressed via learned content maps, avoiding positional mismatches when extending context length. Experiments on character‑level Tiny Shakespeare show that a model trained on 256‑character contexts achieves lower perplexity at 4096 characters using this scheme, and a second diagnostic demonstrates retrieval of multiple serialized facts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

Oct 1

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

arXiv:2609.39929v1 Announce Type: cross Abstract: Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and disting...

By Yuyang Wu, Yufeng Du, Hao Peng

llms benchmarks

Hugging Face Trending Papers

Sep 30

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and distinguishing nearby positions. Determining which weaknes...

llms benchmarks

arXiv Machine Learning

Oct 9

Remember to Forget: Gated Adaptive Positional Encoding

The paper introduces GAPE (Gated Adaptive Positional Encoding), a drop‑in augmentation for rotary positional encodings that adds content‑aware bias to attention logits while preserving rotary geometry. GAPE uses a query‑dependent gate to contract irrelevant context and a key‑dependent gate to keep salient distant tokens, leading to exponential attenuation of weakly protected context. Experiments show that GAPE improves long‑context robustness, extrapolating up to eight times the training length, and can be retrofitted into a pretrained 7B model without losing benchmark performance.

By Riccardo Ali, Alessio Borgi, Mario Severino, Pietro Li\`o, Christopher Irwin

llms benchmarks safety

arXiv AI

Sep 10

Positional Encoding via Token-Aware Phase Attention

arXiv:2509.12635v4 Announce Type: replace-cross Abstract: We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention sco...

By Yu Wang, Sheng Shen, R\'emi Munos, Hongyuan Zhan, Yuandong Tian

rag safety

arXiv Machine Learning

Jul 9

How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

arXiv:2607. 07678v1 Announce Type: new Abstract: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly.

By Xinyi Wu, Siyuan Liu, Ali Jadbabaie

llms rag

arXiv AI

Jul 23

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.

By Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li

llms rag

One email a morning, machine-written

Related stories

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

Remember to Forget: Gated Adaptive Positional Encoding

Positional Encoding via Token-Aware Phase Attention

How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally