arXiv AI June 11, 2026 By Alejandro Garc\'ia-Castellanos, Maurice Weiler, Erik J Bekkers

RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

arXiv:2606. 11275v1 Announce Type: cross Abstract: Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of its distance from the query.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 10

Positional Encoding via Token-Aware Phase Attention

arXiv:2509.12635v4 Announce Type: replace-cross Abstract: We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention sco...

By Yu Wang, Sheng Shen, R\'emi Munos, Hongyuan Zhan, Yuandong Tian

rag safety

arXiv AI

4d ago

How Local Mixing Encodes Relative Position in Global NoPE Attention

arXiv:2609.38109v1 Announce Type: cross Abstract: The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of...

By Cutter Dawes, Nick Alonso, Tom Figliolia, Beren Millidge

llms safety

arXiv Machine Learning

Jun 2

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

arXiv:2602. 23197v2 Announce Type: replace-cross Abstract: Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations.

By Chungpa Lee, Jy-yong Sohn, Kangwook Lee

llms fine-tuning

arXiv Machine Learning

Jun 24

Selective Rotary Position Embedding

arXiv:2511. 17388v3 Announce Type: replace-cross Abstract: Position information is essential for language modeling.

By Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher

llms rag

arXiv Computer Vision

Sep 10

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

arXiv:2609.06712v2 Announce Type: replace Abstract: Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequ...

By Zekun Zhang, Yixiang Cai, Yuxi Liu, Tengxu Sun, Tianle Liu, Zhoutong Wu, Haoyu Li, Baole Ai, Ang Wang, Jiamang Wang, Lin Qu, Kun Yuan

llms rag diffusion efficiency

arXiv AI

Jul 23

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally

arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.

By Shaowen Wang, Yuke Zheng, Tansheng Zhu, Shuang Chen, Shaofan Liu, Suncong Zheng, Jian Li

llms rag

One email a morning, machine-written

Related stories

Positional Encoding via Token-Aware Phase Attention

How Local Mixing Encodes Relative Position in Global NoPE Attention

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

Selective Rotary Position Embedding

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally