arXiv AI
Positional Encoding via Token-Aware Phase Attention
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 11
RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
arXiv:2606. 11275v1 Announce Type: cross Abstract: Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of its distance from the query.
arXiv AI
Jul 23
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
arXiv AI
Jun 18
Dual Dimensionality for Local and Global Attention
arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.
arXiv Machine Learning
2d ago
Content-Based Addressing for Long Context
arXiv:2609.07314v1 Announce Type: cross Abstract: Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local tok...
arXiv AI
Jul 24
LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
arXiv:2604. 00004v2 Announce Type: replace-cross Abstract: The extension of context windows in Large Language Models is typically facilitated by scaling positional encodings followed by lightweight Continual Pre-Training (CPT).
Hugging Face Trending Papers
Jun 2
When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics
Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores.