arXiv Machine Learning
Content-Based Addressing for Long Context
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
2d ago
Positional Encoding via Token-Aware Phase Attention
arXiv:2509.12635v4 Announce Type: replace-cross Abstract: We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention sco...
arXiv Machine Learning
Jul 9
How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
arXiv:2607. 07678v1 Announce Type: new Abstract: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly.
arXiv AI
Jul 23
AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
arXiv:2607. 19363v1 Announce Type: new Abstract: Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads.
arXiv Computation and Language
Aug 25
RIG-RoPE: Relation-Stratified Multimodal Attention with Instance-Local Rotary Geometry and Representation-Aware Traversal Coordinates
arXiv:2608.05154v3 Announce Type: replace Abstract: Multimodal rotary positional encodings apply temporal, height, and width phases to interleaved text, image, and video tokens. This creates two ambi...
arXiv Computation and Language
1d ago
Distance generalization in transformers: why bother with positional encoding?
arXiv:2609.11913v1 Announce Type: new Abstract: Out-of-distribution length generalization, namely to extrapolate a task from short to longer context, has been studied intensively for transformers. He...
arXiv AI
Jun 8
RePo: Language Models with Context Re-Positioning
arXiv:2512. 14391v3 Announce Type: replace-cross Abstract: In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices.