arXiv Computation and Language

Distance generalization in transformers: why bother with positional encoding?

arXiv Machine Learning

Jul 22

Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention

arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.

By Subham Singh, Ashutosh Mishra, Subha Raut

llms safety

arXiv Machine Learning

2d ago

Content-Based Addressing for Long Context

arXiv:2609.07314v1 Announce Type: cross Abstract: Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local tok...

By Mahesh Godavarti

rag benchmarks safety

arXiv Machine Learning

Jul 9

How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization

arXiv:2607. 07678v1 Announce Type: new Abstract: Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly.

By Xinyi Wu, Siyuan Liu, Ali Jadbabaie

llms rag

arXiv Machine Learning

Jun 25

Why Do Accumulated Transformations Extrapolate?

arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.

By Mahesh Godavarti

arXiv Machine Learning

Aug 4

Understanding Rate-Distortion Performance in Distributed Transformer Inference

arXiv:2601. 22002v5 Announce Type: replace Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference.

By Anderson de Andrade, Alon Harell, Ivan V. Baji\'c

llms

arXiv AI

Jun 18

The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior

arXiv:2604. 13082v2 Announce Type: replace-cross Abstract: Grokking in transformers trained on algorithmic tasks is characterized by a long delay between training-set fit and abrupt generalization, but the source of that delay remains poorly understood.

By Laura Gomezjurado Gonzalez

llms safety

arXiv Machine Learning

Aug 11

Barriers to Universal Reasoning With Transformers (And How to Overcome Them)

arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.

By Oliver Kraus, Yash Sarrof, Yuekun Yao, Alexander Koller, Michael Hahn

llms

arXiv Machine Learning

Jul 7

How Much is Left? LLMs Linearly Encode Their Remaining Output Length

arXiv:2607. 05316v1 Announce Type: cross Abstract: Large language models generate one token at a time, yet their responses show remarkably consistent length structure: step-by-step solutions converge in predictable token counts, retrievals stop after a few sentences, retractions extend responses by measurable amounts.

By Mohamed Amine Merzouk, Dmitri Carpov, Mirko Bronzi, Damiano Fornasiere, Adam Oberman

llms

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv Machine Learning

Jul 2

Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization

arXiv:2607. 00479v1 Announce Type: new Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning.

By Peilin Liu, Ding-Xuan Zhou

llms

arXiv Machine Learning

Jun 25

The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

arXiv:2606. 25450v1 Announce Type: new Abstract: Traditional evaluations measure a learning algorithm's final performance on an i.

By Jinghan Zhang, Zerui Cheng, Shiqi Chen, Ge Zhang, Wenhao Huang, Jiashuo Liu, Junxian He, Tianle Cai

reinforcement-learning fine-tuning efficiency benchmarks

Hugging Face Trending Papers

Jul 6

How Much is Left? LLMs Linearly Encode Their Remaining Output Length

Large language models generate one token at a time, yet their responses show remarkably consistent length structure: step-by-step solutions converge in predictable token counts, retrievals stop after a few sentences, retractions extend responses by measurable amounts. We ask whether the model carries an internal estimate of how much response remains.

llms

Related stories