Positional encoding (PE) is widely viewed as necessary for transformers to process ordered sequences: without them, the next-token map appears permutation-invariant in its context tokens. This intuition underlies all prior universality results, which rely on positional information to prove that transformers with chain-of-thought can perform arbitrary computation, i.
arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.
By Oliver Kraus, Yash Sarrof, Yuekun Yao, Alexander Koller, Michael Hahn
arXiv:2607. 17710v1 Announce Type: new Abstract: Large Language Models (LLMs) have had a remarkable impact across many areas of machine learning.
By Ehsan Futuhi, Nathan R. Sturtevant
arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.
By Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford
arXiv:2607. 26988v1 Announce Type: cross Abstract: What types of decision problems can a causally masked, finite-precision transformer solve for inputs of arbitrary length?
By Franz Nowak, Ryan Cotterell, Reda Boumasmoud
arXiv:2608. 11909v1 Announce Type: new Abstract: Two accounts recur in explanations of the success of rotary position embeddings (RoPE).
By Selim Jerad, Anej Svete, Jiaoda Li, Ryan Cotterell
arXiv:2606. 19697v1 Announce Type: cross Abstract: The increasing popularity of \emph{reasoning} models -- language models that output a series of reasoning or thought tokens before producing an answer -- is justified, in part, by theoretical results showing that chain-of-thought (CoT) transformers can simulate Turing machines, and thus perform arbitrary computation.
By Yanhong Li, Anej Svete, Ashish Sabharwal, William Merrill
arXiv:2603. 17019v2 Announce Type: replace Abstract: A central question in the debate over large language models is whether transformers can learn rules they have never seen, or whether they can only interpolate: predict new cases from their similarity to training examples.
By Andy Gray
arXiv:2608. 09432v1 Announce Type: cross Abstract: Transformer-based language models rely on self-attention, whose computation is permutation-equivariant and therefore lacks an intrinsic mechanism for representing token order.
By R\'ois\'in Luo
arXiv:2605. 18079v2 Announce Type: replace Abstract: Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice.
By Moritz Br\"osamle, Stephan Eckstein
arXiv:2608. 03921v1 Announce Type: new Abstract: This paper offers a new interpretation of the Transformer during inference.
By Marco Giunti, Fabrizia Giulia Garavaglia
arXiv:2607. 15893v1 Announce Type: cross Abstract: While the internal mechanisms of autoregressive (AR) transformers have been studied extensively, much less is known about diffusion language models (DLMs), an emerging alternative that generates text by iterative denoising.
By Andy Catruna, Emilian Radoi