The paper investigates how large language models can accurately copy long repeated strings, a task that standard transformers struggle with due to overrun or truncation. It proposes a two‑layer transformer using RoPE‑style rotations (VO‑rotations) with small prime periods to encode distances modulo primes, enabling the model to distinguish all necessary positions via the Chinese Remainder Theorem and correctly terminate copying. Experiments show that baseline transformers fail on long repeats, while the VO‑based model generalizes to lengths up to about 5005 with near‑perfect accuracy.
arXiv:2607. 18759v1 Announce Type: new Abstract: Transformers with relative positional encodings often extrapolate to sequences longer than those seen during training, whereas transformers with learned absolute encodings typically do not.
The paper investigates distance generalization in transformer models, focusing on how well they can handle changes in inter-token distances between training and inference while keeping context length fixed. Using two synthetic delay-copy tasks that require copying tokens after finite delays, the authors evaluate the impact of positional encoding schemes (RoPE, ALiBi, and NoPE), the diversity of distances seen during training, and the conditions under which distance transfer learning is beneficial or detrimental. Their comprehensive study highlights the importance of understanding the underlying mechanisms that govern distance generalization in transformers.
arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.
arXiv:2605. 18079v2 Announce Type: replace Abstract: Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice.
arXiv:2604. 25800v2 Announce Type: replace Abstract: Chain-of-Thought (CoT) has been shown to empirically improve Transformers' performance, and theoretically increase their expressivity to Turing completeness.
The Recurrent Looped Transformer (RLT) divides its layers into a parallel causal encoder and a recurrent decoder, allowing the computation path to grow with sequence length while keeping a fixed per-token cost. Experiments on six algorithmic tasks show that RLT outperforms a standard eight‑layer Transformer, achieving perfect generalization on parity up to 256 bits and high accuracy on swap‑based permutation tracking and modular arithmetic beyond training lengths. Ablation studies confirm that the recurrent feedback is essential for these gains, with chunked updates preserving performance on parity but reducing accuracy on permutation tracking.
arXiv:2610.26974v1 Announce Type: new Abstract: We show that two-layer transformers trained to near-zero loss on a sorting task converge to one of two mechanistically distinct solution classes depend...
The paper introduces Universal Attention, a transformer architecture that unifies and extends decay-based KV‑cache pruning mechanisms while preserving RoPE embeddings and Softmax attention. It achieves highly expressive, end‑to‑end trainable pruning that adaptively removes low‑contribution tokens, yielding up to 10× compression on standard datasets and 25× compression at 16k context length without sacrificing downstream performance. The approach outperforms existing state‑of‑the‑art baselines and unpruned models in both compression and generalization.
arXiv:2604.13082v3 Announce Type: replace-cross Abstract: Large language models today still make mistakes on multi-digit arithmetic when they do not have tools, and part of the reason is how numbers...
arXiv:2608.30310v1 Announce Type: cross Abstract: Hybrid large language models interleave full-attention layers with linear-attention layers to reduce the cost of long-context inference. This structu...
arXiv:2606. 27538v1 Announce Type: cross Abstract: We introduce the context-ready transformer, a new recurrent neural network architecture built from a D-layer transformer block that pre-contextualizes each token before it enters the block.
arXiv:2607. 22361v1 Announce Type: new Abstract: We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive.