← Back to all news
arXiv Machine Learning By Anxin Guo, Tianlong Huang, Chenxiao Yang, Zhiyuan Li

Copying Under Repetition: From Prefix Matching to Position Hashing

Read the original on arXiv Machine Learning →

The paper investigates how large language models can accurately copy long repeated strings, a task that standard transformers struggle with due to overrun or truncation. It proposes a two‑layer transformer using RoPE‑style rotations (VO‑rotations) with small prime periods to encode distances modulo primes, enabling the model to distinguish all necessary positions via the Chinese Remainder Theorem and correctly terminate copying. Experiments show that baseline transformers fail on long repeats, while the VO‑based model generalizes to lengths up to about 5005 with near‑perfect accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 11

Distance generalization in transformers: why bother with positional encoding?

The paper investigates distance generalization in transformer models, focusing on how well they can handle changes in inter-token distances between training and inference while keeping context length fixed. Using two synthetic delay-copy tasks that require copying tokens after finite delays, the authors evaluate the impact of positional encoding schemes (RoPE, ALiBi, and NoPE), the diversity of distances seen during training, and the conditions under which distance transfer learning is beneficial or detrimental. Their comprehensive study highlights the importance of understanding the underlying mechanisms that govern distance generalization in transformers.

By Daniel Henrik Nevermann, Claudius Gros
llmsfine-tuning
More like this →
arXiv Machine Learning
Jun 25

Why Do Accumulated Transformations Extrapolate?

arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.

By Mahesh Godavarti
More like this →
arXiv AI
Oct 7

Recurrent Looped Transformer

The Recurrent Looped Transformer (RLT) divides its layers into a parallel causal encoder and a recurrent decoder, allowing the computation path to grow with sequence length while keeping a fixed per-token cost. Experiments on six algorithmic tasks show that RLT outperforms a standard eight‑layer Transformer, achieving perfect generalization on parity up to 256 bits and high accuracy on swap‑based permutation tracking and modular arithmetic beyond training lengths. Ablation studies confirm that the recurrent feedback is essential for these gains, with chunked updates preserving performance on parity but reducing accuracy on permutation tracking.

By Yifan Zhang, Jichen Feng, Shihan Qin
llmsrag
More like this →