arXiv Machine Learning June 5, 2026 By Akarsh Kumar, Phillip Isola

Pretraining Recurrent Networks without Recurrence

Read the original on arXiv Machine Learning →

arXiv:2606. 06479v1 Announce Type: new Abstract: Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jun 4

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

By Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Ag\"uera y Arcas, Jo\~ao Sacramento

llms benchmarks

arXiv Machine Learning

Aug 5

Maglev: Sliding Recurrent Memory

arXiv:2608. 02870v1 Announce Type: new Abstract: We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training.

By Bo Liu, Qiang Liu

llms benchmarks

arXiv AI

Jul 9

Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization

arXiv:2604. 01577v3 Announce Type: replace-cross Abstract: We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory.

By Shota Takashiro, Masanori Koyama, Takeru Miyato, Yusuke Iwasawa, Yutaka Matsuo, Kohei Hayashi

llms reinforcement-learning safety

arXiv AI

Jun 9

MinMax Recurrent Neural Cascades

arXiv:2605. 06384v3 Announce Type: replace-cross Abstract: We introduce MinMax Recurrent Neural Cascades (MinMax RNCs), a class of recurrent neural networks built from a novel form of recurrence over the MinMax algebra.

By Alessandro Ronca

arXiv AI

Jun 16

Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks

arXiv:2410. 11687v3 Announce Type: replace-cross Abstract: Linear recurrent networks (LRNNs) offer linear-time sequence modeling, but standard recurrent updates do not directly expose the supervised products needed for in-context gradient descent.

By Yudou Tian, Neeraj Mohan Sushma, Harshvardhan Mestha, Nicolo Colombo, David Kappel, Anand Subramoney

safety

arXiv Machine Learning

Jun 16

Next-Latent Prediction Transformers Learn Compact World Models

arXiv:2511. 05963v4 Announce Type: replace Abstract: Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens.

By Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford

llms diffusion benchmarks safety

Related stories

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

Maglev: Sliding Recurrent Memory

Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization

MinMax Recurrent Neural Cascades

Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks

Next-Latent Prediction Transformers Learn Compact World Models