← Back to all news
arXiv AI By Mayank Mishra, Shawn Tan, Ion Stoica, Joseph E. Gonzalez, Tri Dao

M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

Read the original on arXiv AI →

M$^2$RNN introduces matrix‑valued hidden states and expressive non‑linear transitions for recurrent neural networks, addressing the limited expressive power of transformers. The architecture scales efficiently with tensor cores, achieving perfect state‑tracking generalization on unseen sequence lengths and outperforming hybrid Gated DeltaNet models by 0.4–0.5 perplexity points on a 7B MoE model while using three‑times smaller recurrent states. Even a single M$^2$RNN layer in a hybrid architecture yields accuracy gains comparable to full Hybrid M$^2$RNN, and improves long‑context performance on LongBench by up to 8 points over state‑of‑the‑art hybrid linear attention models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Jun 17

Olmo Hybrid: From Theory to Practice and Back

arXiv:2604. 03444v4 Announce Type: replace Abstract: Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention.

By William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, Chuan Li, Kyle Lo, Saumya Malik, DJ Matusz, Benjamin Minixhofer, Jacob Morrison, Luca Soldaini, Finbarr Timbers, Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi, Ashish Sabharwal
llms
More like this →
arXiv AI
Jun 4

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training

arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.

By Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Ag\"uera y Arcas, Jo\~ao Sacramento
llmsbenchmarks
More like this →
arXiv Machine Learning
Oct 23

baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity

The paper introduces baseVultra, a compact recurrent neural network cell that reallocates most of its recurrent capacity to a single dense candidate‑generation matrix while simplifying gate operations to O(H) diagonal interactions. On character‑level language modeling benchmarks such as TinyShakespeare and text8, baseVultra matches or slightly outperforms parameter‑matched LSTM and GRU cells, achieving lower validation loss and competitive bits‑per‑character scores. The authors find that the advantages of this candidate‑focused design are most pronounced in low‑capacity regimes, whereas full‑gate cells remain competitive when hidden size is increased.

By Linke Zhang
benchmarks
More like this →