M$^2$RNN introduces matrix‑valued hidden states and expressive non‑linear transitions for recurrent neural networks, addressing the limited expressive power of transformers. The architecture scales efficiently with tensor cores, achieving perfect state‑tracking generalization on unseen sequence lengths and outperforming hybrid Gated DeltaNet models by 0.4–0.5 perplexity points on a 7B MoE model while using three‑times smaller recurrent states. Even a single M$^2$RNN layer in a hybrid architecture yields accuracy gains comparable to full Hybrid M$^2$RNN, and improves long‑context performance on LongBench by up to 8 points over state‑of‑the‑art hybrid linear attention models.
arXiv:2604. 03444v4 Announce Type: replace Abstract: Recent work has demonstrated the potential of non-transformer language models, especially linear recurrent neural networks (RNNs) and hybrid models that mix recurrence and attention.
arXiv:2506. 05233v2 Announce Type: replace-cross Abstract: Sequence modeling is currently dominated by causal transformer architectures that use softmax self-attention.
arXiv:2609.36529v1 Announce Type: new Abstract: Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memo...
The paper introduces baseVultra, a compact recurrent neural network cell that reallocates most of its recurrent capacity to a single dense candidate‑generation matrix while simplifying gate operations to O(H) diagonal interactions. On character‑level language modeling benchmarks such as TinyShakespeare and text8, baseVultra matches or slightly outperforms parameter‑matched LSTM and GRU cells, achieving lower validation loss and competitive bits‑per‑character scores. The authors find that the advantages of this candidate‑focused design are most pronounced in low‑capacity regimes, whereas full‑gate cells remain competitive when hidden size is increased.
arXiv:2606. 16093v1 Announce Type: cross Abstract: Modeling long-range dependencies remains a central challenge in natural language processing.
Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.
arXiv:2610.15715v1 Announce Type: new Abstract: Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ong...
arXiv:2602. 14814v3 Announce Type: replace Abstract: Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear).
arXiv:2605. 06384v3 Announce Type: replace-cross Abstract: We introduce MinMax Recurrent Neural Cascades (MinMax RNCs), a class of recurrent neural networks built from a novel form of recurrence over the MinMax algebra.
arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...
arXiv:2605. 08696v4 Announce Type: replace-cross Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput.
arXiv:2602. 03681v2 Announce Type: replace-cross Abstract: The quadratic computational complexity of softmax transformers has become a bottleneck in long-context scenarios.