← Back to all news
arXiv Machine Learning By Linke Zhang

baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity

Read the original on arXiv Machine Learning →

The paper introduces baseVultra, a compact recurrent neural network cell that reallocates most of its recurrent capacity to a single dense candidate‑generation matrix while simplifying gate operations to O(H) diagonal interactions. On character‑level language modeling benchmarks such as TinyShakespeare and text8, baseVultra matches or slightly outperforms parameter‑matched LSTM and GRU cells, achieving lower validation loss and competitive bits‑per‑character scores. The authors find that the advantages of this candidate‑focused design are most pronounced in low‑capacity regimes, whereas full‑gate cells remain competitive when hidden size is increased.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 19

M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

M$^2$RNN introduces matrix‑valued hidden states and expressive non‑linear transitions for recurrent neural networks, addressing the limited expressive power of transformers. The architecture scales efficiently with tensor cores, achieving perfect state‑tracking generalization on unseen sequence lengths and outperforming hybrid Gated DeltaNet models by 0.4–0.5 perplexity points on a 7B MoE model while using three‑times smaller recurrent states. Even a single M$^2$RNN layer in a hybrid architecture yields accuracy gains comparable to full Hybrid M$^2$RNN, and improves long‑context performance on LongBench by up to 8 points over state‑of‑the‑art hybrid linear attention models.

By Mayank Mishra, Shawn Tan, Ion Stoica, Joseph E. Gonzalez, Tri Dao
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Aug 27

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.

By Amr Hegazy, Amr Alanwar, Mostafa Elhoushi
llmsefficiency
More like this →