arXiv Machine Learning October 12, 2026 By Shutian Liang, Yue Song

Understanding Mamba-3 Through Oscillatory Dynamics

Read the original on arXiv Machine Learning →

Mamba-3 incorporates input‑dependent rotations into its recurrent state updates, which the authors interpret as underdamped second‑order dynamics. Experiments show that these oscillatory dynamics outperform non‑oscillatory regimes on state‑tracking tasks and improve long‑context prediction and length generalization in language‑model pretraining. Further interventions confirm that the learned phase is crucial, suggesting that rotation controls how retained information combines with new updates, thereby supporting long‑context processing.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Oct 16

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.

By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane

llms

arXiv Machine Learning

Jun 5

Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies

arXiv:2602. 01196v2 Announce Type: replace Abstract: Recurrent neural policies are widely used in partially observable control and meta-RL tasks.

By Jin Li, Yue Wu, Mengsha Huang, Yuhao Sun, Hao He, Xianyuan Zhan

reinforcement-learning

arXiv AI

Jul 9

Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization

arXiv:2604. 01577v3 Announce Type: replace-cross Abstract: We study out of distribution generalization in streaming tasks where models are trained on short sequences but must operate over much longer, unknown horizons under bounded memory.

By Shota Takashiro, Masanori Koyama, Takeru Miyato, Yusuke Iwasawa, Yutaka Matsuo, Kohei Hayashi

llms reinforcement-learning safety

arXiv AI

Sep 16

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

arXiv:2609.16540v1 Announce Type: cross Abstract: State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear comput...

By William L. Tong, Aryo Lotfi, Emmanuel Abbe, Kostas Vaggelakos, Vishnu Banna, Etai Littwin, Josh Susskind, Cengiz Pehlevan, Eran Malach

llms

arXiv Machine Learning

Sep 10

Learning Length-Extrapolatable Recurrent Models

arXiv:2609.09157v1 Announce Type: new Abstract: Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their t...

By Hanwen Jiang

arXiv AI

Jun 2

SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition

arXiv:2606. 00732v1 Announce Type: new Abstract: Learning long-range non-stationary temporal patterns remains a core challenge for modern sequence models, particularly in strict streaming settings.

By Jayanta Dey, Shikhar Srivastava, Itamar Lerner, Christopher Kanan, Dhireesha Kudithipudi

llms benchmarks

One email a morning, machine-written

Related stories

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies

Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization

On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models

Learning Length-Extrapolatable Recurrent Models

SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition