arXiv Machine Learning By Zeyi Huang, Xuehai He, Yong Jae Lee, Yelong Shen

Trading Depth for Time in Recurrent Transformers

Read the original on arXiv Machine Learning →

The paper investigates whether extra computation in recurrent Transformers should be allocated to more temporal steps or greater physical depth. Using Latent Recurrent Transformers (LRTs), the authors insert a latent thought token between vocabulary tokens, allowing each token to pass through the same $L$ layers twice while sharing parameters. Experiments on 16‑ and 20‑layer mixture‑of‑experts NanoChat backbones show that a single thought token brings a shallower model within 0.006–0.004 bits per byte of a double‑depth counterpart, recovering 67–81% of the improvement with roughly 48% fewer parameters.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Nov 6

Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves

The paper introduces Dreamer, a depth‑recurrent attention mixture framework that combines sequence attention, depth attention, and sparse expert attention to overcome the hidden‑size bottleneck in latent reasoning models. Dreamer decouples scaling dimensions, allowing efficient depth‑recurrent scaling, and achieves state‑of‑the‑art performance on language reasoning benchmarks while using 2 to 8× fewer training tokens than FLOP‑, parameter‑, and memory‑matched baselines. It also demonstrates significantly greater expert selection diversity, up to 11× larger than current MoE models.

By Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting