The paper introduces a new language modeling approach that combines the benefits of Transformers and recurrent models by using balanced-tree recursive operators for autoregressive prediction. This method achieves logarithmic depth and linear runtime, allowing all prefix representations to be computed efficiently. Experiments show strong length extrapolation and performance close to ALiBi-based Transformers, suggesting it could serve as a viable alternative architecture for language modeling.
Language modeling using Transformers has become commonplace despite their fixed computational depth and quadratic runtime with respect to input tokens. Recurrent models on the other hand offer linear...
arXiv:2609.27233v1 Announce Type: new Abstract: Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adja...
Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adjacent token spans are highly predictable or freque...
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
The paper introduces a training‑free structured suffix modeling technique to accelerate Diffusion Language Models (DLMs). It partitions the suffix into local, middle, and tail regions, retaining varying numbers of tokens per region and incorporating previous decoding results into current token representations. Experiments on three DLMs show significant speedups—up to 72.81× in long‑sequence inference—while often improving performance, and the method is compatible with existing acceleration strategies.
The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.
arXiv:2608. 15062v1 Announce Type: cross Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency.
Diffusion Language Models (DLMs) exhibit strong parallel decoding capabilities by denoising multiple tokens in a single generation step. However, this parallelism comes with substantial computational...
arXiv:2608.15062v3 Announce Type: replace-cross Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layer...
arXiv:2605. 06384v3 Announce Type: replace-cross Abstract: We introduce MinMax Recurrent Neural Cascades (MinMax RNCs), a class of recurrent neural networks built from a novel form of recurrence over the MinMax algebra.
arXiv:2605. 08696v4 Announce Type: replace-cross Abstract: Over the last two decades, language modeling has experienced a shift from the use of predominantly recurrent architectures that process tokens sequentially during training and inference to non-recurrent models that process sequence elements in parallel during training, which results in greater training efficiency and stability at the expense of lower inference throughput.
arXiv:2602.00612v3 Announce Type: replace Abstract: Diffusion Large Language Models (dLLMs) have demonstrated promising generative capabilities and are increasingly used to produce formal languages d...