arXiv:2608. 02032v1 Announce Type: new Abstract: Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures.
Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.
Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures. Transformers rely on token-level attention memories, while recurrent models such as state space models (SSMs) and linear attention maintain compact recurrent states.
arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.
arXiv:2503. 18970v4 Announce Type: replace Abstract: Structured State Space Models (SSMs) have become a prominent class of sequence models, developed against two long-standing difficulties: the sequential computation and gradient propagation limits of Recurrent Neural Networks (RNNs), and the quadratic time and memory cost of self-attention in Transformers.
arXiv:2608. 16844v1 Announce Type: cross Abstract: The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state.
arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.
arXiv:2608. 12435v1 Announce Type: new Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.