arXiv Machine Learning July 30, 2026 By Suvadeep Hajra

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

Read the original on arXiv Machine Learning →

arXiv:2505. 15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv AI

Jun 29

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

arXiv:2606. 27748v1 Announce Type: cross Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences.

By Haoran Zhang, Feng Zhou

llms

Hugging Face Trending Papers

Jul 9

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.

llms rag

arXiv AI

Jun 9

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

arXiv:2606. 07604v1 Announce Type: cross Abstract: Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs).

By Harry Jake Cunningham, Nicola Muca Cirone

llms safety

arXiv AI

1d ago

Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models

arXiv:2608. 14604v1 Announce Type: cross Abstract: Small language models in the ten to one hundred million parameter range are attractive for on device inference, rapid experimentation, and controlled scientific study, yet most of them reuse the standard transformer block without adaptation to the small scale regime.

By Aryuemaan Kumar Chowdhury, Praveen Oosa, Vineesha Reddy

llms

arXiv AI

2d ago

BCMT: Blockwise Causal Memory Transformer

arXiv:2608. 13578v1 Announce Type: cross Abstract: Transformer architectures rely on dense self-attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length.

By Rachid Arezki

llms efficiency benchmarks

Related stories

Dual Dimensionality for Local and Global Attention

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

Wiola 13M, a Gated Spiral Attention Architecture for Parameter Efficient Small Language Models

BCMT: Blockwise Causal Memory Transformer