arXiv Machine Learning July 30, 2026 By Suvadeep Hajra

Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training

Read the original on arXiv Machine Learning →

arXiv:2505. 15548v2 Announce Type: replace Abstract: Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv Computation and Language

Sep 16

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

The paper proposes a lightweight recurrent memory module inserted between the lower and upper halves of a 6‑layer decoder‑only transformer. This module, which uses cross‑attention to observe hidden states, a GRU to update a persistent state, and gated addition to modulate subsequent layers, adds only 3.7% more parameters. It reduces evaluation loss by 28.5% and narrows the generalization gap, with ablations showing the benefit comes solely from the memory topology rather than auxiliary losses.

By Eduardo Novaes Hering

llms

arXiv Computation and Language

Sep 14

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

arXiv:2609.13141v1 Announce Type: new Abstract: Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context u...

By Zhiwei Li, Lei Zhu, Hao Gu, Xiang Hu, Yan Wang, Haitao Mi, Sirui Han, Leo Liang, Zhijiang Guo

llms agents

arXiv AI

Jun 29

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

arXiv:2606. 27748v1 Announce Type: cross Abstract: Transformer models rely on attention mechanism to capture long-range dependencies but suffer from quadratic complexity, limiting their scalability to long sequences.

By Haoran Zhang, Feng Zhou

llms

arXiv Computation and Language

Sep 22

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

arXiv:2601.06787v2 Announce Type: replace Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...

By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Taesup Kim

llms efficiency

Hugging Face Trending Papers

Jul 9

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone can keep improving by allocating more computation to each token while leaving the Transformer backbone fixed.

llms rag

One email a morning, machine-written

Related stories

Dual Dimensionality for Local and Global Attention

Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

Flexformer: Flexible Linear Transformer with Learnable Attention Kernel

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models