Hugging Face Trending Papers August 12, 2026

Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

Read the original on Hugging Face Trending Papers →

We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs.

Summary generated by The Flow from the publisher's feed. The full article lives at Hugging Face Trending Papers.

llms

arXiv AI

Jun 9

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

arXiv:2603. 22473v2 Announce Type: replace-cross Abstract: Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each component type remains insufficiently characterized.

By Hector Borobia, Elies Segu\'i-Mas, Guillermina Tormo-Carb\'o

llms benchmarks

arXiv Machine Learning

Jun 2

Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization

arXiv:2606. 02288v1 Announce Type: new Abstract: Massive activation spikes in Large Language Models (LLMs) severely degrade quantization by stretching dynamic ranges.

By Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou, Naveen Verma

llms rag efficiency benchmarks safety

arXiv AI

Jun 2

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures

arXiv:2606. 02378v1 Announce Type: cross Abstract: We track the developmental trajectory of attention-head circuit formation across three 1B-class language models spanning two architecture families (dense transformer, mixture-of-experts) and two pretraining corpora (The Pile, DCLM): Pythia 1B, OLMo 1B-0724-hf, and OLMoE 1B-7B-0924.

By Yongzhong Xu

llms safety

arXiv Machine Learning

Jun 15

Beyond a Single Explanation of the Adam--SGD Gap

arXiv:2606. 14259v1 Announce Type: new Abstract: Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties.

By Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni, Jun Pang, Aurelien Lucchi, Antonio Orvieto

arXiv AI

Jul 3

The Wiola Architecture for Efficient Small Language Models

arXiv:2607. 01394v1 Announce Type: new Abstract: We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon.

By Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi, Brahma Kumar

llms

arXiv Machine Learning

Jun 2

Task Structure Reverses Layerwise State Encoding in Sequence Models

arXiv:2606. 00926v1 Announce Type: new Abstract: Mechanistic studies of sequence models often treat layerwise state encodings as architectural traits: recurrent models concentrate readable state, attention-based models distribute it.

By Yuhang Jiang

llms fine-tuning benchmarks

Related stories

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures

Beyond a Single Explanation of the Adam--SGD Gap

The Wiola Architecture for Efficient Small Language Models

Task Structure Reverses Layerwise State Encoding in Sequence Models