arXiv Machine Learning August 18, 2026 By Pratikkumar Babariya

A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

Read the original on arXiv Machine Learning →

arXiv:2608. 14689v1 Announce Type: new Abstract: Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jul 22

A Controlled Study of Attention-Only Transformers

arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.

By Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee

llms rag

arXiv AI

2d ago

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

arXiv:2610.00423v1 Announce Type: cross Abstract: Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transfor...

By S. Aaron McClendon, Jorge Gallego-Feliciano, Antonios Saravanos

llms

arXiv Machine Learning

Jul 1

Review Residuals: Update-Conditioned Residual Gating for Transformers

arXiv:2606. 31859v1 Announce Type: new Abstract: Residual connections add every sublayer's proposed update with a fixed coefficient of one; the network never evaluates whether an update is reliable before committing it.

By Kyle Kramer

llms

arXiv AI

Jul 3

Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits

arXiv:2607. 01940v1 Announce Type: cross Abstract: Mechanistic interpretability often relies on component-level interventions to discover how a model produces a behavior.

By Zhiren Gong, Zihao Zeng, Chau Yuen, Wei Yang Bryan Lim

llms efficiency benchmarks safety

arXiv Computation and Language

Sep 10

Revisiting the Shape Convention of Transformer Language Models

arXiv:2602.06471v2 Announce Type: replace Abstract: The architectural shape of dense Transformers has remained remarkably stable: narrow-wide-narrow feed-forward networks (FFNs) consume most non-embe...

By Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin, Meng-Hsi Chen, Da-shan Shiu

llms rag

arXiv AI

Jun 16

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention

arXiv:2510. 04212v4 Announce Type: replace-cross Abstract: The pursuit of computational efficiency has driven the adoption of low-precision formats for training transformer models.

By Haiquan Qiu, Quanming Yao

llms efficiency safety

One email a morning, machine-written

Related stories

A Controlled Study of Attention-Only Transformers

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation

Review Residuals: Update-Conditioned Residual Gating for Transformers

Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits

Revisiting the Shape Convention of Transformer Language Models

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention