arXiv AI July 15, 2026 By Johannes Knittel, Hanspeter Pfister

Sparse Inter-Layer Dependencies of Transformer FFN Neurons

arXiv:2607. 11990v1 Announce Type: cross Abstract: Feedforward network (FFN) blocks account for a large fraction of the parameters and computation in Transformer architectures, yet their internal structure remains difficult to interpret due to the additive superposition induced by the residual stream.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Jun 8

Sparsely gated tiny linear experts

arXiv:2606. 07414v1 Announce Type: new Abstract: Sparsity allows scaling model parameters without proportionally increasing computational cost.

By Simon Schug

llms efficiency safety

arXiv Machine Learning

Sep 1

Tracing distinguishability through transformer processing with stochastic LayerNorm

arXiv:2608.30720v1 Announce Type: new Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied...

By Kieran Murphy

llms fine-tuning efficiency safety

arXiv AI

Jul 24

Scaling Interpretable Transformers with Parity Bottleneck Layers

arXiv:2607. 20652v1 Announce Type: cross Abstract: Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams.

By Andrew Mack, Kraig Yuheng Tou, Mark Henry, Zhengxun Wu, Lauren Greenspan

llms efficiency safety

arXiv AI

Jul 7

Individual Parameters in Weight-Sparse Transformers Appear Interpretable

arXiv:2607. 02964v1 Announce Type: cross Abstract: A central goal of mechanistic interpretability is to understand how neural networks work and what each individual component does.

By Arnau Marin-Llobet, Stefan Heimersheim

llms safety

arXiv Machine Learning

Jul 31

Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

arXiv:2607. 27591v1 Announce Type: new Abstract: Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification.

By Jinyi Liu, Wei Chen, Pengyu Chen, Xinyi Yuan, Minghe Bai, Guoquan Wu, Jun Wei

llms efficiency

arXiv AI

Jun 26

CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

arXiv:2606. 26538v1 Announce Type: cross Abstract: Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value.

By Huzama Ahmad, Cao Viet Hai Nam, Se-Young Yun

llms efficiency

One email a morning, machine-written

Related stories

Sparsely gated tiny linear experts

Tracing distinguishability through transformer processing with stochastic LayerNorm

Scaling Interpretable Transformers with Parity Bottleneck Layers

Individual Parameters in Weight-Sparse Transformers Appear Interpretable

Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry