CBAM Paper Walkthrough: The Double-Attention Mechanism

Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?

The paper investigates whether recent attention‑mechanism improvements—specifically gated attention, Kimi K3, Kimi Delta Attention, and Attention Residuals—effectively eliminate the attention‑sink problem when scaling language models to a one‑million‑token context window. Using a new diagnostic suite called SinkProbe, the authors evaluate sink mass, massive activation, position‑resolved recall, and the recency gap across four small models that vary only in token mixing and depth. Their findings show that the training objective, rather than the architecture, drives the emergence of attention sinks; gating did not replicate its previously reported benefits at the larger scale, and sink mass, activations, and positional bias behaved independently.

BCMT: Blockwise Causal Memory Transformer

arXiv:2608. 13578v1 Announce Type: cross Abstract: Transformer architectures rely on dense self-attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length.

Profiling in PyTorch (Part 3): Attention is all you profile

Accelerating Attention with Basis Decomposition

arXiv:2510. 01718v2 Announce Type: replace Abstract: Attention is a core operation in large language models (LLMs).

Linearized 2-Simplicial Attention

arXiv:2608. 09307v1 Announce Type: new Abstract: We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over one token axis takes the same form as ordinary softmax attention.

What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

arXiv:2603. 06591v2 Announce Type: replace Abstract: Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks.